This could result in lookups failing, e.g. after a change in max msgs per subject to a lower value.
Also fixed a bug that would not prperly update psim during compact when throwing away the whole block and a subject had more than one message.
Signed-off-by: Derek Collison <derek@nats.io>
- In some failure scenarios where the current leader of a stream sourcing from other stream(s) gets shutdown while publications are happening on the stream(s) being sourced leads to `setLeader(true)` being called on the new leader for the sourcing stream before all the messages having been sourced by the previous leader are completely processed such that when the new leader does it's reverse scan from the last message in it's view of the stream in order to know what sequence number to start the consumer for the stream being sourced from, such that the last message(s) sourced by the previous leader get sourced again, leading to some messages being sourced more than once.
The existing `TestNoRaceJetStreamSuperClusterSources` test would sidestep the issue by relying on the deduplication window in the sourcing stream. Without deduplication the test is a flapper.
This avoid the race condition by adding a small delay before scanning for the last message(s) having been sourced and starting the sources' consumer(s). Now the test (without using the deduplication window) never fails because more messages than expected have been received in the sourcing stream.
- Fix test TestJetStreamWorkQueueSourceRestart that expects the sourcing stream to get all of the expected messages right away by adding a small sleep before checking the number of messages pending on the consumer for that stream.
Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
Test didn't catch this error, as by default, old JS client
sets ack policy to none. If policy is different, it will fail
to create consumer with defaults.
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
- Use Go 1.21 in nightlies
- Both rand.Seed and rand.Read are both deprecated, remove its use to
fix staticcheck errors
```go
server/client.go:95:2: SA1019: rand.Seed has been deprecated since Go 1.20 and an alternative
has been available since Go 1.0: As of Go 1.20 there is no reason to call Seed with a random value.
Programs that call Seed with a known value to get a specific sequence of results should use
New(NewSource(seed)) to obtain a local random generator. (staticcheck)
server/jetstream_test.go:20399:2: SA1019: rand.Read has been deprecated since Go 1.20
because it shouldn't be used: For almost all use cases, crypto/rand.Read is more appropriate. (staticcheck)
```
This uses `TempDir` in more tests. Some were trying to manage the
directory manually, others were just not managing a directory at all.
There may be more that need doing.
Signed-off-by: Neil Twigg <neil@nats.io>
Historically we kept indexing information, either by sequence or by subject, as a per msg block operation. These were the "*.idx" and "*.fss" indexing files. When streams became very large this could have an impact on recovery time. Also, for encryption the fast path for determining if the indexing was current would require loading and decrypting the complete block.
This design moves to a more traditional WAL and snapshot approach. The snapshots for the complete stream, including sumary information, global per subject information maps (PSIM) and per msg block details including summary and dmap, are processed asynchronously. The snapshot includes the msh block and has for the last record considered in the snapshot. On recovery the snapshot is read and processed and any additional records past the point of the snapshot itself are processed. To this end, any removal of a message has to be expressed as a delete tombstone that is always added the the fs.lmb file. These are processed on recovery and our indexing layer knows to skip them.
Changing to this method drastically improves startup and recovery times, and has simplified the code. Some normal performance benefits have been seen as well.
Signed-off-by: Derek Collison <derek@nats.io>
We violated the locking pattern, so we now make sure we do this in a separate Go routine and put checks to only run it once.
Signed-off-by: Derek Collison <derek@nats.io>
- [X] Tests added
- [X] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [X] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
- [X] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)
### Changes proposed in this pull request:
Adds support for multi-filter (and associated transform destinations) to
stream sources
---------
Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
As of Go 1.20, `math/rand.Read` is deprecated. In addition to that, it
also isn't recommended for use in combination with anything
cryptographic.
I haven't replaced all `math/rand` imports with `crypto/rand` imports
because there are still some legitimate uses for the `math/rand` package
in some places.
Signed-off-by: Neil Twigg <neil@nats.io>
As of Go 1.20, `math/rand.Read` is deprecated. In addition to that, it also
isn't recommended for use in combination with anything cryptographic.
I haven't replaced all `math/rand` with `crypto/rand` imports because there
are still some legitimate uses for the `math/rand` package in some places.
Signed-off-by: Neil Twigg <neil@nats.io>
