During normal operation and quick restarts the number of expired messages per cycle is manageable and correct.
However if a server is shutdown for quite a long time and many messages have expired this process is too slow.
This commit introduces an optimized expiration tailored for startup vs running state.
Signed-off-by: Derek Collison <derek@nats.io>
When we created a filestore we would figure out if we should track by subject based on stream config.
This would cause bad results when a stream was updated to multiple subjects or wildcards.
This change tightens when and what we track but turns it on all the time now.
Signed-off-by: Derek Collison <derek@nats.io>
1. We were holding open FDs longer than we should for consumers causing issues with open FD limits. We now do not hold them open and cap updates a bit better.
2. When doing a stream delete, consumer delete was repeating alot of work that was not necessary, causing longer delays. This has been optimized a bit, still more improvements to be made.
3. We cover all JS under a single export, but that was also trapping GetNext for pull based consumers, and since this was a no-op (is handled at user account level) we were creating alot of garbage service import responses and reverse map entries that had to be garbage collected. We have a fix in to avoind this but still looking for a better one.
4. Still had some lingering references to all exports vs single JS export.
Signed-off-by: Derek Collison <derek@nats.io>
We optimized the filtered purge to skip msgBlks that are not in play.
Also optimized msgBlock buffer usage by using two sync.Pools to enhance reuse.
Signed-off-by: Derek Collison <derek@nats.io>
This supports XChaChaPoly1305 for Seal and Open and ChaCha20 for our message blocks which use highway hashes and sequence numbers for authenticity.
We support snapshot and restore as well.
Signed-off-by: Derek Collison <derek@nats.io>
Allow wider scoped filtered subjects.
We introduce a per subject information tracking to filestore to optimize for large mux'd streams and more efficient filtered consumers.
Signed-off-by: Derek Collison <derek@nats.io>
We had issues of instability and incorrect behavior during concurrent operations.
This CL optimizes expiring msgs to be more efficient and hold the lock until completion.
Compact will also now hold the top level lock through completion.
Signed-off-by: Derek Collison <derek@nats.io>
We now make sure to sync state of the replicas when a new leader takes over. We also update ack floors regardless of detection on pending list.
Signed-off-by: Derek Collison <derek@nats.io>
We were not properly enforcing server limits. This commit will allow a server to enforce limits but still remain functional even at the JetStream level.
Also fixed a bug for RAFT replay that could cause instability.
Signed-off-by: Derek Collison <derek@nats.io>
1. Release cache during heavy writes when we move to a new block if no read activity.
2. If we detect a linear scan reading dump the cache on the last seq read for the mb.
Signed-off-by: Derek Collison <derek@nats.io>
