Commit Graph

23 Commits

Author SHA1 Message Date
Derek Collison
1f00d0e3f2 Track deleted with single avl.SeqSet dmap for now vs old method.
Size of encoding may be a bit bigger then we wanted, but still way better then old method and very fast.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-05 12:32:29 -07:00
Neil Twigg
4d8d01949b Add first_seq support to memory store too
Signed-off-by: Neil Twigg <neil@nats.io>
2023-07-25 16:08:13 +01:00
Derek Collison
daacbf5580 Added optimized store NumPending() call.
Optimized and fixed a bug in filestore filteredPending().
Optimized memstore FilteredState().

Added comprehensive tests for NumPending() and FilteredState().

Signed-off-by: Derek Collison <derek@nats.io>
2023-02-25 17:26:26 -08:00
Derek Collison
24c2f3b452 Improved performance of subjects details for stream info.
This version avoids all disk IO in the filestore version.

Signed-off-by: Derek Collison <derek@nats.io>
2023-02-24 17:22:18 -08:00
Derek Collison
0da2a150cc Make sure we adjust per subject info when doing a Compact().
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-10 07:21:02 +02:00
Derek Collison
9c02be2409 Various fixes for snapshots.
Due to bug, in rare circumstances could write an empty snapshot for aplied == 0. This would cause a spinning at the raft layer.

1. Allow Truncate() to also properly do a reset of the store when terms were only mismatch.
2. During testing fixed memstore truncate and also made sure per subject info was also cleaned up.
3. Then added fix to detect a bad snapshot on initialization and remove.
4. Do not allow snapshots for applied == 0.

Signed-off-by: Derek Collison <derek@nats.io>
2023-02-04 13:46:06 -08:00
Derek Collison
1fa5e73177 Honor MaxMsgsPerSubject when a stream config is updated, including enforcing a lower limit.
Signed-off-by: Derek Collison <derek@nats.io>
2022-10-31 17:25:20 -07:00
Derek Collison
4291433a46 General improvements to accounting for the filestore. This in response to tracking issue #3114.
Signed-off-by: Derek Collison <derek@nats.io>
2022-05-12 15:43:11 -07:00
Derek Collison
ef8f543ea5 Improve memory usage through JetStream storage layer.
Previously we would rely more heavily on Go's garbage collector since when we loaded a block for an underlying stream we would pass references upward to avoimd copies.
Now we always copy when passing back to the upper layers which allows us to not only expire our cache blocks but pool and reuse them.

The upper layers also had changes made to allow the pooling layer at that level to interoperate with the storage layer optionally.

Also fixed some flappers and a bug where de-dupe might not be reformed correctly.

Signed-off-by: Derek Collison <derek@nats.io>
2022-03-24 17:45:15 -06:00
Derek Collison
f0cdf89c61 JetStream Clustering WIP
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 01:14:52 -08:00
Derek Collison
7564768027 Added Compact to store interface for WAL functionality
Signed-off-by: Derek Collison <derek@nats.io>
2020-12-03 16:18:58 -08:00
Derek Collison
0d2ca9ba54 Fix race, can't clear direct memory since shared
Signed-off-by: Derek Collison <derek@nats.io>
2020-06-02 06:19:04 -07:00
Derek Collison
eca04c6fce First pass header support for JetStream
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-30 10:04:23 -07:00
R.I.Pienaar
fb37c121d1 expose the arrival timestamp in the message metadata
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:21:27 -07:00
Derek Collison
323b3d0b11 Beginning basics for stream and consumer snapshotting
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
47c28b2fb0 JetStream major refactor for name changes.
MsgSet -> Stream
Observable -> Consumer

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
613d65c58c Erased msg recovery, bug fix on linux for idx file, LoadMsg vs Lookup
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:12:28 -07:00
Derek Collison
e8f465eaf3 Filestore read perf, secure delete for stores
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:12:28 -07:00
Derek Collison
39aba7de92 filestore first pass
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:12:28 -07:00
Derek Collison
da7838368d Tracking usage and limits
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:07:02 -07:00
Ivan Kozlovic
9214fa2995 Update expiration test
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-05-19 14:06:29 -07:00
Ivan Kozlovic
09c1acc36c Fix message expiration
Also add missing `package` for sysmem/windows

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-05-19 14:06:29 -07:00
Derek Collison
dd116fcfd4 JetStream first pass basics.
This is the first checkin for JetStream. Has some rudimentary basics working.

TODO
1. Push vs pull mode for observables. (work queues)
2. Disk/File store, memory only for now.
3. clustering code - design shaping up well.
4. Finalize account import semantics.
5. Lots of other little things.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:06:29 -07:00