gogrlx/nats-server
1
0
Fork 0
mirror of https://github.com/gogrlx/nats-server.git synced 2026-04-15 10:40:41 -07:00
Code Issues Packages Projects Releases Wiki Activity
4,843 Commits 30 Branches 48 Tags
fc9af36df69617e3210cd57dd347dc3eb4308708
Commit Graph

58 Commits

Author SHA1 Message Date
Derek Collison
bf966d84f3 Number of initial pending was off by one when DeliverLastPerSubject was utilized on a consumer. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-08-04 14:20:52 -07:00
Derek Collison
9b0158daf9 Allow delivery policy of DeliverLastPerSubject, which is helpful for scoped watchers for K/V. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-07-28 12:49:02 -07:00
Derek Collison
99fed910f0 Improvements to large numbers of JetStream R1 consumers per stream. 
1. We were holding open FDs longer than we should for consumers causing issues with open FD limits. We now do not hold them open and cap updates a bit better.

2. When doing a stream delete, consumer delete was repeating alot of work that was not necessary, causing longer delays. This has been optimized a bit, still more improvements to be made.

3. We cover all JS under a single export, but that was also trapping GetNext for pull based consumers, and since this was a no-op (is handled at user account level) we were creating alot of garbage service import responses and reverse map entries that had to be garbage collected. We have a fix in to avoind this but still looking for a better one.

4. Still had some lingering references to all exports vs single JS export.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-06-29 05:45:55 -07:00
Derek Collison
c0e47966ab Added in Stream get last message by subject. 
This is to aid in K/V overlay for simple Get ops vs creating a full blown consumer.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-06-24 13:21:39 -07:00
Derek Collison
2e145196b8 Fix for extended purge by sequence. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-06-22 07:38:30 -07:00
Derek Collison
89d930fd0f Updates and fixes to PurgeEx 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-06-20 10:34:27 -07:00
Derek Collison
9398c3ca28 Allow for more advanced purge operations that filter by subject, specify the sequence or number of messages to keep. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-06-19 07:04:44 -07:00
Derek Collison
d0ac1a40ca Added in per subject limits for streams. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-06-15 06:36:34 -07:00
Derek Collison
08cdb2d2ea Make filtered consumers in large mixed streams more efficient. 
Allow wider scoped filtered subjects.

We introduce a per subject information tracking to filestore to optimize for large mux'd streams and more efficient filtered consumers.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-06-15 04:44:05 -07:00
Derek Collison
90989d57d6 Change to report total deleted by default for stream info. 
Allow deleted details if requested.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-04-12 18:10:11 -07:00
Derek Collison
1c368973c8 Merge pull request #2086 from alexpantyukhin/fix_max_bytes_check_memstore 
fix for memstore storeRawMsg be consistent with filestore
 2021-04-08 14:48:56 -07:00
alexpantyukhin
ce34266912 fix for memstore storeRawMsg be consistent with filestore 2021-04-08 23:54:10 +04:00
Derek Collison
c0e8590c0f During startup each filtered consumer could do a linear scan of the stream 
to determine number of messages pending. This improves that with a startup cache.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-04-07 09:15:01 -07:00
Derek Collison
8f671deb91 Calling out to the storage update should not hold lock. 
We had lock inversion with consumers.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-26 04:40:30 -07:00
Derek Collison
e53caee5e8 Enforce server limits even when dynamic limits for accounts in play. 
We were not properly enforcing server limits. This commit will allow a server to enforce limits but still remain functional even at the JetStream level.
Also fixed a bug for RAFT replay that could cause instability.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-25 16:06:27 -07:00
Derek Collison
d31fda5dac Added code to constrain size of WAL under most scenarios. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-06 08:38:56 -08:00
Derek Collison
b9e1a921ff Use internal wildcard inbox for stream and consumer info requests. 
More gateway friendly but suffers from no echo, so added new client based internal send.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-02-28 10:01:01 -08:00
Derek Collison
49c3729abd Detect closed state 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-02-11 13:37:16 -08:00
Derek Collison
c16f6e193d Move JetStream direct APIs to private. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-02-07 15:19:22 -08:00
Derek Collison
f0cdf89c61 JetStream Clustering WIP 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-01-14 01:14:52 -08:00
Derek Collison
7564768027 Added Compact to store interface for WAL functionality 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-12-03 16:18:58 -08:00
Derek Collison
28cb4e8c34 Fix bug when removing the same message from a stream. 
We would release locks and call into upper layers when removing a message. The upper layers may call back into the lower layers to get more information, such as the subject.
This fix has the storage updates optionally supply the subject for filtered consumers and fixes the bug of double deletes.

Signed-off-by: Derek Collison <derek@nats.io>
 2020-11-13 17:05:24 -08:00
Derek Collison
164f9fdf2b Updates to consumer store to support delta updates. 
In preparation for clustering we need to have the consumer filestore update state with deltas vs original design.

Signed-off-by: Derek Collison <derek@nats.io>
 2020-11-10 19:16:55 -08:00
Derek Collison
fe2b354414 Stability and performance updates. 
The original design had a shared filestore write buffer and individual message blocks had a read cache.
This presented some performance and stability issues when lots of reads and deletes were happening to a
message block that was also being written to actively.

This change eliminates the shared write buffer and uses the message block's cache as a write through as
well as read cache and handles partials correctly.

Signed-off-by: Derek Collison <derek@nats.io>
 2020-10-28 07:58:47 -07:00
Derek Collison
df4ee081a5 Track number of stream pending for each consumer. 
This will track the stream pending state for each consumer.
This code does account for filtered consumers.

Signed-off-by: Derek Collison <derek@nats.io>
 2020-10-27 19:30:42 -07:00
Derek Collison
ad247d1853 Add store.SkipMsg() and update no interest retention streams 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-10-22 19:35:28 -07:00
Derek Collison
3a61a0cef8 Fix for https://github.com/nats-io/jetstream/issues/297 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-08-17 10:35:07 -07:00
Derek Collison
95462cde57 Fix for memory expiration and age timer 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-06-18 14:02:11 -07:00
Derek Collison
0d2ca9ba54 Fix race, can't clear direct memory since shared 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-06-02 06:19:04 -07:00
Derek Collison
e584d4efee Merge pull request #1435 from nats-io/js-hdrs 
First pass header support for JetStream
 2020-05-31 06:01:01 -07:00
Derek Collison
eca04c6fce First pass header support for JetStream 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-30 10:04:23 -07:00
Derek Collison
e12907ffa6 Allow snapshots to optionally check all message checksums 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-29 09:57:33 -07:00
Derek Collison
fa59cff105 Add in snapshot and restore JSApi 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-27 20:01:30 -07:00
Derek Collison
8727315eb9 Updated snapshots, added restore, generic hashes 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-26 19:53:16 -07:00
Derek Collison
7b6352c991 Add timestamps to stream state for first and last, addresses #1396 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-19 14:33:06 -07:00
Derek Collison
7f41b5a6ae Added support for DiscardPolicy on streams 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-19 14:26:46 -07:00
R.I.Pienaar
fb37c121d1 expose the arrival timestamp in the message metadata 
Signed-off-by: R.I.Pienaar <rip@devco.net>
 2020-05-19 14:21:27 -07:00
Derek Collison
323b3d0b11 Beginning basics for stream and consumer snapshotting 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-19 14:18:34 -07:00
Derek Collison
5e9f487b61 Fixes for test and locking, bumped version 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-19 14:16:03 -07:00
Derek Collison
623fa5ccec Impl for GetSeqFromTime for FileStore was missing 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-19 14:16:03 -07:00
Derek Collison
c9d3bdd5ab Added Stream Update abilities for certain properties 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-19 14:16:03 -07:00
Derek Collison
47c28b2fb0 JetStream major refactor for name changes. 
MsgSet -> Stream
Observable -> Consumer

Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-19 14:16:03 -07:00
Derek Collison
2237915c97 Added in template support 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-19 14:16:03 -07:00
Derek Collison
f1a81454b5 Added in error for Delete and Stop for store 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-19 14:16:03 -07:00
Derek Collison
e3be6f45c4 Add in delete for observables for filestore impl 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-19 14:16:03 -07:00
R.I.Pienaar
e2a1b18de6 report observable config and observables count on message sets 2020-05-19 14:15:11 -07:00
Derek Collison
e127039622 More API impls, tests for DeleteMsg 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-19 14:13:22 -07:00
Derek Collison
68860b39c3 metadata and checksum storage 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-19 14:12:28 -07:00
Derek Collison
81704d285b Plugin filestore to jetstream tests 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-19 14:12:28 -07:00
Derek Collison
c3a5df135f Observable state storage 
Signed-off-by: Derek Collison <derek@nats.io>
 2020-05-19 14:12:28 -07:00