gogrlx/nats-server
1
0
Fork 0
mirror of https://github.com/gogrlx/nats-server.git synced 2026-04-14 18:20:42 -07:00
Code Issues Packages Projects Releases Wiki Activity
4,666 Commits 30 Branches 48 Tags
v2.3.1-ReloadOptions
Commit Graph

192 Commits

Author SHA1 Message Date
Derek Collison
99fed910f0 Improvements to large numbers of JetStream R1 consumers per stream. 
1. We were holding open FDs longer than we should for consumers causing issues with open FD limits. We now do not hold them open and cap updates a bit better.

2. When doing a stream delete, consumer delete was repeating alot of work that was not necessary, causing longer delays. This has been optimized a bit, still more improvements to be made.

3. We cover all JS under a single export, but that was also trapping GetNext for pull based consumers, and since this was a no-op (is handled at user account level) we were creating alot of garbage service import responses and reverse map entries that had to be garbage collected. We have a fix in to avoind this but still looking for a better one.

4. Still had some lingering references to all exports vs single JS export.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-06-29 05:45:55 -07:00
Derek Collison
08197de9e0 FIXED max consumers was not enforced when set on stream 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-06-25 11:45:36 -07:00
Derek Collison
08cdb2d2ea Make filtered consumers in large mixed streams more efficient. 
Allow wider scoped filtered subjects.

We introduce a per subject information tracking to filestore to optimize for large mux'd streams and more efficient filtered consumers.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-06-15 04:44:05 -07:00
Derek Collison
e238512285 Handling of rewrites for subjects to a globally routed subject was not properly handling c.pa.deliver or reply rewrite. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-06-09 15:34:47 -07:00
Derek Collison
6e17b7a303 Fix for #2213 
We do not want to report consumers that were created for the purpose of sources or mirrors.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-05-12 07:51:53 -07:00
Derek Collison
bbb94467fb Fix for failing test, we needed to account for consumers having filtered subjects configured but not really being filtered. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-05-10 17:01:31 -07:00
Derek Collison
50bc1867bd When dealing with filtered subjects do not assume the current state as others can. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-05-07 15:20:19 -07:00
Derek Collison
9a517194a1 Merge pull request #2191 from nats-io/raft_catchup_snap 
[FIXED] Raft groups could continually spin trying to catch up.
 2021-05-07 14:20:37 -07:00
Derek Collison
4500f7889e For WorkQueues that have direct we need to make sure they have received the message before deleting. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-05-07 14:13:36 -07:00
Derek Collison
bd9172fb61 With WorkQueue policy streams that are mirrored or sourced need to only delete if no regular consumers. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-05-07 13:14:56 -07:00
Derek Collison
70a2521f95 For interest or workqueue streams with ephemerals we need to not reduce replication to 1. 
We need the consumer state on the stream leader.
Also if we can't find the store yet for a consumer fallback to calculate needsAck.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-05-07 12:07:27 -07:00
R.I.Pienaar
b5f846a719 add domain in JS advisories 
Signed-off-by: R.I.Pienaar <rip@devco.net>
 2021-05-07 19:35:46 +02:00
Derek Collison
ceb4a86d57 Check for nil chan 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-05-07 08:36:45 -07:00
Derek Collison
35a60289d9 Fixed mirror/source streams from work queues. 
Fixed deadlock with no-ack consumers on interest retention streams.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-05-07 06:01:52 -07:00
Derek Collison
b5521053e6 Make sure to clean up ephemerals across a Gateway. 
Also report direct consumers in num consumers in stream info.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-04-21 08:47:55 -07:00
Derek Collison
0a3e54c88a Fix subscription leaks for mirrors and sources for streams as well as flow control subjects for consumers. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-04-20 12:13:07 -07:00
Derek Collison
f893345b40 Pull based consumers when deleted were not removing messages from an interest based stream. 
Fix for #2097.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-04-12 11:16:31 -07:00
Derek Collison
deb015ec73 Fix for #2083 to release ack pending when messages expire or hit max redeliveries. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-04-08 11:47:59 -07:00
Derek Collison
c0e8590c0f During startup each filtered consumer could do a linear scan of the stream 
to determine number of messages pending. This improves that with a startup cache.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-04-07 09:15:01 -07:00
Derek Collison
2009578e8d Fix data race 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-04-02 08:42:37 -07:00
Derek Collison
44ada49b16 During repeated server restarts or failures consumer state could drift between replicas. 
We now make sure to sync state of the replicas when a new leader takes over. We also update ack floors regardless of detection on pending list.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-04-02 08:20:29 -07:00
Derek Collison
c3ec8d9fb3 Agressive cache expiration was hindering performance when publishing and receiving from a stream concurrently. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-04-01 16:32:43 -07:00
Derek Collison
bb7a8a5f79 Introduced default max ack pending for ack explicit. 
Fixed a bug that would introduce performance degradation for durable consumers R>1.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-30 11:47:24 -07:00
Derek Collison
0f71c260fb Durable consumers with R>1 had performance challenges. 
This code changes the way we handle raft based proposals for consumers.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-26 12:53:49 -07:00
Derek Collison
2ed53035ed Reworked flow control for sources and mirrors. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-24 07:07:33 -07:00
Derek Collison
022c5b4ab2 Fix for bad behavior with flow control and multiple sources streams. 
Allow chaining of sources and mirrors with filtered consumers.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-22 11:13:51 -07:00
Derek Collison
0f548edcc6 Reduce sliding window for direct consumers and catchup stream windows. 
Remove another possible wire blocking operation in raft.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-21 09:24:27 -07:00
Derek Collison
0bcda04a9d Send both last consumer and stream sequence for idle hb 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-20 16:57:13 -07:00
Derek Collison
ced35e5b8c Reworked sources and mirrors on missed data. 
Add last delivered sequence to consumer idle heartbeats.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-20 14:03:46 -07:00
Derek Collison
1dde8aed5d Do not send idle heartbeats if not active 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-20 09:22:06 -07:00
Derek Collison
cfe2f448ad Fix for when gateways were dropped and we did not detect interest returning. 
Fix for leaked subscription when retrying the source consumers.
Better suppression of old messages from old direct consumers.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-20 08:44:18 -07:00
Derek Collison
4f2b701eb5 Unlock only on return 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-18 11:38:04 -07:00
Derek Collison
702e17d48f Direct consumer updates. 
1. Reduce maximum sliding window for large N->1.
2. Do not create any store for directs.
3. Relax hard requirement for interest at creation for directs.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-18 10:52:40 -07:00
Derek Collison
259cb07284 When a stream was busy but not for a filtered consumer with idle heartbeats they would be suppressed. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-18 07:46:29 -07:00
R.I.Pienaar
d6e63cb683 improve log line when account registration fails 
Signed-off-by: R.I.Pienaar <rip@devco.net>
 2021-03-18 12:19:23 +01:00
Derek Collison
74cc2b581a When a stream is a mirror or has sources we need to check the upstream streams for filter subject correctness. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-17 06:39:21 -07:00
Derek Collison
4d15658ec2 Merge pull request #1996 from nats-io/updates 
JSC Updates
 2021-03-14 17:17:08 -07:00
Derek Collison
0425056c33 Fix deadlock 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-14 05:38:25 -07:00
Derek Collison
10afedcc46 Check in initial pending if our filtered subject directly matches our stream config. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-13 19:59:21 -08:00
Ivan Kozlovic
5edb8b1105 Clear timer in case currently set 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2021-03-13 14:59:38 -07:00
Ivan Kozlovic
9e858ff81f Ephemeral cleanup across GWs 
Watch for interest loss across GWs so ephemeral consumers are removed
when there is no longer local and GW interest.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2021-03-13 14:11:34 -07:00
Derek Collison
c783bf1bed Tweak flow control and slow start 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-10 17:30:25 -05:00
Derek Collison
f95b6481d2 Make source and mirror consumers direct, meaning they are not mapped by the metaleader. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-10 07:12:48 -05:00
Waldemar Quevedo
86a64fbc46 Updates to JS consumer errors 
Signed-off-by: Waldemar Quevedo <wally@synadia.com>
 2021-03-09 09:46:28 -08:00
Waldemar Quevedo
5656ce5ace Merge pull request #1958 from nats-io/consumer-ack-pending-pull 
Prevent pull batch larger than max ack pending
 2021-03-09 07:50:01 -08:00
Ivan Kozlovic
4d1fc3a449 Merge pull request #1974 from nats-io/pae 
Memory cache for commits and WAL size constraints.
 2021-03-08 16:43:20 -07:00
Derek Collison
673543c180 Modified flow control for clustered mode. 
Set channels into and out of RAFT layers to block.

Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-08 12:58:57 -06:00
Waldemar Quevedo
3e61c2e5b1 Prevent pull batch size larger than max ack pending 
Signed-off-by: Waldemar Quevedo <wally@synadia.com>
 2021-03-08 08:55:47 -08:00
Waldemar Quevedo
278c0efc8a Fix check from next request msg 
Signed-off-by: Waldemar Quevedo <wally@synadia.com>
 2021-03-07 00:12:31 -08:00
Derek Collison
0b3c686430 Fixes for data races and some locking. 
Signed-off-by: Derek Collison <derek@nats.io>
 2021-03-05 17:19:51 -08:00