Commit Graph

202 Commits

Author SHA1 Message Date
Derek Collison
98f7d63b84 Fix for data race and corruption
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-12 06:14:19 -07:00
Derek Collison
da3c89efda Add domain to PubAck1
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-11 13:25:24 -07:00
R.I.Pienaar
a3bbb04748 Merge pull request #2409 from ripienaar/js_errors_redux
attempt to improve UX of the error system
2021-08-10 20:01:24 +02:00
Derek Collison
7dd399e355 Merge branch 'main' into max-per-default
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-10 07:33:58 -07:00
R.I.Pienaar
76ab1b8d17 attempt to improve UX of the error system
Previously we had a few confusing functions like NewT
and similar that were quite fragile to use due to minimal
validation and a panic in go stdlib string Replacer.

Now we generate helper methods for every string, these
are used to access errors, fill in templates and conditional
returns of error type using the new Unless() option

We now get compile time errors for some common mistakes
and have better IDE helpers for arguments etc

Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-08-10 16:08:28 +02:00
Derek Collison
633763a202 Fix for #2420
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-10 06:36:08 -07:00
Derek Collison
29536629eb Simplified flow control, avoid stalls due to msg loss
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-09 20:13:17 -07:00
Derek Collison
40c21934c5 Fix for #2416
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-06 14:58:25 -07:00
Derek Collison
154bc40718 Fix for reentrant read lock on a stream that once anyone else wanted the write lock would deadlock.
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-03 15:46:40 -07:00
Derek Collison
98970ac03d Add descriptions to JetStream streams and consumers. 2021-07-27 14:05:25 -07:00
Derek Collison
f13fa767c2 Remove the swapping of accounts during processing of service imports.
When processing service imports we would swap out the accounts during processing.
With the addition of internal subscriptions and internal clients publishing in JetStream we had an issue with the wrong account being used.
This was specific to delyaed pull subscribers trying to unsubscribe due to max of 1 while other JetStream API calls were running concurrently.
2021-07-26 07:57:10 -07:00
Derek Collison
960c45df81 Use of sync.Pool for filestore could cause msg corruption.
Signed-off-by: Derek Collison <derek@nats.io>
2021-07-06 08:41:01 -07:00
Derek Collison
9b73fae5bd We should not sit in place here, no measurable effect on latency in tests
Signed-off-by: Derek Collison <derek@nats.io>
2021-07-01 14:29:09 -07:00
Derek Collison
617f061084 Check not really needed here, so removed
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-30 15:08:56 -07:00
Derek Collison
99fed910f0 Improvements to large numbers of JetStream R1 consumers per stream.
1. We were holding open FDs longer than we should for consumers causing issues with open FD limits. We now do not hold them open and cap updates a bit better.

2. When doing a stream delete, consumer delete was repeating alot of work that was not necessary, causing longer delays. This has been optimized a bit, still more improvements to be made.

3. We cover all JS under a single export, but that was also trapping GetNext for pull based consumers, and since this was a no-op (is handled at user account level) we were creating alot of garbage service import responses and reverse map entries that had to be garbage collected. We have a fix in to avoind this but still looking for a better one.

4. Still had some lingering references to all exports vs single JS export.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-29 05:45:55 -07:00
Derek Collison
bb84ef7d91 Added ability to match based on last expected sequence per subject.
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-28 10:57:50 -07:00
R.I.Pienaar
0d71d35e43 do not log at Error level for some store failures
Some of these are quite generic errors that can happen a lot
in normal circumstances so no need to be too noisy about them

Fixes one missed old style Api Error

Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-06-28 10:18:16 +02:00
Derek Collison
545230bd9e FIXED Clustered streams can become broken with seq mismatch state on low level store failures.
We can account for high level failure properly but were not properly accounting for low level store failures.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-25 13:46:40 -07:00
Derek Collison
bf6335dff9 Add in ability to have encrypted JetStream filestores.
This supports XChaChaPoly1305 for Seal and Open and ChaCha20 for our message blocks which use highway hashes and sequence numbers for authenticity.
We support snapshot and restore as well.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-21 19:28:10 -07:00
Derek Collison
9398c3ca28 Allow for more advanced purge operations that filter by subject, specify the sequence or number of messages to keep.
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-19 07:04:44 -07:00
R.I.Pienaar
53ae604403 further tagged error confusion cleanups
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-06-18 21:35:26 +02:00
Derek Collison
e7fd6961d6 Fix data race condition
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-18 07:47:19 -07:00
Derek Collison
d0ac1a40ca Added in per subject limits for streams.
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-15 06:36:34 -07:00
Derek Collison
08cdb2d2ea Make filtered consumers in large mixed streams more efficient.
Allow wider scoped filtered subjects.

We introduce a per subject information tracking to filestore to optimize for large mux'd streams and more efficient filtered consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-15 04:44:05 -07:00
Derek Collison
326a377b3e When leader changes happened to streams that mirrored or sourced other streams we could continue to try to create consumers.
This could get excessive on a stream that has sourced many upstream origin streams and had several leader changes.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-10 19:47:46 -07:00
Derek Collison
e238512285 Handling of rewrites for subjects to a globally routed subject was not properly handling c.pa.deliver or reply rewrite.
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-09 15:34:47 -07:00
R.I.Pienaar
ee9d10f40b restore old error constants for backwards compat
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-05-26 08:04:50 +02:00
R.I.Pienaar
0d391b02eb richer api errors proposal
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-05-26 08:04:50 +02:00
Derek Collison
308355a2fd Fix for #2242.
When we had a duplicate detected in R>1 mode we set the skip sequence indicator but were not using that when dealing with underlying store.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-24 08:21:41 -07:00
Derek Collison
4bde91e9bd Fix a data race.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-20 07:31:55 -07:00
Derek Collison
a27b0dd673 Move default file and dir perms
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-19 14:46:07 -07:00
Derek Collison
df8835d2d0 Stupid spelling where I introduced this..
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-12 14:32:06 -07:00
Derek Collison
060f2ec127 Fix stream source lookup and add in optional External to StreamSourceInfo.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-12 12:32:56 -07:00
Derek Collison
6e17b7a303 Fix for #2213
We do not want to report consumers that were created for the purpose of sources or mirrors.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-12 07:51:53 -07:00
Derek Collison
494ddeec5b Fix for #2202
When using multiple source streams from either different accounts or domains, the stream name could be the same and would cause bad behavior.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-11 13:12:10 -07:00
Derek Collison
9a517194a1 Merge pull request #2191 from nats-io/raft_catchup_snap
[FIXED] Raft groups could continually spin trying to catch up.
2021-05-07 14:20:37 -07:00
Derek Collison
4500f7889e For WorkQueues that have direct we need to make sure they have received the message before deleting.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-07 14:13:36 -07:00
Derek Collison
bd9172fb61 With WorkQueue policy streams that are mirrored or sourced need to only delete if no regular consumers.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-07 13:14:56 -07:00
R.I.Pienaar
b5f846a719 add domain in JS advisories
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-05-07 19:35:46 +02:00
Derek Collison
35a60289d9 Fixed mirror/source streams from work queues.
Fixed deadlock with no-ack consumers on interest retention streams.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-07 06:01:52 -07:00
Derek Collison
c8d2132ee5 Expand test to target mirrors and sources in different JS domains.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-06 18:45:27 -06:00
Derek Collison
a301d3a892 Check direct consumers directly, stream state back to previous behavior
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-21 09:40:04 -07:00
Derek Collison
b5521053e6 Make sure to clean up ephemerals across a Gateway.
Also report direct consumers in num consumers in stream info.

Signed-off-by: Derek Collison <derek@nats.io>
2021-04-21 08:47:55 -07:00
Derek Collison
ef7a811a71 Check that we have mirror still to not panic and deadlock/hang
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-20 16:41:41 -07:00
Derek Collison
5aa3dd7761 Put back direct consumer subject prefix
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-20 13:35:48 -07:00
Derek Collison
0a3e54c88a Fix subscription leaks for mirrors and sources for streams as well as flow control subjects for consumers.
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-20 12:13:07 -07:00
Derek Collison
68ddd519d2 Process upstream missing messages for mirrors better.
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-19 20:15:21 -07:00
Derek Collison
7adf7bc7d6 Maintain stream sequence properly
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-18 18:38:28 -07:00
Derek Collison
d84af39f16 Make sure source consumers work properly after expiration/purge
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-16 17:28:04 -07:00
Derek Collison
60df8a2fcb Minimize proposals doing skip msgs
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-16 14:19:29 -07:00