Commit Graph

186 Commits

Author SHA1 Message Date
Derek Collison
dfd9548d98 Merge pull request #2406 from nats-io/consumer-panic
Various bug fixes and improvements to clustered filestore consumer stores.
2021-08-04 08:36:24 -07:00
Derek Collison
09012f0847 Fix for #2401.
Leafnodes with same domain and shared system account should behave like flat jetstream network.

Signed-off-by: Derek Collison <derek@nats.io>
2021-08-04 07:07:29 -07:00
Derek Collison
398ae95a4a Various bug fixes and improvements to filestore consumer stores.
Improved behavior around clustered persistent consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2021-08-03 22:17:49 -07:00
Ivan Kozlovic
8d2faadb5c Test to reproduce panic on consumer state recovery
You may have to run the test multiple times:
```
go test -race -v -run=TestJetStreamPanicDecodingConsumerState ./server -count=10
```

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-08-03 15:35:46 -06:00
Derek Collison
925a6fe6b2 Fix for #2388. Leafnodes with no JS can seamlessly access a HUB with JS.
This is the reverse of the early work to have LNs extend a non-JS cluster.
Also have mixed mode tests as well.

Signed-off-by: Derek Collison <derek@nats.io>
2021-08-01 14:57:47 -07:00
Derek Collison
065049e090 Fix for flapper
Signed-off-by: Derek Collison <derek@nats.io>
2021-07-08 07:42:40 -07:00
Derek Collison
6337198119 Fix for multiple concurrent ephemeral consumer requests in clustered mode with max consumers set.
Signed-off-by: Derek Collison <derek@nats.io>
2021-07-08 07:02:09 -07:00
Ivan Kozlovic
b582b13a96 Fixed another flapper
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-07-06 14:20:16 -06:00
Ivan Kozlovic
df3eccbd69 Fix yet another flapper
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-07-06 13:55:28 -06:00
Ivan Kozlovic
710d0c3a1f Fixed another flapper
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-07-06 12:44:08 -06:00
Ivan Kozlovic
d5f5a74438 Fixed some tests
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-07-06 12:20:03 -06:00
Derek Collison
57cfba0964 Make requirement for interest on creation of ephemerals soft vs hard.
Will start the cleanup timer if no interest but allow creation.

Signed-off-by: Derek Collison <derek@nats.io>
2021-07-06 10:48:07 -07:00
Derek Collison
960c45df81 Use of sync.Pool for filestore could cause msg corruption.
Signed-off-by: Derek Collison <derek@nats.io>
2021-07-06 08:41:01 -07:00
Derek Collison
3c16f776dc Protect account prand, use faster Uint64 variant
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-29 06:13:41 -07:00
Derek Collison
cbb97a4f4a Fix for Go client behavior change
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-27 10:52:03 -07:00
Derek Collison
545230bd9e FIXED Clustered streams can become broken with seq mismatch state on low level store failures.
We can account for high level failure properly but were not properly accounting for low level store failures.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-25 13:46:40 -07:00
Derek Collison
08197de9e0 FIXED max consumers was not enforced when set on stream
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-25 11:45:36 -07:00
Derek Collison
2e145196b8 Fix for extended purge by sequence.
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-22 07:38:30 -07:00
Derek Collison
5fcd9c5564 Add test for sequence only
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-20 10:42:35 -07:00
Derek Collison
89d930fd0f Updates and fixes to PurgeEx
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-20 10:34:27 -07:00
Derek Collison
9398c3ca28 Allow for more advanced purge operations that filter by subject, specify the sequence or number of messages to keep.
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-19 07:04:44 -07:00
Derek Collison
08cdb2d2ea Make filtered consumers in large mixed streams more efficient.
Allow wider scoped filtered subjects.

We introduce a per subject information tracking to filestore to optimize for large mux'd streams and more efficient filtered consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-15 04:44:05 -07:00
Derek Collison
ceebc3ae07 When checking limits we would check total ask against the server limits if limits were not set.
We were also dynamically setting account limits based on a single server limit.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-12 10:27:43 -07:00
Derek Collison
4912c8b8f7 Make sure leader ready
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-10 21:29:03 -07:00
Derek Collison
326a377b3e When leader changes happened to streams that mirrored or sourced other streams we could continue to try to create consumers.
This could get excessive on a stream that has sourced many upstream origin streams and had several leader changes.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-10 19:47:46 -07:00
Derek Collison
1270977322 When receiving a response across a gateway that has headers and a globally routed subject (_GR_) we were dropping header information.
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-10 14:29:33 -07:00
Derek Collison
cc63915f54 Added active servers to statsz.
This is generally useful but will also help with cli actions since we know from first response how many total responses are expected.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-09 19:14:37 -07:00
Derek Collison
e238512285 Handling of rewrites for subjects to a globally routed subject was not properly handling c.pa.deliver or reply rewrite.
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-09 15:34:47 -07:00
Derek Collison
30fae4f960 Changes to leafnodes to support multiple domains where the hub is JetStream enabled but the hub account is not, and the leafnode is.
We were incorrectly shutting things down via deny clauses when detecting the remote side/hub had JetStream capabilities.
This change moves that logic to the remote side and is signalled off the connect message which let's the remote side know
if the local leafnode account has JetStream enabled.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-07 08:39:11 -07:00
R.I.Pienaar
0d391b02eb richer api errors proposal
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-05-26 08:04:50 +02:00
Derek Collison
0d6b022780 Fix for varz JetStream usage showing total account usage vs server usage.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-24 14:05:42 -07:00
Derek Collison
8888ab51f4 Fix for #2243. We were not allowing replicated acks processing for workqueues properly, only interest retention.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-24 09:53:31 -07:00
Derek Collison
8f2457fd30 Merge branch 'master' into msgid 2021-05-24 09:22:35 -07:00
Derek Collison
308355a2fd Fix for #2242.
When we had a duplicate detected in R>1 mode we set the skip sequence indicator but were not using that when dealing with underlying store.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-24 08:21:41 -07:00
Derek Collison
a52697170c Fix for corrupting a message when extending an existing header.
We had a report of corrupt message payloads when going across leafnodes between streams that were sourced from one another.
We were incorrectly using the underlying buffer when a header already existed.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-23 11:28:48 -07:00
Derek Collison
7936faf00e Merge pull request #2230 from nats-io/mapped
Mapped cross account test
2021-05-20 11:32:33 -07:00
Derek Collison
a14f07f86c Fix flapping test
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-19 19:09:49 -07:00
Derek Collison
9ccc843382 Removing peers should wait for RemovePeer entry replication.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-19 18:58:19 -07:00
Derek Collison
6e6edd0b47 Mapped cross account test
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-19 15:58:27 -07:00
Derek Collison
bd01f202af Under double import scenarios we could map to the wrong subject.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-19 05:59:24 -07:00
Derek Collison
060f2ec127 Fix stream source lookup and add in optional External to StreamSourceInfo.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-12 12:32:56 -07:00
Derek Collison
908b5ec304 Fix for #2205
When a response was needed from a leafnode cluster back to a hub, we had rules to disallow.
That rule was a bit dated and since we have cluster origin for leafnode clusters and that
is checked before the message is actually sent we could remove the old rule.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-12 06:49:11 -07:00
Derek Collison
494ddeec5b Fix for #2202
When using multiple source streams from either different accounts or domains, the stream name could be the same and would cause bad behavior.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-11 13:12:10 -07:00
Derek Collison
bbb94467fb Fix for failing test, we needed to account for consumers having filtered subjects configured but not really being filtered.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-10 17:01:31 -07:00
Derek Collison
09c4f5841c Bump version and skip test for now, will fix in followup PR.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-10 10:44:07 -07:00
Derek Collison
f74bb53c83 Remove debug
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-07 12:21:35 -07:00
Derek Collison
95a9304ae7 Tweak timing
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-07 12:07:27 -07:00
Derek Collison
35a60289d9 Fixed mirror/source streams from work queues.
Fixed deadlock with no-ack consumers on interest retention streams.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-07 06:01:52 -07:00
Ivan Kozlovic
a9a49cc2d5 MQTT make session streams domain aware
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-05-06 20:02:00 -06:00
Derek Collison
2002ea1ee5 Another test fix
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-06 18:46:32 -06:00