Commit Graph

162 Commits

Author SHA1 Message Date
Derek Collison
925a6fe6b2 Fix for #2388. Leafnodes with no JS can seamlessly access a HUB with JS.
This is the reverse of the early work to have LNs extend a non-JS cluster.
Also have mixed mode tests as well.

Signed-off-by: Derek Collison <derek@nats.io>
2021-08-01 14:57:47 -07:00
Derek Collison
9b0158daf9 Allow delivery policy of DeliverLastPerSubject, which is helpful for scoped watchers for K/V.
Signed-off-by: Derek Collison <derek@nats.io>
2021-07-28 12:49:02 -07:00
Derek Collison
98970ac03d Add descriptions to JetStream streams and consumers. 2021-07-27 14:05:25 -07:00
Derek Collison
f13fa767c2 Remove the swapping of accounts during processing of service imports.
When processing service imports we would swap out the accounts during processing.
With the addition of internal subscriptions and internal clients publishing in JetStream we had an issue with the wrong account being used.
This was specific to delyaed pull subscribers trying to unsubscribe due to max of 1 while other JetStream API calls were running concurrently.
2021-07-26 07:57:10 -07:00
Ivan Kozlovic
a7e0eeaf50 Removed debug.PrintStack() from apiDispatch [ci skip]
It was introduced by mistake in PR #2324 which I had reported
but failed to be actually removed before PR was merged:
https://github.com/nats-io/nats-server/pull/2324#pullrequestreview-695163635

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-07-13 16:49:44 -06:00
Derek Collison
894f26d149 Fix for #2352
Signed-off-by: Derek Collison <derek@nats.io>
2021-07-08 06:13:36 -07:00
Derek Collison
ad4685c84f Fix for crash in test run
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-30 19:30:37 -07:00
Derek Collison
99fed910f0 Improvements to large numbers of JetStream R1 consumers per stream.
1. We were holding open FDs longer than we should for consumers causing issues with open FD limits. We now do not hold them open and cap updates a bit better.

2. When doing a stream delete, consumer delete was repeating alot of work that was not necessary, causing longer delays. This has been optimized a bit, still more improvements to be made.

3. We cover all JS under a single export, but that was also trapping GetNext for pull based consumers, and since this was a no-op (is handled at user account level) we were creating alot of garbage service import responses and reverse map entries that had to be garbage collected. We have a fix in to avoind this but still looking for a better one.

4. Still had some lingering references to all exports vs single JS export.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-29 05:45:55 -07:00
Derek Collison
c0e47966ab Added in Stream get last message by subject.
This is to aid in K/V overlay for simple Get ops vs creating a full blown consumer.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-24 13:21:39 -07:00
Derek Collison
9398c3ca28 Allow for more advanced purge operations that filter by subject, specify the sequence or number of messages to keep.
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-19 07:04:44 -07:00
R.I.Pienaar
53ae604403 further tagged error confusion cleanups
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-06-18 21:35:26 +02:00
R.I.Pienaar
a0fcf0bb65 further tagged error confusion cleanups
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-06-18 20:11:09 +02:00
Derek Collison
c75d97a0d3 Fix for #2290
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-18 07:03:51 -07:00
Matthias Hanel
2caf2303f2 [adding] jetstream info to statsz (#2269)
* [adding] jetstream info to statsz

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-06-10 11:54:56 -04:00
R.I.Pienaar
470969d1b3 ensure correct error handling in non clustered stream add
The addStream() can return an ApiErr but we did not handle
that leading to errors like 'stream name already in use (10058)' instead
of just 'stream name already in use' with the correct error code 10058 set

Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-06-08 13:38:00 +02:00
Derek Collison
30fae4f960 Changes to leafnodes to support multiple domains where the hub is JetStream enabled but the hub account is not, and the leafnode is.
We were incorrectly shutting things down via deny clauses when detecting the remote side/hub had JetStream capabilities.
This change moves that logic to the remote side and is signalled off the connect message which let's the remote side know
if the local leafnode account has JetStream enabled.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-07 08:39:11 -07:00
R.I.Pienaar
ee9d10f40b restore old error constants for backwards compat
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-05-26 08:04:50 +02:00
R.I.Pienaar
0d391b02eb richer api errors proposal
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-05-26 08:04:50 +02:00
Matthias Hanel
b93f5975e8 [fixed] issue with invalid api prefix for source/mirror
addressed by failing

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-05-20 18:01:49 -04:00
Derek Collison
8b50094fde Merge pull request #2227 from nats-io/docker_js
Fixed issue that we would stage snapshot restores in temporary storage.
2021-05-20 06:56:31 -07:00
Derek Collison
9ccc843382 Removing peers should wait for RemovePeer entry replication.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-19 18:58:19 -07:00
Derek Collison
a27b0dd673 Move default file and dir perms
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-19 14:46:07 -07:00
Derek Collison
592066ed0b Fixed issue that we would stage snapshot restore in temporary storage.
This did not work when using our default docker image which does not have /tmp by default.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-19 11:36:20 -07:00
Derek Collison
6e17b7a303 Fix for #2213
We do not want to report consumers that were created for the purpose of sources or mirrors.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-12 07:51:53 -07:00
Matthias Hanel
f4251feb0f [added] check for mirror stream return error when de dupe window is set
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-05-10 19:57:17 -04:00
R.I.Pienaar
b5f846a719 add domain in JS advisories
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-05-07 19:35:46 +02:00
Derek Collison
8499376575 Add in support for JetStream domains.
This allows a domain to be set in the JetStream server block that sets a domain name.
Once set this signals that any leafnode connections should operate as separate JetStream domains.
Each domain <NAME> is accessible via "$JS.<NAME>.API.>", even when connected to the same domain.
Also for mixed mode you can set a jetstream block now that defines a domain but specifies "enabled: false".

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-06 18:46:32 -06:00
Derek Collison
c87eed8792 Create helper function to check on account jetstream status.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-06 18:45:27 -06:00
Derek Collison
0bd92e85da Add in formal support for multiple JetStream domains across leafnodes.
This CL adds in support for multiple JetStream domains using mapped subjects.
Mapping subjects aligns well with the JetStream context APIPrefix in clients.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-06 18:45:27 -06:00
Matthias Hanel
eb3af67031 Delaying stream/consumer error response when there is no contact to meta
this gives a stream/consumer leader time to respond first

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-04-21 20:24:43 -04:00
Derek Collison
3788822f82 Updates based on feedback
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-21 12:41:11 -07:00
Derek Collison
9f1bc5882c Make sure to return no consumer if we are still waiting on a leader to be elected.
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-21 12:13:10 -07:00
Ivan Kozlovic
56d0d9ec87 Do not propagate service import interest across GW and ROUTES
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-04-15 11:34:36 -06:00
Derek Collison
90989d57d6 Change to report total deleted by default for stream info.
Allow deleted details if requested.

Signed-off-by: Derek Collison <derek@nats.io>
2021-04-12 18:10:11 -07:00
Derek Collison
1ea4a430da If we fail to load an account while processing a stream assignment, send error back to metaleader.
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-07 14:23:12 -07:00
Derek Collison
ceca86a79c Fix for #1976
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-31 11:23:31 -07:00
Matthias Hanel
03aee09847 [Added] error when mirror/source stream prefix overlaps with stream subs (#2041)
* [Added] error when mirror/source stream prefix overlaps with stream subs

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-03-27 14:01:51 -04:00
Derek Collison
e53caee5e8 Enforce server limits even when dynamic limits for accounts in play.
We were not properly enforcing server limits. This commit will allow a server to enforce limits but still remain functional even at the JetStream level.
Also fixed a bug for RAFT replay that could cause instability.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-25 16:06:27 -07:00
Derek Collison
2ed53035ed Reworked flow control for sources and mirrors.
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-24 07:07:33 -07:00
Derek Collison
022c5b4ab2 Fix for bad behavior with flow control and multiple sources streams.
Allow chaining of sources and mirrors with filtered consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-22 11:13:51 -07:00
Derek Collison
ee92cc9a5b Properly print when a stream is doing out of band catchup. Print node banner consistently
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-14 07:29:36 -07:00
Derek Collison
cbbe6dc9c5 Make API access determing system not available consistent.
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-14 06:18:04 -07:00
Derek Collison
299f44cddf This changes our behaviors for streams and peer removals in several ways.
First we no longer try to auto-remap stream assignments on peer removals from the system.
We also now can always respond to stream info requests if at least a member is running.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-11 06:52:28 -05:00
Derek Collison
f95b6481d2 Make source and mirror consumers direct, meaning they are not mapped by the metaleader.
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-10 07:12:48 -05:00
Derek Collison
75b0455e0b Enable opt-in flow control for push based consumers.
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-05 08:48:12 -08:00
Derek Collison
207ebd3b3d Changed stream sendq to linked list outq.
Made consumer share streams outq.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-04 17:19:50 -08:00
Derek Collison
bfb8e3432e Move RAFT comms off internal sendq.
Move route and gateway msgs our of fast path for inbound stream msgs.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-04 14:45:34 -08:00
Ivan Kozlovic
0f53bf6580 Fixed data race with nodeInfo
Took the approach of storing struct instead of pointer. Of course,
when changing the offline bool from false to true, it means that
we need to call Store again (with same key).

This is based on the assumption that those Load/Store are not too
frequent. Otherwise, we may need to use locking (and keep *nodeInfo)

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-03-03 13:28:45 -07:00
Derek Collison
66cbf75712 Merge branch 'master' into consumer 2021-03-03 10:29:39 -07:00
R.I.Pienaar
236498a142 restore meta peer remove by name rather than id
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-03-03 15:55:50 +01:00