Commit Graph

177 Commits

Author SHA1 Message Date
Derek Collison
678469b40b Fix for #2644
Signed-off-by: Derek Collison <derek@nats.io>
2021-10-25 13:12:37 -07:00
Derek Collison
aff0f62106 In addition to sealed we add in other stream perms to control purge, msg deletes and rollups.
Signed-off-by: Derek Collison <derek@nats.io>
2021-10-05 17:43:24 -07:00
Derek Collison
1338a167e3 Merge pull request #2584 from nats-io/sealed-stream
Allow streams to be sealed through a stream update.
2021-09-29 16:02:58 -07:00
Derek Collison
5fc2cc5754 Allow streams to be sealed through a stream update.
Sealed streams can not accept new messages, allow you to delete or purge messages, or have messages expire due to age.
Sealed stream can not be unsealed through an update.

Signed-off-by: Derek Collison <derek@nats.io>
2021-09-29 15:25:38 -07:00
R.I.Pienaar
98ef6cb03c fixes the msg get audit event
We resused msg variable for both the request
and later the data loaded from the stream.

The data thus replaces the request and the audit
had the data as both request and response

Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-09-29 17:36:38 +02:00
Derek Collison
9918f16914 ConsumerInfo in clustered mode would return not found if assigned but not properly setup with a leader.
This change returns an info in those states now to better support client logic when binding consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2021-09-16 10:56:34 -07:00
Ivan Kozlovic
47359c453e Merge pull request #2533 from nats-io/js_cons_fc_hb
[CHANGED] JetStream: flow control requires heartbeats
2021-09-16 10:20:48 -06:00
Derek Collison
cfbc69b12c Allow clustered JetStream to allow duplicate stream creation like single server mode.
Resolves #2528

Signed-off-by: Derek Collison <derek@nats.io>
2021-09-15 20:18:44 -07:00
Ivan Kozlovic
108d4060ff [CHANGED] JetStream: flow control requires heartbeats
The server will now reject the creation of a push consumer with
flow control if no heartbeat is set.

Resolves #2520

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-09-15 09:50:15 -06:00
Ivan Kozlovic
d6f9a06f05 Merge pull request #2530 from nats-io/mqtt_del_sess
MQTT: delete the session record even on restart with clean flag
2021-09-15 09:01:49 -06:00
Ivan Kozlovic
4cc2677573 MQTT: delete the session record even on restart with clean flag
When a session record exists and is currently not connected, if
the user reconnects with the same client ID but with now the
clean flag set, we are required to clear the state. In earlier
implementation (where we were using a stream per session), we
were not deleting the stream to be created right away. Since now
we just have a message per session, it is ok to delete that
message when clearing the existing session that is going to be
replaced.

Also made apiDispatch execute in place for any connection that
is not ROUTER or GATEWAY.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-09-14 16:33:48 -06:00
Derek Collison
ebc581012e Fixed a condition where JetStream assets could be created in multiple leafnodes.
Also added in optional Domain to StreamInfo.

Signed-off-by: Derek Collison <derek@nats.io>
2021-09-14 14:17:49 -07:00
Derek Collison
8615820a87 Avoid potential race
Signed-off-by: Derek Collison <derek@nats.io>
2021-09-01 14:00:31 -07:00
Derek Collison
57ef5fd528 Set consumer config defaults early on to avoid a race condition.
Signed-off-by: Derek Collison <derek@nats.io>
2021-09-01 13:06:08 -07:00
R.I.Pienaar
76ab1b8d17 attempt to improve UX of the error system
Previously we had a few confusing functions like NewT
and similar that were quite fragile to use due to minimal
validation and a panic in go stdlib string Replacer.

Now we generate helper methods for every string, these
are used to access errors, fill in templates and conditional
returns of error type using the new Unless() option

We now get compile time errors for some common mistakes
and have better IDE helpers for arguments etc

Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-08-10 16:08:28 +02:00
Derek Collison
925a6fe6b2 Fix for #2388. Leafnodes with no JS can seamlessly access a HUB with JS.
This is the reverse of the early work to have LNs extend a non-JS cluster.
Also have mixed mode tests as well.

Signed-off-by: Derek Collison <derek@nats.io>
2021-08-01 14:57:47 -07:00
Derek Collison
9b0158daf9 Allow delivery policy of DeliverLastPerSubject, which is helpful for scoped watchers for K/V.
Signed-off-by: Derek Collison <derek@nats.io>
2021-07-28 12:49:02 -07:00
Derek Collison
98970ac03d Add descriptions to JetStream streams and consumers. 2021-07-27 14:05:25 -07:00
Derek Collison
f13fa767c2 Remove the swapping of accounts during processing of service imports.
When processing service imports we would swap out the accounts during processing.
With the addition of internal subscriptions and internal clients publishing in JetStream we had an issue with the wrong account being used.
This was specific to delyaed pull subscribers trying to unsubscribe due to max of 1 while other JetStream API calls were running concurrently.
2021-07-26 07:57:10 -07:00
Ivan Kozlovic
a7e0eeaf50 Removed debug.PrintStack() from apiDispatch [ci skip]
It was introduced by mistake in PR #2324 which I had reported
but failed to be actually removed before PR was merged:
https://github.com/nats-io/nats-server/pull/2324#pullrequestreview-695163635

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-07-13 16:49:44 -06:00
Derek Collison
894f26d149 Fix for #2352
Signed-off-by: Derek Collison <derek@nats.io>
2021-07-08 06:13:36 -07:00
Derek Collison
ad4685c84f Fix for crash in test run
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-30 19:30:37 -07:00
Derek Collison
99fed910f0 Improvements to large numbers of JetStream R1 consumers per stream.
1. We were holding open FDs longer than we should for consumers causing issues with open FD limits. We now do not hold them open and cap updates a bit better.

2. When doing a stream delete, consumer delete was repeating alot of work that was not necessary, causing longer delays. This has been optimized a bit, still more improvements to be made.

3. We cover all JS under a single export, but that was also trapping GetNext for pull based consumers, and since this was a no-op (is handled at user account level) we were creating alot of garbage service import responses and reverse map entries that had to be garbage collected. We have a fix in to avoind this but still looking for a better one.

4. Still had some lingering references to all exports vs single JS export.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-29 05:45:55 -07:00
Derek Collison
c0e47966ab Added in Stream get last message by subject.
This is to aid in K/V overlay for simple Get ops vs creating a full blown consumer.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-24 13:21:39 -07:00
Derek Collison
9398c3ca28 Allow for more advanced purge operations that filter by subject, specify the sequence or number of messages to keep.
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-19 07:04:44 -07:00
R.I.Pienaar
53ae604403 further tagged error confusion cleanups
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-06-18 21:35:26 +02:00
R.I.Pienaar
a0fcf0bb65 further tagged error confusion cleanups
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-06-18 20:11:09 +02:00
Derek Collison
c75d97a0d3 Fix for #2290
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-18 07:03:51 -07:00
Matthias Hanel
2caf2303f2 [adding] jetstream info to statsz (#2269)
* [adding] jetstream info to statsz

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-06-10 11:54:56 -04:00
R.I.Pienaar
470969d1b3 ensure correct error handling in non clustered stream add
The addStream() can return an ApiErr but we did not handle
that leading to errors like 'stream name already in use (10058)' instead
of just 'stream name already in use' with the correct error code 10058 set

Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-06-08 13:38:00 +02:00
Derek Collison
30fae4f960 Changes to leafnodes to support multiple domains where the hub is JetStream enabled but the hub account is not, and the leafnode is.
We were incorrectly shutting things down via deny clauses when detecting the remote side/hub had JetStream capabilities.
This change moves that logic to the remote side and is signalled off the connect message which let's the remote side know
if the local leafnode account has JetStream enabled.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-07 08:39:11 -07:00
R.I.Pienaar
ee9d10f40b restore old error constants for backwards compat
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-05-26 08:04:50 +02:00
R.I.Pienaar
0d391b02eb richer api errors proposal
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-05-26 08:04:50 +02:00
Matthias Hanel
b93f5975e8 [fixed] issue with invalid api prefix for source/mirror
addressed by failing

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-05-20 18:01:49 -04:00
Derek Collison
8b50094fde Merge pull request #2227 from nats-io/docker_js
Fixed issue that we would stage snapshot restores in temporary storage.
2021-05-20 06:56:31 -07:00
Derek Collison
9ccc843382 Removing peers should wait for RemovePeer entry replication.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-19 18:58:19 -07:00
Derek Collison
a27b0dd673 Move default file and dir perms
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-19 14:46:07 -07:00
Derek Collison
592066ed0b Fixed issue that we would stage snapshot restore in temporary storage.
This did not work when using our default docker image which does not have /tmp by default.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-19 11:36:20 -07:00
Derek Collison
6e17b7a303 Fix for #2213
We do not want to report consumers that were created for the purpose of sources or mirrors.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-12 07:51:53 -07:00
Matthias Hanel
f4251feb0f [added] check for mirror stream return error when de dupe window is set
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-05-10 19:57:17 -04:00
R.I.Pienaar
b5f846a719 add domain in JS advisories
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-05-07 19:35:46 +02:00
Derek Collison
8499376575 Add in support for JetStream domains.
This allows a domain to be set in the JetStream server block that sets a domain name.
Once set this signals that any leafnode connections should operate as separate JetStream domains.
Each domain <NAME> is accessible via "$JS.<NAME>.API.>", even when connected to the same domain.
Also for mixed mode you can set a jetstream block now that defines a domain but specifies "enabled: false".

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-06 18:46:32 -06:00
Derek Collison
c87eed8792 Create helper function to check on account jetstream status.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-06 18:45:27 -06:00
Derek Collison
0bd92e85da Add in formal support for multiple JetStream domains across leafnodes.
This CL adds in support for multiple JetStream domains using mapped subjects.
Mapping subjects aligns well with the JetStream context APIPrefix in clients.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-06 18:45:27 -06:00
Matthias Hanel
eb3af67031 Delaying stream/consumer error response when there is no contact to meta
this gives a stream/consumer leader time to respond first

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-04-21 20:24:43 -04:00
Derek Collison
3788822f82 Updates based on feedback
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-21 12:41:11 -07:00
Derek Collison
9f1bc5882c Make sure to return no consumer if we are still waiting on a leader to be elected.
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-21 12:13:10 -07:00
Ivan Kozlovic
56d0d9ec87 Do not propagate service import interest across GW and ROUTES
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-04-15 11:34:36 -06:00
Derek Collison
90989d57d6 Change to report total deleted by default for stream info.
Allow deleted details if requested.

Signed-off-by: Derek Collison <derek@nats.io>
2021-04-12 18:10:11 -07:00
Derek Collison
1ea4a430da If we fail to load an account while processing a stream assignment, send error back to metaleader.
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-07 14:23:12 -07:00