147 Commits

Author SHA1 Message Date
Ivan Kozlovic
a294517946 No need to fetch account if we detect that it is valid NKEY
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-05-12 15:57:26 -06:00
Phil Pennock
452bec8e42 Don't fix dir perms yet, punt to separate issue 2021-05-12 13:16:10 -04:00
Phil Pennock
408e009ebb Error-checking, resilience & perms for JetStream move
Jetstream movement can fail, so return that error and abort start-up if there's
a failure in moving precious data, rather than serve without it.

Create the jetstream directory if needed.

Create directories for private data mode 0750 not 0755.

This does not handle a directory layout made with 2.2.3, but does support a
2.2.2 to 2.2.4 migration.  The empty directories made under 2.2.3 will still
hinder the renames we do here.
2021-05-12 12:32:21 -04:00
Derek Collison
58344c666a Update based on feedback.
Only log when actually moving an account in case other files start polluting the directory.
When failing to look up an account and we have a resolver check to see if its a valid account name before attempting lookup.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-11 07:54:10 -07:00
Derek Collison
bad9fba8ec Fix for store directory on disk prior to 2.2.3 without the 'jetstream' directory.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-10 19:24:58 -07:00
Ivan Kozlovic
19e07f7b97 Extend stream/consumer name validation to include more characters
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-05-07 16:50:29 -06:00
R.I.Pienaar
b5f846a719 add domain in JS advisories
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-05-07 19:35:46 +02:00
R.I.Pienaar
cc9b6735a5 remove . from domain names
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-05-07 14:46:22 +02:00
R.I.Pienaar
0bc6ac2e14 small tweaks to domains
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-05-07 11:03:23 +02:00
Derek Collison
c2fcc114a5 Update based on PR feedback, moved to validateOptions
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-06 20:10:44 -07:00
Derek Collison
0c74a850b5 Make storedir consistent with mix of flags etc 2021-05-06 18:46:32 -06:00
Derek Collison
ea5cddd590 Moved the JetStream logic for solicited leafnodes to after we receive first info.
We needed access to the other side's JetStream status.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-06 18:46:32 -06:00
Derek Collison
8499376575 Add in support for JetStream domains.
This allows a domain to be set in the JetStream server block that sets a domain name.
Once set this signals that any leafnode connections should operate as separate JetStream domains.
Each domain <NAME> is accessible via "$JS.<NAME>.API.>", even when connected to the same domain.
Also for mixed mode you can set a jetstream block now that defines a domain but specifies "enabled: false".

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-06 18:46:32 -06:00
Derek Collison
0bd92e85da Add in formal support for multiple JetStream domains across leafnodes.
This CL adds in support for multiple JetStream domains using mapped subjects.
Mapping subjects aligns well with the JetStream context APIPrefix in clients.

Signed-off-by: Derek Collison <derek@nats.io>
2021-05-06 18:45:27 -06:00
Derek Collison
bd9a782701 Fixed issue when limiting bytes with an unlimited account that would always fail.
Signed-off-by: Derek Collison <derek@nats.io>
2021-05-05 07:04:24 -07:00
Derek Collison
d10b054f43 Update comment per feedback
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-30 18:20:29 -07:00
Derek Collison
d43c818bf8 Make sure when we extend the JetStream domain with a single server leafnode we can place things there.
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-30 18:18:16 -07:00
Derek Collison
8bf99224c5 This adds ability to have a single node server with a system leafnode expand an existing JetStream cluster domain.
Signed-off-by: Derek Collison <derek@nats.io>
2021-04-30 16:20:32 -07:00
scottf
486df98373 close tempfiles, fix path print 2021-04-22 12:47:21 -04:00
Waldemar Quevedo
c9ab7ce8a1 Fix for data race when disabling JS running out of resources
Signed-off-by: Waldemar Quevedo <wally@synadia.com>
2021-04-21 14:26:52 -07:00
Derek Collison
c0e8590c0f During startup each filtered consumer could do a linear scan of the stream
to determine number of messages pending. This improves that with a startup cache.

Signed-off-by: Derek Collison <derek@nats.io>
2021-04-07 09:15:01 -07:00
Derek Collison
c05ff97d09 Fixes based on PR feedback
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-25 18:09:35 -07:00
Derek Collison
da7b7f9f85 Make sure to configure global account for reload
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-25 18:06:14 -07:00
Derek Collison
e53caee5e8 Enforce server limits even when dynamic limits for accounts in play.
We were not properly enforcing server limits. This commit will allow a server to enforce limits but still remain functional even at the JetStream level.
Also fixed a bug for RAFT replay that could cause instability.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-25 16:06:27 -07:00
Derek Collison
2ed53035ed Reworked flow control for sources and mirrors.
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-24 07:07:33 -07:00
Derek Collison
d9b78b8bc4 Update info status for JetStream when disabling and restarting
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-20 09:07:47 -07:00
Derek Collison
cfe2f448ad Fix for when gateways were dropped and we did not detect interest returning.
Fix for leaked subscription when retrying the source consumers.
Better suppression of old messages from old direct consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-20 08:44:18 -07:00
Derek Collison
61771e88f8 In operator mode with JetStream we want to load accounts that have stable storage.
Also if an account was registered but not JetStream enabled, update it vs error.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-20 06:53:13 -07:00
Derek Collison
8eefff2b3b Make sure the jetstream accounts use the name as the key to the map.
This prevents possible double adds under reload or restart scenarios.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-18 17:29:26 -07:00
R.I.Pienaar
ac70abdcf3 update banner url
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-03-17 16:39:24 +01:00
Derek Collison
2e9ae92bb5 Fix for #2011 and atomics on 32bit systems.
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-16 13:57:28 -07:00
Derek Collison
e530c98ebe We no longer force remove our peer on out of space.
We also delay restarting JetStream to make sure accounts are enabled.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-14 17:01:38 -07:00
Waldemar Quevedo
775afd01e7 Fix for JS reload and exports
Signed-off-by: Waldemar Quevedo <wally@synadia.com>
2021-03-14 15:16:56 -07:00
Derek Collison
e74eaa02c4 Suppress multiple out of space warnings
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-13 16:30:57 -05:00
Derek Collison
a3a35c0ddb Updated raft processing and dealing with remove peer.
Made sure to not remove us if we were remapped after the peer removal.
Fixed some raft behaviors.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-13 16:28:24 -05:00
Derek Collison
be3514bfdc Do not auto remap
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-13 16:26:26 -05:00
Waldemar Quevedo
93fd0f60f5 reload: Allow re-enabling JS after it was disabled
Signed-off-by: Waldemar Quevedo <wally@synadia.com>
2021-03-12 17:19:34 -08:00
Waldemar Quevedo
d338e9ae30 Disable JS on restart if on invalid state
Signed-off-by: Waldemar Quevedo <wally@synadia.com>
2021-03-10 01:40:32 -08:00
Matthias Hanel
bea37b089c fixing crash when disabling already shut down client
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-03-08 19:57:18 -05:00
Ivan Kozlovic
57af977548 Stabilized stream sources under restart
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-03-08 16:41:02 -07:00
Derek Collison
5fe9ba5e94 Switched to fast state
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-06 09:00:15 -08:00
Derek Collison
0b3c686430 Fixes for data races and some locking.
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-05 17:19:51 -08:00
Derek Collison
ff98984358 Reworked internal consumers used for sources and mirrors.
Now use ephemerals and heartbeats and flowcontrol.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-05 14:32:57 -08:00
Derek Collison
d7201a110b Better handling on out of disk.
Suppress some stream and consumer bad results since they delete the asset.
Allow rehup to re-enable JetStream.
Various bug fixes and improvements.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-03 20:12:10 -08:00
Ivan Kozlovic
e7e756034a Switch Gateway JS accounts to interest-only mode + some other fixes
- Fixed the close of a TLS connection which starting Go 1.16
set the deadline to 5 seconds.

- Fixed an issue with setHeader that was causing these error messages
```
=== RUN   TestServiceImportReplyMatchCycleMultiHops
nats: message could not decode headers on connection [4] for subscription on "foo"
--- PASS: TestServiceImportReplyMatchCycleMultiHops (0.04s)
```

- Fixed names of tests in norace_test.go since they must start with
TestNoRace in order to make sure that we execute them in Travis:
```
go test -v -run=TestNoRace --failfast -p=1 ./...
```

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-03-03 19:15:28 -07:00
Derek Collison
e8029094e3 Cleanup export and import processing
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-03 06:35:52 -08:00
Derek Collison
00a49ce329 Test for existing service import
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-02 20:38:16 -08:00
Derek Collison
4f7fbefc7c In clustered JetStream we need to move API calls out of routes/gateways/leafnodes path.
This moves from explicit imports and subscriptions to one wildcard subscription and a single wildcard export.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-02 17:54:41 -08:00
Derek Collison
d4a8baf030 Stabilize test, wait a bit for migration
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-02 05:28:55 -08:00
Derek Collison
22749fb1e2 Fix data race, stabilize usage test
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-01 16:47:19 -08:00