Commit Graph

3966 Commits

Author SHA1 Message Date
Matthias Hanel
509eec4baa bump server version to 2.9.0-beta.3
Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-07-07 03:36:19 +02:00
Derek Collison
16f788adce Consumer isFiltered could crash if the stream was a mirror with no direct subjects but it was in interest or workqueue retention mode.
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-07 03:35:41 +02:00
Matthias Hanel
70be4b77f9 fixes peer removal, simplifies move, more tests
Make sure when processing a peer removal that the stream assignment agrees.
When a new leader takes over it can resend a peer removal, and if the stream/consumer really was rescheduled we could remove by accident.

Also need to make sure that when we remove a stream we remove the node as part of the stream assignment.
If we didn't, if the same asset returned to this server we would not start up the monitoring loop.

Simplify migration logic in monitorStream, to be driven by leader only

Improved unit tests

Added failure when server not in peer list

Move command does not require server anymore

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-07-07 03:32:13 +02:00
Derek Collison
f8939b40bc Do not unsubscribe from direct access on leader stepdown, only stopping.
Also wait for stream to have replicas and leader for test.

Signed-off-by: Derek Collison <derek@nats.io>
2022-07-06 16:20:12 -07:00
Derek Collison
1ea608eabf Allows direct get to also do get next for subject with starting sequence
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-06 14:22:28 -07:00
Derek Collison
5690059dac Reserve a system queue group
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-06 13:16:13 -07:00
Derek Collison
90caf12d96 Attempt to fix flapping test
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-06 09:11:31 -07:00
Derek Collison
722ae548dd Fix data race
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-06 09:11:22 -07:00
Derek Collison
0c9ed8c10a Bump to 2.9.0-beta.2
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-05 14:01:28 -07:00
Derek Collison
c14fda51e7 Direct access to JetStream resources would be affected if across a leafnode that was down.
This allows a solciting leafnode config to ask that any JetStream cluster assets that are a current leader have the leader stepdown.

Signed-off-by: Derek Collison <derek@nats.io>
2022-07-05 12:35:09 -07:00
Derek Collison
1cc63c023d Merge pull request #3238 from nats-io/mirror_direct
Direct stream access enhancements
2022-07-05 10:38:48 -07:00
Derek Collison
52baa95e2a When stalled for MaxAckPending, expire all pull requests that are one shot, meaning had at least 1 delivered message
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-04 19:14:49 -07:00
Derek Collison
1b580c67f3 Make pull consumers FIFO per message, not per request.
This effectively means that requests with batch > 1 will process a message and go to the end of the line.

Signed-off-by: Derek Collison <derek@nats.io>
2022-07-04 13:05:57 -07:00
Derek Collison
81a9906ad9 Wait a bit longer for the direct sub
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-03 12:54:15 -07:00
Derek Collison
9b7c81c37e Some tests improvements on non-standard JS cluster setup
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-03 12:45:27 -07:00
Derek Collison
47bef915ed Allow all members of a replicated stream to participate in direct access.
We will wait until a non-leader replica is current to subscribe.

Signed-off-by: Derek Collison <derek@nats.io>
2022-07-03 11:08:24 -07:00
Derek Collison
4075721651 Allow direct msg get for stream to operate in queue group and allows mirrors to opt-in to the same group.
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-02 14:16:55 -07:00
R.I.Pienaar
97ad346c34 fix json tag for meta stream move
Signed-off-by: R.I.Pienaar <rip@devco.net>
2022-07-02 11:51:50 +02:00
Derek Collison
b7d94e1b52 Merge pull request #3232 from nats-io/leafnode_cluster_same
Close leafnode connection when same cluster name detected
2022-06-30 15:34:40 -07:00
Derek Collison
e6479dafd2 Close leafnode connection when same cluster name detected
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-30 15:34:22 -07:00
Jean-Noël Moyne
e46b00639a Resolves #3151 plus redo and improve subject mapping destination validation and error handling (#3231)
* 1: Allows spaces to be used inside {{}} subject mapping functions:

2: Rework and improve mapping destinations validation and error handling with much more helpful error messages, e.g.:

* Error adding mapping for "foo.*.*" to "bar.{{wildcard(1)}}" : invalid mapping destination: not using all of the token wildcard(s) in bar.{{wildcard(1)}}

* Error adding mapping for "myservice.request.*" to "myservice.request.{{ partition(10) }}.{{wildcard(1)}}" : invalid mapping destination: not enough arguments passed to the function in {{ partition(10) }}

* Error adding mapping for "myservice.request.*" to "myservice.request.{{ partition(10,2) }}.{{wildcard(1)}}" : invalid mapping destination: wildcard index out of range in {{ partition(10,2) }}: [2]

* Error adding mapping for "myservice.request.*" to "myservice.request.{{ partition(10,1) }}.{{wildcard()}}" : invalid mapping destination: not enough arguments passed to the function in {{wildcard()}}

* Error adding mapping for "myservice.request.*" to "myservice.request.{{ xxxpartition(10,1) }}.{{wildcard(1)}}" : invalid mapping destination: unknown function in {{ xxxpartition(10,1) }}

* Error adding mapping for "myservice.request.*" to "myservice. request.{{ xxxpartition(10,1) }}.{{wildcard(1)}}" : invalid mapping destination: invalid subject

* implement PR comments
2022-06-30 14:21:53 -07:00
Derek Collison
8a94e14fe7 Bump to 2.9.0-beta.1
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-29 15:42:11 -07:00
Ivan Kozlovic
3d3b854d72 Merge pull request #3229 from nats-io/pull_inactive_threshold
[FIXED] Pull consumer may be incorrectly removed after InactiveThreshold
2022-06-29 16:40:42 -06:00
Derek Collison
4a94a172c4 Merge pull request #3227 from nats-io/filtered_mirrors
Allow filtered stream mirrors
2022-06-29 15:39:07 -07:00
Matthias Hanel
6bd14e1b7a removed commented out code (#3228)
Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-06-29 20:31:12 +02:00
Matthias Hanel
6e52608936 [added] support for jwt account option DisallowBearer (#3127)
* [added] support for jwt account option DisallowBearer

change 3 out of 3. Fixes #3084
corresponds to:
https://github.com/nats-io/jwt/pull/177
https://github.com/nats-io/nsc/pull/495

update jwt library to 2.3.0

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-06-29 18:19:14 +02:00
Ivan Kozlovic
c519df7e0d [FIXED] Pull consumer may be incorrectly removed after InactiveThreshold
When creating a pull consumer with InactiveThreshold set, if the
application is doing pull requests with "no_wait" at regular interval
(lower than InactiveThreshold), the JS consumer should be considered
active and not deleted.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-06-29 10:15:09 -06:00
Derek Collison
a77d5941ef Allow filtered stream mirrors
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-29 08:12:38 -07:00
Ivan Kozlovic
f8b16f90be Tweak MQTT test
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-06-28 18:00:25 -06:00
Ivan Kozlovic
5ed212ce04 Rework startupComplete gate from PR #2360
The "InProcess" change make readyForConnections() possibly return
much faster than it used to, which could cause tests to fail.

Restore the original behavior, but in case of DontListen option
wait on the startupComplete gate.

Also fixed some missing checks for leafnode connections.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-06-28 17:36:39 -06:00
Matthias Hanel
dff1a33693 [Fixed] race on unlocked cfg access (#3223)
write originated in updateWithAdvisory while holding the lock

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-06-29 00:05:00 +02:00
Ivan Kozlovic
cb406edfcd Merge pull request #2360 from neilalexander/neilalexander/inprocessconn
[ADDED] In-process connections
2022-06-28 11:47:01 -06:00
Neil Alexander
558293e096 Fix the lock 2022-06-28 18:05:57 +01:00
Derek Collison
abc5905aa9 Merge pull request #3221 from nats-io/direct
Made direct get from a stream part of the $JS.API hierarchy vs separate.
2022-06-28 09:59:44 -07:00
Matthias Hanel
aba1da090b [ADD] account specific in/out msgs/bytes stats to CONNS (#3187)
* [ADD] account specific in/out msgs/bytes stats to CONNS

This subject $SYS.ACCOUNT.%s.SERVER.CONNS will now respond with account
specific datastats for Received and sent messages as well as number of slow
consumers for the account.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-06-28 18:59:29 +02:00
Matthias Hanel
aabaf6f106 [fixed] reload related races (#3222)
account.rm had races caused by reload copying rm from one account to
another

mset.store was used outsisde the lock

in rare cases the stasz message was not received in time.
Trigger automatically now

sometimes a statsz message received before reload cause issues.
try receiving a second time

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-06-28 18:36:13 +02:00
Neil Alexander
cedf08a1b7 Commit properly 2022-06-28 17:03:42 +01:00
Neil Alexander
ff696f00d8 Remainder of time after waiting for startupComplete, add waitgroup done call after createClient 2022-06-28 17:03:42 +01:00
Neil Alexander
5b04c49df9 Re-add startupComplete channel, make readyForConnections wait for it 2022-06-28 17:03:42 +01:00
Neil Alexander
9190feb05f Review comments @kozlovic 2022-06-28 17:03:40 +01:00
Neil Alexander
90d7e007c0 Update comments (re. review) 2022-06-28 17:02:47 +01:00
Neil Alexander
e9abc5801e Add InProcessConn, DontListen 2022-06-28 17:02:47 +01:00
Derek Collison
b8ef9b19a0 Made direct get from a stream part of the $JS.API hierarchy vs separate.
Also for direct get and for pull requests, if we are not on a client connection check how long we have been away from the readloop.
If need be execute in a separate go routine.

Signed-off-by: Derek Collison <derek@nats.io>
2022-06-28 08:53:48 -07:00
Derek Collison
b0c4ec69ab Merge pull request #3216 from nats-io/update_consumer_filtered_subj
Allow consumer filter subjects to be updated
2022-06-28 08:42:08 -07:00
Matthias Hanel
3421c49310 [Add] ability for operator to move streams (#3217)
Also added:
ability to reload tags
special tag (!jetstream) to remove peer from peer placement
$JS.API.SERVER.STREAM.MOVE subject to initiate move away from a server

This changes a detail about regular stream move as well.
Before differing cluster names where used to start/stop a transfer.
Now only the peer list and it's size relative to configured replica matter.
Once a transfer is considered completed, excess peers will be dropped
from the beginning of the list.
This allows transfers within the cluster as well.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-06-28 02:36:32 +02:00
Derek Collison
e02016db26 Fix race
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-27 16:05:03 -07:00
Derek Collison
92cd7821de Convert server mutex to RW.
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-27 16:05:03 -07:00
Derek Collison
830d659aec Do not default to sharing all client info for JetStream, only server and cluster info.
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-26 07:11:02 -07:00
Derek Collison
9154fca7f1 Allow consumer filter subjects to be updated
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-24 12:38:01 -07:00
Derek Collison
4de1766f40 Fix TLS tests and allow older test certs for now
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-24 09:43:42 -07:00