Commit Graph

5626 Commits

Author SHA1 Message Date
Derek Collison
efb91c4ade Upgrade to latest released client
Signed-off-by: Derek Collison <derek@nats.io>
2022-04-10 08:03:11 -07:00
Derek Collison
e7ff38a4ca Add consumerMemStore impl to allow proper replication of state.
Resolves #3006

Signed-off-by: Derek Collison <derek@nats.io>
2022-04-10 08:01:13 -07:00
Derek Collison
3663d595fc Disallow moving a stream that is already being moved
Signed-off-by: Derek Collison <derek@nats.io>
2022-04-07 17:09:55 -07:00
Matthias Hanel
5662141932 Adding unique_tag to ensure matching tags are not used twice (#3011)
Allows to not place a stream in the same availability zone twice.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-07 18:11:00 -04:00
Ivan Kozlovic
47776bdd36 Merge pull request #3013 from nats-io/ln_min_version
[ADDED] LeafNode `min_version` new option
2022-04-07 13:57:29 -06:00
Ivan Kozlovic
b5c9583ee2 Reject configuration with value below 2.8.0
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-07 12:49:34 -06:00
Matthias Hanel
f4c2302301 fix sleep in unit test to ensure updates have propagated (#3012)
Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-07 12:16:18 -04:00
Ivan Kozlovic
7fa2676353 Fixed comment typos and some rewording
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-07 09:22:51 -06:00
Ivan Kozlovic
9e6f965913 [ADDED] LeafNode min_version new option
If set, a server configured to accept leafnode connections will
reject a remote server whose version is below that value. Note
that servers prior to v2.8.0 are not sending their version
in the CONNECT protocol, which means that anything below 2.8.0
would be rejected.

Configuration example:
```
leafnodes {
    port: 7422
    min_version: 2.8.0
}
```
The option is a string and can have the "v" prefix:
```
min_version: "v2.9.1"
```
Note that although suffix such as `-beta` would be accepted,
only the major, minor and update are used for the version comparison.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-06 18:40:33 -06:00
Ivan Kozlovic
3485dcf2b9 Merge pull request #3010 from nats-io/js_add_filter_test
Add test that demonstrated the consumer filter perf degradation
2022-04-06 10:30:40 -06:00
Ivan Kozlovic
c78f7f343c Add test that demonstrated the consumer filter perf degradation
This is a follow up to PR #3008.

This test fails on v2.7.4 but passes on main.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-06 09:27:56 -06:00
Ivan Kozlovic
1691e9aaf6 Bump version to 2.8.0-beta.12
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-05 20:48:22 -06:00
Ivan Kozlovic
9bbb7acb50 Merge pull request #3008 from nats-io/js_first_matching_improvement
[IMPROVED] Stream with multiple subjects and consumer with filter
2022-04-05 20:46:50 -06:00
Derek Collison
ef9728997d During recovery check our guess on the last block.
Signed-off-by: Derek Collison <derek@nats.io>
2022-04-05 19:20:31 -07:00
Derek Collison
ab5e2344e0 When loading blocks in use len(mb.fss) to determine if we can use sfilter optimization.
Also check fs.lmb when the stream config is updated.

Signed-off-by: Derek Collison <derek@nats.io>
2022-04-05 18:49:21 -07:00
Ivan Kozlovic
371ce36712 [IMPROVED] Stream with multiple subjects and consumer with filter
This is more of a regression introduced in v2.7.3 (with PR #2848).
When the store has a list of subjects, finding the next message
to deliver would go through the subjects map and have to match
to find out if it is a subset (if the filter had a wildcard).
In situations where there were lots of subjects (for instance 1
message per subject), but the consumer did not filter on anything
specific, then this processing was becoming slow.

We now check that if the stream has a single subject (even with
wildcard) and the consumer filters on that exact subject, then
we can do a linear scan. We also do a linear scan if the number
of messages in the block is 1/2 the number of subjects in the
subjects map.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-05 18:19:17 -06:00
Matthias Hanel
2db7d9fe2f unit test to make sure tiered limits and stream moves work together (#3007)
This needs testing because stream move adjusts the replication factor

Because adjusting replication factor and moving is illegal, this case
does not need to be tested

In order to support one off configurations, added same modification
callout to super cluster as is used with cluster

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-05 18:11:04 -04:00
Matthias Hanel
d9da66d67e returns -1 for new unlimited/unset limits and tests/fixes info counts (#3002)
iterates on tiered limits

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-05 12:25:55 -04:00
Ivan Kozlovic
5f4f813c53 Bump to 2.8.0-beta.11
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-05 09:19:13 -06:00
Ivan Kozlovic
4211a35533 Merge pull request #3005 from nats-io/undo_send_bad_request
Undo sending bad request on no-interest in apiDispatch
2022-04-05 09:18:29 -06:00
Ivan Kozlovic
9b5797f63c Undo sending bad request on no-interest in apiDispatch
This broke cross-account functionality. Ported the test from the
Go client that showed the failure after PR#2997 was merged.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-05 08:51:28 -06:00
Derek Collison
92813d7370 Bump to 2.8.0-beta.10
Signed-off-by: Derek Collison <derek@nats.io>
2022-04-04 19:15:05 -07:00
Derek Collison
8e6764dfb3 Merge pull request #3001 from nats-io/asset-move
Allow streams and consumers to migrate between clusters.
2022-04-04 19:13:40 -07:00
Derek Collison
7e38ebcb6e Allow assets such as streams and their associated consumers to migrate between clusters.
The system will allow an update to a stream, and subsequently all attached consumers, to be placed in another cluster either directly or via tag placement.
The meta layer will scale the underlying peerset appropriately to straddle the two clusters for both the stream and consumers, taking into account the consumer type.
Control will then pass to the current leaders of the assets who will monitor the catchup status of the new peers.
(Note we can optimize this later to only traverse once across a GW for any given asset, but for now this is simpler)
Once the original leaders have determined the assets are synched it will pass leadership to a member of the new peerset.
Once the new leader has been elected, it will forward a request for the meta layer to shrink the peerset by removing the old peers.

Signed-off-by: Derek Collison <derek@nats.io>
2022-04-04 18:28:36 -07:00
Matthias Hanel
268a29e719 fix unit test that did not fail when header was modified (#3000)
Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-04 19:24:09 -04:00
Matthias Hanel
569328bc18 fix crash on shutdown with meta being nil (#2999)
Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-04 17:40:19 -04:00
Matthias Hanel
b7bc842c8b Add a config modification callback to createJetStreamCluster (#2998)
* Add a config modification callback to createJetStreamCluster

named createJetStreamClusterAndModHook allowing the generated config to
be altered prior to server start

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-04 17:39:58 -04:00
Ivan Kozlovic
e8b118bae5 Merge pull request #2997 from nats-io/fix_2995
[FIXED] JetStream: send "bad request" response for malformed API requests
2022-04-04 12:27:27 -06:00
Ivan Kozlovic
29ea280fe7 [FIXED] JetStream: send "bad request" response for malformed API requests
An example was a "consumer info" request with a consumer name that
had tokens, which is illegal. This results in the request being
dropped in apiDispatch() because there was no interest.
The server will now return a "bad request" error in such case.

Resolves #2995

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-04 11:25:55 -06:00
Ivan Kozlovic
ee1341fa17 Merge pull request #2996 from nats-io/ws_mqtt_in_varz
[ADDED] Monitoring: MQTT and Websocket blocks in `/varz` endpoint
2022-04-04 10:53:19 -06:00
Ivan Kozlovic
14f54b8dd7 [ADDED] Monitoring: MQTT and Websocket blocks in /varz endpoint
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-04 10:11:55 -06:00
Ivan Kozlovic
18bdabff35 Merge pull request #2994 from nats-io/add_rate_limited_warnings
[CHANGED] Rate limit (some) similar warnings
2022-04-01 18:19:34 -06:00
Ivan Kozlovic
366d217f44 Some changes based on review
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-01 17:55:33 -06:00
Ivan Kozlovic
19783a9f11 [CHANGED] Rate limit similar warnings
Some warnings, especially when dealing with JS limits that were
printed on a per-message basis, are now limited to ~1 per second
if the content of the warning is already found in a map.

This is also for "client" warnings, but the client porting of the
warning is not taken into account so that helps with reducing logging
for similar content, but coming from different clients.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-01 15:24:03 -06:00
Matthias Hanel
a77f95faa8 error handling and info when moving a stream from non existing tier (#2992)
adds unit test to test this scenario
improves reporting of correct error
only show info for non existing tiers where streams exist

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-01 14:21:35 -04:00
Matthias Hanel
33d9f189cc start using unit test TestJWTClusteredJetStreamTiers, size was off (#2988)
If both servers sent a remote update of their local use,
the limit was hit. But that limit wass to small by 200

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-01 12:51:28 -04:00
Derek Collison
3c09064117 Merge pull request #2991 from nats-io/seal_restart
[FIXED] Sealed streams would not recover on server restart.
2022-04-01 06:47:24 -07:00
Derek Collison
7f78d3e618 Not allowing streams to be created meant we could not recover on server restart.
Signed-off-by: Derek Collison <derek@nats.io>
2022-04-01 06:41:22 -07:00
Jaime Piña
32b17f7a7e Skip SystemLimitsPlacement if we can't get the desired leader (#2989) 2022-03-31 16:24:29 -07:00
Ivan Kozlovic
c917141df8 Merge pull request #2987 from nats-io/js_tweak_snapshot_defaults
Updated snapshot default chunk size and window size
2022-03-31 16:30:08 -06:00
Ivan Kozlovic
52200ab8bb Updated snapshot default chunk size and window size
Down from 256KB to 128KB for chunk and 32MB to 8MB.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-03-31 16:14:18 -06:00
Matthias Hanel
241bf5df0d Fixed wrong error check (#2986)
Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-03-31 18:03:06 -04:00
Ivan Kozlovic
d4e2fde45c Bump version to beta.7
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-03-31 12:36:15 -06:00
Matthias Hanel
64feb142a9 In Merge validate nkey and subsequent saveIfNewer error on invalid jwt (#2985)
Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-03-31 14:25:25 -04:00
Matthias Hanel
92f4dc986a added max_ack_pending setting to js account limits (#2982)
* added max_ack_penind setting to js account limits

because of the addition, defaults now have to be set later (depend on
these new limits now)

also re-organized the code to closer track how stream create looks

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-03-31 14:17:16 -04:00
Ivan Kozlovic
ee98a81469 Merge pull request #2984 from nats-io/data_race_and_flappers
Fixed data race and some flappers
2022-03-31 11:11:46 -06:00
Ivan Kozlovic
34650e9dd5 Fixed data race and some flappers
Data race that has been seen:
```
Read at 0x00c00134bec0 by goroutine 159:
  github.com/nats-io/nats-server/v2/server.(*client).msgHeaderForRouteOrLeaf()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:2935 +0x254
  github.com/nats-io/nats-server/v2/server.(*client).processMsgResults()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:4364 +0x2147
(...)
Previous write at 0x00c00134bec0 by goroutine 201:
  github.com/nats-io/nats-server/v2/server.(*Server).addRoute()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/route.go:1475 +0xdb4
  github.com/nats-io/nats-server/v2/server.(*client).processRouteInfo()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/route.go:641 +0x1704
```

Also fixed some flappers and removed use of `s.js.` since we have
already captured `js` in Jsz monitoring.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-03-31 10:05:34 -06:00
Derek Collison
96aa5eee7a Merge pull request #2983 from ripienaar/skipjz_nonjs
skips jsz on non js machines when leader only requested
2022-03-31 05:50:46 -07:00
R.I.Pienaar
4c4aa3e87f skips jsz on non js machines when leader only requested
This is a regression introduced in 055703f4fa
that leads to panics in management tooling

Signed-off-by: R.I.Pienaar <rip@devco.net>
2022-03-31 12:02:09 +02:00
Derek Collison
d634d237ca Bump version to 2.8.0-beta.6
Signed-off-by: Derek Collison <derek@nats.io>
2022-03-30 20:26:49 -07:00