Commit Graph

7278 Commits

Author SHA1 Message Date
Derek Collison
ce0d8514be Merge branch 'main' into dev 2023-04-07 05:32:05 -07:00
Derek Collison
84c0d48815 [FIXED] Make sure on reverse match to compensate for wider target subjects. (#4032)
If we have a wider subject, meaning more tokens, but our subs end with a
wildcard token make sure that matches properly.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-06 16:03:52 -07:00
Derek Collison
dbde8aba36 Make sure on reverse match to compensate for wider target subjects
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-06 14:27:04 -07:00
Derek Collison
02122a233d [IMPROVED] Finer grained locking for /healthz with large number of JetStream assets. (#4031)
For checking the health of jetstream, do not hold the lock as we
traverse the streams and consumers.

 Signed-off-by: Derek Collison <derek@nats.io>
2023-04-06 12:37:06 -07:00
Derek Collison
c16915bff4 For checking the health of jetstream, do not hold the lock as we traverse the streams and consumers.
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-06 11:56:55 -07:00
Derek Collison
33451e5d56 Fix for API data race. (#4030)
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-06 10:45:29 -07:00
Derek Collison
f02f379468 Move stepdowns of streams and consumers to not be inline. (#4029)
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-06 10:45:16 -07:00
Derek Collison
a60f623842 Fix flapping test since claims not processed inline now
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-06 10:39:30 -07:00
Derek Collison
dee5229f9b Fix data race
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-06 10:37:31 -07:00
Derek Collison
0d2269b1e9 Move stepdowns of streams and consumers to not be inline
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-06 10:27:36 -07:00
Derek Collison
7547093b24 _EMPTY_
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-06 08:38:30 -07:00
Derek Collison
ff8701b724 Merge branch 'main' into dev 2023-04-06 08:37:11 -07:00
Derek Collison
1e80ea68b3 [IMPROVED] Optimize non-inline direct gets to not use simple go routines. (#4028)
This was causing regression in certain KV performance tests.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-06 08:35:37 -07:00
Derek Collison
2da50512e2 Optimize non-inline direct gets to not use simple go routines
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-06 07:50:57 -07:00
Derek Collison
7b82384fd7 ADDED hard_delete option on resolver configuration (#3783)
- [x] Link to issue, e.g. `Resolves #NNN`
 - [ ] Documentation added (if applicable)
 - [ ] Tests added
- [x] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [x] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [x] Build is green in Travis CI
- [x] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

Resolves #3782

### Changes proposed in this pull request:

- Add hard_delete option in the resolver config and use it to set the
deleteType in NewDirAccResolver

/cc @nats-io/core
2023-04-06 07:04:03 -07:00
Derek Collison
a5326c97ef [IMPROVED] Head of line improvements (#4027)
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-05 22:03:43 -07:00
Derek Collison
e76b0b9b96 Move check for out of resources which would want a read lock out of inline processing
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-05 20:28:19 -07:00
Derek Collison
b6ebf0fe43 RWMutex does not help here and could hurt
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-05 20:26:45 -07:00
Derek Collison
81541361dc Alternative fix for issue 4014, where we always send pings on ROUTER connections (#4016)
Alternative fix for issue
https://github.com/nats-io/nats-server/issues/4014: where we always send
pings for ROUTER, GATEWAY and LEAF spoke connections.

This is my own original work that I license to the project

[ x] Link to issue, e.g. Resolves #NNN
 Documentation added (if applicable)
 Tests added
[ x] Branch rebased on top of current main (git pull --rebase origin
main)
[ x] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
[x] Build is green in Travis CI
[ x] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)
Resolves #

Changes proposed in this pull request:
- in processPingTimer, add explicit test for ROUTER connection, and only
hold off sending ping based on receive ping
2023-04-05 20:16:22 -07:00
Derek Collison
fd14e75379 File store compression (#4004)
This PR adds internal support for compression of message blocks in the
file store. Message blocks will be compressed when they reach their
maximum configured block size and we have moved onto the next block, so
as to avoid slowing down on-going writes as far as possible.

Compressed blocks are preambled with a small amount of metadata
describing the compression algorithm (although at this point only S2
compression is supported) and compression takes place before encryption
(so as to maximise the efficiency of the compression where a stream
contains redundant or repeated information in the messages).

This PR **does not** provide any way to configure compression from the
stream level yet (or indeed any other level exposed to clients or
operators). It also does not go out of its way to compress old
uncompressed blocks after turning on compression unless they are
compacted or truncated. These omissions will be addressed in follow-up
PRs.

New permutations have been added to the file store tests to also ensure
test coverage when compression is enabled.

We should note that, when file store encryption is not in use,
storage-level or filesystem-level deduplication or compression (such as
those provided by some disk arrays or filesystems like ZFS) may end up
being considerably more space-efficient than file store compression
would. Conversely, when file store encryption is enabled, enabling
compression at the file store level may achieve better ratios.

Signed-off-by: Neil Twigg <neil@nats.io>
2023-04-05 12:42:13 -07:00
Neil Twigg
193a69d786 File store compression
Signed-off-by: Neil Twigg <neil@nats.io>
2023-04-05 15:16:31 +01:00
Sandy Kellagher
5ae83b7983 Alternative fix for issue 4014, where we always send pings on ROUTER connections, updating c.lastIn timestamp on receiving client data or ping 2023-04-05 13:03:46 +01:00
Julien Viard de Galbert
1b1610f2da Adding hard_delete option on resolver configuration
Signed-off-by: Julien Viard de Galbert <jviarddegalbert@scaleway.com>
2023-04-05 10:05:48 +02:00
Derek Collison
4525bdea0d [IMPROVE] Improvements and fixes to stream source consumer creation on leadership change or on stream config source updates (#4009)
- [X] Tests added
- [X] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [X] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [x] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)
2023-04-04 20:09:15 -07:00
Derek Collison
ce115ab1e6 Allow republish for mirroring/sourcing streams (#4010)
- [X] Link to issue, e.g. `Resolves #NNN`
 - [ ] Documentation added (if applicable)
 - [ ] Tests added
- [X] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [X] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [ ] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

Resolves #4000

### Changes proposed in this pull request:

Removes the check on the republish filter needing to have an overlap
with the listened subjects, as with the new stream subject
transformation changes we do not assume anymore that the subjects in a
stream must match the subjects being listened to.

This allows the use of republish for mirroring and/or sourcing streams
2023-04-04 18:38:27 -07:00
Derek Collison
87d7263026 Fix to new system internal callback format for new callbacks
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-04 15:21:34 -07:00
Derek Collison
0453596223 Bump to 2.10.0-beta.31
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-04 15:16:43 -07:00
Derek Collison
d14968cb4f Bump to 2.9.16-RC.6
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-04 15:15:24 -07:00
Derek Collison
03d5cf0871 Merge branch 'main' into dev 2023-04-04 15:14:41 -07:00
Derek Collison
2167f39ccd [IMPROVED] Do not block routes, leafnodes or gateways on received internal system msgs. (#4026)
To increase system stability in large NATS systems we do not want to
process inbound system messages that are destined to be processed by the
server itself to be processed inline on the inbound connection.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-04 15:13:02 -07:00
Derek Collison
3036bb8bc5 Fix for test
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-04 14:43:46 -07:00
Derek Collison
208f4625df Possibly de-race sysRequest (#4017)
This may fix a race condition in `sysRequest` where multiple inbox
responses could try to mutate the same input object, so instead we'll
let it instantiate its own, although it isn't clear to me yet why that
would happen in the first place.

Signed-off-by: Neil Twigg <neil@nats.io>
2023-04-04 14:40:11 -07:00
Derek Collison
b14a400df6 Fix for debugSubscribers and claims test
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-04 14:36:16 -07:00
Derek Collison
3551c7b2bf Convert over zReq to not have to call msgparts
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-04 14:23:50 -07:00
Derek Collison
c323ec3086 Non-inline system callbacks need hdr and msg already split due to client context to split
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-04 14:15:30 -07:00
Derek Collison
9f95e993e2 Do not inline inbound system messages to avoid blocking routes, gateways etc.
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-04 13:53:21 -07:00
Neil Twigg
03a5a4deaf Possibly de-race sysRequest
Signed-off-by: Neil Twigg <neil@nats.io>
2023-04-04 10:30:59 +01:00
Derek Collison
d5a525bde0 Bump to 2.9.16-RC.5
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 21:23:35 -07:00
Derek Collison
c5e19e19e7 Merge branch 'main' into dev 2023-04-03 21:22:53 -07:00
Derek Collison
34922df706 [IMPROVED] Make sure to never process next message requests inline. (#4022)
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 21:22:31 -07:00
Derek Collison
ebe4f8957f Spelling based on review feedback
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 21:08:59 -07:00
Derek Collison
dcbefd5cc4 We can receive these on push consumers, so error if we do
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 21:07:08 -07:00
Derek Collison
07b34f707f Make sure to never process next message requests inline
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 20:50:01 -07:00
Derek Collison
c9350a2d57 Only apply consumer entries if not recovering. (#4021)
These entries on restart should not be re-processed. This was causing
some failures on consumer tests that involved restarts.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 19:42:05 -07:00
Derek Collison
b0c3cf0dbd Only apply consumer entries if not recovering
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 17:22:50 -07:00
Derek Collison
1ae51b23a9 [ADDED] Multiple routes and ability to have per-account routes (#4001)
New configuration fields:
```
cluster {
   ...
   pool_size: 5
   accounts: ["A", "B"]
}
```

The configuration `pool_size` in the example above means that this
server will create 5 routes to a remote server, assuming that that
server has the same `pool_size` setting.

Accounts (which are not part of the `accounts[]` configuration)
are assigned a specific route in this pool, and this will be the
same route on all servers in the cluster.

Accounts that are defined in the `accounts` field will each have
a dedicated route connection. This will allow suppression of the
account name in some of the route protocols, reducing bytes transmitted
which may increase performance.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2023-04-03 15:33:46 -07:00
Derek Collison
01a2c0472d Merge branch 'main' into dev 2023-04-03 15:33:12 -07:00
Derek Collison
9db08041c6 [IMPROVED] Raft layer improvements (#4020)
Two changes here.

1. Revert of a change regarding handling of leadership transfer entries
that caused instability in some tests and a bad upgrade experience.

2. Make sure if we detect that a stream or a consumer is already running
the monitoring routines that we do not stop the underlying raft group
since it will be being used in the monitor routine and would stall that
asset.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 15:24:48 -07:00
Derek Collison
7d3ec51d79 Fix for flapping test
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 14:46:59 -07:00
Derek Collison
59175c491f Fix for a datarace
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 14:46:57 -07:00