Commit Graph

7637 Commits

Author SHA1 Message Date
Derek Collison
60d006bbeb Merge branch 'main' into dev 2023-06-09 18:25:17 -07:00
Derek Collison
975f004a74 Fixed issues with leafnode compression negotiation (#4230)
When a server would send an asynchronous INFO to a remote server it
would incorrectly contain compression information that could cause
issues with one side thinking that the connection should be compressed
while the other side was not.

It also caused the authentication timer to be incorrectly set which
would cause a disconnect.

Signed-off-by: Ivan Kozlovic <ijkozlovic@gmail.com>
2023-06-09 18:20:50 -07:00
Derek Collison
8c513ad2c2 [FIXED] DQ weighted subscribers across separate leafnode connections. (#4231)
Fix for properly distributed queue requests over multiple leafnode
connections.
When a leafnode server joins two accounts in a supercluster, we want to
make sure that each connection properly takes into account the weighted
number of subscribers in each account.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-09 17:50:17 -07:00
Derek Collison
2765e534eb Fix test and update copyright
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-09 15:09:15 -07:00
Derek Collison
ce2dcd3394 Fix for properly distributed queue requests over multiple leafnode connections.
When a leafnode server joins two accounts in a supercluster, we want to make sure that each connection properly takes into account the weighted number of subscribers in each account.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-09 14:43:59 -07:00
Derek Collison
13b6e2613c Reduce messages in chaos tests (#4229)
It doesn't really appear as though, for what these tests are trying to
prove, that an excessively large number of messages is required. Instead
let's drop the count a little in the hope that they run a bit faster.

Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-09 13:59:35 -07:00
Ivan Kozlovic
7ff0ea449a Fixed issues with leafnode compression negotiation
When a server would send an asynchronous INFO to a remote server
it would incorrectly contain compression information that could
cause issues with one side thinking that the connection should
be compressed while the other side was not.

It also caused the authentication timer to be incorrectly set
which would cause a disconnect.

Signed-off-by: Ivan Kozlovic <ijkozlovic@gmail.com>
2023-06-09 13:20:44 -06:00
Derek Collison
81154c40f5 Bump to 2.9.18-beta.2
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-09 09:29:48 -07:00
Derek Collison
a1f03513d8 Merge branch 'main' into dev 2023-06-09 09:29:13 -07:00
Neil Twigg
7de3568f39 Reduce messages in chaos tests
It doesn't really appear as though, for what these tests are trying to
prove, that an excessively large number of messages is required. Instead
let's drop the count a little in the hope that they run a bit faster.

Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-09 17:07:53 +01:00
Neil
9d2ae42956 [FIXED] Performance issues with checkAckFloor with large first stream sequence. (#4226)
Bail early if new consumer, meaning stream sequence floor is 0.
Decide which linear space to scan.
Do no work if no pending and we just need to adjust which we do at the
end.

Also realized some tests were named wrong and were not being run, or
were in wrong file.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-09 09:27:52 +01:00
Derek Collison
9eeffbcf56 Fix performance issues with checkAckFloor.
Bail early if new consumer, meaning stream sequence floor is 0.
Decide which linear space to scan.
Do no work if no pending and we just need to adjust which we do at the end.

Also realized some tests were named wrong and were not being run, or were in wrong file.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-08 18:45:03 -07:00
Derek Collison
40619659d5 [FIXED] Check for invalid stream name in sources (#4222)
- [X] Link to issue, e.g. `Resolves #4141`
 - [X] Tests added
 - [X] Branch rebased on top of current dev
- [X] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [x] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

Resolves #4141

### Changes proposed in this pull request:

Adds a check that the stream name of a stream source is valid and
associated new error if it isn't.
2023-06-08 15:14:25 -07:00
Jean-Noël Moyne
a979fdb4ba Fix and improve description for the two new errors
Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-06-08 11:01:54 -07:00
Derek Collison
19eba1b8c8 Merge branch 'main' into dev 2023-06-08 09:34:41 -07:00
Derek Collison
d3bde2c6e1 Send peer state when adding peers (#4224)
Currently `UpdateKnownPeers` doesn't send a peer state when a single
peer add operation is taking place, but it seems like this can
potentially race when there are lots of changes to the replica count
happening in rapid succession. Sending the peer state in all cases seems
to fix this issue and, so far in my testing, fixes the failground stream
update replicas test.

Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-08 09:31:49 -07:00
Jean-Noël Moyne
7ff114162c Adds the same check for valid stream name for Mirror
Fix test using invalid stream names

Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-06-08 07:49:47 -07:00
Jean-Noël Moyne
bd6c15d24e Adds a check that the stream name of a stream source is valid and associated new error if it isn't.
Adresses https://github.com/nats-io/nats-server/issues/4141

Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-06-08 07:49:46 -07:00
Neil Twigg
6d9955d212 Send peer state when adding peers
Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-08 15:25:18 +01:00
Derek Collison
c85db15a2c Update Go to 1.19.10 (#4223) 2023-06-08 04:39:20 -07:00
Derek Collison
b5c0170527 Turn off leaf compression to stabilize test for now
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-08 04:37:07 -07:00
Byron Ruth
3ca99adcca Update Go to 1.19.10
Signed-off-by: Byron Ruth <byron@nats.io>
2023-06-08 07:10:27 -04:00
Derek Collison
fd082ee8a5 Merge branch 'main' into dev 2023-06-07 14:31:53 -07:00
Derek Collison
779978d817 Extended replay leafnode test to confirm mirror functionality
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-07 14:01:43 -07:00
Derek Collison
822ad00d50 Bump to 2.9.18-beta.1
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-05 14:14:35 -07:00
Derek Collison
f342f6a758 Merge branch 'main' into dev 2023-06-05 14:13:18 -07:00
Derek Collison
2e2ac33920 [IMPROVED] When R1 consumers were recreated with the same name when they became inactive. (#4216)
When consumers were R1 and the same name was reused, server restarts
could try to cleanup old ones and effect the new ones. These changes
allow consumer name reuse more effectively during server restarts.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-05 14:04:53 -07:00
Derek Collison
df5df3ce99 Panic fixes (#4214)
- [ ] Link to issue, e.g. `Resolves #NNN`
 - [ ] Documentation added (if applicable)
 - [ ] Tests added
- [ ] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [ ] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [x] Build is green in Travis CI
- [x] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

Resolves panics in the code.

### Changes proposed in this pull request:

 - This PR fixes some of the panics in the code
2023-06-05 13:02:05 -07:00
Derek Collison
4ac45ff6f3 When consumers were R1 and the same name was reused, server restarts could try to cleanup old ones and effect the new ones.
These changes allow consumer name reuse more effectively during server restarts.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-05 12:48:18 -07:00
Nikita Mochalov
5141b87dff Refactor code 2023-06-05 22:42:28 +03:00
Nikita Mochalov
4c181bc99a Use sentinel error 2023-06-05 22:41:09 +03:00
Nikita Mochalov
f71c49511b Fix client panic on absent server field 2023-06-05 15:27:45 +03:00
Derek Collison
af318be5db Merge branch 'main' into dev 2023-06-04 13:30:15 -07:00
Derek Collison
64e3bf82ed Fix PurgeEx replay with sequence & keep succeeds (#4213)
PR https://github.com/nats-io/nats-server/pull/4212 fixed the issue I
reported in https://github.com/nats-io/nats-server/issues/4196.

However, I believe there might be a bug when both `sequence` and `keep`
are set during recovery.
In the `PurgeEx` the following check is done (for both `filestore.go`
and `memstore.go`):
```go
	if sequence > 1 && keep > 0 {
		return 0, ErrPurgeArgMismatch
	}
```

The `TestJetStreamClusterPurgeExReplayAfterRestart` also triggers this
case, meaning that during the test this error is returned but it
succeeds because the purge was already performed. Is this intended
behaviour?

To elaborate a bit more, I believe the following happens:
- when running the purge normally it will properly run the `keep` (since
it's not combined with `sequence` yet)
- when replaying the purge though, the `sequence` is added to the
`keep`, which errors out in the above if

Which means that during normal operation all will be well, but purges
with `keep` will be ignored upon replaying.

I'm proposing to remove the `sequence > 1 && keep > 0` check and
subsequent error. Which, for reference, was introduced in
https://github.com/nats-io/nats-server/pull/3121.
Hoping this ensures that during recovery, purges that haven't executed
yet will still be executed.

An alternative approach, which wouldn't remove the error: not allow
combining `sequence` and `keep` normally and only allowing it during
recovery. Which would preserve the current behaviour, and correctly
apply `sequence+keep` during recovery still. However, not sure if it's
possible to know if we're in "recovery mode" from within `PurgeEx`.

Resolves https://github.com/nats-io/nats-server/issues/4196
2023-06-04 13:29:53 -07:00
Maurice van Veen
132567de39 Fix PurgeEx replay with sequence & keep succeeds 2023-06-04 11:56:28 +02:00
Derek Collison
30d9dfd305 Merge branch 'main' into dev 2023-06-03 18:17:28 -07:00
Derek Collison
e1f8064e9e [FIXED] Make sure to process extended purge operations correctly when being replayed. (#4212)
This is an extension to the excellent work by @MauriceVanVeen and his
original PR #4197 to fully resolve for all use cases.

Signed-off-by: Derek Collison <derek@nats.io>

Resolves #4196
2023-06-03 18:12:22 -07:00
Derek Collison
dee532495d Make sure to process extended purge operations correctly when being replayed on a restart.
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-03 17:49:45 -07:00
Derek Collison
eb09ddd73a [FIXED] Killed server on restart could render encrypted stream unrecoverable (#4210)
When a server was killed on restart before an encrypted stream was
recovered the keyfile was removed and could cause the stream to not be
recoverable.

We only needed to delete the key file when converting ciphers and right
before we add the stream itself.

Signed-off-by: Derek Collison <derek@nats.io>

Resolves #4195
2023-06-03 17:36:10 -07:00
Derek Collison
449b429b58 [FIXED] Data races detected in internal testing (#4211)
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-03 16:20:56 -07:00
Derek Collison
238282d974 Fix some data races detected in internal testing
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-03 13:58:15 -07:00
Derek Collison
4c1b93d023 Make sure to put the keyfile back if we did not recover the stream.
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-03 11:21:58 -07:00
Derek Collison
d5ae96f54d When a server was killed on restart before an encrypted stream was recovered the keyfile was removed and could cause the stream to not be recoverable.
We only needed to delete the key file when converting ciphers and right before we add the stream itself.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-03 11:21:47 -07:00
Derek Collison
df901dc1aa Merge branch 'main' into dev 2023-06-02 16:45:07 -07:00
Derek Collison
22c97d67ff [FIXED] Daisy chained leafnodes sometimes would not propagate interest (#4207)
When we were optimizing for single cluster and large numbers of
leafnodes we inadvertently broke a daisy chained scenario where a server
was a spoke and a hub within a single hub server.

So interest on D would not propagate properly to server A as a
publisher.

```
     B
   /    \
A       C -- D (SUB)
 |
PUB
```
2023-06-02 16:43:21 -07:00
Derek Collison
b2ac621212 Bump to 2.9.18-beta (#4182) 2023-06-02 16:40:23 -07:00
Derek Collison
1bce79750e When we were optimizing for single cluster but large number of leafnodes we inadvertently broke a daisy chained scenarion where a server was a spoke and a hub with a single hub cluster.
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-02 15:16:36 -07:00
Derek Collison
8278a7a0b2 Merge branch 'main' into dev 2023-06-02 12:44:02 -07:00
Derek Collison
25ad3cd4af Only check ack floor if we are interest policy based. (#4206)
Saw performance issue with a user a limits based stream with large
number of consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-02 12:43:06 -07:00
Derek Collison
27bbfb7a85 Only check ack floor if we are interest policy based.
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-02 11:04:00 -07:00