Commit Graph

5420 Commits

Author SHA1 Message Date
Derek Collison
bcf5da04e3 Merge branch 'main' into dev 2023-08-22 06:50:36 -07:00
Derek Collison
e5d208bf33 When moving streams, we could check too soon and be in a gap where the replica peer has not registered a catchup request.
This would cause us to think the replica was caughtup incorrectly and drop our leadership, which would cancel any cacthup requests.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 20:07:48 -07:00
Derek Collison
e088583cd3 Bump to 2.10.0-beta.50
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 15:59:53 -07:00
Derek Collison
f0e2765b44 Fixes for merge conflicts from main
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 15:55:31 -07:00
Derek Collison
fb8525b713 Merge branch 'main' into dev
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 15:55:00 -07:00
Derek Collison
2fc3f45ea1 [FIXED] Durable pull consumers could get cleaned up incorrectly on leader change. (#4412)
Fix for a bug that would allow old leaders of pull based durables to
delete a consumer from an inactivity threshold timer inadvertently.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 15:35:44 -07:00
Derek Collison
6e3ae20650 [FIXED] Fixed deadlock when checkAndSync was being called as part of storing message (#4411)
We violated the locking pattern, so we now make sure we do this in a
separate Go routine and put checks to only run it once.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 15:28:58 -07:00
Derek Collison
0a86bf4a9a Should reset to false, not true when done
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 14:57:17 -07:00
Derek Collison
43314fd439 Fix for a bug that would allow old leaders of pull based durables to delete a consumer from an inactivity threshold.
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 14:53:09 -07:00
Neil Twigg
d720a6931c Use own subject for LDM event
Signed-off-by: Neil Twigg <neil@nats.io>
2023-08-21 22:03:26 +01:00
Neil Twigg
7cc5838a6d Send shutdown event on LDM so that R1 assets do not get assigned to the LDM node
Signed-off-by: Neil Twigg <neil@nats.io>
2023-08-21 21:29:01 +01:00
Derek Collison
e018705a08 Fixed deadlock when checkAndSync was being called as part of storing message.
We violated the locking pattern, so we now make sure we do this in a separate Go routine and put checks to only run it once.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 12:12:36 -07:00
Jean-Noël Moyne
62f62d4071 Adds sfs to sourceInfo
Adds sfs to SourceInfo such that transforms with just a subject filter (and no transformation, meaning that the transform pointer in streamInfo is nil) can still be reflected in SourceInfo, which is important since the filtering is still happening, just no transformation as well.

Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-08-19 12:26:42 -07:00
Neil Twigg
c437157c1f Recover in consumer assignment when asset already existed
Signed-off-by: Neil Twigg <neil@nats.io>
2023-08-17 23:22:10 +01:00
Neil Twigg
3c85490dc0 Backport test helper tweak
Signed-off-by: Neil Twigg <neil@nats.io>
2023-08-17 15:28:48 +01:00
Neil Twigg
c0636d117f Tweak consumer replica scaling, add unit test for orphaned consumer subjects
Signed-off-by: Neil Twigg <neil@nats.io>
2023-08-17 15:27:29 +01:00
Neil Twigg
19397a5683 Don't set block profile rate
Signed-off-by: Neil Twigg <neil@nats.io>
2023-08-16 17:00:07 +01:00
Jean-Noël Moyne
0cc43acb84 Fix Nats-Stream-Source header parsing when using multi-filter transforms
Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-08-15 19:22:09 -07:00
Jean-Noël Moyne
c2d3ef1021 Fix potential out of range for stream source transform update.
Clean up un-needed if statement as it's ok to call NewSubjectTransform with an empty destination (ie no transformation) it will return nil

Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-08-15 16:35:19 -07:00
Waldemar Quevedo
740e5ddc37 Add some jitter to leafnode remotes reconnect
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-15 07:36:37 -07:00
Neil
8717b050e9 [IMPROVED] $SYS.REQ.SERVER.PING.PROFILEZ always honored (#4393) 2023-08-14 22:03:46 +01:00
Jean-Noël Moyne
61a0555336 Call SetBlockProfileRate even it the profiling port is not set
Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-08-14 10:58:20 -07:00
R.I.Pienaar
1d916ef9c7 Adds a missing json encoding tag
Signed-off-by: R.I.Pienaar <rip@devco.net>
2023-08-14 17:41:02 +03:00
Waldemar Quevedo
3a20f66535 config: parsed empty config only show warnings
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-13 23:59:50 -07:00
Waldemar Quevedo
412dee67f1 config: allow empty configs, but prevent bad configs
- Adds reporting the line with the bad key position
  that makes the config invalid.

- Fixes a few tests with trailing braces which were
  being handled as keys and ignored before.

Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-13 23:13:32 -07:00
Jean-Noël Moyne
40b8aa434b Remove part of the test that expects an error since now you can always get the profilez through the system account request
Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-08-13 18:00:08 -07:00
Jean-Noël Moyne
b839c53abc [ADDED] Full StreamSource (filters, transforms) functionality to stream mirror (#4354)
- [X] Tests added
- [X] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [X] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [x] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

Follow up to #4276 extending to Mirror the full StreamSource
functionality.

---------

Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-08-12 15:17:48 -07:00
Jean-Noël Moyne
bb53b54810 Remove the gate on a profiling port being defined in the server config for the profilez request to return profiling data even if the server doesn't have a profiling port set.
Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-08-12 14:48:38 -07:00
Neil Twigg
3c9c124b94 When checking replica count when updating retention, make sure stream assignment is set first
Signed-off-by: Neil Twigg <neil@nats.io>
2023-08-11 14:15:49 +01:00
Jean-Noël Moyne
fc41ab1a5a Adds LDM and KICK server $SYS requests
Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-08-10 17:08:09 -07:00
Waldemar Quevedo
37d3220dfb test: fixes for TestLeafNodeSlowConsumer (#4388)
It would fail sometimes locally otherwise...
```
=== RUN   TestLeafNodeSlowConsumer
    leafnode_test.go:7069: got: 0, expected: 1
--- FAIL: TestLeafNodeSlowConsumer (0.29s)
=== RUN   TestLeafNodeSlowConsumer
    leafnode_test.go:7069: got: 0, expected: 1
--- FAIL: TestLeafNodeSlowConsumer (0.28s)
=== RUN   TestLeafNodeSlowConsumer
--- PASS: TestLeafNodeSlowConsumer (0.28s)
=== RUN   TestLeafNodeSlowConsumer
    leafnode_test.go:7069: got: 0, expected: 1
--- FAIL: TestLeafNodeSlowConsumer (0.28s)
=== RUN   TestLeafNodeSlowConsumer
--- PASS: TestLeafNodeSlowConsumer (0.28s)
```
2023-08-10 01:12:21 -07:00
Waldemar Quevedo
f16582e2a4 test: update TestWSTLSVerifyClientCert for go1.21
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-09 21:50:46 -07:00
Waldemar Quevedo
7c9ea91296 test: fix TestLeafNodeSlowConsumer flake
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-09 21:35:24 -07:00
Waldemar Quevedo
3cec8dc451 test: fix TestNoRaceJetStreamMemstoreWithLargeInteriorDeletes flake
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-09 13:33:48 -07:00
Waldemar Quevedo
af766b78ce test: bump timeout from TestFileStoreNewWriteIndexInfo to 3ms
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-09 11:16:44 -07:00
Waldemar Quevedo
4625234bba test: delay slightly between filestore test permutations
This is to try to prevent test failing due to trying to access
the tempdir while it is being tore down.
(go issue: https://github.com/golang/go/issues/43547)

```
=== RUN   TestFileStoreMsgBlkFailOnKernelFaultLostDataReporting/AES-GCM-S2
    filestore_test.go:5195: ------------> 128
    testing.go:1225: TempDir RemoveAll cleanup: unlinkat ./TestFileStoreMsgBlkFailOnKernelFaultLostDataReportingAES-GCM-S23605508670/001/msgs: directory not empty
--- FAIL: TestFileStoreMsgBlkFailOnKernelFaultLostDataReporting (0.02s)
```

Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-09 10:53:13 -07:00
Piotr Piotrowski
27dc50eb8f [ADDED] Filter Healthz results based on stream and consumer names, add 'details` param
Signed-off-by: Piotr Piotrowski <piotr@synadia.com>
2023-08-09 16:44:45 +02:00
Waldemar Quevedo
8b7dfe7d74 monitoring: track slow consumers per connection type
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-09 05:57:42 -07:00
Neil Twigg
d7f76da597 Allow switching from limits-based to interest-based retention in stream update
Signed-off-by: Neil Twigg <neil@nats.io>
2023-08-09 11:46:49 +01:00
Neil
6eb77fd46b test: fix TestAccountImportCycle flake (#4381)
Add extra flushes to make test more precise and try to avoid timeouts

```
=== RUN   TestAccountImportCycle
    accounts_test.go:3447: require no error, but got: nats: timeout
--- FAIL: TestAccountImportCycle (1.01s)
```
2023-08-09 11:39:52 +01:00
Neil
617d69d6c7 Match --signal PIDs with globular-style expression. (#4370)
When multiple instances are running on the machine a PID argument
suffixed with a '*' character will signal all matching PIDs.

Example: `nats-server --signal reload=*`

 - [ ] Link to issue, e.g. `Resolves #NNN`
 - [ ] Documentation added (if applicable)
 - [X] Tests added
 - [X] Branch rebased on top of current ~~main~~ dev
- [X] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [ ] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)
2023-08-09 11:16:56 +01:00
Waldemar Quevedo
14a56e28dd test: fix TestAccountImportCycle flake
add extra flushes to make test more precise

Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-08 23:41:18 -07:00
Waldemar Quevedo
e68c411b74 test: fix TestMQTTTLSVerifyAndMap on Go 1.21
reported error changed slightly in Go 1.21

```
=== RUN   TestMQTTTLSVerifyAndMap
=== RUN   TestMQTTTLSVerifyAndMap/no_filtering,_client_does_not_provide_cert
    mqtt_test.go:1033: Unexpected error: Error reading: remote error: tls: certificate required
--- FAIL: TestMQTTTLSVerifyAndMap (0.04s)
```

Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-08 23:10:29 -07:00
Waldemar Quevedo
1492cf717f test: fix TestFileStoreNewWriteIndexInfo hanging
t.Fatalf being called while holding a lock would
sometimes leave builds hanging.

Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-08 16:41:15 -07:00
Waldemar Quevedo
0ffd455e32 test: update TestNoRaceJetStreamServiceImportAccountSwapIssue flake (#4376)
Let pull consumer in test fetch messages for slightly longer instead of
at the same time as the producer, to avoid failing due to missing a few
messages:

```
=== RUN   TestNoRaceJetStreamServiceImportAccountSwapIssue
    norace_test.go:1194: Expected to receive 14982 msgs, only got 14981
--- FAIL: TestNoRaceJetStreamServiceImportAccountSwapIssue (3.03s)
```
2023-08-08 02:01:44 -07:00
Waldemar Quevedo
b081f8c2ea test: update TestNoRaceJetStreamServiceImportAccountSwapIssue flake
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-08 01:07:19 -07:00
Tomasz Pietrek
54fe8cb14f Fix race in consumer create
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-08-08 09:16:44 +02:00
Waldemar Quevedo
2630e9b597 test: bump timeout from TestAccountReloadServiceImportPanic
It can take slightly longer in a testing environment.

Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-07 16:42:12 -07:00
Jason Volk
9c4ae764a1 Match --signal PIDs with globular-style expression.
When multiple instances are running on the machine a PID argument suffixed with
a '*' character will signal all matching PIDs.

Example: `nats-server --signal reload=*`

Signed-off-by: Jason Volk <jason@zemos.net>
2023-08-07 10:16:05 -07:00
Derek Collison
6ca7887992 [IMPROVED] Delete blocks performance (#4371)
Track deleted with single avl.SeqSet dmap for now vs old method for
memory store.

For fileStore, we were trying to be too smart to save space at the
expense of encoding time, so revert back to simple version that is much
100x faster.
 
Size of encoding may be a bit bigger then we wanted, but we want to
prefer speed over size.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-07 09:18:48 -07:00