Commit Graph

4983 Commits

Author SHA1 Message Date
Marco Primi
bbf42c1f57 Use dynamic port number in benchmark 2023-09-01 12:58:52 -07:00
Jean-Noël Moyne
db96238ad9 Enables 0s deduplication window duration when the stream has sources (#4476)
- [X] Link to issue, e.g. `Resolves #NNN`
- [X] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [X] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [X] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

Resolves #4459

Allows the user to set the deduplication window duration to 0s when the
stream has sources defined. Remember that if the stream in question is
also listening on subjects as well as sourcing the deduplication window
is the same for sourced and listened messages.

---------

Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-09-01 12:47:14 -07:00
Derek Collison
aca74d0b27 Bump to 2.9.22-RC.4
Signed-off-by: Derek Collison <derek@nats.io>
2023-09-01 09:44:53 -07:00
Waldemar Quevedo
ed8b50d943 Fix monitoring server connz idle time sorting (#4463)
- Changed how [`byIdle`](887a4ae692/server/monitor_sort_opts.go (L97))
struct compares the idle times (was subtracting the start time from the last activity time).

 - Added tests for `byIdle`.

- Changed and simplified the test
[`TestConnzSortedByIdle`](8a9f441c40/server/monitor_test.go (L1185))
(No need for the clients to publish and subscribe if we are manually
changing the client's `last` time, and we only need to test the sorting
order). Also the test was not catching the problem because the clients
`start` time was the same, now every client has a different start time.

Resolves #4462
2023-09-01 09:44:19 -07:00
Derek Collison
4422a95a8e We should update accounting before clearing ebit
Signed-off-by: Derek Collison <derek@nats.io>
2023-09-01 09:31:12 -07:00
Derek Collison
c679f9d7f6 Added in detail info when failing to load in a message for a consumer.
E.g. `Unexpected partial cache error looking up message for consumer '$G > TEST > dlc'`

Signed-off-by: Derek Collison <derek@nats.io>
2023-09-01 09:06:29 -07:00
Derek Collison
3a39786972 When we fail to deliver a message for a consumer, either through didNotDeliver() or LoadMsg() failure re-adjust delivered count and waitingRequest accounting.
Signed-off-by: Derek Collison <derek@nats.io>
2023-09-01 08:48:28 -07:00
Pierre Mdawar
d24d51292f Fix monitoring server connz idle time sorting 2023-09-01 14:32:08 +03:00
Derek Collison
afb052651a Sending too fast to have replicas be caught up enough to register direct subs
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-31 15:16:19 -07:00
Derek Collison
d7ea3b94d9 [FIXED] Check for checksum violations for all records and before any sequence processing. (#4465)
Also small bug fix for leaking fds under certain scenarios during
corruption.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-31 15:08:04 -07:00
Derek Collison
a45281d51f Added check to test
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-31 14:00:14 -07:00
Pierre Mdawar
6d6d3cfa55 Fix Content-Type header in /healthz when status is not 200 OK (#4437)
- Added a new internal function `handleResponse` that accepts the HTTP 
  status code and sets it after setting the headers
- Added tests for the `/healthz` endpoint for the `ok`, `error` and `unavailable` statuses
- Changed the IETF API health check URL to 
https://datatracker.ietf.org/doc/html/draft-inadarei-api-health-check

Resolves #4436
2023-08-31 13:55:20 -07:00
Derek Collison
c110ceea94 Check for checksum violations for all records and before sequence processing.
Also fix for bitrot test and a small bug fix for a leaking fd.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-31 13:53:28 -07:00
Ivan Kozlovic
9a9e84ea5c Fix leaf connection missing LS+ sometimes
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-31 10:06:02 -07:00
Derek Collison
2834142bdd Revert lock guard
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-31 08:59:15 -07:00
Derek Collison
8a9f441c40 Bump to 2.9.22-RC.3
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-31 08:33:22 -07:00
Derek Collison
2b677c231a Unlock needed to be guarded
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-31 08:16:47 -07:00
Derek Collison
2e1392a234 [FIXED] potential message duplication from sources when downgrading back from 2.10 (#4454)
- [X] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [ ] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

2.10 adds a couple space separated fields to the sourcing message header
from 2 to 4 but the current 2.9 code is too strict of checking the
number of fields is exactly 2 rather than at least 2
2023-08-30 16:21:59 -07:00
Derek Collison
abae24086c Make sure order correct
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-30 16:13:56 -07:00
Derek Collison
1de649a690 Remove OCSP debug log on reload (#4453)
When reloading TLS we would always be logging the attempt to plug OCSP:

```
[42801] 2023/08/30 14:52:33.766638 [INF] Reloaded: authorization users
[42801] 2023/08/30 14:52:33.766648 [INF] Reloaded: accounts
[42801] 2023/08/30 14:52:33.766652 [INF] Reloaded: tls = enabled
[42801] 2023/08/30 14:52:33.766756 [DBG] Plugging TLS OCSP peer for [Client]
[42801] 2023/08/30 14:52:33.766763 [INF] Reloaded server configuration
```
2023-08-30 16:05:24 -07:00
Derek Collison
3be9e97760 Bump to 2.9.22-RC.2
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-30 15:32:31 -07:00
Jean-Noël Moyne
003daf3db8 Fixes possible message duplication in sourcing streams if upgrading to 2.10 and then back down to 2.9
2.10 adds a couple space separated fields to the sourcing message header from 2 to 4 but the current 2.9 code is too strict of checking the number of fields is exactly 2 rather than at least 2

Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-08-30 15:27:26 -07:00
Waldemar Quevedo
4109e420d2 Remove ocsp debug log on reload
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-30 14:54:30 -07:00
Derek Collison
8841432d03 Allow 2.10 tombstones to be skipped and allow us to recover on downgrade from 2.10 to 2.9.
Also fixed small bug that could set bad first seq.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-30 14:38:41 -07:00
Neil
bd23469ebe Add benchmark for request-reply workload over encrypted connection (#4399)
- [x] Tests added
- [x] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [x] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
- [x] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

### Changes proposed in this pull request:
- Creates new TLS certificates and private keys for testing with various
key types
    - RSA (1024, 2048, 4096)
    - ED25519
- Adds a benchmark that measures NATS Core request-reply performance
over TLS-encrypted connections
2023-08-30 10:10:25 +01:00
Derek Collison
e4a1b81d30 Fix on rebuild first when rebuild results in empty block (from dev branch)
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-29 19:50:33 -07:00
Derek Collison
8865c2a703 Fix for update to max msgs per where recalculating first was not checking for seq < mb.first.seq
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-29 16:02:52 -07:00
reubenninan
8c10cb3591 Add benchmark for req-reply with various TLS schemes 2023-08-29 17:00:28 -04:00
Derek Collison
b66a7f6e9f When expiring complete blocks make sure to update global subject index psim.
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-27 12:03:44 -07:00
Derek Collison
5b18e80d42 Added CORS support for the monitoring server (#4423)
- [x] Link to issue, e.g. `Resolves #NNN`
 - [ ] Documentation added (if applicable)
 - [x] Tests added
- [ ] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [ ] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [ ] Build is green in Travis CI
- [x] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

Resolves #4422 

### Changes proposed in this pull request:

- Added `Access-Control-Allow-Origin` header to allow CORS requests for
the monitoring server
- Added a check in the tests for the header when the `Content-Type` is
`application/json`
2023-08-25 14:49:09 -07:00
Derek Collison
0d135d4161 Bump to 2.9.22-RC.1
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-25 11:04:37 -07:00
Derek Collison
e19f883120 [FIX] PurgeEx with keep and deleted bug (#4431)
Fix for purge with keep bug with user deletes and improved search for
large number of blocks.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-25 11:03:32 -07:00
Derek Collison
e5625b9d9b If a leader is asked for an item and we have no items left, make sure to also step-down.
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-25 10:20:07 -07:00
Derek Collison
22ed97c6c9 Fix for purge with keep bug and improved search for large number of blocks.
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-25 08:59:47 -07:00
Tomasz Pietrek
6df4403913 Fix flaky TestJetStreamClusterConsumerFollowerStoreStateAckFloorBug
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-08-25 15:31:20 +02:00
Derek Collison
2669f77190 Make sure to reset election timer on catching up
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-24 19:58:08 -07:00
Derek Collison
48bf7ba151 When a consumer reached a max delivered condition, we did not properly synchronize the state such that on a restore or leader switch the ack pending could jump and be higher than max ack pending and block the consumer.
This propagates a delivered update and we updated the store state engine to do the right thing when the condition is reached.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-24 16:00:27 -07:00
reubenninan
bba949e617 Add tls benchmarking configs with certs and keys
Contains certs and keys for RSA (1024, 2048, 4096) and ED25519. As well as root certificate authority cert & key.
2023-08-23 18:11:27 -04:00
Pierre Mdawar
e5836fc98d Added CORS support for the monitoring server 2023-08-23 16:47:30 +03:00
Derek Collison
ddb7f9f9d5 Fix for a peer-remove of an R1 that would brick the stream.
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-22 17:45:19 -07:00
Waldemar Quevedo
2b2fbf7359 Bump to v2.9.22-beta.1
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-08-22 13:37:12 -07:00
Derek Collison
84536761a9 Bump to 2.9.22-beta
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-22 08:27:44 -07:00
Derek Collison
e5d208bf33 When moving streams, we could check too soon and be in a gap where the replica peer has not registered a catchup request.
This would cause us to think the replica was caughtup incorrectly and drop our leadership, which would cancel any cacthup requests.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 20:07:48 -07:00
Derek Collison
2fc3f45ea1 [FIXED] Durable pull consumers could get cleaned up incorrectly on leader change. (#4412)
Fix for a bug that would allow old leaders of pull based durables to
delete a consumer from an inactivity threshold timer inadvertently.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 15:35:44 -07:00
Derek Collison
6e3ae20650 [FIXED] Fixed deadlock when checkAndSync was being called as part of storing message (#4411)
We violated the locking pattern, so we now make sure we do this in a
separate Go routine and put checks to only run it once.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 15:28:58 -07:00
Derek Collison
0a86bf4a9a Should reset to false, not true when done
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 14:57:17 -07:00
Derek Collison
43314fd439 Fix for a bug that would allow old leaders of pull based durables to delete a consumer from an inactivity threshold.
Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 14:53:09 -07:00
Neil Twigg
d720a6931c Use own subject for LDM event
Signed-off-by: Neil Twigg <neil@nats.io>
2023-08-21 22:03:26 +01:00
Neil Twigg
7cc5838a6d Send shutdown event on LDM so that R1 assets do not get assigned to the LDM node
Signed-off-by: Neil Twigg <neil@nats.io>
2023-08-21 21:29:01 +01:00
Derek Collison
e018705a08 Fixed deadlock when checkAndSync was being called as part of storing message.
We violated the locking pattern, so we now make sure we do this in a separate Go routine and put checks to only run it once.

Signed-off-by: Derek Collison <derek@nats.io>
2023-08-21 12:12:36 -07:00