Commit Graph

7276 Commits

Author SHA1 Message Date
Derek Collison
855e1bb14e Allow more tolerance for travis
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-23 14:24:00 -07:00
Derek Collison
2b2e22ed52 When creating a consumer on a stream with a very large number of msg blks, calculating numPending could be slow.
This aims to optimize a bit, more work to be done on streams with a very large (> 200k) number of msg blks.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-23 14:11:56 -07:00
Derek Collison
3183f6ed95 Cert Store (aka wincert) (#4268)
For NATS Server on Windows, provide option for TLS certificate and
handshake signature to be provided by the Windows Certificate Store
instead of PEM files.
2023-06-22 17:20:35 -07:00
Todd Beets
f854e95c11 Cert Store (aka wincert) 2023-06-22 12:25:54 -07:00
Derek Collison
031535b89d Add object store benchmark (#4220)
- [ ] Link to issue, e.g. `Resolves #NNN`
 - [ ] Documentation added (if applicable)
 - [x] Tests added
- [x] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [x] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [x] Build is green in Travis CI
- [x] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

### Changes proposed in this pull request:
 - New benchmark for NATS JetStream Object Store
2023-06-21 14:36:35 -07:00
reubenninan
13e09f2db4 Add benchmarks for object store
Signed-off-by: reubenninan <reuben@nats.io>
2023-06-21 16:05:30 -04:00
Derek Collison
a8fc352a20 Additional check for TLS required in WebSocket INFO (#4264)
This should stop us reporting `tls_required` when we aren't reporting
`tls_available`.

Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-21 12:11:02 -07:00
Neil Twigg
cf715845d1 Additional check for TLS required in WebSocket INFO
Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-21 19:06:49 +01:00
Derek Collison
f4f3cce6cc Release v2.9.19 (#4257) 2023-06-20 08:39:14 -07:00
Byron Ruth
9592b760b9 Release v2.9.19
Signed-off-by: Byron Ruth <byron@nats.io>
2023-06-20 10:55:31 -04:00
Derek Collison
04a79b9b1e WebSocket-specific INFO (#4255)
This fixes #4252 by ensuring that `tls_available`, `tls_required`,
`host` and `port` are populated based on the WebSocket listener rather
than standard listeners.

Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-20 06:33:00 -07:00
Neil Twigg
afdc2478ca WebSocket-specific INFO
This fixes #4252 by ensuring that `tls_available`, `tls_required`, `host` and `port`
are populated based on the WebSocket listener rather than standard listeners.

Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-20 11:47:09 +01:00
Derek Collison
f7896b4969 Test MaxMsgs and MaxMsgsPer in combination (#4253)
This unit test is modelled around issue #4247 and proves that the
`MaxMsgs` and `MaxMsgsPer` limits are correctly enforced together with
`DiscardNew` and `DiscardNewPer`.

Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-19 08:58:42 -07:00
Neil Twigg
e879a9fa0c Test MaxMsgs and MaxMsgsPer in combination
Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-19 09:45:17 +01:00
Derek Collison
f5b06100d3 Added in another DQ test across leafnodes. (#4250)
This test has multiple leafnode connections to different accounts and to
a shared account to make sure behavior is correct.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-16 13:55:45 -07:00
Derek Collison
e8094c9f33 Make utility funcs helpers
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-16 12:07:21 -07:00
Derek Collison
b3f913237c Added in another DQ test across leafnodes.
This test has multiple leafnode connections to different accounts and to a shared account to make sure behavior is correct.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-16 11:54:09 -07:00
Derek Collison
4a1b281412 [IMPROVED] High CPU and Memory usage on replicated mirrors with very high starting sequence. (#4249)
When creating replicated mirrors where the source stream had a very
large starting sequence number, the server would use excessive CPU and
Memory.

This is due to the mirroring functionality trying to skip messages when
it detects a gap. In a replicated stream this puts excessive stress on
the raft system.

This step is not needed at all if the mirror stream has no messages, we
can simply jump ahead.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-15 17:48:32 -07:00
Derek Collison
087a28a13e When creating replicated mirrors where the source stream had a very large starting sequence number, the server would use excessive CPU and Memory.
This is due to the mirroring functionality trying to skip messages when it detects a gap. In a replicated stream this puts excessive stress on the raft system.
This step is not needed at all if the mirror stream has no messages, we can simply jump ahead.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-15 17:20:15 -07:00
Derek Collison
367d857612 Bump v2.9.19-beta.1 (#4245) 2023-06-13 13:50:13 -07:00
Byron Ruth
bbf24a6d98 Bump v2.9.19-beta.1
Signed-off-by: Byron Ruth <byron@nats.io>
2023-06-13 15:48:05 -04:00
Derek Collison
166eeb243c Release v2.9.18 (#4242) 2023-06-13 12:41:10 -07:00
Byron Ruth
af805b57a4 Release v2.9.18
Signed-off-by: Byron Ruth <byron@nats.io>
2023-06-13 15:19:20 -04:00
Derek Collison
876cb6d837 [UPDATED] go mod tidy to update go.sum (#4240)
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-12 17:31:16 -07:00
Derek Collison
232294c3af go mod tidy to update go.sum
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-12 17:16:35 -07:00
Derek Collison
378d5a944e Bump client version to v1.27.0 (#4239)
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-06-12 14:11:24 -07:00
Tomasz Pietrek
13bf12ce64 Bump client version to v1.27.0
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-06-12 22:06:08 +02:00
Derek Collison
1d00ea4fa0 Bump to 2.9.18-beta.3
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-11 13:11:32 -07:00
Derek Collison
ab2ac20fe1 [UPDATED] Dependencies (#4236)
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-11 13:09:03 -07:00
Derek Collison
0980384c97 Update dependencies
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-11 12:25:44 -07:00
Derek Collison
860c481f0f [IMPROVED] Optimize statsz locking and sending in standalone mode. (#4235)
If we know we are in stand alone mode only send out statsz updates if we
know we have external interest.

Signed-off-by: Derek Collison <derek@nats.io>

Resolves: #4234
2023-06-10 20:55:43 -07:00
Derek Collison
11963e51fe Optimize statsz locking and only send if we know we have external interest.
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-10 20:25:05 -07:00
Derek Collison
aae218fe77 [IMPROVED] Only enable JetStream account updates in clustered mode. (#4233)
If we know we are in stand alone mode we do not need to run the updates
for JetStream account resources updates.

Signed-off-by: Derek Collison <derek@nats.io>

Resolves #4227 (Partial)
2023-06-10 17:03:11 -07:00
Derek Collison
783e9491b1 [IMPROVED] Last msg lookup (KV Get) when subject is a literal subject (#4232)
When messages were very small and the key space was very large the
performance of last message gets in the store layer (both file and
memory) would degrade.

If the subject is literal we can optimize and avoid sequence scans that
are needed when multiple subject states need to be considered.

Signed-off-by: Derek Collison <derek@nats.io>

Resolves #4221
2023-06-10 15:23:11 -07:00
Derek Collison
a5de25f213 Only enable JetStream account updates in clustered mode.
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-10 15:21:55 -07:00
Derek Collison
f81bc80676 Improve last msg lookup (KV Get) when subject is a literal subject.
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-10 13:16:13 -07:00
Derek Collison
8c513ad2c2 [FIXED] DQ weighted subscribers across separate leafnode connections. (#4231)
Fix for properly distributed queue requests over multiple leafnode
connections.
When a leafnode server joins two accounts in a supercluster, we want to
make sure that each connection properly takes into account the weighted
number of subscribers in each account.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-09 17:50:17 -07:00
Derek Collison
2765e534eb Fix test and update copyright
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-09 15:09:15 -07:00
Derek Collison
ce2dcd3394 Fix for properly distributed queue requests over multiple leafnode connections.
When a leafnode server joins two accounts in a supercluster, we want to make sure that each connection properly takes into account the weighted number of subscribers in each account.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-09 14:43:59 -07:00
Derek Collison
13b6e2613c Reduce messages in chaos tests (#4229)
It doesn't really appear as though, for what these tests are trying to
prove, that an excessively large number of messages is required. Instead
let's drop the count a little in the hope that they run a bit faster.

Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-09 13:59:35 -07:00
Derek Collison
81154c40f5 Bump to 2.9.18-beta.2
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-09 09:29:48 -07:00
Neil Twigg
7de3568f39 Reduce messages in chaos tests
It doesn't really appear as though, for what these tests are trying to
prove, that an excessively large number of messages is required. Instead
let's drop the count a little in the hope that they run a bit faster.

Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-09 17:07:53 +01:00
Neil
9d2ae42956 [FIXED] Performance issues with checkAckFloor with large first stream sequence. (#4226)
Bail early if new consumer, meaning stream sequence floor is 0.
Decide which linear space to scan.
Do no work if no pending and we just need to adjust which we do at the
end.

Also realized some tests were named wrong and were not being run, or
were in wrong file.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-09 09:27:52 +01:00
Derek Collison
9eeffbcf56 Fix performance issues with checkAckFloor.
Bail early if new consumer, meaning stream sequence floor is 0.
Decide which linear space to scan.
Do no work if no pending and we just need to adjust which we do at the end.

Also realized some tests were named wrong and were not being run, or were in wrong file.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-08 18:45:03 -07:00
Derek Collison
d3bde2c6e1 Send peer state when adding peers (#4224)
Currently `UpdateKnownPeers` doesn't send a peer state when a single
peer add operation is taking place, but it seems like this can
potentially race when there are lots of changes to the replica count
happening in rapid succession. Sending the peer state in all cases seems
to fix this issue and, so far in my testing, fixes the failground stream
update replicas test.

Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-08 09:31:49 -07:00
Neil Twigg
6d9955d212 Send peer state when adding peers
Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-08 15:25:18 +01:00
Derek Collison
c85db15a2c Update Go to 1.19.10 (#4223) 2023-06-08 04:39:20 -07:00
Byron Ruth
3ca99adcca Update Go to 1.19.10
Signed-off-by: Byron Ruth <byron@nats.io>
2023-06-08 07:10:27 -04:00
Derek Collison
779978d817 Extended replay leafnode test to confirm mirror functionality
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-07 14:01:43 -07:00
Derek Collison
822ad00d50 Bump to 2.9.18-beta.1
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-05 14:14:35 -07:00