Commit Graph

7366 Commits

Author SHA1 Message Date
Neil Twigg
57d888eec4 Use AVL tree for consumer redeliver map
Signed-off-by: Neil Twigg <neil@nats.io>
2023-04-18 15:53:44 +01:00
Derek Collison
f3e0d6b070 AVL trees and sequence sets for interior delete optimizations. (#4070)
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-18 07:48:13 -07:00
Derek Collison
333d684c86 Use encoding of avl seqset for writeIndexInfo's delete map.
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-17 20:17:15 -07:00
Neil Twigg
1a24e955d0 Add size to preamble, check capacity instead of length when encoding
* If we don't encode the `size`, it is lost during an encoding-decoding round-trip
* If we don't check capacity, we might reallocate needlessly instead of just growing the slice

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-17 20:17:12 -07:00
Derek Collison
9b9159ab70 Basic swap out of the old dmap (map[uint64]struct{}) for new avl.SequenceSet.
No other optimizations yet.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-17 20:17:10 -07:00
Derek Collison
1f6aa94405 SequenceSet is an AVL tree with variable bitmask nodes to contain large delete maps for streams.
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-17 20:17:03 -07:00
Derek Collison
09afcee9d9 Merge branch 'main' into dev 2023-04-17 08:43:08 -07:00
Waldemar Quevedo
f84ca24fcc 2.9.16 release (#4065)
Don't mind the "bullet" character changes.. that was my Prettier
auto-formatting.
2023-04-17 07:50:08 -07:00
Byron Ruth
202d49d069 2.9.16 release
Signed-off-by: Byron Ruth <byron@nats.io>
2023-04-17 10:05:18 -04:00
Derek Collison
d8389ec4d3 [FIX] Data race when setting up service import subscriptions. (#4068)
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-17 06:57:27 -07:00
Derek Collison
9a3e0b783c Fix for a data race when setting up service import subscriptions.
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-17 06:40:09 -07:00
Neil
7709780987 Bump version to 2.9.16-RC.9 (#4067)
Signed-off-by: Neil Twigg <neil@nats.io>
2023-04-17 13:54:06 +01:00
Neil Twigg
a9aa280d06 Bump version to 2.9.16-RC.9
Signed-off-by: Neil Twigg <neil@nats.io>
2023-04-17 13:38:25 +01:00
Waldemar Quevedo
d62ccc7737 Fix nsc in nightly (#4063)
Pin to the v2 version from nsc instead.
2023-04-17 00:59:56 -07:00
Waldemar Quevedo
dea7fc04f3 Fix nsc in nightly 2023-04-17 00:07:01 -07:00
Derek Collison
33f23be103 Update package dependencies (#4060)
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-16 16:54:11 -07:00
Derek Collison
a6029a090c Update pkg dependencies
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-16 14:25:33 -07:00
Derek Collison
6215ed9bc3 Update to Go 1.19.8 (#4059) 2023-04-16 13:08:44 -07:00
Byron Ruth
dea68595fd Update to Go 1.19.8
Signed-off-by: Byron Ruth <byron@nats.io>
2023-04-16 15:33:13 -04:00
Derek Collison
3b3fac297a Merge branch 'main' into dev 2023-04-15 14:21:39 -07:00
Derek Collison
670da9f7a3 [IMPROVED] Recover on boundary conditions that could cause RAFT layer to spin. (#4058)
Reset our WAL on edge conditions instead of trying to recover.
Also if we are timing out and trying to become a candidate but are doing
a catchup check if we are stalled.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-15 14:19:59 -07:00
Derek Collison
a5f5603645 Reset our WAL on edge conditions instead of trying to recover.
Also if we are timing out and trying to become a candidate but are doing a catchup check if we are stalled.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-15 12:23:44 -07:00
Derek Collison
3f27b67791 Merge branch 'main' into dev 2023-04-15 10:47:29 -07:00
Derek Collison
2ee73f8a38 [FIXED] Fix for a regression in behavior. (#4057)
When tracking information per subject went from >1 back to only 1 we
needed to make sure we cleared firstNeedsUpdate.

Thanks to @scottf for finding it.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-15 10:47:13 -07:00
Derek Collison
034975e767 Fix for a regression in behavior, needed to make sure when we went back to 1 entry for a subject we cleared firstNeedsUpdate.
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-15 10:00:44 -07:00
Derek Collison
8375ab5cde Merge branch 'main' into dev 2023-04-14 16:44:25 -07:00
Derek Collison
816cd82fc3 [FIXED] Do not spin on multi-leader with same term. (#4056)
If we see another leader with same term we should step down, otherwise
we could spin with catchups and WAL truncates.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-14 16:43:50 -07:00
Derek Collison
66ca46e145 If we see another leader with same term we should step down
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-14 16:21:40 -07:00
Derek Collison
b3e1961821 Merge branch 'main' into dev (#4055)
cc @jnmoyne 

 Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-04-14 16:02:27 -07:00
Tomasz Pietrek
82220d9103 Merge branch 'main' into dev
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-04-14 22:38:45 +02:00
Derek Collison
94e62cc30b Add server name / remote server name to routez (#4054)
Add server name to routez info:

```js
{
  "server_id": "NAXUYR7XZFPBQPJGV5HNWIXLOCLBSA2GOQEPGP47AFY62XHWPREKCXWL",
  "server_name": "nats-burn-0",
  "now": "2023-04-14T19:47:52.057269Z",
  "num_routes": 2,
  "routes": [ 
    {
      "rid": 51,
      "remote_id": "NACPYGKLPWMYM7YZA3NCYQ5HL2EKGVKFKNGGE6ECD5LMO26JC3T3TARR",
      "remote_name": "nats-burn-1",
      "did_solicit": true,
      "is_configured": true,
      "ip": "127.0.0.1",
      "port": 57889,
      "start": "2023-04-14T12:47:43.475646-07:00",
      "last_activity": "2023-04-14T19:47:51.983756Z",
      "rtt": "178µs",
      "uptime": "8s",
      "idle": "0s",
      "pending_size": 0,
      "in_msgs": 210,
      "out_msgs": 154,
      "in_bytes": 31854,
      "out_bytes": 15150,
      "subscriptions": 134
    },
    {
      "rid": 53,
      "remote_id": "NCTOFS2M5IVGKYRGYOWP3Q5SQNPGCPMCIIJWOEH5SOIH3XQKRBKP7ITJ",
      "remote_name": "nats-burn-2",
      "did_solicit": true,
      "is_configured": true,
      "ip": "127.0.0.1",
      "port": 57905,
      "start": "2023-04-14T12:47:44.275914-07:00",
      "last_activity": "2023-04-14T19:47:52.022301Z",
      "rtt": "152µs",
      "uptime": "7s",
      "idle": "0s",
      "pending_size": 0,
      "in_msgs": 179,
      "out_msgs": 48,
      "in_bytes": 25629,
      "out_bytes": 7573,
      "subscriptions": 100
    }
  ]
}
```
2023-04-14 13:09:43 -07:00
Waldemar Quevedo
d12152c48f Add server name / remote server name to routez
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-04-14 12:47:00 -07:00
Derek Collison
2699465596 Fix stream sourcing & mirroring overlap errors (#4052)
When adding or updating sources/mirrors, server was checking if the
stream with a given name exists to check for subject overlaps, among
other things.
However, if sourced/mirrored stream was `External`, checks should not be
executed, as not only stream would never be found, but also, if
`External` stream had the same name as the sourcing stream, the check
would be wrongly performed against itself.

cc @jnmoyne 

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-04-14 12:18:04 -07:00
Tomasz Pietrek
a66c67baa5 Fix stream sourcing & mirroring overlap errors
When adding or updating sources/mirrors, server was checking if the stream with
a given name exists to check for subject overlaps, among other things.
However, if sourced/mirrored stream was `External`, checks should
not be executed, as not only stream would never be found,
but also, if `External` stream had the same name as the sourcing stream,
the check would be wrongly performed against itself.

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-04-14 21:00:11 +02:00
Derek Collison
c3c4c8f1f5 Merge branch 'main' into dev 2023-04-14 11:47:15 -07:00
Derek Collison
e881656683 [IMPROVED] Use new server read locks. (#4053)
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-14 10:37:14 -07:00
Derek Collison
0fe48fe91e Use new server read locks now that we have them
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-14 10:11:40 -07:00
Derek Collison
89fc7e3203 Bump to 2.9.16-RC.8
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-13 21:04:33 -07:00
Derek Collison
a319d24345 Merge branch 'main' into dev 2023-04-13 21:03:05 -07:00
Derek Collison
89f6a5c547 [FIXED] The meta layer should snapshot if any outstanding entries are present. (#4050)
Fixes this test [TestJetStreamClusterDeleteAndRestoreAndRestart] which
would flap since it would not snapshot since hash was same but had
entries that would erase stream data.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-13 20:59:00 -07:00
Derek Collison
093564f7e0 The meta layer should snapshot if any oustanding entries are present, regardless of hash.
Fixes this test [TestJetStreamClusterDeleteAndRestoreAndRestart] which would flap since it would not snapshot since hash was same but had entries that would erase stream data.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-13 20:37:00 -07:00
Derek Collison
be4999a281 [IMPROVED] Consumer updates (#4049)
Make sure to process consumer entries on recovery in case state was not
committed.
And sync other consumers when taking over as leader but no need to
process snapshots when we are in fact the leader.
Do not let the consumer redelivered count go down on re-applying state
on restarts.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-13 19:18:11 -07:00
Derek Collison
cc77d662bb Make sure to process consumer entries on recovery in case state was not committed.
Also sync other consumers when taking over as leader but no need to process snapshots when we are in fact the leader.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-13 18:40:17 -07:00
Derek Collison
a77884e4ec Do not let the redelivered count go down on re-applying on restarts
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-13 18:40:15 -07:00
Derek Collison
bafd585ce4 Fix data race
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-13 18:40:06 -07:00
Derek Collison
3f8fc5c5b1 [FIXED] #4043 (#4048)
Improve performance on storing msgs when multiple subjects exists with
multiple messages and we have store limits that are being hit.

This will not fix any performance degradation per se when per subject
limits are being hit constantly across a vast array of keys. That will
be addressed more so in 2.10.

Resolves #4043 

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-13 17:31:49 -07:00
Waldemar Quevedo
3602b3882a Fix raft log debug reloading (#4047)
Fixes being able to reload debug logging for running raft groups.
2023-04-13 16:21:53 -07:00
Derek Collison
b597485102 Improve performance on storing msgs when multiple subjects exists with multiple messages and we have store limits that are being hit.
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-13 15:27:34 -07:00
Waldemar Quevedo
a4833d0889 Fix raft log debug reloading
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-04-13 14:57:04 -07:00
Neil
96fe933846 Limit filestore block size to 2MB when encryption is enabled (#4046)
Some write operations in the filestore require re-encrypting the entire
block, so having large blocks can make these operations slow and hurt
performance.

Signed-off-by: Neil Twigg <neil@nats.io>
2023-04-13 17:24:30 +01:00