If a configuration reload is issued as the server is being shutdown,
we could get 2 different panics. One due to JetStream if an account
is JetStream enabled, and one due to the send to a go channel that
has been closed.
```
panic: send on closed channel [recovered]
panic: send on closed channel
goroutine 440 [running]:
testing.tRunner.func1.2({0x1038d58e0, 0x1039e1270})
/usr/local/go/src/testing/testing.go:1545 +0x274
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:1548 +0x448
panic({0x1038d58e0?, 0x1039e1270?})
/usr/local/go/src/runtime/panic.go:920 +0x26c
github.com/nats-io/nats-server/v2/server.(*Server).reloadAuthorization(0xc00024fb00)
/Users/ik/dev/go/src/github.com/nats-io/nats-server/server/reload.go:1998 +0x788
github.com/nats-io/nats-server/v2/server.(*Server).applyOptions(0xc00024fb00, 0xc00021dc00, {0xc00038e4e0, 0x2, 0xc00021dc28?})
/Users/ik/dev/go/src/github.com/nats-io/nats-server/server/reload.go:1746 +0x2b8
github.com/nats-io/nats-server/v2/server.(*Server).reloadOptions(0xc000293500?, 0xc000118a80, 0xc000293500)
/Users/ik/dev/go/src/github.com/nats-io/nats-server/server/reload.go:1121 +0x178
github.com/nats-io/nats-server/v2/server.(*Server).ReloadOptions(0xc00024fb00, 0xc000293500)
/Users/ik/dev/go/src/github.com/nats-io/nats-server/server/reload.go:1060 +0x368
github.com/nats-io/nats-server/v2/server.(*Server).Reload(0xc00024fb00)
/Users/ik/dev/go/src/github.com/nats-io/nats-server/server/reload.go:995 +0x104
```
and
```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x10077b224]
goroutine 8 [running]:
testing.tRunner.func1.2({0x101351640, 0x101b7d2a0})
/usr/local/go/src/testing/testing.go:1545 +0x274
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:1548 +0x448
panic({0x101351640?, 0x101b7d2a0?})
/usr/local/go/src/runtime/panic.go:920 +0x26c
github.com/nats-io/nats-server/v2/server.(*Account).EnableJetStream(0xc00020fb80, 0xc000220240)
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/jetstream.go:1045 +0xa4
github.com/nats-io/nats-server/v2/server.(*Server).configJetStream(0xc000226d80, 0xc00020fb80)
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/jetstream.go:707 +0xdc
github.com/nats-io/nats-server/v2/server.(*Server).configAllJetStreamAccounts(0xc000226d80)
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/jetstream.go:768 +0x2b0
github.com/nats-io/nats-server/v2/server.(*Server).enableJetStreamAccounts(0xc000226d80?)
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/jetstream.go:637 +0x128
github.com/nats-io/nats-server/v2/server.(*Server).reloadAuthorization(0xc000226d80)
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/reload.go:2039 +0x93c
github.com/nats-io/nats-server/v2/server.(*Server).applyOptions(0xc000226d80, 0xc000171c50, {0xc000074600, 0x2, 0xc000171c78?})
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/reload.go:1746 +0x2b8
github.com/nats-io/nats-server/v2/server.(*Server).reloadOptions(0xc000276000?, 0xc0000a6000, 0xc000276000)
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/reload.go:1121 +0x178
github.com/nats-io/nats-server/v2/server.(*Server).ReloadOptions(0xc000226d80, 0xc000276000)
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/reload.go:1060 +0x368
github.com/nats-io/nats-server/v2/server.(*Server).Reload(0xc000226d80)
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/reload.go:995 +0x104
```
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Several strategies which are listed below.
1. Checking a RaftNode to see if it is the leader now uses atomics.
2. Checking if we are the JetStream meta leader from the server now uses an atomic.
3. Accessing the JetStream context no longer requires a server lock, uses atomic.Pointer.
4. Filestore syncBlocks would hold msgBlock locks during sync, now does not.
Signed-off-by: Derek Collison <derek@nats.io>
In the benchmark on my machine, this added ~300ns per call, but I think that is ok for now vs the memory usage.
Signed-off-by: Derek Collison <derek@nats.io>
Streams with many interior deletes was causing issues due to the fact that the interior deletes were represented as a sorted []uint64.
This approach introduces 3 sub types of delete blocks, avl bitmask tree, a run length encoding, and the legacy format above.
We also take into account large interior deletes such that on receiving a snapshot we can skip things we already know about.
Signed-off-by: Derek Collison <derek@nats.io>
The new field `compression` in the `cluster{}` block allows to
specify which compression mode to use between servers.
It can be simply specified as a boolean or a string for the
simple modes, or as an object for the "s2_auto" mode where
a list of RTT thresholds can be specified.
By default, if no compression field is specified, the server
will use the s2_auto mode with default RTT thresholds of
10ms, 50ms and 100ms for the "uncompressed", "fast", "better"
and "best" modes.
```
cluster {
..
# Possible values are "disabled", "off", "enabled", "on",
# "accept", "s2_fast", "s2_better", "s2_best" or "s2_auto"
compression: s2_fast
}
```
To specify a different list of thresholds for the s2_auto,
here is how it would look like:
```
cluster {
..
compression: {
mode: s2_auto
# This means that for RTT up to 5ms (included), then
# the compression level will be "uncompressed", then
# from 5ms+ to 15ms, the mode will switch to "s2_fast",
# then from 15ms+ to 50ms, the level will switch to
# "s2_better", and anything above 50ms will result
# in the "s2_best" compression mode.
rtt_thresholds: [5ms, 15ms, 50ms]
}
}
```
Note that the "accept" mode means that a server will accept
compression from a remote and switch to that same compression
mode, but will otherwise not initiate compression. That is,
if 2 servers are configured with "accept", then compression
will actually be "off". If one of the server had say s2_fast
then they would both use this mode.
If a server has compression mode set (other than "off") but
connects to an older server, there will be no compression between
those 2 routes.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
New configuration fields:
```
cluster {
...
pool_size: 5
accounts: ["A", "B"]
}
```
The configuration `pool_size` in the example above means that this
server will create 5 routes to a remote server, assuming that that
server has the same `pool_size` setting.
Accounts (which are not part of the `accounts[]` configuration)
are assigned a specific route in this pool, and this will be the
same route on all servers in the cluster.
Accounts that are defined in the `accounts` field will each have
a dedicated route connection. This will allow suppression of the
account name in some of the route protocols, reducing bytes transmitted
which may increase performance.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Fixed one extraneous account update for $G. We sent for the addition before switching but suppressed the change back to 0.
We now suppress all for $G as was designed.
Signed-off-by: Derek Collison <derek@nats.io>
This adds the ability to augment or override the NATS auth system.
A server will send a signed request to $SYS.REQ.USER.AUTH on the specified account. The request will contain client information, all client options sent to the server, and optionally TLS information and client certificates.
The external auth service will respond with an empty message if not authorized, or a signed User JWT that the user will bind to.
The response can change the account the client will be bound to.
Signed-off-by: Derek Collison <derek@nats.io>
This is only added if set by a user or account expiration claim.
It is represented as a duration til expiration vs absolute time which would involve time zone and clock sync issues.
Signed-off-by: Derek Collison <derek@nats.io>
