- [X] Link to issue, e.g. `Resolves #NNN`
- [X] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [ ] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
- [x] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)
Resolves#1556
### Changes proposed in this pull request:
Adds tw new $SYS server API endpoints:
- `$SYS.REQ.SERVER.%s.KICK` (where %s is the server_id) which 'kicks'
(effectiveley 'rebalance' as the client application reconnects itself
right away (potentially to another server in the cluster)). The service
takes a JSON payload containing either an "id" or a "name" field. "id"
disconnects the client connection id, "name" disconnects _all_ of the
clients connected to the server with that name.
- `$SYS.REQ.SERVER.%s.LDM` (where %s is the server_id) and takes a JSON
payload containing either an "id" or a "name" field. "id" sends an LDM
Info message to the client connection id, "name" sends an LDM Info
message to _all_ of the clients connected to the server with that name.
This features allow administrators to manually 're-balance' client
connections between the servers in the cluster (e.g. after a rolling
upgrade of the servers where one server ends up with no client
connections after the upgrade), by kicking some of the client
connections from one of the 'overloaded' (in comparison to other
servers) servers in the cluster, causing them to re-estalibsh their
connection to (hopefully) another server.
Added timeout to test to prevent running into go test timeout in case
messages did not arrive on time.
```
=== RUN TestAccountImportSubjectMapping
panic: test timed out after 30m0s
goroutine 85 [chan receive, 29 minutes]:
github.com/nats-io/nats-server/v2/test.TestAccountImportSubjectMapping(0xc000007d40)
/workspace/build/buildkite/synadia/nats-server-go-v1-21/test/accounts_cycles_test.go:466 +0x5d2
testing.tRunner(0xc000007d40, 0x11e1818)
/usr/local/go/src/testing/testing.go:1595 +0x239
created by testing.(*T).Run in goroutine 1
/usr/local/go/src/testing/testing.go:1648 +0x82b
```
This can sometimes go just above 50ms but have never seen it slower than
90ms:
```
=== RUN TestNoRaceJetStreamMemstoreWithLargeInteriorDeletes
norace_test.go:4078: Took too long to snapshot: 50.838542ms
--- FAIL: TestNoRaceJetStreamMemstoreWithLargeInteriorDeletes (7.64s)
=== RUN TestNoRaceJetStreamMemstoreWithLargeInteriorDeletes
norace_test.go:4078: Took too long to snapshot: 50.920709ms
--- FAIL: TestNoRaceJetStreamMemstoreWithLargeInteriorDeletes (7.06s)
=== RUN TestNoRaceJetStreamMemstoreWithLargeInteriorDeletes
norace_test.go:4078: Took too long to snapshot: 62.469125ms
--- FAIL: TestNoRaceJetStreamMemstoreWithLargeInteriorDeletes (6.25s)
=== RUN TestNoRaceJetStreamMemstoreWithLargeInteriorDeletes
norace_test.go:4078: Took too long to snapshot: 69.397834ms
--- FAIL: TestNoRaceJetStreamMemstoreWithLargeInteriorDeletes (6.49s)
=== FAIL: server TestNoRaceJetStreamMemstoreWithLargeInteriorDeletes (5.66s)
norace_test.go:4078: Took too long to snapshot: 81.595512ms
```
This is to try to prevent test failing due to trying to access the
tempdir while it is being torn down.
(go issue: https://github.com/golang/go/issues/43547)
```
=== RUN TestFileStoreMsgBlkFailOnKernelFaultLostDataReporting/AES-GCM-S2
testing.go:1225: TempDir RemoveAll cleanup: unlinkat ./TestFileStoreMsgBlkFailOnKernelFaultLostDataReportingAES-GCM-S23605508670/001/msgs: directory not empty
--- FAIL: TestFileStoreMsgBlkFailOnKernelFaultLostDataReporting (0.02s)
```
Also increases timeout slightly of `TestFileStoreNewWriteIndexInfo`
which runs close to 1ms deadline sometimes:
```
=== FAIL: server TestFileStoreNewWriteIndexInfo/None-None (4.85s)
| filestore_test.go:5489: Unexpected elapsed time: 1.054065ms
| --- FAIL: TestFileStoreNewWriteIndexInfo/None-None (4.85s)
```
This is to try to prevent test failing due to trying to access
the tempdir while it is being tore down.
(go issue: https://github.com/golang/go/issues/43547)
```
=== RUN TestFileStoreMsgBlkFailOnKernelFaultLostDataReporting/AES-GCM-S2
filestore_test.go:5195: ------------> 128
testing.go:1225: TempDir RemoveAll cleanup: unlinkat ./TestFileStoreMsgBlkFailOnKernelFaultLostDataReportingAES-GCM-S23605508670/001/msgs: directory not empty
--- FAIL: TestFileStoreMsgBlkFailOnKernelFaultLostDataReporting (0.02s)
```
Signed-off-by: Waldemar Quevedo <wally@nats.io>
This PR adds `account` and `stream` options to `HEALTHZ` system request
and `/healthz` monitoring endpoint.
It allows for checking health of a specific stream, without having to
rely on other streams, which (when under stress) may not have reached
consensus yet and would return an error.
Additionally, `HealthzOptions.Details` alters the response, returning an
array of errors containing each skewed stream/consumer, e.g.:
```json
{
"status": "error",
"status_code": 500,
"errors": [
{
"type": "STREAM",
"account": "js",
"stream": "test:123",
"error": "JetStream stream js \u003e test:123 is not current"
},
{
"type": "STREAM",
"account": "js",
"stream": "test:125",
"error": "JetStream stream js \u003e test:125 is not current"
},
{
"type": "STREAM",
"account": "js",
"stream": "test:42",
"error": "JetStream stream js \u003e test:42 is not current"
},
{
"type": "STREAM",
"account": "js",
"stream": "test:126",
"error": "JetStream stream js \u003e test:126 is not current"
},
{
"type": "STREAM",
"account": "js",
"stream": "test:128",
"error": "JetStream stream js \u003e test:128 is not current"
}
]
}
```
Adds a new `slow_consumer_stats` field to varz to get more details about
the types of connections that are becoming slow consumers:
```
"slow_consumer_stats": {
"clients": 0,
"routes": 0,
"gateways": 0,
"leafs": 0
}
```
This should make it possible to switch from limits-based retention to
interest-based retention on an existing stream.
Signed-off-by: Neil Twigg <neil@nats.io>
Add extra flushes to make test more precise and try to avoid timeouts
```
=== RUN TestAccountImportCycle
accounts_test.go:3447: require no error, but got: nats: timeout
--- FAIL: TestAccountImportCycle (1.01s)
```
When multiple instances are running on the machine a PID argument
suffixed with a '*' character will signal all matching PIDs.
Example: `nats-server --signal reload=*`
- [ ] Link to issue, e.g. `Resolves #NNN`
- [ ] Documentation added (if applicable)
- [X] Tests added
- [X] Branch rebased on top of current ~~main~~ dev
- [X] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
- [ ] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)
`t.Fatalf` being called while holding a lock would sometimes leave
builds hanging until test timeout.
```
=== RUN TestFileStoreNewWriteIndexInfo/AES-GCM-None
=== RUN TestFileStoreNewWriteIndexInfo/AES-GCM-S2
filestore_test.go:5483: require true, but got false
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
```
Picks up https://github.com/nats-io/nats-server/pull/4297 into main.
Includes:
- Using Go 1.20 for the nightly images and Travis tests
- Drop Go 1.18
- Updates to GitHub Actions
- Upgrade to golang-ci
Let pull consumer in test fetch messages for slightly longer instead of
at the same time as the producer, to avoid failing due to missing a few
messages:
```
=== RUN TestNoRaceJetStreamServiceImportAccountSwapIssue
norace_test.go:1194: Expected to receive 14982 msgs, only got 14981
--- FAIL: TestNoRaceJetStreamServiceImportAccountSwapIssue (3.03s)
```
This fixes the race condition in consumer create API by adding a missing
return statement, probably introduced while solving conflicts.
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
It can take slightly longer in Travis close to the deadline so bumping
it for this test:
```
=== RUN TestAccountReloadServiceImportPanic
--- PASS: TestAccountReloadServiceImportPanic (10.60s)
=== RUN TestAccountReloadServiceImportPanic
accounts_test.go:3621: Have not received all responses, want 187876 got 182649
--- FAIL: TestAccountReloadServiceImportPanic (14.09s)
```
When multiple instances are running on the machine a PID argument suffixed with
a '*' character will signal all matching PIDs.
Example: `nats-server --signal reload=*`
Signed-off-by: Jason Volk <jason@zemos.net>
Track deleted with single avl.SeqSet dmap for now vs old method for
memory store.
For fileStore, we were trying to be too smart to save space at the
expense of encoding time, so revert back to simple version that is much
100x faster.
Size of encoding may be a bit bigger then we wanted, but we want to
prefer speed over size.
Signed-off-by: Derek Collison <derek@nats.io>
Add distinction between create and update to consumer API
As in the server there is only one API for consumer management create
and update,
if clients want to provide to the users guard against overriding
existing consumer with create operation, or accidentaly creating them
with update, they need to rely on calling `Info`.
That adds latency, traffic and load on the server and is still race'y,
as state on the server can change between the info and create calls.
This PR adds `Action` to CreateConsumerRequest, which is a non-breaking
change that allows client's to present it's intent without spliting
Consumer API into create and update.
This is not a prefect solution, but such split, to not be breaking and
does not require new API version.
TODO:
- [x] Add concrete error types to errors.json and use them
- [ ] Add ADR (after LGTM)
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
