* add the ability to cancel a move in progress
Move to individual subjects for move and cancel_move
New subjects are:
$JS.API.ACCOUNT.STREAM.MOVE.*.*
$JS.API.ACCOUNT.STREAM.CANCEL_MOVE.*.*
last and second to last token are account and stream name
Signed-off-by: Matthias Hanel <mh@synadia.com>
Added http monitoring endpoint /accstatz
It responds with a list of statz for all accounts with local connections
the argument "unused=1" can be provided to get statz for all accounts
This endpoint is also exposed as nats request under:
This monitoring endpoint is exposed via the system account.
$SYS.REQ.ACCOUNT.*.STATZ
Each server will respond with connection statistics for the requested
account. The format of the data section is a list (size 1) identical to the event
$SYS.ACCOUNT.%s.SERVER.CONNS which is sent periodically as well as on
connect/disconnect. Unless requested by options, server without the account,
or server where the account has no local connections, will not respond.
A PING endpoint exists as well. The response format is identical to
$SYS.REQ.ACCOUNT.*.STATZ
(however the data section will contain more than one account, if they exist)
In addition to general filter options the request takes a list of accounts and
an argument to include accounts without local connections (disabled by default)
$SYS.REQ.ACCOUNT.PING.STATZ
Each account has a new system account import where the local subject
$SYS.REQ.ACCOUNT.PING.STATZ essentially responds as if
the importing account name was used for $SYS.REQ.ACCOUNT.*.STATZ
The only difference between requesting ACCOUNT.PING.STATZ from within
the system account and an account is that the later can only retrieve
statz for the account the client requests from.
Also exposed the monitoring /healthz via the system account under
$SYS.REQ.SERVER.*.HEALTHZ
$SYS.REQ.SERVER.PING.HEALTHZ
No dedicated options are available for these.
HEALTHZ also accept general filter options.
Signed-off-by: Matthias Hanel <mh@synadia.com>
In some situations, a server may report that a remote server is
detected as orphaned (and the node is marked as offline). This is
because the orphaned detection relies on conns update to be received,
however, servers would suppress the update if an account does not
have any connections attached.
This PR ensures that the update is sent regardless if the account
is JS configured (not necessarily enabled at the moment).
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
The test was lowering the eventsHBInterval to low value but not
restoring its original value. This would cause some servers to
be considered "orphan" and the shutdown event sent causing their
raft node to be considered "offline", which then would result
in some tests failing with "insufficient resources".
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
A customer experienced and endless failure to have a stream cacthup. The current leader was being asked for a message from a snapshot that was larger then what we had, resulting in EOF which silently failed.
We now detect this and signal end of catchup and redo the bad snapshot if possible.
Signed-off-by: Derek Collison <derek@nats.io>
When increasing the replica count unique tags for already existing peers
where ignored, which could lead to bad placement
Signed-off-by: Matthias Hanel <mh@synadia.com>
We use to ignore if the seq was 0, but now would treat it as
a requirement that the stream be empty if header is present but
set to 0.
This relates to client PR: https://github.com/nats-io/nats.go/pull/958
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Make sure when processing a peer removal that the stream assignment agrees.
When a new leader takes over it can resend a peer removal, and if the stream/consumer really was rescheduled we could remove by accident.
Also need to make sure that when we remove a stream we remove the node as part of the stream assignment.
If we didn't, if the same asset returned to this server we would not start up the monitoring loop.
Simplify migration logic in monitorStream, to be driven by leader only
Improved unit tests
Added failure when server not in peer list
Move command does not require server anymore
Signed-off-by: Matthias Hanel <mh@synadia.com>
This allows a solciting leafnode config to ask that any JetStream cluster assets that are a current leader have the leader stepdown.
Signed-off-by: Derek Collison <derek@nats.io>
This effectively means that requests with batch > 1 will process a message and go to the end of the line.
Signed-off-by: Derek Collison <derek@nats.io>
* 1: Allows spaces to be used inside {{}} subject mapping functions:
2: Rework and improve mapping destinations validation and error handling with much more helpful error messages, e.g.:
* Error adding mapping for "foo.*.*" to "bar.{{wildcard(1)}}" : invalid mapping destination: not using all of the token wildcard(s) in bar.{{wildcard(1)}}
* Error adding mapping for "myservice.request.*" to "myservice.request.{{ partition(10) }}.{{wildcard(1)}}" : invalid mapping destination: not enough arguments passed to the function in {{ partition(10) }}
* Error adding mapping for "myservice.request.*" to "myservice.request.{{ partition(10,2) }}.{{wildcard(1)}}" : invalid mapping destination: wildcard index out of range in {{ partition(10,2) }}: [2]
* Error adding mapping for "myservice.request.*" to "myservice.request.{{ partition(10,1) }}.{{wildcard()}}" : invalid mapping destination: not enough arguments passed to the function in {{wildcard()}}
* Error adding mapping for "myservice.request.*" to "myservice.request.{{ xxxpartition(10,1) }}.{{wildcard(1)}}" : invalid mapping destination: unknown function in {{ xxxpartition(10,1) }}
* Error adding mapping for "myservice.request.*" to "myservice. request.{{ xxxpartition(10,1) }}.{{wildcard(1)}}" : invalid mapping destination: invalid subject
* implement PR comments
When creating a pull consumer with InactiveThreshold set, if the
application is doing pull requests with "no_wait" at regular interval
(lower than InactiveThreshold), the JS consumer should be considered
active and not deleted.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
The "InProcess" change make readyForConnections() possibly return
much faster than it used to, which could cause tests to fail.
Restore the original behavior, but in case of DontListen option
wait on the startupComplete gate.
Also fixed some missing checks for leafnode connections.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
