The monitoring http server is started early and the gateway setup
(when configured) may not be fully ready when the `/gatewayz`
endpoint is inspected and could cause a panic.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
ordering of templates got messed up by a map (now removed)
Also improved error message when template generation fails
Signed-off-by: Matthias Hanel <mh@synadia.com>
Test TestNoRaceJetStreamClusterCorruptWAL() would start to flap
because of the snapshot on cluster shutdown. Disable the snapshot
on exit for this test by stopping the raft node before shutdown.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
- didRemove in applyMetaEntries() could be reset when processing
multiple entries
- change "no race" test names to include JetStream
- separate raft nodes leader stepdown and stop in server
shutdown process
- in InstallSnapshot, call wal.Compact() with lastIndex+1
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This is a regression introduced in v2.8.3. If a message reaches
the max redeliver count, it stops being delivered to the consumer.
However, after a server or cluster restart, those messages would
be redelivered again.
Resolves#3361
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
For security reasons we have introduced scoped signing keys to jwt.
They carry user permissions.
Wich is why jwt issued by those keys are not allowed to carry their own permission.
Instead they are copied from the signing key.
If the scoped signing key gets compromised, an attacker can only issue jwt with the permissions of the key.
With a plain signing key, an attacker can create arbitrary user with permissions.
Because user jwt creation is greatly simplified we added a single utility function to go/java/.net which issues user for such keys.
This is function is documented in ADR-14:
```
/**
* signingKey, is a mandatory account nkey pair to sign the generated jwt.
* accountId, is a mandatory public account nkey. Will return error when not set or not account nkey.
* publicUserKey, is a mandatory public user nkey. Will return error when not set or not user nkey.
* name, optional human readable name. When absent, default to publicUserKey.
* expiration, optional but recommended duration, when the generated jwt needs to expire. If not set, JWT will not expire.
* tags, optional list of tags to be included in the JWT.
*
* Returns:
* error, when issues arose.
* string, resulting jwt.
**/
IssueUserJWT(signingKey nkey, accountId string, publicUserKey string, name string, expiration time.Duration, tags []string) (error, string)
```
Currently the only downside of this is that the permissions are static and can't be tailored to the user.
This PR changes that by allowing the user pub/sub permissions to be parameterized with templates.
templates are for entire tokens only and include:
{{name()}} -> username
{{subject()}} -> user subject (nkey)
{{account-name()}} -> users account name
{{account-subject()}} -> user accoutn subject (nkey)
{{tag(arbitrary-prefix)}}
provided the tag "arbitrary-prefix:value" will result in "value"
provided the tags ["arbitrary-prefix:1", "arbitrary-prefix:2"] will result in two subjects "1" & "2"
If the resulting subject is not valid.
Say a tag is not present or name is not set.
This will result in an error for deny subjects
and result in no subject for allow subject.
Signed-off-by: Matthias Hanel <mh@synadia.com>
Signed-off-by: Matthias Hanel <mh@synadia.com>
In standalone mode, when attempting to create a stream which has
subjects that overlap with an existing stream, the generic
stream create error "10049" was returned instead of the more
accurate "10065" error code corresponding to subject overlap,
as it was the case in clustered mode.
Resolves#3362
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
In normal message get, the returned format is RFC3339Nano, which
is what is being used when using JSON marshaling. However, for
the direct get we had to pass a string to construct the header
and we were using time.Time.String() which was using a different
layout. So use time.Time.MarshalJSON() to be consistent with
the non-direct get message.
Libraries that already parsed the non RFC3339Nano time format
can be updated since none should have been released yet (since
the feature in the server is not released yet)
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
* Fix race between stream stop and monitorStream
monitorCluster stops the stream, when doing so, monitorStream
needs to be stopped to avoid miscounting of store size.
In a test stop and reset of store size happened first and then
was followed by storing more messages via monitorStream
Signed-off-by: Matthias Hanel <mh@synadia.com>
This can happen if the move was initiated by the user.
A subsequent cancel resets the initial peer list.
The original peer list was picked on the old set of tags.
A cancel would then keep the new list of tags but reset
to the old peers. Thus tags and peers diverge.
The problem is that at the time of cancel, the old
placement tags can't be found anymore.
This fix causes cancel to remove the placement tags, if
the old peers do not satisfy the new placement tags.
Signed-off-by: Matthias Hanel <mh@synadia.com>
* fixed consumer restart on source filter update
When a stream source filter subject was updated, the internal consumer
was not re created
If the upstream stream contains a tail of previously filtered messages,
these will now be delivered
Signed-off-by: Matthias Hanel <mh@synadia.com>
* Added check for source/mirror filter subjects
When the origin stream exists, the sourec/mirror filter subject
will be checked against the stream subjects.
If there is no overlap, an error will be returned
Signed-off-by: Matthias Hanel <mh@synadia.com>
Use better indexing for lookups, we used to do simple linear scan backwards, now track first and last block.
Will expire the fss cache at will to reduce memory usage.
Signed-off-by: Derek Collison <derek@nats.io>
If the leader sends messages but the follower for any reason aborts
or retry the snapshot process, it will now send the error that
caused this and the leader can then abort the catchup instead of
waiting for its inactivity threshold of 5 seconds.
Also make the send of a batch be delayed for a bit until the number
of "acks" is 1/2 of the batch size or after reaching 100ms. This
helps avoid trickling of messages. Tested with the new test
TestJetStreamSuperClusterStreamCathupLongRTT() and see better
results both in size of batches and overall time is smaller or
similar but not longer.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
I noticed some contention when I was investigating a catchup bug on the server write lock.
Medium term we could have a separate lock, longer term formal client support in the server will alleviate.
Signed-off-by: Derek Collison <derek@nats.io>
We would send skip messages for a sync request that was completely below our current state, but this could be more traffic then we might want.
Now we only send EOF and the other side can detect the skip forward and adjust on a successful catchup.
We still send skips if we can partially fill the sync request.
Signed-off-by: Derek Collison <derek@nats.io>
```
Replica: Server name unknown at this time (peerID: jZ6RvVRH), outdated, OFFLINE, not seen
```
After discussing with @ripienaar, this text convey better a sense
that this is a transient situation.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
If a cluster is brought down and then partially restarted, the
replica information about the non restarted node would be completely
missing. The CLI could report replicas 3 but then only the leader
and the running replicas, but nothing about the other node.
Since this node's server name is not know, this PR adds an entry
with something similar to this:
```
<unknown (peerID: jZ6RvVRH)>, outdated, OFFLINE, not seen
```
Also, replicas array is now ordered, which will help when using
a watcher or repeating stream info commands in that the replicas
output will be stable in regards to the list of replicas.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
* better error when peer selection fails
It is pretty hard to diagnose what went wrong when not enough peers for
an operation where found. This change now returns counts of reasons why
peers where discarded.
Changed the error to JSClusterNoPeers as it seems more appropriate
of an error for that operation. Not having enough resources is one of
the conditions for a peer not being considered. But so is having a non
matching tag. Which is why JSClusterNoPeers seems more appropriate
In addition, JSClusterNoPeers was already used as error after one call
to selectPeerGroup already.
example:
no suitable peers for placement: peer selection cluster 'C' with 3 peers
offline: 0
excludeTag: 1
noTagMatch: 2
noSpace: 0
uniqueTag: 0
misc: 0
Examle for mqtt:
mid:12 - "mqtt" - unable to connect: create sessions stream for account "$G":
no suitable peers for placement: peer selection cluster 'MQTT' with 3 peers
offline: 0
excludeTag: 0
noTagMatch: 0
noSpace: 0
uniqueTag: 0
misc: 0
(10005)
Signed-off-by: Matthias Hanel <mh@synadia.com>
* review comment
Signed-off-by: Matthias Hanel <mh@synadia.com>
