We are phasing out the optimistic-only mode. Servers accepting
inbound gateway connections will switch the accounts to interest-only
mode.
The servers with outbound gateway connection will check interest
and ignore the "optimistic" mode if it is known that the corresponding
inbound is going to switch the account to interest-only. This is
done using a boolean in the gateway INFO protocol.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This is to avoid a narrow race between adding server and them catching
up where they also register as current.
Also wait for all peers to be caught up.
This also avoids clearing catchup marker once catchup stalled.
A stalled catchup would remove the marker causing the peer to
register as current.
Signed-off-by: Matthias Hanel <mh@synadia.com>
if an origin stream contains:
1M msgs with subject foo and 1M msgs with subject bar
IF the source consumer changes their filter from foo to bar
Then it would have received messages for subject bar.
This happens because this tail was filtered and their
respective seqno was not communicated to the consumer.
This is somewhat unexpected. It is also coincidental.
Had the last message in the stream had subject foo then
this wouldn't happen.
Therefore, when completely changing the subject say,
from foo to bar, we only receive messages received
after the time the change was made.
However, if the old and new subject overlap in any way,
we go by sequence number. Meaning in these cases the
outlined behavior remains in order to not induce artificial
message loss for the part of the subject space that is
covered by old and new filter.
Signed-off-by: Matthias Hanel <mh@synadia.com>
Signed-off-by: Matthias Hanel <mh@synadia.com>
The monitoring http server is started early and the gateway setup
(when configured) may not be fully ready when the `/gatewayz`
endpoint is inspected and could cause a panic.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
ordering of templates got messed up by a map (now removed)
Also improved error message when template generation fails
Signed-off-by: Matthias Hanel <mh@synadia.com>
Test TestNoRaceJetStreamClusterCorruptWAL() would start to flap
because of the snapshot on cluster shutdown. Disable the snapshot
on exit for this test by stopping the raft node before shutdown.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
- didRemove in applyMetaEntries() could be reset when processing
multiple entries
- change "no race" test names to include JetStream
- separate raft nodes leader stepdown and stop in server
shutdown process
- in InstallSnapshot, call wal.Compact() with lastIndex+1
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This is a regression introduced in v2.8.3. If a message reaches
the max redeliver count, it stops being delivered to the consumer.
However, after a server or cluster restart, those messages would
be redelivered again.
Resolves#3361
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
For security reasons we have introduced scoped signing keys to jwt.
They carry user permissions.
Wich is why jwt issued by those keys are not allowed to carry their own permission.
Instead they are copied from the signing key.
If the scoped signing key gets compromised, an attacker can only issue jwt with the permissions of the key.
With a plain signing key, an attacker can create arbitrary user with permissions.
Because user jwt creation is greatly simplified we added a single utility function to go/java/.net which issues user for such keys.
This is function is documented in ADR-14:
```
/**
* signingKey, is a mandatory account nkey pair to sign the generated jwt.
* accountId, is a mandatory public account nkey. Will return error when not set or not account nkey.
* publicUserKey, is a mandatory public user nkey. Will return error when not set or not user nkey.
* name, optional human readable name. When absent, default to publicUserKey.
* expiration, optional but recommended duration, when the generated jwt needs to expire. If not set, JWT will not expire.
* tags, optional list of tags to be included in the JWT.
*
* Returns:
* error, when issues arose.
* string, resulting jwt.
**/
IssueUserJWT(signingKey nkey, accountId string, publicUserKey string, name string, expiration time.Duration, tags []string) (error, string)
```
Currently the only downside of this is that the permissions are static and can't be tailored to the user.
This PR changes that by allowing the user pub/sub permissions to be parameterized with templates.
templates are for entire tokens only and include:
{{name()}} -> username
{{subject()}} -> user subject (nkey)
{{account-name()}} -> users account name
{{account-subject()}} -> user accoutn subject (nkey)
{{tag(arbitrary-prefix)}}
provided the tag "arbitrary-prefix:value" will result in "value"
provided the tags ["arbitrary-prefix:1", "arbitrary-prefix:2"] will result in two subjects "1" & "2"
If the resulting subject is not valid.
Say a tag is not present or name is not set.
This will result in an error for deny subjects
and result in no subject for allow subject.
Signed-off-by: Matthias Hanel <mh@synadia.com>
Signed-off-by: Matthias Hanel <mh@synadia.com>
In standalone mode, when attempting to create a stream which has
subjects that overlap with an existing stream, the generic
stream create error "10049" was returned instead of the more
accurate "10065" error code corresponding to subject overlap,
as it was the case in clustered mode.
Resolves#3362
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
In normal message get, the returned format is RFC3339Nano, which
is what is being used when using JSON marshaling. However, for
the direct get we had to pass a string to construct the header
and we were using time.Time.String() which was using a different
layout. So use time.Time.MarshalJSON() to be consistent with
the non-direct get message.
Libraries that already parsed the non RFC3339Nano time format
can be updated since none should have been released yet (since
the feature in the server is not released yet)
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
* Fix race between stream stop and monitorStream
monitorCluster stops the stream, when doing so, monitorStream
needs to be stopped to avoid miscounting of store size.
In a test stop and reset of store size happened first and then
was followed by storing more messages via monitorStream
Signed-off-by: Matthias Hanel <mh@synadia.com>
This can happen if the move was initiated by the user.
A subsequent cancel resets the initial peer list.
The original peer list was picked on the old set of tags.
A cancel would then keep the new list of tags but reset
to the old peers. Thus tags and peers diverge.
The problem is that at the time of cancel, the old
placement tags can't be found anymore.
This fix causes cancel to remove the placement tags, if
the old peers do not satisfy the new placement tags.
Signed-off-by: Matthias Hanel <mh@synadia.com>
* fixed consumer restart on source filter update
When a stream source filter subject was updated, the internal consumer
was not re created
If the upstream stream contains a tail of previously filtered messages,
these will now be delivered
Signed-off-by: Matthias Hanel <mh@synadia.com>
* Added check for source/mirror filter subjects
When the origin stream exists, the sourec/mirror filter subject
will be checked against the stream subjects.
If there is no overlap, an error will be returned
Signed-off-by: Matthias Hanel <mh@synadia.com>
Use better indexing for lookups, we used to do simple linear scan backwards, now track first and last block.
Will expire the fss cache at will to reduce memory usage.
Signed-off-by: Derek Collison <derek@nats.io>
If the leader sends messages but the follower for any reason aborts
or retry the snapshot process, it will now send the error that
caused this and the leader can then abort the catchup instead of
waiting for its inactivity threshold of 5 seconds.
Also make the send of a batch be delayed for a bit until the number
of "acks" is 1/2 of the batch size or after reaching 100ms. This
helps avoid trickling of messages. Tested with the new test
TestJetStreamSuperClusterStreamCathupLongRTT() and see better
results both in size of batches and overall time is smaller or
similar but not longer.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
