This could happen for stream with R>1 but with a durable that
has an override of R=1.
Fixed a test to make sure assets have an elected leader.
Also fixed a gateway test that would cause a data race.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
The server was not properly handling queue subscriptions internal
notifications which could lead to messages not being delivered
to applications using a queue group to consume from a JetStream
consumer after they were restarted.
Resolves#1066
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
In some rare situations, it is possible that nodes are added
to the cluster but are not properly tracked and not shown as
offline when they exit the cluster.
Relates to #3258
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Internally jetstream may subscribe to some subject and then send
a request with a reply subject matching that subscription.
Due to interest propagation through a super cluster, it is possible
that the reply comes back to a node that is not yet aware of
the subscription interest which would cause the reply to be dropped.
Some code detects that the subscription is recent and "map" the
reply subject so that it can be routed back to the origin server.
However, this was done with the use of the connection object that
created the subscription, but at the time of the send, a different
internal "*client" object may be used which would then cause
the code to not be aware of the recent subscription and not do
the mapping.
This code was changed to scope at the account level instead of
connection.
A recent change in PR #3412 is no longer needed and was reverted
in favor of changes in this PR.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
The original value was hardcoded to 128MB and 32MB per stream. The
per-server limit is lowered to 32MB but is configurable with
a new configuration parameter:
```
jetstream {
max_catchup: 8MB
}
```
The per-stream limit was also lowered from 32MB/128,000msgs to
8MB/32,000 messages.
Tests have shown no difference in performance for fast links.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
When a subscription is recently made, gateway code ensures that if
there is a reply subject, the reply is "mapped" or rewritten to allow
the reply to come back to the origin cluster, regardless of subscription
interest propagation.
The issue was that this uses a map with a `*client` as the key
but the pointer for SYSTEM clients would not always be the same,
which meant that the rewrite would not happen, causing possible
"loss" of replies.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
There was a way to detect a cycle but I believe it needs to be
a stack of "si" not just the one before invoking processServiceImport.
Changes in #3393 would solve issue reported with test TestAccountImportCycle,
but would not address the new reported issue represented by new test
TestLeafNodeSvcImportExportCycle. This current approach seems to solve
all known cases.
Resolves#3397
Replaces #3393
