Added a leafnode lock to allow better traversal without copying of large
leafnodes in a single hub account.
Signed-off-by: Derek Collison <derek@nats.io>
Added a leafnode lock to allow better traversal without copying of large leafnodes in a single hub account.
Signed-off-by: Derek Collison <derek@nats.io>
In #1943 it was adopted to use `UTC()` in some timestamps, but an
unintended side effect from this is that it strips the monotonic time
(e5646b23de),
so it can be prone to clock skews when subtracting time in other areas
of the code.
In #1943 it was adopted to use `UTC()` in some timestamps,
but an unintended side effect from this is that it strips
the monotonic time, so it can be prone to clock skews when
subtracting time in other areas of the code.
e5646b23de
If consumer with multiple subjects encountered a sequnece of messages in
a row from the same subject, it tried to load messages from other
subjects in some cases.
This checks for that scenario and optimizes it by early returning.
I added a temporary instrumentation to check for how many times fetching
new messages is called, and it seems that it cuts those calls according
to assumptions. Though it being internal, it's really hard to show that
in test.
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
If consumer with multiple subjects encountered a sequnece
of messages from the same subject, it tried to load messages
from other subjects in some cases.
This checks for that scenario and optimizes it by early returning.
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
This would impact only cases with accounts defined in configuration file
(as opposed to operator mode). During the configuration reload, new
accounts and sublists were created to later be replaced with existing
ones. That left a window of time where a subscription could have been
added (or attempted to be removed) from the "wrong" sublist. This could
lead to route subscriptions seemingly not being forwarded.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
I have seen cases, maybe due to previous issue with configuration
reload that would miss subscriptions in the sublist because
of the sublist swap, where we would attempt to remove subscriptions
by batch but some were not present. I would have expected that
all present subscriptions would still be removed, even if the
call overall returned an error.
This is now fixed and a test has been added demonstrating that
even on error, we remove all subscriptions that were present.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
One should not access s.opts directly but instead use s.getOpts().
Also, server lock needs to be released when performing an account
lookup (since this may result in server lock being acquired).
A function was calling s.LookupAccount under the client lock, which
technically creates a lock inversion situation.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
When removing a msg and we need to load the msg block and incur IO,
unlock fs lock to avoid stalling other activity on other blocks. E.g
removing and adding msgs at the same time.
Signed-off-by: Derek Collison <derek@nats.io>
This can happen when we reset a stream internally and the stream had a
prior snapshot.
Also make sure to always release resources back to the account
regardless if the store is no longer present.
Signed-off-by: Derek Collison <derek@nats.io>
A new field in `tls{}` blocks force the server to do TLS handshake
before sending the INFO protocol.
```
leafnodes {
port: 7422
tls {
cert_file: ...
...
handshake_first: true
}
remotes [
{
url: tls://host:7423
tls {
...
handshake_first: true
}
}
]
}
```
Note that if `handshake_first` is set in the "accept" side, the first
`tls{}` block in the example above, a server trying to create a LeafNode
connection to this server would need to have `handshake_first` set to
true inside the `tls{}` block of the corresponding remote.
Configuration reload of leafnodes is generally not supported, but TLS
certificates can be reloaded and the support for this new field was also
added.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
A new field in `tls{}` blocks force the server to do TLS handshake
before sending the INFO protocol.
```
leafnodes {
port: 7422
tls {
cert_file: ...
...
handshake_first: true
}
remotes [
{
url: tls://host:7423
tls {
...
handshake_first: true
}
}
]
}
```
Note that if `handshake_first` is set in the "accept" side, the
first `tls{}` block in the example above, a server trying to
create a LeafNode connection to this server would need to have
`handshake_first` set to true inside the `tls{}` block of
the corresponding remote.
Configuration reload of leafnodes is generally not supported,
but TLS certificates can be reloaded and the support for this
new field was also added.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This can happen when we reset a stream internally and the stream had a prior snapshot.
Also make sure to always release resources back to the account regardless if the store is no longer present.
Signed-off-by: Derek Collison <derek@nats.io>
When a fleet of leafnodes are isolated (not routed but using same
cluster) we could do better at optimizing how we update the other
leafnodes since if they are all in the same cluster and we know we are
isolated we can skip.
We can improve further in 2.10.
Signed-off-by: Derek Collison <derek@nats.io>
Under certain scenarios we have witnessed healthz() that will never
return healthy due to a stream or consumer being missing or stopped.
This will now allow the healthz() call to attempt to restart those
assets.
We will also periodically call this in clustered mode from the
monitorCluster routine.
Signed-off-by: Derek Collison <derek@nats.io>
