1. We were holding open FDs longer than we should for consumers causing issues with open FD limits. We now do not hold them open and cap updates a bit better.
2. When doing a stream delete, consumer delete was repeating alot of work that was not necessary, causing longer delays. This has been optimized a bit, still more improvements to be made.
3. We cover all JS under a single export, but that was also trapping GetNext for pull based consumers, and since this was a no-op (is handled at user account level) we were creating alot of garbage service import responses and reverse map entries that had to be garbage collected. We have a fix in to avoind this but still looking for a better one.
4. Still had some lingering references to all exports vs single JS export.
Signed-off-by: Derek Collison <derek@nats.io>
* [fixed] hanging leaf node connection when account can't be found
as a result of the issue, the leaf node connection never got created,
even after the account can be found.
Also tracing account id and name (when available)
Signed-off-by: Matthias Hanel <mh@synadia.com>
The issue was that the subscription created for the MQTT client
was resulting in creation of a shadow subscription which did not
have the mqtt specific object attached, which would cause the
panic when accessing it in the sub's icb.
After that, it was discovered that the wrong subject was passed
to deliverMsg(), so fixed that too so that the icb callback gets
the proper transformed subject.
Resolves#2265
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
* [changed] pinned certs to check the server connected to as well
on reload clients with removed pinned certs will be disconnected.
The check happens only on tls handshake now.
Signed-off-by: Matthias Hanel <mh@synadia.com>
We had a report of corrupt message payloads when going across leafnodes between streams that were sourced from one another.
We were incorrectly using the underlying buffer when a header already existed.
Signed-off-by: Derek Collison <derek@nats.io>
When a response was needed from a leafnode cluster back to a hub, we had rules to disallow.
That rule was a bit dated and since we have cluster origin for leafnode clusters and that
is checked before the message is actually sent we could remove the old rule.
Signed-off-by: Derek Collison <derek@nats.io>
This structure is used in ClientAuthentication, an interface
designed to let 3rd parties extend the authentication mechanisms
of the server
In order to allow those 3rd parties to create unit tests, mocks etc
we need to export this structure so it's accessible externally
Signed-off-by: R.I.Pienaar <rip@devco.net>
This CL adds in support for multiple JetStream domains using mapped subjects.
Mapping subjects aligns well with the JetStream context APIPrefix in clients.
Signed-off-by: Derek Collison <derek@nats.io>
Some issues that have been fixed would manifest by timeouts on
connect, unexpected memory usage on high publish message rate.
Some details:
- Replies were not always GW routed properly because we were looking
at the wrong connection's rsubs
- GW routed replies would not be found because they were tracked
in the subscription's client object, which may not be the same used
to send the reply
- Increased the mqtt timeout to wait for JS replies since in some
tests it was sometimes taking more than the original 2 seconds
- Incoming gateway messages destined for an MQTT internal subscription
may have been rejected as a no interest if the account had service imports
- Don't use time.After(), instead create explicit timer so it can
be stopped when not timing out.
- Unnecessary copy of a slice since we were converting to a string anyway.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
we want to auto-suppress JetStream traffic on normal accounts.
We also now track remote accounts so that client info headers can be remapped.
Signed-off-by: Derek Collison <derek@nats.io>
For internal callbacks we can not rely on the GW routed reply tracking
since internal clients can change who receives vs sends response.
Signed-off-by: Derek Collison <derek@nats.io>
In a setup with a cluster of servers to which 2 different leaf nodes
attach to, and queue subs are attached to one of the leaf, if the
leaf server is restarted and reconnects to another server in the
cluster, there was a risk for an infinite message loop between
some servers in the "hub" cluster.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This could happen when a leafnode has permissions set and another
connection (client, etc..) is about to assign a message to the
leafnode while the leafnode itself is receiving messages and they
both check permissions at the same time.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This was caused by not sending subs across leaf node connections in some
cases but sending unsub in all cases. This imbalance caused
subscriptions to go away too soon. (ref count was off)
Signed-off-by: Matthias Hanel <mh@synadia.com>
If the subscription was foo. > but the server also had an import deny of foo.bar
It was legal to send the subscription. But the other server was unaware
of the restriction and sent the message anyway. The check of the
incoming message did not happen.
Fixing by ignoring messages the server is not supposed to receive.
And exchange deny_import so that the non soliciting leaf node knows to not
send these messages in the first place.
NB. merging of deny_ export/import with perms from INFO happens in processLeafnodeInfo
Signed-off-by: Matthias Hanel <mh@synadia.com>
On connect all subscription where sent by the soliciting leaf node.
If creds contains sub deny permissions, the leaf node would be
disconnected.
This waits for the permissions to be exchanged and checks permissions
before sending subscriptions.
Signed-off-by: Matthias Hanel <mh@synadia.com>
- It was possible that when the server was sending frames to a
webbrowser, it would send empty frames. While technically not wrong,
prevent that from happening.
- Not copying enqueued buffers could cause corruption with LN+WS.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
- Fixed the close of a TLS connection which starting Go 1.16
set the deadline to 5 seconds.
- Fixed an issue with setHeader that was causing these error messages
```
=== RUN TestServiceImportReplyMatchCycleMultiHops
nats: message could not decode headers on connection [4] for subscription on "foo"
--- PASS: TestServiceImportReplyMatchCycleMultiHops (0.04s)
```
- Fixed names of tests in norace_test.go since they must start with
TestNoRace in order to make sure that we execute them in Travis:
```
go test -v -run=TestNoRace --failfast -p=1 ./...
```
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This also applies to times that end up in that json.
Where applicable moved time.Now() to where it is used.
Moved calls to .UTC() to where time is created it that time is converted
later anyway.
Signed-off-by: Matthias Hanel <mh@synadia.com>
