There was a test to prevent an errorneous loop detection when a
remote would reconnect (due to a stale connection) while the accepting
side did not detect the bad connection yet.
However, this test was racy because the test was done prior to add
the connections to the map.
In the case of a misconfiguration where the remote creates 2 different
remote connections that end-up binding to the same account in the
accepting side, then it was possible that this would not be detected.
And when it was, the remote side would be unaware since the disconnect/
reconnect attempts would not show up if not running in debug mode.
This change makes sure that the detection is no longer racy and returns
an error to the remote so at least the log/console of the remote will
show the "duplicate connection" error messages.
Resolves#1730
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Suppose a cluster of 2 servers, let's call them leaf1 and leaf2.
These servers are routed and have a leaf connection to another
server, let's call it srv1.
They share the same cluster name.
If a queue subscriber runs on srv1 and a queue subscriber on the
same subject/group name runs on leaf1, if a requestor runs on
leaf2, the request should reach only one of the 2 queue subs.
The defect was that sometimes both queue subs would receive the
message.
The added test checks that only one reply is ever received and
that the local "leaf" cluster is preferred.
Resolves#1722
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Connections normally suppress sending PINGs if there was some
activity. We now force GATEWAY connections to send PINGs at the
configured interval or 15 seconds, whichever is the smallest.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Addresses stack overflow issue wally was seeing with configs
that mix and match streams and services between each other.
Signed-off-by: Derek Collison <derek@nats.io>
The mappings enable traffic shaping functionality.
Also added a subject transform which can match any subject and transform to a new one, even re-ordering partial wildcards.
Signed-off-by: Derek Collison <derek@nats.io>
When creating shadow subscriptions for import streams, we were
not invoking code for gateway subscription accounting, which means
that when the account (for leafnodes) was switched to interest
only, those shadow subscriptions were not sent.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Users and NKey users will now have the option to specify a list
of allowed connection types.
This will allow for instance a certain user to be allowed to
connect as a standard NATS client, but not as Websocket, or
vice-versa.
This also fixes the websocket auth override. Indeed, with
the original behavior, the websocket users would have been bound
to $G, which would not work when there are accounts defined, since
when that is the case, no app can connect/bind to $G account.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
We previously simply called DialTimeout() on a route's url when
soliciting. If it resolved to the IP of the host, it would create
a route to self, which server detects, but then would not try again
with other IPs that would have allowed to form a cluster with
other servers running on the other IPs.
This PR keeps track of local IPs + cluster port and exclude them
from the list of IPs returned by LookupHost API. This even prevent
solicitation of routes to self. Only non-local IPs will be tried.
Resolves#1586
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Latency reports will include the header(s) responsible for the trace
Updated ADR to have it reflect implementation
Signed-off-by: Matthias Hanel <mh@synadia.com>
because times stored are hh:mm:ss it is possible to end up with start > end where end is actually the next day.
jwt.go line 189
Also, ranges are based on the servers location, not the clients.
Signed-off-by: Matthias Hanel <mh@synadia.com>
Made changes to processSub() to accept subscription properties,
including the icb callback so that it is set prior to add the
subscription to the account's sublist, which prevent races.
Fixed some other racy conditions, notably in addServiceImportSub()
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This change allows the removal of the connection and update of
the server state to be done "in place" but still delay the flushing
of and close of tcp connection to the writeLoop. With ref counting
we ensure that the reconnect happens after the flushing but not
before the state has been updated.
Had to fix some places where we may have called closeConnection()
from under the server lock since it now would deadlock for sure.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
The call sendProtoNow() should not normally be used (only when
setting up a connection when the writeloop is not yet started and
server needs to send something before being able to start the
writeLoop.
Instead, code should use enqueueProto(). For this particular
case though, use queueOutbound() directly and add to the
producer's pcd map.
Also fixed other places where we were using queueOutbound() +
flushSignal() which is what enqueueProto is doing.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
That is, if the server receives "SUB foo 1" more than once from
the same client, we would register in the client map this subscription
only once, and add to the account's sublist only once, however we
would have updated shadow subscriptions and route/gateway maps for
each SUB protocol, which would result in inability to send unsubscribe
to routes when the client goes away or unsubscribes.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
If an IPv6 address contains some "%" characters, this was causing
the connection name in log statement to mess up the Sprintf formatting.
The solution is to escape those "%" characters.
Resolves#1505
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Leafnodes that formed clusters were partially supported. This adds proper support for origin cluster, subscription suppression and data message no echo for the origin cluster.
Signed-off-by: Derek Collison <derek@nats.io>
When a leafnode would connect with credentials that had permissions the spoke did not have a way of knowing what those were.
This could lead to being disconnected when sending subscriptions or messages to the hub which were not allowed.
Signed-off-by: Derek Collison <derek@nats.io>
Added cluster names as required for prep work for clustered JetStream. System can dynamically pick a cluster name and settle on one even in large clusters.
Signed-off-by: Derek Collison <derek@nats.io>
There is a race between the time the processing of a subscription
and the init/send of subscriptions when accepting a leaf node
connection that may cause internally a subscription's subject
to be counted many times, which would then prevent the send of
an LS- when the subscription's interest goes away.
Imagine this sequence of events, each side represents a "thread"
of execution:
```
client readLoop leaf node readLoop
----------------------------------------------------------
recv SUB foo 1
sub added to account's sublist
recv CONNECT
auth, added to acc.
updateSmap
smap["foo"]++ -> 1
no LS+ because !allSubsSent
init smap
finds sub in acc sl
smap["foo"]++ -> 2
sends LS+ foo
allSubsSent == true
recv UNSUB 1
updateSmap
smap["foo"]-- -> 1
no LS- because count != 0
----------------------------------------------------------
```
Equivalent result but with slightly diffent execution:
```
client readLoop leaf node readLoop
----------------------------------------------------------
recv SUB foo 1
sub added to account's sublist
recv CONNECT
auth, added to acc.
init smap
finds sub in acc sl
smap["foo"]++ -> 1
sends LS+ foo
allSubsSent == true
updateSmap
smap["foo"]++ -> 2
no LS+ because count != 1
recv UNSUB 1
updateSmap
smap["foo"]-- -> 1
no LS- because count != 0
----------------------------------------------------------
```
The approach for the fix is delay the creation of the smap
until we actually initialize the map and send the subs on processing
of the CONNECT.
In the meantime, as soon as the LN connection is registered
and available in updateSmap, we check that smap is nil or
not. If nil, we do nothing.
In "init smap" we keep track of the subscriptions that have been
added to smap. This map will be short lived, just enough to
protect against races above.
In updateSmap, when smap is not nil, we need to checki, if we
are adding, that the subscription has not already been handled.
The tempory subscription map will be ultimately emptied/set to
nil with the use of a timer (if not emptied in place when
processing smap updates).
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
