When a cluster of servers are having routes to each other, there
is a chance that the list of leafnode URLs maintained on each
server is not complete. This would result in LN servers connecting
to this cluster to not get the full list of possible URLs the
server could reconnect to.
Also fixed a DATA RACE that appeared when running the updated
TestLeafNodeInfoURLs test. Fixed the race and added specific
test that easily demonstrated the race: TestLeafNodeNoRaceGeneratingNonce
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This has an effect only on connections created by the server,
so routes and gateways (explicit and implicit).
Make sure that an explicit warning is printed if the insecure
property is set, but otherwise allow it.
Resolves#1062
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
When a solicited leafnode comes from multiple servers that themselves are a cluster, cycles were formed.
This change allows solicited leafnodes to behave similar to gateways in that each server of a cluster
is expected to have a solicted leafnode per destination account and cluster.
We no longer forward subscription interest or messages to a cluster from a server that has a solicited leafnode.
Signed-off-by: Derek Collison <derek@nats.io>
One could craft a PUB protocol to cause server to panic. This can
happen if the size in the PUB protocol overruns an int32.
(note that if authorization is enabled, the user would need to
authenticate first, limiting the impact).
Thank you to Aviv Sasson and Ariel Zelivansky from Twistlock
for the security report!
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
When tls is on routes it can cause reloadAuthorization to be called.
We were assuming configured accounts, but did not copy the remote map.
This copies the remote map when transferring for configured accounts
and also handles operator mode. In operator mode we leave the accounts
in place, and if we have a memory resolver we will remove accounts that
are not longer defined or have bad claims.
Signed-off-by: Derek Collison <derek@nats.io>
- TestSystemAccountConnectionUpdatesStopAfterNoLocal: I believe that
the check on number of notifications was wrong. Since we did not
consume the ones for the connect, the expected count after the
disconnect is 8 instead of 4.
- Possible fix GW tests complaining about number of outbound/inbound
I think that it may be possible that connection does not succeed
right away (remote to fully started, etc) and due to dial timeout
and reconnect attempt delay, I suspect that when given a max time
of 1sec to complete, it may not be enough.
Quick change for now is to override to 2secs for now in the
wait helpers. If that proves conclusive, we could remove the
timeout given to these helpers.
- TestGatewaySendAllSubsBadProtocol: used a t.Fatalf() in checkFor
instead of return fmt.Errorf().
- TestLeafNodeResetsMSGProto: this test is not about change to
interest mode only, so to avoid possible mix of protos, delay
a bit creation of gateway after creation of leaf node.
- Some defer s.Shutdown() were missing
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Suppose two servers, SA in cluster A and SB in cluster B. If SA
sends a message to SB on an account for which there is no interest
at all (account not known or no subscription), SB will send an A-
and keep track that it sent an A- for this account.
When a queue subscription is created on SB, SB will send and RS+
to A because A needs to have perfect knowledge of all queue subs
in all clusters.
If then a regular subscription is also created on SB, SB will
think that it needs to send an A+ because it had sent an A- for
this account. However, SA had an entry for this account for the
queue sub. The A+ would clear the entry in the map and would cause
SA to not send messages to SB even if they would have been a
match for the queue sub on SB.
We fix this in two ways:
- Clear the possible A- in SB when sending an RS+ for queue sub
- Processing of A-/A+ to be aware of a possible entry in the map
due to queue subs.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
If the first protocol for an inbound gateway connection is not
CONNECT, reject with auth violation.
Fixes#1006
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Say there are 2 clusters, A and B. A client connects to A and
publishes messages on an account that B has no interest in.
Then a leaf node server connects to B (using same account than
the no-interest is for). Cluster B will ask cluster
A to switch to interest mode only for leaf node account. This
would cause a panic.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Remove sub from rsubs sublist when user UNSUBs.
Fix bench test that was not actually creating a SUB per request
in the Benchmark_Gateways_Requests_CreateOneSubForEach test.
Also UNSUBs older SUBs after a certain threshold to simulate
actual req/reply.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This addresses the following race:
- client connection creates a subscription on a reply subject
- client connection sends a request
- server sends the subscription to inbound gateway
- server sends the message to outbound gateway (those may be
to different servers)
- receiving server sends to sub interested in request subject
- app sends reply
- its server then check for interest on the reply's subject
In interestOnly mode, there is a possibility that this server
has not received the interest on the reply subject yet and would
then drop the reply.
This PR detects above scenario and will prefix the reply subject
to identify the origin cluster if it is detected that the last
subscription from the sending connection was created less than
a second ago.
Once the destination has this prefix, the destination cluster
will always send back that message to origin cluster even if
there is no registered interest.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Also try multiple email and SANs found in cert until one valid
otherwise, default to the subject in the cert.
```
authorization {
users [
{ user = "app.nats.dev", permissions = {
publish {
allow = ["sandbox.>"]
}
subscribe {
allow = ["sandbox.>"]
}
}
}
]
}
```
Signed-off-by: Waldemar Quevedo <wally@synadia.com>
For cluster, we allow to skip hostname verification from certificate.
We now print a warning when this option is enabled, both on startup
or if the property is enabled on config reload.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
A server that creates a LeafNode connection to a remote cluster
will now be notified of all possible LeafNode URLs in that cluster.
The list is updated when nodes in the cluster come and go.
Also support for advertise address, similar to cluster, gateway, etc..
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
When a route is established, it is possible that each server sends
its list of subscriptions to each other at the same time. Doing
it in place from the readLoop could then cause problems because
each side could reach a point where the outbound socket buffer
is full and no one is dequeuing data (since readLoop is doing
the send of the subs list).
We changed sending this list from a go routine. However, for small
number of subscriptions, it is not required and was causing some
of the tests to fail because of timing issues.
We will now send in place if the estimated size of all protocols
is below a give threshold (1MB).
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
