Shuffle the array created when iterating through the gateways URLs
map since map iteration may not be well randomized with small maps.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This is a continuation of #1000. Added a configuration to specify
the number of attempts at which the repeated error is reported.
The algo is now to print only the 1st attempt and when current
attempt % <this config param> == 0.
Resolves#969
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This applies to routes, gateways and leaf node connections.
The failed attempts will be printed at the first, after the first
minute and then every hour.
The connect/error statements now include the attempt number.
Note that in debug mode, all attempts are traced, so you may get
double trace (one for debug, one for info/error).
Resolves#969
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Say there are 2 clusters, A and B. A client connects to A and
publishes messages on an account that B has no interest in.
Then a leaf node server connects to B (using same account than
the no-interest is for). Cluster B will ask cluster
A to switch to interest mode only for leaf node account. This
would cause a panic.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Remove sub from rsubs sublist when user UNSUBs.
Fix bench test that was not actually creating a SUB per request
in the Benchmark_Gateways_Requests_CreateOneSubForEach test.
Also UNSUBs older SUBs after a certain threshold to simulate
actual req/reply.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This addresses the following race:
- client connection creates a subscription on a reply subject
- client connection sends a request
- server sends the subscription to inbound gateway
- server sends the message to outbound gateway (those may be
to different servers)
- receiving server sends to sub interested in request subject
- app sends reply
- its server then check for interest on the reply's subject
In interestOnly mode, there is a possibility that this server
has not received the interest on the reply subject yet and would
then drop the reply.
This PR detects above scenario and will prefix the reply subject
to identify the origin cluster if it is detected that the last
subscription from the sending connection was created less than
a second ago.
Once the destination has this prefix, the destination cluster
will always send back that message to origin cluster even if
there is no registered interest.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
What is not completed:
1. TLS
2. config to bind local account.
3. Info updates for solicitor to track topology changes like a client.
4. CONNECT sent after INFO for nonce authroization.
5. Authorization
6. Services and Streams tests.
7. config file parsing.
Signed-off-by: Derek Collison <derek@nats.io>
Based on @softkbot PR #913.
Removed the command line parameter, which then removes the need for Options.Cluster.TLSInsecure.
Added a test with config reload.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Check sublist only when required.
Send the subs list in place instead of go routine (gateways have
different outbound/inbound connections so they don't suffer same
issue than routes)
Bump the default array size when collecting gateway connections
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
We now send A- if an account does not exists, or if there is no
interest on a given subject and no existing subscription.
An A+ is sent if an A- was previously sent and a subscription
for this account is registered.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Moving some of the connection initialization post TLS handshake
to avoid temporary memory growth when getting repeated failed
connections to any of the client, route and gateway ports.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Otherwise, this may be sent to servers in the cluster and to other
gateways which may result in attempt to connect to self which
in case of TLS would produce error.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Do the swapping to outbound connection only on send.
It means that those subs are stored in the inbound connection and
those are the only type of subs stored there. So on connection close
it is easy to clean them up.
Also instead of having processMsgResults have to return this sub,
simply check the size of r.psubs and if 1, the type of client
associated with it. If gateway, we know we have to do the direct
send.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This is not complete solution and is a bit hacky but is a start
to be able to have service import work at least in some basic
cases.
Also fixed a bug where replySub would not be removed from
connection's list of subs after delivery.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
We can't use a simple sync.Map here because the noInterest map
for inbound gateway connections are used concurrently. Indeed,
whenever an account would have been registered or a new sub created
this could trigger an update of that map in order to clear the
fact that we had sent an A-/RS- and now are sending an A+/RS+.
So changed to simple map but protected by gw connection's lock.
Without this change, server would panic if there are messages
published to cluster A that are sent to server B while a sub
is then created on matching subject on B.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
- If/when splitting buffer to pass to queueOutbound(), it has to
be include full protocol.
- Fix counting of total queue subs
- Fix tests
- Send RS- if no plain sub interest even if there is queue sub
interest.
- Removed a one-liner function
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
- Solve RS+ with wildcards
- Solve issue with messages not send to remote gateways queue subs
if there was a qsub on local server.
- Made rcache a perAccountCache since it is now used by routes and
gateways
- Order outbound gateways only on RTT updates
- Print a server's gateway name on startup
- Augment/add some tests
- Update TLS handling: when connecting, use hostname for ServerName
if url is not IP, otherwise use a hostname that we saved when
parsing/adding URLs for the remote gateway.
- Send big buffer in chunks if needed.
- Add caching for qsubs match
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
