Changed the introduced new option and added a new one. The idea
is to be able to differentiate between never connected and reconnected
event. The never connected situation will be logged at first attempt
and every hour (by default, configurable).
However, once connected and if trying to reconnect, will report every
attempts by default, but this is configurable too.
These two options are supported for config reload.
Related to #1000
Related to #1001Resolves#969
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This is a continuation of #1000. Added a configuration to specify
the number of attempts at which the repeated error is reported.
The algo is now to print only the 1st attempt and when current
attempt % <this config param> == 0.
Resolves#969
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This applies to routes, gateways and leaf node connections.
The failed attempts will be printed at the first, after the first
minute and then every hour.
The connect/error statements now include the attempt number.
Note that in debug mode, all attempts are traced, so you may get
double trace (one for debug, one for info/error).
Resolves#969
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Remove sub from rsubs sublist when user UNSUBs.
Fix bench test that was not actually creating a SUB per request
in the Benchmark_Gateways_Requests_CreateOneSubForEach test.
Also UNSUBs older SUBs after a certain threshold to simulate
actual req/reply.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This addresses the following race:
- client connection creates a subscription on a reply subject
- client connection sends a request
- server sends the subscription to inbound gateway
- server sends the message to outbound gateway (those may be
to different servers)
- receiving server sends to sub interested in request subject
- app sends reply
- its server then check for interest on the reply's subject
In interestOnly mode, there is a possibility that this server
has not received the interest on the reply subject yet and would
then drop the reply.
This PR detects above scenario and will prefix the reply subject
to identify the origin cluster if it is detected that the last
subscription from the sending connection was created less than
a second ago.
Once the destination has this prefix, the destination cluster
will always send back that message to origin cluster even if
there is no registered interest.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
For cluster, we allow to skip hostname verification from certificate.
We now print a warning when this option is enabled, both on startup
or if the property is enabled on config reload.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
A server that creates a LeafNode connection to a remote cluster
will now be notified of all possible LeafNode URLs in that cluster.
The list is updated when nodes in the cluster come and go.
Also support for advertise address, similar to cluster, gateway, etc..
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
When a route is established, it is possible that each server sends
its list of subscriptions to each other at the same time. Doing
it in place from the readLoop could then cause problems because
each side could reach a point where the outbound socket buffer
is full and no one is dequeuing data (since readLoop is doing
the send of the subs list).
We changed sending this list from a go routine. However, for small
number of subscriptions, it is not required and was causing some
of the tests to fail because of timing issues.
We will now send in place if the estimated size of all protocols
is below a give threshold (1MB).
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
What is not completed:
1. TLS
2. config to bind local account.
3. Info updates for solicitor to track topology changes like a client.
4. CONNECT sent after INFO for nonce authroization.
5. Authorization
6. Services and Streams tests.
7. config file parsing.
Signed-off-by: Derek Collison <derek@nats.io>
If each server has a long list of subscriptions, when the route
is established, sending this list could result in each server
treating the peer as a slow consumer, resulting in a reconnect,
etc..
Also bumping the fan-in threshold for route connections.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Moving some of the connection initialization post TLS handshake
to avoid temporary memory growth when getting repeated failed
connections to any of the client, route and gateway ports.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This is not complete solution and is a bit hacky but is a start
to be able to have service import work at least in some basic
cases.
Also fixed a bug where replySub would not be removed from
connection's list of subs after delivery.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Changed account lookup and validation failures to be more understandable by users.
Changed limits to be -1 for unlimited to match jwt pkg.
The limits changed exposed problems with options holding real objects causing issues with reload tests under race mode.
Longer term this code should be reworked such that options only hold config data, not real structs, etc.
Signed-off-by: Derek Collison <derek@nats.io>
Specifically this is to support distributed tracking of number of account connections across clusters.
Gateways may not work yet based on attempts to only generate payloads when we know there is outside interest.
Signed-off-by: Derek Collison <derek@nats.io>
- If/when splitting buffer to pass to queueOutbound(), it has to
be include full protocol.
- Fix counting of total queue subs
- Fix tests
- Send RS- if no plain sub interest even if there is queue sub
interest.
- Removed a one-liner function
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
- Solve RS+ with wildcards
- Solve issue with messages not send to remote gateways queue subs
if there was a qsub on local server.
- Made rcache a perAccountCache since it is now used by routes and
gateways
- Order outbound gateways only on RTT updates
- Print a server's gateway name on startup
- Augment/add some tests
- Update TLS handling: when connecting, use hostname for ServerName
if url is not IP, otherwise use a hostname that we saved when
parsing/adding URLs for the remote gateway.
- Send big buffer in chunks if needed.
- Add caching for qsubs match
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This will allow to signal multiple servers at once to go in
that mode and not have their client reconnect to one of the
servers in the group.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Allow deny clauses for subscriptions to still allow wildcard subscriptions but do not deliver the messages themselves.
Signed-off-by: Derek Collison <derek@nats.io>
