We now share more information about the responder and the requestor. The requestor information by default is not shared, but can be when declaring the import.
Also fixed bug for error handling on old request style requests that would always result on a 408 response.
Signed-off-by: Derek Collison <derek@nats.io>
This contains a rewrite to the services layer for exporting and importing. The code this merges to already had a first significant rewrite that moved from special interest processing to plain subscriptions.
This code changes the prior version's dealing with reverse mapping which was based mostly on thresholds and manual pruning, with some sporadic timer usage. This version uses the jetstream branch's code that understands interest and failed deliveries. So this code is much more tuned to reacting to interest changes. It also removes thresholds and goes only by interest changes or expirations based around a new service export property, response thresholds. This allows a service provider to provide semantics on how long a response should take at a maximum.
This commit also introduces formal support for service export streamed and chunked response types send an empty message to signify EOF.
This commit also includes additions to the service latency tracking such that errors are now sent, not only successful interactions. We have added a Status field and an optional Error fields to ServiceLatency.
We support the following Status codes, these are directly from HTTP.
400 Bad Request (request did not have a reply subject)
408 Request Timeout (when system detects request interest went away, old request style to make dependable)..
503 Service Unavailable (no service responders running)
504 Service Timeout (The new response threshold expired)
Signed-off-by: Derek Collison <derek@nats.io>
This bring these to same level as the JS events, these are the ones
I care for right now but will do this to the rest here in time as well
and document them in JSON schema
Signed-off-by: R.I.Pienaar <rip@devco.net>
This is the first checkin for JetStream. Has some rudimentary basics working.
TODO
1. Push vs pull mode for observables. (work queues)
2. Disk/File store, memory only for now.
3. clustering code - design shaping up well.
4. Finalize account import semantics.
5. Lots of other little things.
Signed-off-by: Derek Collison <derek@nats.io>
Fixes#1372 by updating s.sys.account pointer.
This issue also showed that accounts are unnecessarily reloaded.
This happened because account imports were not copied and thus,
deepEqual detected a difference were none was.
This was addressed by making the copy less shallow.
Furthermore did deepEqual detects a difference when it compared
slices that were appended to while processing a map.
This was fixed by sorting before comparison.
Noticed that Account.clients stored an unnecessary pointer.
Removed duplicated code in systemAccount.
Signed-off-by: Matthias Hanel <mh@synadia.com>
This is needed to put those accounts into interest only mode for inbound gateway connections. Also added code
to make sure we were doing proper account tracking and would track the global account as well, which used to
be excluded.
Fixes#977
Signed-off-by: Derek Collison <derek@nats.io>
Fixed#1296, by altering client state on reload
Detect a trace level change on reload and update all clients.
To avoid data races, read client.trace while holding the lock,
pass the value into functionis that trace while not holding the lock.
Delete unused client.debug.
Signed-off-by: Matthias Hanel <mh@synadia.com>
- Call flushOutbound() for SYSTEM connections
- Flush in place in internalSendLoop when sending the shutdown event
- Fix some tests:
- missing defer client connection Close()
- ensure subs are registered and messages received before shutdown
of leafnode server to check disconnected event's stats.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
We had too much special processing, so reduced to a single wildcard
which will propagate across routes and gateways and is consistent
with gateway handling of globally routed subjects and timeouts.
Signed-off-by: Derek Collison <derek@nats.io>
Want to keep this commit separate so that we can easily remove
when we no longer want to support both prefixes.
- If this server receives a "$GR." message, it takes the subject
and tries to process this locally. If there is no cluster race
reply may be received ok (like before).
- If this server sends a routed reply, it detects if sending to
an older server (then uses $GR.) or not (then uses $GNR)
- Gateway INFO has a new field that indicates if the server is
using the new prefix.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
- Add atomic in client to skip check in processInboundClientMsg()
if value is 0. Avoids getting the lock in fast path if not needed.
- Have a timer per client instead of the global server list that
was expiring: noticed a lot of contention there when running
some perf/profiling tests. The timer is also not reset for
every timestamp that is not yet expired since this too affects
performance. Instead fires are regular interval and cleared
when map is empty after a cycle.
- Move processing of gw map rely on its own function (in inbound msg).
I have verified that this is inlined same way as when code was
directly in processInboundClientMsg.
- Use string(subj[]) for prefix detection: I have verified that
it is actually faster.
- Builds the RMSG with appends to local buffer in handleGatewayReply()
instead of using fmt.Sprintf().
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
- New prefix that includes origin server for the request
- Mapping done if request is service import or requestor has
recent subscription
- Subscription considered recent if less than 250ms
- Destination server strip GW prefix before giving to client
and restore when getting a reply on that subject
- Mapping removed aftert 250ms
- Server rejects client publish on "$GNR." (the new prefix)
- Cluster and server hash are now 8 chars long and from base 62
alphabets
- Mapped replies need to be sent to leafnode servers due to race
(cluster B sends RS+ on GW inbound then RMSG on outbound, the
RS+ may be processed later and cluster A may have given message
to LN before RS+ on reply subject. So LN needs to accept the
mapped reply but will strip to give to client and reassemble
before sending it back)
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
If a client RTT for a requestor is longer than a service RTT, the requestor latency was often zero.
We now wait for the RTT (if zero) before sending out the metric.
Signed-off-by: Derek Collison <derek@nats.io>
This adds a new config option server_name that
when set will be exposed in varz, events and more
as a descriptive name for the server.
If unset though the server_name will default to the pk
Signed-off-by: R.I.Pienaar <rip@devco.net>
Ensure that lookupAccount does not hold server lock during
updateAccount and fetchAccount.
Updating the account cannot have the server lock because it is
possible that during updateAccountClaims(), clients are being
removed, which would try to get the server lock (deep down in
closeConnection/s.removeClient).
Added a test that would have show the deadlock prior to changes
in this PR.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This is the first pass at introducing exported services to the system account for generally debugging of blackbox systems.
The first service reports number of subscribers for a given subject. The payload of the request is the subject, and optional queue group, and can contain wildcards.
Signed-off-by: Derek Collison <derek@nats.io>
Will now breakout the internal NATS latency to show requestor client RTT, responder client RTT and any internal latency caused by hopping between servers, etc.
Signed-off-by: Derek Collison <derek@nats.io>
If account A imports from B and B from A, when the account A
is built, it causes B to be fetch, but since B imports from A,
A was fetch/built again in an infinite loop.
Resolves#1117
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Take into account tracking of response maps that are created and do proper cleanup.
Also fixes#1089 which was discovered while working on this.
Signed-off-by: Derek Collison <derek@nats.io>
The timer was not set with the proper variable, which caused the
check to always think that a new timer should be created, which
would lead to more and more timers being created which translated
to updates being sent more and more frequently.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
What is not completed:
1. TLS
2. config to bind local account.
3. Info updates for solicitor to track topology changes like a client.
4. CONNECT sent after INFO for nonce authroization.
5. Authorization
6. Services and Streams tests.
7. config file parsing.
Signed-off-by: Derek Collison <derek@nats.io>
