If some servers in the cluster have the same connect URLs (due
to the use of client advertise), then it would be possible to
have a server sends the connect_urls INFO update to clients with
missing URLs.
Resolves#1515
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This was discovered with the test TestLeafNodeWithGatewaysServerRestart
that was sometimes failing. Investigation showed that when cluster B
was shutdown, one of the server on A that had a connection from B
that just broke tried to reconnect (as part of reconnect retries of
implicit gateways) to a server in B that was in the process of shuting down.
The connection had been accepted but createGateway not called because
the server's running boolean had been set to false as part of the shutdown.
However, the connection was not closed so the server on A had a valid
connection to a dead server from cluster B. When the B cluster (now single
server) was restarted and a LeafNode connection connected to it, then
the gateway from B to A was created, that server on A did not create outbound
connection to that B server because it already had one (the zombie one).
So this PR strengthens the starting of accept loops and also make sure
that if a connection (all type of connections) is not accepted because
the server is shuting down, that connection is properly closed.
Since all accept loops had almost same code, made a generic function
that accept functions to call specific create connection functions.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Leafnodes that formed clusters were partially supported. This adds proper support for origin cluster, subscription suppression and data message no echo for the origin cluster.
Signed-off-by: Derek Collison <derek@nats.io>
Added cluster names as required for prep work for clustered JetStream. System can dynamically pick a cluster name and settle on one even in large clusters.
Signed-off-by: Derek Collison <derek@nats.io>
We added authentication override block for websocket configuration
in PR #1463 and #1465 which somehow introduced a drop in perf as
reported by the bench tests.
This PR refactors a bit to restore the performance numbers.
This change also fixes the override behavior for websocket auth:
- If websocket's NoAuthUser is configured, the websocket's auth
block MUST define Users, and the user be present.
- If there is any override (username/pwd,token,etc..) then the
whole block config will be used when authenticating a websocket
client, which means that if websocket NoAuthUser is empty we
are not falling back to the regular client's NoAuthUser config.
- TLSMap always override the regular client's config. That is,
whatever TLSMap value specified in the websocket's tls{} block
will be used.
The TLSMap configuration was not used for LeafNodes. The behavior
now will be:
- If LeafNode's auth block contains users and TLSMap is true,
the user is looked up based on the cert's info. If not found,
authentication will fail. If found, it will be authenticated
and bound to associated account.
- If no user is specified in LeafNode's auth block and TLSMap
is true, then the cert's info will be used against the global
users map.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Websocket can now override
- Username/password
- Token
- Users
- NKeys
- no_auth_user
- auth_timeout
For TLS, support for verify and verify_and_map. We used to set
tls config's ClientAuth to NoClientCert. It will now depend
if the config requires client certificate verification, which
is needed if TLSMap is enabled.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
The grace period used to be hardcoded at 10 seconds.
This option allows the user to configure the amount of time the
server will wait before initiating the closing of client connections.
Note that the grace period needs to be strictly lower than the overall
lame_duck_duration. The server deducts the grace period from that
overall duration and spreads the closing of connections during
that time.
For instance, if there are 1000 connections and the lame duck
duration is set to 30 seconds and grace period to 10, then
the server will use 30-10 = 20 seconds to spread the closing
of those 1000 connections, so say roughly 50 clients per second.
Resolves#1459.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This is related to #1408.
Make sure that we close the websocket "accept loop" if configured
before proceeding with the lame duck mode.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This will ensure that there is no race where clients are accepted
after the LDM INFO notification.
Also add to the test to make sure that we don't send INFO when
routes are disconnected due to internal closing of connections
during the shutdown process.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Also send an INFO to routes so that the remotes can remove the
LDM's server client URLs and notify their own clients of this
change.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Websocket support can be enabled with a new websocket
configuration block:
```
websocket {
# Specify a host and port to listen for websocket connections
# listen: "host:port"
# It can also be configured with individual parameters,
# namely host and port.
# host: "hostname"
# port: 4443
# This will optionally specify what host:port for websocket
# connections to be advertised in the cluster
# advertise: "host:port"
# TLS configuration is required
tls {
cert_file: "/path/to/cert.pem"
key_file: "/path/to/key.pem"
}
# If same_origin is true, then the Origin header of the
# client request must match the request's Host.
# same_origin: true
# This list specifies the only accepted values for
# the client's request Origin header. The scheme,
# host and port must match. By convention, the
# absence of port for an http:// scheme will be 80,
# and for https:// will be 443.
# allowed_origins [
# "http://www.example.com"
# "https://www.other-example.com"
# ]
# This enables support for compressed websocket frames
# in the server. For compression to be used, both server
# and client have to support it.
# compression: true
# This is the total time allowed for the server to
# read the client request and write the response back
# to the client. This include the time needed for the
# TLS handshake.
# handshake_timeout: "2s"
}
```
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This contains a rewrite to the services layer for exporting and importing. The code this merges to already had a first significant rewrite that moved from special interest processing to plain subscriptions.
This code changes the prior version's dealing with reverse mapping which was based mostly on thresholds and manual pruning, with some sporadic timer usage. This version uses the jetstream branch's code that understands interest and failed deliveries. So this code is much more tuned to reacting to interest changes. It also removes thresholds and goes only by interest changes or expirations based around a new service export property, response thresholds. This allows a service provider to provide semantics on how long a response should take at a maximum.
This commit also introduces formal support for service export streamed and chunked response types send an empty message to signify EOF.
This commit also includes additions to the service latency tracking such that errors are now sent, not only successful interactions. We have added a Status field and an optional Error fields to ServiceLatency.
We support the following Status codes, these are directly from HTTP.
400 Bad Request (request did not have a reply subject)
408 Request Timeout (when system detects request interest went away, old request style to make dependable)..
503 Service Unavailable (no service responders running)
504 Service Timeout (The new response threshold expired)
Signed-off-by: Derek Collison <derek@nats.io>
This is the first checkin for JetStream. Has some rudimentary basics working.
TODO
1. Push vs pull mode for observables. (work queues)
2. Disk/File store, memory only for now.
3. clustering code - design shaping up well.
4. Finalize account import semantics.
5. Lots of other little things.
Signed-off-by: Derek Collison <derek@nats.io>
