1
0
mirror of https://github.com/taigrr/nats.docs synced 2025-01-18 04:03:23 -08:00

Merge branch 'master' into mqtt

This commit is contained in:
Ginger Collison
2021-03-15 10:52:36 -05:00
committed by GitHub
60 changed files with 3343 additions and 175 deletions

View File

@@ -118,6 +118,8 @@ authorization: {
| [`gateway`](gateways/gateway.md#gateway-configuration-block) | Configuration map for [gateway](gateways/). | |
| [`leafnode`](leafnodes/leafnode_conf.md) | Configuration map for a [leafnode](leafnodes/). | |
| [`mqtt`](mqtt/mqtt_conf.md) | Configuration map for a [mqtt](mqtt/). | |
| [`websocket`](websocket/websocket_conf.md) | Configuration map for [websocket](websocket/). | |
### Connection Timeouts

View File

@@ -27,7 +27,7 @@ If gateways are to be used in a cluster, **all** servers of this cluster need to
A nats-server in a gateway role will specify a port where it will accept gateway connections. If the configuration specifies other _external_ `gateways`, the gateway will create one outbound gateway connection for each gateway in its configuration. It will also gossip other gateways it knows or discovers. Fewer _external_ `gateways` mean less configuration. Yet, the ability to discover more gateways and gateway nodes depends on these servers running. This is similar to _seed server_ in cluster. It is recommended to have all _seed server_ of a cluster listed in the `gateways` section.
If the local cluster has three gateway nodes, this means there will be three outbound connections to each external gateway.
If the local cluster has three gateway nodes, this means there will be three outbound connections from the local cluster to each external gateway cluster.
![Gateway Connections](../../../.gitbook/assets/simple.svg)

View File

@@ -93,6 +93,32 @@ If other form of credentials are used \(jwt, nkey or other\), then the server wi
| `account` | [Account](../securing_nats/accounts.md) name or jwt public key identifying the local account to bind to this remote server. Any traffic locally on this account will be forwarded to the remote server. |
| `credentials` | Credential file for connecting to the leafnode server. |
| `tls` | A [TLS configuration](leafnode_conf.md#tls-configuration-block) block. Leafnode client will use specified TLS certificates when connecting/authenticating. |
| `ws_compression` | If connecting with [Websocket](leafnode_conf#connecting-using-websocket-protocol) protocol, this boolean (`true` or `false`) indicates to the remote server that it wishes to use compression. The default is `false`. |
| `ws_no_masking` | If connecting with [Websocket](leafnode_conf#connecting-using-websocket-protocol) protocol, this boolean indicates to the remote server that it wishes not to mask outbound websocket frames. The default is `false`, which means that outbound frames will be masked. |
### Connecting using Websocket protocol
Since NATS 2.2.0, Leaf nodes support outbound websocket connections by specifying `ws` as the scheme component of the remote server URLs:
```
leafnodes {
remotes [
{urls: ["ws://hostname1:443", "ws://hostname2:443"]}
]
}
```
Note that if a URL has the `ws` scheme, all URLs the list must be `ws`. You cannot mix and match.
Therefore this would be considered an invalid configuration:
```
remotes [
# Invalid configuration that will prevent the server from starting
{urls: ["ws://hostname1:443", "nats-leaf://hostname2:7422"]}
]
```
Note that the decision to make a TLS connection is not based on `wss://` (as opposed to `ws://`) but instead in the presence of a TLS configuration in the `leafnodes{}` or the specific remote configuration block.
To configure Websocket in the remote server, check the [Websocket](../websocket/websocket_conf.md) secion.
### `tls` Configuration Block

View File

@@ -10,6 +10,7 @@ To monitor the NATS messaging system, `nats-server` provides a lightweight HTTP
* [Gateways](monitoring.md#gateway-information)
* [Leaf Nodes](monitoring.md#leaf-nodes-information)
* [Subscription Routing](monitoring.md#subscription-routing-information)
* [JetStream Information](monitoring.md#jetstream-information)
All endpoints return a JSON object.
@@ -56,7 +57,7 @@ http: localhost:8222
For example, to monitor this server locally, the endpoint would be [http://localhost:8222/varz](http://localhost:8222/varz). It reports various general statistics.
## Monitoring endpoints
## Monitoring Endpoints
The following sections describe each supported monitoring endpoint: `varz`, `connz`, `routez`, `subsz`, `gatewayz`, and `leafz`. There are not any required arguments, however use of arguments can let you tailor monitoring to your environment and tooling.
@@ -246,6 +247,61 @@ You can also report detailed subscription information on a per connection basis
}
```
### JetStream Information
The `/jsz` endpoint reports information about the JetStream subsystem.
**Endpoint:** `http://server:port/jsm`
| Result | Return Code |
| :--- | :--- |
| Success | 200 \(OK\) |
| Error | 400 \(Bad Request\) |
#### Arguments
| Argument | Values | Description |
| :--- | :--- | :--- |
| acc | account name | Provide information for a specfic account |
| accounts | true, 1, false, 0 | Provide information for all accounts. The default is false. |
| streams | true, 1, false, 0 | Include stream information. The default is false. |
| consumers | true, 1, false, 0 | Include consumer information. The default is false. |
| config | true, false | Include configuration with streams and consumers. The default is false. |
| offset | integer > 0 | Pagination offset. Default is 0. |
| limit | integer > 0 | Number of results to return. Default is 1024. |
| leader-only | true, false | TODO |
As noted above, the `routez` endpoint does support the `subs` argument from the `/connz` endpoint. For example: [http://demo.nats.io:8222/routez?subs=1](http://demo.nats.io:8222/jsz?accounts=1&streams=1&consumers=1&config=1)
#### Example
* Get JetStream information: [http://host:port/jsz?accounts=1&streams=1&consumers=1&config=1](http://host:port/jsz?accounts=1&streams=1&consumers=1&config=1)
#### Response
```javascript
{
"server_id": "NACDVKFBUW4C4XA24OOT6L4MDP56MW76J5RJDFXG7HLABSB46DCMWCOW",
"now": "2019-06-24T14:29:16.046656-07:00",
"num_routes": 1,
"routes": [
{
"rid": 1,
"remote_id": "de475c0041418afc799bccf0fdd61b47",
"did_solicit": true,
"ip": "127.0.0.1",
"port": 61791,
"pending_size": 0,
"in_msgs": 0,
"out_msgs": 0,
"in_bytes": 0,
"out_bytes": 0,
"subscriptions": 0
}
]
}
```
### Route Information
The `/routez` endpoint reports information on active routes for a cluster. Routes are expected to be low, so there is no paging mechanism with this endpoint.
@@ -432,7 +488,7 @@ The `/gatewayz` endpoint reports information about gateways used to create a NAT
}
```
### Leaf Nodes Information
### Leaf Node Information
The `/leafz` endpoint reports detailed information about the leaf node connections.
@@ -520,6 +576,147 @@ The `/subsz` endpoint reports detailed information about the current subscriptio
}
```
## JetStream Information
The `/jsz` endpoint reports more detailed information on JetStream. For accounts it uses a paging mechanism which defaults to 1024 connections.
**Endpoint:** `http://server:port/connz`
| Result | Return Code |
| :--- | :--- |
| Success | 200 \(OK\) |
| Error | 400 \(Bad Request\) |
#### Arguments
| Argument | Values | Description |
| :--- | :--- | :--- |
| acc | account name | Include metrics for the specified account. Default is unset. |
| accounts | true, 1, false, 0 | Include account specific JetStream information. Default is false. |
| streams | true, 1, false, 0 | Include streams. When set, implies `accounts=true`. Default is false. |
| consumers | true, 1, false, 0 | Include consumer. When set, implies `streams=true`. Default is false. |
| config | true, 1, false, 0 | When stream or consumer are requested, include their respective configuration. Default is false. |
| leader-only | true, 1, false, 0 | Only the leader responds. Default is false.|
| offset | number > 0 | Pagination offset. Default is 0. |
| limit | number > 0 | Number of results to return. Default is 1024. |
#### Examples
Get basic JetStream information: [http://demo.nats.io:8222/jsz](http://demo.nats.io:8222/jsz)
Request accounts and control limit and offset: [http://demo.nats.io:8222/jsz?accounts=true&limit=16&offset=128](http://demo.nats.io:8222/jsz?accounts=true&limit=16&offset=128).
You can also report detailed consumer information on a per connection basis using consumer=true. For example: [http://demo.nats.io:8222/jsz?consumers=true](http://demo.nats.io:8222/jsz/consumer=true).
#### Response
```javascript
{
"server_id": "NCVIDODSZ45C5OD67ZD7EJUIJPQDP6CM74SJX6TJIF2G7NLYS5LCVYHS",
"now": "2021-02-08T19:08:30.555533-05:00",
"config": {
"max_memory": 10485760,
"max_storage": 10485760,
"store_dir": "/var/folders/9h/6g_c9l6n6bb8gp331d_9y0_w0000gn/T/srv_7500251552558"
},
"memory": 0,
"storage": 66,
"api": {
"total": 5,
"errors": 0
},
"total_streams": 1,
"total_consumers": 1,
"total_messages": 1,
"total_message_bytes": 33,
"meta_cluster": {
"name": "cluster_name",
"replicas": [
{
"name": "server_5500",
"current": false,
"active": 2932926000
}
]
},
"account_details": [
{
"name": "BCC_TO_HAVE_ONE_EXTRA",
"id": "BCC_TO_HAVE_ONE_EXTRA",
"memory": 0,
"storage": 0,
"api": {
"total": 0,
"errors": 0
}
},
{
"name": "ACC",
"id": "ACC",
"memory": 0,
"storage": 66,
"api": {
"total": 5,
"errors": 0
},
"stream_detail": [
{
"name": "my-stream-replicated",
"cluster": {
"name": "cluster_name",
"replicas": [
{
"name": "server_5500",
"current": false,
"active": 2931517000
}
]
},
"state": {
"messages": 1,
"bytes": 33,
"first_seq": 1,
"first_ts": "2021-02-09T00:08:27.623735Z",
"last_seq": 1,
"last_ts": "2021-02-09T00:08:27.623735Z",
"consumer_count": 1
},
"consumer_detail": [
{
"stream_name": "my-stream-replicated",
"name": "my-consumer-replicated",
"created": "2021-02-09T00:08:27.427631Z",
"delivered": {
"consumer_seq": 0,
"stream_seq": 0
},
"ack_floor": {
"consumer_seq": 0,
"stream_seq": 0
},
"num_ack_pending": 0,
"num_redelivered": 0,
"num_waiting": 0,
"num_pending": 1,
"cluster": {
"name": "cluster_name",
"replicas": [
{
"name": "server_5500",
"current": false,
"active": 2933232000
}
]
}
}
]
}
]
}
]
}
```
## Creating Monitoring Applications
NATS monitoring endpoints support [JSONP](https://en.wikipedia.org/wiki/JSONP) and [CORS](https://en.wikipedia.org/wiki/Cross-origin_resource_sharing#How_CORS_works). You can easily create single page web applications for monitoring. To do this you simply pass the `callback` query parameter to any endpoint.

View File

@@ -190,3 +190,6 @@ The above example shows how clients without authentication can be associated wit
> Please note that the `no_auth_user` will not work with nkeys. The user referenced can also be part of the [authorization](authorization.md) block.
> Despite `no_auth_user` being set, clients still need to communicate that they will not be using credentials.
> The [authentication timeout](auth_intro/auth_timeout.md) applies to this process as well.
> When your connection is slow, you may run into this timeout and the resulting `Authentication Timeout` error, despite not providing credentials.

View File

@@ -29,12 +29,14 @@ Listening on [>]
Tokens can be bcrypted enabling an additional layer of security, as the clear-text version of the token would not be persisted on the server configuration file.
You can generate bcrypted tokens and passwords using the [`mkpasswd`](../../../../nats-tools/mkpasswd.md) tool:
You can generate bcrypted tokens and passwords using the [`nats`](../../../../nats-tools/natscli.md) tool:
```text
> mkpasswd
pass: dag0HTXl4RGg7dXdaJwbC8
bcrypt hash: $2a$11$PWIFAL8RsWyGI3jVZtO9Nu8.6jOxzxfZo7c/W0eLk017hjgUKWrhy
> nats server passwd
? Enter password [? for help] **********************
? Reenter password [? for help] **********************
$2a$11$PWIFAL8RsWyGI3jVZtO9Nu8.6jOxzxfZo7c/W0eLk017hjgUKWrhy
```
Here's a simple configuration file:

View File

@@ -30,12 +30,14 @@ authorization: {
## Bcrypted Passwords
Username/password also supports bcrypted passwords using the [`mkpasswd`](../../../../nats-tools/mkpasswd.md) tool. Simply replace the clear text password with the bcrypted entries:
Username/password also supports bcrypted passwords using the [`nats`](../../../../nats-tools/natscli.md) tool. Simply replace the clear text password with the bcrypted entries:
```text
> mkpasswd
pass: (Uffs#rG42PAu#Oxi^BNng
bcrypt hash: $2a$11$V1qrpBt8/SLfEBr4NJq4T.2mg8chx8.MTblUiTBOLV3MKDeAy.f7u
> nats server passwd
? Enter password [? for help] **********************
? Reenter password [? for help] **********************
$2a$11$V1qrpBt8/SLfEBr4NJq4T.2mg8chx8.MTblUiTBOLV3MKDeAy.f7u
```
And on the configuration file:
@@ -44,7 +46,7 @@ And on the configuration file:
authorization: {
users: [
{user: a, password: "$2a$11$V1qrpBt8/SLfEBr4NJq4T.2mg8chx8.MTblUiTBOLV3MKDeAy.f7u"},
...
...
]
}
```

View File

@@ -1,9 +1,10 @@
# Account lookup using Resolver
# Account Lookup Using a Resolver
The `resolver` configuration option is used in conjunction with [NATS JWT Authentication](./) and [nsc](../../../../nats-tools/nsc/). The `resolver` option specifies a URL where the nats-server can retrieve an account JWT. There are two built-in resolver implementations:
The `resolver` configuration option is used in conjunction with [NATS JWT Authentication](./) and [nsc](../../../../nats-tools/nsc/). The `resolver` option specifies a URL where the nats-server can retrieve an account JWT. There are three built-in resolver implementations:
* `URL`
* `MEMORY`
* [`URL`](resolver.md#URL-Resolver)
* [`MEMORY`](resolver.md#Memory)
* [NATS Based Resolver](resolver.md#nats-based-resolver)
> If the operator JWT specified in `operator` contains an account resolver URL, `resolver` only needs to be specified in order to overwrite that default.
@@ -34,3 +35,68 @@ The `MEMORY` resolver is recommended when the server has a small number of accou
For more information on how to configure a memory resolver, see [this tutorial](mem_resolver.md).
## NATS Based Resolver
The NATS based resolver embeds the functionality of the [account server](https://github.com/nats-io/nats-account-server) inside the nats-server.
In order to avoid having to store all account JWT on every server, this resolver has two sub types `full` and `cache`.
Their commonalities are that they exchange/lookup account JWT via NATS and the system account, and store them in a local (not shared) directory.
### Full
The Full resolver stores all JWTs and exchanges them in an eventually consistent way with other resolvers of the same type.
[`nsc`](../../../../nats-tools/nsc/README.md) supports push/pull/purge with this resolver type.
[JWTs](../../nats-server/configuration/securing_nats/jwt/), uploaded this way, are stored in a directory the server has exclusive access to.
```yaml
resolver: {
type: full
# Directory in which account jwt will be stored
dir: './jwt'
# In order to support jwt deletion, set to true
# If the resolver type is full delete will rename the jwt.
# This is to allow manual restoration in case of inadvertent deletion.
# To restore a jwt, remove the added suffix .delete and restart or send a reload signal.
# To free up storage you must manually delete files with the suffix .delete.
allow_delete: false
# Interval at which a nats-server with a nats based account resolver will compare
# it's state with one random nats based account resolver in the cluster and if needed,
# exchange jwt and converge on the same set of jwt.
interval: "2m"
# limit on the number of jwt stored, will reject new jwt once limit is hit.
limit: 1000
}
```
This resolver type also supports `resolver_preload`. When present, JWTs are listed and stored in the resolver.
There, they may be subject to updates. Restarts of the `nats-server` will hold on to these more recent versions.
Not every server in a cluster needs to be set to `full`.
You need enough to still serve your workload adequately, while some servers are offline.
### Cache
The Cache resolver only stores a subset of [JWT](../../nats-server/configuration/securing_nats/jwt/) and evicts others based on an LRU scheme.
Missing JWTs are downloaded from `full` nats based resolver.
This resolver is essentially the URL Resolver in NATS.
```yaml
resolver: {
type: cache
# Directory in which account jwt will be store
dir: "./"
# limit on the number of jwt stored, will evict old jwt once limit is hit.
limit: 1000
# How long to hold on to a jwt before discarding it.
ttl: "2m"
}
```
### NATS Based Resolver - Integration
The NATS based resolver utilizes the system account for lookup and upload of account [JWTs](../../nats-server/configuration/securing_nats/jwt/) .
If your application requires tighter integration you can make use of these subjects for tighter integration.
To upload or update any generated account JWT without [`nsc`](../../../../nats-tools/nsc/README.md), send it as a request to `$SYS.REQ.CLAIMS.UPDATE`.
Each participating `full` NATS based account resolver will respond with a message detailing success or failure.
To serve a requested account [JWT](../../nats-server/configuration/securing_nats/jwt/) yourself and essentially implement an account server, subscribe to `$SYS.REQ.ACCOUNT.*.CLAIMS.LOOKUP` and respond with the account JWT corresponding to the requested account id (wildcard).

View File

@@ -0,0 +1,12 @@
# Websocket
*Supported since NATS Server version 2.2*
Websocket support can be enabled in the server and may be used alongside the
traditional TCP socket connections. TLS, compression and
Origin Header checking are supported.
**Important**
- NATS Supports only Websocket data frames in Binary, not Text format (https://tools.ietf.org/html/rfc6455#section-5.6). The server will always send in Binary and your clients MUST send in Binary too.
- For writers of client libraries: a Websocket frame is not guaranteed to contain a full NATS protocol (actually will generally not). Any data from a frame must be going through a parser that can handle partial protocols. See the protocol description [here](../../../nats-protocol/nats-protocol/README.md).

View File

@@ -0,0 +1,158 @@
# Configuration
To enable websocket support in the server, add a `websocket` configuration
block in the server's configuration file like the following:
```
websocket {
# Specify a host and port to listen for websocket connections
#
# listen: "host:port"
# It can also be configured with individual parameters,
# namely host and port.
#
# host: "hostname"
port: 443
# This will optionally specify what host:port for websocket
# connections to be advertised in the cluster.
#
# advertise: "host:port"
# TLS configuration is required by default
#
tls {
cert_file: "/path/to/cert.pem"
key_file: "/path/to/key.pem"
}
# For test environments, you can disable the need for TLS
# by explicitly setting this option to `true`
#
# no_tls: true
# [Cross-origin resource sharing option](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). When set to `true`, the HTTP origin header must match the requests hostname.
# The default is `false`.
#
# same_origin: true
# [Cross-origin resource sharing option](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). List of accepted origins. When empty, and `same_origin` is `false`, clients from any origin are allowed to connect.
# This list specifies the only accepted values for the client's request Origin header. The scheme,
# host and port must match. By convention, the absence of TCP port in the URL will be port 80
# for an "http://" scheme, and 443 for "https://".
#
# allowed_origins [
# "http://www.example.com"
# "https://www.other-example.com"
# ]
# This enables support for compressed websocket frames
# in the server. For compression to be used, both server
# and client have to support it.
#
# compression: true
# This is the total time allowed for the server to
# read the client request and write the response back
# to the client. This includes the time needed for the
# TLS handshake.
#
# handshake_timeout: "2s"
# Name for an HTTP cookie, that if present will be used as a client JWT.
# If the client specifies a JWT in the CONNECT protocol, this option is ignored.
# The cookie should be set by the HTTP server as described [here](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies).
# This setting is useful when generating NATS `Bearer` client JWTs as the
# result of some authentication mechanism. The HTTP server after correct
# authentication can issue a JWT for the user, that is set securely preventing
# access by unintended scripts. Note these JWTs must be [NATS JWTs](https://docs.nats.io/nats-server/configuration/securing_nats/jwt).
#
# jwt_cookie: "my_jwt_cookie_name"
# If no user name is provided when a websocket client connects, will default
# this user name in the authentication phase. If specified, this will
# override, for websocket clients, any `no_auth_user` value defined in the
# main configuration file.
# Note that this is not compatible with running the server in operator mode.
#
# no_auth_user: "my_username_for_apps_not_providing_credentials"
# See below to know what is the normal way of limiting websocket clients
# to specific users.
# If there are no users specified in the configuration, this simple authorization
# block allows you to override the values that would be configured in the
# equivalent block in the main section.
#
# authorization {
# # If this is specified, the client has to provide the same username
# # and password to be able to connect.
# # username: "my_user_name"
# # password: "my_password"
#
# # If this is specified, the password field in the CONNECT has to
# # match this token.
# # token: "my_token"
#
# # This overrides the main's authorization timeout. For consistency
# # with the main's authorization configuration block, this is expressed
# # as a number of seconds.
# # timeout: 2.0
#}
}
```
## Authorization of Websocket Users
A new field when configuring users allows you to restrict which type of connections are allowed for a specific user.
Consider this configuration:
```
authorization {
users [
{user: foo password: foopwd, permission: {...}}
{user: bar password: barpwd, permission: {...}}
]
}
```
If a websocket client were to connect and use the username `foo` and password `foopwd`, it would be accepted.
Now suppose that you would want websocket client to only be accepted if it connected using the username `bar`
and password `barpwd`, then you would use the option `allowed_connection_types` to restrict which type
of connections can bind to this user.
```
authorization {
users [
{user: foo password: foopwd, permission: {...}}
{user: bar password: barpwd, permission: {...}, allowed_connection_types: ["WEBSOCKET"]}
]
}
```
The option `allowed_connection_types` (also can be named `connection_types` or `clients`) as you can see
is a list, and you can allow several type of clients. Suppose you want the user `bar` to accept both
standard NATS clients and websocket clients, you would configure the user like this:
```
authorization {
users [
{user: foo password: foopwd, permission: {...}}
{user: bar password: barpwd, permission: {...}, allowed_connection_types: ["STANDARD", "WEBSOCKET"]}
]
}
```
The absence of `allowed_connection_types` means that all type of connections are allowed (the default behavior).
The possible values are currently:
* `STANDARD`
* `WEBSOCKET`
* `LEAFNODE`
* `MQTT`
## Leaf nodes connections
You can configure remote Leaf node connections so that they connect to the Websocket port instead of the Leaf node port.
See [Leafnode](../leafnodes/leafnode_conf.md#connecting-using-websocket-protocol) section.

View File

@@ -1,9 +1,6 @@
# Lame Duck Mode
In production we recommend that a server is shut down with lame duck mode
as a graceful way to slowly evict clients. With large deployments this
mitigates the "thundering herd" situation that will place CPU pressure on
servers as TLS enabled clients reconnect.
In production we recommend that a server is shut down with lame duck mode as a graceful way to slowly evict clients. With large deployments this mitigates the "thundering herd" situation that will place CPU pressure on servers as TLS enabled clients reconnect.
## Server
@@ -13,16 +10,9 @@ Lame duck mode is initiated by signaling the server:
nats-server --signal ldm
```
After entering lame duck mode, the server will stop accepting new connections,
wait for a 10 second grace period, then begin to evict clients over a period of time
configurable by the [lame_duck_duration](https://docs.nats.io/nats-server/configuration#runtime-configuration)
configuration option. This period defaults to 2 minutes.
After entering lame duck mode, the server will stop accepting new connections, wait for a 10 second grace period, then begin to evict clients over a period of time configurable by the [lame\_duck\_duration](https://docs.nats.io/nats-server/configuration#runtime-configuration) configuration option. This period defaults to 2 minutes.
## Clients
When entering lame duck mode, the server will send a message to clients. Some
maintainer supported clients will invoke an optional callback indicating that
a server is entering lame duck mode. This is used for cases where an application
can benefit from preparing for the short outage between the time it is evicted and
automatically reconnected to another server.
When entering lame duck mode, the server will send a message to clients. Some maintainer supported clients will invoke an optional callback indicating that a server is entering lame duck mode. This is used for cases where an application can benefit from preparing for the short outage between the time it is evicted and automatically reconnected to another server.

View File

@@ -38,7 +38,7 @@ nats-sub -s nats://localhost:4222 ">"
`nats-sub` is a subscriber sample included with all NATS clients. `nats-sub` subscribes to a subject and prints out any messages received. You can find the source code to the go version of `nats-sub` in [GitHub](https://github.com/nats-io/nats.go/tree/master/examples). After starting the subscriber you should see a message on 'A' that a new client connected.
We have two servers and a client. Time to simulate our rolling upgrade. But wait, before we upgrade 'A', let's introduce a new server 'C'. Server 'C' will join the existing cluster while we perform the upgrade. Its sole purpose is to provide an additional place where clients can go other than 'A' and ensure we don't end up with a single server serving all the clients after the upgrade procedure. Clients will randomly select a server when connecting unless a special option is provided that disables that functionality \(usually called 'DontRandomize' or 'noRandomize'\). You can read more about ["Avoiding the Thundering Herd"](). Suffice it to say that clients redistribute themselves about evenly between all servers in the cluster. In our case 1/2 of the clients on 'A' will jump over to 'B' and the remaining half to 'C'.
We have two servers and a client. Time to simulate our rolling upgrade. But wait, before we upgrade 'A', let's introduce a new server 'C'. Server 'C' will join the existing cluster while we perform the upgrade. Its sole purpose is to provide an additional place where clients can go other than 'A' and ensure we don't end up with a single server serving all the clients after the upgrade procedure. Clients will randomly select a server when connecting unless a special option is provided that disables that functionality \(usually called 'DontRandomize' or 'noRandomize'\). You can read more about ["Avoiding the Thundering Herd"](upgrading_cluster.md). Suffice it to say that clients redistribute themselves about evenly between all servers in the cluster. In our case 1/2 of the clients on 'A' will jump over to 'B' and the remaining half to 'C'.
Let's start our temporary server: