taigrr/nats.docs
1
0
Fork 0
mirror of https://github.com/taigrr/nats.docs synced 2025-01-18 04:03:23 -08:00
Code Issues Projects Releases Wiki Activity

wip

Browse Source
This commit is contained in:
Alberto Ricart 2019-05-15 15:45:36 -05:00
parent 12104b7678
commit a9cf7544ff
13 changed files with 1119 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored
View File

@ -4,3 +4,5 @@ _docs/
.vscode
.idea/
node_modules
.DS_Store

27
SUMMARY.md
View File

@ -7,26 +7,31 @@
* [NATS Server](nats_server/README.md)
* [Installing](nats_server/installation.md)
* [Running](nats_server/running.md)
* [Window Service](nats_server/windows_srv.md)
* [Clients](nats_server/clients.md)
* [Flags](nats_server/flags.md)
* [Configuration File](nats_server/configuration.md)
* [Configuration](nats_server/configuration.md)
* [Authentication](nats_server/authentication.md)
* [Authorization](nats_server/authorization.md)
* [Clustering](nats_server/clustering.md)
* [TLS Security](nats_server/tls.md)
* [Logging](nats_server/logging.md)
* [Monitoring](nats_server/monitoring.md)
* [Signals](nats_server/signals.md)
* [Window Service](nats_server/windows_srv.md)
* [Upgrading a Cluster](nats_server/upgrading.md)
## Managing NATS Server
* [Upgrading a Cluster](nats_admin/upgrading_cluster.md)
* [Slow Consumers](nats_admin/slow_consumers.md)
* [Signals](nats_admin/signals.md)
## Containerization
* [NATS.docker](nats_docker/README.md)
* [NATS.docker](nats_docker/README.md)
* [Tutorial](nats_docker/tutorial.md)
## NATS Tools
* [NATS Top](nats_tools/nats_top/README.md)
* [Tutorial](nats_tools/nats_top/tutorial.md)
* [NATS Top](nats_tools/nats_top/README.md)
* [Tutorial](nats_tools/nats_top/tutorial.md)
* [Benchmarking](nats_tools/natsbench.md)
## Developing With NATS
@ -85,3 +90,11 @@
* [Explore NATS Request/Reply](developer/tutorials/reqreply.md)
* [Explore NATS Queueing](developer/tutorials/queues.md)
* [Benchmarking NATS](developer/tutorials/benchmarking.md)
## NATS Protocol
* [Protocol Demo](nats_protocol/nats-protocol-demo.md)
* [Client Protocol](nats_protocol/nats-protocol.md)
* [Developing a Client](nats_protocol/nats-client-dev.md)
* [NATS Cluster Protocol](nats_protocol/nats-server-protocol.md)

0
nats_server/signals.md → nats_admin/signals.md
View File

113
nats_admin/slow_consumers.md Normal file
View File

@ -0,0 +1,113 @@
## Slow Consumers
To support resiliency and high availability, NATS provides built-in mechanisms to automatically prune the registered listener interest graph that is used to keep track of subscribers, including slow consumers and lazy listeners. NATS automatically handles a slow consumer. If a client is not processing messages quick enough, the NATS server cuts it off. To support scaling, NATS provides for auto-pruning of client connections. If a subscriber does not respond to ping requests from the server within the [ping-pong interval](/documentation/internals/nats-protocol/), the client is cut off (disconnected). The client will need to have reconnect logic to reconnect with the server.
# Slow Consumers
In core NATS, consumers that cannot keep up are handled differently from many other messaging systems: NATS favors the approach of protecting the system as a whole over accommodating a particular consumer to ensure message delivery.
__What is a slow consumer?__
A slow consumer is a subscriber that cannot keep up with the message flow delivered from the NATS server. This is a common case in distributed systems because it is often easier to generate data than it is to process it. When consumers cannot process data fast enough, back pressure is applied to the rest of the system. NATS has mechanisms to reduce this back pressure.
NATS identifies slow consumers in the client or the server, providing notification through registered callbacks, log messages, and statistics in the server's monitoring endpoints.
__What happens to slow consumers?__
When detected at the client, the application is notified and messages are dropped to allow the consumer to continue and reduce potential back pressure. When detected in the server, the server will disconnect the connection with the slow consumer to protect itself and the integrity of the messaging system.
## Slow consumers identified in the client
A [client can detect it is a slow consumer](/documentation/writing_applications/advanced#slow-consumers) on a local connection and notify the application through use of the asynchronous error callback. It is better to catch a slow consumer locally in the client rather than to allow the server to detect this condition. This example demonstrates how to define and register an asynchronous error handler that will handle slow consumer errors.
```go
func natsErrHandler(nc *nats.Conn, sub *nats.Subscription, natsErr error) {
fmt.Printf("error: %v\n", natsErr)
if natsErr == nats.ErrSlowConsumer {
pendingMsgs, _, err := sub.Pending()
if err != nil {
fmt.Printf("couldn't get pending messages: %v", err)
return
}
fmt.Printf("Falling behind with %d pending messages on subject %q.\n",
pendingMsgs, sub.Subject)
// Log error, notify operations...
}
// check for other errors
}
// Set the error handler when creating a connection.
nc, err := nats.Connect("nats://localhost:4222",
nats.ErrorHandler(natsErrHandler))
```
With this example code and default settings, a slow consumer error would generate output something like this:
```sh
error: nats: slow consumer, messages dropped
Falling behind with 65536 pending messages on subject "foo".
```
Note that if you are using a synchronous subscriber, `Subscription.NextMsg(timeout time.Duration)` will also return an error indicating there was a slow consumer and messages have been dropped.
## Slow consumers identified by the server
When a client does not process messages fast enough, the server will buffer messages in the outbound connection to the client. When this happens and the server cannot write data fast enough to the client, in order to protect itself, it will designate a subscriber as a "slow consumer" and may drop the associated connection.
When the server initiates a slow consumer error, you'll see the following in the server output:
```sh
[54083] 2017/09/28 14:45:18.001357 [INF] ::1:63283 - cid:7 - Slow Consumer Detected
```
The server will also keep count of the number of slow consumer errors encountered, available through the monitoring `varz` endpoint in the `slow_consumers` field.
## Handling slow consumers
Apart from using [NATS streaming](http://nats.io/doc/streaming/nats-streaming-intro/) or optimizing your consuming application, there are a few options available: scale, meter, or tune NATS to your environment.
__Scaling with queue subscribers__
This is ideal if you do not rely on message order. Ensure your NATS subscription belongs to a [queue group](http://nats.io/documentation/concepts/nats-queueing/), then scale as required by creating more instances of your service or application. This is a great approach for microservices - each instance of your microservice will receive a portion of the messages to process, and simply add more instances of your service to scale. No code changes, configuration changes, or downtime whatsoever.
__Create a subject namespace that can scale__
You can distribute work further through the subject namespace, with some forethought in design. This approach is useful if you need to preserve message order. The general idea is to publish to a deep subject namespace, and consume with wildcard subscriptions while giving yourself room to expand and distribute work in the future.
For a simple example, if you have a service that receives telemetry data from IoT devices located throughout a city, you can publish to a subject namespace like `Sensors.North`, `Sensors.South`, `Sensors.East` and `Sensors.West`. Initially, you'll subscribe to `Sensors.>` to process everything in one consumer. As your enterprise grows and data rates exceed what one consumer can handle, you can replace your single consumer with four consuming applications to subscribe to each subject representing a smaller segment of your data. Note that your publishing applications remain untouched.
__Meter the publisher__
A less favorable option may be to meter the publisher. There are several ways to do this varying from simply slowing down your publisher to a more complex approach periodically issuing a blocking request/reply to match subscriber rates.
__Tune NATS through configuration__
The NATS server can be tuned to determine how much data can be buffered before a consumer is considered slow, and some officially supported clients allow buffer sizes to be adjusted. Decreasing buffer sizes will let you identify slow consumers more quickly. Increasing buffer sizes is not typically recommended unless you are handling temporary bursts of data. Often, increasing buffer capacity will only *postpone* slow consumer problems.
### Server Configuration
The NATS server has a write deadline it uses to write to a connection. When this write deadline is exceeded, a client is considered to have a slow consumer. If you are encountering slow consumer errors in the server, you can increase the write deadline to buffer more data.
The `write_deadline` configuration option in the NATS server configuration file will tune this:
```ascii
write_deadline: 2s
```
Tuning this parameter is ideal when you have bursts of data to accommodate. **_Be sure you are not just postponing a slow consumer error._**
### Client Configuration
Most officially supported clients have an internal buffer of pending messages and will notify your application through an asynchronous error callback if a local subscription is not catching up. Receiving an error locally does not necessarily mean that the server will have identified a subscription as a slow consumer.
This buffer can be configured through setting the pending limits after a subscription has been created:
```go
if err := sub.SetPendingLimits(1024*500, 1024*5000); err != nil {
log.Fatalf("Unable to set pending limits: %v", err)
}
```
The default subscriber pending message limit is `65536`, and the default subscriber pending byte limit is `65536*1024`
If the client reaches this internal limit, it will drop messages and continue to process new messages. This is aligned with NATS at most once delivery. It is up to your application to detect the missing messages and recover from this condition.

0
nats_server/upgrading.md → nats_admin/upgrading_cluster.md
View File

106
nats_protocol/nats-client-dev.md Normal file
View File

@ -0,0 +1,106 @@
## NATS Client Development Guide
This guide provides you with considerations for developing NATS clients, including:
- CONNECT handling
- Authorization
- Verbose (acks)
- Pedantic mode
- Ping/pong interval
- Parsing the protocol
- Deciding on a parsing strategy
- Storing and dispatching subscription callbacks
- Implementing requests/response
- Error handling, disconnecting and reconnecting
- Cluster support
Probably the best way to learn about implementing a client is to look at one of the client's maintained by the Synadia team. These clients are generally full featured, so if you can use them, that is even better, but if you have to write a client these may go beyond your needs while still capturing many of the design considerations discussed here.
- [go](https://github.com/nats-io/go-nats)
- [node](https://github.com/nats-io/node-nats)
- [typescript](https://github.com/nats-io/ts-nats)
- [python2](https://github.com/nats-io/python-nats)
- [python asyncio](https://github.com/nats-io/asyncio-nats)
- [java](https://github.com/nats-io/java-nats)
- [c#](https://github.com/nats-io/csharp-nats)
- [ruby](https://github.com/nats-io/ruby-nats)
- [c](https://github.com/nats-io/cnats)
## Client connection options
Clients can connect in authenticated or unauthenticated mode, as well as verbose mode which enables acknowledgements. See the [protocol documentation](/documentation/internals/nats-protocol/#CONNECT) for details.
## Client authorization
By default clients can connect to the server in unauthenticated mode. You can configure the NATS server to require password authentication to connect.
For example, using the command line:
```
nats-server -DV -m 8222 -user foo -pass bar
```
The client must then authenticate to connect to the server. For example:
```
nats.Connect("nats://foo:bar@localhost:4222")
```
## Verbose mode
When 'verbose' is enabled (via the `CONNECT` message), the NATS server will return `+OK` to acknowledge receipt of a valid protocol message. The NATS server automatically runs in verbose mode. Most client implementations disable verbose mode (set it to `false` in the `CONNECT` message) for performance reasons.
## Pedantic mode
A client may also support 'pedantic' mode. Pedantic mode indicates to the server that strict protocol enforcement is required.
## Ping/pong interval
NATS implements auto-pruning. When a client connects to the server, the server expects that client to be active. Periodically, the NATS server pings each subscriber, expecting a reply. If there is no reply within the configurable time limit, the server disconnects the client.
## Parsing the protocol
NATS provides a text-based message format. The text-based [protocol](/documentation/internals/nats-protocol/) makes it easy to implement NATS clients. The key consideration is deciding on a parsing strategy.
The NATS server implements a [zero allocation byte parser](https://youtu.be/ylRKac5kSOk?t=10m46s) that is fast and efficient. Off the wire, a NATS message is simply a slice of bytes. Across the wire the message is transported as an immutable string over a TCP connection. It is up to the client to implement logic to parse the message.
The NATS message structure includes the Subject string, an optional Reply string, and an optional Data field that is a byte array. The type `Msg` is a structure used by Subscribers and PublishMsg().
```
type Msg struct {
Subject string
Reply string
Data []byte
Sub *Subscription
}
```
A NATS publisher publishes the data argument to the given subject. The data argument is left untouched and needs to be correctly interpreted on the receiver. How the client parses a NATS message depends on the programming language.
## Deciding on a parsing strategy
Generally, protocol parsing for a NATS client is a string operation. In Python, for example, string operations are faster than regex. The Go and Java clients also use string operations to parse the message. But, if you look at the Ruby client, regex is used to parse the protocol because in Ruby regex is faster that string operations.
In sum, there is no magic formula for parsing—it depends on the programming language. But, you need to take into consideration how you are going to parse the message when you write a client.
## Storing and dispatching subscription callbacks
When you make a subscription to the server, you need to store and dispatch callback handlers.
On the client side, you need a hash map for this data structure. The hash map will be storing the callback that maps the subscription ID to the subscription.
The key of the hash map is the subscription ID. The key is used to look up the callback in the hash map. When you process the NATS message off the wire, you pass the parameters subject, reply subject, and the payload to the callback handler, which does its work.
Thus, you must store the mapping of subscription ID to the callback. Inside the subscription you have the callback.
## Implementing request/response
When to use pub/sub vs. req/rep depends on your use case. Run the tutorials for each to understand the differences between each style of implementation.
## Error handling, disconnecting and reconnecting
Considerations for error handling primarily include handling client disconnections and implementing retry logic.
## Cluster support
The NATS client has reconnection logic. So, if you are implementing clustering, you need to implement reconnect callbacks a priori, meaning you cannot modify it during runtime. When you start it, you need to have that information already.

147
nats_protocol/nats-protocol-demo.md Normal file
View File

@ -0,0 +1,147 @@
## Protocol Demo
The virtues of the NATS protocol manifest quickly when you experience how easy it is to use NATS. Because the NATS protocol is text-based, you can use NATS across virtually any platform or language. In the following demo we use [Telnet](https://en.wikipedia.org/wiki/Telnet).
On the wire you can publish and subscribe using a simple [set of protocol commands](/documentation/internals/nats-protocol/).
## Instructions
**1. Open a terminal session.**
You'll use this terminal as the subscriber.
**2. Connect to NATS.**
```
telnet demo.nats.io 4222
```
Expected result:
```
$ telnet demo.nats.io 4222
Trying 107.170.221.32...
Connected to demo.nats.io.
Escape character is '^]'.
INFO {"server_id":"ad29ea9cbb16f2865c177bbd4db446ca","version":"0.6.8","go":"go1.5.1","host":"0.0.0.0","port":4222,"auth_required":false,"ssl_required":false,"max_payload":1048576}
```
**3. Run the subscriber.**
Subscribe to the wildcard subject `foo.*` with subject ID of `90`.
```
sub foo.* 90
```
Subscriber result: `+OK` indicating successful interest registration.
```
sub foo.* 90
+OK
```
**4. Open a second terminal window.**
You'll use this terminal for the publisher.
**5. Connect to NATS.**
```
telnet demo.nats.io 4222
```
Expected result:
```
$ telnet demo.nats.io 4222
Trying 107.170.221.32...
Connected to demo.nats.io.
Escape character is '^]'.
INFO {"server_id":"ad29ea9cbb16f2865c177bbd4db446ca","version":"0.6.8","go":"go1.5.1","host":"0.0.0.0","port":4222,"auth_required":false,"ssl_required":false,"max_payload":1048576}
```
**6. Publish a message.**
The message includes the command (`pub`), subject (`foo.bar`), and length of the payload (`5`). Press enter and provide the payload (`hello`), then press enter again.
```
pub foo.bar 5
hello
```
Publisher result: `+OK` indicating message publication.
```
pub foo.bar 5
hello
+OK
```
Subscriber result: `MSG` + subject name + subscription ID + message payload size + message payload `hello`.
```
sub foo.* 90
+OK
MSG foo.bar 90 5
hello
```
**7. Publish another message with reply subject.**
```
pub foo.bar optional.reply.subject 5
hello
+OK
```
Subscriber result: `MSG` indicating message receipt.
```
MSG foo.bar 90 optional.reply.subject 5
hello
```
**8. Unsubscribe from interest in the subject.**
You can use the `UNSUB` command to unsubscribe from a message.
Run the subscriber to unsubscribe:
```
unsub 90
```
Subscriber result: `+OK` indicating successful deregistration of interest.
```
unsub 90
+OK
```
**9. Reconnect to server and subscribe.**
```
telnet demo.nats.io 4222
```
```
sub foo.* 90
```
**10. Explore the ping/pong interval.**
If you leave your telnet session open for a few minutes, you may notice that your clients receives `ping` requests from the server. If your client is not active, or does not respond to the server pings within the ping/pong interval, the server disconnects the client. The error message is `-ERR 'Stale Connection'`.
You can send a `ping` request to the serve and receive a `PONG` reply. For example:
```
$ telnet demo.nats.io 4222
Trying 107.170.221.32...
Connected to demo.nats.io.
Escape character is '^]'.
INFO {"server_id":"ad29ea9cbb16f2865c177bbd4db446ca","version":"0.6.8","go":"go1.5.1","host":"0.0.0.0","port":4222,"auth_required":false,"ssl_required":false,"max_payload":1048576}
ping
PONG
```

315
nats_protocol/nats-protocol.md Normal file
View File

@ -0,0 +1,315 @@
## Client Protocol
The wire protocol used to communicate between the NATS server and clients is a simple, text-based publish/subscribe style protocol. Clients connect to and communicate with `nats-server` (the NATS server) through a regular TCP/IP socket using a small set of protocol operations that are terminated by a new line.
Unlike traditional messaging systems that use a binary message format that require an API to consume, the text-based NATS protocol makes it easy to implement clients in a wide variety of programming and scripting languages. In fact, refer to the topic [NATS Protocol Demo](/documentation/internals/nats-protocol-demo/) to play with the NATS protocol for yourself using telnet.
The NATS server implements a [zero allocation byte parser](https://youtu.be/ylRKac5kSOk?t=10m46s) that is fast and efficient.
## Protocol conventions
**Control line w/Optional Content**: Each interaction between the client and server consists of a control, or protocol, line of text followed, optionally by message content. Most of the protocol messages don't require content, only `PUB` and `MSG` include payloads.
**Field Delimiters**: The fields of NATS protocol messages are delimited by whitespace characters '` `' (space) or `\t` (tab). Multiple whitespace characters will be treated as a single field delimiter.
**Newlines**: NATS uses `CR` followed by `LF` (`CR+LF`, `\r\n`, `0x0D0A`) to terminate protocol messages. This newline sequence is also used to mark the end of the message payload in a `PUB` or `MSG` protocol message.
**Subject names**: Subject names, including reply subject (INBOX) names, are case-sensitive and must be non-empty alphanumeric strings with no embedded whitespace, and optionally token-delimited using the dot character (`.`), e.g.:
`FOO`, `BAR`, `foo.bar`, `foo.BAR`, `FOO.BAR` and `FOO.BAR.BAZ` are all valid subject names
`FOO. BAR`, `foo. .bar` and`foo..bar` are *not* valid subject names
**Character Encoding**: Subject names should be ascii characters for maximum interoperability. Due to language constraints and performance, some clients may support UTF-8 subject names, as may the server. No guarantees of non-ASCII support are provided.
**Wildcards**: NATS supports the use of wildcards in subject subscriptions.
- The asterisk character (`*`) matches a single token at any level of the subject.
- The greater than symbol (`>`), also known as the _full wildcard_, matches one or more tokens at the tail of a subject, and must be the last token. The wildcarded subject `foo.>` will match `foo.bar` or `foo.bar.baz.1`, but not `foo`.
- Wildcards must be a separate token (`foo.*.baz` or `foo.>` are syntactically valid; `foo*.bar`, `f*o.b*r` and `foo>` are not)
For example, the wildcard subscriptions `foo.*.quux` and `foo.>` both match `foo.bar.quux`, but only the latter matches `foo.bar.baz`. With the full wildcard,
it is also possible to express interest in every subject that may exist in NATS: `sub > 1`, limited of course by authorization settings.
## Protocol messages
The following table briefly describes the NATS protocol messages. NATS protocol operation names are case insensitive, thus `SUB foo 1\r\n` and `sub foo 1\r\n` are equivalent.
Click the name to see more detailed information, including syntax:
| OP Name | Sent By | Description
| -------------------- |:-------------- |:--------------------------------------------
| [`INFO`](#INFO) | Server | Sent to client after initial TCP/IP connection
| [`CONNECT`](#CONNECT)| Client | Sent to server to specify connection information
| [`PUB`](#PUB) | Client | Publish a message to a subject, with optional reply subject
| [`SUB`](#SUB) | Client | Subscribe to a subject (or subject wildcard)
| [`UNSUB`](#UNSUB) | Client | Unsubscribe (or auto-unsubscribe) from subject
| [`MSG`](#MSG) | Server | Delivers a message payload to a subscriber
| [`PING`](#PINGPONG) | Both | PING keep-alive message
| [`PONG`](#PINGPONG) | Both | PONG keep-alive response
| [`+OK`](#OKERR) | Server | Acknowledges well-formed protocol message in `verbose` mode
| [`-ERR`](#OKERR) | Server | Indicates a protocol error. May cause client disconnect.
The following sections explain each protocol message.
## <a name="INFO"></a>INFO
#### Description
As soon as the server accepts a connection from the client, it will send information about itself and the configuration and security requirements that are necessary for the client to successfully authenticate with the server and exchange messages.
When using the updated client protocol (see [`CONNECT`](#CONNECT) below), `INFO` messages can be sent anytime by the server. This means clients with that protocol level need to be able to asynchronously handle `INFO` messages.
#### Syntax
`INFO {["option_name":option_value],...}`
The valid options are as follows:
- `server_id`: The unique identifier of the NATS server
- `version`: The version of the NATS server
- `go`: The version of golang the NATS server was built with
- `host`: The IP address used to start the NATS server, by default this will be `0.0.0.0` and can be configured with `-client_advertise host:port`
- `port`: The port number the NATS server is configured to listen on
- `max_payload`: Maximum payload size, in bytes, that the server will accept from the client.
- `proto`: An integer indicating the protocol version of the server. The server version 1.2.0 sets this to `1` to indicate that it supports the "Echo" feature.
- `client_id`: An optional unsigned integer (64 bits) representing the internal client identifier in the server. This can be used to filter client connections in monitoring, correlate with error logs, etc...
- `auth_required`: If this is set, then the client should try to authenticate upon connect.
- `tls_required`: If this is set, then the client must perform the TLS/1.2 handshake. Note, this used to be `ssl_required` and has been updated along with the protocol from SSL to TLS.
- `tls_verify`: If this is set, the client must provide a valid certificate during the TLS handshake.
- `connect_urls` : An optional list of server urls that a client can connect to.
##### connect_urls
The `connect_urls` field is a list of urls the server may send when a client first connects, and when there are changes to server cluster topology. This field is considered optional, and may be omitted based on server configuration and client protocol level.
When a NATS server cluster expands, an `INFO` message is sent to the client with an updated `connect_urls` list. This cloud-friendly feature asynchronously notifies a client of known servers, allowing it to connect to servers not originally configured.
The `connect_urls` will contain a list of strings with an IP and port, looking like this: ```"connect_urls":["10.0.0.184:4333","192.168.129.1:4333","192.168.192.1:4333"]```
#### Example
Below you can see a sample connection string from a telnet connection to the `demo.nats.io` site.
```sh
% telnet demo.nats.io 4222
Trying 107.170.221.32...
Connected to demo.nats.io.
Escape character is '^]'.
INFO {"server_id":"Zk0GQ3JBSrg3oyxCRRlE09","version":"1.2.0","proto":1,"go":"go1.10.3","host":"0.0.0.0","port":4222,"max_payload":1048576,"client_id":2392}
```
## <a name="CONNECT"></a>CONNECT
#### Description
The `CONNECT` message is the client version of the `INFO` message. Once the client has established a TCP/IP socket connection with the NATS server, and an `INFO` message has been received from the server, the client may send a `CONNECT` message to the NATS server to provide more information about the current connection as well as security information.
#### Syntax
`CONNECT {["option_name":option_value],...}`
The valid options are as follows:
* `verbose`: Turns on [`+OK`](#OKERR) protocol acknowledgements.
* `pedantic`: Turns on additional strict format checking, e.g. for properly formed subjects
* `tls_required`: Indicates whether the client requires an SSL connection.
* `auth_token`: Client authorization token (if `auth_required` is set)
* `user`: Connection username (if `auth_required` is set)
* `pass`: Connection password (if `auth_required` is set)
* `name`: Optional client name
* `lang`: The implementation language of the client.
* `version`: The version of the client.
* `protocol`: *optional int*. Sending `0` (or absent) indicates client supports original protocol. Sending `1` indicates that the client supports dynamic reconfiguration of cluster topology changes by asynchronously receiving `INFO` messages with known servers it can reconnect to.
* `echo`: Optional boolean. If set to `true`, the server (version 1.2.0+) will not send originating messages from this connection to its own subscriptions. Clients should set this to `true` only for server supporting this feature, which is when `proto` in the `INFO` protocol is set to at least `1`.
#### Example
Here is an example from the default string of the Go client:
```
[CONNECT {"verbose":false,"pedantic":false,"tls_required":false,"name":"","lang":"go","version":"1.2.2","protocol":1}]\r\n
```
Most clients set `verbose` to `false` by default. This means that the server should not confirm each message it receives on this connection with a `+OK` back to the client.
## <a name="PUB"></a>PUB
#### Description
The `PUB` message publishes the message payload to the given subject name, optionally supplying a reply subject. If a reply subject is supplied, it will be delivered to eligible subscribers along with the supplied payload. Note that the payload itself is optional. To omit the payload, set the payload size to 0, but the second CRLF is still required.
#### Syntax
`PUB <subject> [reply-to] <#bytes>\r\n[payload]\r\n`
where:
- `subject`: The destination subject to publish to
- `reply-to`: The optional reply inbox subject that subscribers can use to send a response back to the publisher/requestor
- `#bytes`: The payload size in bytes
- `payload`: The message payload data
#### Example
To publish the ASCII string message payload "Hello NATS!" to subject FOO:
`PUB FOO 11\r\nHello NATS!\r\n`
To publish a request message "Knock Knock" to subject FRONT.DOOR with reply subject INBOX.22:
`PUB FRONT.DOOR INBOX.22 11\r\nKnock Knock\r\n`
To publish an empty message to subject NOTIFY:
`PUB NOTIFY 0\r\n\r\n`
## <a name="SUB"></a>SUB
#### Description
`SUB` initiates a subscription to a subject, optionally joining a distributed queue group.
#### Syntax
`SUB <subject> [queue group] <sid>\r\n`
where:
- `subject`: The subject name to subscribe to
- `queue group`: If specified, the subscriber will join this queue group
- `sid`: A unique alphanumeric subscription ID, generated by the client
#### Example
To subscribe to the subject `FOO` with the connection-unique subscription identifier (sid) `1`:
`SUB FOO 1\r\n`
To subscribe the current connection to the subject `BAR` as part of distribution queue group `G1` with sid `44`:
`SUB BAR G1 44\r\n`
## <a name="UNSUB"></a>UNSUB
#### Description
`UNSUB` unsubcribes the connection from the specified subject, or auto-unsubscribes after the specified number of messages has been received.
#### Syntax
`UNSUB <sid> [max_msgs]`
where:
* `sid`: The unique alphanumeric subscription ID of the subject to unsubscribe from
* `max_msgs`: An optional number of messages to wait for before automatically unsubscribing
#### Example
The following examples concern subject `FOO` which has been assigned sid `1`. To unsubscribe from `FOO`:
`UNSUB 1\r\n`
To auto-unsubscribe from `FOO` after 5 messages have been received:
`UNSUB 1 5\r\n`
## <a name="MSG"></a>MSG
#### Description
The `MSG` protocol message is used to deliver an application message to the client.
#### Syntax
`MSG <subject> <sid> [reply-to] <#bytes>\r\n[payload]\r\n`
where:
* `subject`: Subject name this message was received on
* `sid`: The unique alphanumeric subscription ID of the subject
* `reply-to`: The inbox subject on which the publisher is listening for responses
* `#bytes`: Size of the payload in bytes
* `payload`: The message payload data
#### Example
The following message delivers an application message from subject `FOO.BAR`:
`MSG FOO.BAR 9 11\r\nHello World\r\n`
To deliver the same message along with a reply inbox:
`MSG FOO.BAR 9 INBOX.34 11\r\nHello World\r\n`
## <a name="PINGPONG"></a>PING/PONG
#### Description
`PING` and `PONG` implement a simple keep-alive mechanism between client and server. Once a client establishes a connection to the NATS server, the server will continuously send `PING` messages to the client at a configurable interval. If the client fails to respond with a `PONG` message within the configured response interval, the server will terminate its connection. If your connection stays idle for too long, it is cut off.
If the server sends a ping request, you can reply with a pong message to notify the server that you are still interested. You can also ping the server and will receive a pong reply. The ping/pong interval is configurable.
The server uses normal traffic as a ping/pong proxy, so a client that has messages flowing may not receive a ping from the server.
#### Syntax
`PING\r\n`
`PONG\r\n`
#### Example
The following example shows the demo server pinging the client and finally shutting it down.
```
telnet demo.nats.io 4222
Trying 107.170.221.32...
Connected to demo.nats.io.
Escape character is '^]'.
INFO {"server_id":"Zk0GQ3JBSrg3oyxCRRlE09","version":"1.2.0","proto":1,"go":"go1.10.3","host":"0.0.0.0","port":4222,"max_payload":1048576,"client_id":2392}
PING
PING
-ERR 'Stale Connection'
Connection closed by foreign host.
```
## <a name="OKERR"></a>+OK/ERR
#### Description
When the `verbose` connection option is set to `true` (the default value), the server acknowledges each well-formed protocol message from the client with a `+OK` message. Most NATS clients set the `verbose` option to `false` using the [CONNECT](#CONNECT) message
The `-ERR` message is used by the server indicate a protocol, authorization, or other runtime connection error to the client. Most of these errors result in the server closing the connection.
Handling of these errors usually has to be done asynchronously.
#### Syntax
`+OK`
`-ERR <error message>`
Some protocol errors result in the server closing the connection. Upon recieving these errors, the connection is no longer valid and the client should clean up relevant resources. These errors include:
- `-ERR 'Unknown Protocol Operation'`: Unknown protocol error
- `-ERR 'Attempted To Connect To Route Port'`: Client attempted to connect to a route port instead of the client port
- `-ERR 'Authorization Violation'`: Client failed to authenticate to the server with credentials specified in the [CONNECT](#CONNECT) message
- `-ERR 'Authorization Timeout'`: Client took too long to authenticate to the server after establishing a connection (default 1 second)
- `-ERR 'Invalid Client Protocol'`: Client specified an invalid protocol version in the [CONNECT](#CONNECT) message
- `-ERR 'Maximum Control Line Exceeded'`: Message destination subject and reply subject length exceeded the maximum control line value specified by the `max_control_line` server option. The default is 1024 bytes.
- `-ERR 'Parser Error'`: Cannot parse the protocol message sent by the client
- `-ERR 'Secure Connection - TLS Required'`: The server requires TLS and the client does not have TLS enabled.
- `-ERR 'Stale Connection'`: The server hasn't received a message from the client, including a `PONG` in too long.
- `-ERR 'Maximum Connections Exceeded`': This error is sent by the server when creating a new connection and the server has exceeded the maximum number of connections specified by the `max_connections` server option. The default is 64k.
- `-ERR 'Slow Consumer'`: The server pending data size for the connection has reached the maximum size (default 10MB).
- `-ERR 'Maximum Payload Violation'`: Client attempted to publish a message with a payload size that exceeds the `max_payload` size configured on the server. This value is supplied to the client upon connection in the initial [`INFO`](#INFO) message. The client is expected to do proper accounting of byte size to be sent to the server in order to handle this error synchronously.
Protocol error messages where the connection remains open are listed below. The client should not close the connection in these cases.
- `-ERR 'Invalid Subject'`: Client sent a malformed subject (e.g. `sub foo. 90`)
- `-ERR 'Permissions Violation for Subscription to <subject>'`: The user specified in the [CONNECT](#CONNECT) message does not have permission to subscribe to the subject.
- `-ERR 'Permissions Violation for Publish to <subject>'`: The user specified in the [CONNECT](#CONNECT) message does not have permissions to publish to the subject.

261
nats_protocol/nats-server-protocol.md Normal file
View File

@ -0,0 +1,261 @@
## NATS Cluster Protocol
The NATS server clustering protocol describes the messages passed between NATS servers within a [cluster](/nats_server/clustering.md) to share subscription state, forward messages, and share cluster topology. It is a simple, text-based publish/subscribe style protocol. Servers communicate with each other through a regular TCP/IP socket using a small set of protocol operations that are terminated by newline.
The NATS server implements a [zero allocation byte parser](https://youtu.be/ylRKac5kSOk?t=10m46s) that is fast and efficient.
The NATS cluster protocol is very similar to that of the NATS client protocol. In the context of a cluster, it can be helpful to visualize a server being a proxy operating on behalf of its connected clients, subscribing, unsubscribing, sending and receiving messages.
## NATS Cluster protocol conventions
**Subject names and wildcards**: The NATS cluster protocol has the same features and restrictions as the client with respect to subject names and wildcards, with one addition - a server can publish subjects prefixed with `_SYS.`, which is reserved for messages that are internally generated by a server.
**Field Delimiters**: The fields of NATS protocol messages are delimited by whitespace characters '` `' (space) or `\t` (tab).
Multiple whitespace characters will be treated as a single field delimiter.
**Newlines**: Like other text-based protocols, NATS uses `CR` followed by `LF` (`CR+LF`, `\r\n`, `0x0D0A`) to terminate protocol messages. This newline sequence is also used to mark the beginning of the actual message payload in a `PUB` or `MSG` protocol message.
## NATS Cluster protocol messages
The following table briefly describes the NATS cluster protocol messages.
As in the client protocol, the NATS protocol operation names are case insensitive, thus `SUB foo 1\r\n` and `sub foo 1\r\n` are equivalent.
Click the name to see more detailed information, including syntax:
| OP Name | Sent By | Description
| -------------------- |:-----------------|:--------------------------------------------
| [`INFO`](#INFO) | All Servers | Sent after initial TCP/IP connection and to update cluster knowledge
| [`CONNECT`](#CONNECT)| All Servers | Sent to establish a route
| [`PUB`](#PUB) | Sending Server | Sent when a message is published by a client.
| [`SUB`](#SUB) | All Servers | Subscribes to a subject on behalf of interested clients.
| [`UNSUB`](#UNSUB) | All Servers | Unsubscribe (or auto-unsubscribe) from subject.
| [`MSG`](#MSG) | Receiving Server | Delivers a message payload to a subscriber on the local server.
| [`PING`](#PINGPONG) | All Servers | PING keep-alive message
| [`PONG`](#PINGPONG) | All Servers | PONG keep-alive response
| [`+OK/-ERR`](#OKERR) | All Servers | Indicates a protocol error. Will cause the server to disconnect.
The following sections explain each protocol message.
## <a name="INFO"></a>INFO
#### Description
As soon as the server accepts a connection from another server, it will send information about itself and the configuration and security requirements that are necessary for the other server to successfully authenticate with the server and exchange messages.
The connecting server also sends an `INFO` message. The accepting server will add an `ip` field containing the address and port of the connecting server, and forward the new server's `INFO` message to all servers it is routed to.
Any servers in a cluster receiving an `INFO` message with an `ip` field will attempt to connect to the server at that address, unless already connected. This propagation of `INFO` messages on behalf of a connecting server provides automatic discovery of new servers joining a cluster.
#### Syntax
`INFO {["option_name":option_value],...}`
The valid options are as follows:
* `server_id`: The unique identifier of the NATS server
* `version`: The version of the NATS server
* `go`: The version of golang the NATS server was built with
* `host`: The host specified in the cluster parameter/options
* `port`: The port number specified in the cluster parameter/options
* `auth_required`: If this is set, then the server should try to authenticate upon connect.
* `tls_required`: If this is set, then the server must authenticate using TLS.
* `max_payload`: Maximum payload size that the server will accept.
* `connect_urls` : A list of server urls that a client can connect to.
* `ip`: Optional route connection address of a server, `nats-route://<hostname>:<port>`
#### Example
Below is an example of an `INFO` string received by a NATS server, with the `ip` field.
`INFO {"server_id":"KP19vTlB417XElnv8kKaC5","version":"0.9.4","go":"","host":"localhost","port":5222,"auth_required":false,"tls_required":false,"tls_verify":false,"max_payload":1048576,"ip":"nats-route://127.0.0.1:5222/","connect_urls":["localhost:4222"]}`
## <a name="CONNECT"></a>CONNECT
#### Description
The `CONNECT` message is analogous to the `INFO` message. Once the NATS server has established a TCP/IP socket connection with another server, and an `INFO` message has been received, the server will send a `CONNECT` message to provide more information about the current connection as well as security information.
#### Syntax
`CONNECT {["option_name":option_value],...}`
The valid options are as follows:
* `verbose`: Turns on [`+OK`](#OKERR) protocol acknowledgements.
* `pedantic`: Turns on additional strict format checking, e.g. for properly formed subjects
* `tls_required`: Indicates whether the server requires an SSL connection.
* `auth_token`: Authorization token
* `user`: Connection username (if `auth_required` is set)
* `pass`: Connection password (if `auth_required` is set)
* `name`: Generated Server Name
* `lang`: The implementation language of the server (go).
* `version`: The version of the server.
#### Example
Here is an example from the default string from a server.
`CONNECT {"verbose":false,"pedantic":false,"tls_required":false,"name":"wt0vffeQyoDGMVBC2aKX0b"}\r\n`
Servers should set `verbose` to `false` by default. This means that other routed servers will not be sending an `+OK` payload back after the remote server ingested the message.
## <a name="PUB"></a>PUB
#### Description
The `PUB` message publishes the message payload to the given subject name, optionally supplying a reply subject, to another server. If a reply subject is supplied, it will be delivered to eligible subscribers along with the supplied payload. Note that the payload itself is optional. To omit the payload, set the payload size to 0.
#### Syntax
`PUB <subject> [reply-to] <#bytes>\r\n[payload]\r\n`
where:
* `subject`: The destination subject to publish to
* `reply-to`: The reply inbox subject that subscribers can use to send a response back to the publisher/requestor
* `#bytes`: The payload size in bytes
* `payload`: The message payload data
#### Example
To publish the string message payload "Hello NATS!" to subject FOO:
`PUB FOO 11\r\nHello NATS!\r\n`
To publish a request message "Knock Knock" to subject FRONT.DOOR with reply subject INBOX.22:
`PUB FRONT.DOOR INBOX.22 11\r\nKnock Knock\r\n`
To publish an empty message to subject NOTIFY:
`PUB NOTIFY 0\r\n\r\n`
## <a name="SUB"></a>SUB
#### Description
`SUB` initiates a subscription to a subject, optionally joining a distributed queue group.
#### Syntax
**Basic Subscription**: `SUB <subject> RSID:<cid>:<sid>\r\n`
**Queue Subscription**: `SUB <subject> <queue> QRSID:<cid>:<sid>\r\n`
where:
* `subject`: The subject name to subscribe to
* `queue`: If specified, the subscriber will join this queue group
* `cid`: A 64bit unsigned integer representing the client connection
* `sid`: A unique alphanumeric subscription ID representing the server's subscription
#### Example
To subscribe to the subject `FOO` with the local unique subject identifier of `1`, and the connection-unique subject identifier (sid) `1`:
`SUB FOO RSID:1:1\r\n`
To subscribe the current connection to the subject `BAR` as part of distribution queue group `G1` with local sid of `14`, and a client sid `44`:
`SUB BAR G1 QRSID:14:44\r\n`
## <a name="UNSUB"></a>UNSUB
#### Description
`UNSUB` unsubcribes the connection from the specified subject, or auto-unsubscribes after the specified number of messages has been received. It is sent by a server when one of it's clients unsubscribes.
#### Syntax
**Basic Subscription**: `UNSUB <sid> RSID:<cid>:<sid> [max_msgs]\r\n`
**Queue Subscription**: `UNSUB <sid> QRSID:<cid>:<sid> [max_msgs]\r\n`
where:
* `sid`: The unique alphanumeric subscription ID of the subject to unsubscribe from
* `max_msgs`: Number of messages to wait for before automatically unsubscribing
* `cid`: A 64bit unsigned integer representing the client connection
* `sid`: A unique alphanumeric subscription ID representing the server's subscription
#### Example
The following examples concern subject `FOO` which has been assigned an internal subscriber id of `5`, and a client sid of `1`. To unsubscribe from `FOO`:
`UNSUB RSID:5:1\r\n`
To auto-unsubscribe from `FOO` after 5 messages have been received:
`UNSUB RSID:1:1 5\r\n`
## <a name="MSG"></a>MSG
#### Description
The `MSG` protocol message delivers a message from another server.
#### Syntax
`MSG <subject> <sid> [reply-to] <#bytes>\r\n[payload]\r\n`
where:
* `subject`: Subject name this message was received on
* `sid`: The unique alphanumeric subscription ID of the subject
* `reply-to`: The inbox subject on which the publisher is listening for responses
* `#bytes`: Size of the payload in bytes
* `payload`: The message payload data
#### Example
The following message delivers a message from subject `FOO.BAR`:
`MSG FOO.BAR 9 11\r\nHello World\r\n`
Deliver the same message along with a reply inbox:
`MSG FOO.BAR 9 INBOX.34 11\r\nHello World\r\n`
## <a name="PINGPONG"></a>PING/PONG
#### Description
`PING` and `PONG` implement a simple keep-alive mechanism between servers. Once two servers establish a connection with each other, the NATS server will continuously send `PING` messages to other servers at a configurable interval. If another server fails to respond with a `PONG` message within the configured response interval, the server will terminate its connection. If your connection stays idle for too long, it is cut off.
If the another server sends a ping request, a server will reply with a pong message to notify the other server that it is still present.
#### Syntax
`PING\r\n`
`PONG\r\n`
## <a name="OKERR"></a>+OK/ERR
#### Description
When the `verbose` connection option is set to `true` (the default value), the server acknowledges each well-formed protocol message with a `+OK` message. NATS servers set the `verbose` option to `false` using the [CONNECT](#CONNECT) message
The `-ERR` message is used by the server indicate a protocol, authorization, or other runtime connection error to another server. Most of these errors result in the server closing the connection.
Handling of these errors usually has to be done asynchronously.
#### Syntax
`+OK`
`-ERR <error message>`
Protocol error messages which close the connection:
- `-ERR 'Unknown Protocol Operation'`: Unknown protocol error
- `-ERR 'Authorization Violation'`: Server failed to authenticate to another server with credentials specified in the [CONNECT](#CONNECT) message.
- `-ERR 'Authorization Timeout'`: Server took too long to authenticate to another server after establishing a connection (default 1 second)
- `-ERR 'Parser Error'`: Cannot parse the protocol message sent by another server
- `-ERR 'Stale Connection'`: PING/PONG interval expired.
- `-ERR 'Slow Consumer'`: The other server's pending data size for the route connection has been exceeded. The default limit is time based, where data cannot be flushed within a two second write deadline.
- `-ERR 'Maximum Payload Exceeded'`: Server attempted to publish a message with a payload size that exceeds the `max_payload` size configured on another server.
Protocol error messages which do not close the connection:
- `-ERR 'Invalid Subject'`: Server sent a malformed subject (e.g. `sub foo. 90`)

2
nats_server/flags.md
View File

@ -23,7 +23,7 @@ The configuration flags revolve around:
| `-m`, `--http_port` | HTTP port for monitoring dashboard (exclusive of `--https_port`). |
| `-ms`, `--https_port` | HTTPS port monitoring for monitoring dashboard (exclusive of `--http_port`). |
| `-c`, `--config` | Path to NATS server configuration file. |
| `-sl`, `--signal` | Send a signal to nats-server process. See [process signaling](signals.md). |
| `-sl`, `--signal` | Send a signal to nats-server process. See [process signaling](/nats_admin/signals.md). |
| `--client_advertise` | Client HostPort to advertise to other servers. |
| `-t` | Test configuration and exit |

2
nats_server/running.md
View File

@ -32,3 +32,5 @@ If you are running your NATS server in a docker container:
[1] 2019/05/13 14:55:11.982492 [INF] Listening for route connections on 0.0.0.0:6222
...
```
More information on [containerized NATS is available here](/nats_docker/README.md).

2
nats_server/windows_srv.md
View File

@ -25,3 +25,5 @@ The above commands will default to controlling the `nats-server` service. If the
```batch
nats-server.exe -sl stop=<service name>
```
For a complete list of signals, see [process signaling](/nats_admin/signals.md).

150
nats_tools/natsbench.md Normal file
View File

@ -0,0 +1,150 @@
## Benchmarking
NATS is fast and lightweight and places a priority on performance. NATS provides tools for measuring performance. In this tutorial you learn how to benchmark and tune NATS on your systems and environment.
#### Prerequisites
- [Set up your Go environment](https://golang.org/doc/install)
- [Install the NATS server](/nats_server/installation.md)
#### Start the NATS server with monitoring enabled
```sh
% nats-server -m 8222
```
Verify that the NATS server starts successfully, as well as the HTTP monitor:
```sh
[18541] 2016/10/31 13:26:32.037819 [INF] Starting nats-server version 0.9.4
[18541] 2016/10/31 13:26:32.037912 [INF] Starting http monitor on 0.0.0.0:8222
[18541] 2016/10/31 13:26:32.037997 [INF] Listening for client connections on 0.0.0.0:4222
[18541] 2016/10/31 13:26:32.038020 [INF] Server is ready
```
#### Installing and running the benchmark utility
The NATS benchmark can be installed and run via Go. Ensure your golang environment is setup.
There are two approaches; you can either install the `nats-bench` utility in the directory specified in your `GOBIN` environment variable:
```sh
% go install $GOPATH/src/github.com/nats-io/go-nats/examples/nats-bench.go
```
... or you can simply run it via `go run`:
```sh
% go run $GOPATH/src/github.com/nats-io/go-nats/examples/nats-bench.go
```
*On windows use the % environment variable syntax, replacing `$GOPATH` with `%GOPATH%`.*
For the purpose of this tutorial, we'll assume that you chose the first option, and that you've added the `GOBIN` environment variable to your `PATH`.
The `nats-bench` utility is straightforward to use. The options are as follows:
```sh
% nats-bench -h
Usage: nats-bench [-s server (nats://localhost:4222)] [--tls] [-np NUM_PUBLISHERS] [-ns NUM_SUBSCRIBERS] [-n NUM_MSGS] [-ms MESSAGE_SIZE] [-csv csvfile] <subject>
```
The options are self-explanatory. Each publisher or subscriber runs in its own go routine with its own NATS connection.
#### Run a publisher throughput test
Let's run a test to see how fast a single publisher can publish one million 16 byte messages to the NATS server.
```sh
% nats-bench -np 1 -n 100000 -ms 16 foo
```
The output tells you the number of messages and the number of payload bytes that the client was able to publish per second:
```sh
Starting benchmark [msgs=100000, msgsize=16, pubs=1, subs=0]
Pub stats: 7,055,644 msgs/sec ~ 107.66 MB/sec
```
Now increase the number of messages published:
```sh
% nats-bench -np 1 -n 10000000 -ms 16 foo
Starting benchmark [msgs=10000000, msgsize=16, pubs=1, subs=0]
Pub stats: 7,671,570 msgs/sec ~ 117.06 MB/sec
```
#### Run a publish/subscribe throughput test
When using both publishers and subscribers, `nats-bench` reports aggregate, as well as individual publish and subscribe throughput performance.
Let's look at throughput for a single publisher with a single subscriber:
```sh
% nats-bench -np 1 -ns 1 -n 100000 -ms 16 foo
```
Note that the output shows the aggregate throughput as well as the individual publisher and subscriber performance:
```sh
Starting benchmark [msgs=100000, msgsize=16, pubs=1, subs=1]
NATS Pub/Sub stats: 2,009,230 msgs/sec ~ 30.66 MB/sec
Pub stats: 1,076,537 msgs/sec ~ 16.43 MB/sec
Sub stats: 1,004,615 msgs/sec ~ 15.33 MB/sec
```
#### Run a 1:N throughput test
When specifying multiple publishers, or multiple subscribers, `nats-bench` will also report statistics for each publisher and subscriber individually, along with min/max/avg and standard deviation.
Let's increase both the number of messages, and the number of subscribers.:
```sh
% nats-bench -np 1 -ns 5 -n 10000000 -ms 16 foo
```
Output:
```sh
Starting benchmark [msgs=10000000, msgsize=16, pubs=1, subs=5]
NATS Pub/Sub stats: 5,730,851 msgs/sec ~ 87.45 MB/sec
Pub stats: 955,279 msgs/sec ~ 14.58 MB/sec
Sub stats: 4,775,709 msgs/sec ~ 72.87 MB/sec
[1] 955,157 msgs/sec ~ 14.57 MB/sec (10000000 msgs)
[2] 955,150 msgs/sec ~ 14.57 MB/sec (10000000 msgs)
[3] 955,157 msgs/sec ~ 14.57 MB/sec (10000000 msgs)
[4] 955,156 msgs/sec ~ 14.57 MB/sec (10000000 msgs)
[5] 955,153 msgs/sec ~ 14.57 MB/sec (10000000 msgs)
min 955,150 | avg 955,154 | max 955,157 | stddev 2 msgs
```
#### Run a N:M throughput test
When more than 1 publisher is specified, `nats-bench` evenly distributes the total number of messages (`-n`) across the number of publishers (`-np`).
Now let's increase the number of publishers and examine the output:
```sh
% nats-bench -np 5 -ns 5 -n 10000000 -ms 16 foo
```
The output:
```sh
Starting benchmark [msgs=10000000, msgsize=16, pubs=5, subs=5]
NATS Pub/Sub stats: 6,716,465 msgs/sec ~ 102.49 MB/sec
Pub stats: 1,119,653 msgs/sec ~ 17.08 MB/sec
[1] 226,395 msgs/sec ~ 3.45 MB/sec (2000000 msgs)
[2] 225,955 msgs/sec ~ 3.45 MB/sec (2000000 msgs)
[3] 225,889 msgs/sec ~ 3.45 MB/sec (2000000 msgs)
[4] 224,552 msgs/sec ~ 3.43 MB/sec (2000000 msgs)
[5] 223,933 msgs/sec ~ 3.42 MB/sec (2000000 msgs)
min 223,933 | avg 225,344 | max 226,395 | stddev 937 msgs
Sub stats: 5,597,054 msgs/sec ~ 85.40 MB/sec
[1] 1,119,461 msgs/sec ~ 17.08 MB/sec (10000000 msgs)
[2] 1,119,466 msgs/sec ~ 17.08 MB/sec (10000000 msgs)
[3] 1,119,444 msgs/sec ~ 17.08 MB/sec (10000000 msgs)
[4] 1,119,444 msgs/sec ~ 17.08 MB/sec (10000000 msgs)
[5] 1,119,430 msgs/sec ~ 17.08 MB/sec (10000000 msgs)
min 1,119,430 | avg 1,119,449 | max 1,119,466 | stddev 12 msgs
```