nats.docs/nats_protocol/nats-server-protocol.md

NATS Cluster Protocol

The NATS server clustering protocol describes the messages passed between NATS servers within a cluster to share subscription state, forward messages, and share cluster topology. It is a simple, text-based publish/subscribe style protocol. Servers communicate with each other through a regular TCP/IP socket using a small set of protocol operations that are terminated by newline.

The NATS server implements a zero allocation byte parser that is fast and efficient.

The NATS cluster protocol is very similar to that of the NATS client protocol. In the context of a cluster, it can be helpful to visualize a server being a proxy operating on behalf of its connected clients, subscribing, unsubscribing, sending and receiving messages.

NATS Cluster protocol conventions

Subject names and wildcards: The NATS cluster protocol has the same features and restrictions as the client with respect to subject names and wildcards, with one addition - a server can publish subjects prefixed with _SYS., which is reserved for messages that are internally generated by a server.

Field Delimiters: The fields of NATS protocol messages are delimited by whitespace characters ' ' (space) or \t (tab). Multiple whitespace characters will be treated as a single field delimiter.

Newlines: Like other text-based protocols, NATS uses CR followed by LF (CR+LF, \r\n, 0x0D0A) to terminate protocol messages. This newline sequence is also used to mark the beginning of the actual message payload in a PUB or MSG protocol message.

NATS Cluster protocol messages

The following table briefly describes the NATS cluster protocol messages. As in the client protocol, the NATS protocol operation names are case insensitive, thus SUB foo 1\r\n and sub foo 1\r\n are equivalent.

Click the name to see more detailed information, including syntax:

OP Name	Sent By	Description
INFO	All Servers	Sent after initial TCP/IP connection and to update cluster knowledge
CONNECT	All Servers	Sent to establish a route
PUB	Sending Server	Sent when a message is published by a client.
SUB	All Servers	Subscribes to a subject on behalf of interested clients.
UNSUB	All Servers	Unsubscribe (or auto-unsubscribe) from subject.
MSG	Receiving Server	Delivers a message payload to a subscriber on the local server.
PING	All Servers	PING keep-alive message
PONG	All Servers	PONG keep-alive response
+OK/-ERR	All Servers	Indicates a protocol error. Will cause the server to disconnect.

The following sections explain each protocol message.

INFO

Description

As soon as the server accepts a connection from another server, it will send information about itself and the configuration and security requirements that are necessary for the other server to successfully authenticate with the server and exchange messages.

The connecting server also sends an INFO message. The accepting server will add an ip field containing the address and port of the connecting server, and forward the new server's INFO message to all servers it is routed to.

Any servers in a cluster receiving an INFO message with an ip field will attempt to connect to the server at that address, unless already connected. This propagation of INFO messages on behalf of a connecting server provides automatic discovery of new servers joining a cluster.

Syntax

INFO {["option_name":option_value],...}

The valid options are as follows:

• server_id: The unique identifier of the NATS server
• version: The version of the NATS server
• go: The version of golang the NATS server was built with
• host: The host specified in the cluster parameter/options
• port: The port number specified in the cluster parameter/options
• auth_required: If this is set, then the server should try to authenticate upon connect.
• tls_required: If this is set, then the server must authenticate using TLS.
• max_payload: Maximum payload size that the server will accept.
• connect_urls : A list of server urls that a client can connect to.
• ip: Optional route connection address of a server, nats-route://<hostname>:<port>

Example

Below is an example of an INFO string received by a NATS server, with the ip field.

INFO {"server_id":"KP19vTlB417XElnv8kKaC5","version":"0.9.4","go":"","host":"localhost","port":5222,"auth_required":false,"tls_required":false,"tls_verify":false,"max_payload":1048576,"ip":"nats-route://127.0.0.1:5222/","connect_urls":["localhost:4222"]}

CONNECT

Description

The CONNECT message is analogous to the INFO message. Once the NATS server has established a TCP/IP socket connection with another server, and an INFO message has been received, the server will send a CONNECT message to provide more information about the current connection as well as security information.

Syntax

CONNECT {["option_name":option_value],...}

The valid options are as follows:

• verbose: Turns on +OK protocol acknowledgements.
• pedantic: Turns on additional strict format checking, e.g. for properly formed subjects
• tls_required: Indicates whether the server requires an SSL connection.
• auth_token: Authorization token
• user: Connection username (if auth_required is set)
• pass: Connection password (if auth_required is set)
• name: Generated Server Name
• lang: The implementation language of the server (go).
• version: The version of the server.

Example

Here is an example from the default string from a server.

CONNECT {"verbose":false,"pedantic":false,"tls_required":false,"name":"wt0vffeQyoDGMVBC2aKX0b"}\r\n

Servers should set verbose to false by default. This means that other routed servers will not be sending an +OK payload back after the remote server ingested the message.

PUB

Description

The PUB message publishes the message payload to the given subject name, optionally supplying a reply subject, to another server. If a reply subject is supplied, it will be delivered to eligible subscribers along with the supplied payload. Note that the payload itself is optional. To omit the payload, set the payload size to 0.

Syntax

PUB <subject> [reply-to] <#bytes>\r\n[payload]\r\n

where:

• subject: The destination subject to publish to
• reply-to: The reply inbox subject that subscribers can use to send a response back to the publisher/requestor
• #bytes: The payload size in bytes
• payload: The message payload data

Example

To publish the string message payload "Hello NATS!" to subject FOO:

PUB FOO 11\r\nHello NATS!\r\n

To publish a request message "Knock Knock" to subject FRONT.DOOR with reply subject INBOX.22:

PUB FRONT.DOOR INBOX.22 11\r\nKnock Knock\r\n

To publish an empty message to subject NOTIFY:

PUB NOTIFY 0\r\n\r\n

SUB

Description

SUB initiates a subscription to a subject, optionally joining a distributed queue group.

Syntax

Basic Subscription: SUB <subject> RSID:<cid>:<sid>\r\n

Queue Subscription: SUB <subject> <queue> QRSID:<cid>:<sid>\r\n

where:

• subject: The subject name to subscribe to
• queue: If specified, the subscriber will join this queue group
• cid: A 64bit unsigned integer representing the client connection
• sid: A unique alphanumeric subscription ID representing the server's subscription

Example

To subscribe to the subject FOO with the local unique subject identifier of 1, and the connection-unique subject identifier (sid) 1:

SUB FOO RSID:1:1\r\n

To subscribe the current connection to the subject BAR as part of distribution queue group G1 with local sid of 14, and a client sid 44:

SUB BAR G1 QRSID:14:44\r\n

UNSUB

Description

UNSUB unsubcribes the connection from the specified subject, or auto-unsubscribes after the specified number of messages has been received. It is sent by a server when one of it's clients unsubscribes.

Syntax

Basic Subscription: UNSUB <sid> RSID:<cid>:<sid> [max_msgs]\r\n

Queue Subscription: UNSUB <sid> QRSID:<cid>:<sid> [max_msgs]\r\n

where:

• sid: The unique alphanumeric subscription ID of the subject to unsubscribe from
• max_msgs: Number of messages to wait for before automatically unsubscribing
• cid: A 64bit unsigned integer representing the client connection
• sid: A unique alphanumeric subscription ID representing the server's subscription

Example

The following examples concern subject FOO which has been assigned an internal subscriber id of 5, and a client sid of 1. To unsubscribe from FOO:

UNSUB RSID:5:1\r\n

To auto-unsubscribe from FOO after 5 messages have been received:

UNSUB RSID:1:1 5\r\n

MSG

Description

The MSG protocol message delivers a message from another server.

Syntax

MSG <subject> <sid> [reply-to] <#bytes>\r\n[payload]\r\n

where:

• subject: Subject name this message was received on
• sid: The unique alphanumeric subscription ID of the subject
• reply-to: The inbox subject on which the publisher is listening for responses
• #bytes: Size of the payload in bytes
• payload: The message payload data

Example

The following message delivers a message from subject FOO.BAR:

MSG FOO.BAR 9 11\r\nHello World\r\n

Deliver the same message along with a reply inbox:

MSG FOO.BAR 9 INBOX.34 11\r\nHello World\r\n

PING/PONG

Description

PING and PONG implement a simple keep-alive mechanism between servers. Once two servers establish a connection with each other, the NATS server will continuously send PING messages to other servers at a configurable interval. If another server fails to respond with a PONG message within the configured response interval, the server will terminate its connection. If your connection stays idle for too long, it is cut off.

If the another server sends a ping request, a server will reply with a pong message to notify the other server that it is still present.

Syntax

PING\r\n PONG\r\n

+OK/ERR

Description

When the verbose connection option is set to true (the default value), the server acknowledges each well-formed protocol message with a +OK message. NATS servers set the verbose option to false using the CONNECT message

The -ERR message is used by the server indicate a protocol, authorization, or other runtime connection error to another server. Most of these errors result in the server closing the connection.

Handling of these errors usually has to be done asynchronously.

Syntax

+OK

-ERR <error message>

Protocol error messages which close the connection:

• -ERR 'Unknown Protocol Operation': Unknown protocol error
• -ERR 'Authorization Violation': Server failed to authenticate to another server with credentials specified in the CONNECT message.
• -ERR 'Authorization Timeout': Server took too long to authenticate to another server after establishing a connection (default 1 second)
• -ERR 'Parser Error': Cannot parse the protocol message sent by another server
• -ERR 'Stale Connection': PING/PONG interval expired.
• -ERR 'Slow Consumer': The other server's pending data size for the route connection has been exceeded. The default limit is time based, where data cannot be flushed within a two second write deadline.
• -ERR 'Maximum Payload Exceeded': Server attempted to publish a message with a payload size that exceeds the max_payload size configured on another server.

Protocol error messages which do not close the connection:

• -ERR 'Invalid Subject': Server sent a malformed subject (e.g. sub foo. 90)