GitBook: [master] 326 pages and 16 assets modified

nats-streaming-concepts/channels/README.md

@@ -0,0 +1,8 @@
+# Channels
+
+Channels are at the heart of the NATS Streaming Server. Channels are subjects clients send data to and consume from.
+
+_**Note: NATS Streaming server does not support wildcard for channels, that is, one cannot subscribe on \`foo.**_**`, or`>\`, etc...\***
+
+The number of channels can be limited \(and is by default\) through configuration. Messages produced to a channel are stored in a message log inside this channel.
+

nats-streaming-concepts/channels/message-log.md

@@ -0,0 +1,6 @@
+# Message Log
+
+You can view a message log as a First In First Out \(FIFO\) queue. Messages are appended to the end of the log. If a limit is set globally for all channels, or specifically for this channel, when the limit is reached, older messages are removed to make room for the new ones.
+
+But except for the administrative size/age limit set for a message log, messages are not removed due to consumers consuming them. In fact, messages are stored regardless of the presence of subscriptions on that channel.
+

nats-streaming-concepts/channels/subscriptions/README.md

@@ -0,0 +1,12 @@
+# Subscriptions
+
+A client creates a subscription on a given channel. Remember, there is no support for wildcards, so a subscription is really tied to one and only one channel. The server will maintain the subscription state on behalf of the client until the later closes the subscription \(or its connection\).
+
+If there are messages in the log for this channel, messages will be sent to the consumer when the subscription is created. The server will send up to the maximum number of inflight messages as given by the client when creating the subscription.
+
+When receiving ACKs from the consumer, the server will then deliver more messages, if more are available.
+
+A subscription can be created to start at any point in the message log, either by message sequence, or by time.
+
+The following pages describe all the types of subscriptions.
+

nats-streaming-concepts/channels/subscriptions/durable.md

@@ -0,0 +1,10 @@
+# Durable
+
+If an application wishes to resume message consumption from where it previously stopped, it needs to create a durable subscription. It does so by providing a durable name, which is combined with the client ID provided when the client created its connection. The server then maintains the state for this subscription even after the client connection is closed.
+
+_**Note: The starting position given by the client when restarting a durable subscription is ignored.**_
+
+When the application wants to stop receiving messages on a durable subscription, it should close - but _not unsubscribe_ - this subscription. If a given client library does not have the option to close a subscription, the application should close the connection instead.
+
+When the application wants to delete the subscription, it must unsubscribe it. Once unsubscribed, the state is removed and it is then possible to re-use the durable name, but it will be considered a brand new durable subscription, with the start position being the one given by the client when creating the durable subscription.
+

nats-streaming-concepts/channels/subscriptions/queue-group.md

@@ -0,0 +1,12 @@
+# Queue Group
+
+When consumers want to consume from the same channel but each receive a different message, as opposed to all receiving the same messages, they need to create a queue subscription. When a queue group name is specified, the server will send each messages from the log to a single consumer in the group. The distribution of these messages is not specified, therefore applications should not rely on an expected delivery scheme.
+
+After the first queue member is created, any other member joining the group will receive messages based on where the server is in the message log for that particular group. That means that the starting position given by joining members is ignored by the server.
+
+When the last member of the group leaves \(subscription unsubscribed/closed/or connection closed\), the group is removed from the server. The next application creating a subscription with the same name will create a new group, starting at the start position given in the subscription request.
+
+A queue subscription can also be durable. For that, the client needs to provide a queue and durable name. The behavior is, as you would expect, a combination of queue and durable subscription. Though unlike a durable subscription, the client ID is not part of the queue group name since the client ID must be unique, and would prevent more than one connection to participate in the queue group. The main difference between a queue subscription and a durable one, is that when the last member leaves the group, the state of the group will be maintained by the server. Later, when a member rejoins the group, the delivery will resume.
+
+_**Note: For a durable queue subscription, the last member to**_  **unsubscribe**  _**\(not simply close\) causes the group to be removed from the server.**_
+

nats-streaming-concepts/channels/subscriptions/redelivery.md

@@ -0,0 +1,12 @@
+# Redelivery
+
+When the server sends a message to a consumer, it expects to receive an ACK from this consumer. The consumer is the one specifying how long the server should wait before resending all unacknowledged messages to the consumer.
+
+When the server restarts and recovers unacknowledged messages for a subscription, it will first attempt to redeliver those messages before sending new messages. However, if during the initial redelivery some messages don't make it to the client, the server cannot know that and will enable delivery of new messages.
+
+_**So it is possible for an application to receive redelivered messages mixed with new messages. This is typically what happens outside of the server restart scenario.**_
+
+For queue subscriptions, if a member has unacknowledged messages, when this member's `AckWait` \(which is the duration given to the server before the server should attempt to redeliver unacknowledged messages\) time elapses, the messages are redelivered to any other member in the group \(including itself\).
+
+If a queue member leaves the group, its unacknowledged messages are redistributed to other queue members.
+

nats-streaming-concepts/channels/subscriptions/regular.md

@@ -0,0 +1,4 @@
+# Regular
+
+The state of these subscriptions is removed when they are unsubscribed or closed \(which is equivalent for this type of subscription\) or the client connection is closed \(explicitly by the client, or closed by the server due to timeout\). They do, however, survive a _server_ failure \(if running with a persistent store\).
+

nats-streaming-concepts/client-connections.md

@@ -0,0 +1,8 @@
+# Client Connections
+
+As described, clients are not directly connected to the streaming server. Instead, they send connection requests. The request includes a `client ID` which is used by the server to uniquely identify, and restrict, a given client. That is, no two connections with the same client ID will be able to run concurrently.
+
+This client ID links a given connection to its published messages, subscriptions, especially durable subscriptions. Indeed, durable subscriptions are stored as a combination of the client ID and durable name. More on durable subscriptions later.
+
+It is also used to resolve the issue of not having direct client connections to the server. For instance, say that a client crashes without closing the connection. It later restarts with the same client ID. The server will detect that this client ID is already in-use. It will try to contact that known client to its original private inbox. If the server does not receive a response - which would be the case if the client crashed - it will replace the old client with this new one. Otherwise, the server would reject the connection request since the client ID is already in-use.
+

nats-streaming-concepts/clustering/README.md

@@ -0,0 +1,10 @@
+# Clustering
+
+NATS Streaming Server supports clustering and data replication, implemented with the [Raft consensus algorithm](https://raft.github.io/), for the purposes of high availability.
+
+There are two ways to bootstrap a cluster: with an explicit cluster configuration or with "auto" configuration using a seed node. With the first, we provide the IDs of the nodes participating in the cluster. In this case, the participating nodes will elect a leader. With the second, we start one server as a seed node, which will elect itself as leader, and subsequent servers will automatically join the seed \(note that this also works with the explicit cluster configuration once the leader has been established\). With the second method, we need to be careful to avoid starting multiple servers as seed as this will result in a split-brain. Both of these configuration methods are shown in the sections below.
+
+It is recommended to run an odd number of servers in a cluster with a minimum of three servers to avoid split-brain scenarios. Note that if less than a majority of servers are available, the cluster cannot make progress, e.g. if two nodes go down in a cluster of three, the cluster is unavailable until at least one node comes back.
+
+Note about Channels Partitioning and Clustering. These two features are mutually exclusive. Trying to start a server with channels Partitioning and Clustering enabled will result in a startup error. Clustering requires all channels to be replicated in the cluster.
+

nats-streaming-concepts/clustering/auto-configuration.md

@@ -0,0 +1,18 @@
+# Auto Configuration
+
+We can also bootstrap a NATS Streaming cluster by starting **one server** as the seed node using the `-cluster_bootstrap` flag. This node will elect itself leader, **so it's important to avoid starting multiple servers as seed**. Once a seed node is started, other servers will automatically join the cluster. If the server is recovering, it will use the recovered cluster configuration.
+
+Here is an example of starting three servers in a cluster by starting one as the seed and letting the others automatically join:
+
+```text
+nats-streaming-server -store file -dir store-a -clustered -cluster_bootstrap -nats_server nats://localhost:4222
+
+nats-streaming-server -store file -dir store-b -clustered -nats_server nats://localhost:4222
+
+nats-streaming-server -store file -dir store-c -clustered -nats_server nats://localhost:4222
+```
+
+For a given cluster ID, if more than one server is started with `cluster_bootstrap` set to true, each server with this parameter will report the misconfiguration and exit.
+
+The very first server that bootstrapped the cluster can be restarted, however, the operator **must remove the datastores** of the other servers that were incorrectly started with the bootstrap parameter before attempting to restart them. If they are restarted with existing state, even without the `-cluster_bootstrap` parameter, they will once again start as a leader.
+

nats-streaming-concepts/clustering/configuration.md

@@ -0,0 +1,96 @@
+# Configuration
+
+We can bootstrap a NATS Streaming cluster by providing the cluster topology using the `-cluster_peers` flag. This is simply the set of node IDs participating in the cluster. Note that once a leader is established, we can start subsequent servers without providing this configuration as they will automatically join the leader. If the server is recovering, it will use the recovered cluster configuration.
+
+Here is an example of starting three servers in a cluster. For this example, we run a separate NATS server which the Streaming servers connect to.
+
+```text
+nats-streaming-server -store file -dir store-a -clustered -cluster_node_id a -cluster_peers b,c -nats_server nats://localhost:4222
+
+nats-streaming-server -store file -dir store-b -clustered -cluster_node_id b -cluster_peers a,c -nats_server nats://localhost:4222
+
+nats-streaming-server -store file -dir store-c -clustered -cluster_node_id c -cluster_peers a,b -nats_server nats://localhost:4222
+```
+
+Note that once a leader is elected, subsequent servers can be started without providing the cluster configuration. They will automatically join the cluster. Similarly, the cluster node ID does not need to be provided as one will be automatically assigned. As long as the file store is used, this ID will be recovered on restart.
+
+```text
+nats-streaming-server -store file -dir store-d -clustered -nats_server nats://localhost:4222
+```
+
+The equivalent clustering configurations can be specified in a configuration file under the `cluster` group. See the [Configuring](../../nats-streaming-server/configuring/) section for more information.
+
+Here is an example of a cluster of 3 nodes using the following configuration files. The nodes are running on `host1`, `host2` and `host3` respectively. **NOTE** If you have an existing NATS cluster and want to run NATS Streaming Cluster on top of that, see details at the end of this section.
+
+On `host1`, this configuration indicates that the server will accept client connections on port 4222. It will accept route connections on port 6222. It creates 2 routes, to `host2` and `host3` cluster port.
+
+It defines the NATS Streaming cluster name as `mycluster`, uses a store file that points to the `store` directory. The `cluster` section inside `streaming` makes the NATS Streaming server run in cluster mode. This configuration explicitly define each node id \(`a` for `host1`\) and list its peers.
+
+```text
+# NATS specific configuration
+port: 4222
+cluster {
+  listen: 0.0.0.0:6222
+  routes: ["nats://host2:6222", "nats://host3:6222"]
+}
+
+# NATS Streaming specific configuration
+streaming {
+  id: mycluster
+  store: file
+  dir: store
+  cluster {
+    node_id: "a"
+    peers: ["b", "c"]
+  }
+}
+```
+
+Below is the configuration for the server running on `host2`. Notice how the routes are now to `host1` and `host3`. The other thing that changed is the node id that is set to `b` and peers are updated accordingly to `a` and `c`.
+
+Note that the `dir` configuration is also `store` but these are local directories and do not \(actually must not\) be shared. Each node will have its own copy of the datastore. You could have each configuration have a different value for `dir` if desired.
+
+```text
+# NATS specific configuration
+port: 4222
+cluster {
+  listen: 0.0.0.0:6222
+  routes: ["nats://host1:6222", "nats://host3:6222"]
+}
+
+# NATS Streaming specific configuration
+streaming {
+  id: mycluster
+  store: file
+  dir: store
+  cluster {
+    node_id: "b"
+    peers: ["a", "c"]
+  }
+}
+```
+
+As you would expect, for `host3`, the routes are now to `host1` and `host2` and the node id is `c` while its peers are `a` and `b`.
+
+```text
+# NATS specific configuration
+port: 4222
+cluster {
+  listen: 0.0.0.0:6222
+  routes: ["nats://host1:6222", "nats://host2:6222"]
+}
+
+# NATS Streaming specific configuration
+streaming {
+  id: mycluster
+  store: file
+  dir: store
+  cluster {
+    node_id: "c"
+    peers: ["a", "b"]
+  }
+}
+```
+
+In the example above, the configuration assumes no existing NATS Cluster and therefore configure the NATS routes between each node. Should you want to use an existing NATS cluster, do not include the "NATS specific configuration" section, instead, add `nats_server_url` in the `streaming` section to point to the NATS server you want.
+

nats-streaming-concepts/clustering/containers.md

@@ -0,0 +1,12 @@
+# Containers
+
+When running the docker image of NATS Streaming Server, you will want to specify a mounted volume so that the data can be recovered. Your `-dir` parameter then points to a directory inside that mounted volume. However, after a restart you may get a failure with a message similar to this:
+
+```text
+[FTL] STREAM: Failed to start: streaming state was recovered but cluster log path "mycluster/a" is empty
+```
+
+This is because the server recovered the streaming state \(as pointed by `-dir` and located in the mounted volume\), but did not recover the RAFT specific state that is by default stored in a directory named after your cluster id, relative to the current directory starting the executable. In the context of a container, this data will be lost after the container is stopped.
+
+In order to avoid this issue, you need to specify the `-cluster_log_path` and ensure that it points to the mounted volume so that the RAFT state can be recovered along with the Streaming state.
+

nats-streaming-concepts/clustering/supported-stores.md

@@ -0,0 +1,10 @@
+# Supported Stores
+
+In order to run NATS Streaming Server in clustered mode, you need to specify a persistent store. At this time you have the choice between `FILE` and `SQL`
+
+The NATS Streaming stores server meta information, messages and subscriptions to the storage you configure using the `--store` option.
+
+However, in clustered mode, we use RAFT for leader election. The raft layer uses its own stores which are currently necessarily file based. The location of the RAFT stores defaults to the current directory under a sub-directory named after the cluster ID, or you can configure it using `--cluster_log_path`.
+
+This means that even if you select a SQL Store, there will still be a need for storing data on the file system.
+

nats-streaming-concepts/ft/README.md

@@ -0,0 +1,16 @@
+# Fault Tolerance
+
+To minimize the single point of failure, NATS Streaming server can be run in Fault Tolerance mode. It works by having a group of servers with one acting as the active server \(accessing the store\) and handling all communication with clients, and all others acting as standby servers.
+
+It is important to note that is not possible to run Nats Streaming as Fault Tolerance mode and Clustering mode at the same time.
+
+To start a server in Fault Tolerance \(FT\) mode, you specify an FT group name.
+
+Here is an example on how starting 2 servers in FT mode running on the same host and embedding the NATS servers:
+
+```text
+nats-streaming-server -store file -dir datastore -ft_group "ft" -cluster nats://localhost:6222 -routes nats://localhost:6223 -p 4222
+
+nats-streaming-server -store file -dir datastore -ft_group "ft" -cluster nats://localhost:6223 -routes nats://localhost:6222 -p 4223
+```
+

nats-streaming-concepts/ft/active-server.md

@@ -0,0 +1,8 @@
+# Active Server
+
+There is a single Active server in the group. This server was the first to obtain the exclusive lock for storage. For the `FileStore` implementation, it means trying to get an advisory lock for a file located in the shared datastore. For the `SQLStore` implementation, a special table is used in which the owner of the lock updates a column. Other instances will steal the lock if the column is not updated for a certain amount of time.
+
+If the elected server fails to grab this lock because it is already locked, it will go back to standby.
+
+_**Only the active server accesses the store and service all clients.**_
+

nats-streaming-concepts/ft/failover.md

@@ -0,0 +1,10 @@
+# Failover
+
+When the active server fails, all standby servers will try to activate. The process consists of trying to get an exclusive lock on the storage.
+
+The first server that succeeds will become active and go through the process of recovering the store and service clients. It is as if a server in standalone mode was automatically restarted.
+
+All other servers that failed to get the store lock will go back to standby mode and stay in this mode until they stop receiving heartbeats from the current active server.
+
+It is possible that a standby trying to activate is not able to immediately acquire the store lock. When that happens, it goes back into standby mode, but if it fails to receive heartbeats from an active server, it will try again to acquire the store lock. The interval is random but as of now set to a bit more than a second.
+

nats-streaming-concepts/ft/shared-state.md

@@ -0,0 +1,4 @@
+# Shared State
+
+Actual file replication to multiple disks is not handled by the Streaming server. This - if required - needs to be handled by the user. For the FileStore implementation that we currently provide, the data store needs to be mounted by all servers in the FT group \(e.g. an NFS Mount, Gluster in Google Cloud or EFS in Amazon\).
+

nats-streaming-concepts/ft/standby-server.md

@@ -0,0 +1,4 @@
+# Standby Servers
+
+There can be as many standby servers as you want in the same group. These servers do not access the store and do not receive any data from the streaming clients. They are just running waiting for the detection of the active server failure.
+

nats-streaming-concepts/intro.md

@@ -0,0 +1,31 @@
+# Introduction
+
+NATS Streaming is a data streaming system powered by NATS, and written in the Go programming language. The executable name for the NATS Streaming server is `nats-streaming-server`. NATS Streaming embeds, extends, and interoperates seamlessly with the core NATS platform. The [NATS Streaming server](https://github.com/nats-io/nats-streaming-server) is provided as open source software under the Apache-2.0 license. Synadia actively maintains and supports the NATS Streaming server.
+
+![](../.gitbook/assets/nats_streaming.svg)
+
+## Features
+
+In addition to the features of the core NATS platform, NATS Streaming provides the following:
+
+* **Enhanced message protocol** - NATS Streaming implements its own enhanced message format using [Google Protocol Buffers](https://developers.google.com/protocol-buffers/). These messages are transmitted as binary message payloads via core NATS platform, and thus require no changes to the basic NATS protocol.
+* **Message/event persistence** - NATS Streaming offers configurable message persistence: in-memory, flat files or database. The storage subsystem uses a public interface that allows contributors to develop their own custom implementations.
+* **At-least-once-delivery** - NATS Streaming offers message acknowledgements between publisher and server \(for publish operations\) and between subscriber and server \(to confirm message delivery\). Messages are persisted by the server in memory or secondary storage \(or other external storage\) and will be redelivered to eligible subscribing clients as needed.
+* **Publisher rate limiting** - NATS Streaming provides a connection option called `MaxPubAcksInFlight` that effectively limits the number of unacknowledged messages that a publisher may have in-flight at any given time. When this maximum is reached, further async publish calls will block until the number of unacknowledged messages falls below the specified limit.
+* **Rate matching/limiting per subscriber** - Subscriptions may specify a `MaxInFlight` option that designates the maximum number of outstanding acknowledgements \(messages that have been delivered but not acknowledged\) that NATS Streaming will allow for a given subscription. When this limit is reached, NATS Streaming will suspend delivery of messages to this subscription until the number of unacknowledged messages falls below the specified limit.
+* **Historical message replay by subject** - New subscriptions may specify a start position in the stream of messages stored for the subscribed subject's channel. By using this option, message delivery may begin at:
+  * The earliest message stored for this subject
+  * The most recently stored message for this subject, prior to the start of the current subscription. This is commonly thought of as "last value" or "initial value" caching.
+  * A specific date/time in nanoseconds
+  * An historical offset from the current server date/time, e.g. the last 30 seconds.
+  * A specific message sequence number
+* **Durable subscriptions** - Subscriptions may also specify a "durable name" which will survive client restarts. Durable subscriptions cause the server to track the last acknowledged message sequence number for a client and durable name. When the client restarts/resubscribes, and uses the same client ID and durable name, the server will resume delivery beginning with the earliest unacknowledged message for this durable subscription.
+
+## Installation
+
+NATS provides a [server binary](../nats-streaming-server/install.md) for Linux, Mac, and Windows. You can install the server from source on any platform you choose.
+
+## Usage, Configuration and Administration
+
+NATS Streaming provides a rich set of commands and parameters to configure all aspects of the server. Please refer to the [Configuring](../nats-streaming-server/configuring/) for further information on usage, configuration, and administration.
+

nats-streaming-concepts/monitoring/README.md

@@ -0,0 +1,67 @@
+# Monitoring
+
+To monitor the NATS Streaming system, a lightweight HTTP server is used on a dedicated monitoring port. The monitoring server provides several endpoints, all returning a JSON object.
+
+## Enabling from the command line
+
+To enable the monitoring server, start the NATS Streaming Server with the monitoring flag -m \(or -ms\) and specify the monitoring port.
+
+Monitoring options
+
+```text
+-m, --http_port PORT             HTTP PORT for monitoring
+-ms,--https_port PORT            Use HTTPS PORT for monitoring (requires TLS cert and key)
+```
+
+To enable monitoring via the configuration file, use `http: "host:port"` or `https: "host:port"`. There is no explicit configuration flag for the monitoring interface.
+
+For example, after running this:
+
+```text
+nats-streaming-server -m 8222
+```
+
+you should see that the NATS Streaming server starts with the HTTP monitoring port enabled:
+
+```bash
+[19339] 2019/06/24 15:02:38.251091 [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.15.1
+[19339] 2019/06/24 15:02:38.251238 [INF] STREAM: ServerID: 0Z2HXClEM6BPsGaKcoHg5N
+[19339] 2019/06/24 15:02:38.251243 [INF] STREAM: Go version: go1.12
+[19339] 2019/06/24 15:02:38.251862 [INF] Starting nats-server version 2.0.0
+[19339] 2019/06/24 15:02:38.251873 [INF] Git commit [not set]
+[19339] 2019/06/24 15:02:38.252173 [INF] Starting http monitor on 0.0.0.0:8222
+[19339] 2019/06/24 15:02:38.252248 [INF] Listening for client connections on 0.0.0.0:4222
+(...)
+```
+
+You can then point your browser \(or curl\) to [http://localhost:8222/streaming](http://localhost:8222/streaming)
+
+## Enabling from the configuration file
+
+To start via the configuration file you can define the monitoring port as follows:
+
+```text
+http_port = 8222
+```
+
+Then use the `-sc` flag to customize the NATS Streaming configuration:
+
+```bash
+nats-streaming-server -sc nats-streaming.conf -ns nats://demo.nats.io:4222 -SDV
+```
+
+Confirm that the monitoring endpoint is enabled by sending a request:
+
+```bash
+curl 127.0.0.1:8222/streaming/channelsz
+{
+  "cluster_id": "test-cluster",
+  "server_id": "dXUsNRef1z25NpcFmZhBNj",
+  "now": "2019-06-24T15:18:37.388938-07:00",
+  "offset": 0,
+  "limit": 1024,
+  "count": 0,
+  "total": 0
+}
+```
+

nats-streaming-concepts/monitoring/endpoints.md

@@ -0,0 +1,340 @@
+# Endpoints
+
+The following sections describe each supported monitoring endpoint: serverz, storez, clientsz, and channelsz.
+
+## /serverz
+
+The endpoint [http://localhost:8222/streaming/serverz](http://localhost:8222/streaming/serverz) reports various general statistics.
+
+```text
+{
+  "cluster_id": "test-cluster",
+  "server_id": "JEzjfVQS4JIEzM7lZmWHm9",
+  "version": "0.14.2",
+  "go": "go1.11.10",
+  "state": "STANDALONE",
+  "now": "2019-05-21T11:09:35.364637-06:00",
+  "start_time": "2019-05-21T11:09:24.204869-06:00",
+  "uptime": "11s",
+  "clients": 0,
+  "subscriptions": 0,
+  "channels": 0,
+  "total_msgs": 0,
+  "total_bytes": 0
+}
+```
+
+In clustering mode, there is an additional field that indicates the RAFT role of the given node. Here is an example:
+
+```text
+{
+  "cluster_id": "test-cluster",
+  "server_id": "t9W9zbOIIi5Y9Guppxl0lF",
+  "version": "0.14.2",
+  "go": "go1.11.10",
+  "state": "CLUSTERED",
+  "role": "Follower",
+  "now": "2019-05-21T11:10:15.765261-06:00",
+  "start_time": "2019-05-21T11:10:12.21284-06:00",
+  "uptime": "3s",
+  "clients": 0,
+  "subscriptions": 0,
+  "channels": 0,
+  "total_msgs": 0,
+  "total_bytes": 0
+}
+```
+
+The possible values are: `Leader`, `Follower` or `Candidate`.
+
+## /storez
+
+The endpoint [http://localhost:8222/streaming/storez](http://localhost:8222/streaming/storez) reports information about the store.
+
+```text
+{
+  "cluster_id": "test-cluster",
+  "server_id": "8AjZq57k4JY7cfKEvuZ8iF",
+  "now": "2019-04-16T09:57:32.857406-06:00",
+  "type": "MEMORY",
+  "limits": {
+    "max_channels": 100,
+    "max_msgs": 1000000,
+    "max_bytes": 1024000000,
+    "max_age": 0,
+    "max_subscriptions": 1000,
+    "max_inactivity": 0
+  },
+  "total_msgs": 130691,
+  "total_bytes": 19587140
+}
+```
+
+## /clientsz
+
+The endpoint [http://localhost:8222/streaming/clientsz](http://localhost:8222/streaming/clientsz) reports more detailed information about the connected clients.
+
+It uses a paging mechanism which defaults to 1024 clients.
+
+You can control these via URL arguments \(limit and offset\). For example: [http://localhost:8222/streaming/clientsz?limit=1&offset=1](http://localhost:8222/streaming/clientsz?limit=1&offset=1).
+
+```text
+{
+  "cluster_id": "test-cluster",
+  "server_id": "J3Odi0wXYKWKFWz5D5uhH9",
+  "now": "2017-06-07T14:47:44.495254605+02:00",
+  "offset": 1,
+  "limit": 1,
+  "count": 1,
+  "total": 11,
+  "clients": [
+    {
+      "id": "benchmark-sub-0",
+      "hb_inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayQK"
+    }
+  ]
+}
+```
+
+You can also report detailed subscription information on a per client basis using `subs=1`. For example: [http://localhost:8222/streaming/clientsz?limit=1&offset=1&subs=1](http://localhost:8222/streaming/clientsz?limit=1&offset=1&subs=1).
+
+```text
+{
+  "cluster_id": "test-cluster",
+  "server_id": "J3Odi0wXYKWKFWz5D5uhH9",
+  "now": "2017-06-07T14:48:06.157468748+02:00",
+  "offset": 1,
+  "limit": 1,
+  "count": 1,
+  "total": 11,
+  "clients": [
+    {
+      "id": "benchmark-sub-0",
+      "hb_inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayQK",
+      "subscriptions": {
+        "foo": [
+          {
+            "client_id": "benchmark-sub-0",
+            "inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayvC",
+            "ack_inbox": "_INBOX.J3Odi0wXYKWKFWz5D5uhem",
+            "is_durable": false,
+            "is_offline": false,
+            "max_inflight": 1024,
+            "ack_wait": 30,
+            "last_sent": 505597,
+            "pending_count": 0,
+            "is_stalled": false
+          }
+        ]
+      }
+    }
+  ]
+}
+```
+
+You can select a specific client based on its client ID with `client=<id>`, and get also get detailed statistics with `subs=1`. For example: [http://localhost:8222/streaming/clientsz?client=me&subs=1](http://localhost:8222/streaming/clientsz?client=me&subs=1).
+
+```text
+{
+  "id": "me",
+  "hb_inbox": "_INBOX.HG0uDuNtAPxJQ1lVjIC2sr",
+  "subscriptions": {
+    "foo": [
+      {
+        "client_id": "me",
+        "inbox": "_INBOX.HG0uDuNtAPxJQ1lVjIC389",
+        "ack_inbox": "_INBOX.Q9iH2gsDPN57ZEvqswiYSL",
+        "is_durable": false,
+        "is_offline": false,
+        "max_inflight": 1024,
+        "ack_wait": 30,
+        "last_sent": 0,
+        "pending_count": 0,
+        "is_stalled": false
+      }
+    ]
+  }
+}
+```
+
+## /channelsz
+
+The endpoint [http://localhost:8222/streaming/channelsz](http://localhost:8222/streaming/channelsz) reports the list of channels.
+
+```text
+{
+  "cluster_id": "test-cluster",
+  "server_id": "J3Odi0wXYKWKFWz5D5uhH9",
+  "now": "2017-06-07T14:48:41.680592041+02:00",
+  "offset": 0,
+  "limit": 1024,
+  "count": 2,
+  "total": 2,
+  "names": [
+    "bar"
+    "foo"
+  ]
+}
+```
+
+It uses a paging mechanism which defaults to 1024 channels.
+
+You can control these via URL arguments \(limit and offset\). For example: [http://localhost:8222/streaming/channelsz?limit=1&offset=1](http://localhost:8222/streaming/channelsz?limit=1&offset=1).
+
+```text
+{
+  "cluster_id": "test-cluster",
+  "server_id": "J3Odi0wXYKWKFWz5D5uhH9",
+  "now": "2017-06-07T14:48:41.680592041+02:00",
+  "offset": 1,
+  "limit": 1,
+  "count": 1,
+  "total": 2,
+  "names": [
+    "foo"
+  ]
+}
+```
+
+You can also get the list of subscriptions with `subs=1`. For example: [http://localhost:8222/streaming/channelsz?limit=1&offset=0&subs=1](http://localhost:8222/streaming/channelsz?limit=1&offset=0&subs=1).
+
+```text
+{
+  "cluster_id": "test-cluster",
+  "server_id": "J3Odi0wXYKWKFWz5D5uhH9",
+  "now": "2017-06-07T15:01:02.166116959+02:00",
+  "offset": 0,
+  "limit": 1,
+  "count": 1,
+  "total": 2,
+  "channels": [
+    {
+      "name": "bar",
+      "msgs": 0,
+      "bytes": 0,
+      "first_seq": 0,
+      "last_seq": 0,
+      "subscriptions": [
+        {
+          "client_id": "me",
+          "inbox": "_INBOX.S7kTJjOcToXiJAzGWgINit",
+          "ack_inbox": "_INBOX.Y04G5pZxlint3yPXrSTjTV",
+          "is_durable": false,
+          "is_offline": false,
+          "max_inflight": 1024,
+          "ack_wait": 30,
+          "last_sent": 0,
+          "pending_count": 0,
+          "is_stalled": false
+        }
+      ]
+    }
+  ]
+}
+```
+
+You can select a specific channel based on its name with `channel=name`. For example: [http://localhost:8222/streaming/channelsz?channel=foo](http://localhost:8222/streaming/channelsz?channel=foo).
+
+```text
+{
+  "name": "foo",
+  "msgs": 649234,
+  "bytes": 97368590,
+  "first_seq": 1,
+  "last_seq": 649234
+}
+```
+
+And again, you can get detailed subscriptions with `subs=1`. For example: [http://localhost:8222/streaming/channelsz?channel=foo&subs=1](http://localhost:8222/streaming/channelsz?channel=foo&subs=1).
+
+```text
+{
+  "name": "foo",
+  "msgs": 704770,
+  "bytes": 105698990,
+  "first_seq": 1,
+  "last_seq": 704770,
+  "subscriptions": [
+    {
+      "client_id": "me",
+      "inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayvC",
+      "ack_inbox": "_INBOX.J3Odi0wXYKWKFWz5D5uhem",
+      "is_durable": false,
+      "is_offline": false,
+      "max_inflight": 1024,
+      "ack_wait": 30,
+      "last_sent": 704770,
+      "pending_count": 0,
+      "is_stalled": false
+    },
+    {
+      "client_id": "me2",
+      "inbox": "_INBOX.jAHSY3hcL5EGFQGYmfaywG",
+      "ack_inbox": "_INBOX.J3Odi0wXYKWKFWz5D5uhjV",
+      "is_durable": false,
+      "is_offline": false,
+      "max_inflight": 1024,
+      "ack_wait": 30,
+      "last_sent": 704770,
+      "pending_count": 0,
+      "is_stalled": false
+    },
+    (...)
+  ]
+}
+```
+
+For durables that are currently running, the `is_offline` field is set to `false`. Here is an example:
+
+```text
+{
+  "name": "foo",
+  "msgs": 0,
+  "bytes": 0,
+  "first_seq": 0,
+  "last_seq": 0,
+  "subscriptions": [
+    {
+      "client_id": "me",
+      "inbox": "_INBOX.P23kNGFnwC7KRg3jIMB3IL",
+      "ack_inbox": "_STAN.ack.pLyMpEyg7dgGZBS7jGXC02.foo.pLyMpEyg7dgGZBS7jGXCaw",
+      "durable_name": "dur",
+      "is_durable": true,
+      "is_offline": false,
+      "max_inflight": 1024,
+      "ack_wait": 30,
+      "last_sent": 0,
+      "pending_count": 0,
+      "is_stalled": false
+    }
+  ]
+}
+```
+
+When that same durable goes offline, `is_offline` is be set to `true`. Although the client is possibly no longer connected \(and would not appear in the `clientsz` endpoint\), the `client_id` field is still displayed here.
+
+```text
+{
+  "name": "foo",
+  "msgs": 0,
+  "bytes": 0,
+  "first_seq": 0,
+  "last_seq": 0,
+  "subscriptions": [
+    {
+      "client_id": "me",
+      "inbox": "_INBOX.P23kNGFnwC7KRg3jIMB3IL",
+      "ack_inbox": "_STAN.ack.pLyMpEyg7dgGZBS7jGXC02.foo.pLyMpEyg7dgGZBS7jGXCaw",
+      "durable_name": "dur",
+      "is_durable": true,
+      "is_offline": true,
+      "max_inflight": 1024,
+      "ack_wait": 30,
+      "last_sent": 0,
+      "pending_count": 0,
+      "is_stalled": false
+    }
+  ]
+}
+```
+

nats-streaming-concepts/partitioning.md

@@ -0,0 +1,124 @@
+# Partitioning
+
+_**Note, this feature is incompatible with Clustering mode. Trying to start a server with Partitioning and Clustering enabled will result in a startup error.**_
+
+It is possible to limit the list of channels a server can handle. This can be used to:
+
+* Prevent creation of unwanted channels
+* Share the load between several servers running with the same cluster ID
+
+In order to do so, you need to enable the `partitioning` parameter in the configuration file, and also specify the list of allowed channels in the `channels` section of the `store_limits` configuration.
+
+Channels don't need to override any limit, but they need to be specified for the server to service only these channels.
+
+Here is an example:
+
+```text
+partitioning: true
+store_limits: {
+    channels: {
+        "foo": {}
+        "bar": {}
+        # Use of wildcards in configuration is allowed. However, applications cannot
+        # publish to, or subscribe to, wildcard channels.
+        "baz.*": {}
+    }
+}
+```
+
+When partitioning is enabled, multiple servers with the same cluster ID can coexist on the same NATS network, each server handling its own set of channels. _**Note however that in this mode, state is not replicated as it is in Clustering mode. The only communication between servers is to report if a given channel is handled in more than one server.**_
+
+## Wildcards
+
+NATS Streaming does not support sending or subscribing to wildcard channels \(such as `foo.*`\).
+
+However, it is possible to use wildcards to define the partition that a server can handle. For instance, with the following configuration:
+
+```text
+partitioning: true
+store_limits: {
+    channels: {
+        "foo.*": {}
+        "bar.>": {}
+    }
+}
+```
+
+The streaming server would accept subscriptions or published messages to channels such as: 1. `foo.bar` 2. `bar.baz` 3. `bar.baz.bat` 4. ...
+
+But would ignore messages or subscriptions on:
+
+1. `foo`
+2. `foo.bar.baz`
+3. `bar`
+4. `some.other.channel`
+5. ...
+
+## A given channel must be defined in a single server
+
+When a server starts, it sends its list of channels to all other servers on the same cluster in an attempt to detect duplicate channels. When a server receives this list and finds that it has a channel in common, it will return an error to the emitting server, which will then fail to start.
+
+However, on startup, it is possible that the underlying NATS cluster is not fully formed. The server would not get any response from the rest of the cluster and therefore start successfully and service clients. Anytime a Streaming server detects that a NATS server was added to the NATS cluster, it will resend its list of channels. It means that currently running servers may suddenly fail with a message regarding duplicate channels. Having the same channel on different servers means that a subscription would be created on all servers handling the channel, but only one server will receive and process message acknowledgements. Other servers would then redeliver messages \(since they would not get the acknowledgements\), which would cause duplicates.
+
+_**In order to avoid issues with channels existing on several servers, it is ultimately the responsibility of the administrator to ensure that channels are unique.**_
+
+## Fault Tolerance and Partitioning
+
+You can easily combine the Fault Tolerance and Partitioning feature.
+
+To illustrate, suppose that we want two partitions, one for `foo.>` and one for `bar.>`.
+
+The configuration for the first server `foo.conf` would look like:
+
+```text
+partitioning: true
+store_limits: {
+    channels: {
+        foo.>: {}
+    }
+}
+```
+
+The second configuration `bar.conf` would be:
+
+```text
+partitioning: true
+store_limits: {
+    channels: {
+        bar.>: {}
+    }
+}
+```
+
+If you remember, Fault Tolerance is configured by specifying a name \(`ft_group_name`\). Suppose there is an NFS mount called `/nss/datastore` on both `host1` and `host2`.
+
+Starting an FT pair for the partition `foo` could look like this:
+
+```text
+host1$ nats-streaming-server -store file -dir /nss/datastore/foodata -sc foo.conf -ft_group_name foo -cluster nats://host1:6222 -routes nats://host2:6222,nats://host2:6223
+
+host2$ nats-streaming-server -store file -dir /nss/datastore/foodata -sc foo.conf -ft_group_name foo -cluster nats://host2:6222 -routes nats://host1:6222,nats://host1:6223
+```
+
+Notice that each server on each node points to each other \(the `-routes` parameter\). The reason why we also point to `6223` will be explained later. They both listen for routes connections on their host's `6222` port.
+
+We now start the FT pair for `bar`. Since we are running from the same machines \(we don't have to\), we need to use a different port:
+
+```text
+host1$ nats-streaming-server -store file -dir /nss/datastore/bardata -sc bar.conf -ft_group_name bar -p 4223 -cluster nats://host1:6223 -routes nats://host2:6222,nats://host2:6223
+
+host2$ nats-streaming-server -store file -dir /nss/datastore/bardata -sc bar.conf -ft_group_name bar -p 4223 -cluster nats://host2:6223 -routes nats://host1:6222,nats://host1:6223
+```
+
+You will notice that the `-routes` parameter points to both `6222` and `6223`, this is so that both partitions belong to the same cluster and be viewed as "one" by a Streaming application connecting to this cluster. Effectively, we have created a full mesh of 4 NATS servers that can all communicate with each other. Two of these servers are backups for servers running on the same FT group.
+
+## Applications behavior
+
+When an application connects, it specifies a cluster ID. If several servers are running with that same cluster ID, the application will be able to publish/subscribe to any channel handled by the cluster \(as long as those servers are all connected to the NATS network\).
+
+A published message will be received by only the server that has that channel defined. If no server is handling this channel, no specific error is returned, instead the publish call will timeout. Same goes for message acknowledgements. Only the server handling the subscription on this channel should receive those.
+
+However, other client requests \(such as connection and subscription requests\) are received by all servers. For connections, all servers handle them and the client library will receive a response from all servers in the cluster, but will use the first one that it received.
+
+For subscriptions, a server receiving the request for a channel that it does not handle will simply ignore the request. Again, if no server handles this channel, the client's subscription request will simply time out.
+

nats-streaming-concepts/relation-to-nats.md

@@ -0,0 +1,14 @@
+# Relation to NATS
+
+NATS Streaming Server by default embeds a [NATS](https://github.com/nats-io/nats-server) server. That is, the Streaming server is not a server per-se, but instead, a client to a NATS Server.
+
+It means that Streaming clients are not directly connected to the streaming server, but instead communicate with the streaming server _through_ NATS Server.
+
+This detail is important when it comes to Streaming clients connections to the Streaming server. Indeed, since there is no direct connection, the server knows if a client is connected based on heartbeats.
+
+_**It is therefore strongly recommended for clients to close their connection when the application exits, otherwise the server will consider these clients connected \(sending data, etc...\) until it detects missing heartbeats.**_
+
+The streaming server creates internal subscriptions on specific subjects to communicate with its clients and/or other servers.
+
+Note that NATS clients and NATS Streaming clients cannot exchange data between each other. That is, if a streaming client publishes on `foo`, a NATS client subscribing on that same subject will not receive the messages. Streaming messages are NATS messages made of a protobuf. The streaming server is expected to send ACKs back to producers and receive ACKs from consumers. If messages were freely exchanged with the NATS clients, this would cause problems.
+

nats-streaming-concepts/store-encryption.md

@@ -0,0 +1,34 @@
+# Store Encryption
+
+The server can be configured to encrypt a message's payload when storing them, providing encryption at rest. This can be done from the command line or from the configuration file. Check `encrypt` and `encryption_key` in the [Configuring](store-encryption.md#configuring) section.
+
+It is recommended to provide the encryption key through the environment variable `NATS_STREAMING_ENCRYPTION_KEY` instead of `encryption_key`. If encryption is enabled and `NATS_STREAMING_ENCRYPTION_KEY` is found, this will take precedence over `encryption_key` value.
+
+You can pass this from the command line this way:
+
+```text
+$ env NATS_STREAMING_ENCRYPTION_KEY="mykey" nats-streaming-server -store file -dir datastore -encrypt
+```
+
+We currently support two ciphers for encryption: [AES](https://godoc.org/crypto/aes) and [CHACHA](https://godoc.org/golang.org/x/crypto/chacha20poly1305). The default selected cipher depends on the platform. For ARM, we use `CHACHA`, otherwise we default to `AES`. You can always override that decision by explicitly specifying the cipher like this:
+
+```text
+$ env NATS_STREAMING_ENCRYPTION_KEY="mykey" nats-streaming-server -store file -dir datastore -encrypt -encryption_cipher "CHACHA"
+```
+
+or, to select `AES`:
+
+```text
+$ env NATS_STREAMING_ENCRYPTION_KEY="mykey" nats-streaming-server -store file -dir datastore -encrypt -encryption_cipher "AES"
+```
+
+Note that only message payload is encrypted, all other data stored by NATS Streaming server is not.
+
+When running in clustering mode \(see below\), the server uses RAFT, which uses its own log files. Those will be encrypted too.
+
+Starting a server with `encrypt` against a datastore that was not encrypted may result in failures when it comes to decrypt a message, which may not happen immediately upon startup. Instead, it will happen when attempting to deliver messages to consumers. However, when possible, the server will detect if the data was not encrypted and return the data without attempting to decrypt it. The server will also detect which cipher was used to encrypt the data and use the proper cipher to decrypt, even if this is not the currently selected cipher.
+
+If the data is encrypted with a key and the server is restarted with a different key, the server will fail to decrypt messages when attempting to load them from the store.
+
+Performance considerations: As expected, encryption is likely to decrease performance, but by how much is hard to define. In some performance tests on a MacbookPro 2.8 GHz Intel Core i7 with SSD, we have observed as little as 1% decrease to more than 30%. In addition to CPU cycles required for encryption, the encrypted payload is bigger, which result in more data being stored or read.
+

nats-streaming-concepts/store-interface.md

@@ -0,0 +1,16 @@
+# Store Interface
+
+Every store implementation follows the [Store interface](https://github.com/nats-io/nats-streaming-server/blob/master/stores/store.go).
+
+On startup, the server creates a unique instance of the `Store`. The constructor of a store implementation can do some initialization and configuration check, but _must not_ access, or attempt to recover, the storage at this point. This is important because when the server runs on Fault Tolerance mode, the storage must be shared across many servers but only one server can be using it.
+
+After instantiating the store, the server will then call `Recover()` in order to recover the persisted state. For implementations that do not support persistence, such as the provided `MemoryStore`, this call will simply return `nil` \(without error\) to indicate that no state was recovered.
+
+The `Store` is used to add/delete clients, create/lookup channels, etc...
+
+Creating/looking up a channel will return a `ChannelStore`, which points to two other interfaces, the `SubStore` and `MsgStore`. These stores, for a given channel, handle subscriptions and messages respectively.
+
+If you wish to contribute to a new store type, your implementation must include all these interfaces. For stores that allow recovery \(such as file store as opposed to memory store\), there are additional structures that have been defined and should be returned by `Recover()`.
+
+The memory and the provided file store implementations both use a generic store implementation to avoid code duplication. When writing your own store implementation, you can do the same for APIs that don't need to do more than what the generic implementation provides. You can check [MemStore](https://github.com/nats-io/nats-streaming-server/blob/master/stores/memstore.go) and [FileStore](https://github.com/nats-io/nats-streaming-server/blob/master/stores/filestore.go) implementations for more details.
+