GitBook: [master] 326 pages and 16 assets modified

nats-streaming-concepts/clustering/README.md

@@ -0,0 +1,10 @@
+# Clustering
+
+NATS Streaming Server supports clustering and data replication, implemented with the [Raft consensus algorithm](https://raft.github.io/), for the purposes of high availability.
+
+There are two ways to bootstrap a cluster: with an explicit cluster configuration or with "auto" configuration using a seed node. With the first, we provide the IDs of the nodes participating in the cluster. In this case, the participating nodes will elect a leader. With the second, we start one server as a seed node, which will elect itself as leader, and subsequent servers will automatically join the seed \(note that this also works with the explicit cluster configuration once the leader has been established\). With the second method, we need to be careful to avoid starting multiple servers as seed as this will result in a split-brain. Both of these configuration methods are shown in the sections below.
+
+It is recommended to run an odd number of servers in a cluster with a minimum of three servers to avoid split-brain scenarios. Note that if less than a majority of servers are available, the cluster cannot make progress, e.g. if two nodes go down in a cluster of three, the cluster is unavailable until at least one node comes back.
+
+Note about Channels Partitioning and Clustering. These two features are mutually exclusive. Trying to start a server with channels Partitioning and Clustering enabled will result in a startup error. Clustering requires all channels to be replicated in the cluster.
+

nats-streaming-concepts/clustering/auto-configuration.md

@@ -0,0 +1,18 @@
+# Auto Configuration
+
+We can also bootstrap a NATS Streaming cluster by starting **one server** as the seed node using the `-cluster_bootstrap` flag. This node will elect itself leader, **so it's important to avoid starting multiple servers as seed**. Once a seed node is started, other servers will automatically join the cluster. If the server is recovering, it will use the recovered cluster configuration.
+
+Here is an example of starting three servers in a cluster by starting one as the seed and letting the others automatically join:
+
+```text
+nats-streaming-server -store file -dir store-a -clustered -cluster_bootstrap -nats_server nats://localhost:4222
+
+nats-streaming-server -store file -dir store-b -clustered -nats_server nats://localhost:4222
+
+nats-streaming-server -store file -dir store-c -clustered -nats_server nats://localhost:4222
+```
+
+For a given cluster ID, if more than one server is started with `cluster_bootstrap` set to true, each server with this parameter will report the misconfiguration and exit.
+
+The very first server that bootstrapped the cluster can be restarted, however, the operator **must remove the datastores** of the other servers that were incorrectly started with the bootstrap parameter before attempting to restart them. If they are restarted with existing state, even without the `-cluster_bootstrap` parameter, they will once again start as a leader.
+

nats-streaming-concepts/clustering/configuration.md

@@ -0,0 +1,96 @@
+# Configuration
+
+We can bootstrap a NATS Streaming cluster by providing the cluster topology using the `-cluster_peers` flag. This is simply the set of node IDs participating in the cluster. Note that once a leader is established, we can start subsequent servers without providing this configuration as they will automatically join the leader. If the server is recovering, it will use the recovered cluster configuration.
+
+Here is an example of starting three servers in a cluster. For this example, we run a separate NATS server which the Streaming servers connect to.
+
+```text
+nats-streaming-server -store file -dir store-a -clustered -cluster_node_id a -cluster_peers b,c -nats_server nats://localhost:4222
+
+nats-streaming-server -store file -dir store-b -clustered -cluster_node_id b -cluster_peers a,c -nats_server nats://localhost:4222
+
+nats-streaming-server -store file -dir store-c -clustered -cluster_node_id c -cluster_peers a,b -nats_server nats://localhost:4222
+```
+
+Note that once a leader is elected, subsequent servers can be started without providing the cluster configuration. They will automatically join the cluster. Similarly, the cluster node ID does not need to be provided as one will be automatically assigned. As long as the file store is used, this ID will be recovered on restart.
+
+```text
+nats-streaming-server -store file -dir store-d -clustered -nats_server nats://localhost:4222
+```
+
+The equivalent clustering configurations can be specified in a configuration file under the `cluster` group. See the [Configuring](../../nats-streaming-server/configuring/) section for more information.
+
+Here is an example of a cluster of 3 nodes using the following configuration files. The nodes are running on `host1`, `host2` and `host3` respectively. **NOTE** If you have an existing NATS cluster and want to run NATS Streaming Cluster on top of that, see details at the end of this section.
+
+On `host1`, this configuration indicates that the server will accept client connections on port 4222. It will accept route connections on port 6222. It creates 2 routes, to `host2` and `host3` cluster port.
+
+It defines the NATS Streaming cluster name as `mycluster`, uses a store file that points to the `store` directory. The `cluster` section inside `streaming` makes the NATS Streaming server run in cluster mode. This configuration explicitly define each node id \(`a` for `host1`\) and list its peers.
+
+```text
+# NATS specific configuration
+port: 4222
+cluster {
+  listen: 0.0.0.0:6222
+  routes: ["nats://host2:6222", "nats://host3:6222"]
+}
+
+# NATS Streaming specific configuration
+streaming {
+  id: mycluster
+  store: file
+  dir: store
+  cluster {
+    node_id: "a"
+    peers: ["b", "c"]
+  }
+}
+```
+
+Below is the configuration for the server running on `host2`. Notice how the routes are now to `host1` and `host3`. The other thing that changed is the node id that is set to `b` and peers are updated accordingly to `a` and `c`.
+
+Note that the `dir` configuration is also `store` but these are local directories and do not \(actually must not\) be shared. Each node will have its own copy of the datastore. You could have each configuration have a different value for `dir` if desired.
+
+```text
+# NATS specific configuration
+port: 4222
+cluster {
+  listen: 0.0.0.0:6222
+  routes: ["nats://host1:6222", "nats://host3:6222"]
+}
+
+# NATS Streaming specific configuration
+streaming {
+  id: mycluster
+  store: file
+  dir: store
+  cluster {
+    node_id: "b"
+    peers: ["a", "c"]
+  }
+}
+```
+
+As you would expect, for `host3`, the routes are now to `host1` and `host2` and the node id is `c` while its peers are `a` and `b`.
+
+```text
+# NATS specific configuration
+port: 4222
+cluster {
+  listen: 0.0.0.0:6222
+  routes: ["nats://host1:6222", "nats://host2:6222"]
+}
+
+# NATS Streaming specific configuration
+streaming {
+  id: mycluster
+  store: file
+  dir: store
+  cluster {
+    node_id: "c"
+    peers: ["a", "b"]
+  }
+}
+```
+
+In the example above, the configuration assumes no existing NATS Cluster and therefore configure the NATS routes between each node. Should you want to use an existing NATS cluster, do not include the "NATS specific configuration" section, instead, add `nats_server_url` in the `streaming` section to point to the NATS server you want.
+

nats-streaming-concepts/clustering/containers.md

@@ -0,0 +1,12 @@
+# Containers
+
+When running the docker image of NATS Streaming Server, you will want to specify a mounted volume so that the data can be recovered. Your `-dir` parameter then points to a directory inside that mounted volume. However, after a restart you may get a failure with a message similar to this:
+
+```text
+[FTL] STREAM: Failed to start: streaming state was recovered but cluster log path "mycluster/a" is empty
+```
+
+This is because the server recovered the streaming state \(as pointed by `-dir` and located in the mounted volume\), but did not recover the RAFT specific state that is by default stored in a directory named after your cluster id, relative to the current directory starting the executable. In the context of a container, this data will be lost after the container is stopped.
+
+In order to avoid this issue, you need to specify the `-cluster_log_path` and ensure that it points to the mounted volume so that the RAFT state can be recovered along with the Streaming state.
+

nats-streaming-concepts/clustering/supported-stores.md

@@ -0,0 +1,10 @@
+# Supported Stores
+
+In order to run NATS Streaming Server in clustered mode, you need to specify a persistent store. At this time you have the choice between `FILE` and `SQL`
+
+The NATS Streaming stores server meta information, messages and subscriptions to the storage you configure using the `--store` option.
+
+However, in clustered mode, we use RAFT for leader election. The raft layer uses its own stores which are currently necessarily file based. The location of the RAFT stores defaults to the current directory under a sub-directory named after the cluster ID, or you can configure it using `--cluster_log_path`.
+
+This means that even if you select a SQL Store, there will still be a need for storing data on the file system.
+