taigrr/nats.docs
1
0
Fork 0
mirror of https://github.com/taigrr/nats.docs synced 2025-01-18 04:03:23 -08:00
Code Issues Projects Releases Wiki Activity

Update NATS Streaming documentation

Browse Source 
Brings missing parts that were in the README.md
Added some details in the TLS section based on issue https://github.com/nats-io/nats-streaming-server/issues/919

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
This commit is contained in:
Ivan Kozlovic 2019-09-03 18:32:56 -06:00
parent e6da4dc7d4
commit 4ef08006bc
13 changed files with 793 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
SUMMARY.md
View File

@ -152,6 +152,8 @@
* [Shared State](nats_streaming/fault-tolerance/shared-state.md)
* [Failover](nats_streaming/fault-tolerance/failover.md)
* [Partitioning](nats_streaming/partitioning.md)
* [Monitoring](nats_streaming/monitoring/monitoring.md)
* [Endpoints](nats_streaming/monitoring/endpoints.md)
## Developing With NATS Streaming
@ -166,16 +168,28 @@
## NATS Streaming Server
* [Important Changes](nats_streaming/gettingstarted/changes.md)
* [Installing](nats_streaming/gettingstarted/install.md)
* [Running](nats_streaming/gettingstarted/run.md)
* [Configuring](nats_streaming/gettingstarted/configuring.md)
* [Securing](nats_streaming/gettingstarted/tls.md)
* [Configuring](nats_streaming/configuring/configuring.md)
* [Command line arguments](nats_streaming/configuring/cmdline.md)
* [Configuration file](nats_streaming/configuring/cfgfile.md)
* [Store Limits](nats_streaming/configuring/storelimits.md)
* [Limits inheritance](nats_streaming/configuring/storelimits.md#limits-inheritance)
* [Persistence](nats_streaming/configuring/persistence.md)
* [File Store](nats_streaming/configuring/filestore.md)
* [Options](nats_streaming/configuring/filestore.md#options)
* [Recovery Errors](nats_streaming/configuring/filestore.md#recovery-errors)
* [SQL Store](nats_streaming/configuring/sqlstore.md)
* [Read and Write Timeouts](nats_streaming/configuring/sqlstore.md#read-and-write-timeouts)
* [Options](nats_streaming/configuring/sqlstore.md#options)
* [Securing](nats_streaming/configuring/tls.md)
* [Authenticating Users](nats_streaming/configuring/tls.md#authenticating-users)
* [TLS](nats_streaming/configuring/tls.md#using-tls)
* [Process Signaling](nats_streaming/gettingstarted/process-signaling.md)
* [Windows Service](nats_streaming/gettingstarted/windows-service.md)
* [Embedding NATS Streaming Server](developer/streaming/embedding.md)
* [Docker Swarm](nats_streaming/swarm.md)
* [Monitoring](nats_streaming/monitoring/monitoring.md)
* [Endpoints](nats_streaming/monitoring/endpoints.md)
## NATS Protocol

2
nats_streaming/clustering/configuration.md
View File

@ -18,7 +18,7 @@ Note that once a leader is elected, subsequent servers can be started without pr
nats-streaming-server -store file -dir store-d -clustered -nats_server nats://localhost:4222
```
The equivalent clustering configurations can be specified in a configuration file under the `cluster` group. See the [README Configuring](https://github.com/nats-io/nats-streaming-server#configuring) section for more information.
The equivalent clustering configurations can be specified in a configuration file under the `cluster` group. See the [Configuring](../configuring/configuring.md) section for more information.
Here is an example of a cluster of 3 nodes using the following configuration files. The nodes are running on `host1`, `host2` and `host3` respectively.
**NOTE** If you have an existing NATS cluster and want to run NATS Streaming Cluster on top of that, see details at the end of this section.

161
nats_streaming/configuring/cfgfile.md Normal file
View File

@ -0,0 +1,161 @@
# Configuration file
You can use a configuration file to configure the options specific to the NATS Streaming server.
Use the `-sc` or `-stan_config` command line parameter to specify the file to use.
For the embedded NATS Server, you can use another configuration file and pass it to the Streaming server using `-c` or `--config` command line parameters.
Since most options do not overlap, it is possible to combine all options into a single file and specify this file using either the `-sc` or `-c` command line parameter.
However, the option named `tls` is common to NATS Server and NATS Streaming Server. If you plan to use a single configuration file and configure TLS,
you should have all the streaming configuration included in a `streaming` map. This is actually a good practice regardless if you use TLS
or not, to protect against possible addition of new options in NATS Server that would conflict with the names of NATS Streaming options.
For instance, you could use a single configuration file with such content:
```
# Some NATS Server TLS Configuration
listen: localhost:5222
tls: {
cert_file: "/path/to/server/cert_file"
key_file: "/path/to/server/key_file"
verify: true
timeout: 2
}
# NATS Streaming Configuration
streaming: {
cluster_id: my_cluster
tls: {
client_cert: "/path/to/client/cert_file"
client_key: "/path/to/client/key_file"
}
}
```
However, if you want to avoid any possible conflict, simply use two different configuration files!
Note the order in which options are applied during the start of a NATS Streaming server:
1. Start with some reasonable default options.
2. If a configuration file is specified, override those options
with all options defined in the file. This include options that are defined
but have no value specified. In this case, the zero value for the type of the
option will be used.
3. Any command line parameter override all of the previous set options.
In general the configuration parameters are the same as the command line arguments. Below is the list of NATS Streaming parameters:
| Parameter | Meaning | Possible values | Usage example |
|:----|:----|:----|:----|
| cluster_id | Cluster name | String, underscore possible | `cluster_id: "my_cluster_name"` |
| discover_prefix | Subject prefix for server discovery by clients | NATS Subject | `discover_prefix: "_STAN.Discovery"` |
| store | Store type | `memory`, `file` or `sql` | `store: "file"` |
| dir | When using a file store, this is the root directory | File path | `dir: "/path/to/storage` |
| sd | Enable debug logging | `true` or `false` | `sd: true` |
| sv | Enable trace logging | `true` or `false` | `sv: true` |
| nats_server_url | If specified, connects to an external NATS Server, otherwise starts an embedded one | NATS URL | `nats_server_url: "nats://localhost:4222"` |
| secure | If true, creates a TLS connection to the server but without the need to use TLS configuration (no NATS Server certificate verification) | `true` or `false` | `secure: true` |
| tls | TLS Configuration | Map: `tls: { ... }` | **See details below** |
| store_limits | Store Limits | Map: `store_limits: { ... }` | **See details below** |
| file_options | File Store specific options | Map: `file_options: { ... }` | **See details below** |
| sql_options | SQL Store specific options | Map: `sql_options: { ... }` | **See details below** |
| hb_interval | Interval at which the server sends an heartbeat to a client | Duration | `hb_interval: "10s"` |
| hb_timeout | How long the server waits for a heartbeat response from the client before considering it a failed heartbeat | Duration | `hb_timeout: "10s"` |
| hb_fail_count | Count of failed heartbeats before server closes the client connection. The actual total wait is: (fail count + 1) * (hb interval + hb timeout) | Number | `hb_fail_count: 2` |
| ft_group | In Fault Tolerance mode, you can start a group of streaming servers with only one server being active while others are running in standby mode. This is the name of this FT group | String | `ft_group: "my_ft_group"` |
| partitioning | If set to true, a list of channels must be defined in store_limits/channels section. This section then serves two purposes, overriding limits for a given channel or adding it to the partition | `true` or `false` | `partitioning: true` |
| cluster | Cluster Configuration | Map: `cluster: { ... }` | **See details below** |
| encrypt | Specify if server should encrypt messages (only the payload) when storing them | `true` or `false` | `encrypt: true` |
| encryption_cipher | Cipher to use for encryption. Currently support AES and CHAHA (ChaChaPoly). Defaults to AES | `AES` or `CHACHA` | `encryption_cipher: "AES"` |
| encryption_key | Encryption key. It is recommended to specify the key through the `NATS_STREAMING_ENCRYPTION_KEY` environment variable instead | String | `encryption_key: "mykey"` |
TLS Configuration:
Note that the Streaming server uses a connection to a NATS Server, and so the NATS Streaming TLS Configuration
is in fact a client-side TLS configuration.
| Parameter | Meaning | Possible values | Usage example |
|:----|:----|:----|:----|
| client_cert | Client key for the streaming server | File path | `client_cert: "/path/to/client/cert_file"` |
| client_key | Client certificate for the streaming server | File path | `client_key: "/path/to/client/key_file"` |
| client_ca | Client certificate CA for the streaming server | File path | `client_ca: "/path/to/client/ca_file"` |
Store Limits Configuration:
| Parameter | Meaning | Possible values | Usage example |
|:----|:----|:----|:----|
| max_channels | Maximum number of channels, 0 means unlimited | Number >= 0 | `max_channels: 100` |
| max_subs | Maximum number of subscriptions per channel, 0 means unlimited | Number >= 0 | `max_subs: 100` |
| max_msgs | Maximum number of messages per channel, 0 means unlimited | Number >= 0 | `max_msgs: 10000` |
| max_bytes | Total size of messages per channel, 0 means unlimited | Number >= 0 | `max_bytes: 1GB` |
| max_age | How long messages can stay in the log | Duration | `max_age: "24h"` |
| max_inactivity | How long without any subscription and any new message before a channel can be automatically deleted | Duration | `max_inactivity: "24h"` |
| channels | A map of channel names with specific limits | Map: `channels: { ... }` | **See details below** |
The `channels` section is a map with the key being the channel name. For instance:
```
channels: {
"foo": {
max_msgs: 100
}
}
```
For a given channel, the possible parameters are:
| Parameter | Meaning | Possible values | Usage example |
|:----|:----|:----|:----|
| max_subs | Maximum number of subscriptions per channel, 0 means unlimited | Number >= 0 | `max_subs: 100` |
| max_msgs | Maximum number of messages per channel, 0 means unlimited | Number >= 0 | `max_msgs: 10000` |
| max_bytes | Total size of messages per channel, 0 means unlimited | Bytes | `max_bytes: 1GB` |
| max_age | How long messages can stay in the log | Duration | `max_age: "24h"` |
| max_inactivity | How long without any subscription and any new message before a channel can be automatically deleted | Duration | `max_inactivity: "24h"` |
File Options Configuration:
| Parameter | Meaning | Possible values | Usage example |
|:----|:----|:----|:----|
| compact | Enable/disable file compaction. Only some of the files (`clients.dat` and `subs.dat`) are subject to compaction | `true` or `false` | `compact: true` |
| compact_fragmentation | Compaction threshold (in percentage) | Number >= 0 | `compact_fragmentation: 50` |
| compact_interval | Minimum interval between attempts to compact files | Expressed in seconds | `compact_interval: 300` |
| compact_min_size | Minimum size of a file before compaction can be attempted | Bytes | `compact_min_size: 1GB` |
| buffer_size | Size of buffers that can be used to buffer write operations | Bytes | `buffer_size: 2MB` |
| crc | Define if CRC of records should be computed on reads | `true` or `false` | `crc: true` |
| crc_poly | You can select the CRC polynomial. Note that changing the value after records have been persisted would result in server failing to start complaining about data corruption | Number >= 0 | `crc_poly: 3988292384` |
| sync_on_flush | Define if server should perform "file sync" operations during a flush | `true` or `false` | `sync_on_flush: true` |
| slice_max_msgs | Define the file slice maximum number of messages. If set to 0 and a channel count limit is set, then the server will set a slice count limit automatically | Number >= 0 | `slice_max_msgs: 10000` |
| slice_max_bytes | Define the file slice maximum size (including the size of index file). If set to 0 and a channel size limit is set, then the server will set a slice bytes limit automatically | Bytes | `slice_max_bytes: 64MB` |
| slice_max_age | Define the period of time covered by a file slice, starting at when the first message is stored. If set to 0 and a channel age limit is set, then the server will set a slice age limit automatically | Duration | `slice_max_age: "24h"` |
| slice_archive_script | Define the location and name of a script to be invoked when the server discards a file slice due to limits. The script is invoked with the name of the channel, the name of data and index files. It is the responsibility of the script to then remove the unused files | File path | `slice_archive_script: "/home/nats-streaming/archive/script.sh"` |
| file_descriptors_limit | Channels translate to sub-directories under the file store's root directory. Each channel needs several files to maintain the state so the need for file descriptors increase with the number of channels. This option instructs the store to limit the concurrent use of file descriptors. Note that this is a soft limit and there may be cases when the store will use more than this number. A value of 0 means no limit. Setting a limit will probably have a performance impact | Number >= 0 | `file_descriptors_limit: 100` |
| parallel_recovery | When the server starts, the recovery of channels (directories) is done sequentially. However, when using SSDs, it may be worth setting this value to something higher than 1 to perform channels recovery in parallel | Number >= 1 | `parallel_recovery: 4` |
| read_buffer_size | Size of buffers used to read ahead from message stores. This can significantly speed up sending messages to consumers after messages have been published. Default is 2MB. Set to 0 to disable | Bytes | `read_buffer_size: 2MB` |
| auto_sync | Interval at which the store should be automatically flushed and sync'ed on disk. Default is every minute. Set to <=0 to disable | Duration | `auto_sync: "2m"` |
Cluster Configuration:
| Parameter | Meaning | Possible values | Usage example |
|:----|:----|:----|:----|
| node_id | ID of the node within the cluster if there is no stored ID | String (no whitespace) | `node_id: "node-a"` |
| bootstrap | Bootstrap the cluster if there is no existing state by electing self as leader | `true` or `false` | `bootstrap: true` |
| peers | List of cluster peer node IDs to bootstrap cluster state | List of node IDs | `peers: ["node-b", "node-c"]` |
| log_path | Directory to store log replication data | File path | `log_path: "/path/to/storage"` |
| log_cache_size | Number of log entries to cache in memory to reduce disk IO | Number >= 0 | `log_cache_size: 1024` |
| log_snapshots | Number of log snapshots to retain | Number >= 0 | `log_snapshots: 1` |
| trailing_logs | Number of log entries to leave after a snapshot and compaction | Number >= 0 | `trailing_logs: 256` |
| sync | Do a file sync after every write to the replication log and message store | `true` or `false` | `sync: true` |
| raft_logging | Enable logging from the Raft library (disabled by default) | `true` or `false` | `raft_logging: true` |
| raft_heartbeat_timeout | Specifies the time in follower state without a leader before attempting an election | Duration | `raft_heartbeat_timeout: "2s"` |
| raft_election_timeout | Specifies the time in candidate state without a leader before attempting an election | Duration | `raft_election_timeout: "2s"` |
| raft_lease_timeout | Specifies how long a leader waits without being able to contact a quorum of nodes before stepping down as leader | Duration | `raft_lease_timeout: "1s"` |
| raft_commit_timeout | Specifies the time without an Apply() operation before sending an heartbeat to ensure timely commit. Due to random staggering, may be delayed as much as 2x this value | Duration | `raft_commit_timeout: "100ms"` |
SQL Options Configuration:
| Parameter | Meaning | Possible values | Usage example |
|:----|:----|:----|:----|
| driver | Name of the SQL driver to use | `mysql` or `postgres` | `driver: "mysql"` |
| source | How to connect to the database. This is driver specific | String | `source: "ivan:pwd@/nss_db"` |
| no_caching | Enable/Disable caching for messages and subscriptions operations. The default is `false`, which means that caching is enabled | `true` or `false` | `no_caching: false` |
| max_open_conns | Maximum number of opened connections to the database. Value <= 0 means no limit. The default is 0 (unlimited) | Number | `max_open_conns: 5` |

125
nats_streaming/configuring/cmdline.md Normal file
View File

@ -0,0 +1,125 @@
# Command line arguments
The NATS Streaming Server accepts command line arguments to control its behavior. There is a set of parameters specific to the NATS Streaming Server and some to the embedded NATS Server.
***Note about parameters types***
| Type | Remark |
|:----|:----|
|`<bool>`| For booleans, either simply specify the parameter with value to enable (e.g `-SD`), or specify `=false` to disable|
|`<size>` | You can specify as a number `1024` or as a size `1KB`|
|`<duration>` | Values must be expressed in the form `_h_m_s`, such as `1h` or `20s` or `1h30m`, or `1.5h`, etc...|
```
Usage: nats-streaming-server [options]
Streaming Server Options:
-cid, --cluster_id <string> Cluster ID (default: test-cluster)
-st, --store <string> Store type: MEMORY|FILE|SQL (default: MEMORY)
--dir <string> For FILE store type, this is the root directory
-mc, --max_channels <int> Max number of channels (0 for unlimited)
-msu, --max_subs <int> Max number of subscriptions per channel (0 for unlimited)
-mm, --max_msgs <int> Max number of messages per channel (0 for unlimited)
-mb, --max_bytes <size> Max messages total size per channel (0 for unlimited)
-ma, --max_age <duration> Max duration a message can be stored ("0s" for unlimited)
-mi, --max_inactivity <duration> Max inactivity (no new message, no subscription) after which a channel can be garbage collected (0 for unlimited)
-ns, --nats_server <string> Connect to this external NATS Server URL (embedded otherwise)
-sc, --stan_config <string> Streaming server configuration file
-hbi, --hb_interval <duration> Interval at which server sends heartbeat to a client
-hbt, --hb_timeout <duration> How long server waits for a heartbeat response
-hbf, --hb_fail_count <int> Number of failed heartbeats before server closes the client connection
--ft_group <string> Name of the FT Group. A group can be 2 or more servers with a single active server and all sharing the same datastore
-sl, --signal <signal>[=<pid>] Send signal to nats-streaming-server process (stop, quit, reopen)
--encrypt <bool> Specify if server should use encryption at rest
--encryption_cipher <string> Cipher to use for encryption. Currently support AES and CHAHA (ChaChaPoly). Defaults to AES
--encryption_key <sting> Encryption Key. It is recommended to specify it through the NATS_STREAMING_ENCRYPTION_KEY environment variable instead
Streaming Server Clustering Options:
--clustered <bool> Run the server in a clustered configuration (default: false)
--cluster_node_id <string> ID of the node within the cluster if there is no stored ID (default: random UUID)
--cluster_bootstrap <bool> Bootstrap the cluster if there is no existing state by electing self as leader (default: false)
--cluster_peers <string, ...> Comma separated list of cluster peer node IDs to bootstrap cluster state
--cluster_log_path <string> Directory to store log replication data
--cluster_log_cache_size <int> Number of log entries to cache in memory to reduce disk IO (default: 512)
--cluster_log_snapshots <int> Number of log snapshots to retain (default: 2)
--cluster_trailing_logs <int> Number of log entries to leave after a snapshot and compaction
--cluster_sync <bool> Do a file sync after every write to the replication log and message store
--cluster_raft_logging <bool> Enable logging from the Raft library (disabled by default)
Streaming Server File Store Options:
--file_compact_enabled <bool> Enable file compaction
--file_compact_frag <int> File fragmentation threshold for compaction
--file_compact_interval <int> Minimum interval (in seconds) between file compactions
--file_compact_min_size <size> Minimum file size for compaction
--file_buffer_size <size> File buffer size (in bytes)
--file_crc <bool> Enable file CRC-32 checksum
--file_crc_poly <int> Polynomial used to make the table used for CRC-32 checksum
--file_sync <bool> Enable File.Sync on Flush
--file_slice_max_msgs <int> Maximum number of messages per file slice (subject to channel limits)
--file_slice_max_bytes <size> Maximum file slice size - including index file (subject to channel limits)
--file_slice_max_age <duration> Maximum file slice duration starting when the first message is stored (subject to channel limits)
--file_slice_archive_script <string> Path to script to use if you want to archive a file slice being removed
--file_fds_limit <int> Store will try to use no more file descriptors than this given limit
--file_parallel_recovery <int> On startup, number of channels that can be recovered in parallel
--file_truncate_bad_eof <bool> Truncate files for which there is an unexpected EOF on recovery, dataloss may occur
--file_read_buffer_size <size> Size of messages read ahead buffer (0 to disable)
--file_auto_sync <duration> Interval at which the store should be automatically flushed and sync'ed on disk (<= 0 to disable)
Streaming Server SQL Store Options:
--sql_driver <string> Name of the SQL Driver ("mysql" or "postgres")
--sql_source <string> Datasource used when opening an SQL connection to the database
--sql_no_caching <bool> Enable/Disable caching for improved performance
--sql_max_open_conns <int> Maximum number of opened connections to the database
Streaming Server TLS Options:
-secure <bool> Use a TLS connection to the NATS server without
verification; weaker than specifying certificates.
-tls_client_key <string> Client key for the streaming server
-tls_client_cert <string> Client certificate for the streaming server
-tls_client_cacert <string> Client certificate CA for the streaming server
Streaming Server Logging Options:
-SD, --stan_debug=<bool> Enable STAN debugging output
-SV, --stan_trace=<bool> Trace the raw STAN protocol
-SDV Debug and trace STAN
--syslog_name On Windows, when running several servers as a service, use this name for the event source
(See additional NATS logging options below)
Embedded NATS Server Options:
-a, --addr <string> Bind to host address (default: 0.0.0.0)
-p, --port <int> Use port for clients (default: 4222)
-P, --pid <string> File to store PID
-m, --http_port <int> Use port for http monitoring
-ms,--https_port <int> Use port for https monitoring
-c, --config <string> Configuration file
Logging Options:
-l, --log <string> File to redirect log output
-T, --logtime=<bool> Timestamp log entries (default: true)
-s, --syslog <string> Enable syslog as log method
-r, --remote_syslog <string> Syslog server addr (udp://localhost:514)
-D, --debug=<bool> Enable debugging output
-V, --trace=<bool> Trace the raw protocol
-DV Debug and trace
Authorization Options:
--user <string> User required for connections
--pass <string> Password required for connections
--auth <string> Authorization token required for connections
TLS Options:
--tls=<bool> Enable TLS, do not verify clients (default: false)
--tlscert <string> Server certificate file
--tlskey <string> Private key for server certificate
--tlsverify=<bool> Enable TLS, verify client certificates
--tlscacert <string> Client certificate CA for verification
NATS Clustering Options:
--routes <string, ...> Routes to solicit and connect
--cluster <string> Cluster URL for solicited routes
Common Options:
-h, --help Show this message
-v, --version Show version
--help_tls TLS help.
```

3
nats_streaming/configuring/configuring.md Normal file
View File

@ -0,0 +1,3 @@
## Configuration and Administration
NATS Streaming provides a rich set of commands and parameters to configure all aspects of the server.

112
nats_streaming/configuring/filestore.md Normal file
View File

@ -0,0 +1,112 @@
# File Store
For a higher level of message delivery, the server should be configured with a file store. NATS Streaming Server comes with a basic file store implementation. Various file store implementations may be added in the future.
To start the server with a file store, you need to provide two parameters:
```sh
nats-streaming-server -store file -dir datastore
```
The parameter `-store` indicates what type of store to use, in this case `file`. The other (`-dir`) indicates in which directory the state should be stored.
The first time the server is started, it will create two files in this directory, one containing some server related information (`server.dat`) another to record clients information (`clients.dat`).
When a streaming client connects, it uses a client identification, which the server registers in this file. When the client disconnects, the client is cleared from this file.
When the client publishes or subscribe to a new subject (also called channel), the server creates a sub-directory whose name is the subject. For instance, if the client subscribes to `foo`, and assuming that you started the server with `-dir datastore`, then you will find a directory called `datastore/foo`. In this directory you will find several files: one to record subscriptions information (`subs.dat`), and a series of files that logs the messages `msgs.1.dat`, etc...
The number of sub-directories, which again correspond to channels, can be limited by the configuration parameter `-max_channels`. When the limit is reached, any new subscription or message published on a new channel will produce an error.
On a given channel, the number of subscriptions can also be limited with the configuration parameter `-max_subs`. A client that tries to create a subscription on a given channel (subject) for which the limit is reached will receive an error.
Finally, the number of stored messages for a given channel can also be limited with the parameter `-max_msgs` and/or `-max_bytes`. However, for messages, the client does not get an error when the limit is reached. The oldest messages are discarded to make room for the new messages.
## Options
As described in the [Configuring](nats_streaming/configuring/configuring.md) section, there are several options that you can use to configure a file store.
Regardless of channel limits, you can configure message logs to be split in individual files (called file slices). You can configure
those slices by number of messages it can contain (`--file_slice_max_msgs`), the size of the file - including the corresponding index file
(`--file_slice_max_bytes`), or the period of time that a file slice should cover - starting at the time the first message is stored in
that slice (`--file_slice_max_age`). The default file store options are defined such that only the slice size is configured to 64MB.
Note: If you don't configure any slice limit but you do configure channel limits, then the server will automatically
set some limits for file slices.
When messages accumulate in a channel, and limits are reached, older messages are removed. When the first file slice
becomes empty, the server removes this file slice (and corresponding index file).
However, if you specify a script (`--file_slice_archive_script`), then the server will rename the slice files (data and index)
with a `.bak` extension and invoke the script with the channel name, data and index file names.<br>
The files are left in the channel's directory and therefore it is the script responsibility to delete those files when done.
At any rate, those files will not be recovered on a server restart, but having lots of unused files in the directory may slow
down the server restart.
For instance, suppose the server is about to delete file slice `datastore/foo/msgs.1.dat` (and `datastore/foo/msgs.1.idx`),
and you have configured the script `/home/nats-streaming/archive_script.sh`. The server will invoke:
``` bash
/home/nats-streaming/archive_script.sh foo datastore/foo/msgs.1.dat.bak datastore/foo/msgs.2.idx.bak
```
Notice how the files have been renamed with the `.bak` extension so that they are not going to be recovered if
the script leave those files in place.
As previously described, each channel corresponds to a sub-directory that contains several files. It means that the need
for file descriptors increase with the number of channels. In order to scale to ten or hundred thousands of channels,
the option `fds_limit` (or command line parameter `--file_fds_limit`) may be considered to limit the total use of file descriptors.
Note that this is a soft limit. It is possible for the store to use more file descriptors than the given limit if the
number of concurrent read/writes to different channels is more than the said limit. It is also understood that this
may affect performance since files may need to be closed/re-opened as needed.
## Recovery Errors
We have added the ability for the server to truncate any file that may otherwise report an `unexpected EOF`
error during the recovery process.
Since dataloss is likely to occur, the default behavior for the server on startup is to report recovery error and stop.
It will now print the content of the first corrupted record before exiting.
With the `-file_truncate_bad_eof` parameter, the server will still print those bad records but truncate each file at
the position of the first corrupted record in order to successfully start.
To prevent the use of this parameter as the default value, this option is not available in the configuration file.
Moreover, the server will fail to start if started more than once with that parameter.<br>
This flag may help recover from a store failure, but since data may be lost in that process, we think that the
operator needs to be aware and make an informed decision.
Note that this flag will not help with file corruption due to bad CRC for instance. You have the option to disable
CRC on recovery with the `-file_crc=false` option.
Let's review the impact and suggested steps for each of the server's corrupted files:
* `server.dat`: This file contains meta data and NATS subjects used to communicate with client applications. If
a corruption is reported with this file, we would suggest that you stop all your clients, stop the server, remove
this file, restart the server. This will create a new `server.dat` file, but will not attempt to recover the
rest of the channels because the server assumes that there is no state. So you should stop and restart the
server once more. Then, you can restart all your clients.
* `clients.dat`: This contains information about client connections. If the file is truncated to move past
an `unexpected EOF` error, this can result in no issue at all, or in client connections not being recovered,
which means that the server will not know about possible running clients, and therefore it will not try
to deliver any message to those non recovered clients, or reject incoming published messages from those clients.
It is also possible that the server recovers a client connection that was actually closed. In this case, the
server may attempt to deliver or redeliver messages unnecessarily.
* `subs.dat`: This is a channel's subscriptions file (under the channel's directory). If this file is truncated
and some records are lost, it may result in no issue at all, or in client applications not receiving their messages
since the server will not know about them. It is also possible that acknowledged messages get redelivered
(since their ack may have been lost).
* `msgs.<n>.dat`: This is a channel's message log (several per channel). If one of those files is truncated, then
message loss occurs. With the `unexpected EOF` errors, it is likely that only the last "file slice" of a channel will
be affected. Nevertheless, if a lower sequence file slice is truncated, then gaps in message sequence will occur.
So it would be possible for a channel to have now messages 1..100, 110..300 for instance, with messages 101 to 109
missing. Again, this is unlikely since we expect the unexpected end-of-file errors to occur on the last slice.
For *Clustered* mode, this flag would work only for the NATS Streaming specific store files. As you know, NATS
Streaming uses RAFT for consensus, and RAFT uses its own logs. You could try the option if the server reports
`unexpected EOF` errors for NATS Streaming file stores, however, you may want to simply delete all NATS Streaming
and RAFT stores for the failed node and restart it. By design, the other nodes in the cluster have replicated the
data, so this node will become a follower and catchup with the rest of the cluster, getting the data from the
current leader and recreating its local stores.

3
nats_streaming/configuring/persistence.md Normal file
View File

@ -0,0 +1,3 @@
# Persistence
By default, the NATS Streaming Server stores its state in memory, which means that if the streaming server is stopped, all state is lost. Still, this level of persistence allows applications to stop and later resume the stream of messages, and protect against applications disconnect (network or applications crash).

86
nats_streaming/configuring/sqlstore.md Normal file
View File

@ -0,0 +1,86 @@
# SQL Store
Using a SQL Database for persistence is another option.
In order to do so, `-store` simply needs to be set to `sql` and `-sql_driver` set to `mysql` or `postgres`
(the two drivers supported at the moment). The parameter `-sql_source` is driver specific, but generally
contains the information required to connect to a specific database on the given SQL database server.
Note that the NATS Streaming Server does not need root privileges to connect to the database since it does not create
the database, tables or indexes. This has to be done by the Database Administrator.
We provide 2 files (`scripts/mysql.db.sql` and `scripts/postgres.db.sql`) that can be used to create the tables and indexes to the
database of your choice. However, administrators are free to configure and optimize the database as long as the name of tables
and columns are preserved, since the NATS Streaming Server is going to issue SQL statements based on those.
Here is an example of creating an user `nss` with password `password` for the MySQL database:
```
mysql -u root -e "CREATE USER 'nss'@'localhost' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON *.* TO 'nss'@'localhost'; CREATE DATABASE nss_db;"
```
The above has gives all permissions to user `nss`. Once this user is created, we can then create the tables using this user
and selecting the `nss_db` database. We then execute all the SQL statements creating the tables from the sql file that
is provided in this repo:
```
mysql -u nss -p -D nss_db -e "$(cat ./scripts/mysql.db.sql)"
```
## Example - Postgres
Run a local dockerized instance of postgres if you do not already have one:
```
ID=$(docker run -d -e POSTGRES_PASSWORD=password -p 5432:5432 postgres)
```
[Optional] Drop any previous tables to clear data from previous sessions:
```
cat scripts/drop_postgres.db.sql | docker exec -i $ID psql -h 127.0.1.1 -U postgres
```
Run the appropriate database migrations for Postgres:
```
cat scripts/postgres.db.sql | docker exec -i $ID psql -h 127.0.1.1 -U postgres
```
Run the nats streaming server with postgres at the sql_source:
```
DOCKER_BRIDGE_IP=$(docker inspect --format '{{(index .IPAM.Config 0).Gateway}}' bridge) docker run -d --name nats-streaming -p 4222:4222 -p 8222:32768 nats-streaming-local -SDV --store sql --sql_driver postgres --sql_source="user=postgres password=postgres host=$DOCKER_BRIDGE_IP port=5432 sslmode=disable"
```
## Read and Write Timeouts
Sometimes, it is possible that a DB connection between the streaming server and the DB server is stale but the connection is not dropped.
This would cause the server to block while trying to store or lookup a message, or any other operation involving the database.
Because of internal locking in the store implementation, this could cause the server to seemingly be unresponsive.
To mitigate that, you can pass `readTimeout` and `writeTimeout` options to the `sql_source` when starting the server. The MySQL driver had always had those options, but we have extended the Postgres driver that we use to provide those options in NATS Streaming `v0.16.0`.
You pass the values as a duration, for instance `5s` for 5 seconds.
Here is what a `sql_source` would look like for `MySQL` driver:
```
nats-streaming-server -store sql -sql_driver mysql -sql_source "nss:password@/nss_db?readTimeout=5s&writeTimeout=5s" ..
```
Or, for `Postgres` driver:
```
nats-streaming-server -store sql -sql_driver postgres -sql_source "dbname=nss_db readTimeout=5s writeTimeout=5s sslmode=disable" ..
```
Be careful to not make those values too small otherwise you could cause unwanted failures.
## Options
Aside from the driver and datasource, the available options are the maximum number of opened connections to the database (`max_open_conns`)
that you may need to set to avoid errors due to `too many opened files`.
The other option is `no_caching` which is a boolean that enables/disables caching. By default caching is enabled. It means
that some operations are buffered in memory before being sent to the database. For storing messages, this still offers the
guarantee that if a producer gets an OK ack back, the message will be successfully persisted in the database.
For subscriptions, the optimization may lead to messages possibly redelivered if the server were to be restarted before
some of the operations were "flushed" to the database. The performance improvement is significant to justify the risk
of getting redelivered messages (which is always possible with NATS Streaming regardless of this option). Still,
if you want to ensure that each operation is immediately committed to the database, you should set `no_caching` to true.

149
nats_streaming/configuring/storelimits.md Normal file
View File

@ -0,0 +1,149 @@
# Store Limits
The `store_limits` section in the configuration file (or the command line parameters
`-mc`, `-mm`, etc..) allow you to configure the global limits.
These limits somewhat offer some upper bound on the size of the storage. By multiplying
the limits per channel with the maximum number of channels, you will get a total limit.
It is not the case, though, if you override limits of some channels. Indeed, it is possible
to define specific limits per channel. Here is how:
```
...
store_limits: {
# Override some global limits
max_channels: 10
max_msgs: 10000
max_bytes: 10MB
max_age: "1h"
# Per channel configuration.
# Can be channels, channels_limits, per_channel, per_channel_limits or ChannelsLimits
channels: {
"foo": {
# Possible options are the same than in the store_limits section, except
# for max_channels. Not all limits need to be specified.
max_msgs: 300
max_subs: 50
}
"bar": {
max_msgs:50
max_bytes:1KB
}
"baz": {
# Set to 0 for ignored (or unlimited)
max_msgs: 0
# Override with a lower limit
max_bytes: 1MB
# Override with a higher limit
max_age: "2h"
}
# When using partitioning, channels need to be listed.
# They don't have to override any limit.
"bozo": {}
# Wildcards are possible in configuration. This says that any channel
# that will start with "foo" but with at least 2 tokens, will be
# able to store 400 messages. Other limits are inherited from global.
"foo.>": {
max_msgs:400
}
# This one says that if the channel name starts with "foo.bar" but has
# at least one more token, the sever will hold it for 2 hours instead
# of one. The max number of messages is inherited from "foo.>", so the
# limit will be 400. All other limits are inherited from global.
"foo.bar.>": {
max_age: "2h"
}
# Delete channels with this prefix once they don't have any
# subscription and no new message for more than 1 hour
"temp.>": {
max_inactivity: "1h"
}
}
}
...
```
Note that the number of defined channels cannot be greater than the stores' maximum number
of channels. ***This is true only for channels without wildcards.***
Channels limits can override global limits by being either higher, lower or even set to
unlimited.
***An unlimited value applies to the specified limit, not to the whole channel***
That is, in the configuration above, `baz` has the maximum number of messages set
to 0, which means ignored or unlimited. Yet, other limits such as max bytes, max age
and max subscriptions (inherited in this case) still apply. What that means is that
the store will not check the number of messages but still check the other limits.
For a truly unlimited channel *all* limits need to be set to 0.
## Limits inheritance
When starting the server from the command line, global limits that are not specified
(configuration file or command line parameters) are inherited from default limits
selected by the server.
Per-channel limits that are not explicitly configured inherit from the corresponding
global limit (which can itself be inherited from default limit).
If a per-channel limit is set to 0 in the configuration file (or negative value
programmatically), then it becomes unlimited, regardless of the corresponding
global limit.
On startup the server displays the store limits. Notice the `*` at the right of a
limit to indicate that the limit was inherited from the default store limits.
For channels that have been configured, their name is displayed and only the
limits being specifically set are displayed to minimize the output.
### Wildcards
Wildcards are allowed for channels configuration. Limits for `foo.>`
will apply to any channel that starts with `foo` (but has at least one more token).
If `foo.bar.>` is specified, it will inherit from `foo.>` and from global limits.
Below is what would be displayed with the above store limits configuration. Notice
how `foo.bar.>` is indented compared to `foo.>` to show the inheritance.
```
[INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.16.0
[INF] STREAM: ServerID: bFHdJP9hesjHIR0RheCl7W
[INF] STREAM: Go version: go1.11.13
[INF] STREAM: Git commit: [not set]
[INF] Starting nats-server version 2.0.2
[INF] Git commit [not set]
[INF] Listening for client connections on 0.0.0.0:4222
[INF] Server id is NDMRMUBKS37GDEPOZP4YVMTRLS6ROZS4O2JQVFOGDRJTGIY44OUV7ZSD
[INF] Server is ready
[INF] STREAM: Recovering the state...
[INF] STREAM: No recovered state
[INF] STREAM: Message store is MEMORY
[INF] STREAM: ---------- Store Limits ----------
[INF] STREAM: Channels: 10
[INF] STREAM: --------- Channels Limits --------
[INF] STREAM: Subscriptions: 1000 *
[INF] STREAM: Messages : 10000
[INF] STREAM: Bytes : 10.00 MB
[INF] STREAM: Age : 1h0m0s
[INF] STREAM: Inactivity : unlimited *
[INF] STREAM: -------- List of Channels ---------
[INF] STREAM: baz
[INF] STREAM: |-> Messages unlimited
[INF] STREAM: |-> Bytes 1.00 MB
[INF] STREAM: |-> Age 2h0m0s
[INF] STREAM: bozo
[INF] STREAM: temp.>
[INF] STREAM: |-> Inactivity 1h0m0s
[INF] STREAM: foo.>
[INF] STREAM: |-> Messages 400
[INF] STREAM: foo.bar.>
[INF] STREAM: |-> Age 2h0m0s
[INF] STREAM: bar
[INF] STREAM: |-> Messages 50
[INF] STREAM: |-> Bytes 1.00 KB
[INF] STREAM: -----------------------------------
```

28
nats_streaming/gettingstarted/tls.md → nats_streaming/configuring/tls.md
View File

@ -33,7 +33,7 @@ The embedded NATS server specifies TLS server certificates with these:
--tlscacert <file> Client certificate CA for verification
```
The server parameters are used the same way you'd [secure a typical NATS server](https://github.com/nats-io/nats-server#securing-nats).
The server parameters are used the same way you'd [secure a typical NATS server](../../nats_server/tls.md).
Proper usage of the NATS Streaming Server requires the use of both client and server parameters.
@ -43,4 +43,28 @@ For example:
% nats-streaming-server -tls_client_cert client-cert.pem -tls_client_key client-key.pem -tls_client_cacert ca.pem -tlscert server-cert.pem -tlskey server-key.pem -tlscacert ca.pem
```
Further TLS related functionality can be found in [Securing NATS > TLS](https://github.com/nats-io/nats-server#securing-nats). Note that if specifying cipher suites is required, a configuration file for the embedded NATS server can be passed through the `-config` command line parameter.
Further TLS related functionality can be found in [Securing NATS > TLS](../../nats_server/tls.md). Note that if specifying cipher suites is required, a configuration file for the embedded NATS server can be passed through the `-config` command line parameter.
### Connecting to Remote NATS Server with TLS Enabled
If that is the case, it is not necessary to configure the server-side TLS parameters. You only need to specify the client-side parameters (`-tls_client_cert`, etc...).
However, NATS Streaming Server uses the NATS Server command line parsing code and currently would not allow specifying the client-side parameters alone. The server would fail to start with a message similar to this:
```sh
$ nats-streaming-server -tls_ca_cert test/certs/ca.pem
TLS Server certificate must be present and valid
```
The solution is to include the required client-side parameters in a configuration file, say `c.conf`:
```
streaming {
tls {
client_ca: "test/certs/ca.pem"
}
}
```
And then start the server with this configuration file:
```sh
$ nats-streaming-server -c c.conf
```

107
nats_streaming/gettingstarted/changes.md Normal file
View File

@ -0,0 +1,107 @@
# Important Changes
## Version `0.10.0`
The server needs to persist more state for a client connection. Therefore, the Store interface has been changed:
* Changed `AddClient(clientID, hbInbox string)` to `AddClient(info *spb.ClientInfo)`
For SQL Stores, the `Clients` table has been altered to add a `proto` column.<br>
You can update the SQL table manually or run the provided scripts that create the tables if they don't exists
and alter the `Clients` table adding the new column. For instance, with MySQL, you would run something similar to:
```
mysql -u root nss_db < scripts/mysql.db.sql
```
The above assumes you are in the NATS Streaming Server directory, and the streaming database is called `nss_db`.
Otherwise, from the mysql CLI, you can run the command:
```
mysql> alter table Clients add proto blob;
Query OK, 0 rows affected (0.05 sec)
Records: 0 Duplicates: 0 Warnings: 0
```
For Postgres, it would be:
```
nss_db=# alter table Clients add proto bytea;
ALTER TABLE
```
If you run the server version with `0.10.0` a database that has not been updated, you would get the following error:
```
[FTL] STREAM: Failed to start: unable to prepare statement "INSERT INTO Clients (id, hbinbox, proto) VALUES (?, ?, ?)": Error 1054: Unknown column 'proto' in 'field list'
```
## Version `0.9.0`
Additions to the Store interface to support deletion of channels.
* Added `Store.GetChannelLimits()` API to return the store limits for a given channel.
* Added `Store.DeleteChannel()` API to delete a channel.
Protocol was added to support replication of deletion of a channel in the cluster.
## Version `0.8.0-beta`
The Store interface has been slightly changed to accommodate the clustering feature.
* Changed `MstStore.Store()` API to accept a `*pb.MsgProto` instead of a byte array. This is because the server is now assigning the sequence number.
The store implementation should ignore the call if the given sequence number is below or equal to what has been already stored.
* Added `MsgStore.Empty()` API to empty a given channel message store.
## Version `0.6.0`
The Store interface has been heavily modified. Some of the responsibilities have been moved into the server
resulting on deletion of some Store APIs and removal of `UserData` fields in `Client` and `ChannelStore` (renamed `Channel`) objects.
NOTE: Although the interface has changed, the file format of the FileStore implementation has not, which means
that there is backward/forward compatibility between this and previous releases.
The Store interface was updated:
* Added error `ErrAlreadyExists` that `CreateChannel()` should return if channel already exists.
* `RecoveredState` has now `Channels` (instead of `Subs`) and is a map of `*RecoveredChannel` keyed by channel name.
* `RecoveredChannel` has a pointer to a `Channel` (formely `ChannelStore`) and an array of pointers to `RecoveredSubscription` objects.
* `RecoveredSubscription` replaces `RecoveredSubState`.
* `Client` no longer stores a `UserData` field.
* `Channel` (formely `ChannelStore`) no longer stores a `UserData` field.
* `CreateChannel()` no longer accepts a `userData interface{}` parameter. It returns a `*Channel` and an `error`. If the channel
already exists, the error `ErrAlreadyExists` is returned.
* `LookupChannel()`, `HasChannel()`, `GetChannels()`, `GetChannelsCount()`, `GetClient()`, `GetClients`, `GetClientsCount()` and `MsgsState()` APIs
have all been removed. The server keeps track of clients and channels and therefore does not need those APIs.
* `AddClient()` is now simply returning a `*Client` and `error`. It no longer accepts a `userData interface{}` parameter.
* `DeleteClient()` now returns an error instead of returning the deleted `*Client`. This will allow the server to
report possible errors.
The SubStore interface was updated:
* `DeleteSub()` has been modified to return an error. This allows the server to report possible errors during deletion
of a subscription.
The MsgStore interface was updated:
* `Lookup()`, `FirstSequence()`, `LastSequence()`, `FirstAndLastSequence()`, `GetSequenceFromTimestamp()`, `FirstMsg()` and `LastMsg()`
have all been modified to return an error. This is so that implementations that may fail to lookup, get the first sequence, etc...
have a way to report the error to the caller.
## Version `0.5.0`
The Store interface was updated. There are 2 news APIs:
* `GetChannels()`: Returns a map of `*ChannelStore`, keyed by channel names.<br>
The implementation needs to return a copy to make it safe for the caller to manipulate
the map without a risk of concurrent access.
* `GetChannelsCount()`: Returns the number of channels currently stored.
## Version `0.4.0`
The Store interface was updated. There are 2 news APIs:
* `Recover()`: The recovery of persistent state was previously done in the constructor of the store implementation.<br>
It is now separate and specified with this API. The server will first instantiate the store, in
which some initialization or checks can be made.<br>
If no error is reported, the server will then proceed with calling `Recover()`, which will returned the recovered state.<br>
* `GetExclusiveLock()`: In Fault Tolerance mode, when a server is elected leader, it will attempt to get an exclusive
lock to the shared storage before proceeding.<br>
Check the [Store interface](https://github.com/nats-io/nats-streaming-server/blob/master/stores/store.go) for more information.

3
nats_streaming/gettingstarted/configuring.md
View File

@ -1,3 +0,0 @@
## Configuration and Administration
NATS Streaming provides a rich set of commands and parameters to configure all aspects of the server. Please refer to the [README](https://github.com/nats-io/nats-streaming-server#configuring) for further information on usage, configuration, and administration.

2
nats_streaming/intro.md
View File

@ -52,4 +52,4 @@ NATS provides a [server binary](gettingstarted/install.md) for Linux, Mac, and W
## Usage, Configuration and Administration
NATS Streaming provides a rich set of commands and parameters to configure all aspects of the server. Please refer to the [README](https://github.com/nats-io/nats-streaming-server#configuring) for further information on usage, configuration, and administration.
NATS Streaming provides a rich set of commands and parameters to configure all aspects of the server. Please refer to the [Configuring](configuring/configuring.md) for further information on usage, configuration, and administration.