taigrr/nats.docs
1
0
Fork 0
mirror of https://github.com/taigrr/nats.docs synced 2025-01-18 04:03:23 -08:00
Code Issues Projects Releases Wiki Activity

Merge branch 'master' into accountz

Browse Source
This commit is contained in:
Colin Sullivan 2021-03-15 10:59:32 -06:00 committed by GitHub
commit 2010bbd6b6
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
62 changed files with 3722 additions and 264 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
.gitbook/assets/consumer-groups.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

BIN
.gitbook/assets/meta-group.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 19 KiB

BIN
.gitbook/assets/replication-setup.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

BIN
.gitbook/assets/replication.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

BIN
.gitbook/assets/stream-groups.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 19 KiB

BIN
.gitbook/assets/streams-and-consumers-75p.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 53 KiB

32
SUMMARY.md
View File

@ -96,8 +96,10 @@
* [Configuration](nats-server/configuration/leafnodes/leafnode_conf.md)
* [Logging](nats-server/configuration/logging.md)
* [Monitoring](nats-server/configuration/monitoring.md)
* [MQTT](nats-server/configuration/mqtt.md)
* [System Events](nats-server/configuration/sys_accounts/README.md)
* [System Events & Decentralized JWT Tutorial](nats-server/configuration/sys_accounts/sys_accounts.md)
* [Websocket](nats-server/configuration/websocket.md)
* [Managing A NATS Server](nats-server/nats_admin/README.md)
* [Upgrading a Cluster](nats-server/nats_admin/upgrading_cluster.md)
* [Slow Consumers](nats-server/nats_admin/slow_consumers.md)
@ -108,10 +110,38 @@
* [Docker Swarm](nats-server/nats_docker/docker_swarm.md)
* [Python and NGS Running in Docker](nats-server/nats_docker/ngs-docker-python.md)
## JetStream
* [About Jetstream](jetstream/jetstream.md)
* [Concepts](jetstream/concepts/README.md)
* [Streams](jetstream/concepts/streams.md)
* [Consumers](jetstream/concepts/consumers.md)
* [Example Configuration](jetstream/concepts/configuration.md)
* [Getting Started](jetstream/getting_started/README.md)
* [Using Docker](jetstream/getting_started/using_docker.md)
* [Using Source](jetstream/getting_started/using_source.md)
* [Administration & Usage from CLI](jetstream/administration/README.md)
* [Account Information](jetstream/administration/account.md)
* [Streams](jetstream/administration/streams.md)
* [Consumers](jetstream/administration/consumers.md)
* [Monitoring](jetstream/monitoring.md)
* [Data Replication](jetstream/replication.md)
* [Clustering](jetstream/clustering/README.md)
* [Administration](jetstream/clustering/administration.md)
* [Configuration Management](jetstream/configuration_mgmt/README.md)
* [NATS Admin CLI](jetstream/configuration_mgmt/nats-admin-cli.md)
* [Terraform](jetstream/configuration_mgmt/terraform.md)
* [GitHub Actions](jetstream/configuration_mgmt/github_actions.md)
* [Kubernetes Controller](jetstream/configuration_mgmt/kubernetes_controller.md)
* [Disaser Recovery](jetstream/disaster_recovery.md)
* [Model Deep Dive](jetstream/model_deep_dive.md)
* [NATS API Reference](jetstream/nats_api_reference.md)
* [Multi-tenancy & Resource Mgmt](jetstream/resource_management.md)
## NATS Tools
* [Introduction](nats-tools/nats-tools.md)
* [mkpasswd](nats-tools/mkpasswd.md)
* [nats](nats-tools/natscli.md)
* [nk](nats-tools/nk.md)
* [nsc](nats-tools/nsc/README.md)
* [Basics](nats-tools/nsc/nsc.md)

2
developing-with-nats-streaming/connecting.md
View File

@ -6,7 +6,7 @@ NATS Streaming is a service on top of NATS. To connect to the service you first
Connecting to a streaming server requires a cluster id, defined by the server configuration, and a client ID defined by the client.
_Client ID should contain only alphanumeric characters, `-` or `_`_
_Client ID should contain only alphanumeric characters, `-` or \`_\`\_
Connecting to a server running locally on the default port is as simple as this:

10
developing-with-nats/security/userpass.md
View File

@ -6,12 +6,14 @@ For this example, start the server using:
> nats-server --user myname --pass password
```
You can encrypt passwords to pass to `nats-server` using a simple [tool](../../nats-tools/mkpasswd.md) provided by the server:
You can encrypt passwords to pass to `nats-server` using a simple [tool](../../nats-tools/natscli.md):
```bash
> go run mkpasswd.go -p
> password: password
> bcrypt hash: $2a$11$1oJy/wZYNTxr9jNwMNwS3eUGhBpHT3On8CL9o7ey89mpgo88VG6ba
> nats server passwd
? Enter password [? for help] **********************
? Reenter password [? for help] **********************
$2a$11$qbtrnb0mSG2eV55xoyPqHOZx/lLBlryHRhU3LK2oOPFRwGF/5rtGK
```
and use the hashed password in the server config. The client still uses the plain text version.

2
docs/nats_tools/README.md
View File

@ -2,7 +2,7 @@
The NATS Ecosystem has many tools to support server configuration, enhance monitoring or tune performance:
- [mkpasswd](nats_tools/mkpasswd.md) - Generates or bcrypts passwords
- [nats](nats_tools/natscli.md) - Interact with and manage NATS
- [nk](nats_tools/nk.md) - Generate NKeys
- [nsc](nats_tools/nsc/README.md) - Configure Operators, Accounts and Users
- [nats account server](nats_tools/nas/README.md) - Serve Account JWTs

4
faq.md
View File

@ -159,9 +159,9 @@ NATS 2.0 is completely backwards compatible with NATS < 2.x configure files a
The default setting for a single server is 65,536. Although there is no specified limit to the number of connections supported by NATS, there are some environmental factors that will influence your decision as to how many connections to allow per server.
Most systems can handle several thousand NATS connections per server without any changes although some have a very low default such as OS X. You'll want to look at kernel/OS settings to increase that limit. You'll also want to look at default TCP buffer sizes to best optimize your machine for your traffic characteristics.
Most systems can handle several thousand NATS connections per server without any changes although some have a very low default such as OS X. You'll want to look at kernel/OS settings to increase that limit. You'll also want to look at default TCP buffer sizes to best optimize your machine for your traffic characteristics.
If you are using TLS you'll want to be sure the hardware can handle the CPU load created by TLS negotiation when there is the thundering herd of inbound connections after an outage or network partition event. This often overlooked factor is usually the constraint limiting the number of connections a single server should support. Choosing a cipher suite that is supported by TLS acceleration can mitigate this (e.g. AES with x86). Thinking of the entire system, you'll also want to look at a range of reconnect delay times or add reconnect jitter to the NATS clients to even out the distribution of connection attempts over time and reduce CPU spikes.
If you are using TLS you'll want to be sure the hardware can handle the CPU load created by TLS negotiation when there is the thundering herd of inbound connections after an outage or network partition event. This often overlooked factor is usually the constraint limiting the number of connections a single server should support. Choosing a cipher suite that is supported by TLS acceleration can mitigate this \(e.g. AES with x86\). Thinking of the entire system, you'll also want to look at a range of reconnect delay times or add reconnect jitter to the NATS clients to even out the distribution of connection attempts over time and reduce CPU spikes.
All said, each server can be tuned to handle a large number of clients, and given the flexibility and scalability of NATS with clusters, superclusters, and leaf nodes one can build a NATS deployment supporting many millions of connections.

33
jetstream/administration/README.md Normal file
View File

@ -0,0 +1,33 @@
# Administration & Usage from CLI
Once the server is running it's time to use the management tool. This can be downloaded from the [GitHub Release Page](https://github.com/nats-io/natscli/releases/) or you can use the `natsio/nats-box:latest` docker image. On OS X homebrew can be used to install the latest version:
```text
$ brew tap nats-io/nats-tools
$ brew install nats-io/nats-tools/nats
$ nats --help
usage: nats [<flags>] <command> [<args> ...]
NATS Utility
Flags:
--help Show context-sensitive help (also try --help-long and --help-man).
--version Show application version.
-s, --server="localhost:4222" NATS servers
--creds=CREDS User credentials
--tlscert=TLSCERT TLS public certifcate
--tlskey=TLSKEY TLS private key
--tlsca=TLSCA TLS certifcate authority chain
--timeout=2s Time to give JetStream to respond to queries
Commands:
help [<command>...]
Show help.
...
```
We'll walk through the above scenario and introduce features of the CLI and of JetStream as we recreate the setup above.
Throughout this example, we'll show other commands like `nats pub` and `nats sub` to interact with the system. These are normal existing core NATS commands and JetStream is fully usable by only using core NATS.
We'll touch on some additional features but please review the section on the design model to understand all possible permutations.

275
jetstream/administration/account.md Normal file
View File

@ -0,0 +1,275 @@
# Account Information
## Account Information
JetStream is multi-tenant so you will need to check that your account is enabled for JetStream and is not limited. You can view your limits as follows:
```text
$ nats account info
Connection Information:
Client ID: 8
Client IP: 127.0.0.1
RTT: 178.545µs
Headers Supported: true
Maximum Payload: 1.0 MiB
Connected URL: nats://localhost:4222
Connected Address: 127.0.0.1:4222
Connected Server ID: NCCOHA6ONXJOGAEZP4WPU4UJ3IQP2VVXEPRKTQCGBCW4IL4YYW4V4KKL
JetStream Account Information:
Memory: 0 B of 5.7 GiB
Storage: 0 B of 11 GiB
Streams: 0 of Unlimited
Max Consumers: unlimited
```
## Streams
The first step is to set up storage for our `ORDERS` related messages, these arrive on a wildcard of subjects all flowing into the same Stream and they are kept for 1 year.
### Creating
```text
$ nats str add ORDERS
? Subjects to consume ORDERS.*
? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Message count limit -1
? Message size limit -1
? Maximum message age limit 1y
? Maximum individual message size [? for help] (-1) -1
? Number of replicas to store 3
Stream ORDERS was created
Information for Stream ORDERS
Configuration:
Subjects: ORDERS.*
Acknowledgements: true
Retention: File - Limits
Replicas: 3
Maximum Messages: -1
Maximum Bytes: -1
Maximum Age: 8760h0m0s
Maximum Message Size: -1
Maximum Consumers: -1
Statistics:
Messages: 0
Bytes: 0 B
FirstSeq: 0
LastSeq: 0
Active Consumers: 0
```
You can get prompted interactively for missing information as above, or do it all on one command. Pressing `?` in the CLI will help you map prompts to CLI options:
```text
$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --replicas 3
```
Additionally, one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`:
```text
$ nats str add ORDERS --config orders.json
```
### Listing
We can confirm our Stream was created:
```text
$ nats str ls
Streams:
ORDERS
```
### Querying
Information about the configuration of the Stream can be seen, and if you did not specify the Stream like below, it will prompt you based on all known ones:
```text
$ nats str info ORDERS
Information for Stream ORDERS
Configuration:
Subjects: ORDERS.*
No Acknowledgements: false
Retention: File - Limits
Replicas: 1
Maximum Messages: -1
Maximum Bytes: -1
Maximum Age: 8760h0m0s
Maximum Consumers: -1
Statistics:
Messages: 0
Bytes: 0 B
FirstSeq: 0
LastSeq: 0
Active Consumers: 0
```
Most commands that show data as above support `-j` to show the results as JSON:
```text
$ nats str info ORDERS -j
{
"config": {
"name": "ORDERS",
"subjects": [
"ORDERS.*"
],
"retention": "limits",
"max_consumers": -1,
"max_msgs": -1,
"max_bytes": -1,
"max_age": 31536000000000000,
"storage": "file",
"num_replicas": 1
},
"stats": {
"messages": 0,
"bytes": 0,
"first_seq": 0,
"last_seq": 0,
"consumer_count": 0
}
}
```
This is the general pattern for the entire `nats` utility as it relates to JetStream - prompting for needed information but every action can be run non-interactively making it usable as a cli api. All information output like seen above can be turned into JSON using `-j`.
In clustered mode additional information will be included:
```text
$ nats str info ORDERS
...
Cluster Information:
Name: JSC
Leader: S1
Replica: S3, current, seen 0.04s ago
Replica: S2, current, seen 0.04s ago
```
Here the cluster name is configured as `JSC`, there is a server `S1` that's the current leader with `S3` and `S2` are replicas. Both replicas are current and have been seen recently.
### Copying
A stream can be copied into another, which also allows the configuration of the new one to be adjusted via CLI flags:
```text
$ nats str cp ORDERS ARCHIVE --subjects "ORDERS_ARCVHIVE.*" --max-age 2y
Stream ORDERS was created
Information for Stream ARCHIVE
Configuration:
Subjects: ORDERS_ARCVHIVE.*
...
Maximum Age: 17520h0m0s
...
```
### Editing
A stream configuration can be edited, which allows the configuration to be adjusted via CLI flags. Here I have a incorrectly created ORDERS stream that I fix:
```text
$ nats str info ORDERS -j | jq .config.subjects
[
"ORDERS.new"
]
$ nats str edit ORDERS --subjects "ORDERS.*"
Stream ORDERS was updated
Information for Stream ORDERS
Configuration:
Subjects: ORDERS.*
....
```
Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`:
```text
$ nats str edit ORDERS --config orders.json
```
### Publishing Into a Stream
Now let's add in some messages to our Stream. You can use `nats pub` to add messages, pass the `--wait` flag to see the publish ack being returned.
You can publish without waiting for acknowledgement:
```text
$ nats pub ORDERS.scratch hello
Published [sub1] : 'hello'
```
But if you want to be sure your messages got to JetStream and were persisted you can make a request:
```text
$ nats req ORDERS.scratch hello
13:45:03 Sending request on [ORDERS.scratch]
13:45:03 Received on [_INBOX.M8drJkd8O5otORAo0sMNkg.scHnSafY]: '+OK'
```
Keep checking the status of the Stream while doing this and you'll see it's stored messages increase.
```text
$ nats str info ORDERS
Information for Stream ORDERS
...
Statistics:
Messages: 3
Bytes: 147 B
FirstSeq: 1
LastSeq: 3
Active Consumers: 0
```
After putting some throw away data into the Stream, we can purge all the data out - while keeping the Stream active:
### Deleting All Data
To delete all data in a stream use `purge`:
```text
$ nats str purge ORDERS -f
...
Statistics:
Messages: 0
Bytes: 0 B
FirstSeq: 1,000,001
LastSeq: 1,000,000
Active Consumers: 0
```
### Deleting A Message
A single message can be securely removed from the stream:
```text
$ nats str rmm ORDERS 1 -f
```
### Deleting Sets
Finally for demonstration purposes, you can also delete the whole Stream and recreate it so then we're ready for creating the Consumers:
```text
$ nats str rm ORDERS -f
$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1
```

220
jetstream/administration/consumers.md Normal file
View File

@ -0,0 +1,220 @@
# Consumers
Consumers is how messages are read or consumed from the Stream. We support pull and push-based Consumers and the example scenario has both, lets walk through that.
## Creating Pull-Based Consumers
The `NEW` and `DISPATCH` Consumers are pull-based, meaning the services consuming data from them have to ask the system for the next available message. This means you can easily scale your services up by adding more workers and the messages will get spread across the workers based on their availability.
Pull-based Consumers are created the same as push-based Consumers, just don't specify a delivery target.
```text
$ nats con ls ORDERS
No Consumers defined
```
We have no Consumers, lets add the `NEW` one:
I supply the `--sample` options on the CLI as this is not prompted for at present, everything else is prompted. The help in the CLI explains each:
```text
$ nats con add --sample 100
? Select a Stream ORDERS
? Consumer name NEW
? Delivery target
? Start policy (all, last, 1h, msg sequence) all
? Filter Stream by subject (blank for all) ORDERS.received
? Maximum Allowed Deliveries 20
Information for Consumer ORDERS > NEW
Configuration:
Durable Name: NEW
Pull Mode: true
Subject: ORDERS.received
Deliver All: true
Deliver Last: false
Ack Policy: explicit
Ack Wait: 30s
Replay Policy: instant
Maximum Deliveries: 20
Sampling Rate: 100
State:
Last Delivered Message: Consumer sequence: 1 Stream sequence: 1
Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0
Pending Messages: 0
Redelivered Messages: 0
```
This is a pull-based Consumer \(empty Delivery Target\), it gets messages from the first available message and requires specific acknowledgement of each and every message.
It only received messages that originally entered the Stream on `ORDERS.received`. Remember the Stream subscribes to `ORDERS.*`, this lets us select a subset of messages from the Stream.
A Maximum Delivery limit of 20 is set, this means if the message is not acknowledged it will be retried but only up to this maximum total deliveries.
Again this can all be done in a single CLI call, lets make the `DISPATCH` Consumer:
```text
$ nats con add ORDERS DISPATCH --filter ORDERS.processed --ack explicit --pull --deliver all --sample 100 --max-deliver 20
```
Additionally, one can store the configuration in a JSON file, the format of this is the same as `$ nats con info ORDERS DISPATCH -j | jq .config`:
```text
$ nats con add ORDERS MONITOR --config monitor.json
```
## Creating Push-Based Consumers
Our `MONITOR` Consumer is push-based, has no ack and will only get new messages and is not sampled:
```text
$ nats con add
? Select a Stream ORDERS
? Consumer name MONITOR
? Delivery target monitor.ORDERS
? Start policy (all, last, 1h, msg sequence) last
? Acknowledgement policy none
? Replay policy instant
? Filter Stream by subject (blank for all)
? Maximum Allowed Deliveries -1
Information for Consumer ORDERS > MONITOR
Configuration:
Durable Name: MONITOR
Delivery Subject: monitor.ORDERS
Deliver All: false
Deliver Last: true
Ack Policy: none
Replay Policy: instant
State:
Last Delivered Message: Consumer sequence: 1 Stream sequence: 3
Acknowledgment floor: Consumer sequence: 0 Stream sequence: 2
Pending Messages: 0
Redelivered Messages: 0
```
Again you can do this with a single non interactive command:
```text
$ nats con add ORDERS MONITOR --ack none --target monitor.ORDERS --deliver last --replay instant --filter ''
```
Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats con info ORDERS MONITOR -j | jq .config`:
```text
$ nats con add ORDERS --config monitor.json
```
## Listing
You can get a quick list of all the Consumers for a specific Stream:
```text
$ nats con ls ORDERS
Consumers for Stream ORDERS:
DISPATCH
MONITOR
NEW
```
## Querying
All details for an Consumer can be queried, lets first look at a pull-based Consumer:
```text
$ nats con info ORDERS DISPATCH
Information for Consumer ORDERS > DISPATCH
Configuration:
Durable Name: DISPATCH
Pull Mode: true
Subject: ORDERS.processed
Deliver All: true
Deliver Last: false
Ack Policy: explicit
Ack Wait: 30s
Replay Policy: instant
Sampling Rate: 100
State:
Last Delivered Message: Consumer sequence: 1 Stream sequence: 1
Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0
Pending Messages: 0
Redelivered Messages: 0
```
More details about the `State` section will be shown later when discussing the ack models in depth.
## Consuming Pull-Based Consumers
Pull-based Consumers require you to specifically ask for messages and ack them, typically you would do this with the client library `Request()` feature, but the `nats` utility has a helper:
First we ensure we have a message:
```text
$ nats pub ORDERS.processed "order 1"
$ nats pub ORDERS.processed "order 2"
$ nats pub ORDERS.processed "order 3"
```
We can now read them using `nats`:
```text
$ nats con next ORDERS DISPATCH
--- received on ORDERS.processed
order 1
Acknowledged message
$ nats con next ORDERS DISPATCH
--- received on ORDERS.processed
order 2
Acknowledged message
```
You can prevent ACKs by supplying `--no-ack`.
To do this from code you'd send a `Request()` to `$JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH`:
```text
$ nats req '$JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH' ''
Published [$JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH] : ''
Received [ORDERS.processed] : 'order 3'
```
Here `nats req` cannot ack, but in your code you'd respond to the received message with a nil payload as an Ack to JetStream.
## Consuming Push-Based Consumers
Push-based Consumers will publish messages to a subject and anyone who subscribes to the subject will get them, they support different Acknowledgement models covered later, but here on the `MONITOR` Consumer we have no Acknowledgement.
```text
$ nats con info ORDERS MONITOR
...
Delivery Subject: monitor.ORDERS
...
```
The Consumer is publishing to that subject, so lets listen there:
```text
$ nats sub monitor.ORDERS
Listening on [monitor.ORDERS]
[#3] Received on [ORDERS.processed]: 'order 3'
[#4] Received on [ORDERS.processed]: 'order 4'
```
Note the subject here of the received message is reported as `ORDERS.processed` this helps you distinguish what you're seeing in a Stream covering a wildcard, or multiple subject, subject space.
This Consumer needs no ack, so any new message into the ORDERS system will show up here in real time.

260
jetstream/administration/streams.md Normal file
View File

@ -0,0 +1,260 @@
# Streams
The first step is to set up storage for our `ORDERS` related messages, these arrive on a wildcard of subjects all flowing into the same Stream and they are kept for 1 year.
## Creating
```text
$ nats str add ORDERS
? Subjects to consume ORDERS.*
? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Message count limit -1
? Message size limit -1
? Maximum message age limit 1y
? Maximum individual message size [? for help] (-1) -1
Stream ORDERS was created
Information for Stream ORDERS
Configuration:
Subjects: ORDERS.*
Acknowledgements: true
Retention: File - Limits
Replicas: 1
Maximum Messages: -1
Maximum Bytes: -1
Maximum Age: 8760h0m0s
Maximum Message Size: -1
Maximum Consumers: -1
Statistics:
Messages: 0
Bytes: 0 B
FirstSeq: 0
LastSeq: 0
Active Consumers: 0
```
You can get prompted interactively for missing information as above, or do it all on one command. Pressing `?` in the CLI will help you map prompts to CLI options:
```text
nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --dupe-window="0s" --replicas 1
```
Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`:
```text
$ nats str add ORDERS --config orders.json
```
## Listing
We can confirm our Stream was created:
```text
$ nats str ls
Streams:
ORDERS
```
## Querying
Information about the configuration of the Stream can be seen, and if you did not specify the Stream like below, it will prompt you based on all known ones:
```text
$ nats str info ORDERS
Information for Stream ORDERS created 2021-02-27T16:49:36-07:00
Configuration:
Subjects: ORDERS.*
Acknowledgements: true
Retention: File - Limits
Replicas: 1
Discard Policy: Old
Duplicate Window: 2m0s
Maximum Messages: unlimited
Maximum Bytes: unlimited
Maximum Age: 1y0d0h0m0s
Maximum Message Size: unlimited
Maximum Consumers: unlimited
State:
Messages: 0
Bytes: 0 B
FirstSeq: 0
LastSeq: 0
Active Consumers: 0
```
Most commands that show data as above support `-j` to show the results as JSON:
```text
$ nats str info ORDERS -j
{
"config": {
"name": "ORDERS",
"subjects": [
"ORDERS.*"
],
"retention": "limits",
"max_consumers": -1,
"max_msgs": -1,
"max_bytes": -1,
"max_age": 31536000000000000,
"max_msg_size": -1,
"storage": "file",
"discard": "old",
"num_replicas": 1,
"duplicate_window": 120000000000
},
"created": "2021-02-27T23:49:36.700424Z",
"state": {
"messages": 0,
"bytes": 0,
"first_seq": 0,
"first_ts": "0001-01-01T00:00:00Z",
"last_seq": 0,
"last_ts": "0001-01-01T00:00:00Z",
"consumer_count": 0
}
}
```
This is the general pattern for the entire `nats` utility as it relates to JetStream - prompting for needed information but every action can be run non-interactively making it usable as a cli api. All information output like seen above can be turned into JSON using `-j`.
## Copying
A stream can be copied into another, which also allows the configuration of the new one to be adjusted via CLI flags:
```text
$ nats str cp ORDERS ARCHIVE --subjects "ORDERS_ARCVHIVE.*" --max-age 2y
Stream ORDERS was created
Information for Stream ORDERS created 2021-02-27T16:52:46-07:00
Configuration:
Subjects: ORDERS_ARCHIVE.*
Acknowledgements: true
Retention: File - Limits
Replicas: 1
Discard Policy: Old
Duplicate Window: 2m0s
Maximum Messages: unlimited
Maximum Bytes: unlimited
Maximum Age: 2y0d0h0m0s
Maximum Message Size: unlimited
Maximum Consumers: unlimited
State:
Messages: 0
Bytes: 0 B
FirstSeq: 0
LastSeq: 0
Active Consumers: 0
```
## Editing
A stream configuration can be edited, which allows the configuration to be adjusted via CLI flags. Here I have a incorrectly created ORDERS stream that I fix:
```text
$ nats str info ORDERS -j | jq .config.subjects
[
"ORDERS.new"
]
$ nats str edit ORDERS --subjects "ORDERS.*"
Stream ORDERS was updated
Information for Stream ORDERS
Configuration:
Subjects: ORDERS.*
....
```
Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`:
```text
$ nats str edit ORDERS --config orders.json
```
## Publishing Into a Stream
Now let's add in some messages to our Stream. You can use `nats pub` to add messages, pass the `--wait` flag to see the publish ack being returned.
You can publish without waiting for acknowledgement:
```text
$ nats pub ORDERS.scratch hello
Published [sub1] : 'hello'
```
But if you want to be sure your messages got to JetStream and were persisted you can make a request:
```text
$ nats req ORDERS.scratch hello
13:45:03 Sending request on [ORDERS.scratch]
13:45:03 Received on [_INBOX.M8drJkd8O5otORAo0sMNkg.scHnSafY]: '+OK'
```
Keep checking the status of the Stream while doing this and you'll see it's stored messages increase.
```text
$ nats str info ORDERS
Information for Stream ORDERS
...
Statistics:
Messages: 3
Bytes: 147 B
FirstSeq: 1
LastSeq: 3
Active Consumers: 0
```
After putting some throw away data into the Stream, we can purge all the data out - while keeping the Stream active:
## Deleting All Data
To delete all data in a stream use `purge`:
```text
$ nats str purge ORDERS -f
...
State:
Messages: 0
Bytes: 0 B
FirstSeq: 1,000,001
LastSeq: 1,000,000
Active Consumers: 0
```
## Deleting A Message
A single message can be securely removed from the stream:
```text
$ nats str rmm ORDERS 1 -f
```
## Deleting Sets
Finally for demonstration purposes, you can also delete the whole Stream and recreate it so then we're ready for creating the Consumers:
```text
$ nats str rm ORDERS -f
$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --dupe-window="0s" --replicas 1
```

111
jetstream/clustering/README.md Normal file
View File

@ -0,0 +1,111 @@
# Clustering
Clustering in JetStream is required for a highly available and scalable system. Behind clustering is RAFT. There's no need to understand RAFT in depth to use clustering, but knowing a little explains some of the requirements behind setting up JetStream clusters.
## RAFT
JetStream uses a NATS optimized RAFT algorithm for clustering. Typically raft generates a lot of traffic, but the NATS server optimizes this by combining the data plane for replicating messages with the messages RAFT would normally use to ensure consensus.
### Raft groups
The RAFT groups include API handlers, streams, consumers, and an internal algorithm designates which servers handle which streams and consumers.
The raft algorithm has a few requirements:
* A log to persist state
* A quorum for consensus.
### The Quorum
In order to ensure data consistency across complete restarts, a quorum of servers is required. A quorum is ½ cluster size + 1. This is the minimum number of nodes to ensure at least one node has the most recent data and state after a catastrophic failure. So for a cluster size of 3, you’ll need at least two Jetstream enabled NATS servers available to store new messages. For a cluster size of 5, you’ll need at least 3 NATS servers, and so forth.
### RAFT Groups
**Meta Group** - all servers join the Meta Group and the JetStream API is managed by this group. A leader is elected and this owns the API and takes care of server placement.
![Meta Group](../../.gitbook/assets/meta-group.png)
**Stream Group** - each Stream creates a RAFT group, this group synchronizes state and data between its members. The elected leader handles ACKs and so forth, if there is no leader the stream will not accept messages.
![Stream Groups](../../.gitbook/assets/stream-groups.png)
**Consumer Group** - each Consumer creates a RAFT group, this group synchronizes consumer state between its members. The group will live on the machines where the Stream Group is and handle consumption ACKs etc. Each Consumer will have their own group.
![Consumer Groups](../../.gitbook/assets/consumer-groups.png)
### Cluster Size
Generally, we recommend 3 or 5 JetStream enabled servers in a NATS cluster. This balances scalability with a tolerance for failure. For example, if 5 servers are JetStream enabled You would want two servers is one “zone”, two servers in another, and the remaining server in a third. This means you can lose any one “zone” at any time and continue operating.
### Mixing JetStream enabled servers with standard NATS servers
This is possible and even recommended in some cases. By mixing server types you can dedicate certain machines optimized for storage for Jetstream and others optimized solely for compute for standard NATS servers, reducing operational expense. With the right configuration, the standard servers would handle non-persistent NATS traffic and the JetStream enabled servers would handle JetStream traffic.
## Configuration
To configure JetStream clusters, just configure clusters as you normally would by specifying a cluster block in the configuration. Any JetStream enabled servers in the list of clusters will automatically chatter and set themselves up. Unlike core NATS clustering though, each JetStream node **must specify** a server name and cluster name.
Below are explicitly listed server configuration for a three-node cluster across three machines, `n1-c1`, `n2-c1`, and `n3-c1`.
### Server 1 \(host\_a\)
```text
server_name=n1-c1
listen=4222
jetstream {
store_dir=/nats/storage
}
cluster {
name: C1
listen: localhost:6222
routes: [
nats-route://host_b:6222
nats-route://host_c:6222
]
}
```
### Server 2 \(host\_b\)
```text
server_name=n2-c1
listen=4222
jetstream {
store_dir=/nats/storage
}
cluster {
name: C1
listen: localhost:6222
routes: [
nats-route://host_a:6222
nats-route://host_c:6222
]
}
```
### Server 3 \(host\_c\)
```text
server_name=n3-c1
listen=4222
jetstream {
store_dir=/nats/storage
}
cluster {
name: C1
listen: localhost:6222
routes: [
nats-route://host_a:6222
nats-route://host_b:6222
]
}
```
Add nodes as necessary. Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use JetStream.

184
jetstream/clustering/administration.md Normal file
View File

@ -0,0 +1,184 @@
# Administration
Once a JetStream cluster is operating interactions with the CLI and with `nats` CLI is the same as before. For these examples, lets assume we have a 5 server cluster, n1-n5 in a cluster named C1.
## Account Level
Within an account there are operations and reports that show where users data is placed and which allow them some basic interactions with the RAFT system.
## Creating clustered streams
When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, \(we suggest 1, 3 or 5\), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. The replica count must be less than the maximum number of servers.
```text
$ nats str add ORDERS --replicas 3
....
Information for Stream ORDERS created 2021-02-05T12:07:34+01:00
....
Configuration:
....
Replicas: 3
Cluster Information:
Name: C1
Leader: n1-c1
Replica: n4-c1, current, seen 0.07s ago
Replica: n3-c1, current, seen 0.07s ago
```
Above you can see that the cluster information will be reported in all cases where Stream info is shown such as after add or using `nats stream info`.
Here we have a stream in the NATS cluster `C1`, its current leader is a node `n1-c1` and it has 2 followers - `n4-c1` and `n3-c1`.
The `current` indicates that followers are up to date and have all the messages, here both cluster peers were seen very recently.
The replica count cannot be edited once configured.
### Viewing Stream Placement and Stats
Users can get overall statistics about their streams and also where these streams are placed:
```text
$ nats stream report
Obtaining Stream stats
+----------+-----------+----------+--------+---------+------+---------+----------------------+----------+
| Stream | Consumers | Messages | Bytes | Storage | Lost | Deleted | Cluster | Template |
+----------+-----------+----------+--------+---------+------+---------+----------------------+----------+
| ORDERS | 4 | 0 | 0 B | File | 0 | 0 | n1-c1*, n2-c1, n3-c1 | |
| ORDERS_3 | 4 | 0 | 0 B | File | 0 | 0 | n1-c1*, n2-c1, n3-c1 | |
| ORDERS_4 | 4 | 0 | 0 B | File | 0 | 0 | n1-c1*, n2-c1, n3-c1 | |
| ORDERS_5 | 4 | 0 | 0 B | File | 0 | 0 | n1-c1, n2-c1, n3-c1* | |
| ORDERS_2 | 4 | 1,385 | 13 MiB | File | 0 | 1 | n1-c1, n2-c1, n3-c1* | |
| ORDERS_0 | 4 | 1,561 | 14 MiB | File | 0 | 0 | n1-c1, n2-c1*, n3-c1 | |
+----------+-----------+----------+--------+---------+------+---------+----------------------+----------+
```
#### Forcing Stream and Consumer leader election
Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process, but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time.
Moving leadership away from a node does not remove it from the cluster and does not prevent it from becoming a leader again, this is merely a triggered leader election.
```text
$ nats stream cluster step-down ORDERS
14:32:17 Requesting leader step down of "n1-c1" in a 3 peer RAFT group
14:32:18 New leader elected "n4-c1"
Information for Stream ORDERS created 2021-02-05T12:07:34+01:00
...
Cluster Information:
Name: c1
Leader: n4-c1
Replica: n1-c1, current, seen 0.12s ago
Replica: n3-c1, current, seen 0.12s ago
```
The same is true for consumers, `nats consumer cluster step-down ORDERS NEW`.
## System Level
Systems users can view state of the Meta Group - but not individual Stream or Consumers.
### Viewing the cluster state
We have a high level report of cluster state:
```text
$ nats server report jetstream --user system
+--------------------------------------------------------------------------------------------------+
| JetStream Summary |
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
| Server | Cluster | Streams | Consumers | Messages | Bytes | Memory | File | API Req | API Err |
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
| n3-c2 | c2 | 0 | 0 | 0 | 0 B | 0 B | 0 B | 1 | 0 |
| n3-c1 | c1 | 6 | 24 | 2,946 | 27 MiB | 0 B | 27 MiB | 3 | 0 |
| n2-c2 | c2 | 0 | 0 | 0 | 0 B | 0 B | 0 B | 3 | 0 |
| n1-c2 | c2 | 0 | 0 | 0 | 0 B | 0 B | 0 B | 14 | 2 |
| n2-c1 | c1 | 6 | 24 | 2,946 | 27 MiB | 0 B | 27 MiB | 15 | 0 |
| n1-c1* | c1 | 6 | 24 | 2,946 | 27 MiB | 0 B | 27 MiB | 31 | 0 |
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
| | | 18 | 72 | 8,838 | 80 MiB | 0 B | 80 MiB | 67 | 2 |
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
+---------------------------------------------------+
| RAFT Meta Group Information |
+-------+--------+---------+---------+--------+-----+
| Name | Leader | Current | Offline | Active | Lag |
+-------+--------+---------+---------+--------+-----+
| n1-c1 | yes | true | false | 0.00s | 0 |
| n1-c2 | | true | false | 0.05s | 0 |
| n2-c1 | | true | false | 0.05s | 0 |
| n2-c2 | | true | false | 0.05s | 0 |
| n3-c1 | | true | false | 0.05s | 0 |
| n3-c2 | | true | false | 0.05s | 0 |
+-------+--------+---------+---------+--------+-----+
```
This is a full cluster wide report, the report can be limited to a specific account using `--account`.
Here we see the distribution of streams, messages, api calls etc by across 2 super clusters and an overview of the RAFT meta group.
In the Meta Group report the server `n2-c1` is not current and has not been seen for 9 seconds, it's also behind by 2 raft operations.
This report is built using raw data that can be obtained from the monitor port on the `/jsz` url, or over nats using:
```text
$ nats server req jetstream --help
...
--name=NAME Limit to servers matching a server name
--host=HOST Limit to servers matching a server host name
--cluster=CLUSTER Limit to servers matching a cluster name
--tags=TAGS ... Limit to servers with these configured tags
--account=ACCOUNT Show statistics scoped to a specific account
--accounts Include details about accounts
--streams Include details about Streams
--consumer Include details about Consumers
--config Include details about configuration
--leader Request a response from the Meta-group leader only
--all Include accounts, streams, consumers and configuration
$ nats server req jetstream --leader
```
This will produce a wealth of raw information about the current state of your cluster - here requesting it from the leader only.
#### Forcing Meta Group leader election
Similar to Streams and Consumers above the Meta Group allows leader stand down. The Meta Group is cluster wide and spans all accounts, therefore to manage the meta group you have to use a `SYSTEM` user.
```text
$ nats server raft step-down --user system
17:44:24 Current leader: n2-c2
17:44:24 New leader: n1-c2
```
### Evicting a peer
Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down.
There might be a case though where you know a machine will never return, and you want to signal to JetStream that the machine will not return. This will remove it from the Stream in question and all it's Consumers.
After the node is removed the cluster will notice that the replica count is not honored anymore and will immediately pick a new node and start replicating data to it. The new node will be selected using the same placement rules as the existing stream.
```text
$ nats s cluster peer-remove ORDERS
? Select a Peer n4-c1
14:38:50 Removing peer "n4-c1"
14:38:50 Requested removal of peer "n4-c1"
```
At this point the stream and all consumers will have removed `n4-c1` from the group, they will all start new peer selection and data replication.
```text
$ nats stream info ORDERS
....
Cluster Information:
Name: c1
Leader: n3-c1
Replica: n1-c1, current, seen 0.02s ago
Replica: n2-c1, outdated, seen 0.42s ago
```
We can see a new replica was picked, the stream is back to replication level of 3 and `n4-c1` is not active any more in this Stream or any of its Consumers.

20
jetstream/concepts/README.md Normal file
View File

@ -0,0 +1,20 @@
# Concepts
In JetStream the configuration for storing messages is defined separately from how they are consumed. Storage is defined in a _Stream_ and consuming messages is defined by multiple _Consumers_.
We'll discuss these 2 subjects in the context of this architecture.
![Orders](../../.gitbook/assets/streams-and-consumers-75p.png)
While this is an incomplete architecture it does show a number of key points:
* Many related subjects are stored in a Stream
* Consumers can have different modes of operation and receive just subsets of the messages
* Multiple Acknowledgement modes are supported
A new order arrives on `ORDERS.received`, gets sent to the `NEW` Consumer who, on success, will create a new message on `ORDERS.processed`. The `ORDERS.processed` message again enters the Stream where a `DISPATCH` Consumer receives it and once processed it will create an `ORDERS.completed` message which will again enter the Stream. These operations are all `pull` based meaning they are work queues and can scale horizontally. All require acknowledged delivery ensuring no order is missed.
All messages are delivered to a `MONITOR` Consumer without any acknowledgement and using Pub/Sub semantics - they are pushed to the monitor.
As messages are acknowledged to the `NEW` and `DISPATCH` Consumers, a percentage of them are Sampled and messages indicating redelivery counts, ack delays and more, are delivered to the monitoring system.

11
jetstream/concepts/configuration.md Normal file
View File

@ -0,0 +1,11 @@
# Example Configuration
[Additional documentation](../clustering/administration.md) introduces the `nats` utility, but for completeness and reference this is how you'd create the ORDERS scenario. We'll configure a 1 year retention for order related messages:
```bash
$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard=old
$ nats con add ORDERS NEW --filter ORDERS.received --ack explicit --pull --deliver all --max-deliver=-1 --sample 100
$ nats con add ORDERS DISPATCH --filter ORDERS.processed --ack explicit --pull --deliver all --max-deliver=-1 --sample 100
$ nats con add ORDERS MONITOR --filter '' --ack none --target monitor.ORDERS --deliver last --replay instant
```

34
jetstream/concepts/consumers.md Normal file
View File

@ -0,0 +1,34 @@
# Consumers
Each Consumer, or related group of Consumers, of a Stream will need an Consumer defined. It's ok to define thousands of these pointing at the same Stream.
Consumers can either be `push` based where JetStream will deliver the messages as fast as possible to a subject of your choice or `pull` based for typical work queue like behavior. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging.
In the orders example above we have 3 Consumers. The first two select a subset of the messages from the Stream by specifying a specific subject like `ORDERS.processed`. The Stream consumes `ORDERS.*` and this allows you to receive just what you need. The final Consumer receives all messages in a `push` fashion.
Consumers track their progress, they know what messages were delivered, acknowledged, etc., and will redeliver messages they sent that were not acknowledged. When first created, the Consumer has to know what message to send as the first one. You can configure either a specific message in the set \(`StreamSeq`\), specific time \(`StartTime`\), all \(`DeliverAll`\) or last \(`DeliverLast`\). This is the starting point and from there, they all behave the same - delivering all of the following messages with optional Acknowledgement.
Acknowledgements default to `AckExplicit` - the only supported mode for pull-based Consumers - meaning every message requires a distinct acknowledgement. But for push-based Consumers, you can set `AckNone` that does not require any acknowledgement, or `AckAll` which quite interestingly allows you to acknowledge a specific message, like message `100`, which will also acknowledge messages `1` through `99`. The `AckAll` mode can be a great performance boost.
Some messages may cause your applications to crash and cause a never ending loop forever poisoning your system. The `MaxDeliver` setting allow you to set a upper bound to how many times a message may be delivered.
To assist with creating monitoring applications, one can set a `SampleFrequency` which is a percentage of messages for which the system should sample and create events. These events will include delivery counts and ack waits.
When defining Consumers the items below make up the entire configuration of the Consumer:
| Item | Description |
| :--- | :--- |
| AckPolicy | How messages should be acknowledged, `AckNone`, `AckAll` or `AckExplicit` |
| AckWait | How long to allow messages to remain un-acknowledged before attempting redelivery |
| DeliverPolicy | The initial starting mode of the consumer, `DeliverAll`, `DeliverLast`, `DeliverNew`, `DeliverByStartSequence` or `DeliverByStartTime` |
| DeliverySubject | The subject to deliver observed messages, when not set, a pull-based Consumer is created |
| Durable | The name of the Consumer |
| FilterSubject | When consuming from a Stream with many subjects, or wildcards, select only a specific incoming subjects, supports wildcards |
| MaxDeliver | Maximum amount times a specific message will be delivered. Use this to avoid poison pills crashing all your services forever |
| OptStartSeq | When first consuming messages from the Stream start at this particular message in the set |
| ReplayPolicy | How messages are sent `ReplayInstant` or `ReplayOriginal` |
| SampleFrequency | What percentage of acknowledgements should be samples for observability, 0-100 |
| OptStartTime | When first consuming messages from the Stream start with messages on or after this time |
| RateLimit | The rate of message delivery in bits per second |
| MaxAckPending | The maximum number of messages without acknowledgement that can be outstanding, once this limit is reached message delivery will be suspended |

32
jetstream/concepts/streams.md Normal file
View File

@ -0,0 +1,32 @@
# Streams
Streams define how messages are stored and retention duration. Streams consume normal NATS subjects, any message found on those subjects will be delivered to the defined storage system. You can do a normal publish to the subject for unacknowledged delivery, else if you send a Request to the subject the JetStream server will reply with an acknowledgement that it was stored.
As of January 2020, in the tech preview we have `file` and `memory` based storage systems, we do not yet support clustering.
In the diagram above we show the concept of storing all `ORDERS.*` in the Stream even though there are many types of order related messages. We'll show how you can selectively consume subsets of messages later. Relatively speaking the Stream is the most resource consuming component so being able to combine related data in this manner is important to consider.
Streams can consume many subjects. Here we have `ORDERS.*` but we could also consume `SHIPPING.state` into the same Stream should that make sense \(not shown here\).
Streams support various retention policies - they can be kept based on limits like max count, size or age but also more novel methods like keeping them as long as any Consumers have them unacknowledged, or work queue like behavior where a message is removed after first ack.
Streams support deduplication using a `Nats-Msg-Id` header and a sliding window within which to track duplicate messages. See the [Message Deduplication](../model_deep_dive.md#message-deduplication) section.
When defining Streams the items below make up the entire configuration of the set.
| Item | Description |
| :--- | :--- |
| MaxAge | Maximum age of any message in the stream, expressed in microseconds |
| MaxBytes | How big the Stream may be, when the combined stream size exceeds this old messages are removed |
| MaxMsgSize | The largest message that will be accepted by the Stream |
| MaxMsgs | How many messages may be in a Stream, oldest messages will be removed if the Stream exceeds this size |
| MaxConsumers | How many Consumers can be defined for a given Stream, `-1` for unlimited |
| Name | A name for the Stream that may not have spaces, tabs or `.` |
| NoAck | Disables acknowledging messages that are received by the Stream |
| Replicas | How many replicas to keep for each message in a clustered JetStream, maximum 5 |
| Retention | How message retention is considered, `LimitsPolicy` \(default\), `InterestPolicy` or `WorkQueuePolicy` |
| Discard | When a Stream reached it's limits either, `DiscardNew` refuses new messages while `DiscardOld` \(default\) deletes old messages |
| Storage | The type of storage backend, `file` and `memory` as of January 2020 |
| Subjects | A list of subjects to consume, supports wildcards |
| Duplicates | The window within which to track duplicate messages |

11
jetstream/configuration_mgmt/README.md Normal file
View File

@ -0,0 +1,11 @@
# Configuration Management
In many cases managing the configuration in your application code is the best model, many teams though wish to pre-create Streams and Consumers.
We support a number of tools to assist with this:
* [CLI with Configuration Files](nats-admin-cli.md)
* [Terraform](terraform.md)
* [GitHub Actions](github_actions.md)
* [Kubernetes JetStream Controller](kubernetes_controller.md)

56
jetstream/configuration_mgmt/github_actions.md Normal file
View File

@ -0,0 +1,56 @@
# GitHub Actions
We have a pack of GitHub Actions that let you manage an already running JetStream Server, useful for managing releases or standing up test infrastructure.
Full details and examples are in the [jetstream-gh-actions](https://github.com/nats-io/jetstream-gh-action) repository, here's an example.
```yaml
on: push
name: orders
jobs:
# First we delete the ORDERS stream and consumer if they already exist
clean_orders:
runs-on: ubuntu-latest
steps:
- name: orders_stream
uses: nats-io/jetstream-gh-action/delete/stream@master
with:
missing_ok: 1
stream: ORDERS
server: js.example.net
# Now we create the Stream and Consumers using the same configuration files the
# nats CLI utility would use as shown above
create_orders:
runs-on: ubuntu-latest
needs: clean_orders
steps:
- uses: actions/checkout@master
- name: orders_stream
uses: nats-io/jetstream-gh-action/create/stream@master
with:
config: ORDERS.json
server: js.example.net
- name: orders_new_consumer
uses: nats-io/jetstream-gh-action/create/consumer@master
with:
config: ORDERS_NEW.json
stream: ORDERS
server: js.example.net
# We publish a message to a specific Subject, perhaps some consumer is
# waiting there for it to kick off tests
publish_message:
runs-on: ubuntu-latest
needs: create_orders
steps:
- uses: actions/checkout@master
- name: orders_new_consumer
uses: nats-io/jetstream-gh-action@master
with:
subject: ORDERS.deployment
message: Published new deployment via "${{ github.event_name }}" in "${{ github.repository }}"
server: js.example.net
```

58
jetstream/configuration_mgmt/kubernetes_controller.md Normal file
View File

@ -0,0 +1,58 @@
# Kubernetes Controller
The JetStream controllers allow you to manage NATS JetStream Streams and Consumers via K8S CRDs. You can find more info on how to deploy and usage [here](https://github.com/nats-io/nack#getting-started). Below you can find an example of how to create a stream and a couple of consumers:
```yaml
---
apiVersion: jetstream.nats.io/v1beta1
kind: Stream
metadata:
name: mystream
spec:
name: mystream
subjects: ["orders.*"]
storage: memory
maxAge: 1h
---
apiVersion: jetstream.nats.io/v1beta1
kind: Consumer
metadata:
name: my-push-consumer
spec:
streamName: mystream
durableName: my-push-consumer
deliverSubject: my-push-consumer.orders
deliverPolicy: last
ackPolicy: none
replayPolicy: instant
---
apiVersion: jetstream.nats.io/v1beta1
kind: Consumer
metadata:
name: my-pull-consumer
spec:
streamName: mystream
durableName: my-pull-consumer
deliverPolicy: all
filterSubject: orders.received
maxDeliver: 20
ackPolicy: explicit
```
Once the CRDs are installed you can use `kubectl` to manage the streams and consumers as follows:
```bash
$ kubectl get streams
NAME STATE STREAM NAME SUBJECTS
mystream Created mystream [orders.*]
$ kubectl get consumers
NAME STATE STREAM CONSUMER ACK POLICY
my-pull-consumer Created mystream my-pull-consumer explicit
my-push-consumer Created mystream my-push-consumer none
# If you end up in an Errored state, run kubectl describe for more info.
# kubectl describe streams mystream
# kubectl describe consumers my-pull-consumer
```

30
jetstream/configuration_mgmt/nats-admin-cli.md Normal file
View File

@ -0,0 +1,30 @@
# NATS Admin CLI
## nats Admin CLI
The `nats` CLI can be used to manage Streams and Consumers easily using it's `--config` flag, for example:
## Add a new Stream
This creates a new Stream based on `orders.json`. The `orders.json` file can be extracted from an existing stream using `nats stream info ORDERS -j | jq .config`
```text
$ nats str add ORDERS --config orders.json
```
## Edit an existing Stream
This edits an existing stream ensuring it complies with the configuration in `orders.json`
```text
$ nats str edit ORDERS --config orders.json
```
## Add a New Consumer
This creates a new Consumer based on `orders_new.json`. The `orders_new.json` file can be extracted from an existing stream using `nats con info ORDERS NEW -j | jq .config`
```text
$ nats con add ORDERS NEW --config orders_new.json
```

68
jetstream/configuration_mgmt/terraform.md Normal file
View File

@ -0,0 +1,68 @@
# Terraform
Terraform is a Cloud configuration tool from Hashicorp found at [terraform.io](https://www.terraform.io/), we maintain a Provider for Terraform called [terraform-provider-jetstream](https://github.com/nats-io/terraform-provider-jetstream/) that can maintain JetStream using Terraform.
### Setup
Our provider is not hosted by Hashicorp so installation is a bit more complex than typical. Browse to the [Release Page](https://github.com/nats-io/terraform-provider-jetstream/releases) and download the release for your platform and extract it into your Terraform plugins directory.
```text
$ unzip -l terraform-provider-jetstream_0.0.2_darwin_amd64.zip
Archive: terraform-provider-jetstream_0.0.2_darwin_amd64.zip
Length Date Time Name
--------- ---------- ----- ----
11357 03-09-2020 10:48 LICENSE
1830 03-09-2020 12:53 README.md
24574336 03-09-2020 12:54 terraform-provider-jetstream_v0.0.2
```
Place the `terraform-provider-jetstream_v0.0.2` file in `~/.terraform.d/plugins/terraform-provider-jetstream_v0.0.2`
In your project you can configure the Provider like this:
```text
provider "jetstream" {
servers = "connect.ngs.global"
credentials = "ngs_jetstream_admin.creds"
}
```
And start using it, here's an example that create the `ORDERS` example. Review the [Project README](https://github.com/nats-io/terraform-provider-jetstream#readme) for full details.
```text
resource "jetstream_stream" "ORDERS" {
name = "ORDERS"
subjects = ["ORDERS.*"]
storage = "file"
max_age = 60 * 60 * 24 * 365
}
resource "jetstream_consumer" "ORDERS_NEW" {
stream_id = jetstream_stream.ORDERS.id
durable_name = "NEW"
deliver_all = true
filter_subject = "ORDERS.received"
sample_freq = 100
}
resource "jetstream_consumer" "ORDERS_DISPATCH" {
stream_id = jetstream_stream.ORDERS.id
durable_name = "DISPATCH"
deliver_all = true
filter_subject = "ORDERS.processed"
sample_freq = 100
}
resource "jetstream_consumer" "ORDERS_MONITOR" {
stream_id = jetstream_stream.ORDERS.id
durable_name = "MONITOR"
deliver_last = true
ack_policy = "none"
delivery_subject = "monitor.ORDERS"
}
output "ORDERS_SUBJECTS" {
value = jetstream_stream.ORDERS.subjects
}
```

86
jetstream/disaster_recovery.md Normal file
View File

@ -0,0 +1,86 @@
# Disaser Recovery
Disaster Recovery of the JetStream system is a topic we are still exploring and fleshing out and that will be impacted by the clustering work. For example replication will extend the options available to you.
Today we have a few approaches to consider:
* `nats` CLI + Configuration Backups + Data Snapshots
* Configuration Management + Data Snapshots
## Data Backup
In all scenarios you can perform data snapshots and restores over the NATS protocol. This is good if you do not manage the NATS servers hosting your data, and you wish to do a backup of your data.
The backup includes:
* Stream configuration and state
* Stream Consumer configuration and state
* All data including metadata like timestamps and headers
```text
$ nats stream backup ORDERS /data/js-backup/ORDERS.tgz
Starting backup of Stream "ORDERS" with 13 data blocks
2.4 MiB/s [====================================================================] 100%
Received 13 MiB bytes of compressed data in 3368 chunks for stream "ORDERS" in 1.223428188s, 813 MiB uncompressed
```
During the backup the Stream is in a state where it's configuration cannot change and no data will be expired from it based on Limits or Retention Policies.
Progress using the terminal bar can be disabled using `--no-progress`, it will then issue log lines instead.
## Restoring Data
The backup made above can be restored into another server - but into the same Stream name.
```text
$ nats str restore ORDERS /data/js-backup/ORDERS.tgz
Starting restore of Stream "ORDERS" from file "/data/js-backup/ORDERS.tgz"
13 MiB/s [====================================================================] 100%
Restored stream "ORDERS" in 937.071149ms
Information for Stream ORDERS
Configuration:
Subjects: ORDERS.>
...
```
The `/data/js-backup/ORDERS.tgz` file can also be extracted into the data dir of a stopped NATS Server.
Progress using the terminal bar can be disabled using `--no-progress`, it will then issue log lines instead.
## Interactive CLI
In environments where the `nats` CLI is used interactively to configure the server you do not have a desired state to recreate the server from. This is not the ideal way to administer the server, we recommend Configuration Management, but many will use this approach.
Here you can back up the configuration into a directory from where you can recover the configuration later. The data for File backed stores can also be backed up.
```text
$ nats backup /data/js-backup
15:56:11 Creating JetStream backup into /data/js-backup
15:56:11 Stream ORDERS to /data/js-backup/stream_ORDERS.json
15:56:11 Consumer ORDERS > NEW to /data/js-backup/stream_ORDERS_consumer_NEW.json
15:56:11 Configuration backup complete
```
This backs up Stream, Consumer and Stream Template configuration.
During the same process the data can also be backed up by passing `--data`, this will create files like `/data/js-backup/stream_ORDERS.tgz`.
Later the data can be restored, for Streams we support editing the Stream configuration in place to match what was in the backup.
```text
$ nats restore /tmp/backup --update-streams
15:57:42 Reading file /tmp/backup/stream_ORDERS.json
15:57:42 Reading file /tmp/backup/stream_ORDERS_consumer_NEW.json
15:57:42 Updating Stream ORDERS configuration
15:57:42 Restoring Consumer ORDERS > NEW
```
The `nats restore` tool does not support restoring data, the same process using `nats stream restore`, as outlined earlier, can be used which will also restore Stream and Consumer configurations and state.

27
jetstream/getting_started/README.md Normal file
View File

@ -0,0 +1,27 @@
# Getting Started
Getting started with JetStream is straightforward. While we speak of JetStream as if it is a separate component, it's actually a subsystem built into the NATS server that needs to be enabled.
## Command Line
Enable JetStream by specifying the `-js` flag when starting the NATS server.
`$ nats-server -js`
## Configuration File
You can also enable JetStream through a configuration file. By default, the JetStream subsytem will store data in the /tmp directory. Here's a minimal file that will store data in a local "nats" directory, suitable for development and local testing.
`$ nats-server -c js.conf`
```text
# js.conf
jetstream {
store_dir=nats
}
```
Normally JetStream will be run in clustered mode and will replicate data, so the best place to store JetStream data would be locally on a fast SSD. One should specifically avoid NAS or NFS storage for JetStream.
See [Using Docker](using_docker.md) and [Using Source](using_source.md) for more information.

41
jetstream/getting_started/using_docker.md Normal file
View File

@ -0,0 +1,41 @@
# Using Docker
The `natsio/nats-box:latest` docker image contains the `nats` utility this guide covers.
In one window start a JetStream enabled nats server:
```text
$ docker run --network host -p 4222:4222 nats -js
```
And in another log into the utilities:
```text
$ docker run -ti --network host natsio/nats-box
```
This shell has the `nats` utility and all other NATS cli tools used in the rest of this guide.
Now skip to the `Administer JetStream` section.
## Using Docker with NGS
You can join a JetStream instance to your [NGS](https://synadia.com/ngs/pricing) account, first we need a credential for testing JetStream:
You'll want to do this outside of docker to keep the credentials that are generated.
```text
$ nsc add user -a YourAccount --name leafnode --expiry 1M
```
You'll get a credential file somewhere like `~/.nkeys/creds/synadia/YourAccount/leafnode.creds`, mount this file into the docker container for JetStream using `-v ~/.nkeys/creds/synadia/YourAccount/leafnode.creds:/leafnode.creds`.
```text
$ docker run -ti -v ~/.nkeys/creds/synadia/YourAccount/leafnode.creds:/leafnode.creds --name jetstream synadia/jsm:latest server
[1] 2020/01/20 12:44:11.752465 [INF] Starting nats-server version 2.2.0
...
[1] 2020/01/20 12:55:01.849033 [INF] Connected leafnode to "connect.ngs.global"
```
Your JSM shell will still connect locally, other connections in your NGS account can use JetStream at this point.

66
jetstream/getting_started/using_source.md Normal file
View File

@ -0,0 +1,66 @@
# Using Source
You will also want to have installed from the nats.go repo the examples/tools such as nats-pub, nats-sub, nats-req and possibly nats-bench. One of the design goals of JetStream was to be native to core NATS, so even though we will most certainly add in syntactic sugar to clients to make them more appealing, for this tech preview we will be using plain old NATS.
You will need a copy of the nats-server source locally and will need to be in the jetstream branch.
```text
$ git clone https://github.com/nats-io/nats-server.git
$ cd nats-server
$ git checkout master
$ go build
$ ls -l nats-server
```
Starting the server you can use the `-js` flag. This will setup the server to reasonably use memory and disk. This is a sample run on my machine. JetStream will default to 1TB of disk and 75% of available memory for now.
```text
$ ./nats-server -js
[16928] 2019/12/04 19:16:29.596968 [INF] Starting nats-server version 2.2.0
[16928] 2019/12/04 19:16:29.597056 [INF] Git commit [not set]
[16928] 2019/12/04 19:16:29.597072 [INF] Starting JetStream
[16928] 2019/12/04 19:16:29.597444 [INF] ----------- JETSTREAM (Beta) -----------
[16928] 2019/12/04 19:16:29.597451 [INF] Max Memory: 96.00 GB
[16928] 2019/12/04 19:16:29.597454 [INF] Max Storage: 1.00 TB
[16928] 2019/12/04 19:16:29.597461 [INF] Store Directory: "/var/folders/m0/k03vs55n2b54kdg7jm66g27h0000gn/T/jetstream"
[16928] 2019/12/04 19:16:29.597469 [INF] ----------------------------------------
[16928] 2019/12/04 19:16:29.597732 [INF] Listening for client connections on 0.0.0.0:4222
[16928] 2019/12/04 19:16:29.597738 [INF] Server id is NAJ5GKP5OBVISP5MW3BFAD447LMTIOAHFEWMH2XYWLL5STVGN3MJHTXQ
[16928] 2019/12/04 19:16:29.597742 [INF] Server is ready
```
You can override the storage directory if you want.
```text
$ ./nats-server -js -sd /tmp/test
[16943] 2019/12/04 19:20:00.874148 [INF] Starting nats-server version 2.2.0
[16943] 2019/12/04 19:20:00.874247 [INF] Git commit [not set]
[16943] 2019/12/04 19:20:00.874273 [INF] Starting JetStream
[16943] 2019/12/04 19:20:00.874605 [INF] ----------- JETSTREAM (Beta) -----------
[16943] 2019/12/04 19:20:00.874613 [INF] Max Memory: 96.00 GB
[16943] 2019/12/04 19:20:00.874615 [INF] Max Storage: 1.00 TB
[16943] 2019/12/04 19:20:00.874620 [INF] Store Directory: "/tmp/test/jetstream"
[16943] 2019/12/04 19:20:00.874625 [INF] ----------------------------------------
[16943] 2019/12/04 19:20:00.874868 [INF] Listening for client connections on 0.0.0.0:4222
[16943] 2019/12/04 19:20:00.874874 [INF] Server id is NCR6KDDGWUU2FXO23WAXFY66VQE6JNWVMA24ALF2MO5GKAYFIMQULKUO
[16943] 2019/12/04 19:20:00.874877 [INF] Server is ready
```
These options can also be set in your configuration file:
```text
// enables jetstream, an empty block will enable and use defaults
jetstream {
// jetstream data will be in /data/nats-server/jetstream
store_dir: "/data/nats-server"
// 1GB
max_memory_store: 1073741824
// 10GB
max_file_store: 10737418240
}
```

35
jetstream/jetstream.md Normal file
View File

@ -0,0 +1,35 @@
# About Jetstream
JetStream was created to solve the problems identified with streaming in technology today - complexity, fragility, and a lack of scalability. Some technologies address these better than others, but no current streaming technology is truly multi-tenant, horizontally scalable, and supports multiple deployment models. No technology we are aware of can scale from edge to cloud under the same security context while having complete deployment observability for operations.
## Goals
JetStream was developed with the following goals in mind:
* The system must be easy to configure and operate and be observable.
* The system must be secure and operate well with NATS 2.0 security models.
* The system must scale horizontally and be applicable to a high ingestion rate.
* The system must support multiple use cases.
* The system must self heal and always be available.
* The system must have an API that is closer to core NATS.
* The system must allow NATS messages to be part of a stream as desired.
* The system must display payload agnostic behavior.
* The system must not have third party dependencies.
## High-Level Design and Features
In terms of deployment, a JetStream server is simply a NATS server with the JetStream subsystem enabled, launched with the `-js` flag with a configured server name and cluster name. From a client perspective, it does not matter which servers are running JetStream so long as there is some route to a JetStream enabled server or servers. This allows for a flexible deployment which to optimize resources for particular servers that will store streams versus very low overhead stateless servers, reducing OpEx and ultimately creating a scalable and manageable system.
## Feature List
* At-least-once delivery; exactly once within a window
* Store messages and replay by time or sequence
* Wildcard support
* Account aware
* Data at rest encryption
* Cleanse specific messages \(GDPR\)
* Horizontal scalability
* Persist Streams and replay via Consumers
JetStream is designed to bifurcate ingestion and consumption of messages to provide multiple ways to consume data from the same stream. To that end, JetStream functionality is composed of server streams and server consumers.

481
jetstream/model_deep_dive.md Normal file
View File

@ -0,0 +1,481 @@
# Model Deep Dive
The Orders example touched on a lot of features, but some like different Ack models and message limits, need a bit more detail. This section will expand on the above and fill in some blanks.
## Stream Limits, Retention, and Policy
Streams store data on disk, but we cannot store all data forever so we need ways to control their size automatically.
There are 3 features that come into play when Streams decide how long they store data.
The `Retention Policy` describes based on what criteria a set will evict messages from its storage:
| Retention Policy | Description |
| :--- | :--- |
| `LimitsPolicy` | Limits are set for how many messages, how big the storage and how old messages may be |
| `WorkQueuePolicy` | Messages are kept until they were consumed by any one single observer and then removed |
| `InterestPolicy` | Messages are kept as long as there are Consumers active for them |
In all Retention Policies the basic limits apply as upper bounds, these are `MaxMsgs` for how many messages are kept in total, `MaxBytes` for how big the set can be in total and `MaxAge` for what is the oldest message that will be kept. These are the only limits in play with `LimitsPolicy` retention.
One can then define additional ways a message may be removed from the Stream earlier than these limits. In `WorkQueuePolicy` the messages will be removed as soon as any Consumer received an Acknowledgement. In `InterestPolicy` messages will be removed as soon as there are no more Consumers.
In both `WorkQueuePolicy` and `InterestPolicy` the age, size and count limits will still apply as upper bounds.
A final control is the Maximum Size any single message may have. NATS have it's own limit for maximum size \(1 MiB by default\), but you can say a Stream will only accept messages up to 1024 bytes using `MaxMsgSize`.
The `Discard Policy` sets how messages are discarded when limits set by `LimitsPolicy` are reached. The `DiscardOld` option removes old messages making space for new, while `DiscardNew` refuses any new messages.
The `WorkQueuePolicy` mode is a specialized mode where a message, once consumed and acknowledged, is discarded from the Stream. In this mode, there are a few limits on consumers. Inherently it's about 1 message to one consumer, this means you cannot have overlapping consumers defined on the Stream - needs unique filter subjects.
## Message Deduplication
JetStream support idempotent message writes by ignoring duplicate messages as indicated by the `Nats-Msg-Id` header.
```text
% nats req -H Nats-Msg-Id:1 ORDERS.new hello1
% nats req -H Nats-Msg-Id:1 ORDERS.new hello2
% nats req -H Nats-Msg-Id:1 ORDERS.new hello3
% nats req -H Nats-Msg-Id:1 ORDERS.new hello4
```
Here we set a `Nats-Msg-Id:1` header which tells JetStream to ensure we do not have duplicates of this message - we only consult the message ID not the body.
```text
$ nats str info ORDERS
....
State:
Messages: 1
Bytes: 67 B
```
The default window to track duplicates in is 2 minutes, this can be set on the command line using `--dupe-window` when creating a stream, though we would caution against large windows.
## Acknowledgement Models
Streams support acknowledging receiving a message, if you send a `Request()` to a subject covered by the configuration of the Stream the service will reply to you once it stored the message. If you just publish, it will not. A Stream can be set to disable Acknowledgements by setting `NoAck` to `true` in it's configuration.
Consumers have 3 acknowledgement modes:
| Mode | Description |
| :--- | :--- |
| `AckExplicit` | This requires every message to be specifically acknowledged, it's the only supported option for pull-based Consumers |
| `AckAll` | In this mode if you acknowledge message `100` it will also acknowledge message `1`-`99`, this is good for processing batches and to reduce ack overhead |
| `AckNone` | No acknowledgements are supported |
To understand how Consumers track messages we will start with a clean `ORDERS` Stream and `DISPATCH` Consumer.
```text
$ nats str info ORDERS
...
Statistics:
Messages: 0
Bytes: 0 B
FirstSeq: 0
LastSeq: 0
Active Consumers: 1
```
The Set is entirely empty
```text
$ nats con info ORDERS DISPATCH
...
State:
Last Delivered Message: Consumer sequence: 1 Stream sequence: 1
Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0
Pending Messages: 0
Redelivered Messages: 0
```
The Consumer has no messages outstanding and has never had any \(Consumer sequence is 1\).
We publish one message to the Stream and see that the Stream received it:
```text
$ nats pub ORDERS.processed "order 4"
Published 7 bytes to ORDERS.processed
$ nats str info ORDERS
...
Statistics:
Messages: 1
Bytes: 53 B
FirstSeq: 1
LastSeq: 1
Active Consumers: 1
```
As the Consumer is pull-based, we can fetch the message, ack it, and check the Consumer state:
```text
$ nats con next ORDERS DISPATCH
--- received on ORDERS.processed
order 4
Acknowledged message
$ nats con info ORDERS DISPATCH
...
State:
Last Delivered Message: Consumer sequence: 2 Stream sequence: 2
Acknowledgment floor: Consumer sequence: 1 Stream sequence: 1
Pending Messages: 0
Redelivered Messages: 0
```
The message got delivered and acknowledged - `Acknowledgement floor` is `1` and `1`, the sequence of the Consumer is `2` which means its had only the one message through and got acked. Since it was acked, nothing is pending or redelivering.
We'll publish another message, fetch it but not Ack it this time and see the status:
```text
$ nats pub ORDERS.processed "order 5"
Published 7 bytes to ORDERS.processed
$ nats con next ORDERS DISPATCH --no-ack
--- received on ORDERS.processed
order 5
$ nats con info ORDERS DISPATCH
State:
Last Delivered Message: Consumer sequence: 3 Stream sequence: 3
Acknowledgment floor: Consumer sequence: 1 Stream sequence: 1
Pending Messages: 1
Redelivered Messages: 0
```
Now we can see the Consumer has processed 2 messages \(obs sequence is 3, next message will be 3\) but the Ack floor is still 1 - thus 1 message is pending acknowledgement. Indeed this is confirmed in the `Pending messages`.
If I fetch it again and again do not ack it:
```text
$ nats con next ORDERS DISPATCH --no-ack
--- received on ORDERS.processed
order 5
$ nats con info ORDERS DISPATCH
State:
Last Delivered Message: Consumer sequence: 4 Stream sequence: 3
Acknowledgment floor: Consumer sequence: 1 Stream sequence: 1
Pending Messages: 1
Redelivered Messages: 1
```
The Consumer sequence increases - each delivery attempt increases the sequence - and our redelivered count also goes up.
Finally, if I then fetch it again and ack it this time:
```text
$ nats con next ORDERS DISPATCH
--- received on ORDERS.processed
order 5
Acknowledged message
$ nats con info ORDERS DISPATCH
State:
Last Delivered Message: Consumer sequence: 5 Stream sequence: 3
Acknowledgment floor: Consumer sequence: 1 Stream sequence: 1
Pending Messages: 0
Redelivered Messages: 0
```
Having now Acked the message there are no more pending.
Additionally, there are a few types of acknowledgements:
| Type | Bytes | Description |
| :--- | :--- | :--- |
| `AckAck` | nil, `+ACK` | Acknowledges a message was completely handled |
| `AckNak` | `-NAK` | Signals that the message will not be processed now and processing can move onto the next message, NAK'd message will be retried |
| `AckProgress` | `+WPI` | When sent before the AckWait period indicates that work is ongoing and the period should be extended by another equal to `AckWait` |
| `AckNext` | `+NXT` | Acknowledges the message was handled and requests delivery of the next message to the reply subject. Only applies to Pull-mode. |
| `AckTerm` | `+TERM` | Instructs the server to stop redelivery of a message without acknowledging it as successfully processed |
So far all of the examples were the `AckAck` type of acknowledgement, by replying to the Ack with the body as indicated in `Bytes` you can pick what mode of acknowledgement you want.
All of these acknowledgement modes, except `AckNext`, support double acknowledgement - if you set a reply subject when acknowledging the server will in turn acknowledge having received your ACK.
The `+NXT` acknowledgement can have a few formats: `+NXT 10` requests 10 messages and `+NXT {"no_wait": true}` which is the same data that can be sent in a Pull Request.
## Exactly Once Delivery
JetStream supports Exactly Once delivery by combining Message Deduplication and double acks.
On the publishing side you can avoid duplicate message ingestion using the [Message Deduplication](model_deep_dive.md#message-deduplication) feature.
Consumers can be 100% sure a message was correctly processed by requesting the server Acknowledge having received your acknowledgement by setting a reply subject on the Ack. If you receive this response you will never receive that message again.
## Consumer Starting Position
When setting up a Consumer you can decide where to start, the system supports the following for the `DeliverPolicy`:
| Policy | Description |
| :--- | :--- |
| `all` | Delivers all messages that are available |
| `last` | Delivers the latest message, like a `tail -n 1 -f` |
| `new` | Delivers only new messages that arrive after subscribe time |
| `by_start_time` | Delivers from a specific time onward. Requires `OptStartTime` to be set |
| `by_start_sequence` | Delivers from a specific stream sequence. Requires `OptStartSeq` to be set |
Regardless of what mode you set, this is only the starting point. Once started it will always give you what you have not seen or acknowledged. So this is merely how it picks the very first message.
Let's look at each of these, first we make a new Stream `ORDERS` and add 100 messages to it.
Now create a `DeliverAll` pull-based Consumer:
```text
$ nats con add ORDERS ALL --pull --filter ORDERS.processed --ack none --replay instant --deliver all
$ nats con next ORDERS ALL
--- received on ORDERS.processed
order 1
Acknowledged message
```
Now create a `DeliverLast` pull-based Consumer:
```text
$ nats con add ORDERS LAST --pull --filter ORDERS.processed --ack none --replay instant --deliver last
$ nats con next ORDERS LAST
--- received on ORDERS.processed
order 100
Acknowledged message
```
Now create a `MsgSetSeq` pull-based Consumer:
```text
$ nats con add ORDERS TEN --pull --filter ORDERS.processed --ack none --replay instant --deliver 10
$ nats con next ORDERS TEN
--- received on ORDERS.processed
order 10
Acknowledged message
```
And finally a time-based Consumer. Let's add some messages a minute apart:
```text
$ nats str purge ORDERS
$ for i in 1 2 3
do
nats pub ORDERS.processed "order ${i}"
sleep 60
done
```
Then create a Consumer that starts 2 minutes ago:
```text
$ nats con add ORDERS 2MIN --pull --filter ORDERS.processed --ack none --replay instant --deliver 2m
$ nats con next ORDERS 2MIN
--- received on ORDERS.processed
order 2
Acknowledged message
```
## Ephemeral Consumers
So far, all the Consumers you have seen were Durable, meaning they exist even after you disconnect from JetStream. In our Orders scenario, though the `MONITOR` a Consumer could very well be a short-lived thing there just while an operator is debugging the system, there is no need to remember the last seen position if all you are doing is wanting to observe the real-time state.
In this case, we can make an Ephemeral Consumer by first subscribing to the delivery subject, then creating a durable and giving it no durable name. An Ephemeral Consumer exists as long as any subscription is active on its delivery subject. It is automatically be removed, after a short grace period to handle restarts, when there are no subscribers.
Ephemeral Consumers can only be push-based.
Terminal 1:
```text
$ nats sub my.monitor
```
Terminal 2:
```text
$ nats con add ORDERS --filter '' --ack none --target 'my.monitor' --deliver last --replay instant --ephemeral
```
The `--ephemeral` switch tells the system to make an Ephemeral Consumer.
## Consumer Message Rates
Typically what you want is if a new Consumer is made the selected messages are delivered to you as quickly as possible. You might want to replay messages at the rate they arrived though, meaning if messages first arrived 1 minute apart and you make a new Consumer it will get the messages a minute apart.
This is useful in load testing scenarios etc. This is called the `ReplayPolicy` and have values of `ReplayInstant` and `ReplayOriginal`.
You can only set `ReplayPolicy` on push-based Consumers.
```text
$ nats con add ORDERS REPLAY --target out.original --filter ORDERS.processed --ack none --deliver all --sample 100 --replay original
...
Replay Policy: original
...
```
Now let's publish messages into the Set 10 seconds apart:
```text
$ for i in 1 2 3 <15:15:35
do
nats pub ORDERS.processed "order ${i}"
sleep 10
done
Published [ORDERS.processed] : 'order 1'
Published [ORDERS.processed] : 'order 2'
Published [ORDERS.processed] : 'order 3'
```
And when we consume them they will come to us 10 seconds apart:
```text
$ nats sub -t out.original
Listening on [out.original]
2020/01/03 15:17:26 [#1] Received on [ORDERS.processed]: 'order 1'
2020/01/03 15:17:36 [#2] Received on [ORDERS.processed]: 'order 2'
2020/01/03 15:17:46 [#3] Received on [ORDERS.processed]: 'order 3'
^C
```
## Stream Templates
When you have many similar streams it can be helpful to auto-create them, let's say you have a service by client and they are on subjects `CLIENT.*`, you can construct a template that will auto-generate streams for any matching traffic.
```text
$ nats str template add CLIENTS --subjects "CLIENT.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size 2048 --max-streams 1024 --discard old
Stream Template CLIENTS was created
Information for Stream Template CLIENTS
Configuration:
Subjects: CLIENT.*
Acknowledgements: true
Retention: File - Limits
Replicas: 1
Maximum Messages: -1
Maximum Bytes: -1
Maximum Age: 8760h0m0s
Maximum Message Size: 2048
Maximum Consumers: -1
Maximum Streams: 1024
Managed Streams:
No Streams have been defined by this template
```
You can see no streams currently exist, let's publish some data:
```text
$ nats pub CLIENT.acme hello
```
And we'll have 1 new Stream:
```text
$ nats str ls
Streams:
CLIENTS_acme
```
When the template is deleted all the streams it created will be deleted too.
## Ack Sampling
In the earlier sections we saw that samples are being sent to a monitoring system. Let's look at that in depth; how the monitoring system works and what it contains.
As messages pass through a Consumer you'd be interested in knowing how many are being redelivered and how many times but also how long it takes for messages to be acknowledged.
Consumers can sample Ack'ed messages for you and publish samples so your monitoring system can observe the health of a Consumer. We will add support for this to [NATS Surveyor](https://github.com/nats-io/nats-surveyor).
### Configuration
You can configure a Consumer for sampling bypassing the `--sample 80` option to `nats consumer add`, this tells the system to sample 80% of Acknowledgements.
When viewing info of a Consumer you can tell if it's sampled or not:
```text
$ nats con info ORDERS NEW
...
Sampling Rate: 100
...
```
### Consuming
Samples are published to `$JS.EVENT.METRIC.CONSUMER_ACK.<stream>.<consumer>` in JSON format containing `api.ConsumerAckMetric`. Use the `nats con events` command to view samples:
```text
$ nats con events ORDERS NEW
Listening for Advisories on $JS.EVENT.ADVISORY.*.ORDERS.NEW
Listening for Metrics on $JS.EVENT.METRIC.*.ORDERS.NEW
15:08:31] [Ph0fsiOKRg1TS0c2k0mMz2] Acknowledgement Sample
Consumer: ORDERS > NEW
Stream Sequence: 40
Consumer Sequence: 161
Deliveries: 1
Delay: 1.009ms
```
```text
$ nats con events ORDERS NEW --json
{
"stream": "ORDERS",
"consumer": "NEW",
"consumer_seq": 155,
"stream_seq": 143,
"ack_time": 5387000,
"delivered": 1
}
{
"stream": "ORDERS",
"consumer": "NEW",
"consumer_seq": 156,
"stream_seq": 144,
"ack_time": 5807800,
"delivered": 1
}
```
## Storage Overhead
JetStream file storage is very efficient, storing as little extra information about the message as possible.
**NOTE:** This might change once clustering is supported.
We do store some message data with each message, namely:
* Message headers
* The subject it was received on
* The time it was received
* The message payload
* A hash of the message
* The message sequence
* A few other bits like the length of the subject and the length of headers
Without any headers the size is:
```text
length of the message record (4bytes) + seq(8) + ts(8) + subj_len(2) + subj + msg + hash(8)
```
A 5 byte `hello` message without headers will take 39 bytes.
With headers:
```text
length of the message record (4bytes) + seq(8) + ts(8) + subj_len(2) + subj + hdr_len(4) + hdr + msg + hash(8)
```
So if you are publishing many small messages the overhead will be, relatively speaking, quite large, but for larger messages the overhead is very small. If you publish many small messages it's worth trying to optimize the subject length.

35
jetstream/monitoring.md Normal file
View File

@ -0,0 +1,35 @@
# Monitoring
## Server Metrics
Typically, NATS is monitored via HTTP endpoints like `/varz`, we do not at this moment have a JetStream equivelant, but it's planned that server and account level metrics will be made available.
## Advisories
JetStream publish a number of advisories that can inform operations about health and state of the Streams. These advisories are published to normal NATS subjects below `$JS.EVENT.ADVISORY.>` and one can store these advisories in JetStream Streams if desired.
The command `nats event --js-advisory` can view all these events on your console. The Golang package [jsm.go](https://github.com/nats-io/jsm.go) can consume and render these events and have data types for each of these events.
All these events have JSON Schemas that describe them, schemas can be viewed on the CLI using the `nats schema show <schema kind>` command.
| Description | Subject | Kind |
| :--- | :--- | :--- |
| API interactions | `$JS.EVENT.ADVISORY.API` | `io.nats.jetstream.advisory.v1.api_audit` |
| Stream CRUD operations | `$JS.EVENT.ADVISORY.STREAM.CREATED.<STREAM>` | `io.nats.jetstream.advisory.v1.stream_action` |
| Consumer CRUD operations | `$JS.EVENT.ADVISORY.CONSUMER.CREATED.<STREAM>.<CONSUMER>` | `io.nats.jetstream.advisory.v1.consumer_action` |
| Snapshot started using `nats stream backup` | `$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_CREATE.<STREAM>` | `io.nats.jetstream.advisory.v1.snapshot_create` |
| Snapshot completed | `$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_COMPLETE.<STREAM>` | `io.nats.jetstream.advisory.v1.snapshot_complete` |
| Restore started using `nats stream restore` | `$JS.EVENT.ADVISORY.STREAM.RESTORE_CREATE.<STREAM>` | `io.nats.jetstream.advisory.v1.restore_create` |
| Restore completed | `$JS.EVENT.ADVISORY.STREAM.RESTORE_COMPLETE.<STREAM>` | `io.nats.jetstream.advisory.v1.restore_complete` |
| Consumer maximum delivery reached | `$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.<STREAM>.<CONSUMER>` | `io.nats.jetstream.advisory.v1.max_deliver` |
| Message delivery terminated using AckTerm | `$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.<STREAM>.<CONSUMER>` | `io.nats.jetstream.advisory.v1.terminated` |
| Message acknowledged in a sampled Consumer | `$JS.EVENT.METRIC.CONSUMER.ACK.<STREAM>.<CONSUMER>` | `io.nats.jetstream.metric.v1.consumer_ack` |
| Clustered Stream elected a new leader | `$JS.EVENT.ADVISORY.STREAM.LEADER_ELECTED.<STREAM>` | `io.nats.jetstream.advisory.v1.stream_leader_elected` |
| Clustered Stream lost quorum | `$JS.EVENT.ADVISORY.STREAM.QUORUM_LOST.<STREAM>` | `io.nats.jetstream.advisory.v1.stream_quorum_lost` |
| Clustered Consumer elected a new leader | `$JS.EVENT.ADVISORY.CONSUMER.LEADER_ELECTED.<STREAM>.<CONSUMER>` | `io.nats.jetstream.advisory.v1.consumer_leader_elected` |
| Clustered Consumer lost quorum | `$JS.EVENT.ADVISORY.CONSUMER.QUORUM_LOST.<STREAM>.<CONSUMER>` | `io.nats.jetstream.advisory.v1.consumer_quorum_lost` |
## Dashboards
The [NATS Surveyor](https://github.com/nats-io/nats-surveyor) system has initial support for passing JetStream metrics to Prometheus, dashboards and more will be added towards final release.

236
jetstream/nats_api_reference.md Normal file
View File

@ -0,0 +1,236 @@
# NATS API Reference
Thus far we saw a lot of CLI interactions. The CLI works by sending and receiving specially crafted messages over core NATS to configure the JetStream system. In time we will look to add file based configuration but for now the only method is the NATS API.
**NOTE:** Some NATS client libraries may need to enable an option to use old style requests when interacting with the JetStream server. Consult the libraries README's for more information.
## Reference
All of these subjects are found as constants in the NATS Server source, so for example the `$JS.API.STREAM.LIST` is a constant in the nats-server source `api.JetStreamListStreams` tables below will reference these constants and likewise data structures in the server for payloads.
## Error Handling
The APIs used for administrative tools all respond with standardised JSON and these include errors.
```text
$ nats req '$JS.API.STREAM.INFO.nonexisting' ''
Published 11 bytes to $JS.API.STREAM.INFO.nonexisting
Received [_INBOX.lcWgjX2WgJLxqepU0K9pNf.mpBW9tHK] : {
"type": "io.nats.jetstream.api.v1.stream_info_response",
"error": {
"code": 404,
"description": "stream not found"
}
}
```
```text
$ nats req '$JS.STREAM.INFO.ORDERS' ''
Published 6 bytes to $JS.STREAM.INFO.ORDERS
Received [_INBOX.fwqdpoWtG8XFXHKfqhQDVA.vBecyWmF] : '{
"type": "io.nats.jetstream.api.v1.stream_info_response",
"config": {
"name": "ORDERS",
...
}
```
Here the responses include a `type` which can be used to find the JSON Schema for each response.
Non admin APIs - like those for adding a message to the stream will respond with `-ERR` or `+OK` with an optional reason after.
## Admin API
All the admin actions the `nats` CLI can do falls in the sections below. The API structure are kept in the `api` package in the `jsm.go` repository.
Subjects that and in `T` like `api.JSApiConsumerCreateT` are formats and would need to have the Stream Name and in some cases also the Consumer name interpolated into them. In this case `t := fmt.Sprintf(api.JSApiConsumerCreateT, streamName)` to get the final subject.
The command `nats events` will show you an audit log of all API access events which includes the full content of each admin request, use this to view the structure of messages the `nats` command sends.
The API uses JSON for inputs and outputs, all the responses are typed using a `type` field which indicates their Schema. A JSON Schema repository can be found in `nats-io/jetstream/schemas`.
### General Info
| Subject | Constant | Description | Request Payload | Response Payload |
| :--- | :--- | :--- | :--- | :--- |
| `$JS.API.INFO` | `api.JSApiAccountInfo` | Retrieves stats and limits about your account | empty payload | `api.JetStreamAccountStats` |
### Streams
| Subject | Constant | Description | Request Payload | Response Payload |
| :--- | :--- | :--- | :--- | :--- |
| `$JS.API.STREAM.LIST` | `api.JSApiStreamList` | Paged list known Streams including all their current information | `api.JSApiStreamListRequest` | `api.JSApiStreamListResponse` |
| `$JS.API.STREAM.NAMES` | `api.JSApiStreamNames` | Paged list of Streams | `api.JSApiStreamNamesRequest` | `api.JSApiStreamNamesResponse` |
| `$JS.API.STREAM.CREATE.*` | `api.JSApiStreamCreateT` | Creates a new Stream | `api.StreamConfig` | `api.JSApiStreamCreateResponse` |
| `$JS.API.STREAM.UPDATE.*` | `api.JSApiStreamUpdateT` | Updates an existing Stream with new config | `api.StreamConfig` | `api.JSApiStreamUpdateResponse` |
| `$JS.API.STREAM.INFO.*` | `api.JSApiStreamInfoT` | Information about config and state of a Stream | empty payload, Stream name in subject | `api.JSApiStreamInfoResponse` |
| `$JS.API.STREAM.DELETE.*` | `api.JSApiStreamDeleteT` | Deletes a Stream and all its data | empty payload, Stream name in subject | `api.JSApiStreamDeleteResponse` |
| `$JS.API.STREAM.PURGE.*` | `api.JSApiStreamPurgeT` | Purges all of the data in a Stream, leaves the Stream | empty payload, Stream name in subject | `api.JSApiStreamPurgeResponse` |
| `$JS.API.STREAM.MSG.DELETE.*` | `api.JSApiMsgDeleteT` | Deletes a specific message in the Stream by sequence, useful for GDPR compliance | `api.JSApiMsgDeleteRequest` | `api.JSApiMsgDeleteResponse` |
| `$JS.API.STREAM.MSG.GET.*` | `api.JSApiMsgGetT` | Retrieves a specific message from the stream | `api.JSApiMsgGetRequest` | `api.JSApiMsgGetResponse` |
| `$JS.API.STREAM.SNAPSHOT.*` | `api.JSApiStreamSnapshotT` | Initiates a streaming backup of a streams data | `api.JSApiStreamSnapshotRequest` | `api.JSApiStreamSnapshotResponse` |
| `$JS.API.STREAM.RESTORE.*` | `api.JSApiStreamRestoreT` | Initiates a streaming restore of a stream | `{}` | `api.JSApiStreamRestoreResponse` |
### Stream Templates
| Subject | Constant | Description | Request Payload | Response Payload |
| :--- | :--- | :--- | :--- | :--- |
| `$JS.API.STREAM.TEMPLATE.CREATE.*` | `api.JSApiTemplateCreateT` | Creates a Stream Template | `api.StreamTemplateConfig` | `api.JSApiStreamTemplateCreateResponse` |
| `$JS.API.STREAM.TEMPLATE.NAMES` | `api.JSApiTemplateNames` | Paged list all known templates | `api.JSApiStreamTemplateNamesRequest` | `api.JSApiStreamTemplateNamesResponse` |
| `$JS.API.STREAM.TEMPLATE.INFO.*` | `api.JSApiTemplateInfoT` | Information about the config and state of a Stream Template | empty payload, Template name in subject | `api.JSApiStreamTemplateInfoResponse` |
| `$JS.API.STREAM.TEMPLATE.DELETE.*` | `api.JSApiTemplateDeleteT` | Delete a specific Stream Template **and all streams created by this template** | empty payload, Template name in subject | `api.JSApiStreamTemplateDeleteResponse` |
### Consumers
| Subject | Constant | Description | Request Payload | Response Payload |
| :--- | :--- | :--- | :--- | :--- |
| `$JS.API.CONSUMER.CREATE.*` | `api.JSApiConsumerCreateT` | Create an ephemeral Consumer | `api.ConsumerConfig`, Stream name in subject | `api.JSApiConsumerCreateResponse` |
| `$JS.API.CONSUMER.DURABLE.CREATE.*` | `api.JSApiDurableCreateT` | Create an Consumer | `api.ConsumerConfig`, Stream name in subject | `api.JSApiConsumerCreateResponse` |
| `$JS.API.CONSUMER.LIST.*` | `api.JSApiConsumerListT` | Paged list of known Consumers including their current info | `api.JSApiConsumerListRequest` | `api.JSApiConsumerListResponse` |
| `$JS.API.CONSUMER.NAMES.*` | `api.JSApiConsumerNamesT` | Paged list of known Consumer names | `api.JSApiConsumerNamesRequest` | `api.JSApiConsumerNamesResponse` |
| `$JS.API.CONSUMER.INFO.*.*` | `api.JSApiConsumerInfoT` | Information about an Consumer | empty payload, Stream and Consumer names in subject | `api.JSApiConsumerInfoResponse` |
| `$JS.API.CONSUMER.DELETE.*.*` | `api.JSApiConsumerDeleteT` | Deletes an Consumer | empty payload, Stream and Consumer names in subject | `api.JSApiConsumerDeleteResponse` |
### ACLs
It's hard to notice here but there is a clear pattern in these subjects, lets look at the various JetStream related subjects:
General information
```text
$JS.API.INFO
```
Stream and Consumer Admin
```text
$JS.API.STREAM.CREATE.<stream>
$JS.API.STREAM.UPDATE.<stream>
$JS.API.STREAM.DELETE.<stream>
$JS.API.STREAM.INFO.<stream>
$JS.API.STREAM.PURGE.<stream>
$JS.API.STREAM.LIST
$JS.API.STREAM.NAMES
$JS.API.STREAM.MSG.DELETE.<stream>
$JS.API.STREAM.MSG.GET.<stream>
$JS.API.STREAM.SNAPSHOT.<stream>
$JS.API.STREAM.RESTORE.<stream>
$JS.API.CONSUMER.CREATE.<stream>
$JS.API.CONSUMER.DURABLE.CREATE.<stream>.<consumer>
$JS.API.CONSUMER.DELETE.<stream>.<consumer>
$JS.API.CONSUMER.INFO.<stream>.<consumer>
$JS.API.CONSUMER.LIST.<stream>
$JS.API.CONSUMER.MSG.NEXT.<stream>.<consumer>
$JS.API.CONSUMER.NAMES.<stream>
$JS.API.STREAM.TEMPLATE.CREATE.<stream template>
$JS.API.STREAM.TEMPLATE.DELETE.<stream template>
$JS.API.STREAM.TEMPLATE.INFO.<stream template>
$JS.API.STREAM.TEMPLATE.NAMES
```
Stream and Consumer Use
```text
$JS.API.CONSUMER.MSG.NEXT.<stream>.<consumer>
$JS.ACK.<stream>.<consumer>.x.x.x
$JS.SNAPSHOT.ACK.<stream>.<msg id>
$JS.SNAPSHOT.RESTORE.<stream>.<msg id>
```
Events and Advisories:
```text
$JS.EVENT.METRIC.CONSUMER_ACK.<stream>.<consumer>
$JS.EVENT.ADVISORY.MAX_DELIVERIES.<stream>.<consumer>
$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.<stream>.<consumer>
$JS.EVENT.ADVISORY.STREAM.CREATED.<stream>
$JS.EVENT.ADVISORY.STREAM.DELETED.<stream>
$JS.EVENT.ADVISORY.STREAM.UPDATED.<stream>
$JS.EVENT.ADVISORY.CONSUMER.CREATED.<stream>.<consumer>
$JS.EVENT.ADVISORY.CONSUMER.DELETED.<stream>.<consumer>
$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_CREATE.<stream>
$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_COMPLETE.<stream>
$JS.EVENT.ADVISORY.STREAM.RESTORE_CREATE.<stream>
$JS.EVENT.ADVISORY.STREAM.RESTORE_COMPLETE.<stream>
$JS.EVENT.ADVISORY.STREAM.LEADER_ELECTED.<stream>
$JS.EVENT.ADVISORY.STREAM.QUORUM_LOST.<stream>
$JS.EVENT.ADVISORY.CONSUMER.LEADER_ELECTED.<stream>.<consumer>
$JS.EVENT.ADVISORY.CONSUMER.QUORUM_LOST.<stream>.<consumer>
$JS.EVENT.ADVISORY.API
```
This design allows you to easily create ACL rules that limit users to a specific Stream or Consumer and to specific verbs for administration purposes. For ensuring only the receiver of a message can Ack it we have response permissions ensuring you can only Publish to Response subject for messages you received.
## Acknowledging Messages
Messages that need acknowledgement will have a Reply subject set, something like `$JS.ACK.ORDERS.test.1.2.2`, this is the prefix defined in `api.JetStreamAckPre` followed by `<stream>.<consumer>.<delivered count>.<stream sequence>.<consumer sequence>.<timestamp>.<pending messages>`.
In all of the Synadia maintained API's you can simply do `msg.Respond(nil)` \(or language equivalent\) which will send nil to the reply subject.
## Fetching The Next Message From a Pull-based Consumer
If you have a pull-based Consumer you can send a standard NATS Request to `$JS.API.CONSUMER.MSG.NEXT.<stream>.<consumer>`, here the format is defined in `api.JetStreamRequestNextT` and requires populating using `fmt.Sprintf()`.
```text
$ nats req '$JS.API.CONSUMER.MSG.NEXT.ORDERS.test' '1'
Published 1 bytes to $JS.API.CONSUMER.MSG.NEXT.ORDERS.test
Received [js.1] : 'message 1'
```
Here we ask for just 1 message - `nats req` only shows 1 - but you can fetch a batch of messages by varying the argument. This combines well with the `AckAll` Ack policy.
The above request for the next message will stay in the server for as long as the client is connected and future pulls from the same client will accumulate on the server, meaning if you ask for 1 message 100 times and 1000 messages arrive you'll get sent 100 messages not 1.
This is often not desired, pull consumers support a mode where a JSON document is sent describing the pull request.
```javascript
{
"expires": "2020-11-10T12:41:00.075933464Z",
"batch": 10,
}
```
This requests 10 messages and asks the server to keep this request until the specific `expires` time, this is useful when you poll the server frequently and do not want the pull requests to accumulate on the server. Set the expire time to now + your poll frequency.
```javascript
{
"batch": 10,
"no_wait": true
}
```
Here we see a second format of the Pull request that will not store the request on the queue at all but when there are no messages to deliver will send a nil bytes message with a `Status` header of `404`, this way you can know when you reached the end of the stream for example. A `409` is returned if the Consumer has reached `MaxAckPending` limits.
```text
[rip@dev1]% nats req '$JS.API.CONSUMER.MSG.NEXT.ORDERS.NEW' '{"no_wait": true, "batch": 10}'
test --password test
13:45:30 Sending request on "$JS.API.CONSUMER.MSG.NEXT.ORDERS.NEW"
13:45:30 Received on "_INBOX.UKQGqq0W1EKl8inzXU1naH.XJiawTRM" rtt 594.908µs
13:45:30 Status: 404
13:45:30 Description: No Messages
```
## Fetching From a Stream By Sequence
If you know the Stream sequence of a message you can fetch it directly, this does not support acks. Do a Request\(\) to `$JS.API.STREAM.MSG.GET.ORDERS` sending it the message sequence as payload. Here the prefix is defined in `api.JetStreamMsgBySeqT` which also requires populating using `fmt.Sprintf()`.
```text
$ nats req '$JS.API.STREAM.MSG.GET.ORDERS' '{"seq": 1}'
Published 1 bytes to $JS.STREAM.ORDERS.MSG.BYSEQ
Received [_INBOX.cJrbzPJfZrq8NrFm1DsZuH.k91Gb4xM] : '{
"type": "io.nats.jetstream.api.v1.stream_msg_get_response",
"message": {
"subject": "x",
"seq": 1,
"data": "aGVsbG8=",
"time": "2020-05-06T13:18:58.115424+02:00"
}
}'
```
The Subject shows where the message was received, Data is base64 encoded and Time is when it was received.
## Consumer Samples
Samples are published to a specific subject per Consumer, something like `$JS.EVENT.METRIC.CONSUMER_ACK.<stream>.<consumer>` you can just subscribe to that and get `api.ConsumerAckMetric` messages in JSON format. The prefix is defined in `api.JetStreamMetricConsumerAckPre`.

170
jetstream/replication.md Normal file
View File

@ -0,0 +1,170 @@
# Data Replication
Replication allows you to move data between streams in either a 1:1 mirror style or by multiplexing multiple source streams into a new stream. In future builds this will allow data to be replicated between accounts as well, ideal for sending data from a Leafnode into a central store.
![](../.gitbook/assets/replication.png)
Here we have 2 main streams - _ORDERS_ and _RETURNS_ - these streams are clustered across 3 nodes. These Streams have short retention periods and are memory based.
We create a _ARCHIVE_ stream that has 2 _sources_ set, the _ARCHIVE_ will pull data from the sources into itself. This stream has a very long retention period and is file based and replicated across 3 nodes. Additional messages can be added to the ARCHIVE by sending to it directly.
Finally, we create a _REPORT_ stream mirrored from _ARCHIVE_ that is not clustered and retains data for a month. The _REPORT_ Stream does not listen for any incoming messages, it can only consume data from _ARCHIVE_.
## Mirrors
A _mirror_ copies data from 1 other stream, as far as possible IDs and ordering will match exactly the source. A _mirror_ does not listen on a subject for any data to be added. The Start Sequence and Start Time can be set, but no subject filter. A stream can only have 1 _mirror_ and if it is a mirror it cannot also have any _source_.
## Sources
A _source_ is a stream where data is copied from, one stream can have multiple sources and will read data in from them all. The stream will also listen for messages on it's own subject. We can therefore not maintain absolute ordering, but data from 1 single source will be in the correct order but mixed in with other streams. You might also find the timestamps of streams can be older and newer mixed in together as a result.
A Stream with sources may also listen on subjects, but could have no listening subject. When using the `nats` CLI to create sourced streams use `--subjects` to supply subjects to listen on.
A source can have Start Time or Start Sequence and can filter by a subject.
## Configuration
The ORDERS and RETURNS streams as normal, I will not show how to create them.
```text
$ nats s report
Obtaining Stream stats
+---------+---------+----------+-----------+----------+-------+------+---------+----------------------+
| Stream | Storage | Template | Consumers | Messages | Bytes | Lost | Deleted | Cluster |
+---------+---------+----------+-----------+----------+-------+------+---------+----------------------+
| ORDERS | Memory | | 0 | 0 | 0 B | 0 | 0 | n1-c2, n2-c2*, n3-c2 |
| RETURNS | Memory | | 0 | 0 | 0 B | 0 | 0 | n1-c2*, n2-c2, n3-c2 |
+---------+---------+----------+-----------+----------+-------+------+---------+----------------------+
```
We now add the ARCHIVE:
```text
$ nats s add ARCHIVE --source ORDERS --source RETURNS
? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Message count limit -1
? Message size limit -1
? Maximum message age limit -1
? Maximum individual message size -1
? Duplicate tracking time window 2m
? Number of replicas to store 3
? ORDERS Source Start Sequence 0
? ORDERS Source UTC Time Stamp (YYYY:MM:DD HH:MM:SS)
? ORDERS Source Filter source by subject
? RETURNS Source Start Sequence 0
? RETURNS Source UTC Time Stamp (YYYY:MM:DD HH:MM:SS)
? RETURNS Source Filter source by subject
```
And we add the REPORT:
```text
$ nats s add REPORT --mirror ARCHIVE
? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Message count limit -1
? Message size limit -1
? Maximum message age limit 1M
? Maximum individual message size -1
? Duplicate tracking time window 2m
? Number of replicas to store 1
? Mirror Start Sequence 0
? Mirror Start Time (YYYY:MM:DD HH:MM:SS)
? Mirror subject filter
```
When configured we'll see some additional information in a `nats stream info` output:
```text
$ nats stream info ARCHIVE
...
Source Information:
Stream Name: ORDERS
Lag: 0
Last Seen: 2m23s
Stream Name: RETURNS
Lag: 0
Last Seen: 2m15s
...
$ nats stream info REPORT
...
Mirror Information:
Stream Name: ARCHIVE
Lag: 0
Last Seen: 2m35s
...
```
Here the `Lag` is how far behind we were reported as being last time we saw a message.
We can confirm all our setup using a `nats stream report`:
```text
$ nats s report
+-------------------------------------------------------------------------------------------------------------------+
| Stream Report |
+---------+---------+----------+-------------+-----------+----------+-------+------+---------+----------------------+
| Stream | Storage | Template | Replication | Consumers | Messages | Bytes | Lost | Deleted | Cluster |
+---------+---------+----------+-------------+-----------+----------+-------+------+---------+----------------------+
| ARCHIVE | File | | Sourced | 1 | 0 | 0 B | 0 | 0 | n1-c2*, n2-c2, n3-c2 |
| ORDERS | Memory | | | 1 | 0 | 0 B | 0 | 0 | n1-c2, n2-c2*, n3-c2 |
| REPORT | File | | Mirror | 0 | 0 | 0 B | 0 | 0 | n1-c2* |
| RETURNS | Memory | | | 1 | 0 | 0 B | 0 | 0 | n1-c2, n2-c2, n3-c2* |
+---------+---------+----------+-------------+-----------+----------+-------+------+---------+----------------------+
+---------------------------------------------------------+
| Replication Report |
+---------+--------+---------------+--------+-----+-------+
| Stream | Kind | Source Stream | Active | Lag | Error |
+---------+--------+---------------+--------+-----+-------+
| ARCHIVE | Source | ORDERS | never | 0 | |
| ARCHIVE | Source | RETURNS | never | 0 | |
| REPORT | Mirror | ARCHIVE | never | 0 | |
+---------+--------+---------------+--------+-----+-------+
```
We then create some data in both ORDERS and RETURNS:
```text
$ nats req ORDERS.new "ORDER {{Count}}" --count 100
$ nats req RETURNS.new "RETURN {{Count}}" --count 100
```
We can now see from a Stream Report that the data has been replicated:
```text
$ nats s report --dot replication.dot
Obtaining Stream stats
+---------+---------+----------+-----------+----------+---------+------+---------+----------------------+
| Stream | Storage | Template | Consumers | Messages | Bytes | Lost | Deleted | Cluster |
+---------+---------+----------+-----------+----------+---------+------+---------+----------------------+
| ORDERS | Memory | | 1 | 100 | 3.3 KiB | 0 | 0 | n1-c2, n2-c2*, n3-c2 |
| RETURNS | Memory | | 1 | 100 | 3.5 KiB | 0 | 0 | n1-c2*, n2-c2, n3-c2 |
| ARCHIVE | File | | 1 | 200 | 27 KiB | 0 | 0 | n1-c2, n2-c2, n3-c2* |
| REPORT | File | | 0 | 200 | 27 KiB | 0 | 0 | n1-c2* |
+---------+---------+----------+-----------+----------+---------+------+---------+----------------------+
+---------------------------------------------------------+
| Replication Report |
+---------+--------+---------------+--------+-----+-------+
| Stream | Kind | Source Stream | Active | Lag | Error |
+---------+--------+---------------+--------+-----+-------+
| ARCHIVE | Source | ORDERS | 14.48s | 0 | |
| ARCHIVE | Source | RETURNS | 9.83s | 0 | |
| REPORT | Mirror | ARCHIVE | 9.82s | 0 | |
+---------+--------+---------------+--------+-----+-------+
```
Here we also pass the `--dot replication.dot` argument that writes a GraphViz format map of the replication setup.
![](../.gitbook/assets/replication-setup.png)

134
jetstream/resource_management.md Normal file
View File

@ -0,0 +1,134 @@
# Multi-tenancy & Resource Mgmt
## Multi Tenancy and Resource Management
JetStream is compatible with NATS 2.0 Multi Tenancy using Accounts. A JetStream enabled server supports creating fully isolated JetStream environments for different accounts.
To enable JetStream in a server we have to configure it at the top level first:
```text
jetstream: enabled
```
This will dynamically determine the available resources. It's recommended that you set specific limits though:
```text
jetstream {
store_dir: /data/jetstream
max_mem: 1G
max_file: 100G
}
```
At this point JetStream will be enabled and if you have a server that does not have accounts enabled all users in the server would have access to JetStream
```text
jetstream {
store_dir: /data/jetstream
max_mem: 1G
max_file: 100G
}
accounts {
HR: {
jetstream: enabled
}
}
```
Here the `HR` account would have access to all the resources configured on the server, we can restrict it:
```text
jetstream {
store_dir: /data/jetstream
max_mem: 1G
max_file: 100G
}
accounts {
HR: {
jetstream {
max_mem: 512M
max_file: 1G
max_streams: 10
max_consumers: 100
}
}
}
```
Now the `HR` account it limited in various dimensions.
If you try to configure JetStream for an account without enabling it globally you'll get a warning and the account designated as System cannot have JetStream enabled.
## `nats` CLI
As part of the JetStream efforts a new `nats` CLI is being developed to act as a single point of access to the NATS eco system.
This CLI has been seen throughout the guide, it's available in the Docker containers today and downloadable on the [Releases](https://github.com/nats-io/jetstream/releases) page.
### Configuration Contexts
The CLI has a number of environment configuration settings - where your NATS server is, credentials, TLS keys and more:
```text
$ nats --help
...
-s, --server=NATS_URL NATS servers
--user=NATS_USER Username of Token
--password=NATS_PASSWORD Password
--creds=NATS_CREDS User credentials
--nkey=NATS_NKEY User NKEY
--tlscert=NATS_CERT TLS public certificate
--tlskey=NATS_KEY TLS private key
--tlsca=NATS_CA TLS certificate authority chain
--timeout=NATS_TIMEOUT Time to wait on responses from NATS
--context=CONTEXT NATS Configuration Context to use for access
...
```
You can set these using the CLI flag, the environmet variable - like **NATS\_URL** - or using our context feature.
A context is a named configuration that stores all these settings, you can switch between access configurations and designate a default.
Creating one is easy, just specify the same settings to the `nats context save`
```text
$ nats context save example --server nats://nats.example.net:4222 --description 'Example.Net Server'
$ nats context save local --server nats://localhost:4222 --description 'Local Host' --select
$ nats context ls
Known contexts:
example Example.Net Server
local* Local Host
```
We passed `--select` to the `local` one meaning it will be the default when nothing is set.
```text
$ nats rtt
nats://localhost:4222:
nats://127.0.0.1:4222: 245.115µs
nats://[::1]:4222: 390.239µs
$ nats rtt --context example
nats://nats.example.net:4222:
nats://192.0.2.10:4222: 41.560815ms
nats://192.0.2.11:4222: 41.486609ms
nats://192.0.2.12:4222: 41.178009ms
```
The `nats context select` command can be used to set the default context.
All `nats` commands are context aware and the `nats context` command has various commands to view, edit and remove contexts.
Server URLs and Credential paths can be resolved via the `nsc` command by specifying an url, for example to find user `new` within the `orders` account of the `acme` operator you can use this:
```text
$ nats context save example --description 'Example.Net Server' --nsc nsc://acme/orders/new
```
The server list and credentials path will now be resolved via `nsc`, if these are specifically set in the context, the specific context configuration will take precedence.

12
nats-on-kubernetes/nats-external-nlb.md
View File

@ -1,3 +1,5 @@
# Using a Load Balancer for External Access to NATS
## Using a Load Balancer for External Access to NATS
In the example below, you can find how to use an [AWS Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) to connect externally to a cluster that has TLS setup.
@ -42,7 +44,7 @@ Also, it would be recommended to set [no\_advertise](../nats-server/configuratio
With the following, you can create a 3-node NATS Server cluster:
```sh
```bash
kubectl apply -f https://raw.githubusercontent.com/nats-io/k8s/b55687a97a5fd55485e1af302fbdbe43d2d3b968/nats-server/leafnodes/nats-cluster.yaml
```
@ -85,13 +87,13 @@ data:
Now let's expose the NATS Server by creating an L4 load balancer on Azure:
```sh
```bash
kubectl apply -f https://raw.githubusercontent.com/nats-io/k8s/b55687a97a5fd55485e1af302fbdbe43d2d3b968/nats-server/leafnodes/lb.yaml
```
Confirm the public IP that was allocated to the `nats-lb` service that was created, in this case it is `52.155.49.45`:
```
```text
$ kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 81d <none>
@ -113,13 +115,13 @@ leaf {
You can also add a NATS Streaming cluster into the cluster connecting to the port 4222:
```sh
```bash
kubectl apply -f https://raw.githubusercontent.com/nats-io/k8s/b55687a97a5fd55485e1af302fbdbe43d2d3b968/nats-server/leafnodes/stan-server.yaml
```
Now if you create two NATS Servers that connect to the same leafnode port, they will be able to receive messages to each other:
```sh
```bash
nats-server -c leafnodes/leaf.conf -p 4222 &
nats-server -c leafnodes/leaf.conf -p 4223 &

164
nats-on-kubernetes/stan-ft-k8s-aws.md
View File

@ -60,8 +60,9 @@ metadata:
name: run-efs-provisioner
subjects:
- kind: ServiceAccount
name: efs-provisioner
# replace with namespace where provisioner is deployed
name:
efs-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: ClusterRole
@ -102,13 +103,16 @@ data:
dns.name: ""
---
kind: Deployment
apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:
name: efs-provisioner
spec:
replicas: 1
selector:
matchLabels:
app: efs-provisioner
strategy:
type: Recreate
type: Recreate
template:
metadata:
labels:
@ -172,14 +176,14 @@ spec:
Result of deploying the manifest:
```bash
serviceaccount/efs-provisioner created
clusterrole.rbac.authorization.k8s.io/efs-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-efs-provisioner created
role.rbac.authorization.k8s.io/leader-locking-efs-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-efs-provisioner created
configmap/efs-provisioner created
deployment.extensions/efs-provisioner created
storageclass.storage.k8s.io/aws-efs created
serviceaccount/efs-provisioner created
clusterrole.rbac.authorization.k8s.io/efs-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-efs-provisioner created
role.rbac.authorization.k8s.io/leader-locking-efs-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-efs-provisioner created
configmap/efs-provisioner created
deployment.extensions/efs-provisioner created
storageclass.storage.k8s.io/aws-efs created
persistentvolumeclaim/efs created
```
@ -200,14 +204,14 @@ spec:
app: stan
clusterIP: None
ports:
- name: client
port: 4222
- name: cluster
port: 6222
- name: monitor
port: 8222
- name: metrics
port: 7777
- name: client
port: 4222
- name: cluster
port: 6222
- name: monitor
port: 8222
- name: metrics
port: 7777
---
apiVersion: v1
kind: ConfigMap
@ -270,69 +274,69 @@ spec:
terminationGracePeriodSeconds: 30
containers:
- name: stan
image: nats-streaming:alpine
- name: stan
image: nats-streaming:alpine
ports:
# In case of NATS embedded mode expose these ports
- containerPort: 4222
name: client
- containerPort: 6222
name: cluster
- containerPort: 8222
name: monitor
args:
- "-sc"
- "/etc/stan-config/stan.conf"
ports:
# In case of NATS embedded mode expose these ports
- containerPort: 4222
name: client
- containerPort: 6222
name: cluster
- containerPort: 8222
name: monitor
args:
- "-sc"
- "/etc/stan-config/stan.conf"
# Required to be able to define an environment variable
# that refers to other environment variables. This env var
# is later used as part of the configuration file.
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: CLUSTER_ADVERTISE
value: $(POD_NAME).stan.$(POD_NAMESPACE).svc
volumeMounts:
- name: config-volume
mountPath: /etc/stan-config
- name: efs
mountPath: /data/stan
resources:
requests:
cpu: 0
livenessProbe:
httpGet:
path: /
port: 8222
initialDelaySeconds: 10
timeoutSeconds: 5
- name: metrics
image: synadia/prometheus-nats-exporter:0.5.0
args:
- -connz
- -routez
- -subz
- -varz
- -channelz
- -serverz
- http://localhost:8222
ports:
- containerPort: 7777
name: metrics
# Required to be able to define an environment variable
# that refers to other environment variables. This env var
# is later used as part of the configuration file.
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: CLUSTER_ADVERTISE
value: $(POD_NAME).stan.$(POD_NAMESPACE).svc
volumeMounts:
- name: config-volume
mountPath: /etc/stan-config
- name: efs
mountPath: /data/stan
resources:
requests:
cpu: 0
livenessProbe:
httpGet:
path: /
port: 8222
initialDelaySeconds: 10
timeoutSeconds: 5
- name: metrics
image: synadia/prometheus-nats-exporter:0.5.0
args:
- -connz
- -routez
- -subz
- -varz
- -channelz
- -serverz
- http://localhost:8222
ports:
- containerPort: 7777
name: metrics
volumes:
- name: config-volume
configMap:
name: stan-config
- name: efs
persistentVolumeClaim:
claimName: efs
- name: config-volume
configMap:
name: stan-config
- name: efs
persistentVolumeClaim:
claimName: efs
```
Your cluster now will look something like this:

2
nats-server/configuration/README.md
View File

@ -117,6 +117,8 @@ authorization: {
| [`cluster`](clustering/cluster_config.md) | Configuration map for [cluster](clustering/). | |
| [`gateway`](gateways/gateway.md#gateway-configuration-block) | Configuration map for [gateway](gateways/). | |
| [`leafnode`](leafnodes/leafnode_conf.md) | Configuration map for a [leafnode](leafnodes/). | |
| [`mqtt`](https://github.com/nats-io/nats.docs/tree/53202d44215a11c4c4ad7caea03a703d302bc954/nats-server/configuration/mqtt/mqtt_conf.md) | Configuration map for a [mqtt](mqtt.md). | |
| [`websocket`](https://github.com/nats-io/nats.docs/tree/53202d44215a11c4c4ad7caea03a703d302bc954/nats-server/configuration/websocket/websocket_conf.md) | Configuration map for [websocket](websocket.md). | |
### Connection Timeouts

2
nats-server/configuration/gateways/README.md
View File

@ -27,7 +27,7 @@ If gateways are to be used in a cluster, **all** servers of this cluster need to
A nats-server in a gateway role will specify a port where it will accept gateway connections. If the configuration specifies other _external_ `gateways`, the gateway will create one outbound gateway connection for each gateway in its configuration. It will also gossip other gateways it knows or discovers. Fewer _external_ `gateways` mean less configuration. Yet, the ability to discover more gateways and gateway nodes depends on these servers running. This is similar to _seed server_ in cluster. It is recommended to have all _seed server_ of a cluster listed in the `gateways` section.
If the local cluster has three gateway nodes, this means there will be three outbound connections to each external gateway.
If the local cluster has three gateway nodes, this means there will be three outbound connections from the local cluster to each external gateway cluster.
![Gateway Connections](../../../.gitbook/assets/simple.svg)

27
nats-server/configuration/leafnodes/leafnode_conf.md
View File

@ -93,6 +93,33 @@ If other form of credentials are used \(jwt, nkey or other\), then the server wi
| `account` | [Account](../securing_nats/accounts.md) name or jwt public key identifying the local account to bind to this remote server. Any traffic locally on this account will be forwarded to the remote server. |
| `credentials` | Credential file for connecting to the leafnode server. |
| `tls` | A [TLS configuration](leafnode_conf.md#tls-configuration-block) block. Leafnode client will use specified TLS certificates when connecting/authenticating. |
| `ws_compression` | If connecting with [Websocket](https://github.com/nats-io/nats.docs/tree/0ab2bb72b305dcf4817b8f1fc4f37ab9b0c8a2db/nats-server/configuration/leafnodes/leafnode_conf/README.md#connecting-using-websocket-protocol) protocol, this boolean \(`true` or `false`\) indicates to the remote server that it wishes to use compression. The default is `false`. |
| `ws_no_masking` | If connecting with [Websocket](https://github.com/nats-io/nats.docs/tree/0ab2bb72b305dcf4817b8f1fc4f37ab9b0c8a2db/nats-server/configuration/leafnodes/leafnode_conf/README.md#connecting-using-websocket-protocol) protocol, this boolean indicates to the remote server that it wishes not to mask outbound websocket frames. The default is `false`, which means that outbound frames will be masked. |
### Connecting using Websocket protocol
Since NATS 2.2.0, Leaf nodes support outbound websocket connections by specifying `ws` as the scheme component of the remote server URLs:
```text
leafnodes {
remotes [
{urls: ["ws://hostname1:443", "ws://hostname2:443"]}
]
}
```
Note that if a URL has the `ws` scheme, all URLs the list must be `ws`. You cannot mix and match. Therefore this would be considered an invalid configuration:
```text
remotes [
# Invalid configuration that will prevent the server from starting
{urls: ["ws://hostname1:443", "nats-leaf://hostname2:7422"]}
]
```
Note that the decision to make a TLS connection is not based on `wss://` \(as opposed to `ws://`\) but instead in the presence of a TLS configuration in the `leafnodes{}` or the specific remote configuration block.
To configure Websocket in the remote server, check the [Websocket](https://github.com/nats-io/nats.docs/tree/0ab2bb72b305dcf4817b8f1fc4f37ab9b0c8a2db/nats-server/configuration/websocket/websocket_conf.md) secion.
### `tls` Configuration Block

211
nats-server/configuration/monitoring.md
View File

@ -11,6 +11,7 @@ To monitor the NATS messaging system, `nats-server` provides a lightweight HTTP
* [Leaf Nodes](monitoring.md#leaf-nodes-information)
* [Subscription Routing](monitoring.md#subscription-routing-information)
* [Account Information](monitoring.md#account-information)
* [JetStream Information](monitoring.md#jetstream-information)
All endpoints return a JSON object.
@ -57,7 +58,7 @@ http: localhost:8222
For example, to monitor this server locally, the endpoint would be [http://localhost:8222/varz](http://localhost:8222/varz). It reports various general statistics.
## Monitoring endpoints
## Monitoring Endpoints
The following sections describe each supported monitoring endpoint: `varz`, `connz`, `routez`, `subsz`, `gatewayz`, and `leafz`. There are not any required arguments, however use of arguments can let you tailor monitoring to your environment and tooling.
@ -247,6 +248,61 @@ You can also report detailed subscription information on a per connection basis
}
```
### JetStream Information
The `/jsz` endpoint reports information about the JetStream subsystem.
**Endpoint:** `http://server:port/jsm`
| Result | Return Code |
| :--- | :--- |
| Success | 200 \(OK\) |
| Error | 400 \(Bad Request\) |
#### Arguments
| Argument | Values | Description |
| :--- | :--- | :--- |
| acc | account name | Provide information for a specfic account |
| accounts | true, 1, false, 0 | Provide information for all accounts. The default is false. |
| streams | true, 1, false, 0 | Include stream information. The default is false. |
| consumers | true, 1, false, 0 | Include consumer information. The default is false. |
| config | true, false | Include configuration with streams and consumers. The default is false. |
| offset | integer &gt; 0 | Pagination offset. Default is 0. |
| limit | integer &gt; 0 | Number of results to return. Default is 1024. |
| leader-only | true, false | TODO |
As noted above, the `routez` endpoint does support the `subs` argument from the `/connz` endpoint. For example: [http://demo.nats.io:8222/routez?subs=1](http://demo.nats.io:8222/jsz?accounts=1&streams=1&consumers=1&config=1)
#### Example
* Get JetStream information: [http://host:port/jsz?accounts=1&streams=1&consumers=1&config=1](http://host:port/jsz?accounts=1&streams=1&consumers=1&config=1)
#### Response
```javascript
{
"server_id": "NACDVKFBUW4C4XA24OOT6L4MDP56MW76J5RJDFXG7HLABSB46DCMWCOW",
"now": "2019-06-24T14:29:16.046656-07:00",
"num_routes": 1,
"routes": [
{
"rid": 1,
"remote_id": "de475c0041418afc799bccf0fdd61b47",
"did_solicit": true,
"ip": "127.0.0.1",
"port": 61791,
"pending_size": 0,
"in_msgs": 0,
"out_msgs": 0,
"in_bytes": 0,
"out_bytes": 0,
"subscriptions": 0
}
]
}
```
### Route Information
The `/routez` endpoint reports information on active routes for a cluster. Routes are expected to be low, so there is no paging mechanism with this endpoint.
@ -433,7 +489,7 @@ The `/gatewayz` endpoint reports information about gateways used to create a NAT
}
```
### Leaf Nodes Information
### Leaf Node Information
The `/leafz` endpoint reports detailed information about the leaf node connections.
@ -520,7 +576,6 @@ The `/subsz` endpoint reports detailed information about the current subscriptio
"avg_fanout": 0
}
```
### Account Information
The `/accountz` endpoint reports information on a servers active accounts.
@ -528,15 +583,6 @@ The default behavior is to return a list of all accounts known to the server.
**Endpoint:** `http://server:port/accountz`
| Result | Return Code |
| :--- | :--- |
| Success | 200 \(OK\) |
| Error | 400 \(Bad Request\) |
#### Arguments
| Argument | Values | Description |
| :--- | :--- | :--- |
| acc | account name | Include metrics for the specified account. Default is empty. When not set, a list of all accounts is included. |
#### Example
@ -626,6 +672,147 @@ Retrieve specific account:
}
```
### JetStream Information
The `/jsz` endpoint reports more detailed information on JetStream. For accounts it uses a paging mechanism which defaults to 1024 connections.
**Endpoint:** `http://server:port/jsz`
| Result | Return Code |
| :--- | :--- |
| Success | 200 \(OK\) |
| Error | 400 \(Bad Request\) |
#### Arguments
| Argument | Values | Description |
| :--- | :--- | :--- |
| acc | account name | Include metrics for the specified account. Default is unset. |
| accounts | true, 1, false, 0 | Include account specific JetStream information. Default is false. |
| streams | true, 1, false, 0 | Include streams. When set, implies `accounts=true`. Default is false. |
| consumers | true, 1, false, 0 | Include consumer. When set, implies `streams=true`. Default is false. |
| config | true, 1, false, 0 | When stream or consumer are requested, include their respective configuration. Default is false. |
| leader-only | true, 1, false, 0 | Only the leader responds. Default is false. |
| offset | number &gt; 0 | Pagination offset. Default is 0. |
| limit | number &gt; 0 | Number of results to return. Default is 1024. |
#### Examples
Get basic JetStream information: [http://demo.nats.io:8222/jsz](http://demo.nats.io:8222/jsz)
Request accounts and control limit and offset: [http://demo.nats.io:8222/jsz?accounts=true&limit=16&offset=128](http://demo.nats.io:8222/jsz?accounts=true&limit=16&offset=128).
You can also report detailed consumer information on a per connection basis using consumer=true. For example: [http://demo.nats.io:8222/jsz?consumers=true](http://demo.nats.io:8222/jsz/consumer=true).
#### Response
```javascript
{
"server_id": "NCVIDODSZ45C5OD67ZD7EJUIJPQDP6CM74SJX6TJIF2G7NLYS5LCVYHS",
"now": "2021-02-08T19:08:30.555533-05:00",
"config": {
"max_memory": 10485760,
"max_storage": 10485760,
"store_dir": "/var/folders/9h/6g_c9l6n6bb8gp331d_9y0_w0000gn/T/srv_7500251552558"
},
"memory": 0,
"storage": 66,
"api": {
"total": 5,
"errors": 0
},
"total_streams": 1,
"total_consumers": 1,
"total_messages": 1,
"total_message_bytes": 33,
"meta_cluster": {
"name": "cluster_name",
"replicas": [
{
"name": "server_5500",
"current": false,
"active": 2932926000
}
]
},
"account_details": [
{
"name": "BCC_TO_HAVE_ONE_EXTRA",
"id": "BCC_TO_HAVE_ONE_EXTRA",
"memory": 0,
"storage": 0,
"api": {
"total": 0,
"errors": 0
}
},
{
"name": "ACC",
"id": "ACC",
"memory": 0,
"storage": 66,
"api": {
"total": 5,
"errors": 0
},
"stream_detail": [
{
"name": "my-stream-replicated",
"cluster": {
"name": "cluster_name",
"replicas": [
{
"name": "server_5500",
"current": false,
"active": 2931517000
}
]
},
"state": {
"messages": 1,
"bytes": 33,
"first_seq": 1,
"first_ts": "2021-02-09T00:08:27.623735Z",
"last_seq": 1,
"last_ts": "2021-02-09T00:08:27.623735Z",
"consumer_count": 1
},
"consumer_detail": [
{
"stream_name": "my-stream-replicated",
"name": "my-consumer-replicated",
"created": "2021-02-09T00:08:27.427631Z",
"delivered": {
"consumer_seq": 0,
"stream_seq": 0
},
"ack_floor": {
"consumer_seq": 0,
"stream_seq": 0
},
"num_ack_pending": 0,
"num_redelivered": 0,
"num_waiting": 0,
"num_pending": 1,
"cluster": {
"name": "cluster_name",
"replicas": [
{
"name": "server_5500",
"current": false,
"active": 2933232000
}
]
}
}
]
}
]
}
]
}
```
## Creating Monitoring Applications
NATS monitoring endpoints support [JSONP](https://en.wikipedia.org/wiki/JSONP) and [CORS](https://en.wikipedia.org/wiki/Cross-origin_resource_sharing#How_CORS_works). You can easily create single page web applications for monitoring. To do this you simply pass the `callback` query parameter to any endpoint.

128
nats-server/configuration/mqtt.md Normal file
View File

@ -0,0 +1,128 @@
# MQTT
_Supported since NATS Server version 2.2_
NATS follows as closely as possible to the MQTT v3.1.1 [specification](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html).
## When to Use MQTT
MQTT support in NATS is intended to be an enabling technology allowing users to leverage existing investments in their IoT deployments. Updating software on the edge or endpoints can be onerous and risky, especially when embedded applications are involved.
In greenfield IoT deployments, when possible, we prefer NATS extended out to endpoints and devices for a few reasons. There are significant advantages with security and observability when using a single technology end to end. Compared to MQTT, NATS is nearly as lightweight in terms of protocol bandwidth and maintainer supported clients efficiently utilize resources so we consider NATS to be a good choice to use end to end, including use on resource constrained devices.
In existing MQTT deployments or in situations when endpoints can only support MQTT, using a NATS server as a drop-in MQTT server replacement to securely connect to a remote NATS cluster or supercluster is compelling. You can keep your existing IoT investment and use NATS for secure, resilient, and scalable access to your streams and services.
## JetStream Requirements
For an MQTT client to connect to the NATS server, the user's account must be JetStream enabled. This is because persistence is needed for the sessions and retained messages since even retained messages of QoS 0 are persisted.
## MQTT Topics and NATS Subjects
MQTT Topics are similar to NATS Subjects, but have distinctive differences.
MQTT topic uses "`/`" as a level separator. For instance `foo/bar` would translate to NATS subject `foo.bar`. But in MQTT, `/foo/bar/` is a valid subject, which, if simply translated, would become `.foo.bar.`, which is NOT a valid NATS Subject.
NATS Server will convert an MQTT topic following those rules:
| MQTT character | NATS character\(s\) | Topic \(MQTT\) | Subject \(NATS\) |
| :---: | :---: | :---: | :---: |
| `/` between two levels | `.` | `foo/bar` | `foo.bar` |
| `/` as first level | `/.` | `/foo/bar` | `/.foo.bar` |
| `/` as last level | `./` | `foo/bar/` | `foo.bar./` |
| `/` next to another | `./` | `foo//bar` | `foo./.bar` |
| `/` next to another | `/.` | `//foo/bar` | `/./.foo.bar` |
| `.` | Not Support | `foo.bar` | Not Supported |
| | Not Support | `foo bar` | Not Supported |
As indicated above, if an MQTT topic contains the character ``` or``.\`, NATS will reject it, causing the connection to be closed for published messages, and returning a failure code in the SUBACK packet for a subscriptions.
### MQTT Wildcards
As in NATS, MQTT wildcards represent either multi or single levels. As in NATS, they are allowed only for subscriptions, not for published messages.
| MQTT Wildcard | NATS Wildcard |
| :---: | :---: |
| `#` | `>` |
| `+` | `*` |
The wildcard `#` matches any number of levels within a topic, which means that a subscription on `foo/#` would receive messages on `foo/bar`, or `foo/bar/baz`, but also on `foo`. This is not the case in NATS where a subscription on `foo.>` can receive messages on `foo/bar` or `foo/bar/baz`, but not on `foo`. To solve this, NATS Server will create two subscriptions, one on `foo.>` and one on `foo`. If the MQTT subscription is simply on `#`, then a single NATS subscription on `>` is enough.
The wildcard `+` matches a single level, which means `foo/+` can receive message on `foo/bar` or `foo/baz`, but not on `foo/bar/baz` nor `foo`. This is the same with NATS subscriptions using the wildcard `*`. Therefore `foo/+` would translate to `foo.*`.
## Communication Between MQTT and NATS
When an MQTT client creates a subscription on a topic, the NATS server creates the similar NATS subscription \(with conversion from MQTT topic to NATS subject\) so that the interest is registered in the cluster and known to any NATS publishers.
That is, say an MQTT client connects to server "A" and creates a subscription of `foo/bar`, server "A" creates a subscription on `foo.bar`, which interest is propagated as any other NATS subscription. A publisher connecting anywhere in the cluster and publishing on `foo.bar` would cause server "A" to deliver a QoS 0 message to the MQTT subscription.
This works the same way for MQTT publishers. When the server receives an MQTT publish message, it is converted to the NATS subject and published, which means that any matching NATS subscription will receive the MQTT message.
If the MQTT subscription is QoS1 and an MQTT publisher publishes an MQTT QoS1 message on the same or any other server in the cluster, the message will be persisted in the cluster and routed and delivered as QoS 1 to the MQTT subscription.
## QoS 1 Redeliveries
When the server delivers a QoS 1 message to a QoS 1 subscription, it will keep the message until it receives the PUBACK for the corresponding packet identifier. If it does not receive it within the "ack\_wait" interval, that message will be resent.
## Max Ack Pending
This is the amount of QoS 1 messages the server can send to a subscription without receiving any PUBACK for those messages. The maximum value is 65535.
The total of subscriptions' `max_ack_pending` on a given session cannot exceed 65535. Attempting to create a subscription that would bring the total above the limit would result in the server returning a failure code in the SUBACK for this subscription.
Due to how the NATS server handles the MQTT "`#`" wildcard, each subscription ending with "`#`" will use 2 times the `max_ack_pending` value.
## Sessions
NATS Server will persist all sessions, even if they are created with the "clean session" flag, meaning that sessions only last for the duration of the network connection between the client and the server.
A session is identified by a client identifier. If two connections try to use the same client identifier, the server, per specification, will close the existing connection and accept the new one.
If the user incorrectly starts two applications that use the same client identifier, this would result in a very quick flapping if the MQTT client has a reconnect feature and quickly reconnects.
To prevent this, the NATS server will accept the new session and will delay the closing of the old connection to reduce the flapping rate.
Detection of the concurrent use of sessions also works in cluster mode.
## Retained Messages
When a server receives an MQTT publish packet with the RETAIN flag set \(regardless of its QoS\), it stores the application message and its QoS, so that it can be delivered to future subscribers whose subscriptions match its topic name.
When a new subscription is established, the last retained message, if any, on each matching topic name will be sent to the subscriber.
A PUBLISH Packet with a RETAIN flag set to 1 and a payload containing zero bytes will be processed as normal and sent to clients with a subscription matching the topic name. Additionally any existing retained message with the same topic name will be removed and any future subscribers for the topic will not receive a retained message.
## Clustering
NATS supports MQTT in a NATS cluster. The replication factor is automatically set based on the size of the cluster.
### Connections with Same Client ID
If a client is connected to a server "A" in the cluster and another client connects to a server "B" and uses the same client identifier, server "A" will close its client connection upon discovering the use of an active client identifier.
Users should avoid this situation as this is not as easy and immediate as if the two applications are connected to the same server.
There may be cases where the server will reject the new connection if there is no safe way to close the existing connection, such as when it is in the middle of processing some MQTT packets.
### Retained Messages
Retained messages are stored in the cluster and available to any server in the cluster. However, this is not immediate and if a producer connects to a server and produces a retained message and another connection connects to another server and starts a matching subscription, it may not receive the retained message if the server it connects to has not yet been made aware of this retained message.
In other words, retained messages in clustering mode is best-effort, and applications that rely on the presence of a retained message should connect on the server that produced them.
## Limitations
* NATS does not support QoS 2 messages. If it receives a published message with QoS greater than 1,
it will close the connection.
* NATS messages published to MQTT subscriptions are always delivered as QoS 0 messages.
* MQTT published messages on topic names containing "```" or "``.\`" characters will cause the
connection to be closed. Presence of those characters in MQTT subscriptions will result in error
code in the SUBACK packet.
* MQTT wildcard `#` may cause the NATS server to create two subscriptions.
* MQTT concurrent sessions may result in the new connection to be evicted instead of the existing one.
* MQTT retained messages in clustering mode is best effort.

179
nats-server/configuration/mqtt/mqtt_conf.md Normal file
View File

@ -0,0 +1,179 @@
# Configuration
To enable MQTT support in the server, add a `mqtt` configuration
block in the server's configuration file like the following:
```
mqtt {
# Specify a host and port to listen for websocket connections
#
# listen: "host:port"
# It can also be configured with individual parameters,
# namely host and port.
#
# host: "hostname"
port: 1883
# TLS configuration.
#
tls {
cert_file: "/path/to/cert.pem"
key_file: "/path/to/key.pem"
# Root CA file
#
# ca_file: "/path/to/ca.pem"
# If true, require and verify client certificates.
#
# verify: true
# TLS handshake timeout in fractional seconds.
#
# timeout: 2.0
# If true, require and verify client certificates and map certificate
# values for authentication purposes.
#
# verify_and_map: true
}
# If no user name is provided when an MQTT client connects, will default
# this user name in the authentication phase. If specified, this will
# override, for MQTT clients, any `no_auth_user` value defined in the
# main configuration file.
# Note that this is not compatible with running the server in operator mode.
#
# no_auth_user: "my_username_for_apps_not_providing_credentials"
# See below to know what is the normal way of limiting MQTT clients
# to specific users.
# If there are no users specified in the configuration, this simple authorization
# block allows you to override the values that would be configured in the
# equivalent block in the main section.
#
# authorization {
# # If this is specified, the client has to provide the same username
# # and password to be able to connect.
# # username: "my_user_name"
# # password: "my_password"
#
# # If this is specified, the password field in the CONNECT packet has to
# # match this token.
# # token: "my_token"
#
# # This overrides the main's authorization timeout. For consistency
# # with the main's authorization configuration block, this is expressed
# # as a number of seconds.
# # timeout: 2.0
#}
# This is the amount of time after which a QoS 1 message sent to
# a client is redelivered as a DUPLICATE if the server has not
# received the PUBACK packet on the original Packet Identifier.
# The value has to be positive.
# Zero will cause the server to use the default value (30 seconds).
# Note that changes to this option is applied only to new MQTT subscriptions.
#
# Expressed as a time duration, with "s", "m", "h" indicating seconds,
# minutes and hours respectively. For instance "10s" for 10 seconds,
# "1m" for 1 minute, etc...
#
# ack_wait: "1m"
# This is the amount of QoS 1 messages the server can send to
# a subscription without receiving any PUBACK for those messages.
# The valid range is [0..65535].
#
# The total of subscriptions' max_ack_pending on a given session cannot
# exceed 65535. Attempting to create a subscription that would bring
# the total above the limit would result in the server returning 0x80
# in the SUBACK for this subscription.
# Due to how the NATS Server handles the MQTT "#" wildcard, each
# subscription ending with "#" will use 2 times the max_ack_pending value.
# Note that changes to this option is applied only to new subscriptions.
#
# max_ack_pending: 100
}
```
## Authorization of MQTT Users
A new field when configuring users allows you to restrict which type of connections are allowed for a specific user.
Consider this configuration:
```
authorization {
users [
{user: foo password: foopwd, permission: {...}}
{user: bar password: barpwd, permission: {...}}
]
}
```
If an MQTT client were to connect and use the username `foo` and password `foopwd`, it would be accepted.
Now suppose that you would want an MQTT client to only be accepted if it connected using the username `bar`
and password `barpwd`, then you would use the option `allowed_connection_types` to restrict which type
of connections can bind to this user.
```
authorization {
users [
{user: foo password: foopwd, permission: {...}}
{user: bar password: barpwd, permission: {...}, allowed_connection_types: ["MQTT"]}
]
}
```
The option `allowed_connection_types` (also can be named `connection_types` or `clients`) as you can see
is a list, and you can allow several type of clients. Suppose you want the user `bar` to accept both
standard NATS clients and MQTT clients, you would configure the user like this:
```
authorization {
users [
{user: foo password: foopwd, permission: {...}}
{user: bar password: barpwd, permission: {...}, allowed_connection_types: ["STANDARD", "MQTT"]}
]
}
```
The absence of `allowed_connection_types` means that all type of connections are allowed (the default behavior).
The possible values are currently:
* `STANDARD`
* `WEBSOCKET`
* `LEAFNODE`
* `MQTT`
### Special permissions
When an MQTT client creates a QoS 1 subscription, this translates to the creation
of a JetStream durable subscription. To receive messages for this durable, the NATS Server
creates a subscription with a subject such as `$MQTT.sub.<nuid>` and sets it as the
JetStream durable's delivery subject.
Therefore, if you have set some permissions for the MQTT user, you need to allow
subscribe permissions on `$MQTT.sub.>`.
Here is an example of a basic configuration that sets some permissions to a user named "mqtt".
As you can see, the subscribe permission `$MQTT.sub.>` is added to allow this client to
create QoS 1 subscriptions.
```
listen: 127.0.0.1:4222
jetstream: enabled
authorization {
mqtt_perms = {
publish = ["baz"]
subscribe = ["foo", "bar", "$MQTT.sub.>"]
}
users = [
{user: mqtt, password: pass, permissions: $mqtt_perms, allowed_connection_types: ["MQTT"]}
]
}
mqtt {
listen: 127.0.0.1:1883
}
```

2
nats-server/configuration/securing_nats/accounts.md
View File

@ -189,4 +189,6 @@ no_auth_user: a
The above example shows how clients without authentication can be associated with the user `a` within account `A`.
> Please note that the `no_auth_user` will not work with nkeys. The user referenced can also be part of the [authorization](authorization.md) block.
>
> Despite `no_auth_user` being set, clients still need to communicate that they will not be using credentials. The [authentication timeout](auth_intro/auth_timeout.md) applies to this process as well. When your connection is slow, you may run into this timeout and the resulting `Authentication Timeout` error, despite not providing credentials.

10
nats-server/configuration/securing_nats/auth_intro/tokens.md
View File

@ -29,12 +29,14 @@ Listening on [>]
Tokens can be bcrypted enabling an additional layer of security, as the clear-text version of the token would not be persisted on the server configuration file.
You can generate bcrypted tokens and passwords using the [`mkpasswd`](../../../../nats-tools/mkpasswd.md) tool:
You can generate bcrypted tokens and passwords using the [`nats`](../../../../nats-tools/natscli.md) tool:
```text
> mkpasswd
pass: dag0HTXl4RGg7dXdaJwbC8
bcrypt hash: $2a$11$PWIFAL8RsWyGI3jVZtO9Nu8.6jOxzxfZo7c/W0eLk017hjgUKWrhy
> nats server passwd
? Enter password [? for help] **********************
? Reenter password [? for help] **********************
$2a$11$PWIFAL8RsWyGI3jVZtO9Nu8.6jOxzxfZo7c/W0eLk017hjgUKWrhy
```
Here's a simple configuration file:

12
nats-server/configuration/securing_nats/auth_intro/username_password.md
View File

@ -30,12 +30,14 @@ authorization: {
## Bcrypted Passwords
Username/password also supports bcrypted passwords using the [`mkpasswd`](../../../../nats-tools/mkpasswd.md) tool. Simply replace the clear text password with the bcrypted entries:
Username/password also supports bcrypted passwords using the [`nats`](../../../../nats-tools/natscli.md) tool. Simply replace the clear text password with the bcrypted entries:
```text
> mkpasswd
pass: (Uffs#rG42PAu#Oxi^BNng
bcrypt hash: $2a$11$V1qrpBt8/SLfEBr4NJq4T.2mg8chx8.MTblUiTBOLV3MKDeAy.f7u
> nats server passwd
? Enter password [? for help] **********************
? Reenter password [? for help] **********************
$2a$11$V1qrpBt8/SLfEBr4NJq4T.2mg8chx8.MTblUiTBOLV3MKDeAy.f7u
```
And on the configuration file:
@ -44,7 +46,7 @@ And on the configuration file:
authorization: {
users: [
{user: a, password: "$2a$11$V1qrpBt8/SLfEBr4NJq4T.2mg8chx8.MTblUiTBOLV3MKDeAy.f7u"},
...
...
]
}
```

63
nats-server/configuration/securing_nats/jwt/resolver.md
View File

@ -1,9 +1,10 @@
# Account lookup using Resolver
The `resolver` configuration option is used in conjunction with [NATS JWT Authentication](./) and [nsc](../../../../nats-tools/nsc/). The `resolver` option specifies a URL where the nats-server can retrieve an account JWT. There are two built-in resolver implementations:
The `resolver` configuration option is used in conjunction with [NATS JWT Authentication](./) and [nsc](../../../../nats-tools/nsc/). The `resolver` option specifies a URL where the nats-server can retrieve an account JWT. There are three built-in resolver implementations:
* `URL`
* `MEMORY`
* [`URL`](resolver.md#URL-Resolver)
* [`MEMORY`](resolver.md#Memory)
* [NATS Based Resolver](resolver.md#nats-based-resolver)
> If the operator JWT specified in `operator` contains an account resolver URL, `resolver` only needs to be specified in order to overwrite that default.
@ -34,3 +35,59 @@ The `MEMORY` resolver is recommended when the server has a small number of accou
For more information on how to configure a memory resolver, see [this tutorial](mem_resolver.md).
## NATS Based Resolver
The NATS based resolver embeds the functionality of the [account server](https://github.com/nats-io/nats-account-server) inside the nats-server. In order to avoid having to store all account JWT on every server, this resolver has two sub types `full` and `cache`. Their commonalities are that they exchange/lookup account JWT via NATS and the system account, and store them in a local \(not shared\) directory.
### Full
The Full resolver stores all JWTs and exchanges them in an eventually consistent way with other resolvers of the same type. [`nsc`](../../../../nats-tools/nsc/) supports push/pull/purge with this resolver type. [JWTs](https://github.com/nats-io/nats.docs/tree/8c85d9c047d2203c7867b62a8415cdfa4d117f04/nats-server/configuration/nats-server/configuration/securing_nats/jwt/README.md), uploaded this way, are stored in a directory the server has exclusive access to.
```yaml
resolver: {
type: full
# Directory in which account jwt will be stored
dir: './jwt'
# In order to support jwt deletion, set to true
# If the resolver type is full delete will rename the jwt.
# This is to allow manual restoration in case of inadvertent deletion.
# To restore a jwt, remove the added suffix .delete and restart or send a reload signal.
# To free up storage you must manually delete files with the suffix .delete.
allow_delete: false
# Interval at which a nats-server with a nats based account resolver will compare
# it's state with one random nats based account resolver in the cluster and if needed,
# exchange jwt and converge on the same set of jwt.
interval: "2m"
# limit on the number of jwt stored, will reject new jwt once limit is hit.
limit: 1000
}
```
This resolver type also supports `resolver_preload`. When present, JWTs are listed and stored in the resolver. There, they may be subject to updates. Restarts of the `nats-server` will hold on to these more recent versions.
Not every server in a cluster needs to be set to `full`. You need enough to still serve your workload adequately, while some servers are offline.
### Cache
The Cache resolver only stores a subset of [JWT](https://github.com/nats-io/nats.docs/tree/8c85d9c047d2203c7867b62a8415cdfa4d117f04/nats-server/configuration/nats-server/configuration/securing_nats/jwt/README.md) and evicts others based on an LRU scheme. Missing JWTs are downloaded from `full` nats based resolver. This resolver is essentially the URL Resolver in NATS.
```yaml
resolver: {
type: cache
# Directory in which account jwt will be store
dir: "./"
# limit on the number of jwt stored, will evict old jwt once limit is hit.
limit: 1000
# How long to hold on to a jwt before discarding it.
ttl: "2m"
}
```
### NATS Based Resolver - Integration
The NATS based resolver utilizes the system account for lookup and upload of account [JWTs](https://github.com/nats-io/nats.docs/tree/8c85d9c047d2203c7867b62a8415cdfa4d117f04/nats-server/configuration/nats-server/configuration/securing_nats/jwt/README.md) . If your application requires tighter integration you can make use of these subjects for tighter integration.
To upload or update any generated account JWT without [`nsc`](../../../../nats-tools/nsc/), send it as a request to `$SYS.REQ.CLAIMS.UPDATE`. Each participating `full` NATS based account resolver will respond with a message detailing success or failure.
To serve a requested account [JWT](https://github.com/nats-io/nats.docs/tree/8c85d9c047d2203c7867b62a8415cdfa4d117f04/nats-server/configuration/nats-server/configuration/securing_nats/jwt/README.md) yourself and essentially implement an account server, subscribe to `$SYS.REQ.ACCOUNT.*.CLAIMS.LOOKUP` and respond with the account JWT corresponding to the requested account id \(wildcard\).

11
nats-server/configuration/websocket.md Normal file
View File

@ -0,0 +1,11 @@
# Websocket
_Supported since NATS Server version 2.2_
Websocket support can be enabled in the server and may be used alongside the traditional TCP socket connections. TLS, compression and Origin Header checking are supported.
**Important**
* NATS Supports only Websocket data frames in Binary, not Text format \([https://tools.ietf.org/html/rfc6455\#section-5.6](https://tools.ietf.org/html/rfc6455#section-5.6)\). The server will always send in Binary and your clients MUST send in Binary too.
* For writers of client libraries: a Websocket frame is not guaranteed to contain a full NATS protocol \(actually will generally not\). Any data from a frame must be going through a parser that can handle partial protocols. See the protocol description [here](../../nats-protocol/nats-protocol/).

158
nats-server/configuration/websocket/websocket_conf.md Normal file
View File

@ -0,0 +1,158 @@
# Configuration
To enable websocket support in the server, add a `websocket` configuration
block in the server's configuration file like the following:
```
websocket {
# Specify a host and port to listen for websocket connections
#
# listen: "host:port"
# It can also be configured with individual parameters,
# namely host and port.
#
# host: "hostname"
port: 443
# This will optionally specify what host:port for websocket
# connections to be advertised in the cluster.
#
# advertise: "host:port"
# TLS configuration is required by default
#
tls {
cert_file: "/path/to/cert.pem"
key_file: "/path/to/key.pem"
}
# For test environments, you can disable the need for TLS
# by explicitly setting this option to `true`
#
# no_tls: true
# [Cross-origin resource sharing option](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). When set to `true`, the HTTP origin header must match the request’s hostname.
# The default is `false`.
#
# same_origin: true
# [Cross-origin resource sharing option](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). List of accepted origins. When empty, and `same_origin` is `false`, clients from any origin are allowed to connect.
# This list specifies the only accepted values for the client's request Origin header. The scheme,
# host and port must match. By convention, the absence of TCP port in the URL will be port 80
# for an "http://" scheme, and 443 for "https://".
#
# allowed_origins [
# "http://www.example.com"
# "https://www.other-example.com"
# ]
# This enables support for compressed websocket frames
# in the server. For compression to be used, both server
# and client have to support it.
#
# compression: true
# This is the total time allowed for the server to
# read the client request and write the response back
# to the client. This includes the time needed for the
# TLS handshake.
#
# handshake_timeout: "2s"
# Name for an HTTP cookie, that if present will be used as a client JWT.
# If the client specifies a JWT in the CONNECT protocol, this option is ignored.
# The cookie should be set by the HTTP server as described [here](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies).
# This setting is useful when generating NATS `Bearer` client JWTs as the
# result of some authentication mechanism. The HTTP server after correct
# authentication can issue a JWT for the user, that is set securely preventing
# access by unintended scripts. Note these JWTs must be [NATS JWTs](https://docs.nats.io/nats-server/configuration/securing_nats/jwt).
#
# jwt_cookie: "my_jwt_cookie_name"
# If no user name is provided when a websocket client connects, will default
# this user name in the authentication phase. If specified, this will
# override, for websocket clients, any `no_auth_user` value defined in the
# main configuration file.
# Note that this is not compatible with running the server in operator mode.
#
# no_auth_user: "my_username_for_apps_not_providing_credentials"
# See below to know what is the normal way of limiting websocket clients
# to specific users.
# If there are no users specified in the configuration, this simple authorization
# block allows you to override the values that would be configured in the
# equivalent block in the main section.
#
# authorization {
# # If this is specified, the client has to provide the same username
# # and password to be able to connect.
# # username: "my_user_name"
# # password: "my_password"
#
# # If this is specified, the password field in the CONNECT has to
# # match this token.
# # token: "my_token"
#
# # This overrides the main's authorization timeout. For consistency
# # with the main's authorization configuration block, this is expressed
# # as a number of seconds.
# # timeout: 2.0
#}
}
```
## Authorization of Websocket Users
A new field when configuring users allows you to restrict which type of connections are allowed for a specific user.
Consider this configuration:
```
authorization {
users [
{user: foo password: foopwd, permission: {...}}
{user: bar password: barpwd, permission: {...}}
]
}
```
If a websocket client were to connect and use the username `foo` and password `foopwd`, it would be accepted.
Now suppose that you would want websocket client to only be accepted if it connected using the username `bar`
and password `barpwd`, then you would use the option `allowed_connection_types` to restrict which type
of connections can bind to this user.
```
authorization {
users [
{user: foo password: foopwd, permission: {...}}
{user: bar password: barpwd, permission: {...}, allowed_connection_types: ["WEBSOCKET"]}
]
}
```
The option `allowed_connection_types` (also can be named `connection_types` or `clients`) as you can see
is a list, and you can allow several type of clients. Suppose you want the user `bar` to accept both
standard NATS clients and websocket clients, you would configure the user like this:
```
authorization {
users [
{user: foo password: foopwd, permission: {...}}
{user: bar password: barpwd, permission: {...}, allowed_connection_types: ["STANDARD", "WEBSOCKET"]}
]
}
```
The absence of `allowed_connection_types` means that all type of connections are allowed (the default behavior).
The possible values are currently:
* `STANDARD`
* `WEBSOCKET`
* `LEAFNODE`
* `MQTT`
## Leaf nodes connections
You can configure remote Leaf node connections so that they connect to the Websocket port instead of the Leaf node port.
See [Leafnode](../leafnodes/leafnode_conf.md#connecting-using-websocket-protocol) section.

16
nats-server/nats_admin/lame_duck_mode.md
View File

@ -1,9 +1,6 @@
# Lame Duck Mode
In production we recommend that a server is shut down with ​lame duck mode​
as a graceful way to slowly evict clients. With large deployments this
mitigates the "thundering herd" situation that will place CPU pressure on
servers as TLS enabled clients reconnect.
In production we recommend that a server is shut down with ​lame duck mode​ as a graceful way to slowly evict clients. With large deployments this mitigates the "thundering herd" situation that will place CPU pressure on servers as TLS enabled clients reconnect.
## Server
@ -13,16 +10,9 @@ Lame duck mode is initiated by signaling the server:
nats-server --signal ldm
```
After entering lame duck mode, the server will stop accepting new connections,
wait for a 10 second grace period, then begin to evict clients over a period of time
configurable by the [lame_duck_duration](https://docs.nats.io/nats-server/configuration#runtime-configuration)
configuration option. This period defaults to 2 minutes.
After entering lame duck mode, the server will stop accepting new connections, wait for a 10 second grace period, then begin to evict clients over a period of time configurable by the [lame\_duck\_duration](https://docs.nats.io/nats-server/configuration#runtime-configuration) configuration option. This period defaults to 2 minutes.
## Clients
When entering lame duck mode, the server will send a message to clients. Some
maintainer supported clients will invoke an optional callback indicating that
a server is entering lame duck mode. This is used for cases where an application
can benefit from preparing for the short outage between the time it is evicted and
automatically reconnected to another server.
When entering lame duck mode, the server will send a message to clients. Some maintainer supported clients will invoke an optional callback indicating that a server is entering lame duck mode. This is used for cases where an application can benefit from preparing for the short outage between the time it is evicted and automatically reconnected to another server.

2
nats-server/nats_admin/upgrading_cluster.md
View File

@ -38,7 +38,7 @@ nats-sub -s nats://localhost:4222 ">"
`nats-sub` is a subscriber sample included with all NATS clients. `nats-sub` subscribes to a subject and prints out any messages received. You can find the source code to the go version of `nats-sub` in [GitHub](https://github.com/nats-io/nats.go/tree/master/examples). After starting the subscriber you should see a message on 'A' that a new client connected.
We have two servers and a client. Time to simulate our rolling upgrade. But wait, before we upgrade 'A', let's introduce a new server 'C'. Server 'C' will join the existing cluster while we perform the upgrade. Its sole purpose is to provide an additional place where clients can go other than 'A' and ensure we don't end up with a single server serving all the clients after the upgrade procedure. Clients will randomly select a server when connecting unless a special option is provided that disables that functionality \(usually called 'DontRandomize' or 'noRandomize'\). You can read more about ["Avoiding the Thundering Herd"](). Suffice it to say that clients redistribute themselves about evenly between all servers in the cluster. In our case 1/2 of the clients on 'A' will jump over to 'B' and the remaining half to 'C'.
We have two servers and a client. Time to simulate our rolling upgrade. But wait, before we upgrade 'A', let's introduce a new server 'C'. Server 'C' will join the existing cluster while we perform the upgrade. Its sole purpose is to provide an additional place where clients can go other than 'A' and ensure we don't end up with a single server serving all the clients after the upgrade procedure. Clients will randomly select a server when connecting unless a special option is provided that disables that functionality \(usually called 'DontRandomize' or 'noRandomize'\). You can read more about ["Avoiding the Thundering Herd"](upgrading_cluster.md). Suffice it to say that clients redistribute themselves about evenly between all servers in the cluster. In our case 1/2 of the clients on 'A' will jump over to 'B' and the remaining half to 'C'.
Let's start our temporary server:

2
nats-streaming-concepts/channels/subscriptions/queue-group.md
View File

@ -8,5 +8,5 @@ When the last member of the group leaves \(subscription unsubscribed/closed/or c
A queue subscription can also be durable. For that, the client needs to provide a queue and durable name. The behavior is, as you would expect, a combination of queue and durable subscription. Though unlike a durable subscription, the client ID is not part of the queue group name since the client ID must be unique, and would prevent more than one connection to participate in the queue group. The main difference between a queue subscription and a durable one, is that when the last member leaves the group, the state of the group will be maintained by the server. Later, when a member rejoins the group, the delivery will resume.
_**Note: For a durable queue subscription, the last member to**_ **unsubscribe** _**\(not simply close\) causes the group to be removed from the server.**_
_**Note: For a durable queue subscription, the last member to**_ **unsubscribe** _**\(not simply close\) causes the group to be removed from the server.**_

12
nats-streaming-concepts/monitoring/endpoints.md
View File

@ -98,7 +98,8 @@ You can control these via URL arguments \(limit and offset\). For example: [http
"clients": [
{
"id": "benchmark-sub-0",
"hb_inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayQK"
"hb_inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayQK",
"subs_count": 1
}
]
}
@ -119,6 +120,7 @@ You can also report detailed subscription information on a per client basis usin
{
"id": "benchmark-sub-0",
"hb_inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayQK",
"subs_count": 1,
"subscriptions": {
"foo": [
{
@ -146,6 +148,7 @@ You can select a specific client based on its client ID with `client=<id>`, and
{
"id": "me",
"hb_inbox": "_INBOX.HG0uDuNtAPxJQ1lVjIC2sr",
"subs_count": 1,
"subscriptions": {
"foo": [
{
@ -222,6 +225,7 @@ You can also get the list of subscriptions with `subs=1`. For example: [http://l
"bytes": 0,
"first_seq": 0,
"last_seq": 0,
"subs_count": 1,
"subscriptions": [
{
"client_id": "me",
@ -249,7 +253,8 @@ You can select a specific channel based on its name with `channel=name`. For exa
"msgs": 649234,
"bytes": 97368590,
"first_seq": 1,
"last_seq": 649234
"last_seq": 649234,
"subs_count": 1
}
```
@ -262,6 +267,7 @@ And again, you can get detailed subscriptions with `subs=1`. For example: [http:
"bytes": 105698990,
"first_seq": 1,
"last_seq": 704770,
"subs_count": 10,
"subscriptions": [
{
"client_id": "me",
@ -301,6 +307,7 @@ For durables that are currently running, the `is_offline` field is set to `false
"bytes": 0,
"first_seq": 0,
"last_seq": 0,
"subs_count": 1,
"subscriptions": [
{
"client_id": "me",
@ -328,6 +335,7 @@ When that same durable goes offline, `is_offline` is be set to `true`. Although
"bytes": 0,
"first_seq": 0,
"last_seq": 0,
"subs_count": 1,
"subscriptions": [
{
"client_id": "me",

15
nats-streaming-server/configuring/cfgfile.md
View File

@ -68,20 +68,20 @@ Note: You may need to scroll horizontally to see all columns.
| hb\_interval | Interval at which the server sends an heartbeat to a client | Duration | `hb_interval: "10s"` | `30s` | v0.3.6 |
| hb\_timeout | How long the server waits for a heartbeat response from the client before considering it a failed heartbeat | Duration | `hb_timeout: "10s"` | `10s` | v0.3.6 |
| hb\_fail\_count | Count of failed heartbeats before server closes the client connection. The actual total wait is: \(fail count + 1\) \* \(hb interval + hb timeout\) | Number | `hb_fail_count: 2` | `10` | v0.3.6 |
| ft\_group | In Fault Tolerance mode, you can start a group of streaming servers with only one server being active while others are running in standby mode. This is the name of this FT group | String | `ft_group: "my_ft_group"` | N/A | v0.4.0|
| ft\_group | In Fault Tolerance mode, you can start a group of streaming servers with only one server being active while others are running in standby mode. This is the name of this FT group | String | `ft_group: "my_ft_group"` | N/A | v0.4.0 |
| partitioning | If set to true, a list of channels must be defined in store\_limits/channels section. This section then serves two purposes, overriding limits for a given channel or adding it to the partition | `true` or `false` | `partitioning: true` | `false` | v0.5.0 |
| sql\_options | SQL Store specific options | Map: `sql_options: { ... }` | [**See details below**](cfgfile.md#sql-options-configuration) | | v0.7.0 |
| cluster | Cluster Configuration | Map: `cluster: { ... }` | [**See details below**](cfgfile.md#cluster-configuration) | | v0.9.0 |
| syslog_name | On Windows, when running several servers as a service, use this name for the event source | String | | v0.11.0 |
| syslog\_name | On Windows, when running several servers as a service, use this name for the event source | String | | v0.11.0 | |
| encrypt | Specify if server should encrypt messages \(only the payload\) when storing them | `true` or `false` | `encrypt: true` | `false` | v0.12.0 |
| encryption\_cipher | Cipher to use for encryption. Currently support AES and CHAHA \(ChaChaPoly\). Defaults to AES | `AES` or `CHACHA` | `encryption_cipher: "AES"` | Depends on platform | v0.12.0 |
| encryption\_key | Encryption key. It is recommended to specify the key through the `NATS_STREAMING_ENCRYPTION_KEY` environment variable instead | String | `encryption_key: "mykey"` | N/A | v0.12.0 |
| credentials | Credentials file to connect to external NATS 2.0+ Server | String | `credentials: "streaming_server.creds"` | N/A | v0.16.2 |
| username | Username is used to connect to a NATS Server when authentication with multiple users is enabled | String | `username: "streaming_server"` | N/A | v0.19.0 |
| password | Password used with above `username` | String | `password: "password"` | N/A | v0.19.0 |
| password | Password used with above `username` | String | `password: "password"` | N/A | v0.19.0 |
| token | Authentication token if the NATS Server requires a token | String | `token: "some_token"` | N/A | v0.19.0 |
| nkey\_seed\_file | Path to an NKey seed file \(1\) if NKey authentication is used | File Path | `nkey_seed_file: "/path/to/some/seedfile"` | N/A | v0.19.0 |
| replace_durable | Replace the existing durable subscription instead of reporting a duplicate durable error | `true` or `false` | `replace_durable: true` | `false` | v0.20.0 |
| replace\_durable | Replace the existing durable subscription instead of reporting a duplicate durable error | `true` or `false` | `replace_durable: true` | `false` | v0.20.0 |
Notes:
@ -196,7 +196,10 @@ For a given channel, the possible parameters are:
| raft\_lease\_timeout | Specifies how long a leader waits without being able to contact a quorum of nodes before stepping down as leader | Duration | `raft_lease_timeout: "1s"` | `1s` | v0.11.2 |
| raft\_commit\_timeout | Specifies the time without an Apply\(\) operation before sending an heartbeat to ensure timely commit. Due to random staggering, may be delayed as much as 2x this value | Duration | `raft_commit_timeout: "100ms"` | `100ms` | v0.11.2 |
| proceed\_on\_restore\_failure | Allow a non leader node to proceed with restore failures, do not use unless you understand the risks! | `true` or `false` | `proceed_on_restore_failure: true` | `false` | v0.17.0 |
| allow_add_remove_node| Enable the ability to send NATS requests to the leader to add/remove cluster nodes | `true` or `false` | `allow_add_remove_node: true` | `false` | v0.19.0 |
| allow\_add\_remove\_node | Enable the ability to send NATS requests to the leader to add/remove cluster nodes | `true` or `false` | `allow_add_remove_node: true` | `false` | v0.19.0 |
| bolt\_free\_list\_sync | Causes the RAFT log to synchronize the free list on write operations. Reduces performance at runtime, but makes the recovery faster | `true` or `false` | `bolt_free_list_sync: true` | `false` | v0.21.0 |
| bolt\_free\_list\_map | Sets the backend freelist type to use a map instead of the default array type. Improves performance for large RAFT logs, with fragmented free list | `true` or `false` | `bolt_free_list_map: true` | `false` | v0.21.0 |
| nodes\_connections | Enable creation of dedicated NATS connections to communicate with other nodes | `true` or `false` | `nodes_connections: true` | `false` | v0.21.0 |
## SQL Options Configuration
@ -206,5 +209,5 @@ For a given channel, the possible parameters are:
| source | How to connect to the database. This is driver specific | String | `source: "ivan:pwd@/nss_db"` | N/A | v0.7.0 |
| no\_caching | Enable/Disable caching for messages and subscriptions operations. | `true` or `false` | `no_caching: false` | `false` \(caching enabled\) | v0.7.0 |
| max\_open\_conns | Maximum number of opened connections to the database. Value &lt;= 0 means no limit. | Number | `max_open_conns: 5` | unlimited | v0.7.0 |
| bulk\_insert\_limit | Maximum number of messages stored with a single SQL "INSERT" statement. The default behavior is to send individual insert commands as part of a SQL transaction. | Number | `bulk_insert_limit: 1000` | `0` (not enabled) | v0.20.0 |
| bulk\_insert\_limit | Maximum number of messages stored with a single SQL "INSERT" statement. The default behavior is to send individual insert commands as part of a SQL transaction. | Number | `bulk_insert_limit: 1000` | `0` \(not enabled\) | v0.20.0 |

52
nats-tools/mkpasswd.md
View File

@ -1,52 +0,0 @@
# mkpasswd
The server supports hashing of passwords and authentication tokens using `bcrypt`. To take advantage of this, simply replace the plaintext password in the configuration with its `bcrypt` hash, and the server will automatically utilize `bcrypt` as needed.
A utility for creating `bcrypt` hashes is included with the nats-server distribution \(`util/mkpasswd.go`\). Running it with no arguments will generate a new secure password along with the associated hash. This can be used for a password or a token in the configuration.
## Installing `mkpasswd`
If you have [go installed](https://golang.org/doc/install), you can easily install the `mkpasswd` tool by doing:
```text
> go get github.com/nats-io/nats-server/util/mkpasswd
```
Alternatively, you can:
```text
> git clone git@github.com:nats-io/nats-server
> cd nats-server/util/mkpasswd
> go install mkpasswd.go
```
## Generating bcrypted passwords
With `mkpasswd` installed:
```text
> mkpasswd
pass: #IclkRPHUpsTmACWzmIGXr
bcrypt hash: $2a$11$3kIDaCxw.Glsl1.u5nKa6eUnNDLV5HV9tIuUp7EHhMt6Nm9myW1aS
```
If you already have a password selected, you can supply the `-p` flag on the command line, enter your desired password, and a `bcrypt` hash will be generated for it:
```text
> mkpasswd -p
Enter Password: *******
Reenter Password: ******
bcrypt hash: $2a$11$3kIDaCxw.Glsl1.u5nKa6eUnNDLV5HV9tIuUp7EHhMt6Nm9myW1aS
```
To use the password on the server, add the hash into the server configuration file's authorization section.
```text
authorization {
user: derek
password: $2a$11$3kIDaCxw.Glsl1.u5nKa6eUnNDLV5HV9tIuUp7EHhMt6Nm9myW1aS
}
```
Note the client will still have to provide the plain text version of the password, the server however will only store the hash to verify that the password is correct when supplied.

11
nats-tools/nas/README.md
View File

@ -1,11 +1,12 @@
# nats-account-server
The [NATS Account Server](https://github.com/nats-io/nats-account-server) is an HTTP server that hosts and vends [JWTs](../../nats-server/configuration/securing_nats/jwt/) for nats-server 2.0 account authentication. The server supports an number of stores which enable it to serve account [JWTs](../../nats-server/configuration/securing_nats/jwt/) from:
The [NATS Account Server](https://github.com/nats-io/nats-account-server) is an HTTP server that hosts and vends [JWTs](../../nats-server/configuration/securing_nats/jwt/) for nats-server 2.0 account authentication. The server supports an number of stores which enable it to serve account [JWTs](../../nats-server/configuration/securing_nats/jwt/) from a [directory](nas_conf.md#directory-configuration)
* a [directory](nas_conf.md#directory-configuration)
* an [NSC](../nsc/nsc.md) [directory](nas_conf.md#nsc-configuration)
> The nats server can be configured with a [memory resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#memory) as well. This avoids usage of the account server.
> The nats server can be configured with a [memory resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#memory) as well. This avoids usage of the account server. The NATS server can be configured with a [NATS based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) for the same purpose as well.
>
> Usage of [full NATS based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) over [NATS Account Server](https://github.com/nats-io/nats-account-server) is recommended.
>
> The [NATS Account Server](https://github.com/nats-io/nats-account-server) also speaks the [full nats based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) protocol and can be used as such.
The server can operate in a _READ ONLY_ mode where it serves content from a directory, or in [notification mode](notifications.md), where it can notify a NATS server that a JWT in the store has been modified, updating the NATS server with the updated JWT.

2
nats-tools/nas/dir_store.md
View File

@ -60,7 +60,7 @@ Generated account key - private key stored "~/.nkeys/AAA/accounts/B/B.nk"
Success! - added account "B"
```
With the account and a couple of users in place, let's push all the accounts to the nats-account-server. If the account JWT server URL is not set on the operator, you may want to set it. Note that account servers typically require the path `/jwt/v1` in addition to the protocol and hostport (or you can specify the `--account-jwt-server-url` flag to nsc's `push` command).
With the account and a couple of users in place, let's push all the accounts to the nats-account-server. If the account JWT server URL is not set on the operator, you may want to set it. Note that account servers typically require the path `/jwt/v1` in addition to the protocol and hostport \(or you can specify the `--account-jwt-server-url` flag to nsc's `push` command\).
```text
❯ nsc edit operator --account-jwt-server-url http://localhost:9090/jwt/v1

61
nats-tools/nas/nas_conf.md
View File

@ -7,63 +7,7 @@ Basic configuration revolves around 4 settings:
* NATS \(for cases where updates are enabled\)
* Logging
For complete information on please refer to the project's [Github](https://github.com/nats-io/nats-account-server).
## `nsc` Configuration
For a basic usage of the server you can specify the `-nsc` flag, and specify the path to an operator in your environment.
> If you have not yet created an operator or accounts, you'll need to do so before continuing. See [NSC](../nsc/)
You can easily locate the path by running `nsc env` to print your `nsc` settings:
```text
> nsc env
╭──────────────────────────────────────────╮
│ NSC Environment │
├──────────────────┬─────┬─────────────────┤
│ Setting │ Set │ Effective Value │
├──────────────────┼─────┼─────────────────┤
│ $NKEYS_PATH │ No │ ~/.nkeys │
│ $NSC_HOME │ No │ ~/.nsc │
│ Config │ │ ~/.nsc/nsc.json │
├──────────────────┼─────┼─────────────────┤
│ Stores Dir │ │ ~/.nsc/nats │
│ Default Operator │ │ Test │
│ Default Account │ │ TestAccount │
│ Default Cluster │ │ │
╰──────────────────┴─────┴─────────────────╯
```
The path you are interested in the `Stores Dir`. This is the root of all operators, you'll also need the name of your operator. If your current operator is not listed, you can list all your available operators by doing:
```text
> nsc list operators
```
To start the `nats-account-server` with the operator `Test`:
```text
> nats-account-server -nsc ~/.nsc/nats/Test
2019/05/10 11:22:41.845148 [INF] starting NATS Account server, version 0.0-dev
2019/05/10 11:22:41.845241 [INF] server time is Fri May 10 11:22:41 CDT 2019
2019/05/10 11:22:41.845245 [INF] loading operator from /Users/synadia/.nsc/nats/Test/Test.jwt
2019/05/10 11:22:41.846367 [INF] creating a read-only store for the NSC folder at /Users/synadia/.nsc/nats/Test
2019/05/10 11:22:41.846459 [INF] NATS is not configured, server will not fire notifications on update
2019/05/10 11:22:41.855291 [INF] http listening on port 9090
2019/05/10 11:22:41.855301 [INF] nats-account-server is running
2019/05/10 11:22:41.855303 [INF] configure the nats-server with:
2019/05/10 11:22:41.855306 [INF] resolver: URL(http://localhost:9090/jwt/v1/accounts/)
```
By default the server will serve JWTs on the localhost at port 9090. The last line in the shown in the printout is important, that is the resolver URL you'll have to provide on your NATS server configuration. You'll also need the matching operator JWT which is on `~/.nsc/nats/Test/Test.jwt` if you are following the example above. On the server configuration you'll need to expand the `~` as necessary. Here's what my NATS server configuration looks like:
```text
operator: /Users/synadia/.nsc/nats/Test/Test.jwt
resolver: URL(http://localhost:9090/jwt/v1/accounts/)
```
Note that servers you create with the `-nsc` option \(or store option\) are read-only. This means that the server will not accept POST requests to update the JWT store.
For complete information, please refer to the project's [Github](https://github.com/nats-io/nats-account-server).
## Directory Configuration
@ -95,7 +39,7 @@ A step by step tutorial using directory configuration can be found [here](dir_st
## Configuration File
While the `-nsc` and `-dir` store flags are sufficient for some very simple developer setups, any production or non-read-only server will require a configuration file.
While the `-dir` store flag is sufficient for some very simple developer setups, any production or non-read-only server will require a configuration file.
Let's take a look at the configuration options:
@ -117,7 +61,6 @@ Let's take a look at the configuration options:
| Option | Description |
| :--- | :--- |
| `dir` | Configures a directory as a store. |
| `nsc` | Configures an nsc read-only store. The value should be the path to an operator _directory_. Option is mutually exclusive with `dir`. |
| `readonly` | If `true`, the store will not accept POST requests. Note that to receive requests, the store must also have `operatorjwtpath` specified as a root option. |
| `shard` | If `true`, JWTs will be stored in multiple sub directories of the store directory. |

2
nats-tools/nats-tools.md
View File

@ -2,7 +2,7 @@
The NATS Ecosystem has many tools to support server configuration, enhance monitoring or tune performance:
* [mkpasswd](mkpasswd.md) - Generates or bcrypts passwords
* [nats](natscli.md) - Interact with and manage NATS
* [nk](nk.md) - Generate NKeys
* [nsc](nsc/) - Configure Operators, Accounts and Users
* [nats account server](nas/) - Serve Account JWTs

60
nats-tools/natscli.md Normal file
View File

@ -0,0 +1,60 @@
# nats
A command line utility to interact with and manage NATS.
This utility replaces various past tools that were named in the form `nats-sub` and `nats-pub`, adds several new capabilities and support full JetStream management.
Check out the repo for more details: [github.com/nats-io/natscli](https://github.com/nats-io/natscli).
## Installing `nats`
For macOS:
```text
> brew tap nats-io/nats-tools
> brew install nats-io/nats-tools/nats
```
For Arch Linux:
```text
> yay natscli
```
For Docker:
```text
docker pull synadia/nats-box:latest
docker run -ti synadia/nats-box
```
Binaries are also available as [GitHub Releases](https://github.com/nats-io/natscli/releases).
## Generating bcrypted passwords
The server supports hashing of passwords and authentication tokens using `bcrypt`. To take advantage of this, simply replace the plaintext password in the configuration with its `bcrypt` hash, and the server will automatically utilize `bcrypt` as needed.
The `nats` utility has a command for creating `bcrypt` hashes. This can be used for a password or a token in the configuration.
With `nats` installed:
```text
> nats server passwd
? Enter password [? for help] **********************
? Reenter password [? for help] **********************
$2a$11$3kIDaCxw.Glsl1.u5nKa6eUnNDLV5HV9tIuUp7EHhMt6Nm9myW1aS
```
To use the password on the server, add the hash into the server configuration file's authorization section.
```text
authorization {
user: derek
password: $2a$11$3kIDaCxw.Glsl1.u5nKa6eUnNDLV5HV9tIuUp7EHhMt6Nm9myW1aS
}
```
Note the client will still have to provide the plain text version of the password, the server however will only store the hash to verify that the password is correct when supplied.

6
whats_new_20.md
View File

@ -54,12 +54,10 @@ Services and streams are mechanisms to share messages between accounts.
Think of a service as an RPC endpoint into an account. Behind that account there might be many microservices working in concert to handle requests, but from outside the account there is simply one subject exposed.
**Services** definition to share an endpoint:
**Service** definitions share an endpoint:
* Export a service to allow other accounts to import
* Import a service to allow requests to be sent securely and seamlessly to
another account
* Import a service to allow requests to be sent securely and seamlessly to another account
Use cases include most applications - anything that accepts a request and returns a response.