taigrr/nats.docs
1
0
Fork 0
mirror of https://github.com/taigrr/nats.docs synced 2025-01-18 04:03:23 -08:00
Code Issues Projects Releases Wiki Activity

adding JetStream Docs

Browse Source
This commit is contained in:
ainsley
2020-11-24 15:12:44 -06:00
parent 8eb1e6a171
commit 1fd9cab1a5
19 changed files with 2109 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

247
jetstream/administration/account.md Normal file
View File

@@ -0,0 +1,247 @@
### Account Information
JetStream is multi-tenant so you will need to check that your account is enabled for JetStream and is not limited. You can view your limits as follows:
```nohighlight
$ nats account info
Memory: 0 B of 6.4 GB
Storage: 0 B of 1.1 TB
Streams: 1 of Unlimited
```
### Streams
The first step is to set up storage for our `ORDERS` related messages, these arrive on a wildcard of subjects all flowing into the same Stream and they are kept for 1 year.
#### Creating
```nohighlight
$ nats str add ORDERS
? Subjects to consume ORDERS.*
? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Message count limit -1
? Message size limit -1
? Maximum message age limit 1y
? Maximum individual message size [? for help] (-1) -1
Stream ORDERS was created
Information for Stream ORDERS
Configuration:
Subjects: ORDERS.*
Acknowledgements: true
Retention: File - Limits
Replicas: 1
Maximum Messages: -1
Maximum Bytes: -1
Maximum Age: 8760h0m0s
Maximum Message Size: -1
Maximum Consumers: -1
Statistics:
Messages: 0
Bytes: 0 B
FirstSeq: 0
LastSeq: 0
Active Consumers: 0
```
You can get prompted interactively for missing information as above, or do it all on one command. Pressing `?` in the CLI will help you map prompts to CLI options:
```
$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old
```
Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`:
```
$ nats str add ORDERS --config orders.json
```
#### Listing
We can confirm our Stream was created:
```nohighlight
$ nats str ls
Streams:
ORDERS
```
#### Querying
Information about the configuration of the Stream can be seen, and if you did not specify the Stream like below, it will prompt you based on all known ones:
```nohighlight
$ nats str info ORDERS
Information for Stream ORDERS
Configuration:
Subjects: ORDERS.*
No Acknowledgements: false
Retention: File - Limits
Replicas: 1
Maximum Messages: -1
Maximum Bytes: -1
Maximum Age: 8760h0m0s
Maximum Consumers: -1
Statistics:
Messages: 0
Bytes: 0 B
FirstSeq: 0
LastSeq: 0
Active Consumers: 0
```
Most commands that show data as above support `-j` to show the results as JSON:
```nohighlight
$ nats str info ORDERS -j
{
"config": {
"name": "ORDERS",
"subjects": [
"ORDERS.*"
],
"retention": "limits",
"max_consumers": -1,
"max_msgs": -1,
"max_bytes": -1,
"max_age": 31536000000000000,
"storage": "file",
"num_replicas": 1
},
"stats": {
"messages": 0,
"bytes": 0,
"first_seq": 0,
"last_seq": 0,
"consumer_count": 0
}
}
```
This is the general pattern for the entire `nats` utility as it relates to JetStream - prompting for needed information but every action can be run non-interactively making it usable as a cli api. All information output like seen above can be turned into JSON using `-j`.
#### Copying
A stream can be copied into another, which also allows the configuration of the new one to be adjusted via CLI flags:
```nohighlight
$ nats str cp ORDERS ARCHIVE --subjects "ORDERS_ARCVHIVE.*" --max-age 2y
Stream ORDERS was created
Information for Stream ARCHIVE
Configuration:
Subjects: ORDERS_ARCVHIVE.*
...
Maximum Age: 17520h0m0s
...
```
#### Editing
A stream configuration can be edited, which allows the configuration to be adjusted via CLI flags. Here I have a incorrectly created ORDERS stream that I fix:
```nohighlight
$ nats str info ORDERS -j | jq .config.subjects
[
"ORDERS.new"
]
$ nats str edit ORDERS --subjects "ORDERS.*"
Stream ORDERS was updated
Information for Stream ORDERS
Configuration:
Subjects: ORDERS.*
....
```
Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`:
```
$ nats str edit ORDERS --config orders.json
```
#### Publishing Into a Stream
Now let's add in some messages to our Stream. You can use `nats pub` to add messages, pass the `--wait` flag to see the publish ack being returned.
You can publish without waiting for acknowledgement:
```nohighlight
$ nats pub ORDERS.scratch hello
Published [sub1] : 'hello'
```
But if you want to be sure your messages got to JetStream and were persisted you can make a request:
```nohighlight
$ nats req ORDERS.scratch hello
13:45:03 Sending request on [ORDERS.scratch]
13:45:03 Received on [_INBOX.M8drJkd8O5otORAo0sMNkg.scHnSafY]: '+OK'
```
Keep checking the status of the Stream while doing this and you'll see it's stored messages increase.
```nohighlight
$ nats str info ORDERS
Information for Stream ORDERS
...
Statistics:
Messages: 3
Bytes: 147 B
FirstSeq: 1
LastSeq: 3
Active Consumers: 0
```
After putting some throw away data into the Stream, we can purge all the data out - while keeping the Stream active:
#### Deleting All Data
To delete all data in a stream use `purge`:
```nohighlight
$ nats str purge ORDERS -f
...
Statistics:
Messages: 0
Bytes: 0 B
FirstSeq: 1,000,001
LastSeq: 1,000,000
Active Consumers: 0
```
#### Deleting A Message
A single message can be securely removed from the stream:
```nohighlight
$ nats str rmm ORDERS 1 -f
```
#### Deleting Sets
Finally for demonstration purposes, you can also delete the whole Stream and recreate it so then we're ready for creating the Consumers:
```
$ nats str rm ORDERS -f
$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1
```

30
jetstream/administration/administration.md Normal file
View File

@@ -0,0 +1,30 @@
## Administration and Usage from the CLI
Once the server is running it's time to use the management tool. This can be downloaded from the [GitHub Release Page](https://github.com/nats-io/jetstream/releases/) or you can use the `synadia/jsm:latest` docker image.
```
$ nats --help
usage: nats [<flags>] <command> [<args> ...]
NATS Management Utility
Flags:
--help Show context-sensitive help (also try --help-long and --help-man).
--version Show application version.
-s, --server="localhost:4222" NATS servers
--creds=CREDS User credentials
--tlscert=TLSCERT TLS public certifcate
--tlskey=TLSKEY TLS private key
--tlsca=TLSCA TLS certifcate authority chain
--timeout=2s Time to give JetStream to respond to queries
Commands:
help [<command>...]
Show help.
...
```
We'll walk through the above scenario and introduce features of the CLI and of JetStream as we recreate the setup above.
Throughout this example, we'll show other commands like `nats pub` and `nats sub` to interact with the system. These are normal existing core NATS commands and JetStream is fully usable by only using core NATS.
We'll touch on some additional features but please review the section on the design model to understand all possible permutations.

219
jetstream/administration/consumers.md Normal file
View File

@@ -0,0 +1,219 @@
### Consumers
Consumers is how messages are read or consumed from the Stream. We support pull and push-based Consumers and the example scenario has both, lets walk through that.
#### Creating Pull-Based Consumers
The `NEW` and `DISPATCH` Consumers are pull-based, meaning the services consuming data from them have to ask the system for the next available message. This means you can easily scale your services up by adding more workers and the messages will get spread across the workers based on their availability.
Pull-based Consumers are created the same as push-based Consumers, just don't specify a delivery target.
```
$ nats con ls ORDERS
No Consumers defined
```
We have no Consumers, lets add the `NEW` one:
I supply the `--sample` options on the CLI as this is not prompted for at present, everything else is prompted. The help in the CLI explains each:
```
$ nats con add --sample 100
? Select a Stream ORDERS
? Consumer name NEW
? Delivery target
? Start policy (all, last, 1h, msg sequence) all
? Filter Stream by subject (blank for all) ORDERS.received
? Maximum Allowed Deliveries 20
Information for Consumer ORDERS > NEW
Configuration:
Durable Name: NEW
Pull Mode: true
Subject: ORDERS.received
Deliver All: true
Deliver Last: false
Ack Policy: explicit
Ack Wait: 30s
Replay Policy: instant
Maximum Deliveries: 20
Sampling Rate: 100
State:
Last Delivered Message: Consumer sequence: 1 Stream sequence: 1
Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0
Pending Messages: 0
Redelivered Messages: 0
```
This is a pull-based Consumer (empty Delivery Target), it gets messages from the first available message and requires specific acknowledgement of each and every message.
It only received messages that originally entered the Stream on `ORDERS.received`. Remember the Stream subscribes to `ORDERS.*`, this lets us select a subset of messages from the Stream.
A Maximum Delivery limit of 20 is set, this means if the message is not acknowledged it will be retried but only up to this maximum total deliveries.
Again this can all be done in a single CLI call, lets make the `DISPATCH` Consumer:
```
$ nats con add ORDERS DISPATCH --filter ORDERS.processed --ack explicit --pull --deliver all --sample 100 --max-deliver 20
```
Additionally, one can store the configuration in a JSON file, the format of this is the same as `$ nats con info ORDERS DISPATCH -j | jq .config`:
```
$ nats con add ORDERS MONITOR --config monitor.json
```
#### Creating Push-Based Consumers
Our `MONITOR` Consumer is push-based, has no ack and will only get new messages and is not sampled:
```
$ nats con add
? Select a Stream ORDERS
? Consumer name MONITOR
? Delivery target monitor.ORDERS
? Start policy (all, last, 1h, msg sequence) last
? Acknowledgement policy none
? Replay policy instant
? Filter Stream by subject (blank for all)
? Maximum Allowed Deliveries -1
Information for Consumer ORDERS > MONITOR
Configuration:
Durable Name: MONITOR
Delivery Subject: monitor.ORDERS
Deliver All: false
Deliver Last: true
Ack Policy: none
Replay Policy: instant
State:
Last Delivered Message: Consumer sequence: 1 Stream sequence: 3
Acknowledgment floor: Consumer sequence: 0 Stream sequence: 2
Pending Messages: 0
Redelivered Messages: 0
```
Again you can do this with a single non interactive command:
```
$ nats con add ORDERS MONITOR --ack none --target monitor.ORDERS --deliver last --replay instant --filter ''
```
Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats con info ORDERS MONITOR -j | jq .config`:
```
$ nats con add ORDERS --config monitor.json
```
#### Listing
You can get a quick list of all the Consumers for a specific Stream:
```
$ nats con ls ORDERS
Consumers for Stream ORDERS:
DISPATCH
MONITOR
NEW
```
#### Querying
All details for an Consumer can be queried, lets first look at a pull-based Consumer:
```
$ nats con info ORDERS DISPATCH
Information for Consumer ORDERS > DISPATCH
Configuration:
Durable Name: DISPATCH
Pull Mode: true
Subject: ORDERS.processed
Deliver All: true
Deliver Last: false
Ack Policy: explicit
Ack Wait: 30s
Replay Policy: instant
Sampling Rate: 100
State:
Last Delivered Message: Consumer sequence: 1 Stream sequence: 1
Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0
Pending Messages: 0
Redelivered Messages: 0
```
More details about the `State` section will be shown later when discussing the ack models in depth.
#### Consuming Pull-Based Consumers
Pull-based Consumers require you to specifically ask for messages and ack them, typically you would do this with the client library `Request()` feature, but the `jsm` utility has a helper:
First we ensure we have a message:
```
$ nats pub ORDERS.processed "order 1"
$ nats pub ORDERS.processed "order 2"
$ nats pub ORDERS.processed "order 3"
```
We can now read them using `nats`:
```
$ nats con next ORDERS DISPATCH
--- received on ORDERS.processed
order 1
Acknowledged message
$ nats con next ORDERS DISPATCH
--- received on ORDERS.processed
order 2
Acknowledged message
```
You can prevent ACKs by supplying `--no-ack`.
To do this from code you'd send a `Request()` to `$JS.NEXT.ORDERS.DISPATCH`:
```
$ nats req '$JS.NEXT.ORDERS.DISPATCH' ''
Published [$JS.NEXT.ORDERS.DISPATCH] : ''
Received [ORDERS.processed] : 'order 3'
```
Here `nats req` cannot ack, but in your code you'd respond to the received message with a nil payload as an Ack to JetStream.
#### Consuming Push-Based Consumers
Push-based Consumers will publish messages to a subject and anyone who subscribes to the subject will get them, they support different Acknowledgement models covered later, but here on the `MONITOR` Consumer we have no Acknowledgement.
```
$ nats con info ORDERS MONITOR
...
Delivery Subject: monitor.ORDERS
...
```
The Consumer is publishing to that subject, so lets listen there:
```
$ nats sub monitor.ORDERS
Listening on [monitor.ORDERS]
[#3] Received on [ORDERS.processed]: 'order 3'
[#4] Received on [ORDERS.processed]: 'order 4'
```
Note the subject here of the received message is reported as `ORDERS.processed` this helps you distinguish what you're seeing in a Stream covering a wildcard, or multiple subject, subject space.
This Consumer needs no ack, so any new message into the ORDERS system will show up here in real time.

235
jetstream/administration/streams.md Normal file
View File

@@ -0,0 +1,235 @@
### Streams
The first step is to set up storage for our `ORDERS` related messages, these arrive on a wildcard of subjects all flowing into the same Stream and they are kept for 1 year.
#### Creating
```nohighlight
$ nats str add ORDERS
? Subjects to consume ORDERS.*
? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Message count limit -1
? Message size limit -1
? Maximum message age limit 1y
? Maximum individual message size [? for help] (-1) -1
Stream ORDERS was created
Information for Stream ORDERS
Configuration:
Subjects: ORDERS.*
Acknowledgements: true
Retention: File - Limits
Replicas: 1
Maximum Messages: -1
Maximum Bytes: -1
Maximum Age: 8760h0m0s
Maximum Message Size: -1
Maximum Consumers: -1
Statistics:
Messages: 0
Bytes: 0 B
FirstSeq: 0
LastSeq: 0
Active Consumers: 0
```
You can get prompted interactively for missing information as above, or do it all on one command. Pressing `?` in the CLI will help you map prompts to CLI options:
```
$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old
```
Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`:
```
$ nats str add ORDERS --config orders.json
```
#### Listing
We can confirm our Stream was created:
```nohighlight
$ nats str ls
Streams:
ORDERS
```
#### Querying
Information about the configuration of the Stream can be seen, and if you did not specify the Stream like below, it will prompt you based on all known ones:
```nohighlight
$ nats str info ORDERS
Information for Stream ORDERS
Configuration:
Subjects: ORDERS.*
No Acknowledgements: false
Retention: File - Limits
Replicas: 1
Maximum Messages: -1
Maximum Bytes: -1
Maximum Age: 8760h0m0s
Maximum Consumers: -1
Statistics:
Messages: 0
Bytes: 0 B
FirstSeq: 0
LastSeq: 0
Active Consumers: 0
```
Most commands that show data as above support `-j` to show the results as JSON:
```nohighlight
$ nats str info ORDERS -j
{
"config": {
"name": "ORDERS",
"subjects": [
"ORDERS.*"
],
"retention": "limits",
"max_consumers": -1,
"max_msgs": -1,
"max_bytes": -1,
"max_age": 31536000000000000,
"storage": "file",
"num_replicas": 1
},
"stats": {
"messages": 0,
"bytes": 0,
"first_seq": 0,
"last_seq": 0,
"consumer_count": 0
}
}
```
This is the general pattern for the entire `nats` utility as it relates to JetStream - prompting for needed information but every action can be run non-interactively making it usable as a cli api. All information output like seen above can be turned into JSON using `-j`.
#### Copying
A stream can be copied into another, which also allows the configuration of the new one to be adjusted via CLI flags:
```nohighlight
$ nats str cp ORDERS ARCHIVE --subjects "ORDERS_ARCVHIVE.*" --max-age 2y
Stream ORDERS was created
Information for Stream ARCHIVE
Configuration:
Subjects: ORDERS_ARCVHIVE.*
...
Maximum Age: 17520h0m0s
...
```
#### Editing
A stream configuration can be edited, which allows the configuration to be adjusted via CLI flags. Here I have a incorrectly created ORDERS stream that I fix:
```nohighlight
$ nats str info ORDERS -j | jq .config.subjects
[
"ORDERS.new"
]
$ nats str edit ORDERS --subjects "ORDERS.*"
Stream ORDERS was updated
Information for Stream ORDERS
Configuration:
Subjects: ORDERS.*
....
```
Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`:
```
$ nats str edit ORDERS --config orders.json
```
#### Publishing Into a Stream
Now let's add in some messages to our Stream. You can use `nats pub` to add messages, pass the `--wait` flag to see the publish ack being returned.
You can publish without waiting for acknowledgement:
```nohighlight
$ nats pub ORDERS.scratch hello
Published [sub1] : 'hello'
```
But if you want to be sure your messages got to JetStream and were persisted you can make a request:
```nohighlight
$ nats req ORDERS.scratch hello
13:45:03 Sending request on [ORDERS.scratch]
13:45:03 Received on [_INBOX.M8drJkd8O5otORAo0sMNkg.scHnSafY]: '+OK'
```
Keep checking the status of the Stream while doing this and you'll see it's stored messages increase.
```nohighlight
$ nats str info ORDERS
Information for Stream ORDERS
...
Statistics:
Messages: 3
Bytes: 147 B
FirstSeq: 1
LastSeq: 3
Active Consumers: 0
```
After putting some throw away data into the Stream, we can purge all the data out - while keeping the Stream active:
#### Deleting All Data
To delete all data in a stream use `purge`:
```nohighlight
$ nats str purge ORDERS -f
...
Statistics:
Messages: 0
Bytes: 0 B
FirstSeq: 1,000,001
LastSeq: 1,000,000
Active Consumers: 0
```
#### Deleting A Message
A single message can be securely removed from the stream:
```nohighlight
$ nats str rmm ORDERS 1 -f
```
#### Deleting Sets
Finally for demonstration purposes, you can also delete the whole Stream and recreate it so then we're ready for creating the Consumers:
```
$ nats str rm ORDERS -f
$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1
```