From 1fd9cab1a5cc8f8e54417352b16bf210681469d7 Mon Sep 17 00:00:00 2001 From: ainsley Date: Tue, 24 Nov 2020 15:12:44 -0600 Subject: [PATCH 01/84] adding JetStream Docs --- jetstream/administration/account.md | 247 +++++++++ jetstream/administration/administration.md | 30 ++ jetstream/administration/consumers.md | 219 ++++++++ jetstream/administration/streams.md | 235 +++++++++ jetstream/concepts/concepts.md | 19 + jetstream/concepts/configuration.md | 10 + jetstream/concepts/consumers.md | 33 ++ jetstream/concepts/streams.md | 31 ++ .../configuration_mgmt/configuration_mgmt.md | 105 ++++ .../configuration_mgmt/github_actions.md | 55 ++ .../kubernetes_controller.md | 57 +++ .../disaster_recovery/disaster_recovery.md | 85 ++++ jetstream/getting_started/getting_started.md | 3 + jetstream/getting_started/using_docker.md | 50 ++ jetstream/getting_started/using_source.md | 65 +++ jetstream/model_deep_dive/model_deep_dive.md | 481 ++++++++++++++++++ jetstream/monitoring/monitoring.md | 30 ++ .../multi-tenancy/resource_management.md | 133 +++++ .../nats_api_reference/nats_api_reference.md | 221 ++++++++ 19 files changed, 2109 insertions(+) create mode 100644 jetstream/administration/account.md create mode 100644 jetstream/administration/administration.md create mode 100644 jetstream/administration/consumers.md create mode 100644 jetstream/administration/streams.md create mode 100644 jetstream/concepts/concepts.md create mode 100644 jetstream/concepts/configuration.md create mode 100644 jetstream/concepts/consumers.md create mode 100644 jetstream/concepts/streams.md create mode 100644 jetstream/configuration_mgmt/configuration_mgmt.md create mode 100644 jetstream/configuration_mgmt/github_actions.md create mode 100644 jetstream/configuration_mgmt/kubernetes_controller.md create mode 100644 jetstream/disaster_recovery/disaster_recovery.md create mode 100644 jetstream/getting_started/getting_started.md create mode 100644 jetstream/getting_started/using_docker.md create mode 100644 jetstream/getting_started/using_source.md create mode 100644 jetstream/model_deep_dive/model_deep_dive.md create mode 100644 jetstream/monitoring/monitoring.md create mode 100644 jetstream/multi-tenancy/resource_management.md create mode 100644 jetstream/nats_api_reference/nats_api_reference.md diff --git a/jetstream/administration/account.md b/jetstream/administration/account.md new file mode 100644 index 0000000..33761ac --- /dev/null +++ b/jetstream/administration/account.md @@ -0,0 +1,247 @@ +### Account Information + +JetStream is multi-tenant so you will need to check that your account is enabled for JetStream and is not limited. You can view your limits as follows: + +```nohighlight +$ nats account info + + Memory: 0 B of 6.4 GB + Storage: 0 B of 1.1 TB + Streams: 1 of Unlimited +``` + +### Streams + +The first step is to set up storage for our `ORDERS` related messages, these arrive on a wildcard of subjects all flowing into the same Stream and they are kept for 1 year. + +#### Creating + +```nohighlight +$ nats str add ORDERS +? Subjects to consume ORDERS.* +? Storage backend file +? Retention Policy Limits +? Discard Policy Old +? Message count limit -1 +? Message size limit -1 +? Maximum message age limit 1y +? Maximum individual message size [? for help] (-1) -1 +Stream ORDERS was created + +Information for Stream ORDERS + +Configuration: + + Subjects: ORDERS.* + Acknowledgements: true + Retention: File - Limits + Replicas: 1 + Maximum Messages: -1 + Maximum Bytes: -1 + Maximum Age: 8760h0m0s + Maximum Message Size: -1 + Maximum Consumers: -1 + +Statistics: + + Messages: 0 + Bytes: 0 B + FirstSeq: 0 + LastSeq: 0 + Active Consumers: 0 +``` + +You can get prompted interactively for missing information as above, or do it all on one command. Pressing `?` in the CLI will help you map prompts to CLI options: + +``` +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old +``` + +Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: + +``` +$ nats str add ORDERS --config orders.json +``` + +#### Listing + +We can confirm our Stream was created: + +```nohighlight +$ nats str ls +Streams: + + ORDERS +``` + +#### Querying + +Information about the configuration of the Stream can be seen, and if you did not specify the Stream like below, it will prompt you based on all known ones: + +```nohighlight +$ nats str info ORDERS +Information for Stream ORDERS + +Configuration: + + Subjects: ORDERS.* + No Acknowledgements: false + Retention: File - Limits + Replicas: 1 + Maximum Messages: -1 + Maximum Bytes: -1 + Maximum Age: 8760h0m0s + Maximum Consumers: -1 + +Statistics: + + Messages: 0 + Bytes: 0 B + FirstSeq: 0 + LastSeq: 0 + Active Consumers: 0 +``` + +Most commands that show data as above support `-j` to show the results as JSON: + +```nohighlight +$ nats str info ORDERS -j +{ + "config": { + "name": "ORDERS", + "subjects": [ + "ORDERS.*" + ], + "retention": "limits", + "max_consumers": -1, + "max_msgs": -1, + "max_bytes": -1, + "max_age": 31536000000000000, + "storage": "file", + "num_replicas": 1 + }, + "stats": { + "messages": 0, + "bytes": 0, + "first_seq": 0, + "last_seq": 0, + "consumer_count": 0 + } +} +``` + +This is the general pattern for the entire `nats` utility as it relates to JetStream - prompting for needed information but every action can be run non-interactively making it usable as a cli api. All information output like seen above can be turned into JSON using `-j`. + +#### Copying + +A stream can be copied into another, which also allows the configuration of the new one to be adjusted via CLI flags: + +```nohighlight +$ nats str cp ORDERS ARCHIVE --subjects "ORDERS_ARCVHIVE.*" --max-age 2y +Stream ORDERS was created + +Information for Stream ARCHIVE + +Configuration: + + Subjects: ORDERS_ARCVHIVE.* +... + Maximum Age: 17520h0m0s +... +``` + +#### Editing + +A stream configuration can be edited, which allows the configuration to be adjusted via CLI flags. Here I have a incorrectly created ORDERS stream that I fix: + +```nohighlight +$ nats str info ORDERS -j | jq .config.subjects +[ + "ORDERS.new" +] + +$ nats str edit ORDERS --subjects "ORDERS.*" +Stream ORDERS was updated + +Information for Stream ORDERS + +Configuration: + + Subjects: ORDERS.* +.... +``` + +Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: + +``` +$ nats str edit ORDERS --config orders.json +``` + +#### Publishing Into a Stream + +Now let's add in some messages to our Stream. You can use `nats pub` to add messages, pass the `--wait` flag to see the publish ack being returned. + +You can publish without waiting for acknowledgement: + +```nohighlight +$ nats pub ORDERS.scratch hello +Published [sub1] : 'hello' +``` + +But if you want to be sure your messages got to JetStream and were persisted you can make a request: + +```nohighlight +$ nats req ORDERS.scratch hello +13:45:03 Sending request on [ORDERS.scratch] +13:45:03 Received on [_INBOX.M8drJkd8O5otORAo0sMNkg.scHnSafY]: '+OK' +``` + +Keep checking the status of the Stream while doing this and you'll see it's stored messages increase. + +```nohighlight +$ nats str info ORDERS +Information for Stream ORDERS +... +Statistics: + + Messages: 3 + Bytes: 147 B + FirstSeq: 1 + LastSeq: 3 + Active Consumers: 0 +``` + +After putting some throw away data into the Stream, we can purge all the data out - while keeping the Stream active: + +#### Deleting All Data + +To delete all data in a stream use `purge`: + +```nohighlight +$ nats str purge ORDERS -f +... +Statistics: + + Messages: 0 + Bytes: 0 B + FirstSeq: 1,000,001 + LastSeq: 1,000,000 + Active Consumers: 0 +``` + +#### Deleting A Message + +A single message can be securely removed from the stream: + +```nohighlight +$ nats str rmm ORDERS 1 -f +``` + +#### Deleting Sets + +Finally for demonstration purposes, you can also delete the whole Stream and recreate it so then we're ready for creating the Consumers: + +``` +$ nats str rm ORDERS -f +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 +``` diff --git a/jetstream/administration/administration.md b/jetstream/administration/administration.md new file mode 100644 index 0000000..8dcd564 --- /dev/null +++ b/jetstream/administration/administration.md @@ -0,0 +1,30 @@ +## Administration and Usage from the CLI + +Once the server is running it's time to use the management tool. This can be downloaded from the [GitHub Release Page](https://github.com/nats-io/jetstream/releases/) or you can use the `synadia/jsm:latest` docker image. + +``` +$ nats --help +usage: nats [] [ ...] +NATS Management Utility + +Flags: + --help Show context-sensitive help (also try --help-long and --help-man). + --version Show application version. + -s, --server="localhost:4222" NATS servers + --creds=CREDS User credentials + --tlscert=TLSCERT TLS public certifcate + --tlskey=TLSKEY TLS private key + --tlsca=TLSCA TLS certifcate authority chain + --timeout=2s Time to give JetStream to respond to queries + +Commands: + help [...] + Show help. + ... +``` + +We'll walk through the above scenario and introduce features of the CLI and of JetStream as we recreate the setup above. + +Throughout this example, we'll show other commands like `nats pub` and `nats sub` to interact with the system. These are normal existing core NATS commands and JetStream is fully usable by only using core NATS. + +We'll touch on some additional features but please review the section on the design model to understand all possible permutations. \ No newline at end of file diff --git a/jetstream/administration/consumers.md b/jetstream/administration/consumers.md new file mode 100644 index 0000000..e3e21ac --- /dev/null +++ b/jetstream/administration/consumers.md @@ -0,0 +1,219 @@ +### Consumers + +Consumers is how messages are read or consumed from the Stream. We support pull and push-based Consumers and the example scenario has both, lets walk through that. + +#### Creating Pull-Based Consumers + +The `NEW` and `DISPATCH` Consumers are pull-based, meaning the services consuming data from them have to ask the system for the next available message. This means you can easily scale your services up by adding more workers and the messages will get spread across the workers based on their availability. + +Pull-based Consumers are created the same as push-based Consumers, just don't specify a delivery target. + +``` +$ nats con ls ORDERS +No Consumers defined +``` + +We have no Consumers, lets add the `NEW` one: + +I supply the `--sample` options on the CLI as this is not prompted for at present, everything else is prompted. The help in the CLI explains each: + +``` +$ nats con add --sample 100 +? Select a Stream ORDERS +? Consumer name NEW +? Delivery target +? Start policy (all, last, 1h, msg sequence) all +? Filter Stream by subject (blank for all) ORDERS.received +? Maximum Allowed Deliveries 20 +Information for Consumer ORDERS > NEW + +Configuration: + + Durable Name: NEW + Pull Mode: true + Subject: ORDERS.received + Deliver All: true + Deliver Last: false + Ack Policy: explicit + Ack Wait: 30s + Replay Policy: instant + Maximum Deliveries: 20 + Sampling Rate: 100 + +State: + + Last Delivered Message: Consumer sequence: 1 Stream sequence: 1 + Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0 + Pending Messages: 0 + Redelivered Messages: 0 +``` + +This is a pull-based Consumer (empty Delivery Target), it gets messages from the first available message and requires specific acknowledgement of each and every message. + +It only received messages that originally entered the Stream on `ORDERS.received`. Remember the Stream subscribes to `ORDERS.*`, this lets us select a subset of messages from the Stream. + +A Maximum Delivery limit of 20 is set, this means if the message is not acknowledged it will be retried but only up to this maximum total deliveries. + +Again this can all be done in a single CLI call, lets make the `DISPATCH` Consumer: + +``` +$ nats con add ORDERS DISPATCH --filter ORDERS.processed --ack explicit --pull --deliver all --sample 100 --max-deliver 20 +``` + +Additionally, one can store the configuration in a JSON file, the format of this is the same as `$ nats con info ORDERS DISPATCH -j | jq .config`: + +``` +$ nats con add ORDERS MONITOR --config monitor.json +``` + +#### Creating Push-Based Consumers + +Our `MONITOR` Consumer is push-based, has no ack and will only get new messages and is not sampled: + +``` +$ nats con add +? Select a Stream ORDERS +? Consumer name MONITOR +? Delivery target monitor.ORDERS +? Start policy (all, last, 1h, msg sequence) last +? Acknowledgement policy none +? Replay policy instant +? Filter Stream by subject (blank for all) +? Maximum Allowed Deliveries -1 +Information for Consumer ORDERS > MONITOR + +Configuration: + + Durable Name: MONITOR + Delivery Subject: monitor.ORDERS + Deliver All: false + Deliver Last: true + Ack Policy: none + Replay Policy: instant + +State: + + Last Delivered Message: Consumer sequence: 1 Stream sequence: 3 + Acknowledgment floor: Consumer sequence: 0 Stream sequence: 2 + Pending Messages: 0 + Redelivered Messages: 0 +``` + +Again you can do this with a single non interactive command: + +``` +$ nats con add ORDERS MONITOR --ack none --target monitor.ORDERS --deliver last --replay instant --filter '' +``` + +Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats con info ORDERS MONITOR -j | jq .config`: + +``` +$ nats con add ORDERS --config monitor.json +``` + +#### Listing + +You can get a quick list of all the Consumers for a specific Stream: + +``` +$ nats con ls ORDERS +Consumers for Stream ORDERS: + + DISPATCH + MONITOR + NEW +``` + +#### Querying + +All details for an Consumer can be queried, lets first look at a pull-based Consumer: + +``` +$ nats con info ORDERS DISPATCH +Information for Consumer ORDERS > DISPATCH + +Configuration: + + Durable Name: DISPATCH + Pull Mode: true + Subject: ORDERS.processed + Deliver All: true + Deliver Last: false + Ack Policy: explicit + Ack Wait: 30s + Replay Policy: instant + Sampling Rate: 100 + +State: + + Last Delivered Message: Consumer sequence: 1 Stream sequence: 1 + Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0 + Pending Messages: 0 + Redelivered Messages: 0 +``` + +More details about the `State` section will be shown later when discussing the ack models in depth. + +#### Consuming Pull-Based Consumers + +Pull-based Consumers require you to specifically ask for messages and ack them, typically you would do this with the client library `Request()` feature, but the `jsm` utility has a helper: + +First we ensure we have a message: + +``` +$ nats pub ORDERS.processed "order 1" +$ nats pub ORDERS.processed "order 2" +$ nats pub ORDERS.processed "order 3" +``` + +We can now read them using `nats`: + +``` +$ nats con next ORDERS DISPATCH +--- received on ORDERS.processed +order 1 + +Acknowledged message + +$ nats con next ORDERS DISPATCH +--- received on ORDERS.processed +order 2 + +Acknowledged message +``` + +You can prevent ACKs by supplying `--no-ack`. + +To do this from code you'd send a `Request()` to `$JS.NEXT.ORDERS.DISPATCH`: + +``` +$ nats req '$JS.NEXT.ORDERS.DISPATCH' '' +Published [$JS.NEXT.ORDERS.DISPATCH] : '' +Received [ORDERS.processed] : 'order 3' +``` + +Here `nats req` cannot ack, but in your code you'd respond to the received message with a nil payload as an Ack to JetStream. + +#### Consuming Push-Based Consumers + +Push-based Consumers will publish messages to a subject and anyone who subscribes to the subject will get them, they support different Acknowledgement models covered later, but here on the `MONITOR` Consumer we have no Acknowledgement. + +``` +$ nats con info ORDERS MONITOR +... + Delivery Subject: monitor.ORDERS +... +``` + +The Consumer is publishing to that subject, so lets listen there: + +``` +$ nats sub monitor.ORDERS +Listening on [monitor.ORDERS] +[#3] Received on [ORDERS.processed]: 'order 3' +[#4] Received on [ORDERS.processed]: 'order 4' +``` + +Note the subject here of the received message is reported as `ORDERS.processed` this helps you distinguish what you're seeing in a Stream covering a wildcard, or multiple subject, subject space. + +This Consumer needs no ack, so any new message into the ORDERS system will show up here in real time. \ No newline at end of file diff --git a/jetstream/administration/streams.md b/jetstream/administration/streams.md new file mode 100644 index 0000000..72e8689 --- /dev/null +++ b/jetstream/administration/streams.md @@ -0,0 +1,235 @@ +### Streams + +The first step is to set up storage for our `ORDERS` related messages, these arrive on a wildcard of subjects all flowing into the same Stream and they are kept for 1 year. + +#### Creating + +```nohighlight +$ nats str add ORDERS +? Subjects to consume ORDERS.* +? Storage backend file +? Retention Policy Limits +? Discard Policy Old +? Message count limit -1 +? Message size limit -1 +? Maximum message age limit 1y +? Maximum individual message size [? for help] (-1) -1 +Stream ORDERS was created + +Information for Stream ORDERS + +Configuration: + + Subjects: ORDERS.* + Acknowledgements: true + Retention: File - Limits + Replicas: 1 + Maximum Messages: -1 + Maximum Bytes: -1 + Maximum Age: 8760h0m0s + Maximum Message Size: -1 + Maximum Consumers: -1 + +Statistics: + + Messages: 0 + Bytes: 0 B + FirstSeq: 0 + LastSeq: 0 + Active Consumers: 0 +``` + +You can get prompted interactively for missing information as above, or do it all on one command. Pressing `?` in the CLI will help you map prompts to CLI options: + +``` +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old +``` + +Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: + +``` +$ nats str add ORDERS --config orders.json +``` + +#### Listing + +We can confirm our Stream was created: + +```nohighlight +$ nats str ls +Streams: + + ORDERS +``` + +#### Querying + +Information about the configuration of the Stream can be seen, and if you did not specify the Stream like below, it will prompt you based on all known ones: + +```nohighlight +$ nats str info ORDERS +Information for Stream ORDERS + +Configuration: + + Subjects: ORDERS.* + No Acknowledgements: false + Retention: File - Limits + Replicas: 1 + Maximum Messages: -1 + Maximum Bytes: -1 + Maximum Age: 8760h0m0s + Maximum Consumers: -1 + +Statistics: + + Messages: 0 + Bytes: 0 B + FirstSeq: 0 + LastSeq: 0 + Active Consumers: 0 +``` + +Most commands that show data as above support `-j` to show the results as JSON: + +```nohighlight +$ nats str info ORDERS -j +{ + "config": { + "name": "ORDERS", + "subjects": [ + "ORDERS.*" + ], + "retention": "limits", + "max_consumers": -1, + "max_msgs": -1, + "max_bytes": -1, + "max_age": 31536000000000000, + "storage": "file", + "num_replicas": 1 + }, + "stats": { + "messages": 0, + "bytes": 0, + "first_seq": 0, + "last_seq": 0, + "consumer_count": 0 + } +} +``` + +This is the general pattern for the entire `nats` utility as it relates to JetStream - prompting for needed information but every action can be run non-interactively making it usable as a cli api. All information output like seen above can be turned into JSON using `-j`. + +#### Copying + +A stream can be copied into another, which also allows the configuration of the new one to be adjusted via CLI flags: + +```nohighlight +$ nats str cp ORDERS ARCHIVE --subjects "ORDERS_ARCVHIVE.*" --max-age 2y +Stream ORDERS was created + +Information for Stream ARCHIVE + +Configuration: + + Subjects: ORDERS_ARCVHIVE.* +... + Maximum Age: 17520h0m0s +... +``` + +#### Editing + +A stream configuration can be edited, which allows the configuration to be adjusted via CLI flags. Here I have a incorrectly created ORDERS stream that I fix: + +```nohighlight +$ nats str info ORDERS -j | jq .config.subjects +[ + "ORDERS.new" +] + +$ nats str edit ORDERS --subjects "ORDERS.*" +Stream ORDERS was updated + +Information for Stream ORDERS + +Configuration: + + Subjects: ORDERS.* +.... +``` + +Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: + +``` +$ nats str edit ORDERS --config orders.json +``` + +#### Publishing Into a Stream + +Now let's add in some messages to our Stream. You can use `nats pub` to add messages, pass the `--wait` flag to see the publish ack being returned. + +You can publish without waiting for acknowledgement: + +```nohighlight +$ nats pub ORDERS.scratch hello +Published [sub1] : 'hello' +``` + +But if you want to be sure your messages got to JetStream and were persisted you can make a request: + +```nohighlight +$ nats req ORDERS.scratch hello +13:45:03 Sending request on [ORDERS.scratch] +13:45:03 Received on [_INBOX.M8drJkd8O5otORAo0sMNkg.scHnSafY]: '+OK' +``` + +Keep checking the status of the Stream while doing this and you'll see it's stored messages increase. + +```nohighlight +$ nats str info ORDERS +Information for Stream ORDERS +... +Statistics: + + Messages: 3 + Bytes: 147 B + FirstSeq: 1 + LastSeq: 3 + Active Consumers: 0 +``` + +After putting some throw away data into the Stream, we can purge all the data out - while keeping the Stream active: + +#### Deleting All Data + +To delete all data in a stream use `purge`: + +```nohighlight +$ nats str purge ORDERS -f +... +Statistics: + + Messages: 0 + Bytes: 0 B + FirstSeq: 1,000,001 + LastSeq: 1,000,000 + Active Consumers: 0 +``` + +#### Deleting A Message + +A single message can be securely removed from the stream: + +```nohighlight +$ nats str rmm ORDERS 1 -f +``` + +#### Deleting Sets + +Finally for demonstration purposes, you can also delete the whole Stream and recreate it so then we're ready for creating the Consumers: + +``` +$ nats str rm ORDERS -f +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 +``` diff --git a/jetstream/concepts/concepts.md b/jetstream/concepts/concepts.md new file mode 100644 index 0000000..00785d4 --- /dev/null +++ b/jetstream/concepts/concepts.md @@ -0,0 +1,19 @@ +## Concepts + +In JetStream the configuration for storing messages is defined separately from how they are consumed. Storage is defined in a *Stream* and consuming messages is defined by multiple *Consumers*. + +We'll discuss these 2 subjects in the context of this architecture. + +![Orders](../.gitbook/assets/images/streams-and-consumers-75p.png) + +While this is an incomplete architecture it does show a number of key points: + + * Many related subjects are stored in a Stream + * Consumers can have different modes of operation and receive just subsets of the messages + * Multiple Acknowledgement modes are supported + +A new order arrives on `ORDERS.received`, gets sent to the `NEW` Consumer who, on success, will create a new message on `ORDERS.processed`. The `ORDERS.processed` message again enters the Stream where a `DISPATCH` Consumer receives it and once processed it will create an `ORDERS.completed` message which will again enter the Stream. These operations are all `pull` based meaning they are work queues and can scale horizontally. All require acknowledged delivery ensuring no order is missed. + +All messages are delivered to a `MONITOR` Consumer without any acknowledgement and using Pub/Sub semantics - they are pushed to the monitor. + +As messages are acknowledged to the `NEW` and `DISPATCH` Consumers, a percentage of them are Sampled and messages indicating redelivery counts, ack delays and more, are delivered to the monitoring system. \ No newline at end of file diff --git a/jetstream/concepts/configuration.md b/jetstream/concepts/configuration.md new file mode 100644 index 0000000..aa4824c --- /dev/null +++ b/jetstream/concepts/configuration.md @@ -0,0 +1,10 @@ +### Configuration + +The rest of this document introduces the `nats` utility, but for completeness and reference this is how you'd create the ORDERS scenario. We'll configure a 1 year retention for order related messages: + +```bash +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard=old +$ nats con add ORDERS NEW --filter ORDERS.received --ack explicit --pull --deliver all --max-deliver=-1 --sample 100 +$ nats con add ORDERS DISPATCH --filter ORDERS.processed --ack explicit --pull --deliver all --max-deliver=-1 --sample 100 +$ nats con add ORDERS MONITOR --filter '' --ack none --target monitor.ORDERS --deliver last --replay instant +``` \ No newline at end of file diff --git a/jetstream/concepts/consumers.md b/jetstream/concepts/consumers.md new file mode 100644 index 0000000..7ce7fde --- /dev/null +++ b/jetstream/concepts/consumers.md @@ -0,0 +1,33 @@ +### Consumers + +Each Consumer, or related group of Consumers, of a Stream will need an Consumer defined. It's ok to define thousands of these pointing at the same Stream. + +Consumers can either be `push` based where JetStream will deliver the messages as fast as possible to a subject of your choice or `pull` based for typical work queue like behavior. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging. + +In the orders example above we have 3 Consumers. The first two select a subset of the messages from the Stream by specifying a specific subject like `ORDERS.processed`. The Stream consumes `ORDERS.*` and this allows you to receive just what you need. The final Consumer receives all messages in a `push` fashion. + +Consumers track their progress, they know what messages were delivered, acknowledged, etc., and will redeliver messages they sent that were not acknowledged. When first created, the Consumer has to know what message to send as the first one. You can configure either a specific message in the set (`StreamSeq`), specific time (`StartTime`), all (`DeliverAll`) or last (`DeliverLast`). This is the starting point and from there, they all behave the same - delivering all of the following messages with optional Acknowledgement. + +Acknowledgements default to `AckExplicit` - the only supported mode for pull-based Consumers - meaning every message requires a distinct acknowledgement. But for push-based Consumers, you can set `AckNone` that does not require any acknowledgement, or `AckAll` which quite interestingly allows you to acknowledge a specific message, like message `100`, which will also acknowledge messages `1` through `99`. The `AckAll` mode can be a great performance boost. + +Some messages may cause your applications to crash and cause a never ending loop forever poisoning your system. The `MaxDeliver` setting allow you to set a upper bound to how many times a message may be delivered. + +To assist with creating monitoring applications, one can set a `SampleFrequency` which is a percentage of messages for which the system should sample and create events. These events will include delivery counts and ack waits. + +When defining Consumers the items below make up the entire configuration of the Consumer: + +|Item|Description| +|----|-----------| +|AckPolicy|How messages should be acknowledged, `AckNone`, `AckAll` or `AckExplicit`| +|AckWait|How long to allow messages to remain un-acknowledged before attempting redelivery| +|DeliverPolicy|The initial starting mode of the consumer, `DeliverAll`, `DeliverLast`, `DeliverNew`, `DeliverByStartSequence` or `DeliverByStartTime`| +|DeliverySubject|The subject to deliver observed messages, when not set, a pull-based Consumer is created| +|Durable|The name of the Consumer| +|FilterSubject|When consuming from a Stream with many subjects, or wildcards, select only a specific incoming subjects, supports wildcards| +|MaxDeliver|Maximum amount times a specific message will be delivered. Use this to avoid poison pills crashing all your services forever| +|OptStartSeq|When first consuming messages from the Stream start at this particular message in the set| +|ReplayPolicy|How messages are sent `ReplayInstant` or `ReplayOriginal`| +|SampleFrequency|What percentage of acknowledgements should be samples for observability, 0-100| +|OptStartTime|When first consuming messages from the Stream start with messages on or after this time| +|RateLimit|The rate of message delivery in bits per second| +|MaxAckPending|The maximum number of messages without acknowledgement that can be outstanding, once this limit is reached message delivery will be suspended| diff --git a/jetstream/concepts/streams.md b/jetstream/concepts/streams.md new file mode 100644 index 0000000..861ec64 --- /dev/null +++ b/jetstream/concepts/streams.md @@ -0,0 +1,31 @@ +### Streams + +Streams define how messages are stored and retention duration. Streams consume normal NATS subjects, any message found on those subjects will be delivered to the defined storage system. You can do a normal publish to the subject for unacknowledged delivery, else if you send a Request to the subject the JetStream server will reply with an acknowledgement that it was stored. + +As of January 2020, in the tech preview we have `file` and `memory` based storage systems, we do not yet support clustering. + +In the diagram above we show the concept of storing all `ORDERS.*` in the Stream even though there are many types of order related messages. We'll show how you can selectively consume subsets of messages later. Relatively speaking the Stream is the most resource consuming component so being able to combine related data in this manner is important to consider. + +Streams can consume many subjects. Here we have `ORDERS.*` but we could also consume `SHIPPING.state` into the same Stream should that make sense (not shown here). + +Streams support various retention policies - they can be kept based on limits like max count, size or age but also more novel methods like keeping them as long as any Consumers have them unacknowledged, or work queue like behavior where a message is removed after first ack. + +Streams support deduplication using a `Msg-Id` header and a sliding window within which to track duplicate messages. See the [Message Deduplication](#message-deduplication) section. + +When defining Streams the items below make up the entire configuration of the set. + +|Item|Description| +|----|-----------| +|MaxAge|Maximum age of any message in the stream, expressed in microseconds| +|MaxBytes|How big the Stream may be, when the combined stream size exceeds this old messages are removed| +|MaxMsgSize|The largest message that will be accepted by the Stream| +|MaxMsgs|How many messages may be in a Stream, oldest messages will be removed if the Stream exceeds this size| +|MaxConsumers|How many Consumers can be defined for a given Stream, `-1` for unlimited| +|Name|A name for the Stream that may not have spaces, tabs or `.`| +|NoAck|Disables acknowledging messages that are received by the Stream| +|Replicas|How many replicas to keep for each message (not implemented as of January 2020)| +|Retention|How message retention is considered, `LimitsPolicy` (default), `InterestPolicy` or `WorkQueuePolicy`| +|Discard|When a Stream reached it's limits either, `DiscardNew` refuses new messages while `DiscardOld` (default) deletes old messages| +|Storage|The type of storage backend, `file` and `memory` as of January 2020| +|Subjects|A list of subjects to consume, supports wildcards| +|Duplicates|The window within which to track duplicate messages| diff --git a/jetstream/configuration_mgmt/configuration_mgmt.md b/jetstream/configuration_mgmt/configuration_mgmt.md new file mode 100644 index 0000000..f654c60 --- /dev/null +++ b/jetstream/configuration_mgmt/configuration_mgmt.md @@ -0,0 +1,105 @@ +## Configuration Management + +In many cases managing the configuration in your application code is the best model, many teams though wish to pre-create Streams and Consumers. + +We support a number of tools to assist with this: + + * `nats` CLI with configuration files + * [Terraform](https://www.terraform.io/) + * [GitHub Actions](https://github.com/features/actions) + * [Kubernetes JetStream Controller](https://github.com/nats-io/nack#jetstream-controller) + +### nats Admin CLI + +The `nats` CLI can be used to manage Streams and Consumers easily using it's `--config` flag, for example: + +### Add a new Stream + +This creates a new Stream based on `orders.json`. The `orders.json` file can be extracted from an existing stream using `nats stream info ORDERS -j | jq .config` + +``` +$ nats str add ORDERS --config orders.json +``` + +### Edit an existing Stream + +This edits an existing stream ensuring it complies with the configuration in `orders.json` +``` +$ nats str edit ORDERS --config orders.json +``` + +### Add a New Consumer + +This creates a new Consumer based on `orders_new.json`. The `orders_new.json` file can be extracted from an existing stream using `nats con info ORDERS NEW -j | jq .config` + +``` +$ nats con add ORDERS NEW --config orders_new.json +``` + +### Terraform + +Terraform is a Cloud configuration tool from Hashicorp found at [terraform.io](https://www.terraform.io/), we maintain a Provider for Terraform called [terraform-provider-jetstream](https://github.com/nats-io/terraform-provider-jetstream/) that can maintain JetStream using Terraform. + +#### Setup + +Our provider is not hosted by Hashicorp so installation is a bit more complex than typical. Browse to the [Release Page](https://github.com/nats-io/terraform-provider-jetstream/releases) and download the release for your platform and extract it into your Terraform plugins directory. + +``` +$ unzip -l terraform-provider-jetstream_0.0.2_darwin_amd64.zip +Archive: terraform-provider-jetstream_0.0.2_darwin_amd64.zip + Length Date Time Name +--------- ---------- ----- ---- + 11357 03-09-2020 10:48 LICENSE + 1830 03-09-2020 12:53 README.md + 24574336 03-09-2020 12:54 terraform-provider-jetstream_v0.0.2 +``` + +Place the `terraform-provider-jetstream_v0.0.2` file in `~/.terraform.d/plugins/terraform-provider-jetstream_v0.0.2` + +In your project you can configure the Provider like this: + +```terraform +provider "jetstream" { + servers = "connect.ngs.global" + credentials = "ngs_jetstream_admin.creds" +} +``` + +And start using it, here's an example that create the `ORDERS` example. Review the [Project README](https://github.com/nats-io/terraform-provider-jetstream#readme) for full details. + +```terraform +resource "jetstream_stream" "ORDERS" { + name = "ORDERS" + subjects = ["ORDERS.*"] + storage = "file" + max_age = 60 * 60 * 24 * 365 +} + +resource "jetstream_consumer" "ORDERS_NEW" { + stream_id = jetstream_stream.ORDERS.id + durable_name = "NEW" + deliver_all = true + filter_subject = "ORDERS.received" + sample_freq = 100 +} + +resource "jetstream_consumer" "ORDERS_DISPATCH" { + stream_id = jetstream_stream.ORDERS.id + durable_name = "DISPATCH" + deliver_all = true + filter_subject = "ORDERS.processed" + sample_freq = 100 +} + +resource "jetstream_consumer" "ORDERS_MONITOR" { + stream_id = jetstream_stream.ORDERS.id + durable_name = "MONITOR" + deliver_last = true + ack_policy = "none" + delivery_subject = "monitor.ORDERS" +} + +output "ORDERS_SUBJECTS" { + value = jetstream_stream.ORDERS.subjects +} +``` diff --git a/jetstream/configuration_mgmt/github_actions.md b/jetstream/configuration_mgmt/github_actions.md new file mode 100644 index 0000000..81975bd --- /dev/null +++ b/jetstream/configuration_mgmt/github_actions.md @@ -0,0 +1,55 @@ +### GitHub Actions + +We have a pack of GitHub Actions that let you manage an already running JetStream Server, useful for managing releases or standing up test infrastructure. + +Full details and examples are in the [jetstream-gh-actions](https://github.com/nats-io/jetstream-gh-action) repository, here's an example. + +```yaml +on: push +name: orders +jobs: + + # First we delete the ORDERS stream and consumer if they already exist + clean_orders: + runs-on: ubuntu-latest + steps: + - name: orders_stream + uses: nats-io/jetstream-gh-action/delete/stream@master + with: + missing_ok: 1 + stream: ORDERS + server: js.example.net + + # Now we create the Stream and Consumers using the same configuration files the + # nats CLI utility would use as shown above + create_orders: + runs-on: ubuntu-latest + needs: clean_orders + steps: + - uses: actions/checkout@master + - name: orders_stream + uses: nats-io/jetstream-gh-action/create/stream@master + with: + config: ORDERS.json + server: js.example.net + - name: orders_new_consumer + uses: nats-io/jetstream-gh-action/create/consumer@master + with: + config: ORDERS_NEW.json + stream: ORDERS + server: js.example.net + + # We publish a message to a specific Subject, perhaps some consumer is + # waiting there for it to kick off tests + publish_message: + runs-on: ubuntu-latest + needs: create_orders + steps: + - uses: actions/checkout@master + - name: orders_new_consumer + uses: nats-io/jetstream-gh-action@master + with: + subject: ORDERS.deployment + message: Published new deployment via "${{ github.event_name }}" in "${{ github.repository }}" + server: js.example.net +``` diff --git a/jetstream/configuration_mgmt/kubernetes_controller.md b/jetstream/configuration_mgmt/kubernetes_controller.md new file mode 100644 index 0000000..9501fe6 --- /dev/null +++ b/jetstream/configuration_mgmt/kubernetes_controller.md @@ -0,0 +1,57 @@ +### Kubernetes JetStream Controller + +The JetStream controllers allows you to manage NATS JetStream Streams and Consumers via K8S CRDs. You can find more info on how to deploy and usage [here](https://github.com/nats-io/nack#getting-started). Below you can find an example on how to create a stream and a couple of consumers: + +```yaml +--- +apiVersion: jetstream.nats.io/v1beta1 +kind: Stream +metadata: + name: mystream +spec: + name: mystream + subjects: ["orders.*"] + storage: memory + maxAge: 1h +--- +apiVersion: jetstream.nats.io/v1beta1 +kind: Consumer +metadata: + name: my-push-consumer +spec: + streamName: mystream + durableName: my-push-consumer + deliverSubject: my-push-consumer.orders + deliverPolicy: last + ackPolicy: none + replayPolicy: instant +--- +apiVersion: jetstream.nats.io/v1beta1 +kind: Consumer +metadata: + name: my-pull-consumer +spec: + streamName: mystream + durableName: my-pull-consumer + deliverPolicy: all + filterSubject: orders.received + maxDeliver: 20 + ackPolicy: explicit +``` + +Once the CRDs are installed you can use `kubectl` to manage the streams and consumers as follows: + +```sh +$ kubectl get streams +NAME STATE STREAM NAME SUBJECTS +mystream Created mystream [orders.*] + +$ kubectl get consumers +NAME STATE STREAM CONSUMER ACK POLICY +my-pull-consumer Created mystream my-pull-consumer explicit +my-push-consumer Created mystream my-push-consumer none + +# If you end up in an Errored state, run kubectl describe for more info. +# kubectl describe streams mystream +# kubectl describe consumers my-pull-consumer +``` diff --git a/jetstream/disaster_recovery/disaster_recovery.md b/jetstream/disaster_recovery/disaster_recovery.md new file mode 100644 index 0000000..92b851b --- /dev/null +++ b/jetstream/disaster_recovery/disaster_recovery.md @@ -0,0 +1,85 @@ +## Disaster Recovery + +Disaster Recovery of the JetStream system is a topic we are still exploring and fleshing out and that will be impacted by the clustering work. For example replication will extend the options available to you. + +Today we have a few approaches to consider: + + * `nats` CLI + Configuration Backups + Data Snapshots + * Configuration Management + Data Snapshots + +### Data Backup + +In all scenarios you can perform data snapshots and restores over the NATS protocol. This is good if you do not manage the NATS servers hosting your data, and you wish to do a backup of your data. + +The backup includes: + + * Stream configuration and state + * Stream Consumer configuration and state + * All data including metadata like timestamps and headers + +```nohighlight +$ nats stream backup ORDERS /data/js-backup/ORDERS.tgz +Starting backup of Stream "ORDERS" with 13 data blocks + +2.4 MiB/s [====================================================================] 100% + +Received 13 MiB bytes of compressed data in 3368 chunks for stream "ORDERS" in 1.223428188s, 813 MiB uncompressed +``` + +During the backup the Stream is in a state where it's configuration cannot change and no data will be expired from it based on Limits or Retention Policies. + +Progress using the terminal bar can be disabled using `--no-progress`, it will then issue log lines instead. + +### Restoring Data + +The backup made above can be restored into another server - but into the same Stream name. + +```nohighlight +$ nats str restore ORDERS /data/js-backup/ORDERS.tgz +Starting restore of Stream "ORDERS" from file "/data/js-backup/ORDERS.tgz" + +13 MiB/s [====================================================================] 100% + +Restored stream "ORDERS" in 937.071149ms + +Information for Stream ORDERS + +Configuration: + + Subjects: ORDERS.> +... +``` + +The `/data/js-backup/ORDERS.tgz` file can also be extracted into the data dir of a stopped NATS Server. + +Progress using the terminal bar can be disabled using `--no-progress`, it will then issue log lines instead. + +### Interactive CLI + +In environments where the `nats` CLI is used interactively to configure the server you do not have a desired state to recreate the server from. This is not the ideal way to administer the server, we recommend Configuration Management, but many will use this approach. + +Here you can back up the configuration into a directory from where you can recover the configuration later. The data for File backed stores can also be backed up. + +```nohighlight +$ nats backup /data/js-backup +15:56:11 Creating JetStream backup into /data/js-backup +15:56:11 Stream ORDERS to /data/js-backup/stream_ORDERS.json +15:56:11 Consumer ORDERS > NEW to /data/js-backup/stream_ORDERS_consumer_NEW.json +15:56:11 Configuration backup complete +``` + +This backs up Stream, Consumer and Stream Template configuration. + +During the same process the data can also be backed up by passing `--data`, this will create files like `/data/js-backup/stream_ORDERS.tgz`. + +Later the data can be restored, for Streams we support editing the Stream configuration in place to match what was in the backup. + +``` +$ nats restore /tmp/backup --update-streams +15:57:42 Reading file /tmp/backup/stream_ORDERS.json +15:57:42 Reading file /tmp/backup/stream_ORDERS_consumer_NEW.json +15:57:42 Updating Stream ORDERS configuration +15:57:42 Restoring Consumer ORDERS > NEW +``` + +The `nats restore` tool does not support restoring data, the same process using `nats stream restore`, as outlined earlier, can be used which will also restore Stream and Consumer configurations and state. \ No newline at end of file diff --git a/jetstream/getting_started/getting_started.md b/jetstream/getting_started/getting_started.md new file mode 100644 index 0000000..0f95734 --- /dev/null +++ b/jetstream/getting_started/getting_started.md @@ -0,0 +1,3 @@ +## Getting Started + +This tech preview is limited to a single server and defaults to the global account. JetStream is NATS 2.0 aware and is scoped to accounts from a resource limit perspective. This is not the same as an individual server's resources, but may feel that way starting out. Don't worry, clustering is coming next but we wanted to get input early from the community. \ No newline at end of file diff --git a/jetstream/getting_started/using_docker.md b/jetstream/getting_started/using_docker.md new file mode 100644 index 0000000..d8923ea --- /dev/null +++ b/jetstream/getting_started/using_docker.md @@ -0,0 +1,50 @@ +### Using Docker + +The `synadia/jsm:latest` docker image contains both the JetStream enabled NATS Server and the `nats` utility this guide covers. + +In one window start JetStream: + +``` +$ docker run -ti -p 4222:4222 --name jetstream synadia/jsm:latest server +[1] 2020/01/20 12:44:11.752465 [INF] Starting nats-server version 2.2.0-beta +[1] 2020/01/20 12:44:11.752694 [INF] Git commit [19dc3eb] +[1] 2020/01/20 12:44:11.752875 [INF] Starting JetStream +[1] 2020/01/20 12:44:11.753692 [INF] ----------- JETSTREAM (Beta) ----------- +[1] 2020/01/20 12:44:11.753794 [INF] Max Memory: 1.46 GB +[1] 2020/01/20 12:44:11.753822 [INF] Max Storage: 1.00 TB +[1] 2020/01/20 12:44:11.753860 [INF] Store Directory: "/tmp/jetstream" +[1] 2020/01/20 12:44:11.753893 [INF] ---------------------------------------- +[1] 2020/01/20 12:44:11.753988 [INF] JetStream state for account "$G" recovered +[1] 2020/01/20 12:44:11.754148 [INF] Listening for client connections on 0.0.0.0:4222 +[1] 2020/01/20 12:44:11.754279 [INF] Server id is NDYX5IMGF2YLX6RC4WLZA7T3JGHPZR2RNCCIFUQBT6C4TP27Z6ZIC73V +[1] 2020/01/20 12:44:11.754308 [INF] Server is ready +``` + +And in another log into the utilities: + +``` +$ docker run -ti --link jetstream synadia/jsm:latest +``` + +This shell has the `nats` utility and all other NATS cli tools used in the rest of this guide. + +Now skip to the `Administer JetStream` section. + +### Using Docker with NGS + +You can join a JetStream instance to your [NGS](https://synadia.com/ngs/pricing) account, first we need a credential for testing JetStream: + +``` +$ nsc add user -a YourAccount --name leafnode --expiry 1M +``` + +You'll get a credential file somewhere like `~/.nkeys/creds/synadia/YourAccount/leafnode.creds`, mount this file into the docker container for JetStream using `-v ~/.nkeys/creds/synadia/YourAccount/leafnode.creds:/leafnode.creds`. + +``` +$ docker run -ti -v ~/.nkeys/creds/synadia/YourAccount/leafnode.creds:/leafnode.creds --name jetstream synadia/jsm:latest server +[1] 2020/01/20 12:44:11.752465 [INF] Starting nats-server version 2.2.0-beta +... +[1] 2020/01/20 12:55:01.849033 [INF] Connected leafnode to "connect.ngs.global" +``` + +Your JSM shell will still connect locally, other connections in your NGS account can use JetStream at this point. \ No newline at end of file diff --git a/jetstream/getting_started/using_source.md b/jetstream/getting_started/using_source.md new file mode 100644 index 0000000..4f6bee2 --- /dev/null +++ b/jetstream/getting_started/using_source.md @@ -0,0 +1,65 @@ +### Using Source + +You will also want to have installed from the nats.go repo the examples/tools such as nats-pub, nats-sub, nats-req and possibly nats-bench. One of the design goals of JetStream was to be native to core NATS, so even though we will most certainly add in syntactic sugar to clients to make them more appealing, for this tech preview we will be using plain old NATS. + +You will need a copy of the nats-server source locally and will need to be in the jetstream branch. + +``` +$ git clone https://github.com/nats-io/nats-server.git +$ cd nats-server +$ git checkout master +$ go build +$ ls -l nats-server +``` + +Starting the server you can use the `-js` flag. This will setup the server to reasonably use memory and disk. This is a sample run on my machine. JetStream will default to 1TB of disk and 75% of available memory for now. + +``` +$ ./nats-server -js + +[16928] 2019/12/04 19:16:29.596968 [INF] Starting nats-server version 2.2.0-beta +[16928] 2019/12/04 19:16:29.597056 [INF] Git commit [not set] +[16928] 2019/12/04 19:16:29.597072 [INF] Starting JetStream +[16928] 2019/12/04 19:16:29.597444 [INF] ----------- JETSTREAM (Beta) ----------- +[16928] 2019/12/04 19:16:29.597451 [INF] Max Memory: 96.00 GB +[16928] 2019/12/04 19:16:29.597454 [INF] Max Storage: 1.00 TB +[16928] 2019/12/04 19:16:29.597461 [INF] Store Directory: "/var/folders/m0/k03vs55n2b54kdg7jm66g27h0000gn/T/jetstream" +[16928] 2019/12/04 19:16:29.597469 [INF] ---------------------------------------- +[16928] 2019/12/04 19:16:29.597732 [INF] Listening for client connections on 0.0.0.0:4222 +[16928] 2019/12/04 19:16:29.597738 [INF] Server id is NAJ5GKP5OBVISP5MW3BFAD447LMTIOAHFEWMH2XYWLL5STVGN3MJHTXQ +[16928] 2019/12/04 19:16:29.597742 [INF] Server is ready +``` + +You can override the storage directory if you want. + +``` +$ ./nats-server -js -sd /tmp/test + +[16943] 2019/12/04 19:20:00.874148 [INF] Starting nats-server version 2.2.0-beta +[16943] 2019/12/04 19:20:00.874247 [INF] Git commit [not set] +[16943] 2019/12/04 19:20:00.874273 [INF] Starting JetStream +[16943] 2019/12/04 19:20:00.874605 [INF] ----------- JETSTREAM (Beta) ----------- +[16943] 2019/12/04 19:20:00.874613 [INF] Max Memory: 96.00 GB +[16943] 2019/12/04 19:20:00.874615 [INF] Max Storage: 1.00 TB +[16943] 2019/12/04 19:20:00.874620 [INF] Store Directory: "/tmp/test/jetstream" +[16943] 2019/12/04 19:20:00.874625 [INF] ---------------------------------------- +[16943] 2019/12/04 19:20:00.874868 [INF] Listening for client connections on 0.0.0.0:4222 +[16943] 2019/12/04 19:20:00.874874 [INF] Server id is NCR6KDDGWUU2FXO23WAXFY66VQE6JNWVMA24ALF2MO5GKAYFIMQULKUO +[16943] 2019/12/04 19:20:00.874877 [INF] Server is ready +``` + +These options can also be set in your configuration file: + +``` +// enables jetstream, an empty block will enable and use defaults +jetstream { + // jetstream data will be in /data/nats-server/jetstream + store_dir: "/data/nats-server" + + // 1GB + max_memory_store: 1073741824 + + // 10GB + max_file_store: 10737418240 +} +``` \ No newline at end of file diff --git a/jetstream/model_deep_dive/model_deep_dive.md b/jetstream/model_deep_dive/model_deep_dive.md new file mode 100644 index 0000000..aa87c20 --- /dev/null +++ b/jetstream/model_deep_dive/model_deep_dive.md @@ -0,0 +1,481 @@ +## Model Deep Dive + +The Orders example touched on a lot of features, but some like different Ack models and message limits, need a bit more detail. This section will expand on the above and fill in some blanks. + +### Stream Limits, Retention Modes and Discard Policy + +Streams store data on disk, but we cannot store all data forever so we need ways to control their size automatically. + +There are 3 features that come into play when Streams decide how long they store data. + +The `Retention Policy` describes based on what criteria a set will evict messages from its storage: + +|Retention Policy|Description| +|----------------|-----------| +|`LimitsPolicy` |Limits are set for how many messages, how big the storage and how old messages may be| +|`WorkQueuePolicy`|Messages are kept until they were consumed by any one single observer and then removed| +|`InterestPolicy`|Messages are kept as long as there are Consumers active for them| + +In all Retention Policies the basic limits apply as upper bounds, these are `MaxMsgs` for how many messages are kept in total, `MaxBytes` for how big the set can be in total and `MaxAge` for what is the oldest message that will be kept. These are the only limits in play with `LimitsPolicy` retention. + +One can then define additional ways a message may be removed from the Stream earlier than these limits. In `WorkQueuePolicy` the messages will be removed as soon as any Consumer received an Acknowledgement. In `InterestPolicy` messages will be removed as soon as there are no more Consumers. + +In both `WorkQueuePolicy` and `InterestPolicy` the age, size and count limits will still apply as upper bounds. + +A final control is the Maximum Size any single message may have. NATS have it's own limit for maximum size (1 MiB by default), but you can say a Stream will only accept messages up to 1024 bytes using `MaxMsgSize`. + +The `Discard Policy` sets how messages are discard when limits set by `LimitsPolicy` are reached. The `DiscardOld` option removes old messages making space for new, while `DiscardNew` refuses any new messages. + +The `WorkQueuePolicy` mode is a specialized mode where a message, once consumed and acknowledged, is discarded from the Stream. In this mode there are a few limits on consumers. Inherently it's about 1 message to one consumer, this means you cannot have overlapping consumers defined on the Stream - needs unique filter subjects. + +### Message Deduplication + +JetStream support idempotent message writes by ignoring duplicate messages as indicated by the `Msg-Id` header. + +```nohighlight +% nats req -H Msg-Id:1 ORDERS.new hello1 +% nats req -H Msg-Id:1 ORDERS.new hello2 +% nats req -H Msg-Id:1 ORDERS.new hello3 +% nats req -H Msg-Id:1 ORDERS.new hello4 +``` + +Here we set a `Msg-Id:1` header which tells JetStream to ensure we do not have duplicates of this message - we only consult the message ID not the body. + +```nohighlight +$ nats str info ORDERS +.... +State: + + Messages: 1 + Bytes: 67 B +``` + +The default window to track duplicates in is 2 minutes, this can be set on the command line using `--dupe-window` when creating a stream, though we would caution against large windows. + +### Acknowledgement Models + +Streams support acknowledging receiving a message, if you send a `Request()` to a subject covered by the configuration of the Stream the service will reply to you once it stored the message. If you just publish, it will not. A Stream can be set to disable Acknowledgements by setting `NoAck` to `true` in it's configuration. + +Consumers have 3 acknowledgement modes: + +|Mode|Description| +|----|-----------| +|`AckExplicit`|This requires every message to be specifically acknowledged, it's the only supported option for pull-based Consumers| +|`AckAll`|In this mode if you acknowledge message `100` it will also acknowledge message `1`-`99`, this is good for processing batches and to reduce ack overhead| +|`AckNone`|No acknowledgements are supported| + +To understand how Consumers track messages we will start with a clean `ORDERS` Stream and `DISPATCH` Consumer. + +``` +$ nats str info ORDERS +... +Statistics: + + Messages: 0 + Bytes: 0 B + FirstSeq: 0 + LastSeq: 0 + Active Consumers: 1 +``` + +The Set is entirely empty + +``` +$ nats con info ORDERS DISPATCH +... +State: + + Last Delivered Message: Consumer sequence: 1 Stream sequence: 1 + Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0 + Pending Messages: 0 + Redelivered Messages: 0 +``` + +The Consumer has no messages oustanding and has never had any (Consumer sequence is 1). + +We publish one message to the Stream and see that the Stream received it: + +``` +$ nats pub ORDERS.processed "order 4" +Published 7 bytes to ORDERS.processed +$ nats str info ORDERS +... +Statistics: + + Messages: 1 + Bytes: 53 B + FirstSeq: 1 + LastSeq: 1 + Active Consumers: 1 +``` + +As the Consumer is pull-based, we can fetch the message, ack it, and check the Consumer state: + +``` +$ nats con next ORDERS DISPATCH +--- received on ORDERS.processed +order 4 + +Acknowledged message + +$ nats con info ORDERS DISPATCH +... +State: + + Last Delivered Message: Consumer sequence: 2 Stream sequence: 2 + Acknowledgment floor: Consumer sequence: 1 Stream sequence: 1 + Pending Messages: 0 + Redelivered Messages: 0 +``` + +The message got delivered and acknowledged - `Acknowledgement floor` is `1` and `1`, the sequence of the Consumer is `2` which means its had only the one message through and got acked. Since it was acked, nothing is pending or redelivering. + +We'll publish another message, fetch it but not Ack it this time and see the status: + +``` +$ nats pub ORDERS.processed "order 5" +Published 7 bytes to ORDERS.processed + +$ nats con next ORDERS DISPATCH --no-ack +--- received on ORDERS.processed +order 5 + +$ nats con info ORDERS DISPATCH +State: + + Last Delivered Message: Consumer sequence: 3 Stream sequence: 3 + Acknowledgment floor: Consumer sequence: 1 Stream sequence: 1 + Pending Messages: 1 + Redelivered Messages: 0 +``` + +Now we can see the Consumer have processed 2 messages (obs sequence is 3, next message will be 3) but the Ack floor is still 1 - thus 1 message is pending acknowledgement. Indeed this is confirmed in the `Pending messages`. + +If I fetch it again and again do not ack it: + +``` +$ nats con next ORDERS DISPATCH --no-ack +--- received on ORDERS.processed +order 5 + +$ nats con info ORDERS DISPATCH +State: + + Last Delivered Message: Consumer sequence: 4 Stream sequence: 3 + Acknowledgment floor: Consumer sequence: 1 Stream sequence: 1 + Pending Messages: 1 + Redelivered Messages: 1 +``` + +The Consumer sequence increases - each delivery attempt increase the sequence - and our redelivered count also goes up. + +Finally if I then fetch it again and ack it this time: + +``` +$ nats con next ORDERS DISPATCH +--- received on ORDERS.processed +order 5 + +Acknowledged message +$ nats con info ORDERS DISPATCH +State: + + Last Delivered Message: Consumer sequence: 5 Stream sequence: 3 + Acknowledgment floor: Consumer sequence: 1 Stream sequence: 1 + Pending Messages: 0 + Redelivered Messages: 0 +``` + +Having now Acked the message there are no more pending. + +Additionally there are a few types of acknowledgements: + +|Type|Bytes|Description| +|----|-----|-----------| +|`AckAck`|nil, `+ACK`|Acknowledges a message was completely handled| +|`AckNak`|`-NAK`|Signals that the message will not be processed now and processing can move onto the next message, NAK'd message will be retried| +|`AckProgress`|`+WPI`|When sent before the AckWait period indicates that work is ongoing and the period should be extended by another equal to `AckWait`| +|`AckNext`|`+NXT`|Acknowledges the message was handled and requests delivery of the next message to the reply subject. Only applies to Pull-mode.| +|`AckTerm`|`+TERM`|Instructs the server to stop redelivery of a message without acknowledging it as successfully processed| + +So far all the examples was the `AckAck` type of acknowledgement, by replying to the Ack with the body as indicated in `Bytes` you can pick what mode of acknowledgement you want. + +All of these acknowledgement modes, except `AckNext`, support double acknowledgement - if you set a reply subject when acknowledging the server will in turn acknowledge having received your ACK. + +The `+NXT` acknowledgement can have a few formats: `+NXT 10` requests 10 messages and `+NXT {"no_wait": true}` which is the same data that can be sent in a Pull request. + +### Exactly Once Delivery + +JetStream supports Exactly Once delivery by combining Message Deduplication and double acks. + +On the publishing side you can avoid duplicate message ingestion using the [Message Deduplication](#message-deduplication) feature. + +Consumers can be 100% sure a message was correctly processed by requesting the server Acknowledge having received your acknowledgement by setting a reply subject on the Ack. If you receive this response you will never receive that message again. + +### Consumer Starting Position + +When setting up an Consumer you can decide where to start, the system supports the following for the `DeliverPolicy`: + +|Policy|Description| +|------|-----------| +|`all`|Delivers all messages that are available| +|`last`|Delivers the latest message, like a `tail -n 1 -f`| +|`new`|Delivers only new messages that arrive after subscribe time| +|`by_start_time`|Delivers from a specific time onward. Requires `OptStartTime` to be set| +|`by_start_sequence`|Delivers from a specific stream sequence. Requires `OptStartSeq` to be set| + +Regardless of what mode you set, this is only the starting point. Once started it will always give you what you have not seen or acknowledged. So this is merely how it picks the very first message. + +Lets look at each of these, first we make a new Stream `ORDERS` and add 100 messages to it. + +Now create a `DeliverAll` pull-based Consumer: + +``` +$ nats con add ORDERS ALL --pull --filter ORDERS.processed --ack none --replay instant --deliver all +$ nats con next ORDERS ALL +--- received on ORDERS.processed +order 1 + +Acknowledged message +``` + +Now create a `DeliverLast` pull-based Consumer: + +``` +$ nats con add ORDERS LAST --pull --filter ORDERS.processed --ack none --replay instant --deliver last +$ nats con next ORDERS LAST +--- received on ORDERS.processed +order 100 + +Acknowledged message +``` + +Now create a `MsgSetSeq` pull-based Consumer: + +``` +$ nats con add ORDERS TEN --pull --filter ORDERS.processed --ack none --replay instant --deliver 10 +$ nats con next ORDERS TEN +--- received on ORDERS.processed +order 10 + +Acknowledged message +``` + +And finally a time-based Consumer. Let's add some messages a minute apart: + +``` +$ nats str purge ORDERS +$ for i in 1 2 3 +do + nats pub ORDERS.processed "order ${i}" + sleep 60 +done +``` + +Then create an Consumer that starts 2 minutes ago: + +``` +$ nats con add ORDERS 2MIN --pull --filter ORDERS.processed --ack none --replay instant --deliver 2m +$ nats con next ORDERS 2MIN +--- received on ORDERS.processed +order 2 + +Acknowledged message +``` + +### Ephemeral Consumers + +So far, all the Consumers you have seen were Durable, meaning they exist even after you disconnect from JetStream. In our Orders scenario, though the `MONITOR` Consumer could very well be a short-lived thing there just while an operator is debugging the system, there is no need to remember the last seen position if all you are doing is wanting to observe the real-time state. + +In this case, we can make an Ephemeral Consumer by first subscribing to the delivery subject, then creating a durable and giving it no durable name. An Ephemeral Consumer exists as long as any subscription is active on its delivery subject. It is automatically be removed, after a short grace period to handle restarts, when there are no subscribers. + +Ephemeral Consumers can only be push-based. + +Terminal 1: + +``` +$ nats sub my.monitor +``` + +Terminal 2: + +``` +$ nats con add ORDERS --filter '' --ack none --target 'my.monitor' --deliver last --replay instant --ephemeral +``` + +The `--ephemeral` switch tells the system to make an Ephemeral Consumer. + +### Consumer Message Rates + +Typically what you want is if a new Consumer is made the selected messages are delivered to you as quickly as possible. You might want to replay messages at the rate they arrived though, meaning if messages first arrived 1 minute apart and you make a new Consumer it will get the messages a minute apart. + +This is useful in load testing scenarios etc. This is called the `ReplayPolicy` and have values of `ReplayInstant` and `ReplayOriginal`. + +You can only set `ReplayPolicy` on push-based Consumers. + +``` +$ nats con add ORDERS REPLAY --target out.original --filter ORDERS.processed --ack none --deliver all --sample 100 --replay original +... + Replay Policy: original +... +``` + +Now lets publish messages into the Set 10 seconds apart: + +``` +$ for i in 1 2 3 <15:15:35 +do + nats pub ORDERS.processed "order ${i}" + sleep 10 +done +Published [ORDERS.processed] : 'order 1' +Published [ORDERS.processed] : 'order 2' +Published [ORDERS.processed] : 'order 3' +``` + +And when we consume them they will come to us 10 seconds apart: + +``` +$ nats sub -t out.original +Listening on [out.original] +2020/01/03 15:17:26 [#1] Received on [ORDERS.processed]: 'order 1' +2020/01/03 15:17:36 [#2] Received on [ORDERS.processed]: 'order 2' +2020/01/03 15:17:46 [#3] Received on [ORDERS.processed]: 'order 3' +^C +``` + +### Stream Templates + +When you have many similar streams it can be helpful to auto create them, lets say you have a service by client and they are on subjects `CLIENT.*`, you can construct a template that will auto generate streams for any matching traffic. + +``` +$ nats str template add CLIENTS --subjects "CLIENT.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size 2048 --max-streams 1024 --discard old +Stream Template CLIENTS was created + +Information for Stream Template CLIENTS + +Configuration: + + Subjects: CLIENT.* + Acknowledgements: true + Retention: File - Limits + Replicas: 1 + Maximum Messages: -1 + Maximum Bytes: -1 + Maximum Age: 8760h0m0s + Maximum Message Size: 2048 + Maximum Consumers: -1 + Maximum Streams: 1024 + +Managed Streams: + + No Streams have been defined by this template +``` + +You can see no streams currently exist, let's publish some data: + +``` +$ nats pub CLIENT.acme hello +``` + +And we'll have 1 new Stream: + +``` +$ nats str ls +Streams: + + CLIENTS_acme +``` + +When the template is deleted all the streams it created will be deleted too. + +### Ack Sampling + +In the earlier sections we saw that samples are being sent to a monitoring system. Let's look at that in depth; how the monitoring system works and what it contains. + +As messages pass through an Consumer you'd be interested in knowing how many are being redelivered and how many times but also how long it takes for messages to be acknowledged. + +Consumers can sample Ack'ed messages for you and publish samples so your monitoring system can observe the health of an Consumer. We will add support for this to [NATS Surveyor](https://github.com/nats-io/nats-surveyor). + +#### Configuration + +You can configure an Consumer for sampling by passing the `--sample 80` option to `nats consumer add`, this tells the system to sample 80% of Acknowledgements. + +When viewing info of an Consumer you can tell if it's sampled or not: + +``` +$ nats con info ORDERS NEW +... + Sampling Rate: 100 +... +``` + +#### Consuming + +Samples are published to `$JS.EVENT.METRIC.CONSUMER_ACK..` in JSON format containing `api.ConsumerAckMetric`. Use the `nats con events` command to view samples: + +```nohighlight +$ nats con events ORDERS NEW +Listening for Advisories on $JS.EVENT.ADVISORY.*.ORDERS.NEW +Listening for Metrics on $JS.EVENT.METRIC.*.ORDERS.NEW + +15:08:31] [Ph0fsiOKRg1TS0c2k0mMz2] Acknowledgement Sample + Consumer: ORDERS > NEW + Stream Sequence: 40 + Consumer Sequence: 161 + Deliveries: 1 + Delay: 1.009ms +``` + +```nohighlight +$ nats con events ORDERS NEW --json +{ + "stream": "ORDERS", + "consumer": "NEW", + "consumer_seq": 155, + "stream_seq": 143, + "ack_time": 5387000, + "delivered": 1 +} +{ + "stream": "ORDERS", + "consumer": "NEW", + "consumer_seq": 156, + "stream_seq": 144, + "ack_time": 5807800, + "delivered": 1 +} +``` + +### Storage Overhead + +JetStream file storage is very efficient storing as little extra information about the message as possible. + +**NOTE:** This might change once clustering is supported. + +We do store some message data with each message, namely: + + * Message headers + * The subject it was received on + * The time it was received + * The message payload + * A hash of the message + * The message sequence + * A few other bits like length of the subject and lengh of headers + +Without any headers the size is: + +``` +length of the message record (4bytes) + seq(8) + ts(8) + subj_len(2) + subj + msg + hash(8) +``` + +A 5 byte `hello` message without headers will take 39 bytes. + +With headers: + +``` +length of the message record (4bytes) + seq(8) + ts(8) + subj_len(2) + subj + hdr_len(4) + hdr + msg + hash(8) +``` + +So if you are publishing many small messages the overhead will be, relatively speaking, quite large, but for larger messages +the overhead is very small. If you publish many small messages it's worth trying to optimise the subject length. diff --git a/jetstream/monitoring/monitoring.md b/jetstream/monitoring/monitoring.md new file mode 100644 index 0000000..13b5185 --- /dev/null +++ b/jetstream/monitoring/monitoring.md @@ -0,0 +1,30 @@ +## Monitoring + +### Server Metrics + +Typically, NATS is monitored via HTTP endpoints like `/varz`, we do not at this moment have a JetStream equivelant, but it's planned that server and account level metrics will be made available. + +### Advisories + +JetStream publish a number of advisories that can inform operations about health and state of the Streams. These advisories are published to normal NATS subjects below `$JS.EVENT.ADVISORY.>` and one can store these advisories in JetStream Streams if desired. + +The command `nats event --js-advsiroy` can view all these events on your console. The Golang package [jsm.go](https://github.com/nats-io/jsm.go) can consume and render these events and have data types for each of these events. + +All these events have JSON Schemas that describe them, schemas can be viewed on the CLI using the `nats schema show ` command. + +|Description|Subject|Kind| +|-----------|-------|----| +|API interactions|`$JS.EVENT.ADVISORY.API`|`io.nats.jetstream.advisory.v1.api_audit`| +|Stream CRUD operations|`$JS.EVENT.ADVISORY.STREAM.CREATED.`|`io.nats.jetstream.advisory.v1.stream_action`| +|Consumer CRUD operations|`$JS.EVENT.ADVISORY.CONSUMER.CREATED..`|`io.nats.jetstream.advisory.v1.consumer_action`| +|Snapshot started using `nats stream backup`|`$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_CREATE.`|`io.nats.jetstream.advisory.v1.snapshot_create`| +|Snapshot completed|`$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_COMPLETE.`|`io.nats.jetstream.advisory.v1.snapshot_complete`| +|Restore started using `nats stream restore`|`$JS.EVENT.ADVISORY.STREAM.RESTORE_CREATE.`|`io.nats.jetstream.advisory.v1.restore_create`| +|Restore completed|`$JS.EVENT.ADVISORY.STREAM.RESTORE_COMPLETE.`|`io.nats.jetstream.advisory.v1.restore_complete`| +|Consumer maximum delivery reached|`$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES..`|`io.nats.jetstream.advisory.v1.max_deliver`| +|Message delivery terminated using AckTerm|`$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED..`|`io.nats.jetstream.advisory.v1.terminated`| +|Message acknowledged in a sampled Consumer|`$JS.EVENT.METRIC.CONSUMER.ACK..`|`io.nats.jetstream.metric.v1.consumer_ack`| + +### Dashboards + +The [NATS Surveyor](https://github.com/nats-io/nats-surveyor) system has initial support for passing JetStream metrics to Prometheus, dashboards and more will be added towards final release. \ No newline at end of file diff --git a/jetstream/multi-tenancy/resource_management.md b/jetstream/multi-tenancy/resource_management.md new file mode 100644 index 0000000..20d9838 --- /dev/null +++ b/jetstream/multi-tenancy/resource_management.md @@ -0,0 +1,133 @@ +## Multi Tenancy and Resource Management + +JetStream is compatible with NATS 2.0 Multi Tenancy using Accounts. A JetStream enabled server supports creating fully isolated JetStream environments for different accounts. + +To enable JetStream in a server we have to configure it at the top level first: + +``` +jetstream: enabled +``` + +This will dynamically determine the available resources. It's recommended that you set specific limits though: + +``` +jetstream { + store_dir: /data/jetstream + max_mem: 1G + max_file: 100G +} +``` + +At this point JetStream will be enabled and if you have a server that does not have accounts enabled all users in the server would have access to JetStream + +``` +jetstream { + store_dir: /data/jetstream + max_mem: 1G + max_file: 100G +} + +accounts { + HR: { + jetstream: enabled + } +} +``` + +Here the `HR` account would have access to all the resources configured on the server, we can restrict it: + +``` +jetstream { + store_dir: /data/jetstream + max_mem: 1G + max_file: 100G +} + +accounts { + HR: { + jetstream { + max_mem: 512M + max_file: 1G + max_streams: 10 + max_consumers: 100 + } + } +} +``` + +Now the `HR` account it limited in various dimensions. + +If you try to configure JetStream for an account without enabling it globally you'll get a warning and the account designated as System cannot have JetStream enabled. + +## `nats` CLI + +As part of the JetStream efforts a new `nats` CLI is being developed to act as a single point of access to the NATS eco system. + +This CLI has been seen throughout the guide, it's available in the Docker containers today and downloadable on the [Releases](https://github.com/nats-io/jetstream/releases) +page. + +### Configuration Contexts + +The CLI has a number of environment configuration settings - where your NATS server is, credentials, TLS keys and more: + +```nohighlight +$ nats --help +... + -s, --server=NATS_URL NATS servers + --user=NATS_USER Username of Token + --password=NATS_PASSWORD Password + --creds=NATS_CREDS User credentials + --nkey=NATS_NKEY User NKEY + --tlscert=NATS_CERT TLS public certificate + --tlskey=NATS_KEY TLS private key + --tlsca=NATS_CA TLS certificate authority chain + --timeout=NATS_TIMEOUT Time to wait on responses from NATS + --context=CONTEXT NATS Configuration Context to use for access +... +``` + +You can set these using the CLI flag, the environmet variable - like **NATS_URL** - or using our context feature. + +A context is a named configuration that stores all these settings, you can switch between access configurations and +designate a default. + +Creating one is easy, just specify the same settings to the `nats context save` + +```nohighlight +$ nats context save example --server nats://nats.example.net:4222 --description 'Example.Net Server' +$ nats context save local --server nats://localhost:4222 --description 'Local Host' --select +$ nats context ls +Known contexts: + + example Example.Net Server + local* Local Host +``` + +We passed `--select` to the `local` one meaning it will be the default when nothing is set. + +```nohighlight +$ nats rtt +nats://localhost:4222: + + nats://127.0.0.1:4222: 245.115µs + nats://[::1]:4222: 390.239µs + +$ nats rtt --context example +nats://nats.example.net:4222: + + nats://192.0.2.10:4222: 41.560815ms + nats://192.0.2.11:4222: 41.486609ms + nats://192.0.2.12:4222: 41.178009ms +``` + +The `nats context select` command can be used to set the default context. + +All `nats` commands are context aware and the `nats context` command has various commands to view, edit and remove contexts. + +Server URLs and Credential paths can be resolved via the `nsc` command by specifying an url, for example to find user `new` within the `orders` account of the `acme` operator you can use this: + +```nohighlight +$ nats context save example --description 'Example.Net Server' --nsc nsc://acme/orders/new +``` + +The server list and credentials path will now be resolved via `nsc`, if these are specifically set in the context, the specific context configuration will take precedence. diff --git a/jetstream/nats_api_reference/nats_api_reference.md b/jetstream/nats_api_reference/nats_api_reference.md new file mode 100644 index 0000000..1e994ed --- /dev/null +++ b/jetstream/nats_api_reference/nats_api_reference.md @@ -0,0 +1,221 @@ +## NATS API Reference + +Thus far we saw a lot of CLI interactions. The CLI works by sending and receiving specially crafted messages over core NATS to configure the JetStream system. In time we will look to add file based configuration but for now the only method is the NATS API. + +**NOTE:** Some NATS client libraries may need to enable an option to use old style requests when interacting withe JetStream server. Consult the libraries README's for more information. + +### Reference + +All of these subjects are found as constants in the NATS Server source, so for example the `$JS.API.STREAM.LIST` is a constant in the nats-server source `api.JetStreamListStreams` tables below will reference these constants and likewise data structures in the server for payloads. + +### Error Handling + +The APIs used for administrative tools all respond with standardised JSON and these include errors. + +```nohighlight +$ nats req '$JS.API.STREAM.INFO.nonexisting' '' +Published 11 bytes to $JS.API.STREAM.INFO.nonexisting +Received [_INBOX.lcWgjX2WgJLxqepU0K9pNf.mpBW9tHK] : { + "type": "io.nats.jetstream.api.v1.stream_info_response", + "error": { + "code": 404, + "description": "stream not found" + } +} +``` + +```nohighlight +$ nats req '$JS.STREAM.INFO.ORDERS' '' +Published 6 bytes to $JS.STREAM.INFO.ORDERS +Received [_INBOX.fwqdpoWtG8XFXHKfqhQDVA.vBecyWmF] : '{ + "type": "io.nats.jetstream.api.v1.stream_info_response", + "config": { + "name": "ORDERS", + ... +} +``` + +Here the responses include a `type` which can be used to find the JSON Schema for each response. + +Non admin APIs - like those for adding a message to the stream will respond with `-ERR` or `+OK` with an optional reason after. + +### Admin API + +All the admin actions the `nats` CLI can do falls in the sections below. The API structure are kept in the `api` package in the `jsm.go` repository. + +Subjects that and in `T` like `api.JSApiConsumerCreateT` are formats and would need to have the Stream Name and in some cases also the Consumer name interpolated into them. In this case `t := fmt.Sprintf(api.JSApiConsumerCreateT, streamName)` to get the final subject. + +The command `nats events` will show you an audit log of all API access events which includes the full content of each admin request, use this to view the structure of messages the `nats` command sends. + +The API uses JSON for inputs and outputs, all the responses are typed using a `type` field which indicates their Schema. A JSON Schema repository can be found in `nats-io/jetstream/schemas`. + +#### General Info + +|Subject|Constant|Description|Request Payload|Response Payload| +|-------|--------|-----------|---------------|----------------| +|`$JS.API.INFO`|`api.JSApiAccountInfo`|Retrieves stats and limits about your account|empty payload|`api.JetStreamAccountStats`| + +#### Streams + +|Subject|Constant|Description|Request Payload|Response Payload| +|-------|--------|-----------|---------------|----------------| +|`$JS.API.STREAM.LIST`|`api.JSApiStreamList`|Paged list known Streams including all their current information|`api.JSApiStreamListRequest`|`api.JSApiStreamListResponse`| +|`$JS.API.STREAM.NAMES`|`api.JSApiStreamNames`|Paged list of Streams|`api.JSApiStreamNamesRequest`|`api.JSApiStreamNamesResponse`| +|`$JS.API.STREAM.LOOKUP`|`api.JSApiStreamLookup`|Finds a stream with interest on a certain subject|`api.JSApiStreamLookupRequest`|`api.JSApiStreamLookupResponse`| +|`$JS.API.STREAM.CREATE.*`|`api.JSApiStreamCreateT`|Creates a new Stream|`api.StreamConfig`|`api.JSApiStreamCreateResponse`| +|`$JS.API.STREAM.UPDATE.*`|`api.JSApiStreamUpdateT`|Updates an existing Stream with new config|`api.StreamConfig`|`api.JSApiStreamUpdateResponse`| +|`$JS.API.STREAM.INFO.*`|`api.JSApiStreamInfoT`|Information about config and state of a Stream|empty payload, Stream name in subject|`api.JSApiStreamInfoResponse`| +|`$JS.API.STREAM.DELETE.*`|`api.JSApiStreamDeleteT`|Deletes a Stream and all its data|empty payload, Stream name in subject|`api.JSApiStreamDeleteResponse`| +|`$JS.API.STREAM.PURGE.*`|`api.JSApiStreamPurgeT`|Purges all of the data in a Stream, leaves the Stream|empty payload, Stream name in subject|`api.JSApiStreamPurgeResponse`| +|`$JS.API.STREAM.MSG.DELETE.*`|`api.JSApiMsgDeleteT`|Deletes a specific message in the Stream by sequence, useful for GDPR compliance|`api.JSApiMsgDeleteRequest`|`api.JSApiMsgDeleteResponse`| +|`$JS.API.STREAM.MSG.GET.*`|`api.JSApiMsgGetT`|Retrieves a specific message from the stream|`api.JSApiMsgGetRequest`|`api.JSApiMsgGetResponse`| +|`$JS.API.STREAM.SNAPSHOT.*`|`api.JSApiStreamSnapshotT`|Initiates a streaming backup of a streams data|`api.JSApiStreamSnapshotRequest`|`api.JSApiStreamSnapshotResponse`| +|`$JS.API.STREAM.RESTORE.*`|`api.JSApiStreamRestoreT`|Initiates a streaming restore of a stream|`{}`|`api.JSApiStreamRestoreResponse`| + +#### Stream Templates + +|Subject|Constant|Description|Request Payload|Response Payload| +|-------|--------|-----------|---------------|----------------| +|`$JS.API.STREAM.TEMPLATE.CREATE.*`|`api.JSApiTemplateCreateT`|Creates a Stream Template|`api.StreamTemplateConfig`|`api.JSApiStreamTemplateCreateResponse`| +|`$JS.API.STREAM.TEMPLATE.NAMES`|`api.JSApiTemplateNames`|Paged list all known templates|`api.JSApiStreamTemplateNamesRequest`|`api.JSApiStreamTemplateNamesResponse`| +|`$JS.API.STREAM.TEMPLATE.INFO.*`|`api.JSApiTemplateInfoT`|Information about the config and state of a Stream Template|empty payload, Template name in subject|`api.JSApiStreamTemplateInfoResponse`| +|`$JS.API.STREAM.TEMPLATE.DELETE.*`|`api.JSApiTemplateDeleteT`|Delete a specific Stream Template **and all streams created by this template**|empty payload, Template name in subject|`api.JSApiStreamTemplateDeleteResponse`| + +#### Consumers + +|Subject|Constant|Description|Request Payload|Response Payload| +|-------|--------|-----------|---------------|----------------| +|`$JS.API.CONSUMER.CREATE.*`|`api.JSApiConsumerCreateT`|Create an ephemeral Consumer|`api.ConsumerConfig`, Stream name in subject|`api.JSApiConsumerCreateResponse`| +|`$JS.API.CONSUMER.DURABLE.CREATE.*`|`api.JSApiDurableCreateT`|Create an Consumer|`api.ConsumerConfig`, Stream name in subject|`api.JSApiConsumerCreateResponse`| +|`$JS.API.CONSUMER.LIST.*`|`api.JSApiConsumerListT`|Paged list of known Consumers including their current info|`api.JSApiConsumerListRequest`|`api.JSApiConsumerListResponse`| +|`$JS.API.CONSUMER.NAMES.*`|`api.JSApiConsumerNamesT`|Paged list of known Consumer names|`api.JSApiConsumerNamesRequest`|`api.JSApiConsumerNamesResponse`| +|`$JS.API.CONSUMER.INFO.*.*`|`api.JSApiConsumerInfoT`|Information about an Consumer|empty payload, Stream and Consumer names in subject|`api.JSApiConsumerInfoResponse`| +|`$JS.API.CONSUMER.DELETE.*.*`|`api.JSApiConsumerDeleteT`|Deletes an Consumer|empty payload, Stream and Consumer names in subject|`api.JSApiConsumerDeleteResponse`| + +#### ACLs + +It's hard to notice here but there is a clear pattern in these subjects, lets look at the various JetStream related subjects: + +General information + +``` +$JS.API.INFO +``` + +Stream and Consumer Admin + +``` +$JS.API.STREAM.CREATE. +$JS.API.STREAM.UPDATE. +$JS.API.STREAM.DELETE. +$JS.API.STREAM.INFO. +$JS.API.STREAM.PURGE. +$JS.API.STREAM.LIST +$JS.API.STREAM.NAMES +$JS.API.STREAM.MSG.DELETE. +$JS.API.STREAM.MSG.GET. +$JS.API.STREAM.SNAPSHOT. +$JS.API.STREAM.RESTORE. +$JS.API.CONSUMER.CREATE. +$JS.API.CONSUMER.DURABLE.CREATE.. +$JS.API.CONSUMER.DELETE.. +$JS.API.CONSUMER.INFO.. +$JS.API.CONSUMER.LIST. +$JS.API.CONSUMER.MSG.NEXT.. +$JS.API.CONSUMER.NAMES. +$JS.API.STREAM.TEMPLATE.CREATE. +$JS.API.STREAM.TEMPLATE.DELETE. +$JS.API.STREAM.TEMPLATE.INFO. +$JS.API.STREAM.TEMPLATE.NAMES +``` + +Stream and Consumer Use + +``` +$JS.API.CONSUMER.MSG.NEXT.. +$JS.ACK...x.x.x +$JS.SNAPSHOT.ACK.. +$JS.SNAPSHOT.RESTORE.. +``` + +Events and Advisories: + +``` +$JS.EVENT.METRIC.CONSUMER_ACK.. +$JS.EVENT.ADVISORY.MAX_DELIVERIES.. +``` + +This allow you to easily create ACL rules that limit users to a specific Stream or Consumer and to specific verbs for administration purposes. For ensuring only the receiver of a message can Ack it we have response permissions ensuring you can only Publish to Response subject for messages you received. + +### Acknowledging Messages + +Messages that need acknowledgement will have a Reply subject set, something like `$JS.ACK.ORDERS.test.1.2.2`, this is the prefix defined in `api.JetStreamAckPre` followed by `......`. + +In all the Synadia maintained API's you can simply do `msg.Respond(nil)` (or language equivalent) which will send nil to the reply subject. + +### Fetching The Next Message From a Pull-based Consumer + +If you have a pull-based Consumer you can send a standard NATS Request to `$JS.API.CONSUMER.MSG.NEXT..`, here the format is defined in `api.JetStreamRequestNextT` and requires populating using `fmt.Sprintf()`. + +```nohighlight +$ nats req '$JS.API.CONSUMER.MSG.NEXT.ORDERS.test' '1' +Published 1 bytes to $JS.API.CONSUMER.MSG.NEXT.ORDERS.test +Received [js.1] : 'message 1' +``` + +Here we ask for just 1 message - `nats req` only shows 1 - but you can fetch a batch of messages by varying the argument. This combines well with the `AckAll` Ack policy. + +The above request for the next message will stay in the server for as long as the client is connected and future pulls from the same client will accumulate on the server, meaning if you ask for 1 message 100 times and 1000 messages arrive you'll get sent 100 messages not 1. + +This is often not desired, pull consumers support a mode where a JSON document is sent describing the pull request. + +```json +{ + "expires": "2020-11-10T12:41:00.075933464Z", + "batch": 10, +} +``` + +This requests 10 messages and asks the server to keep this request until the specific `expires` time, this is useful when you poll the server frequently and do not want the pull requests to accumulate on the server. Set the expire time to now + your poll frequency. + +```json +{ + "batch": 10, + "no_wait": true +} +``` + +Here we see a second format of the Pull request that will not store the request on the queue at all but when there are no messages to deliver will send a nil bytes message with a `Status` header of `404`, this way you can know when you reached the end of the stream for example. A `409` is returned if the Consumer has reached `MaxAckPending` limits. + +``` +[rip@dev1]% nats req '$JS.API.CONSUMER.MSG.NEXT.ORDERS.NEW' '{"no_wait": true, "batch": 10}' + test --password test +13:45:30 Sending request on "$JS.API.CONSUMER.MSG.NEXT.ORDERS.NEW" +13:45:30 Received on "_INBOX.UKQGqq0W1EKl8inzXU1naH.XJiawTRM" rtt 594.908µs +13:45:30 Status: 404 +13:45:30 Description: No Messages +``` + +### Fetching From a Stream By Sequence + +If you know the Stream sequence of a message you can fetch it directly, this does not support acks. Do a Request() to `$JS.API.STREAM.MSG.GET.ORDERS` sending it the message sequence as payload. Here the prefix is defined in `api.JetStreamMsgBySeqT` which also requires populating using `fmt.Sprintf()`. + +```nohighlight +$ nats req '$JS.API.STREAM.MSG.GET.ORDERS' '{"seq": 1}' +Published 1 bytes to $JS.STREAM.ORDERS.MSG.BYSEQ +Received [_INBOX.cJrbzPJfZrq8NrFm1DsZuH.k91Gb4xM] : '{ + "type": "io.nats.jetstream.api.v1.stream_msg_get_response", + "message": { + "subject": "x", + "seq": 1, + "data": "aGVsbG8=", + "time": "2020-05-06T13:18:58.115424+02:00" + } +}' +``` + +The Subject shows where the message was received, Data is base64 encoded and Time is when it was received. + +### Consumer Samples + +Samples are published to a specific subject per Consumer, something like `$JS.EVENT.METRIC.CONSUMER_ACK..` you can just subscribe to that and get `api.ConsumerAckMetric` messages in JSON format. The prefix is defined in `api.JetStreamMetricConsumerAckPre`. From 5a1f9b9c6caa4d70a19eead01c6c20fd568eaa82 Mon Sep 17 00:00:00 2001 From: ainsley Date: Tue, 24 Nov 2020 16:56:10 -0600 Subject: [PATCH 02/84] adding JetStream to outline --- SUMMARY.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/SUMMARY.md b/SUMMARY.md index 0d6e9d0..d35f1ef 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -204,3 +204,37 @@ * [Creating a NATS Super Cluster in Digital Ocean with Helm](nats-on-kubernetes/super-cluster-on-digital-ocean.md) * [From Zero to K8S to Leafnodes using Helm](nats-on-kubernetes/from-zero-to-leafnodes.md) +## JetStream + +* [Concepts](jetstream/concepts/concepts.md) + * [Streams](jetstream/concepts/streams.md) + * [Consumes](jetstream/concepts/consumers.md) + * [Configuration](jetstream/concepts/configuration.md) +* [Getting Started](jetstream/getting_started/getting_started.md) + * [Using Docker](jetstream/getting_started/using_docker.md) + * [Using Docker with NGS](jetstream/getting_started/using_docker.md#using-docker-with-ngs) + * [Using Source](jetstream/getting_started/using_source.md) +* [Administration & Usage from CLI](jetstream/administration/administration.md) + * [Account Information](jetstream/administration/account.md) + * [Streams](jetstream/administration/streams.md) + * [Consumers](jetstream/administration/consumers.md) +* [Monitoring](jetstream/monitoring/monitoring.md) +* [Configuration Management](jetstream/configuration_mgmt/configuration_mgmt.md) + * [nats CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) + * [Terraform](jetstream/configuration_mgmt/configuration_mgmt.md#terraform) + * [GitHub Actions](jetstream/configuration_mgmt/github_actions.md) + * [Kubernetes Controller](jetstream/configuration_mgmt/kubernetes_controller.md) +* [Disaser Recovery](jetstream/disaster_recovery/disaster_recovery.md) +* [Model Deep Dive](jetstream/model_deep_dive/model_deep_dive.md) + * [Stream Limits & Retention Modes](jetstream/model_deep_dive/model_deep_dive.md#stream-limits-retention-modes-and-discard-policy) + * [Message Deduplication](jetstream/model_deep_dive/model_deep_dive.md#message-deduplication) + * [Acknowledgment Models](jetstream/model_deep_dive/model_deep_dive.md#acknowledgement-models) + * [Exactly Once Delivery](jetstream/model_deep_dive/model_deep_dive.md#exactly-once-delivery) + * [Consumer Starting Position](jetstream/model_deep_dive/model_deep_dive.md#consumer-starting-position) + * [Ephemeral Consumers](jetstream/model_deep_dive/model_deep_dive.md#ephemeral-consumers) + * [Consumer Message Rates](jetstream/model_deep_dive/model_deep_dive.md#consumer-message-rates) + * [Stream Templates](jetstream/model_deep_dive/model_deep_dive.md#stream-templates) + * [Ack Sampling](jetstream/model_deep_dive/model_deep_dive.md#ack-sampling) + * [Storage Overhead](jetstream/model_deep_dive/model_deep_dive.md#storage-overhead) +* [NATS API Reference](jetstream/nats_api_reference/nats_api_reference.md) +* [Multi-tenancy & Resource Mgmt](jetstream/multi-tenancy/resource_management.md) From fa388104238e1942df212cdf0f4cba04f3f2c0a1 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 11 Dec 2020 14:28:53 -0600 Subject: [PATCH 03/84] Update SUMMARY.md add updates from https://github.com/nats-io/jetstream/pull/392 --- SUMMARY.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SUMMARY.md b/SUMMARY.md index d35f1ef..c0c55b0 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -220,13 +220,13 @@ * [Consumers](jetstream/administration/consumers.md) * [Monitoring](jetstream/monitoring/monitoring.md) * [Configuration Management](jetstream/configuration_mgmt/configuration_mgmt.md) - * [nats CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) + * [nats Admin CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) * [Terraform](jetstream/configuration_mgmt/configuration_mgmt.md#terraform) * [GitHub Actions](jetstream/configuration_mgmt/github_actions.md) * [Kubernetes Controller](jetstream/configuration_mgmt/kubernetes_controller.md) * [Disaser Recovery](jetstream/disaster_recovery/disaster_recovery.md) * [Model Deep Dive](jetstream/model_deep_dive/model_deep_dive.md) - * [Stream Limits & Retention Modes](jetstream/model_deep_dive/model_deep_dive.md#stream-limits-retention-modes-and-discard-policy) + * [Stream Limits, Retention Modes and Discard Policy](jetstream/model_deep_dive/model_deep_dive.md#stream-limits-retention-modes-and-discard-policy) * [Message Deduplication](jetstream/model_deep_dive/model_deep_dive.md#message-deduplication) * [Acknowledgment Models](jetstream/model_deep_dive/model_deep_dive.md#acknowledgement-models) * [Exactly Once Delivery](jetstream/model_deep_dive/model_deep_dive.md#exactly-once-delivery) From c69c4e404daca8977e0df966291921675066ac51 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jaime=20Pi=C3=B1a?= Date: Mon, 25 Jan 2021 11:55:30 -0800 Subject: [PATCH 04/84] Remove shell prompt character Currently, our copy-able snippets have a shell prompt. When someone clicks on the copy button, the shell prompt gets copied along with the command we want copied. This removes the shell prompt so that users don't have to edit that out themselves when they paste one of our commands. --- nats-tools/mkpasswd.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/nats-tools/mkpasswd.md b/nats-tools/mkpasswd.md index 267df17..0da545f 100644 --- a/nats-tools/mkpasswd.md +++ b/nats-tools/mkpasswd.md @@ -9,15 +9,15 @@ A utility for creating `bcrypt` hashes is included with the nats-server distribu If you have [go installed](https://golang.org/doc/install), you can easily install the `mkpasswd` tool by doing: ```text -> go get github.com/nats-io/nats-server/util/mkpasswd +go get github.com/nats-io/nats-server/util/mkpasswd ``` Alternatively, you can: ```text -> git clone git@github.com:nats-io/nats-server -> cd nats-server/util/mkpasswd -> go install mkpasswd.go +git clone git@github.com:nats-io/nats-server +cd nats-server/util/mkpasswd +go install mkpasswd.go ``` ## Generating bcrypted passwords From f1a92bdcd6997fc6cac1973fef00c11480febd84 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Thu, 4 Feb 2021 17:48:40 -0700 Subject: [PATCH 05/84] add about jetstream Signed-off-by: Colin Sullivan --- jetstream/about_jetstream/jetstream.md | 31 ++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 jetstream/about_jetstream/jetstream.md diff --git a/jetstream/about_jetstream/jetstream.md b/jetstream/about_jetstream/jetstream.md new file mode 100644 index 0000000..29261b8 --- /dev/null +++ b/jetstream/about_jetstream/jetstream.md @@ -0,0 +1,31 @@ +# Jetstream + +JetStream was created to solve the problems identified with streaming in technology today - complexity, fragility, and a lack of scalability. Some technologies address these better than others, but no current streaming technology is truly multi-tenant, horizontally scalable, and supports multiple deployment models. No technology we are aware of can scale from edge to cloud under the same security context while having complete deployment observability for operations. + +## Goals +JetStream was developed with the following goals in mind: + +- The system must be easy to configure and operate and be observable. +- The system must be secure and operate well with NATS 2.0 security models. +- The system must scale horizontally and be applicable to a high ingestion rate. +- The system must support multiple use cases. +- The system must self heal and always be available. +- The system must have an API that is closer to core NATS. +- The system must allow NATS messages to be part of a stream as desired. +- The system must display payload agnostic behavior. +- The system must not have third party dependencies. + +## High-Level Design and Features +In terms of deployment, a JetStream server is simply a NATS server with the JetStream subsystem enabled, launched with the `-js` flag with a configured server name and cluster name. From a client perspective, it does not matter which servers are running JetStream so long as there is some route to a JetStream enabled server or servers. This allows for a flexible deployment which to optimize resources for particular servers that will store streams versus very low overhead stateless servers, reducing OpEx and ultimately creating a scalable and manageable system. + +## Feature List +- At-least-once delivery; exactly once within a window +- Store messages and replay by time or sequence +- Wildcard support +- Account aware +- Data at rest encryption +- Cleanse specific messages (GDPR) +- Horizontal scalability +- Persist Streams and replay via Consumers + +JetStream is designed to bifurcate ingestion and consumption of messages to provide multiple ways to consume data from the same stream. To that end, JetStream functionality is composed of server streams and server consumers. From d422ccb39435c2c0059ae67dbe7d1c24ae55e84b Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Thu, 4 Feb 2021 17:48:57 -0700 Subject: [PATCH 06/84] update getting started Signed-off-by: Colin Sullivan --- jetstream/getting_started/getting_started.md | 27 ++++++++++++++++++-- jetstream/getting_started/using_docker.md | 2 +- jetstream/getting_started/using_source.md | 2 +- 3 files changed, 27 insertions(+), 4 deletions(-) diff --git a/jetstream/getting_started/getting_started.md b/jetstream/getting_started/getting_started.md index 0f95734..91b2364 100644 --- a/jetstream/getting_started/getting_started.md +++ b/jetstream/getting_started/getting_started.md @@ -1,3 +1,26 @@ -## Getting Started +# Getting Started -This tech preview is limited to a single server and defaults to the global account. JetStream is NATS 2.0 aware and is scoped to accounts from a resource limit perspective. This is not the same as an individual server's resources, but may feel that way starting out. Don't worry, clustering is coming next but we wanted to get input early from the community. \ No newline at end of file +Getting started with Jetstream is straightforward. While we speak of Jetstream as if it is a seperate component, it's actually a subsystem built into the NATS server that needs to be enabled. + +## Command line + +Enable Jetstream by specifying the `-js` flag when starting the NATS server. + +`$ nats-server -js` + +## Configuration File + +Enable Jetstream through a configuration file. By default, the Jetstream subsytem will store data in the /tmp directory. Here's a minimal file that will store data in a local "nats" directory, suitable for development and local testing. + +`$ nats-server -c js.conf` + +```text +# js.conf +jetstream { + store_dir=nats +} +``` + +Normally Jetstream will be run in clustered mode and will replicate data, so the best place to store Jetstream data would be locally on a fast SSD. One should specifically avoid NAS or NFS storage for Jetstream. + +See [Using Docker](./using_docker.md) and [Using Source](./using_source.md) for more information. \ No newline at end of file diff --git a/jetstream/getting_started/using_docker.md b/jetstream/getting_started/using_docker.md index d8923ea..a16549e 100644 --- a/jetstream/getting_started/using_docker.md +++ b/jetstream/getting_started/using_docker.md @@ -1,4 +1,4 @@ -### Using Docker +# Using Docker The `synadia/jsm:latest` docker image contains both the JetStream enabled NATS Server and the `nats` utility this guide covers. diff --git a/jetstream/getting_started/using_source.md b/jetstream/getting_started/using_source.md index 4f6bee2..b21eb8a 100644 --- a/jetstream/getting_started/using_source.md +++ b/jetstream/getting_started/using_source.md @@ -1,4 +1,4 @@ -### Using Source +# Using Source or a Binary You will also want to have installed from the nats.go repo the examples/tools such as nats-pub, nats-sub, nats-req and possibly nats-bench. One of the design goals of JetStream was to be native to core NATS, so even though we will most certainly add in syntactic sugar to clients to make them more appealing, for this tech preview we will be using plain old NATS. From 7bf9e7ce655aa5336f9509149cf408012faa3102 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Thu, 4 Feb 2021 22:12:21 -0700 Subject: [PATCH 07/84] add clustering doc Signed-off-by: Colin Sullivan --- jetstream/clustering/clustering.md | 92 ++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 jetstream/clustering/clustering.md diff --git a/jetstream/clustering/clustering.md b/jetstream/clustering/clustering.md new file mode 100644 index 0000000..1cb9fce --- /dev/null +++ b/jetstream/clustering/clustering.md @@ -0,0 +1,92 @@ +# Clustering + +Clustering in Jetstream is required for a highly available and scalable system. Behind +clustering is RAFT. There's no need to understand RAFT in depth to use clustering, but knowing a little explains some of the requirements behind setting up Jetstream clusters. + +## RAFT +JetStream uses a NATS optimized RAFT algorithm for clustering. Typically raft generates a lot of traffic, but the NATS server optimizes this by combining the data plane for replicating messages with the messages RAFT would normally use to ensure consensus. + +### Raft groups +The RAFT groups include API handlers sstreams, consumers, and an internal algorithm designates which servers handle which streams and consumers. + +The raft algorithm has a few requirements: +- A log to persist state +- A quorum for consensus. + +### The Quorum +In order to ensure data consistency across complete restarts, a quorum of servers is required. A quorum is ½ cluster size + 1. This is the minimum number of nodes to ensure at least one node has the most recent data and state after a catastrophic failure. So for a cluster size of 3, you’ll need at least two Jetstream enabled NATS servers available to store new messages. For a cluster size of 5, you’ll need at least 3 NATS servers, and so forth. + +### Cluster Size +Generally we recommend 3 or 5 Jetstream enabled servers in a NATS cluster. This balances scalability with a tolerance for failure. For example, if 5 servers are Jetstream enabled You would want two servers is one “zone”, two servers in another, and the remaining server in a third. This means you can lose any one “zone” at any time and continue operating. + +### Mixing Jetstream enabled servers with standard NATS servers + +This is possible, and even recommended in some cases. By mixing server types you can dedicate certain machines optimized for storage for Jetstream and others optimized solely for compute for standard NATS servers, reducing operational expense. With the right configuration, the standard servers would handle non-persistent NATS traffic and the Jetstream enabled servers would handle Jetstream traffic. + +## Configuration + +To configure Jetstream clusters, just configure clusters as you normally would by specifying a cluster block in the configuration. Any Jetstream enabled servers in the list of clusters will automatically chatter and set themselves up. Unlike core NATS clustering though, each Jetstream node must specify a server name and cluster name. + +Below are explicity listed server configuration for a three node cluster across three machinges, `host_a`, `host_b`, and `host_c`. + +### Server 1 (host_a) + +```text +server_name=server1 +listen=4222 + +jetstream { + store_dir=/nats/storage +} + +cluster { + name: c1 + listen: localhost:6222 + routes: [ + nats-route://host_b:6222 + nats-route://host_c:6222 + ] +} +``` + +### Server 2 (host_b) + +```text +server_name=server2 +listen=4222 + +jetstream { + store_dir=/nats/storage +} + +cluster { + name: c1 + listen: localhost:6222 + routes: [ + nats-route://host_a:6222 + nats-route://host_c:6222 + ] +} +``` + +### Server 3 (host_c) + +```text +server_name=server3 +listen=4222 + +jetstream { + store_dir=/nats/storage +} + +cluster { + name: c1 + listen: localhost:6222 + routes: [ + nats-route://host_a:6222 + nats-route://host_b:6222 + ] +} +``` + +Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use Jetstream. \ No newline at end of file From 34db7d248bdaaf1d163e3860c4e3af2dd906fef8 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Fri, 5 Feb 2021 10:43:22 -0700 Subject: [PATCH 08/84] clustering updates Signed-off-by: Colin Sullivan --- jetstream/clustering/clustering.md | 124 ++++++++++++++++++++++++++--- 1 file changed, 112 insertions(+), 12 deletions(-) diff --git a/jetstream/clustering/clustering.md b/jetstream/clustering/clustering.md index 1cb9fce..6e6ed08 100644 --- a/jetstream/clustering/clustering.md +++ b/jetstream/clustering/clustering.md @@ -16,6 +16,20 @@ The raft algorithm has a few requirements: ### The Quorum In order to ensure data consistency across complete restarts, a quorum of servers is required. A quorum is ½ cluster size + 1. This is the minimum number of nodes to ensure at least one node has the most recent data and state after a catastrophic failure. So for a cluster size of 3, you’ll need at least two Jetstream enabled NATS servers available to store new messages. For a cluster size of 5, you’ll need at least 3 NATS servers, and so forth. +### RAFT Groups + +**Meta Group** - all servers join the Meta Group and the JetStream API is managed by this group. A leader is elected and this owns the API and takes care of server placement. + +![](images/meta-group.png) FIXME + +**Stream Group** - each Stream creates a RAFT group, this group synchronizes state and data between its members. The elected leader handles ACKs and so forth, if there is no leader the stream will not accept messages. + +![](images/stream-groups.png) FIXME + +**Consumer Group** - each Consumer creates a RAFT group, this group synchronizes consumer state between its members. The group will live on the machines where the Stream Group is and handle consumption ACKs etc. Each Consumer will have its own group. + +![](images/consumer-groups.png) FIXME + ### Cluster Size Generally we recommend 3 or 5 Jetstream enabled servers in a NATS cluster. This balances scalability with a tolerance for failure. For example, if 5 servers are Jetstream enabled You would want two servers is one “zone”, two servers in another, and the remaining server in a third. This means you can lose any one “zone” at any time and continue operating. @@ -25,14 +39,14 @@ This is possible, and even recommended in some cases. By mixing server types yo ## Configuration -To configure Jetstream clusters, just configure clusters as you normally would by specifying a cluster block in the configuration. Any Jetstream enabled servers in the list of clusters will automatically chatter and set themselves up. Unlike core NATS clustering though, each Jetstream node must specify a server name and cluster name. +To configure Jetstream clusters, just configure clusters as you normally would by specifying a cluster block in the configuration. Any Jetstream enabled servers in the list of clusters will automatically chatter and set themselves up. Unlike core NATS clustering though, each Jetstream node **must specify** a server name and cluster name. -Below are explicity listed server configuration for a three node cluster across three machinges, `host_a`, `host_b`, and `host_c`. +Below are explicity listed server configuration for a three node cluster across three machines, `n1-c1`, `n2-c1`, and `n3-c1`. ### Server 1 (host_a) -```text -server_name=server1 +```nohighlight +server_name=n1-c1 listen=4222 jetstream { @@ -40,7 +54,7 @@ jetstream { } cluster { - name: c1 + name: C1 listen: localhost:6222 routes: [ nats-route://host_b:6222 @@ -51,8 +65,8 @@ cluster { ### Server 2 (host_b) -```text -server_name=server2 +```nohighlight +server_name=n2-c1 listen=4222 jetstream { @@ -60,7 +74,7 @@ jetstream { } cluster { - name: c1 + name: C1 listen: localhost:6222 routes: [ nats-route://host_a:6222 @@ -71,8 +85,8 @@ cluster { ### Server 3 (host_c) -```text -server_name=server3 +```nohighlight +server_name=n3-c1 listen=4222 jetstream { @@ -80,7 +94,7 @@ jetstream { } cluster { - name: c1 + name: C1 listen: localhost:6222 routes: [ nats-route://host_a:6222 @@ -89,4 +103,90 @@ cluster { } ``` -Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use Jetstream. \ No newline at end of file +Add nodes as necessary. Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use Jetstream. + +## Administration + +Once a JetStream cluster is operating interactions with the CLI and with `nats` CLI is the same as before. For these examples, lets assume we have a 5 server cluster, n1-n5 in a cluster named C1. + +### Creating clustered streams + +When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. + +```nohighlight +$ nats str add ORDERS --replicas 3 +.... +Information for Stream ORDERS_4 created 2021-02-05T12:07:34+01:00 +.... +Configuration: +.... + Replicas: 3 + +Cluster Information: + + Name: C1 + Leader: n1-c1 + Replica: n4-c1, current, seen 0.07s ago + Replica: n3-c1, current, seen 0.07s ago + +``` + +Above you can see that the cluster information will be reported in all cases where Stream info is shown such as after add or using `nats stream info`. + +Here we have a stream in the NATS cluster `C1`, its current leader is a node `n1-c1` and it has 2 followers - `n4-c1` and `n3-c1`. + +The `current` indicates that followers are up to date and have all the messages, here both cluster peers were seen very recently. + +The replica count cannot be edited once configured. + +### Forcing leader election + +Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time. + +Moving leadership away from a node does not remove it from the cluster and does not prevent it from becoming a leader again, this is merely a triggered leader election. + +```nohighlight +$ nats stream cluster step-down ORDERS +14:32:17 Requesting leader step down of "n1-c1" in a 3 peer RAFT group +14:32:18 New leader elected "n4-c1" + +Information for Stream ORDERS created 2021-02-05T12:07:34+01:00 +... +Cluster Information: + + Name: c1 + Leader: n4-c1 + Replica: n1-c1, current, seen 0.12s ago + Replica: n3-c1, current, seen 0.12s ago +``` + +### Evicting a peer + +Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down. + +There might be a case though where you know a machine will never return, and you want to signal to JetStream that the machine will not return. This will remove it from the Stream in question and all it's Consumers. + +After the node is removed the cluster will notice that the replica count is not honored anymore and will immediately pick a new node and start replicating data to it. The new node will be selected using the same placement rules as the existing stream. + +```nohighlight +$ nats s cluster peer-remove ORDERS +? Select a Peer n4-c1 +14:38:50 Removing peer "n4-c1" +14:38:50 Requested removal of peer "n4-c1" +``` + +At this point the stream and all consumers will have removed `n4-c1` from the group, they will all start new peer selection and data replication. + +```nohighlight +$ nats stream info ORDERS +.... +Cluster Information: + + Name: c1 + Leader: n3-c1 + Replica: n1-c1, current, seen 0.02s ago + Replica: n2-c1, outdated, seen 0.42s ago +``` + +We can see a new replica was picked, the stream is back to replication level of 3 and `n4-c1` is not active any more in this Stream or any of its Consumers. + From 70a9b61e4ec75a1b51ec908956f0a4b5ec4b88cf Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Fri, 5 Feb 2021 13:13:30 -0700 Subject: [PATCH 09/84] clustering updates and pictures Signed-off-by: Colin Sullivan --- assets/images/consumer-groups.png | Bin 0 -> 16607 bytes assets/images/meta-group.png | Bin 0 -> 19301 bytes assets/images/stream-groups.png | Bin 0 -> 19445 bytes assets/images/streams-and-consumers-75p.png | Bin 0 -> 53849 bytes jetstream/clustering/administration.md | 85 ++++++++++++++++++++ 5 files changed, 85 insertions(+) create mode 100644 assets/images/consumer-groups.png create mode 100644 assets/images/meta-group.png create mode 100644 assets/images/stream-groups.png create mode 100644 assets/images/streams-and-consumers-75p.png create mode 100644 jetstream/clustering/administration.md diff --git a/assets/images/consumer-groups.png b/assets/images/consumer-groups.png new file mode 100644 index 0000000000000000000000000000000000000000..d3691663bd3ce8e4173c5649e95b2f11f7543cea GIT binary patch literal 16607 zcmch9cT|&2)Mr3J0Z~ATqJSbOAiY~F-9PpmHOVvc%-lP3?{97!qu^=^q{Otu5D0`+Nl{i40>RS; ze@!kEf+uImCW;UUp{9qNo`<8iwH?X|!ptXg{*9TJ+t%6LgPBj3nU~kh#f8(-&dkQm z%+a0G$;tyf0_~2@mUh;5R+i^|c)59bIJkK^xcLw~0?d5U+ezc~*=Pux>manX z9gt=!Fb{P-AAOhzpPsC@fR>gZuNe2sLRwuzO%G+q?e2*76&6u3*LPEJw6RdsQBYIkQI^#Nr0Uu$fq%gS zLQ1@RP#0@gb-0qQGQW$vlNQp(ms?MPS4hxF#1(WvTf&4b_@LmMxf{yW+1moGFDRtL zgVOSr)rUISXt_FQxblj)q210%EQjaMps?dQ_aKH2DHoSA<-U!R>C&As`CHR9{mKrR6E4rr~Yp*OXcXoF5baLWR2NDLg7BrW&kww6it%POO^vtCdoy?^bEbVNC zK%0-Shm(%9GWez`$iu7VB zb8jIYOEX2dCEQa(4rZh7T7G^f)X$j zG<90<8&g<*t1{bkbwiG}Mx@+;Oddv9oxPVb~JdnJa9&k51 zc^z{-GhKZiM|&PGn6$T!9*?b~mztWd8lMNZo2#aehOCmdf|nr5LfKrzi`N{jiga{? z+VOip&3)vY7479b^jzF5!BpN>9(Qq9Ia{B)PZnj%XREE@>}w&X?G3e3g*ijjoZV5ft{M&+X0|}Q#E7fD4J>FD7JM$J17DKgsQ7qAnYwg6dlYI+>jn3 z{2KbI2!yq>tAdb*m7bQYGr~gw>Fy?{q+zQ8pj2MpUdvHI)?G3)TBz$;3V5JUPzPC6@PJ3yli%H0-;!TT$4baS zP#z|aTSy%hdqISyn!Ki~4^&=A&_>(SOkZ2q z+gSj-V=ZH;r!M3rgy6T9bwIhh2RQfb7PjV^`sz3Vg(@rQ$)G^1 zf*V}G!HLHT*V9uK%I7I83!oMJ_z!gb3$@_)KgS23944M)4g$FYQIeHLc&Bbm`N0vo ztz28BU)iu^{@hwNUGqN@3tnq-Et=H*F^<*tZ9a(i!wQc&SO}R8JKoVC)S!jmPK;uC zM8Vf}ksvcF?t0X-Uqd^Qu9Qg&!Cf?5(qy+fUtM~}-J3#2`Nt1?$8S=7Y^tT{VC|q_ z0hi{o1zgfl#KBKWnm~;`#oZ9=;c}RFt)p)TanNt&@7t? zyL?wX3(g!$Mz4`i0lgg0f*`sZsz6yNOG$g{mYHBmh0PawS$TOzHa21s8qSVppItIa z%FvM_-Fqx7mv?{le{2sUA*Z4W+glsG$H75rU3F7cRTbJs0ENKwDIV*U=vZ1>nr=+g zT)uiW{DWz*THeRjfD^c$USw|W{py;U%%UO^85tQ!aAxLR4>UTnq=Ym!HkO5j<^6Pv zU*_;i#^@T4!*}-l5qnE(>vsDZ;?t{9ni}~D!7CdZokQ6Q6_z~|wlzqm`}eb&n{S7N zgf#yMGBt%*Sy^$@#tz!@>ot0?^78QwjgEHrVTfj*K$8xQl$A-$&CR*Jej73|GiR2T za^5L3Ffbs=9*Puk|8?j7eZrEG5IX(CyL&0vBSz1|{b;r>`g+6lCI!=bq zcgK@VU18J!Y+LB->yK4h#SHdu71h@ZFDxuv1})N{dLSSmpjTs`G}uqp5l!`QaB%Pr=qw95t62}EvL1Ek zHjm8A#E)0m;ASPKEVP6*NF$TJ{|aB1#!7v&Su!n6OUs$2vt+|2$26`I4I5xLI%6I&vaQE@rty zXO1F_&6VQ%^$RhbfL+;=j0{MMpd$t79|}fEo1QlQoF=BbY^?}&ap89EcDR0&0cZA% zqnC(&_l_wujq z)6;|fVUo)p%4lgZTDB%52U1Y{^yw8K;>1%|l*TKqNGhwU!kh@n$%(bwkxiZ)y?uRQ zPT}uTMcfH#W99#Oh5cA?()~~MMOUEW5D3tz!8ITND*ui0BK-APD{)7?*sr!21iDi z0kZ|!*%x+~zToze2?!444uBQ_3b>aDFI^(PefvMDS{*40b9(Y_D4oF!MngXBPYL$+bsYU$+qK#e%0(r zDgdwQYXG>NBZ7vA=lTr_3JM(MJU}Z8?1ppA{QV7q?zAs2vsizAg&W`#z-y(&#l^d< ztV*B-3Wef-wywghf9qs7+^dqp^o6)1}5?W4m|+G={30J{DU6LaM|{o zVQjPPl9DE_@%nP|@?TeZ=fG;sJ^%B^12tKzx@>(hbjViL+Bz4EOab&v-1GF%wWG5W zC|6r_Y;0#w;sYST&+mYDl2TJ|Gd*|!5TXt20A+rM1HcIfg}B?#jba6Oc@rM(Tu%pU zp1!cOWYHN-wY<4W)5!V=Z0m=jvR0T0j65BzNxA?(zv=$^I12{{RU@l$g&zB5BBIRh z+{;{meCPJ|c2j;@9zntT9$LaT>FDGD*Cj6)o53 zPu&|V#wp{YNAwyR8s&QIP@zsW4GnUju8uPbr(oF+Ro&hBs*xPj9LUfoPw=HD6Oxl7 z2uP^!Ffu+7e04t%$o=VneU15Wt|~ZdyS|qgxPs$3xcPVNEg>=nhWLz(I~`qJUteo7 z-C;o}y}xqxD&xb4Bz-SALgceQyyP%&`StncpZ)zEH|*QD^m^>n0737uvtQ{;=8EXC z%(8f=Bn5buuCA-QCnk1_HkQzRX&_83PsZC@*mH9_7U00Qk&(=r8Y((Ex*NA|U*sn5 z1>bBP9pwT7BzAUop1pY!y|v}C>>(N&8Y)#Z(r(E+3l1jyHkm90&e{N=1UEE%ia9xu zDH{cFeFq$}0234dr0SFz$Nv0j`Z-;aAu}^`tlaFv6~YTwZwm*{&6yV_QVapK{h!R2 z|4VD<|5Nwpme87t3UXj1`R%VuTUs(@Wo4DS%tAk>2wjtwmey~ zAn3IPN&q-A6-*iW_KoR}!5!I(7yN;8w+g{|b@S#;J%IjCLPE^AZ)6Ow;LsXaCs_pr zL;!J@S6ABy2gx9&wzijzjErm@9Aqpk?hz9cn4FBsR){z6NnjqduTfM{k@fKr0YL22fPMcS(%ah$1Uqxcw%lVy6BuP#Wo7Zn zC)cwq`2!zBG9nC+NV4nKLmJ$d7=a-JjyXJ^sfh{XjaYeL#a!ay0iKp4>8CXc#s~a? z8$`5)b_+QkEWGM%Wl6An189#p+6%Wr$C z#*vZPXx&c>^#0Nga{%w7M{(eI28Jat!y7gzoz$HKo1vh`7#L_5S!vByvD?1x= zM1hz0A2!$JZ86%m>1~62>vOZ~tg$8IKg}-aX?R!J!6`-IKbDuI69=uUtLvwp9`eSU zp<7^XXYXqxA0w{y?K=xA^z`%yyDi{Om{1}9z<_QC9Q(bcgnU#j#_A@<Y0O9Y_XK(jy9`1=C+}+)oYdreX0rKa!!kEIkHkJKj zP10qwb0|7r7H4q_3Wl!?=K*wu3TJPhP?*e*=1C5Lqi1MS=eq6vM568OZE$uyV;wp~ zcXc1`i6WeVksb2k=WtA45`YwUciiF7!9m;h`$#mf<*IfKlB1E-zO&uD0ssdvf}lUL zw%6X4&5str9qsHs0NH5?)L_=Dwtb$j?$6XKH}7TN`qZQhDgbPG>X^d* zIDW|HLc2D<`QL1#kFa-k=K3{|wl)&Os`}S*+ofo~MBDbmM=Vg{>Il#PAi+KJG@4o9 zSOi$93|?-}+U`rMn;)Hr8&z5=y&u?n*_a~aYz~Yvzz`XAtjX_Yx;FZ-!V!BpZ}0kU zt}IJ{lp7lxxK`RW<1On`U&=zDxIn4?EEAw&Nv~xeIFU@mkc=0B&o0@@=O33p@p;sG zW$&Bw`JrpQEIbxoj~@IuH~M@~%y*}^(Wh+Bh2}@I;YR1SpM)mzr`uC?ExDRb-*-6aFo9b`D{e}t(Z!>=&>}cNReTBgj zT#1r*OS75gDb#Y3#xN&tS~&`Qy2rIikpvR4b0W@NTuX3A6e`rXY{`HAgM!KYM4Tf^ z56SnH#~<_Xso*yeqmQF)PXz9zVa6TAYR$HWM(JC~)g^wf%MC>=jh8p4deL2@ZSAh} z3UXTaa6ck-u#~` zi;o{s89FWGu@7hl8m?${ul0nvuYcYHBm|~Z#KnnQT-ZY(iGCZlZ1Sqmm5J$YR`HEy zUHjtxwgTui@bt#GHp9p`#c#HCyv>_WcF+GkUUlhYe*=tMx4&+FVPs_ePgAe*;aN;|5c30w8L6Mqly^;?&y_xDLkPgiGXiOi^vGnxB5yxk{) zK#S77dGjXsRK)tFeZOKINpJhL=?GX(`;d^=-1dmQN1%RGh*H$pQP&!$O|eRB&FrA` zfO9yB^-Euo%|;`5pvRw0cmZOpZSBtQ@8VMPHBH`PRzkI1_Zw3ycm=-iGU%!wy#Zif z-A_wSIl8T!`j@>`5E~f!{#|h8%Tv;8ReUPl@fZ?gAA$;Gtd{rwO#QFgEAM~AlufJx zk@pN=Ijq25Ys22V^1MtW%*Np3y*ew%FJr9{9pIY-zoEC!?@q@p!!9n@{=W<@^`}IX zmX<07g&!@O?8#RfN$ndp5huaI+{g7(Tg%Nkb8CBbiGq)ogV;(sIPYwZJPL*7K&PYd z)lBqwXtWH{3NH>04Lv^EUE1BDzr0PDysd1w>v8IKbp#knS4c@!kNK`e69a|H{=Jdr zIqQ8;#hZ^{`$qXjf_S8{&MOOA)FrfZD6&$KJ9kxu)%t~{VI@iTD#>W3W`RrQyRX@B zj-L99TG2OEM-n%igoTA8fdykcec|r(Pu{>NhqrMi1DTD!d#kKL;Y8ZqzHEG25-CY5 z8u`5FkxOy}sWXF?ui)t7JYK#L^|%xXV(r{gq?W5q@93|cC56Z~ICeMoK~Nyx>jGYZ zksEu+JZ#IC{wKjw1Blme?BCt*ZOBDp1%O|)Fr+f#7noAui_n_7F;Dp{Z^jQ7ieOk* zl=ebXwYK<^Z!*zN1)gkvd1LQ{EYOmo7fZbW*08#6Kl{k~)@tGA&l1O`AAu+80@{qs z2(7)hGr-FVJi)pLaKz`b-CEat+>OCqVyy%2Z;IHRv@GlWdtg|4mR^$BPsx@gu&dV>3_d5)LH0~tK>}wGO>sMKu znVJUgeu)PPn*;G&0_&9TchYa-c@~tI_L_IJM52jXQ2c2eSKF&Uhg6T+zmKT!36)Uh zyiV459drSFjK5}W+6}7R9l)xNe2)rhn|ivwVQuLgbTdDhxOCPe=tL}jLM~F$TQrbp zaCkW6*)y{E!0v3gcF_n3pzSUbKz=PRGr|={dJVi=8;biW+LCSPX41 z($Lq%3;_&$MD=rL!b$%1%Feql*}PTLsEC_u3eG9yb$36k)snZd4}>mx?b?V$n7Kc>dnSTQ6uwKiJ=D)OJl7wA_97elKm3XXlWoY{PZoup+S1)QXzs zg^jLgfXA};*1er;Z*vkLb?)eFb+iAO~5 z*)J~_m$qB|swlFR2y8|t)(MWAJLF6@Tw0?c5Q4uoT5+?<*sfU0FKdpu?o6ej6nyda z%)IVE^VE3XJe(kS)FddaPNoBuT%$%^{24Xk={%Aic!)RZI zT*Di$JT4xz%Bcqolv(G^z-VA>)*axeBu<~#E#2AOeL&ZixXp$*`hkub4;h)7*HkI@ zoN0_AE}2-FJfpF&D1SR(6?A4_NiH)J(5E~ZoNV&QDsSx4D&j-qd5k2e_e2X2NW1Yy zP@r^vl&TC$1pz*G1n@P25$f5^ck95_6q>3CUzce7Z_nS zHu9qGADCA0Hn@7;IvdW&N(k!n11_mStoUQjVR#E=`^o9czrQm1k~llVIlfj%T7$H%@PI-ij2 zx2=y~z6a#@hs5^tDT+}b*B|CjgkM5*iuK3`Wk$eThh0mDNx=T)a?T`HDTe->=OGPx z0YPEB*Y2}r@8u5`x{jCAr;TXSV_F@Fexfn*A9V8S*@&*}VVM*SBL8d)1m-Y-P}aeA zpF=*rl&`NS@HKz!2Apc(mkR*hcFH{qOREvIdS`*M4Bg!f(%^P&?yU$2+EqTOcVj0? z^)9-+m*iEzB-PwLdYFU{k|Uo#t4we9`GGy&p6@27p;;Q1IMoJR%pJK`)6irwn$xjW z+asJ~WrF+w8mUGF^=)WDpg{a)Nf_pJ*FkXJb!S1T?Z+fnCNV@rL6; zlfKDDbG|&9AqqBqk7>AmoHY;GcJ#ERrMle>Cc~}8&$UA;zU-mpIw^wF8-);;g-_(^ z_WWm@cw=scbI2yi{zZz>PoEU`?_JQscc~$hQbbVP@!vw9MnB&;+@om=XXDm5HLfrx zI6gU%IGd7s0a;h6?5N4hyHwVCTxg_mGU;cO#fB%Gl>pgJ;+of0F^n|{+6Jsj$;e!= z{_-~V>sQVAK(1%Z5CfbLPt$gDx=J*8T?@n%nTSYuZl4~JXuO2n1_*Z8KDqsnnmUHy z?%lh1e(OAUGJ7wRkc0qsUR}C2b*b{AsRPRLV%@N0xxw7=ZsG$q$p_ylg6{#*H1XjJ z4C5CSjRx_tUcr(PUxJ=-@PtNgY#$ z#DN9!pItemBIXM`nvMyH365a9_ir3A@yswo86qbq2g3ZGlR>ekhZjmWRa|{gF$mcC zs~^y(gl~$rZ`nQxV}tMu3CR?Vm=9&ifgGI6iTh&~yoJR@vky-QA|oSR8l*R3DtMZh zOHG(>@%+JfLEeGomIWF?QaUe0mP_0An8`Muw^-+)pyNa+FnCc@jZ{L;6X{P^%P8Le zcX3EJ(Zt-QU(`(pryxHqDe(VeEGD9I-aWup6M>{zeo>K(gTq5nQBiJT z;mf9TlJ*J;mq6x11EdasLHRB>mr-2&wrJW_u)iQU#{F8w`*15$eZw7yAvs-0B9VT= z!@qJg8!~@5vf^?1+O_WAQ%!O_ff%Lg%+>tX<|_aXd;dHKSqYFdyi7{kd3v-9QXEfl zDOF+N8@QOKtu2cnY77$cA^5_js~Nxw9JQ9Tr}?VQAk}6~hM-j0T?3qZEezhp!SH2SnQj#(aN!9b{?;Vml#?s(uKi>c zRfJRAhD208{tgS?813B6)EPck5S|8k>(Bbjt<2%XcbUP4KqbTKkjQ51o9lmtOFh}K%2E~R4zVY2{U zK7}pFXvYBLy;>Ne9-H=b3gs zTgye$KG!H<{Tx^}b&QaM*PmHZ6+*zGusTK+;g0jnPn?KOWBg^*=2bOT|+R#Zq=>|*=0M?;S2^nBu2tgQI66Eyb z_SnnZr}v^oYCEaGo^*!d9xC*lFW91{NM>3j3I~WFmW?C5h$W93MXW<2`-nlJW z+bO|D^eKcM+sVdc`Jd- znIjOC)SQNqKD)o}f(&<&ZkaqBP6o{Jh%gcwT!GEnXqf^MN&DHF55(%$f7=+~my>=2 z*7xpLFJ8I?Qf5+l!)7zB0W9q7vRYc# zCnhF9dOZx()j-~ZL{MX6S2ZOSR+Y-K%$}ibLafO_@IU`81^9TzsyBmNw#oPiX ziJ0E2hki{M$-eCv5+7C~kX+9wF1{k4 zO%7t}xNNw!_5I`HV{hL}y&22!MI^qhot?Cb$`#y^4l+9+9eI=^@%zb0H7*2^4jgQG ztr}iyG7nUr^6z64?7Vbd!ux}RBR^P1Z7nqz)&19J-0MgKJ1ik5C0*hB1?FH-IwDh| zW3fHg$->EbRnZR>BH(wD_et%@!@Tym?k!rqb0+Xx_bHMs})P0=KyIA^i6a?*W zF8!BEAuSc`?#nru5hJ2I|K_pqoD(g|CkBu)k`fodLEoPubg9`0x&qXU(^b2-g5&Le z8*q*5(V0XH8|({zUPyt>073jY*CtS5L^+BT&pCvn+3^3_u11E6=gR6jUYp1{KfGk5YPf4l#P)!mJ+NFbUTx{nH{hB4Ez0# zh=~63ukzat#clzq(GMHAT#c&2+s=fvl`@t?q7?q@OdoyD<_#{{-gX6*Q_>|myI-vv zGT;R8z3C?4kZi|SPl2>`&y2g9*>M$S9s$JuBS`4-SnqcB11Z5}9#cipu#A_41BqRe z>n5RYdUHrEt&rDzHgoIaRn600l?lO)N-Oq0E$U@jIy&JX9F^HRU3c;+Y)(c1lu@l= zN3oO}#*l!3fWxvNy{Gr080L?c>Ri3CdU-(&?`EY!oPvN^|nS-1hFoU5m zy3=unFXxr(>^CZq`gvG{ZO9yh_xJX;$I*$61+^Y*D-CTBaEtC-S>{!$3ghLRMIfW5+PH+Y8LcMMsrW1i-T~9z0!tDK@T?w*w5ck$oO6}9Z%Ie<5=4#T z7-a-N*59CZEa=;&LsfL)#Vrss2%6FT?D-w!SGF@|{rvq7F6FIBe>|v9^};ODAQ|2& zXr6lF>OC-m`0_9S>#IR$-^Zt8o4p>Tc+~p1C5KJnZMfQSHO*Yyll;QMi2wnsb+_<4 zxfH=0glr&FPkbAEn$e(yPspU;XV4&C8QhmoBIKX5pI?#n@OV*< z%^*Tx`0VTfO*GNSn>Vz@$Qt{v>Etp*8C5MQ8$aUsLO&m<9`2O|k|x0jKvdkP4WpaN zyC{NQ|C?tKoouQ&PpZ}j^n=yTZy9@<*u-9WSzF3q*|6b{WKg~CF4+SweYjtRPw$e3 z*ksd?M`!PMy$d7Q1$mxKQ2;os3B~~+0Y#<($+LLTZ48SFweXIYdf1-fw z4q<^H8eMA9Js|@b2A>DS!U1OQ4OJXt^k>puJ$o82$=&qB7#>%JFo4oc&e>siX{>8S z_3MwDu*^(MELp8=CvfcoalaVoONc$&@Am+$mAaUl2J{1FMG3{{u5#>Q^ss;AQr@-Sc2kFZCfiglYXm$mUtp(7Ptk*D@Jsx;pI1rWUJ{_(m2n-@`yMT^ek174y@wkRW$M z{Gg5UM2_G*%Z*2;`}qDOyd~g7@CspwZreiy(fcWtqyjrR`>a$4j3OWHkW=v4`oGuekHt68srvNcaq-U657_Was4bn^fO}PWj?Q2W8afE(PB4X47R&lee>w)Icq=bKf8 z=jr~)Hrn{4EvP-JjdHwSZ{c_Yi#Rl%>s7$cCl)zl9Y4Mx&p*2aiJ|65*oZ2ryfX6Y z)eTU61V%=H?A@zhdZU#sAP<#MdVDp{_dC!4qgFL(X}me}`HfzDSXv2hV<`fN{$XWD zQv6x>gUKa?QCLBV)P9?4ir~>22MNscZ#@gBM>+Oq$W(^pQZ7+(7;<$}ar_2WlZA`R zGY*nZc5+E|x8off{O0b72OYoI-P|3)9<~?fzhP6k3H>A4@Z2b$9@sLVh`srTa(*7< zl{UxU91)?L*S|TUW^vZ{Y3G#!F&LYMV$4AjS9k5!IQGz`>{n)Ycg|&?_E68vMo_W5 zZ|Gs7s)K*y7JZjuomWJzsui#pn#SdvkgQ8LWn>m5YGr5wTmpMPcF0zd9SPl^`clb@ zB|NE;V@O@{5GRCyqXoxz;wcm;?5f#h8c?C#rQsKC8@7owt~nx}^= zF228LANC#lKoOTFmyG7o1ysTKxIk8k5PVeeUHHZJrS#A8{4|KmJWEU4==QoB5T|Td zG$+_-IJX*q_JCR>y%iMz~t@;(f`|Z`5XB!{@<%^x3?9*?^0r z`>!kfw6P3G4oTpzYEadYgWw2=KjXsCTXO^_o0uVNNAPjF4ep!B|zl_cTxL{&`pOZShd*5l`y8-8tr@nQBB=33{XPN%H+ z=I;Fv;Ph@+w=o2toQ)1M#*fH+!`44n%@-6DgqDq-_@{$Hh3uUg7SF-*vaI|IfgD8xzFnaUwHjEd7a=Sf z8$Q`fac|#<@2LZoZc3?M z8-bo53SF;iH~7E15Ioi7Wr01zfU3-d)Og47xb#48A5>pR){`iV!9149^kJu}6GmMy zllJD#sP=ZK@#YGzUQom7k+62g;?mNtrl>}xq=K#MUCloF$*=knuBbkc#T$Dx43cU1 zb(*OQ-?5h03qeQ=6z(_Aj2e~qDj6B2jZQKp5P^q#*rW9NR`(_l@mT@U`;IfdejTH8 zVmZ%idjlKlEMf(o#cL5jLF5c7BZDhO9^Y>}N&nSQ(mjkCdHv%!9R)>CH~WC1w4F(e zV~BQScfsj@R{bXY8pDF*7%@%IIAl@~VOP7Va?TLAX#o6}wpPOH#g0dLLw@VzA!G|JnX zQN>ZzXXMsUzMtQN;+reDkHYE5+LHqTyMlz*W}h5$$!tNiO>f^BOW?>%{o~}2$!TMJ z@mRzrrD(p7%W|fyKH=})9v9_`9xgk0wgiRgkUGeVhL;6}4TX#5IUFAAmRrATJB2!Q z*9J4VZ;&>OREPd7#Nw)h&NG#3l(k8ObgdIFS=ItOlm0H}2*b?Tq*I3Go;?3sL*b>$ zVKHNM&QuGV8>616>zL0_aBqS5L*khbNH`Z>7wl^VT-D*Y8u?ody|DN8bI2qp?P*Gt z^VjtunhA0p-&lWg6Y{ncLw}3@>TwKMON(WtRV{69ZogSe8lj^8cn( zg<6H97?i5im08`&1<(P~#XI|t>J?R0J0D-Tgn#_TO|wRX8Z->Z*$x^T3a@nUNbNr? zc=mTcb{dpNK+Tibb(d4`+Mh|fWJM#w=$*4$qdx)-Eq*TwkKO)=$QPJ%n`CAge3h2} zs%S@`rO$;F(Uw5_SIzB#lkJ?s%gr!^lZUlW5~HQK0z^Yk>O3y$S1|>Tc^F21gcXog zA*2FN^oeJ8x^CoQU&hU@ZU!=IyR8LP5@|zT2{=#_5)%5iUaK-Pn3@#Z5Z#{o(hDko z;99o(`MaZ_{t8gv%2`oQs`YDj>$r`+1L%FQk(ZmK{CWbGdT{i`yf$_;MD5yB<-_Og6su2U6!r4U^T90wGMXBRag`um6tPZiIcQxiG_meuPaaSk^%0aj zUH}`}A&zHyt5D^Y=m-Myt?Zp*97bISM2P>XidR2)wQewxzUJU6?kM!cX23H2fgy1J z4%hX<`{E==>aeBOrH_yIv0bsZ^jiIWq>PmZZnt6D^5d&XwB$36lMtV3usxiJBQ`w# z+iwnqm-MieEx7U+K+~s!-vT4?LUg8ua_zVME zUzosJ(i8%}^c9~?+?Rvbs6~>|z7#5Lm3l(+TG~hn+eIgHxsks3BdtBjyEsMqJ0IRp zP@BKBNG9bOyWe%APV90+6Al#$P2L*)`SUJVsu=U-%NGR{>SuaX(k<(awvvM5bT|g4 zvhV|Id9%=e!

zmji3hI^?Hl)nP?(=*HowW`dmlF0$Md@Apg_7r35iS9tbd37|&m zNjkgCzd7slO+FkJLAYzU2TqK0nPS{#`m(YZWo2bSg{Xc?A-1OiAjZMjgGu3&zcW%9n`qEB2m6gtuLZjTY9i&;)UG-o zqzEf7|9n?uO?#D&axeK3xNIZ?g+3YVhhHqks9(iIKV<$RsUN`xQArUbou8iv=Ewf4 zr~g@&5fezr12SSa`K$p|a=-;qsxfiRd2#mO1m54*C!?vE0Immwg@#^y)eo*)WwLz3 z8f(I^z%ju;b*yHza{EyaWy{C17GYT#5qM{{1TPpxoXYMCe8avN#!_C#6&C%MD+b<@ zl+V4xkJH_b)6Jn&uk<=Uj9TbulHa`9(+92^h^+?oX5@pag#Xob1HpW3Px>6F2#8~l zj05+jGE-)C!L2Z1pPhH0l9dyVO%wuQL|W4wD;n&+YR zLfmn>I6PnU)4vxH2L8E-AXl8DL$Mn4RUva#9^}?QeY$CdUZ_)a8W0XpH9qc~p6l@q z`z{8Grs(hQ2N^I>bxRJa$}c)98ynM8v+D&%Mqbu;-cmNsI(wi>S$G@V00f1%L?1qU z7;8j71oxe8J{xUuohJlWUdX}m4hl8K->%5qSHP$vSr8H+=@waB%nnT6DcEi zVnc(IlYM;{KVM%!3hpLDc{w+(u;qTD(#!5ciZ2@2L_%V1uA2Yjn+ZHDAiXy@F%b>2 z@gTW*S3rPjc6Junc;Nm9h*gI5ShfSuAqb3+{hW`Dq6uSD|9`!lphcbz$6mkoe{t>d gf4GRSjJ@zIf_kT2F-8jf#|022IW^f*sOgjc23u=O-2eap literal 0 HcmV?d00001 diff --git a/assets/images/meta-group.png b/assets/images/meta-group.png new file mode 100644 index 0000000000000000000000000000000000000000..1e802be26efb0851026ee1112c9d2ee17c73bf89 GIT binary patch literal 19301 zcmbrm1yq#X`!A_4*e0@7WA)F@JdG((4gfHX+w ze0%i$z32O%^PRJPYn`plH;O*jX>)>pI!NM>1_a_!U9(y-0Zx()e7Ct@;cXuvp2Maq- z3l}dgR~v6|3EX#avv#m`u(AHzhL4Ajmy?H&lZPL|E6Bnx!@~>y5#ZzE;T1Ca+up*? z#`T{L1-W=Y2M;tXtQ=gu-Ms7{{%zrFyk%KRE_-Cv-TKXPZo{BC| zMP)l{SzB2(PbH@454li z4Fv%wMLu3%VL@3xXFgp^du1af10H`ZsFPNJt+lADs=oqwX+;?yK36_1@VFKFlasrE zijjzXfVHEhfVQ88jF20+2yW^*@CyejsmeR>!C?Bj{sR8ah8C`F8sMg%y`8rqKaZxi zsxnl|!OdOAU*Ai}$;iOd+g45=0=0DH(-QPFu(k~F1hXW|E5mE+U}R~}XQOWq=10-Z zQAAbG7N+ZB=WffVE~f~_APX+L=vithI;;C@Yik)fz*GYR`2;D)jT|9 zA>P7HdWLT9wvJA2yu#X=iWU$#KbVE6zN?IeoszG%vLCNSpk|I)lFB!Ud4mo$IHjaPS#IeQC3OKNkEHV-9pYMP+3sX%05sj&|1UO&r@4f*Fnfo zTSmaqj?YR<-cX0vQ^{W)qGF?^i)jO=0Rioyo!a?V=*GQu7j3R)JP z8uE^wGV-!6a?1SHc7lq$)+#DO!geCo_WqVW0agM2wlbpDj;eN`3kOYCZw+60K6@8+ z4Ivfq-I{j%wq8P_+CF*~;FGqFpobctC!B^l%CAc2c%c zc9B(aa5c2G77*~TR#5i$cTw=yv=Q(Nuny3)HB_|JxAU~I;ZyMN0Gk9o`4BrdKNWs$ zJ5_#5H*duNMFT&5cW_xjLq}LcL0DK#2fdI40_F9*WWD+P0$^HvD)L%hHeN4 z5CIuJZ6&CHx4fUVuA+jMuc|0|b=&(}*o&&E^2r!@_*?3^2%@bcWU1;J0D86%wBR%H zRns@pbr$6ZpY{C(6=1r`dOmJ`F8<0ohF%t$E&{#|JdRp2wmOFH_F%I8EYTao#zWpK zKs6BjD9fuPU=?8RB=4=F6QHYQ1%;|;@+is(dVp58x>hjhBm} zGcS4w`VKzMf6UBl3x(N31vULWyYLMZo zP)j3Un1>Y<_&7yX5e=w_dLZb~2HkmpyNaltf`W^Uk+Zy}kAjwhuBW5DA-Lh}Z>TJ< z1Qxu7D%6P2&QOR)*cTe0i=IO(LsfNQTYYa}7Fiwl06k$DsIVNA#}}rc#Sc@E6;#mi zcaZhb(K3|b*S9mY1ii`H%3DDNb^U=qS9B2-6@qyRY6}Q?0ne%E%&(;F=%?t1cBtq- za^Tv3{OBL2ihktJkpX;cgRnyk1_mRBio6WO|J_zgh&H+1N0b^=H%S_cRC3jC-Dpm) zV0|?9A-fOUr}(wrHv`s(8*}p){D(Eqt`BkkjN=!kRiGY5oa>oA`YnD>niI zvq5HvnVOW&NMW+)JiYnex}y9tT^K&i<(#z8>^45l1o!Nv_pw=4arw7L`x1D^M&_CD zGealrr;IqyUd1ww-5|n}p}a)`ea47G#iYXmE)xIOA7n`(FxWj-)|*X!JDpQ?PF3*B zKn~-Nx7Ct4WZd0(^vg{mfBbmNVOWjTo6Pz3>~N#XbyjS1vgW3!sAyqPk;NrD5f%pc zHewA84P|xp_L~gBPhPzul2=gpV$EkU-x|{0+bgG`fuEM1K7>SuFDzJKeBRyVX>4p9 zoSBJPUw8buvSPV2*TR#k{xT+}y)7J%n3_83!v|&oK|$Ut{K!Y)@=4mUt*uhb%*;Q^ z>gQzr{Y5@HPd=-$8?dyuS5Dr$dGn@@fx*j$29dDv@P)1Vc|~1alC$%38>%cuiKpP< zcV}mT&J&f_qA56KgM%e{dV8U%^`)h^Jv==8x9Y|#Exz6m5D+Nj<0hVXRY(%01BHeU z4r(`_uB7v%_o=&l>+Sved%3sDwvT!$l!XjEoUmWN?7<6WS605Nd&u=pDuh}`M`xne zfz(M4=ecZ-YPYJQ;*Dp|p1E&M@Q&qwS1~f8HVa&1v9`6X77FK~WG)VKE;XvdHLiEH zaf#?s*VMc#B}Jc|on5NIbQ>#@Z9zCMFVCRz@wL|TZSk@E6L|{@CJHW-mr+r;WBEik zp2#QdM#se5Zbn_5Q&Le`xI~%XCLmaBIi8dA^%Vj`$dJoXJ^7w3_hNU$9FVdU)k@2jg{*HMLOGU~8MSblki-6$=z%#MkP>3@iXp~r+p0<|>L zFEh%UetzlcvGSd&)T{-JI*b53u)=)ro*b;GK()<-cB_J99~k$$iZ;=wmoH~syL!11 z-@(Y2wA+blNa*OwP960e6VuY7-oB*-!={dqPjXj-LdkP-atzY5Rn+g3uCK2fq|4Zo z0V8k%bEmQMl#&15-zTQ0M_V8lSb)El~GD6vyQ5f zQAmj5;lqdM2hb*KGj?kU|NYx@%=Xu_bYZu}48J)GmmAr<%8*-;`T6(d<>f1d0+nUlp)#ODh@p=0B~a zPJUp%lMG6(qq46$TAKJI=HAS+;j?K%XH5~%F6!|6?(bh^kD zwmW+l5-nKGpC}Adwx0-(FYX_Zr0lS!mAoxCS*yk3r6Yk}$Iv!4jRkXUyKl@uxb#>* zl{pzSe!ty{&pDgOU*32wWNq4bZ#Iahm7=`-XJ;Y))bv4m=3*?2$H&{V6ZLKk ziJwTnC%t~(|4}SstGLN6F-&mCq@(@Hw8_^3b#Rb2jtW!FSGsvUo*?Mj1dUcL@qUS5XW zPk05U2m=E+NI5UBnqsfGsm)>Bk<+SB8F!q!V<-yL1ve_H^XAe6Hm9%8eh`%!S*1ZC z8{5+#sz&FGUF_{Y1OEceNF3z)^Sa+&v;10L*Y(ld8$y8@xbxHm41P&;b&n8rqIJTy z&mq(q5;5Ny{Hz8pl&QhRIq?*56d3ROl7M}f zTh%3=dWBZ(DnlIdRPC>!SI)E~l5Q>8PyuvpXAS%V3Zoy6MXvgsd!+6E4ODl z+D8$puTMdD#mf9=%$brwm3}*B_r0PGq2j}wQp;C6%Jol{u$kYor%oWeRyAow+@E}# z2|GehTVv9++pIZ^Jm2n|2 z5n{$KzvU>+i2^4{|KN7q!{1FJ zBh%B-9np8#r3kYb(B1(YOJo!jJ$%Vc$U^@PdM;TGZbsjA)XrIB`?8&J>7_CcjOnQI z{0dU_g3j$U!(26QV0dKxu|8FpYKf&W=IQAv4k01T+F3RW7@tH%Mdf95^nTDgu=$hE zUhwN@4|T6hsG4%sf*g} zc&q2tPS2G##Id`w^`aWrSzZ0MuAZI^P5huVIOXK^^b$8q$}_+4lHrt19Qs5~Coy;7 zamBComq7_^llmwJL@hoW8e|~2qBfk_XTL|sR=eN>?@zt8wN)LXK#2()b$Iis7cT&pMgZ#jc!T=iL~zZ75hq z^n-48?G}931^djjKMQ!lxGjtVr4y}q3N>lr%RUtAw{fCgya(kWPxe{px|=6nf^GCt z+!Nc!FOaN??tLx9(p4D}X*dZYso~*aiB`nrmnU7v{XVr0jg?WyQCqO~7X7_E{05Ev`7InAgs@Q%YA3AZXY+kkZ7H^tyxHdne91n-F5#HeXz6R z?d|Q=z(aqniCn$)zys4n=rBHnTUQ>Fe;a-T#sIAgUhoXu+p>47%3Lj%2 zQ?l#hhZ1o6Yc8o-pRh$}#JvKxy|p3Bc$XCs0uVZz^~qs@)2+SvyT1Lh2deN5yzw{; zyG2UMQ#$mDHY?O;cFQ6ZV|e@>HuUE-9=>ud7swj+;4=nbg>k8PiUTX+$faZN4pJ?F%8V4D3w z?20`Xv?G~PQRhRz_z*$8bYusQb;%4okPgX1R(J$k1Y454@a!qjwRf{{4qm~l1z{YI zLzMC_Qm`Yfz;S6KZM%H&Ykq9)bnTjGRjfGrEXJO0*iida=QQ5)+EfvX0PKI*u0*`c z^v7>vwo2oS-xZu|5@+c4qN1baK!lFzgd+*uq`&v;Cp@%@;>ket<6PT z2y8VzmFr~?V%&+r-y6|qi`nJjWveQ=&5(raQjk@Ky~f7N0E}H~ZD+T%oMN6??>DmS zsg#~{;PpFly)42Nn69FKtUt)j)059EV3|DL^K?g=I$8gVY@W~DdE{phCu@g+n7Py( zK@W}ySH!?76daZ_jgtIX^RUdgNu9X^`+k>jy#h^;(gm2dR&-3oTdl6|Zc=eVBR_$6 z*!5O#{n*z(ibOI=NYJ`9@7?h9^2)1-kI@AEw9^sKEL2Ip~~h-2ce4iyKq(h6?9dpL1}v38;)MC?onh2+9KACj(ZPJ<& zw$>Y45tkfBEag=PMfEHEx(VLq=a)p{;J{i+K^HB9^c)<*%$UcL&q${BEzjlcGvGe``@TnHc?ad*AW}mNd z9~C05;^LgJ#h8n4D5$if7N+S4WMiuGG3J-`|wW{-l~$klZz$g=N()_G-Kk4Si>DxTGe+Ba`J z=sO-qB$WuQ4vh#ZOX}sGJj8hB9wCo>GPC`wV@$p)`)ZV5uu`kqylC0f^+p$7gZ~DX zlODuo=E`}@s5Xq3Nk@<}i8DU$o>b`hr^q@{67l@?*S^R7rs+uK3D&HI?jqIT*TMK> zN|G*dyE10^0FMja&6JuA`^FWZM8lxR&9erT|o{YJ)@oVejt;d;!|1dmPe08(%; zfn?0GO#A2KQQPsbA9}vz*(3L;}Lz7J?uR@@4VNXGQ(JvA-&`~PLsO*{c__t-i3b=||pY&-Yc(sR}*-S}NX5-9%b zeXm4_!@30q)aNeFJz!#{&BTEOcsT(WsahNPD0S_05yj)wYE?=a5@_t>bTDsPbTVhA zzXfZJZW^RpE2nKAu5mS~;@u((TYXjN7zTpkbHt}e=?|t|U+DvX!f;nN*(7Nj7-+KE zv-s5#F7@sK@P&{`2T9cO-0BwTa)U|X?A?Spu9^TFC9E5FC0P;TUdV; z;*FGs2jQjkY*z`{Xnxpa%YdA)Ic+M=ys4zXDJ$(`S^?vN{&Hbn|$}u zBx7Y*f3*#SyTGcujx%Y#s4KTXrYwS1K>e#AD`?`pQw;XWOrX*$6QO01z#!=6;)(7 z-krMw0%Ra-v3JxBl`eZ^pM(Y7IFFmtou0#tyR+LBV+`;!fhWXE<9Y%t#UUXf%F4>0 z(OApwuDkPOHMZSAX2;;*D~=^X0)m{%N+N*&;o#wYopwXS0=V(u__&wv%2^`l$Rto8 zPWE{p6p;Z#I{Nw%0D^2jnsS<`w7{gM ze&f({w0Y{qybQAd+d^QL-!*;ZWB;{-+k*)_byRhe9>>`dZOeFy@#bz8Y{gJ06bB!_ z10YYgY9_xAll@#?wmAL0Y*1qxd-q)UW2DN=hVAM}7%vrAYgcZri-|v-e))?OuFpIr zL(F|vhveqw5|fjMx3$UY-)~qxSQ`OgVOzdNxT)YMd9Ofq?`NvPKXM=3Z&9taV3 z>|7MG6_@CjwT+b-Bi{{k-)ChdW?+a1X=!|@t-tqmX`{mX`DP7}6?@GS;S6h7U-*Mk zgcfJ#=kA8hrJJ%BJx|$kjEsz;u<rxqd?>@cdp(k`3%~sM!H5oL zK>*Zr313~Z_$Q$PiEo4wE9B!Ve9I9rNkrH*|KSVm0mO4%EYM42PEO#jL8gndafH4n zE;2GE25VrS%iO7sKzULg#ABw2gWZVoOyZ)e5(=^Wv#67a;MNb%fycV5y$Ew9BSpS> zL*_i+YFf0ATJ16|^r4_Y4Q0OR#$cZz7g{@m-G$fPT)1PDMHrNo^ReZ{nS&^?@*cZz z)JsQIRa_yrxzhH*G>fIqSY8_AX8XaQtMdKTdv;eSAg)q2&RaJOZ(!N$xY@8podP(kqC zy|3;;Jp05{R7E{2^<$K@R9zj$etTX;Rk&X%6BdSUt%WZFsNuGxPpHxJMHsy`DROvN z7mX-;;uomg-yGO&Nq@$~u3!4d#)btm8mEmPWDsgymq(e+qD$VYjO{zl-E@fR$n|H~ zc;2C#mRulA+ye*q^Cj$eXH8BO??DjVe70T^!*TDY^~{y}_^Rt_PUysaI)<82Y46j_ zp(Q`ApK<1f@;=C7XrmVzrLS$mNM6Qwacm?OM}qI5 z`59%7xKaRHOVGT`R%nG(*{&7g(Y3@;G&%}tZ*MDo!W(%BTXkU7#i>yJsq3tbww)4|8bN18k}azg_tmL+8XjShS9 zf}KxD?VWf&b%0IHY6CWFYlWRl1bE)fZ%6OqsxX&Q8)W;P$yHtRCxOW)}omtKz*o*VS;QZg@0Uw<3;lPu#Kl7D;g_)DUNv(~2@ad)&CYx%0x z#ZO2PQjzNs_%yGwvUWzoj2QPV?}31%Kq)$};#QJA$TDi%j0{)MPP>aCl%vak=W?4##bCh9$?kEg?=CC6g#rm^9S)<~?t*W%p9}lqvxwts<|7UK~+?*m3@qUAQRD7O(lmT&O_IJd63)<=^LQ zMtK^+vE~&Df=o=;K*nyS^?@A3vn(>p{?|hLIly*2VP0c15?|h8pr@gU27A}A%eoI3 zdF(zB{D;Z~bpT?9=1&KQfMe1vtEeYoeoGfSXIUUGn0)6Cl+gIoZ6|1dMWcDYU%WbA zji@!}_ykI7zBJCR&!oZKWrCt$#(0q+C z-djEo$^)KoWZe1TtGKh*d)<5yJg)(Gvt&-==CnsDAPTHCI`{Z8%w$?Nc4at_73ISJ z4%?@(Axbjebq@ZZS9A;zG)upRa^dlEh4DkxB^QzNN+G!oX6KLegPLQ_DNaZbW|8Y? z8V*oofr3v-Y7NdkHce)4^Ln?&Wt7N8BYon%$H!Up*rC^E7E+~aB$ug&2asQMQQ9BL zo&Hh-)>P5jcX*q6Og||}E;?NRBWrZ?GQOzIVNvYgT49gvO|m)y~A{*rmJC zyKC9ws-(szs!Vb9t2JMCt63J6ScVoYou1!^`Nu9%>k^>^YcnfP`khL9h!Op5MpJ`Z z$bmB&1fVD2liP%IofTaRjJh=AOL5a}Gf&$(MtlHjLGQT$c%zmtd5h|LteOo zg-xC1p%7e>0y*2{X9eg`DXL}eMc%@J5jES(4WVhM-+aVLd&x`5P5(E$5jyFn$!8c% zzFTcqm*sEm zNe)J-y<)9BcHENsJ@z|)-mB4f%Ls%Lr`B(zUk04faw~@}P++}y3}^|q>9f4hi|->T z#DLp?I;rzZy)DyIcqC%BYro??YTJdac2i2yh$Fht*3OQ9R%|cnre6FN2ptwg1=W#) zo*fl=S4^Nm3|?}aC@c=x62N)CNDx`f=KBxf(T+haQRbia(C-ZeDREjFbym#pn|~RV zgZO4f5iGzWOnC7ao=tIwlCK}heFE05-pkV60*$3@@Pp=0V{6Wi#3E)GS*s6BS2fkG3bqrRl5jCHGBOMry^EZeh{Ni^;ma*w zqa2mNsZ{>sAW6SS{=*vsuLflj*qq4*^g_&hajtmE7~{_(pa$T{f`w1K;5+WMS5hz6 za?V`7umpcR=RTDB)?2O+T&BW5mC4_Z_Cpo=W*0Ynd1I`yuL= zMlO5ivjVD!XDqq`>^LWuwk}bh))CtNd4?YW;#{1L-Qyfh!ApQz3RMUsLqwPtRnO?n zCb*_TAsH2G3{fXc%?(jRy-n~RS@fLA)wb^{X1h46FX;oP_2>eNcQ%<=-22PZu5$DU z!TecrKAF_d=MJmInW=qiOn#ZCwC8jp2`y{6^gGU$zZ4o_vB($d?9DwNj?Sk6#SAe3e1gC$7mLdox9%^R#S;I`(M$DnXEb~`t>?2%O&Ms0SODQV@=b`5|a`nkRb#g3@KH|6gIhijq_HjS}-*iKYEH= z5vc#oxO@hcA?~;M`6(tmKKF)0xU<;}qMp033!`;O?zzm<+}!i>lb`#BMv-KRnNh`F z7yZ7LG7jKSzJwH|1=da^xbFFP7(u4lLl|%Js|^hazC06`q@>A$AmJ^bMWwVH0LI)U zeAET0ITrn#gGuVPB!B&)ZU>+b@rK<6b-5}OC~3|Ei#3CU3{$37i@HuM2@fCJ7L~8_ zRxj$zKRC*0j^xJIeUs{ct)pV+fxZ?w23k1sMdq|AF0iMtkIM!7<#?QTFJyVwcX!2T zjXT|+1#qJJ1UmrbUhIfY?c2c%TmS82HRNvL34<)kQ~;$A!#IXtGQ^&zMfQV@PQ39ZHE zA+4g5D9P0IT=?_I<>e(>B-EU#tQPFoGF!roVQ!%_*?Xsb>??GM1e*B{&gnH|}{LO%eLy}fHKmwLZ})s;_0 zuDwUF=dW!VMChc~EW36h`Of$nh`9#}S(<78gxtk*#atS8&NDifj=c%&Z-evI(+9`R zL+Lmt-lwwX&jGsdtK#a6D|NrXO|1$^B?yT%^p)hazV3iD0E>8NPlV~;MpzS$dfs*0+_k3Uzs zq?U_czQj3w;1rAnu%mIr8hcX@mFVKJ2kODui$mCq$@WQ-285lBZ5JR5-vr;W0ZhU>g6h4_SIZeH0aelKS$8rf074PAsu958g@QPyv)cBGK(v6mEH#~9e3_~?>8^OdbL z1~m#WTRV*=$bNOLMKztk76KWdz6@PWYPClee2n z_gLWENupPUKTTcryV(MC_~7KkPz-=hd<9rf61AC->+BI6Npa^+fw2s^cz!1xF=U5L z?bl7WrQ-=m&>Q3WI+T*a+6h!0a_t9@ho#M1g^UVEwXkxCz%~t2s{t#x{yV|^L-i>y zhzmTQwwjd^uPTlMhS zE6gbK9c=}Of_{ZrEbvVeTW5te_BE5AJ!oxIiLoBQB^99H3*$+-5f{}fLBcGotV@9Y z%{GCzu>(NE<4l!ZcVHB_2I2?sRe+U|M*fed^;NDMEfi0ioY&{O! zWYcgs)K)aXkyY;k%bf-9*c)1>l!FfJm(OShSX z9sxKS%NQZmB+X&t1Cboag3-w=8UzKM>S${}0r|LLCpZff;BYBGyP@%y=JV|)WetrE zpxIDT3KV+OkHAGGMsD(nN>1>kjI`SQrr zj~|7AE{)8>!or}|o){>mYc=5pRzaZ57Bsb^9 z$#DRvXuG$;*0U97#`Zpk51{h5;_kRt1ih?#81?&n)DpSPDL&ErYz;gOUZ%;1hgU;HIth|QoVLX z<6wg=^eir4%m7c}zQX3C4vo$a3ZbN=M?)ojO^f#dM3`M8)EVI(7*Q+y+^$NcDq zhv0%53)i!^_{F{}^dwN$9_psO*+XB6&5)icvPu7meKG)7DV^b?C7nwa1A5?ip|gpF zY<;H!B;I2{i%^RNf^(4Q8i4m@h{9q386MUqFt_z}P|pfcd)EvQ`9!#$zc03)beI40 z>8>p-#OGr&U}jWb6*eyXB&@GmZ3wTUf)5E>%SG2^)lprGe``vf^&_aujI$bz7lk(T z0i~mC&N`YT`X!a8x4B)EqUfw>JFlYT3%$&sAItBS1&C_TU^)=v2){eYrU#2HuncY- zcj4!X4fb-t$i1?lbB5e9b7A0o`Cy)f=QQk^;JjJ+cDd zz$5zcz*9F@z$DIu*l$|$wxWj(Kz%Df=784VBpzGw*YK37gT%!${X8BOFjEjUhVFu@fAsJFX+S(%EQsbdX_F6!0);sK3coRy@83R0O7ZSQM!F^V zdV(yS9n8kW9-|Ad&>^U8icCW;N9-nu(?Je$lsO3ZNQx+mD*{RjLjd{8l+-Z*bbJI_4_v$8I;93kTlWN(lh+FKu`WOOFpzhQhKauxDAC1I2IbfJjPU`I z5bQ*RTI-D|{7f1WWB^f3;`&rw%gy_~>RU9`RYKu!Uqr;2Nzm2Q2z_ODoT4?h_=b%v z)fkdsF5Y+lG7Q3w-OEGqQjOFkaIk`|LPs6SuFIakYZ)T|EC0*zmnpJ7Fn&Uf4^M%%9<40lmaCMr)uD zWTey`ISVU}q`D1ML3 zE~(3!(%yKhzwuGUe%uQq@=#oXjq;vSHvnd4q z`j_cKxXC7LO*esI@DI(#1<5tyU%fGp?QEKofN)FU4`qiAfaXNg?aYh**F2gr&Y92e z@q~Wk0wG|bvydWpfNT)poE>^r53ffS-vR6{*z!D~qRdifU92YA{BdEHwdzR^FZh~b z0DB3zXuNWW&!SSVsRra(u|tULxk*^g0siD}?lKp+Q7xnZLmNq^8>o)|;}CB6&qznb z;NhofQ+ltH^}a>=TWcULCNdf*zbo-v9;pV{ULa!>2~oK0 zs;%yHbyjOs?Yq1kHF4btuRu)G+#y`sAy0Zy>QT-*3r8YG>s5KCSU&aJ-}y>u-BVst z2UKL4y)ok;dhLf;3Q4{X7*NLXrAs{bz(K>PS<4sJ&l6M}f+FH2wU z=oQ8J40XUM+L%z}vh!s^WE!~uI0@h%esc-)*U449Dm;7SU?mKE%6cj_a^?FU72B@d zEgyB3H!O$h_}~V(4H)5Dpx=YrF~AXkoss6~wPNO38Yj*$l5Qbu2v6A*2_NL=>6{`n z1^(fl@OtBJp&JA#dQ+(rnh}qq4(Q3859QH7mJO8rj5xB2U}4JUz7cdR@F83f1$xhH znWJ}=+P+DeXgVMN_ZGw#2?VsjOPk5&6nl7b2`;da-y#rc=gJSwEkhL2dBxCu%!0wb5J^kw9-pm+uqi4Q%2PE$xc6hMSfVB{}n06CvfIp`L*{od7 zI;(6Rav>(1;^)twXf-xkVm(%29&gi|Yys5DI0OWpKqgEKa3&VD<2tD9}->5m8v&MePGU8#mAi z$;rxIp9|dJ1cZteP;rX{9dQ6jku_DQyE_rcet}F@&cJ|j4u0-}?h=t@1|j|#8u(Nm zi3#L8RiJlu^P#|eX&|> z|67_4?b-h)={uU-+$_Oi-b(8J3FWN%+M z;6*ZJ!G&Tku4H>na#?x&k9MWy%J8@J@vFk?4>&nH0C%_x6uPGpo%PaUs&%uw_inxS zEw*xe_G!2zi~V0>cwv7AOI%_P!Frhs7A4^MOvgUCjrxG?Z+-RaTiHpNCF@0I`x zk5b39^6}I8uKa_Z&z~{S3D%e`mVoD(xh}kpqwD?X-N6fP#y|8eoNhBE8j&)ewO0qk z;wR0xqaPnmJ8ykY4rrEd@bGYGNz{@+p);Iak0p@7k@%NL=8vpAA3_rn=qvwq9S$~BBxd2@q zTi{O@1JOHA7egZ`kU*1IQKAv>6_A%$UMzs%8m*O-8F;@GT>KV`5#p=*tPwygfj7E7 z@5Pb0-mK{v3`PtL@gCaqW|oZd0e42u%gbB*o&48>wb1$bhUbZ^ZcATk>23eeDI)w( zqu@`6Kr?K@Rj(>_3j<_YAczE22IOm+p}R?pL>B{Rz>UYuw_e1-bRhTA&q9os3Ji+6O`A!k80lQq#5PaV|ufT0M!R! zs_dZRiT@%O6!zRIuBi~cw_VaHX4(&jG1m4t?gUd#qHaGdrV(;}1c$e-??=TORB5)Y zH~V2@04UP7+!jg!!T5)sF6}XS8bNX;>{w$3sLu!VoKH^CkP^H(K&+h~ zbB&Mc4VXatKS-awm^{mPsdG8to|E?K)svGGf7|^guV4>p=tb+k6{T8NTx_g@s%q5w zJ~dU|<-jE319`y+gS(&Gl|p+6w9Z_;FXGR`XJf{0MPqkDgaC2R)h|BlaXBeRS2t-B zlPC7NWHe=y35c^PqoFcq}gm*n#=g0@q5U_CdRKwpAHOLq* zCOON6)&=IKrpYYbGAXGx#)}=04VqrW50tbE-wAg6%_^*TAT9SVhg~g%37}_MQbJ<} z-(y{wM^mY3qA>E24oFc=7?Ak__#)Bg`y+*0sC2G(BOS=o^F_UcXw;`Y$vD3csx|P7 zn=xyl5hTztH&Re^2DmMgcPF5%3c%l>QVny?BpOh$Caqb3#WoCSAl2KGD48jNJlS%% zm7Dzr6;x;#w9?iY8EDQO%#ztyzqJCeg2W1&wO8LGS*8D$s>D7+j z;!?g^-=ikOREkendcduvD1DmLF9Z9S>$uM^^k)Q0w)2xQ{rt|>$y0Yk4eFa@>w~na zk&8A%lL3nQB~WM(awMHxfB|3=5N4$IaSKNfBj3HF1^N+C+LU+Mp7f`Bu3zGB^&D;c zT@G&jg0|X6X#st{_dU+mo}X@;qfi~}VsOTpoy}0scxlHP#BQ_$irp*+b+R5hqJ~+l zA}^2i-IhP`Q_^RioW`JI*Ujnr_-`|oQYgTB*OU|3Hmz?@Gk49Q`Z!ZcQ}5Urc08Yj zf^r}xQ?Ih%cRT4jSKLQeR0!#}3wxJyaSS-e;^$d2ONj0^c2Cg;Zr58|sFi$x=4sXDy#;~hJ`dJ;-S2#{7sgO~@o>SJc}l0EGID%1NG@V}Mmxo@J&^kTr#0@k8w zJMvUVD-tOWr1~Rer1##xLB&pO@|)>LSvnSl7QRFk#SUEz@Z>lGzow^`d)BypY`Nz6^Hm<(L7{EL%&bMGz?uB%&li9@a8vSt?sTo zFS*VzYP8M%$_^Zbf7s-7XvRoDMek{q)Bk&2AF}9AU0;5MVnzov66q)>hY8Rou8p^7 zceC28n!%O7c#PbY&f`r(Q7J-FI1Hbamvcc^$rxIsw75My27whsGQH?gyYZ;73cdkL zs~Ni!m+!1EXNdSq^y3t)zpvzZFmw`{c^WZ;(+EwS&U6;P(Mw&!)b%Sb4TreFUS66DuGw$WXs=i3J9u_U^URuc^WTurhFI>i*=f{L)nMA zeeXX^;`)ALul?SUWya9!q$Hcduj3qysC~NJQ?Pk`Ky}VUlOK2Dr;gHOx;2U{&kDZ} z#w#2R*#hEz+wD&=Z@&HRLiX=B+nD7!7Dungc}I7#M}VdHE!)tyPQWOMkpjGF+@IBE zL|^H!fq{Yb9sPR&^z^>AW)@FmrrmkvQxuXJn7lRI0uGI$%)EZ4{|wA_y;YwQs#R4 zee;|?C~5yaY7NsZaQ)L z$TLuVjb{MS=M(!GnrQ_=@xLp){@D$_Pjg=b;}E@(b8>URFqFGdD_z~_pg_$SQ;F}a#8;>Shg+^_eKAyKO3{9r2dVV z|GjkW@kqV~ATod&J6~B-^G26PX&(+Slr0I)m7To~_7k+;6(la;y|9GR@hh5t<2vzQ z$Ni1CgB69MGK17qR}(QX3;>uE&<38Ko(Ac}+OJS8jM^6iyE(Y_1%udXF$N8?+^q04k*lW-6LT}7@MzINHa4vx?Ni_H zHd|A5Z)S6eyHVGlKOuo){0N|*{^$aR%+}t~!SwK9B%n@7e};Cl{s3Ir@ZVx7P`|JG zZ?RMxKDc|IQ32c?9!6(q;4WAICZEM>%@|SqT-Vr(G2DN46Mx=jt@Tt194R|TQU5Id&(_it>BS15sH?4)NR0_Iiye&RxJ0XA4LXW{Zb1b`$VqNk4& zp7X~9MZe%zGJt}cjS55@q4dSzAT~Vm&MVX!$m9v_|38@$kUGLv!I^xtGmrkY3P3#iM+@HQ{ZLj`wi0+^fMPse z8_qLm4Ho@hv-|Zz8K~ePa7hC2w2CeMu0Mb|vJ7K9o0JnlK&OcO{LlCLe=4oS_GGrdNZJwt}M4?HUg|Kb&JS9`c3?-W= zNkWl1GVgb7-S_i8?{WN(?|Z-RIPPwBUDr9Tb*^)*-})_4I$FxdjxZlVAP~n?RTOm* z2r@MMv7kHzPxhW$s2~uB$i0w8UKl?+M;BWJr;x(mzi|rkJ77J%IE55B1qChL+^*U< zTH1S9Vmz<9+Iqnwc#px_INCYd+Wh@SkY7;X3cuhLej$AU5l$gFej(ET#IA}7i<$iW z-qPOI^`8z!uJS_%T$+~Fj;>x?{UY7QMpL%#$ zd5X#*HLds!)r1tpoLwBR{4*Ude>dB|-`LptI9l71S_&ayLI2Dg26X&quA*|H&PW$e zYcGF{tCfzbzLT1;<=@Z63|y?NB(T29I{X4YNO&l3b&YrsbzD_<8?ux$ZIy#<)x<+2c zmWCKjJ!OCR+TB>$31jUiWU1_K=^vmT;2xl7jBz)H!9>wYIyQ2imJWJe9@u~YM}JRW zRRc$DXydC9;BFh>>8`74qNC(2pl9G@C!p@)q-lULKr8wBSS#vix@&r((QZb@68z2{ zT1Mh78vM2aPQEsZ-hTR;5?(I8qF7NK1y5&3*i|JhCvhKde`h1n0IaZ)i-3!_nun4B zR@_M4)8EcR&0kQ_&BsJb1t}n^>gMjP=cMZDsp}zTtEyvX=x<|a1%j8?2Bnf0HnK7b za15|?^YU_4*0Mv|IP1tGZ7sb}!rtB<3f|(j7#B2FBS6K)#YDtg6&k?Ms@8HwI(|YT z-qxD%K*ZP5Nn24@+eld0ir-b<)6v7zT1!mdSx8Aq$VW@uKs3OfU*5)0Sk=eDSI9@x zNDJDi>N{&F3t*rxjK7dNzn#3bvIa)NS=q(fMBPWi*wzwbpzk7P5TI(SC~TvQ@zN4? z5JjojS=o5_*?9R13uqd9s2gDnmE~2Ol$>mB?1lC8+=O*C3|0MAT?FMMTzoa4Z+Q=U z7{C&1&o654Jtv73XIi!gLyC#t1mtSiQECv1XI6%yAL^TK*4 z!9aR88YmAJM_n-iLkV$=JyO*jWvi#@po~>jS2l2$Gg1-sHxjbuw=^;qQ-jZ~jFq7? zIYmRPs;Y{kf{MDTsjE+~lAMJc(eC<+<#D_QHQ ziiqggBJGjN^77hpnrLHN4`po=Zx?SbUvDE#2|Zf}AuA(MVNE+77a=zze<2SSV?iGu zbr?@c3uWairlPEYLMn-g3n)3s85*PbvG#Iy&H~m(7#{-K z>YBz#5nJ&9q`ik=fU~ZWp@M>&0xZ{7&)H4E(^c8n1+A;%s&0$bR6{A)3uFAXEnOsB zC9GX^O)S+FT+k@eh6dQ^irFdKi>Z04_!%K}{JpePO(XpKA9F8+be;CQi$Jg;R2Aj){Zi%|g7x)%lNRd(dpPE3zTWVtSCF2`(f)Jf zYOJS$M_iY{{7=Qc9f8D84W?l?RmiHQW5c-aX-TIpR1|-_);>gs49|bVm`zT3Uijr5 zW+AJ=>$ERVJ4DN0ry#%h>LlOoR}ACZhZis?oxx9E);6UK9hHP(M z<71(`%|`kDtI?V3vXATjAx^iSNy;Khw+OKKOe@&-yN&>y>R(5^~A)) z((3A+;$ps64Gna(v`pTx zRV(t>FPC3q^{1F)4$;xkCCG-#+`fJLVM@w_h=@b&;}4hgVHMTyKYhBNr5tCqxiUq0 z>{#1S;SH3L(Y;%@$WRO8{?PxA4TCzis@$XZ#sj|ZZ7*d@HZ zy+4*%9$>s;5*{9YP~X6yZmPB#JFXNIB+Yo$^xm;EqAy;*{&J@+Drg8*&q(Y%k_Z~b@l#?TK*2{U6o*viPsp2wP z5mm_@uEvhyROGyH;Q&j=#9*FYfo9^9iVESLWU=EjeW})%BN>9Qp7%x#0U}kHPwk`B z^@#}y*X8BOVbMm(@;FL*bc-zy56{=Bsk@n(=ZA8{kub`1e|j7q&oh)m%M+#;?-upo z0rjih^-q4kf6zJWnPxUNvSJ(77%NP!IO|c~RmfBm@6Aw%ED6~S7_z&krl@!@mPMjv zlv=*|A&y?_(c{Or&r11}h$_%mVaH>iT+%KRy`#g$fe&we`SRt>)#=`X-iJ>U5>CI& z&Mt*FZ16^bMbb;9Qc7PP<{a_zCC88*?U6f*@g>PH5ki{|1_UP;X&9~m{rkY)+^QZA zNv>qlFsk+7T=-nu3PZD#UQJ9*Wp7k)+y9)KYrRcz)F@ew>maQ6V!mN9coU6Army_y znwy&odf84QqId)@F)=Yk^IVI6TvH<+T}&?uJ3Add*@^UP@90?W5$Z1yz4K6_!iq9q z5sB=2{po4Fs-`B=#pSZI5G50ALXVoax5)Vm7mSnH)rFalIXOEUC(Cu7q(@sBz-HSF z-!!@UeQJtbSomorQ@93nZl_~t7ztu86;&9zY*zR5VikS4JoIz3HR4$3+qc%3h=Q)Z zzV`N*Go+4ap(Eyoh6bYw2Rdh=legmH;v6O`sn)J%s%cFAc%P-BqH+@xasFeO?NP8m zu+~m$Qj53G`IK#KIiZD|zrTb*fl)K;jved{`LScijOu-^z#F8B3T0AK5~&U7gZ$Tz zrPd$oym_64I9i3B-smWE*WuTX48u>2ET!DVTK`qM6mfTXMP90GST-e`zs!=7hm}m3 zLWAzdFow5ZXwMXw!%zM}_uh#>iyAjFc6Rn}XEeDOq4VpQ+>spAg>ylS{F>5>PQmXD zDvp>9XOVvRR=)nQx6^D-Ml)~9edBfsSQP}~hnJ+Hq9SOIuu@(A`^Us2wrB9UWq{kU zAGj3v#B-(!!)p!`t_ z8lP(CIeIiX;_Li;OmnjwOjgm5kd|k-o{1KkeAz2jdQFlAX8cv0CPX%vsd_IcKnsQ8 zO*+E)d+=iK=4UXX8yg!nGwSqZk)#bfc<`g|z!iHxRON$Qle9g zLmtWQgd+Sk(e%QIMp=0l(iWwa>WL87dRw%c+*uaB@VrRnVdo(&-n&gdxq`2kwo}af zMud=R;8Mx7ph-9JLFYq@uaSJ0+dm$dlR`)5(f7fDLFf>F60wuxr@rNZ*}TNZmnf8@ zmP-sThjW1lyLXQgv<}v#@hSd#zB{1{i=C_|*4$onZY=R%SyV}LLZcbrN|=ejDsz#M zwqof^;3B(}6xg7v)`gkfkKVqI#`pASsO?=kxUWrV*8)p?7tKMN$Su%>m%&|5_V`-! zZJ4hnVeeFN*_^+W@4}nC-EGcYt#A)E6ej{#s!qtiT{2p4eHMR>o6<>rFMlvQ+U*Ew zG`QTd@HuRHyZRQvquP~&O0opq)ewBKt}MF65yrcwZP86ib`CnfaP6d}9` z-)>7D6WZV3AMQbhaR>QQlIiH`>be&fXOD@n(a_Xuh)5gd@IueaX=CUkB#S2G-V?oW z9~vb}OG{NWHRGF1XLWMS}*AP-M(|@{pZh@AK8Y}gC)#a-kK|(T%Z3` z(CfxKZ5j4#XLu>NE%JwRPt(Vy28qz3f&yyn>&=|N2OKBiiWN9k8Hu>e^s^|9&H^J{ zzwxu#U_I_Vs?O5jQMO^Im8gn-l;i0brOT!@V3l00mqMDKhNTIf7}6U$UeSXbS|qOM z2HEq;2f=qtuZVlw_|xPL$Ajvdn4Tw=4E3{DMU)?f-oE51T$N9SsS7Q=EK)z4JvZ|u zwXgIJrgL36eS&(f`1-zqH5e)|zO%CvG-Rg|YwI4iiN;QiH_EJf2G#rwEfr0D$jyO& zQ-CXUAuxZS!3l;^21`a_8>vHe`DumcT-WZ029kLb#Eg=)NEby5PqHXR-o%8NWO-lZ zd4tgX219JsOew8*|kuts3V(HLAN59rfZ+owY}Vw zRrU5kBL%w{Y=}fxPmdz_2QY<ZM78;Po5ys&e<71Tk@Hsxt_jv zC(rCxq4>>#?~L_S{$PPV-W?q!H))kV$iHL8gm{s)&LXqx_N5@5wZ@n3Sm;)Lo|Yl! zVv}&rUgHHTSJ3Wy93J0t1KZFxN?lEw(Uup@c<3rS;^tI$JgyVk z)-~(O$AZ{t+`o6yxs1O~tz>#EC?Rk}Awul;PZlj!V(OTzJ@cHhbC;CfVWZr1q{XBJ zRUrpyF_#MVJUDV5wl0)m3K z!G+3irO~v$^DaGvaqBnhO{0t^O!AwRi9tSmakL-MoH(#(YtR zhW3B@HK}D&oRU1YjGrqYrm1CLMYVhK={V*efwRPsn~_0wM$9!lF79}h%gF7O6_0A^ zYA{1!)t1)RmGt#bGM_nG4_vnxLd4{Py5CM303g9$?0bHI=#-0}uAxsGjT&q;gj70dsN`qmg8V*iQEm0>f zk(wj#Gs%GhhLN*m_bBXTP(UiHI=oMHV@ydS8DG06u7&xhmU z#q;OSlS2?iiaj24s(+oFyb~6NfUEju*JDo7!#eLdS|+Bb-R(aY#KeX(#Pfuf*u(ZN z{|!_R6^oddmXegqetI%3!7dIaXm_w@L||b%OOBByKp4}+OmWlC-eH^MiggSC9ROh0#?{Uwn2M6 zYaMy9UktndAt_j8d5PLS52rbzk=J@tJeoSp5SR>d9dKWqX96tv{!EyXg_foz$g5jgd-?BffY8E_EsKa5X)#IwKXQXHKBbgU6iy7 z&q))YJ1sKHq+!B9cb_zYS!tL6JAH4hdLleKu`kP6mi=ctFA1>&YtSjx->2fybZCg_ z3koj(`qFR)Oe5v-=zi=Iy`(TBqRi0bp$RpL|zMAXXE8PDk36M z&`6%)0Wyg`MwYN#ldm>7V4!W%SjIpAEPzGIhx*p7TT9E!ayB*`5YT~p5}A_1GB`Nc zbd&!@R@Q+#ck~=D-??+=$B!Rj%{Rleaus}hMAsL_NQV!6$ewgrRyjF2u+>Tm3I`^! zD!50Ww9&`OmTGiIXS?PM4Khdn|G^A!?m$HgH0dam)owWMYOxGzbTlXf3L`RR)u4Mi1AMuiu?z>SO}?502IIy+!%4+*exyk*kxRyqR$h9HVCT^$gWN4r0#{vba^Gj zHCL$@^d^zLU7IsNnf=8!ru&rZ-n@Bpke;5N_T!Xqri2T(dMvbJ4r;Q!Y`W9Ps?av?ExGQ?@S^&pd7Dx8~fsRn{ z`!!x?U`GTFEmjN+^%Ln%WCgocyp#D{8^XyzP>7_l+}&E!m1&uWAopwMqYHpVe2+Dz zyg7)o$HeG~mxUZQS;>&yt=3k%u`4h-e!0+H>s#y^Zz3)Su#=A1v$|tx+@Gqk9bHc@ zkw5|#nS<{NrJ6=hM!MVW(s+Nl4wq{B(<+&`jJ`AU#xu;i#TKfHBAlEQ$zs@N--gQA zFJ3(4=%|*Us#ucQ$Q{U&Tkb~TtjE$4za}+0{$N^~cqzLu^#1Gi^7A(?(dZUgyhpCR z%TU0NWtfyGQ6(M_ZW$B|lge?ZNzL21x z{wak@BmhHa&FKsrulOqqfJ{vrpX5+^eHlGZ>HO(bGk?tSC=aZ_Bdgu$i+BbG2EXn3 z>bAPn%>Tl4B2LeJ05^2$g&%{n-n1fq?b8ikmM!f)k_&j{6j7@ET;=^eYD2m}QWxgt z=F!^89?xFCmbq+H5(bb9HQmL+Zs~(;Q(I?U>JKWk?_AQmVbL;q^^dS=>8aA;Vy?`( zfHkMKi|hB>;OeAfVd(*+Bmyp;DQ0rFj;*slcX<~=w&>y3VBY&3+F4YA?pd2F!`a!` zaN&RJt9|?Sjq9~q&x>49)r3tBF%WRaxyr1mZ=ugIj`>D&!?vg?apMY`c#6r%Nx$ht z`+bCl{h-^HG28&g$wJ3KnLcdLj=rXRX1m`YyYW2^$2H=c>>&ns_`R4ITg)MLA-K#! z^+cWZ4t^{gCpkzLVTsC?nSS^sB|!uiq9GS>z%`xVTh&j^#|rgU2VYpg_{5d#EgP z16Y4q$UHY@9zS~La5_az??uDmV!ce0zu1+;u8#K2p99%tX&Wjv>C>OH!RvM%^X`rK zJhbQ#DwO20cb_mYbhs@a1P7r*KSs2+EF_hl03awEA76jo;eaC}BO@Pc+%J$&v}SD+ z+q2{kC%00Eevf8W#1k99*t?c@#IMbj;MUjIt=?xMwXXGi0HaKYk><+?g420f0pVfZ z_)N3-(`(B1uTS0!gEl|_Hx}S9^=r%9NI&k}wP=80BU!&kxr~%iaC39R8M9Gb^_K+? zSlWr#awUkfzyfX)U)ePVMRa;!M($AXu?Hz;jWqXXSEk;Mi4NcR`Skg7>$W4dmBANY z-X*yh_n=jF!FcmFg@gDm%kSJ8$IQ{EyVlhsUtvrY{WHwWB|h^ek*wdk00MrImls8X z{NO^289g<&=K|w>z|jTAFgbMZEei_}ay@aOE84#2eoK^xoTVpNcf?xG4HeQ{Yf9;` zR;U@rGv$`xpiEDEYafkMiyLTCf5ly=bhf*3-qu+dU z$RFPPlvgue`1<6_qo=O2ad3pg+#mdSMb~OGfG?jLK1=gCV13WV47gY1rx|Nt_YeaK9eV!IYIu70gwbR(%tv9i# zU?`LaxDS4`n9cvbQv@%>D?GjJc1N$^B~xht7WNshDQ{%BRD9`(RU(Xo|HQjI9O*4o zG`p_$eN5s?(E9f^MX&X0$=}HD4`(HqdiD{)?4yWPjkZAhW>!%_sq zR~4ok*xN=vW^oZen?+t^d9>RC@(2Lr-TtcYjnnVO>$B#DT~WQ=-M5vpO6SVsIwB6r zT??8z@Y$(Wlms$Xn-oz=4L=z^fiAZ;1GJTNyOLnmkO6$XJR-KhqSl&OBy)2|_lLIT zyoU1c^gf=PXT$gQ^eksa0qh);GEg2a9?Pr?{NwuTQ+0n)e_DgL3a$RzF-OgMnqe;@ zjhS;L!Hk!$0QS-4a3^*;0RR#xXpr%AxCv8j+9p*&Rlc|w64kWV3>+{wbJjdd+5 zrN!sQW0fl{PFPv8{Q;O686)7%Pfo8NA7gvfp1i2jSbHPZ0$JR;$|rq1b`rog`Ig}x zo`P&&KB_-DtrPJE`RL#Ir`XBt*u%g~taOajhfMW>=qE(J1VxTwx!S65ZaYhN+it(! zDZl7uU`fKUKDVZ|^6rm^fA0V=mi4OV&Q+1czMqcrwILq@{QLtacCuN5JgZr9S7n;{ zhIa;>xWeK}r}epH&*lNp>7Ohj9|GQw>lj&Ay6xgtywuLs%DI5w!Wlcy6DT+Q0w;XB zBz~V!b6*SMX}Lq}+HYNDMpdz?u|MkU%*^3C&HOX==i-i0?c<}w(v@QcG-en>m-yoc zmj37lxKzG=eQs}Wk7Oj2z|w-bUtmzP@V4)V(Evg&X{`vFKW-t3jyzznXL`2Pa$&UcQd-|)pxH}+lbKG@;T#l4;4gAeaGuVE4|e()=m z6nfQS{PNC00BaWt$u}4D&qn8&e-Th%Bkol;cCg0oo*Q;ncdz|MZxC73PEuI2Pro+t zC-pd`DfSd^bkl|ZKDpW9J>KT~|AHSy&&KO;}3D^sCLHl2J+t*02Rf8AP3ZF6%$dkr*W zE)fjRyPsa7vd{J0TbxWQi^>@-2xc8Ib;T}eJNew&l@ObQLAQqcTQ8ip>8HIV^OI?Y zu`g@9zN1A4O=9x3LITf>?{+W*S#00!eG;m?qJP$`(dYgwDO!PaWpaAC0cxr~+sCd0_zR#3O9DSG&Z`?p>GiISjssAg@!%+& zoZn{ntBWOp9VP*P=xHpz+zM_(iSJD+UBY7`o~>Yg z!6?#6`^asuVm#Yd(GYB1UDIpO5%HBJ97Sty=jE7vx^ux|2Rkv`QMAVh*6YDr@0q)F zm(^yg)pqr%0v-Cyct)r%!r-3)=w%EiUCjCafx5f1(s>Z10Bd)5*C%vu`Qt;{Bear4 zn3}fhxi@$5A2z;E%T1sye(Z$K(ZXSu8xC;pTjqUs`-T(OW7}ufwY84}szq|}$iTtN zK0+p57cv#=I4Eu{c>T#bX9WYri+o{b43ey+Pyfe`#jEa&K4M>iMWiBlNt6^6u|$jC zA-;DWGTfZ+&J!1y)QKMdxxgDOZ{l_^s9B}?;uSL97|#f`BLib!o)8*pQ5CFjv!bd6$K z$6C_o16TUTLmBUY8A{pSqTGrMM3y66y1ToV001Lx4{hssyd20EwVlcF=TzWsS-Yvu z+E11(YJ0`{YH&F$Q4HG*-*nCT9-Pjzb*U`b>Zy=j{1q5B=v@EEh$ZF@Cl$`Qn6~$v zK!#yx3lnoCpL}|s+Ax21MCi30Nx6z`srrz!jOeWQ_wF7&CAq`qEZs|U zkKQNqusVHCwgyvVy?Y6V+=dY1v{;PhiSI|zxi-E=*%5u$!?%CFqPsBi5?wmX#|v3V zOYwTQ2we1t{QD7-UrOGzz1YY%N-HoyNRz!+cD^9!hTL18{AWc09Kp-Z9E{2re%`le zUD7$X)@xG82&Qr~w+2yEhOqhF4F?2A#@l6x>xlbvqyv=5RVHte#qFVCy zK781P<+UuVE`Ow@dB*Vq*8X5y`rH}$M+*f@p2)iL)2XN3M`R5Brs56XeZ<8^^SAUS zXqv>IKltK%kz)>O^4INv=o7m5_CpJ`J1mAn#UILE+x0SkpP7-`oOLFp znv1$!GHZhEm7r)^(ur(+o*PhdF?5s8BrwUetlzjVT{g~lJotH6IhA*lHmP^Sz4DN; zUVZ;rX9Hi{&Xf}}DSvL*5yg=8D<(8stdS<@HHC$knUdui6Z;pZs|8ivdaNtod=j_LyEgWt0r*CgTd?%?koK-(f%iOm!_TXh` zV0%{2UFbg>@g_$z`QfwaplXj*dIPh$D;c-DSAI8^HxedHi1@=66HPPA+KT?NlH2FR zF}W95WJ6fo-Q8XN88fVU7%-HO@Q(5N}Kq#6@MV;InOE&kGWAQw5@5MJLh|U} z^Jc$h{n6Alx1Ih5Gl^RJ%8Odmj?tqm6^J<|AG&4Sb*Yf2^nLfotg1`B_S3C4kA4c> zz%?;i6bWWa$fgydhbGKaE%FJZ>)i zl9~+htRs2Y3)KcG1Pr| zT0-@$DOC$IoTHad3}#vq*LWHFUg}OwO<68~Yjaf(^tz8*Ie7T+1sR!h0CI@R&J_*f zYfm#V$prh&J@yxTdj8~UmSn{jrC(3@im-R`;luh)3 zL&YCIK2Z}H-ts=xqsV{&;tm}%b9A=)^Yc=H(|I{kdijDvLd)H}#b3vK$16u(teZbK z=Q=GOw0U4KI6v)CWB98H1w`Dano+Wq=WcYbDaf(FBF~ZsFAWqM60{K%6jWmSmJvXN z0}o{?@&BDgda5AgczMQZ$}qL1bED*kD1ErZj%(p+M9|~a!a@sy#4GXL@gU+dT67w? z?j*ugTQmbD6;(3{19-XQo0)B}_WL(LDaRo(MM*=`o{W7RH8`mMyj?|1rhsnRiWBaj z&vTD+)TQJQO^}c!WkpH&?^C9pst`WYny@eC!(xg-JOnW&?7g~>?CS=ZPFi|;C$v(; zZ{fzB9>V>4T=z@IXr*(h^YB%NN)zC-WkQGqAxZ)bPW zXW?^HrSq^nB&|RzAOVMTa^gu!PDYk{J9r^R7`Q)ln|3gDyrNtizy+hqRP(Kxi^m^6 zq`hoXLCGA`p1pQssaPG*A~&~B#4B4{3};57dHS2ICkLapExx(Sh#iiKin@32-gN-w zfyYv6(?t(CDmFGYgp7;~DX1bnbRCl1Q0|8F){>tmQ|zV(z0f9-wX;^1@9~H}YQT}Z=;gDti4Ss~q%KLCR>LJj z0!Z(PLH~BCXxo*0g!pbVE_r?9mj<=Br%Vg7oJop$1{G~r1Uto+m?wk`}D=Jf9iJ{N9zLi`yCg>Q?@FWKX_ZZj3_-JK;%p+X9 zmEDXNCR;njlJBEmO3e)l8@71gy0d$vR(o_fi<9_7r@ zUFOo#Cp!m_)Yrk7z71p)#) zw2-vE4wqv>`**^PL8hlkYHeBKD2l;&6WgPIMT6r{)2Qi&{4Z%u*vBCSQG;9}u%<_u z_;y%RLVug2(c`eub*xdcV&^@8ET#eUmd_*2`y{$D$6%zz<0>OeRvPRnza#o5J0hDj zTU~uZs)_BH)qROAL39$&LayV%!w`2K$)FSJ6n_QG1;fuitKVHuRa=B6ERz3yZ-peB z(9QqV47$GZoE4Rre^_Y1>`FM|#k*)}&Rpo(K>r#y2yU9BD1`EVZPqt_HSO$gr~({$ z+COX4pPo(pH^t(_n)fP-`S{ z8uhhpfdiA*L%0LB2G5KT%eK5E ztcDb5mY&&vXRZ&EmVriRFRNZ3tFAUqR#dqQ&Ukh>V(=l1>4y@b@zryRPlZvtizSh^ zMLgMAUJX|A2q#q-^cDE$QK93=JT%Qs8ItG;QMBZpX5FJFlX09NErv|BJanwJ1nCtz zf|PmTTI7+cgjwO_Nh2HQ^tY+)tz3W;RxTP%8wI&-5*3D&WO*uqQU|&4D8_>e*&1|p z2%@H-yhat6!HY)j8|zOD2v1>xM^F@8mEC9%8!USTj!8Fwe@Spc^e^C_nsCzW%LBa$ zk`8owVw5b%>!b$rj=||V#ryrUla?&ARv5;KgF9L}vFwP-Y?z#qK5Q9%a-dm2M$a0! zy|{oS23!x?7>>tN4CQqUs2QFf4WF>WmdXTCUUx12=%fxE$CEnt0`1HcG_-AwM3|_5 zvBY!>4?csb@}RN@c^peq?#EkrAAtj2(0ks6|9_u_5e*ZxHLMt~$nav2#~1dK7eX7o z#C%dpS{^b5gaqhBlGMqcn=G?10LA-4DYkro1vFsfR9sT70iCXbZIPPgAnw$~!ZI2% z9O=Y(XC9H(EX<@>I0s^0wk^u(0&JU(5q88l5-~_Rd20K8f|TY4*LL-%d4zzG+6ghv zkUq~jbTm)4K`HD%9H#E0tF(yIqCF^Ql3)krIzhAs)DF1rrEIfwh|_j=RAiN$V`)N#R&x>gA(Fd-At zg#0b{HEIMdvPWGn(Uqj04fcCJ1Fi;G<725}R?u!~$@!0~_#o+oIFZ^(2&&o^J(##u zSSNMx#X4yT_etZOn$&r|Z=J>-I6hFNygpBoAtT7}vDkihgy1E1gzr4_HFnrDLgo=2 zr+9k)^!uRk3%8Dw3C~>he!D77dbdW>>F%a|^P={^k@`P8p~td)CrFCN2a1<{`R_rj zs0d{`1pXu#DAn{yZqk^i&|$jyPb2n^V|)b3(bO@D%50tY9I*Zdw40`d@sqf5XY1Nl1<{sN=vgqN`7f=?QV6o z7&$pPut|>re7do@NqPm0|8wiVM(#jfe{fKTe8)856tzy`oJPu8V~^|}J*0R-d$+KU zKTfH_d|}+MjXX3sTjD;yPLm%vFag{ZV8}ZNGXrLR9QgFOyu4e@H?J!zA3k(Ax7`qG z0x}_gf9MVwBn?REr+QyLNbmyzhCg&K=pY3JyMzQ2beH)t?;@KXz!2vH=4oJ ztusG9goY8z-Y9z!S6`C0gbppBn;#Gu=>Gc$>axXamW+%H5_$T_(W3(V{A|%NfKYL9 zaa{-Uzr4Ks67U=^K<7#q8Wv=1;bMEq7ZMX_HjsO;i!5&^j+9zbr5__obH_(_SSu3Z z2k~`vlBDGQea0)$oE*qMoF#4-|EFpKG&zWc{(hEw-c{vF+ald<*2ZL)EB^MQi}kai zX>htH;pP@kGeMh~#k_dIcKJpXb!}}e{S7g(|1LP#ItWVd#Ovw(oEUT58*mi4&@HTh zfaG@3FcWdLg0x_Iw6bn~W@aXd3=_#sMFms`HZHDvmE2`;>kW}&=cy}(40_Yu1H?2fleH>l z@)Db?#ML){r+ssM5FZ^4V6}e+4tFoZwogw-Cyd0E#p9!)3pg*&KEE}Ej4d!=fra z#VN2OJTLF|p8&T)T1&rIFQ4YDWcb?LaJi_eKB$-*unLFlIuHFdE~m>@*9@G#Plryk zhjmfSn2%h5FNA-_ol69Px$oP>Eq-??aQ-Z}*XPCkS2&!7Y~K8?HHzd5ZF<#1O65!W z{JQDX>H;@5IH+3VlW~nl*E&fnK1q;u3eDXrLJJ58sHm^LGAynwNx9z!!TVnVfX}w) z{4MP}v#;nTSFT+ph7!3lZM&N%6-g=n4Y$zS%3g(|RhTo6A3p|$A}<_-+WH>SyRy$2 zE7YNe>oxvQ^8Z)5|Kv)zhd#9FDZ( zS9V6993n3=t)&NMq9rEc7px&Z08?#aTl8%2l09UAk2m_f(r$`(u748IKO9$N-VmT*f4?7PluRhu)72is?MC+u}x7e*^!21Etf#$Z5 z_KGawzLSzt1Yw|Yq|U&5z53ftRdT?cYu_qX(?L}0zI;VlApL&Z^IG!0vmX(j+vzFb ze@*x50H?RYj7lB|#edno8Yc{HnB_^g5y3{Y*zfY1_!NGlSo=sAuX!t7Ehr-sYoF#H zQOQ)+;c<1Sw(Nq|^V+_tP>+ax^H0IgiGw-_P3MOmV%xT-`>vi0Cm0y=XfMm!?wzWB z0%U@WPyH^%zo*Uey$Y+tkzVUV?2OZOcOhJS)81Ytsz3);R$On|X(3e2+QaH(1T~hF zBH!CPBS(T`HL~=2@0~kYOdnF*@7C^Q$g`?*>(WNL(>{3!h=VP~tZ~OCzOTLEe!sKD zzh@eJ2oS*bTa7MqR;Z_JI6$?yM7S>PO;JPT6;Lw^dUX!Mwf5_ZXXS5M386I;R#Kk# zMZObXfODJ`F6$)5;vfm^DjG7R)mCW#aNyEw!!WnhZ|~fu7vJf5o<5s>X6b7=f}N%5 zsKAR5qFy2H5CuiA!;u$N3XKSb3#m<88f^t3?@18d2Q16cxiqk>U(lOwWY{*?iBmeO-q8FT&isD zixnJ!sDB9$)(t6tJ9x?)Hn1i0;~z!+_^NWZyH{>hv2k+Vtz?RB^VpB1H>wBYD2S7x zjU`*uHxyEvMq>%Omsvwn`6nwwmtF_A9wE$z;!hB!-aeNSJ~bvC->nxwwzN;8h-Lu6 zOT$&Q&ZxwmD|~i*eJpv}@K1gGDcO`O?M|xc6C`k7D5Z3J&uL8-9pwbvIUuyoE%hfJ z&8JTtXMk()-QeJxQR+4!L74{+20~Bl{MdNjn0yfrNu-`p#a#YVLnA!d8$3M96!FS2 zd$~NQYv2}CxsJ^a_OBGX|EWJ#iQ`7a6jEs^yo`34k zxb6AOrqt%oU~};vrvfKhD1ZS1xYy-kW+pbGHDG=hM3=HSSLv- z=2z{dUnK5DHH^@|`TTj&wWI6Qbgx|U_XDpWuf19y;>{3sJq^inJ^!}G1pqhWau5p zxw4SLC=<96kcl@r;Iv<31!ZfGPDj>b76kt=C{tM?ULQ#Myo-HVA7IiD3M7q$bEXF* zdF)Md4iCTczBXDAJu6bLi}xFx4#m?Ez^?=EtX{_1yeMF!`*01x17l!c^~$x<^n?!$ zus85_9d$sU`995)`%*V{hg*!{Of@odP$yj7TS7c=?%~5+Cx0`-%-d07kCZMf%}SB@ zv%t#9Kjw!lgHI6{!I1`QbTTndN#A$zOHAcmy0vOjR`V)j}2T?p=AD8@G9JGjYzp!nwJN`6V=jw37>#{S)z{JK7?d`}7JS-x;>AKmYP8V(b2A ze#LD5Y1YZ^ypl8;tE)MBxs59>A!f$Lc?qK49^s+A#NqyLZeu#qGtRYAyZuZXW=_Lv zKOE&Z`2^^eme*V6AA9}~d-`p5Q$PGL@d-f(!D=nSif}kqxe5A2*e^!2?CXbY75(0E z?B?j2tikJBgpB<8Xv}i4k;Os$t*T<}SYQu)+^ffI z{IV-Gi|y^G+A!pIw&IyF<3UtZS6c&>9n6ogRG(tc2G*4uyyW!!rQ$sjH%4(q%>4|| zTxfz$Mn)80{m5Am_1kP|=p?Dd|ljIRS2Z##c6-J4ZfrC~xP@D#IlmNIg$^Z_2JLX)hMrxJEuM z_72=*EkALerFbGjZj|3t}L|H`ECv)p0{P6?ysRfeUNb$QKt*S?ye*WVm=mIr!L z^ZZ-H%+h>%_r`bQp67HMw*78=W3}8)T*`D}(yo-=E#BC!mJ0$ISZiG-N)>+7H-hsuH>|l%jDD*JLDwv-OE$^9$M7W@qbn7$HQW%P0XrN zqFTNjJGqo@v8V0XUg46u*Z;Y8;Af*r;{5M%Qw{&mQ=xr=!~at`AP*Z{G_XhkdPj_Yj{Bshudl` ze$gvqzq)#Pr=P9>tuNsr0ieUa;xCMp9}fJpz^~NTfYY~Cuy#-QFVJje?D}Xr#>xr zzjMj4?AaBXdS7Z1(~Cya3{UAf_&s#`f__p3A6V7@Do0fN30(M^LG3kp!sp$~eEKM; zKa_M@ZOjY7Ye(*R$B+I11^J@AhMvvEGm=Wm_x3{SWr=<+Dt-);#jVeMzdlJm?4B_z zEK}>R^O0$37`z<)as`U8w6t@V&0i%{GU4I$+5bP6Fk0;2@W)7ni?@P>I(ln?zn0Q% zQV$}-c_^X(C9`H68JrDOI6E9(fN(a1vd&yP{k-=?eNOy8)Qhc|+#F-k7q6yx)Q%y! zu(i?KW5@8Li6-=rOalLS$PPzdxt$JNL?F$=>614koP%;!OWb6Vf|Q>6UoQZP1_-+r z+D?CK8-ZMczP>jjzJHDLAYScl&D8)QpHzr#Y|Qi}XjKzBJ+dOZjmjtHY3Z8=0f zZp^kZ|L0MR(gpZ^0iC1N$eY^sd@TZ<#9zvVd%2R3sQWV>I$p#cmiE9P9|@&M&O-WB zo9URrU!iz_mc1Ju-8MA$r-+G(0iERh9$JMeC*}dI@5DEbW{H;K{hp@EypIrEmo!`mtq1wwQiFyqr0 zz;9>()@6d+UTz44eh-~(zkp;w3=9mE0J|M3I-%{W z4#^5#HliNiIbJ;C;nSxt>rNXQ86_yc_TZ*lk@f!e9+IwS&lNy;8XO!9btzGxygR?_ zWmjty$g&moa%H5=VZgprDWj5`WLM2!LFGj*lz_E{p<`~o@NDK;d^|gFi-2@^FFIOxUW3)j z6~3m3!XJS8Gl&W(cnu1!!!HmJ#eR7Jgy9#})rL>E|NKumNp$QoCrjjxHq%3#r0PE@ z#{VS#4ml4OwH2AwzaTiFjE$qVwgO1^_05}zrol`HELIuH*711!!^Fg>R4HF8D2Ib3 z8olxp0IK?Ak9S?=RhpyaJM03o}&OS^|zW?S8lvOCQL2U{!I!Vbv$i&jZ zuNPruWxZ@%c98V2+Vw0Hmz0JQL%vT>>!w7zQLV^sLWwx-LH6@daU@539!f9&KmTKe bc%Z(xjc!KoyD|JaFN7*mOR-4)=I#FlZ+tR< literal 0 HcmV?d00001 diff --git a/assets/images/streams-and-consumers-75p.png b/assets/images/streams-and-consumers-75p.png new file mode 100644 index 0000000000000000000000000000000000000000..a6a7c9a083588cb5306e5725379fa14f05a7bc6a GIT binary patch literal 53849 zcmeGEdo+~m8wZR}B_x$}(h)gSl4iyjrj%xenQ=ZdCa0M(2g4XM7-JHJ4kE-Zl_IrE zC4@=`$8xA7sU)f7luCu9D2MlYYJb;y-?iSgerx@HzxTg)uf2;oJkR~y_kCU0=lXm; z_fB+jpw3fSp@Kr8=2_D$T~Mf5>L}EVo63stO3UjO0lfY=+=?D9lyL)tIVe42^538I zuxP$0EL_jnQV)xbjEpo2kT4^S0!8o}yd@L`1abp80e|1dqOs^rXv`)I-VKA+GbS1v z!yi~vBQp~`72au=XGeNv_Nwp3E{(Xd8vL z=DNseXr_A*m26LQGV%11(`=Yrs>t3%LgWdZc!3f$-%KR3mSUL3fh@8qf#g7e+gZDq zN~yMX5s@TXu)xffWkck-;UZj#Q4%v#dos>iCilX7+R50FCJqui0>dsK0E3YSxS=D7 zflf>lYcFe0WKDwz0(XWi5`Ho!1QVR1I1-^PALr%iN^r8KMA=YeOv_N%NF)d&j^gZ2ceHm7;xfEUT#X4{LGZc@9~b53=w>UDk=cAFMg)&S z3JG!xkl=a2R0qD?0Y1WxaAI1zlFS16A}>ZD&&HZBFfkQ{SOpM?5-(bmi4)erjKU!? zXg0wdFPyQEY08u1FkV(Bo|o`>qjdxT7q=>`4$o6DD#galNP-L!bES@aFQDjsvB2UB#l+dWI;_$#IF5An@g&HPc zN}U2FEDXUmfX!#y(D_!}KuLrx+l%Jz?(8lK31&qEl7oeoLSe8-=4^#01i&bXQKn(! zK+_0mFowfrgxQn1&MsW4t(%9nmyIxzinb(1@`UydAsCU3m9uNG9fjy-MhLMdTfyj@ z-P{};BOD?1P>^5kj9nc}=}vOEs?3fh4W`PitRqQYw$3CGQ%JIxW0|fbCli{xy=^3e zPUbs!SQ$t0EM2hzhMkyX%)_HYY^_XfMGg|CB_+xhE*Rw=Vdg6Lbhb4mdIiX>MQE%m z+tI~D5*lpD!Fn=notPd0Tq}E~NFWI1^Jq~F8~Bn*uz*G|wPv7goj7bqS57EB5>2K! zM36a5ybV!kY0C-5+FRoSgmON{!^$cUS!;ongB|?s9_h+*u}7O&hni90N*s4Q0kTpS`2E&Bs8pxK}TZT|f za5jMyPe%#7DsXg>%IJ6(8y4C&B9v+@1&h%c1z2n0Ho zP8NHHv%)1zF4@g7M9PD5KoNzB>5d$9IDAwX$&_>CA+8uco8*DBvvhW$W9&psR})iW zs05Aiw4t%ECODzBYZ%L!6$W2$rMR0IvyG!*zHL2%q>iC*Hl4K@u zcfOEqOt)d#vzQ#2mqTb&kUfLI51=wlm|;vb-ksu!r8Dd~6ub=MYA*?pcsa4i;aHA| zV?-3r#M;gdZR{2hfDQ}5I-7)e(9swg)zTx3?!t-mK#&Lt>q51+_vCOqjKkbz3^YN; z4hqLeOqd)Nhu|njhZ!3Ok;y`PM^j5Xf~N~sYU)A{bP_q4x$tNtCpv*kpgI_Hsm1{T za;WJP5y6rGd7k7-<%cjab}~<;Bqs@lVHtr%i&<{is0jFUWK@I;0sfZ4 ze}Zs|gK0R0fb;O+xx9P>KNE9hzN^x$0D6kY#xTmeVF)xHq=Lj9$LWLY#4%!pv zNN}QIolHZ;_y{T%XX+jt=3r_a#CH(5axKy52s?)WG1Cdl6GQ}i2GbdIzBIzji4}o2 z;YtILTf?2)2)Ia37deMO;KK_NM>D*H;ep&Qf)GMQ9tw67#Y5s1LAG6;ND)iCBwU&$Odp9zBt?+9QX$LLAy^zJunG$`b|Yc%;jVTVYdXUc;|c?GbLBJ5 zyc}WrDXt7FTaia_a2Ojlsk^OBn9R<^F)TQO=4BFTj9>tkOvo?}qDruV*49B@NJs>5 z7(!8Kn3M=BM8aZ(4l>tJsRM=#?1Jn9q5|BFo%vBr2h&Kh zJ2Wz;uEr~61h~i4+&UBic znUgVz$l)+7u_p3xwihW3?V`MZ_CYC^9g@tf}MR?P2x|eY% zQeTiNEn$S(I)sUkfFoc%-AJCM9D6SERh%o=S&k!kQkdZ^Hyks<*_7etfDK?s&7@9v zp);2$5eMPKE>R)m08`fx^I1!c+LL<-}?aU&htmPypQ!hEe zi66{1rC_bOkyLjHHHw85*bxA7kqGuQPa#qRq;xh*!gaF?6A)SU!9iFroO1w`EtS(` z5w21+-Yb+yaFyUaL%jfPk!T3yvLSQAkqs4&4|ZXN;RtqQ5i2Ar$WtWZdI{-Ka=MFa z2#e)JG3B5=Xmm5W4aS*^5k`jL;k#y5LGC;c63&$4DwhYDncxK0?yO*`lu0vzP$x&= zoXjvXH)NLO3_ONs8*b&sCcD~jxy)dKE7?tmq4OkB;b>bSA59Hnkt0LJAuNWojiUf< zhC>Dz6lxkwQP(#K@>_0 zWo=1xlb!nf#?Q{RqtbNZNSH-4!Lvdjk;+?=^s{!8o+c#VvWE5_IG}b{TY)6gJ8t#v z%;wtx%V+Gi>`Ps`SS4>}xB6!f?`~$7pFH#qaYwa}l~sN*{LbGzM|uXkZf;1@3Zvx?JKdRC5(cxk9j#m z&m1jsD=H@Qt9+fE7fK`&f9#3MUqJ=+jhUNspM5;@>?7LLw5{)gK~ zYq+J_Teq%kZ*L#_Qr3w=H8nNefBcvd82Ir}v*wBwBnD#zi9~wW*SGJ<6Q@<%qcsLQ ztNg-F7%rU)*H6pHh|kTLJ} zUq@_=Zx!@$Zbpw|QCb@}&P~RwKl+f96NFol`MPX2aYtC#$jOJ#J31QMdN$;WjDI78 z{_?IS-anaG;IaA6siLCI8tj+1!q)96+Lip~P=kn6X`-SY@Qfamrj`@-<&_Dl`txVQ zoIICHq2GF~|H;g}*U)gFxk^52uxDRsaCiA;Le`xV^a2lEJw0as7mQDT|y?vBfKxBo6rez+$?0^2;4gT?mC5g`VHfqRo%z16JXm;Z0Xh9Xv zuR{=T856Tz`t^0a+mZWwjLLS?rc}@$%)nKfZJ!{t+F%NJOPlm7YF* zns)T)zNx90fPerkW8<*8ZwKF2>^hK~tZGT2tTixDkyotYmY%Tib#;yZ{yp5gT()rD zym>Sm8Y12Gz>2je^7`?6A{d7$Yal50cPk#H4 z_I9ri;aiYVU@!||^gA)MYh`6R@Nq`L(u#_T`wt&>cXZLUD`CCiF`Qgn?)H!V`0-7+ zkX-ZVk*dnvxeJ#r-Cte31ER?G@zbq}bKvpc+=cTo21Z$ z_W9NIB#WZTO5bmE4o5H6P)BE0_1(&2$BylX$$xz(a={(xm+rm0_Vt`iHSMK5W@-DFWZ%AhG+W#F2LYsr2on~I^`xU?F6zVJAl1f3W#-J8?*<1`dwT=k z;vPANg&F_&+#a8nrUs95F92I3xc;qy$K^s?t1Vo(_v%$NGRybw-Aj)^W=UInModi1 z!PHc$mDOwtg)-J7y|dc0(3=<@j+>a6xGL)0{`#hP*0X2NP^YSRC?^Kv5UeiF#AJBz zX`#=Z(BD6V*5kF!#(4bFg$ozn6!$0=df$qJeFht#rSJD#Lxs-nZdPz-M_Q zUW~`Jr^5lau$!)j4DUL5?AVQ7%mR9h#mW^c?(UtXq^7POcl@}o*SLH4u7}}*t4|(#UoqR9U2AijzC7H#)X<@(rp94pRqUhH z6;UIUr~lRs69m^^U0T&qCQ^F$x)&B`rsUSDWX$~ltew3*^Y@Q04nG0UrAud~DJIqxt;=|z z>$zd=+O=c9uVwdn2c-@D_@T&Fotm0*NNxWjU4PUDT@N!b^WD$Ef5yhK^-gYX37MPy zjBe#@QA~vSxD#!@ZQE+p(A4Cv?T*pHhQT78;oaivoeb!GxPD!BQe06(! z`-#ERTO-fjzI}39tr2F{&hE_l^ZPc3egEL>`*Cz+B=(8vfGs+!35_xq-miIen%s;-Mf#X=`gwOv@``+;i%> zg`WAX_nVtDCul|bw2N*Aix)2*QZZ&H=U{y7XHT}AqozNsFXc6>tyuA}Q)kBtT{nXz zs*1DDo;^EeUYzj(h9NsIlFQ{jjkk_z4UdoTrcOcH&upx%t)<;)9p0*^JQH@Klc(o_ zn>Wq2TVS#3P>$0dW;_VEdtia)n0ah$toJR+?Be3$=o5v7M%HQm$u%7vE9cCaW7POw z_QMRa8G%4ZM?ZW1T;bhMmF%0VTUuHmg&bfMyzdZq*UmX*9+ht4m+ORtQ2Jm3R5^%I z$SlqM6)}@5PMkO~F&Gy&Q$tm;at-Fm#L&G3(W{$tEU518OJX1Io4d^aI%E8&0fDd_ zc@fgO--W8G*mkWNjeEuH1}AUtN1g29u-M`5eZ7;01_tR0-_OS-^ekz6^vLeowQEB? zXKsEypCU||42c=*QaVKl+b_x?<)XCasl#Tuc=2M|=V{T~@)_{E?P|7Xn7P*_JKKVe zH(Fgcb1LMchpKog`(`d#vgH1Q2VISm6C=$=ox4btygwDtjJC=wbkFqUyP8>$@oZzv zzP`TIQS>KygD{`p5iROe!;_$&+W!Y)`Kl4`~G1etWvD^ zm46Rv6DRg6q7GAL)h{hPbFj7)ZuY>ab8~}DhU~=?jM~8^rAf;+e~TJ?Td}ib5#Pln zy?5JFKJIug zon5ygEqlY#jd8G8{f!^Pw^Tlct@&;I=U`{1+gRO~)oa$&r>x$3mEADkeD|+^l++JX z?q%II^WB?u@+65)*O=Z4h2!Yaqc?Z`G`fA~4zrl9yPFq)T`qd~?CskW82_V3j|^Ap zu4;a#Nsw56J8>Eoboxza=Ykz8)@G`xL-|2Ug+L$}>JU|ReQM30s93BpqxxwBURQfJZQb$Cb1%UL)#O#xeTgQ zW$eLqmQb;dy=j;;z6Q|pnvYYHzma&nrsKCtcdd?&&wf9BPt=z!>;EP$4ooU5Gu_i!<(iT^`Yg|J z;=1GMEp`2%OO1@{XUu_W$oOIGXk(MEFbDI<$ST^emg~88Sg_~iciQX0x~dsx3JX)J z9Cq%^YQ*~84!4&4psUs-ZHkZ(Px)5G9RBr7o|&q*ZXH`9QAg=|;%lw-Glnm0ZJeE0 zvFqozZ^&eb9j{zXIvVTj?|-IoF!O#$na;+IjhqbN?e7l1)Gyq+em(zEYHFsfxpd}^ zojX_hs9`@NKH;UEt(rn!mywc^()gCT zhetGYSN**H@nc}+>mxeUvFL`MbpcrG{Z5Y?8^eA*+PQP59&C!7xQ(ndWmVO1ucgJE z`)+n7#m=gZS6XoD?dBaT&`9h;ip(4Q3b$XgO`bqaUBiryKIQjuWOOv**5;($KECC$ zy;rUn=EYc~H0}KQwjv2&oAsjY!*|NO(*WW14A58UI-D}=3J+F@sUqD1r0}|-ySv-T z)%D(Q#!VIVlm>h;;apE-8>@Ker!XewZQn)vpn2-~gzPI9>sc+Lc{=;I=eZW;q`JEu9fy`2o|uTPd;|}HGW33M zFe!-)fL&2ZNsmY}zu#FEo6~G-ZQaGwx7huwcH{2u<%|Z+wZ&_1KTrK4c&L+Iw=^rN zeE08T7Ch)yOzg8fR_Y!Tyuv{^H9y~yU0eYr^>Ufdl^w?Qp@VO?0g``kkpXMQ;c!qW zfHjvE&q%wHUskpZpJo4ac^IH(Jv}`qgQHIQt;e&n;@^Z&t*z(y)d%B?3}^=@J-&^M zRM`;PR_LzEN(UVGZTF3$BFCpwmFLgT-ZP*zVStKHSTD|6 zw0mr52Gl|1*~sL3C*u;v2!OJBdLLv&`vMq1wJkAk*s)^8ibKQ0V!%5#g;Vq9&$mh0 z$nsm(pz;Iy;_*F-aau!zWoFnXU*Fupn3^h`cbRkEMvcoB>VlN;NSLU7h-o_PRva9+`V4qEd4d99xCeaX8KR%v5HIQ zNV$8OxH~how7(YpZzaEsonG7-*)SP=OABLR(VV%c=u~?!{@=Vj9U~(L$?Vf<1#w7# zWn^U7;xTGlmsQ11`${P7!;j9$jy()1vwdZt2{_*)-~DNgi<*M<0d{eTGHv5qUteGJ zio>>3FAFc2v+C^C6m<3VZG275%~w^|D<(qv-|8L;=v+k6@`$M`&FS7Y$D{O1619af zFr_(Xpe1=wm|1HM!lP75b%C0Bu_J8=mkGVm=Vph0Vch%PFE%WEx&hDE? zU-^_El}SwqzBLR38nnTi@66A5ppez1zkEhxLj$#B)46l!g669ybuJTMLsv)4UXb=f zk3JDy{B(<{>ALa91nI>kj!rmJQ(NCQ@t!#Hf%!Ai@TnssBN+-c&?V6gmp&Lcuy3Dr z_6XDu6Qw!B`dPNS@wT%Q^F52caUf`PJ6n#%C9wT2)HT=+Q<5P}3R8;zH^0JS`mxDu zqh-3NcOP2LrB619YfF#*5R)pYi**k++>V*N?W(uYz~E41t->A%W5VP`MFjA4rA=|yAJvd!^QBqQpc=A_7rp1l?4UUnKW&g6e2L_Ua9XE#C zZqN;1R_rS4Ra%6CzP9E#j&^Xs%Yc@goNP8(O~_udq@gYH^%}^u#Tj1`7p+6ln!67z zdDa@e6oC9O^EIniHM*>Fw zO*6wE5SLHKn>kKIFD>iyYnDJ4u+g&+g(@j6jq2kNpyM&>=r_I*+E-oNepGhsmDq2V zR^a8e4k-iWk%MbG8n)+~BWZt3)%m)ab>0^$sGg2a zqAOsWf9E*yczL=z&^OSZo&ylo&FSk_;YuV}zh(#=J)NjT*XU?-P1>2_;+DP-p+H?I zUX9jFoc#TBq{J+H)3S=#={0a4HHE!7|ER^=*x4mF{)`L>all4jv(wKw)hY@J@%`~q z3am-Kb#kH5q{ZN)65zZniV5AEUCx_e83eab1KP8c5R zXr8}%OYiqvhvLy{C9S~9tr3QV%>VM`%T0JtQSM zt#2;Lpo3ezdNn^Jgd!3RrkgG)p&M$fS#vDm`1varpqtv}V0!b{nkzp_fgZ_xT^>Y8 zSrYD_JBI;GN!5wxENGWyQxoGQrUy@+I%NqjG8hbK*$2W)i>{BY=$lZ7HTdx96KVJE zZGfKIzHeEYG<$*Oc7Xb|C5=#z=R>{U>{t2EYxBpeS!rs6rDz;3p}#rDc1vMV{qyIG zUv+c`@#yu%bU%VHJC?CK6RxwPT-`d_jC*#4j?(JPh zo((TsS2=pP>U$1k=j1d1#C;K-8R_3u)&ngK0NFEHEk`yXG-vI(O@(K6U2bx5aJ+6t zQ)^pQWbo_PuWsOw(%+jD6cnuVu_WDJN~)42sHNWUGWd{n{#5p%xY|#gfPkC9x24}d zw3zrUT$_+~{fwVE=^%CGnl%(3pVgY$4_$@FB=XCTA3Hp6ZMFVIPCM^a;9(v;rUrPQ z9~?Yklx3q%GS~e7l}Y`7fSmn*FZusJqWk~Lxx5J>4gKYr+qYK$vUno6juUo$%=C-d z+rqB}jBP4!(@l_R{h`@WsOhl=3*gDlv5~AM&Mo)sJPH8uFba;HG+RhQbc6dI9&ovnCr=(W+Fv^~gqDPb9Z5^u^YG!r z!$yx!9&h3zTp+h}2I@jy-jSS~1*r9#Hq}9Uq?geiyuZf(UGru`!-V!=MU?8idD@#d z4`zw(xzsl_hBttJ4R+dt0A*!m6k8Qog_S;rW$y0o zz`N(fPAx~FA+Z2JLOtQ*;6_$R`v#dnE91dfNiyd1n>));p~Kx)!5~kt@d0z-k%4ns z>7z=zulPq9+4&qQI4r_2)b+zgD$5owoIzAWZ`(Ex1??P_#afGc2Sg!|VEY*b^}sx! z4Bbj5CnnSuFILz`+I#4b3aa{qG4G6~MP*Umv!sw=?PbEDg zL$3prhJAo}fE+@~#cZTu&l7sCUo0E{rdQ}!l>+2B5J0Hv$EUaId2TeXfkBS(b||;s z{GKnT&Ds@F3+n?J&wbS5l&$i~-`S3*;cDOKwOR zSgbM|VT1wY15LPcWm0zb8W0n-^z|t&ZvjXBd*Q;_D_2yZ20IXui=1~M0)}o#S&dpc z^WJ&e%(Q?0Nw|G`%h>PVJ$;RtaLo?`1HT^*mhVEI1@&i#bD~&j2UAk^r=~8Lz5`$$ z=oz@X6(_W9qdB1tF*>G)SmnI zX;3X^pMef##VzrP`ifvHE>Z5X|B+||$0gT%Ys(wermWk5cpYqhkuCytLt0hk|& zSqyRW@_Nv+e{A$FT)2>ynW?Q z73UOhjJ3tjtAnwsp zHVb`+Lfuq_Mt)qmD^!kqwCMK@P~RIEP~Ww zop%5J{db;LmMO%UsFYQNwR6;g*HqaT=Li4L^G-|TZPfP3@?ck@ClbD4INPlwOddIoylyeIq;l3X4kj~_83Km9j!veQ6<-0zWpK5&U^RMa>n z*1MMcH!DBPh^qy(3Opv63qH zd$k6c&DMcoi+_C&giZ7D*}8xK{Z~s*Z#HmM6ktbTha;prXjibAUcF*$KVf%)(-r|R z*V57wpq&yV&V3M1pxv6rYI={qjjM&a@izr~nCtj?cSY>*#eLJC6R+~tw6Zyl@K7=%`kk8*FVffKX$bagS|Id0t|YUtd6wW~wgA}DLj_&-05fdX`j6JJ|EQ-PjY zW97=8pZjG$J}(D4phkln9Be2bTDsBqe_jB&9KtFAp~!XZv*3Bgy(R&%#M~ob_5^S^xsa(KIy=J) zi;EXRg#a;22?Y~v%zJzW@@z5&G{sI%)SB(l%6@)+56WR9!Saqgtb(%sr?Nw?fc@`4 zJbUZba%X4fi9@v$6XrRvsY(W5qsFxgKuTADiZvga_`Q4g7LClf`Eggioom_|h|m3h zBHuaNO&gLSBsoNWgl-A8?vKIF=Jby>BF*j5V|ze%?-}lC0B~Ry zGhqe+01HGW&w@EmOG`ufYt#NYdURN;<;9ErAa$ME8aY2_`|j?68y#OD|6eRB>R&Pi zH2})l78B}B6jbvgM;2PA9S71mG!(6>c<*u+o=~`P`}Q?eQ@=K%{GpkHMS$cz@GU0x zQmfmR-_$!8W7E;YHrLA6&5Fd}x83gTS*-iQl=kS6ebLD=1yM`NKR0QU;fze7KRc*zYZ{D>MSd!`TaU*4HBXE<%G zEiE%NH8mkwkNx_^g8k~(Q{b1;XK;G*+#9o>Kh00w+10%=QxFKh-oIj9J=&m{xi>DZ z_LoB9iCY_9k5lJKL{l^(YnIQh;Of}c!G?^s*y&G2R{teZ74@?~YUYVQ3Gpj0T=33m zztJ8HF_iIWn@9!V}*4L=#H@#Cf@_rszDe#|1eZ=k`7tgZf+RwxWsXwuJ zRY5!-_T}}PMdiUsvu4ju+&ovEwdek=TPshP%vpcMaq^^<-bRhAHGRM3%-yd%=Pmw| z8vx@a2RaT}2fY^`N=xNwX{|_U3Sx4}t`x zn?GJ^wYODyoPqtPteyYd{fF^a&0iLsP2mIgwZUUyZ{&S*Pl>J7QCr-pf0Yn*ki7P?<(PcY)t%dH(#`<^KNO z-ZQzmH3e;ke}Q^&>~stwXMQCYgGf}^+u8K^QsA?*JDlk`H){04_3^@op~`CZ(>M3R zju5;`xc>8q3>%uMq%vG_{(}8V$KY50h;_R16_ghu5W`DPMLpYgRmI2eAD^MBslulp z!}H~pq}_X}w#8YJC5 z$><=r?A6Mr72_Y4M|S&ud@qCiHU0R-Mt2sE$Abn_ZT0HI(59S&j9z`WNvlpj9+dk7 z9{E|;d)TpnYIBjK$4eVevGc~yln5_(Ea?9>yv>| z*>Lx4*tGC=Rp4@E?M_o+1!0i{HPJ@oUZ-#27BKyC&v8X7&cYoC{@x<~%e>DY~miA@QR>68+W zLTT1nyLRU2=qOT+0E7TJRAmj2^#3!=x;wbaZ|BZN@KId*d>dRV`vI+)%Z5;(P7MtY zkH3#-+q@9GC_wWH!!ecSH1MuCyD&w;!3n-aR6v87h4NO!>;w>oRMa+-rMUao_e2Qj z03YYHA#WD{Zy(%Nt%u&f=c=e@Da=k3t<(kcg0}KZfN*2wB_-I}x;ny} zFFvZjihL_b;QT>^>Cd2I0KDlZvWoj&96fOYk@Hwm>C$SjcsJ+|1EWs`y*4^J8oDVZ zFg8J}ny_yl;tT*K6SnZsw{M7`d;89^!kNAA-;*x6t<};(LA68f0n9D5Q%GZT=goXj zi~#C`?9vTw?eP;Q5a|v8k&BB9dh6DKZ&7dFyis1T0PF<&Gc((MF4sk!Id^UrxWb?x zNk82i^x@;jy)d1DftPw8pF`@yz~Q)r{`AJa_3PK~pimSw*g&wpCw7Bm2R#iW2~hMQ zC(cCm_Vo={E{@ZB2QEJ7J>wY#V18L~cz@29faI=;@sael=X@N%!4s}tSS%Kx9XJLn z&S&9;`ipfdg=rVBT(JxfzZUQom`kn9#;86EfDaG^unvD{{5-AU4=p-Sd#6^rUvi7a zYN$3A4nTlGpzhaFR-QAb26nD)Um%~q$K_Hj82dn5hZvk!+OA4ed-CSZ;gpp53bSW3 zgOtR^lHNpBk-2(E9L zod}!M(#mRhqdKJB=#q{vjfMI7`Ewyb1G2ptGYS38iGLGWfc(_e)lqwXUczQIL6)C| zTD)}W*^3vIt4VJhAeUafss_%s^e^eaC8?@fIytG8S5$~cKe@lY9WfWADo}a%!Nh@@ zdPCTuc*{FQGiGcC^nnh|`OvLtDNksC>uh+p(j5P;Brk7o)e7ILuNH35H$oNKKs;oQ zp_#`{t%4R8oEm15US;Tr+}&F#%aYVeY+ws|4OPsZgSvR>(%!#~Uqr;{vWhk^z(Jz{ zd8YiX=JMr;r!wz7t@`fJBd{i>Yhg@G_Pu-Ouc5l%-Bc*?_Jk0LnH=>*Rl^`58lc<9 z7#qg}^aEhH<6MhBgktKNZ3ujQ#5Jxu>kR%KFfq+UA=>ZDmvrPo6B83rdw_}Yu8dv| zN@96@S&cu^xq)6Y0|im)w0<$-vp^X8ty`18{R50+P2ULI&9_26JF7`?A8B|J<{)<) z1Ju&*^2yu5SKov*0FupZ8;DvE>k0FKWY5i;Q>Q0kufL@{3C8QJU#~oS_H6KeuB6+L z?z;dxQu1z>s(Nq7<$?kzWQYiBaFk|e2V4G1OdxBXJz?oq87hPos%A#!llCq9?E|3LatgO2Q_x={VJ zGaf+5>5a@LRznW}trx=T&(-A{ZKiDs_foGD>H=2ZQnOcQN>7w;Xo%^YkqHe7nBB9$_A9>+2CT;P<+pwSiF}wQt@v z^lk=Nt4iOuo!25(M#6G0U%nh@-PE7<@d{UmYQSPL(YHBPyk%?EKj7YlUr%j`5VZ2% zZ`(uz8GQz1c*y5|D`14YX7<{f(1Q`6j>3izxyNk~hB z>ND7R?&3v$`vsfsrZ2UasyAJUKz~Ay6AI!0B;ce41R0<;#IMN#$wW<4v-!YE_@^fo zjB(HgRR`n$IQTVJt?D&(WhNu^vzM1wd4q*^ZzIgJfX@dRBpD;ywQD3-ZPu(=<5v^G z3=Q<;dm;U%Jj*65D*8t2XNDHSK#>8Zg2#W81@{%i9>&R|GR?gYfVg+fPe#RW8Vna3!gx4 z*}Qo(5Fx|EKl1YO;7+g$>wmvPW}>g}mN)M&(VYwh4kp|MXq4#Ud2`jPzfG=roV{%> zz!ZR4g@JFfSLk|g*tjv{+NSN&&l)M37Aat*Cmb$a1zhPLPNf@o-9IY4z6X>86D;?u63Fp(0M@J}791_D9cQ$G)Jk|$$8@3x7vl*w-D(%@?&N8D<+ zE5M9{zZ`HQl>0a>;AEid*?Rb9$;k`$5IPeo1@MgXET$)nVFJf}1RWh6#k}SXS>hbaM6OY?LP~ih+1c?f4GK~?6vu6)S9Q)@Vgad>ro2R}f z+^h|b2%zX@J3fxekK0`^DT*|nomhuxgCH*fD+v-32sQ7(8jA4!(Ytpu%l@3=u+22; zy1Q>K(rN+*KxvtnxR<;E2@OH4Q1_w1fiRj42?`LS`%h=>1jCnhM1Z32nq#Mof%T6I zYAYlhR-73fl?H&Fx?=|l%3;gPm&)}Th=UF4aEvGambcLyb!dSY92}C{VOmdrZFmMd88NqyKqZHi1|i|lbflKK zY(`)jB9Zwu2(s{r0{|N*K&*rI6XY1M^8jT4JMt_{b58FDL&M(Hk9RT{PCIud0YLaR zwIDDs5Ihrc0K%bzgd7Fd4u$;Ihw*Sse$&L{B%E)#*1KiFDzj?G#}G6BZOsMhXQ4bm zt=SHzQFgcd!iS})fqmfG_3PTutO6x!CL7X%iZ;e~EZncND4s;4DT8|21(r={xMI9s zb4$DK9asPz-VQ6PI>q26+F5PTo>h~Mt7Wah)DJaJPc(z+yr=8z_3NWaaUi1Z|8qPD zDiHJ-ph|#R0_h9#10W#eJXTQ9M$P`ynEe0(LCG4{VqRy!&aKJ31gCV6&iE+^4RuM& zqV#;O9OpqAe0c2~IOrkmdgOx`7?OOnbf4y?%XAVO9*l@A=@neuU zKnR0(08l4D0fdK{Vfg(t5^6R!Hld%MZ$Qnp_eJDVH8m6zRS=0l1Rlg+e{d0S%5}dlIzMGnCr{Ckb7Sw8g+zL~f zt#Z3JekY`9&DaTNb^WQ2Ia7(#)16aaICc6K*STspfWUIpXJfnjQG_B-T|NA~e~v#jWVTFI>_w=}wAXIL64M$@(e z(kp;+;2iFP7j6Dtz?K702F@NCMVZz@;R+8AzcSV{u?^02X(Iv$ zAP&^rCAo;j_K2f-%|Khxk<82ka0mx>WTt7wLg+G>Q0&@Ag-6AsRTr9;>aMYfQT5NA zSIIsOI%I0>gbuPf(N9;_1Ym(ZK6~SalxrRi@M2S=xFw^)HZhgMNv6A&z4{6@5< z$D<*(r)YI1+PMEMx^VaI-QdxA7u#7Lm0`ME?Y9$Z2egYIed{(a87yB7CxZ6uL2W;Ew}CJT>@)Y-6M4nP8E73zVms=R5QU5lZCcU}u9cI=5{CBzGebQB<> z{L~|WrVcFv{0j#Mt9r}9DeyLB9IOPWrCUgsfqeh{n|!Y23>03l@FQv;QfF7JFqjjZ zV-X9S10*0gDnx+@?#x-1S#bICp;YO!r%y`)8(KS6VJ47+kI9&ckB+f-Q!Sq|Cp_x^%%5M(DF=z)+V5!Th}h!h97#81`lu86T7yDg0SIf zVPU5A7uu#(s-fZTo?dFVEo3$#*mhvPMJYThEs;n9l$w8;GYrD1N;}XEnAwhPMu1C zrGyzu^|>P*b{GaR3urgsags3zWCysCxm3u?M?(ma!8(nvn5ezCK7Q z5dY0Z3C#nC#l}I_hJ#_7+o$)bm0Wj9d#Pn^H_?gGc9@<#qp+I10KLt0ytQ%YQ|m^6 zB{NV+=l$vxK+i+xsHsSkU>*PmuaJPbb7y5Tgegu7G)zc8kTuKWw%~BR&z-Lp6)gZK zpWp7Cuk{oY!C7vHg!%uff3_fw+Y-Yg!~)1BY;`?jk2oMU}Hr zz+wX%_(@Vi?}-1l$^ z%zx^gbwoeE9IDbB)L2l1-T}3Rsh~{M7Eg~)kCo?Sl{$AuEZ;qN$NzfM?$#1$nzi7Z z5Cn(@oRb5by>Q8reSq8oxZG|qy#gSr`_l?oOnzU4kbL>ADu5cmhzP_*HGo`rtUpBZ zsx9FMO~;3IS$~>SU~IpA>mC7r5G>s2K-=*5kFJ1{&*bGPZIyoB0h~{j*aMCsJSzvk z5-6+K6KO||z*#y?bu0#R#FYGWVAL#n3=K;L4J#O}jFwvkak+{B+W;cgJ$a%AkpWJa z&!58$vg%lSp`0!-%@TS~HudgQNQ8D6SfjMaf284KL(I3{4^EF3+*sZYVGc)(R0}+k zKze{?@EjZryRc~sR6-=?fZo{$QwxdDCs8Yb7A;N#{VN_=?i&B<-dW_aSd<2bCZl$SwTFU(`D-Lmlq_l<|+akK`coTg!BNqKfN`uiLBknQAJbq9>0W zeSC^A3v@oP=EAubEeJbkIh3wGsST9LG(aMT1`W>m&I0xcIC^L*$I;gS<^n1*YMX|T zm^b$fiw_znU{_!g5J`Se2O@slYnpsusX}mtd|b$F@u+D=(Q@4z zqRy1q$+p98wwxHtXQ{VlF z4vYCMr0iTs-2)!`ILHpDGaz0tnM~-Vt)Qdv^MI}!(#OOjDTpSpFM!t>{`zLCoui`; z4!0QY+WGJ2Ak4OHhe5}DP!5uC-lSmIeE34_S=h{QG&uM2Wfjn2{FAL;oh=G~Ftlv> z|GWSma9p`Ikml>j$vS?VjyW#8As*e-QFzs+J zf$b9_7Hyxe_49oyXlRk`efr!6X3S)yw{Y`;ucF_wt7fe-jj}CMD=S1ftdOYzY22Ds zt7?HHMxme%5(5J_#QgfW4hT1J+~r=lpa|z{qrZaz5wTDqFG1f9gm5jS#~O_uUb_Ww zA3h5#q5zoz;0Lz>91FiyzL78-4Tklk1Dwg9vC}_`oVLLKC!n|EtX3ECEu8MCsFasw zk9>Wm_I0%N+_x#~-sfLQpw*Tf-WvLKFpB@Npzg^_+m~Pb20ssi9S~R!IBbXp~fQ4CLlo0&y zbvTpu^=a&{XE`sPH}8f2O~EiQPnA0O-{!u%KUh7p2&sN@GRcs=7O7ynMyPJv%=R6a zwD;ri+W{v*<1~w|yp+W=Kkt$VXDyJWg9$=~LSqL&4~~^D;ypgK7cjJzGISgbK=_Pp zQ$pxfkm=vmA6^M5vrlKu!jdD$RMY|YjWzmB-95nZF;^G>ues0sMjKedU@i=Zozu^2 z`Tb-0)C;eDPcNvxRkexzjY@v65CwzvIm&}NO0(Uzom3^-u=J8IcvWIbwHAUy>bv| z0j(x5nE5eH;06nqN((P_0`a^A`Z1)=0`8dF4*oUf^wcC`T0&Z~U%&Qk1gh63ItHvU zA)S>?ZSFw4X)ekeA0O9in{D`dC31Y^k3jtD)hD+~U_Ayuy;UfTtA(`QW^T9<{x1_H zC!Et>k}hnIy<1xI`{q>d=Ji)>eQqxIZHhfxZk1I3bwD=-He$!F8MZH4Em= zLri(V*~P=k1HfJmO(d8ws6f$*emVaiti5?u&+Y#I8#fY}5;9gq85oq)|&)4%>yx0qb!R>@bOi^WJ7f^wI+Adv*(yW|WqqZZSVz~x4fV*xE`EI^^d6yKq zALw0JCc{uR8c`Soi# zXjgOR^FuJq5QCplH(q#+Tf6;@XUonx?hwF2eUQ4|_uJEZE$*e$`seTL^J{&$ZFFwK zm8rzhl0jOBNtO`QAZ-$mvL9|0B`tw!eVA1dj^a z$L^s!YbZuQBfJ*=AG9@;$au%kUFnRVc|^%F*=Pfl3f!KMlc53x6Ja+%*Mg}A{r&f} zxfbjXVD75;yLazy&AdU0&g5!!Dtxh!Em51#P_x;%aX%2{uDuG1`D+qi%q?e_(w$r> z?{p0oUezMbM|%rq2m;<^ndbc(qn+yZE7i?#-|E@T1uZrg59+g5T-tPyX(K^{rZy(u zJ3HTR{Kl7spIg2N*@vTB?tUBZjH1%_lZj1vUTfCr}71|(+WR6E$ui)-XUv={rt1ffPMB2`wM>$iK!>9 znw%T|fZE+ucCXNAGJgZZQ0f?zwcg9yy99Kbr^o&}4EBDv^8O$cd5pFq7iO_4ju>(1 z(j}$H?~%jLA(-youODOJJv;CJ3)(Z<^hhsNi$hu;XN6?VUA{acXxc%FUS;SA{zph( zG)RNo_hgppUwnb;&k7kL z^2>|(Y4NqFyjL$0Zss2S@neh0=Z3&Z+YJ^kesjg9>*a%)r3BW!6ko{qZY~t zzBfPa@m^xP{ag2}s6nIkUi-!D)W3P}4Z3yYyiYIB_G98B8IVW?aJv&Us^-T0G-}X1 z_dP)>h~WZ>SQlTcwIAPay$|ee!YYjySy|KIWS#H&vmB65utP1guFXeqF}0Hn^a0sc zh-+{Gc~M?NU@vminy};N7%E4*ie-`OdvN2|XLOU7H$f4pwpnW~{{_wK+);zOFY z#hLk;=V;NQ_8n(8Z+bsXyLV(&>CXE-zJB#;JKaR8Lx;J40oo_8$(dm2BaD3}`zR<5 z`MY9A$GS2*htho0&cAF29qp~yuRhmcZe{iP|t_+e2bKeTM#N9jHR!~&jJ!yL_65m%sMJ=)3hbRkHa6zR*&x3n`ut44y z-LY3Z5X7%ANByEi<%^7hWal#r;XY}=*TFTMsoiVBrcB20UB;nk~ZK2k{UuUg;G(CJxT)9CFQ zIo-~5)BAE={;})n#D*d{r}8(C4*6s(s0`>tGMFB!gF8%0spFCHTk4g+AvEVH7zb!y z&97W?=vP(E{_y^Ca&q>Kvhi21u6u`n=7tY+q1>z z-#O{gMvC25^p}*3<&z1L{a!`R+gzSAMy6wElAgZW(HwWLwDxE0+g8TF==X_uGs3Bj z^&FBLNVL5A$g3bRPoh_!P~a2vTorWnj6!bhOYEXsRH>NgWWK}aL+K~42~erF$x)$X z%*^Jv|K9VuOtx-*+Qz)ZsjCdyPp^*hZC3EQJ9a|%{PY>^PZhK@*KK%xV>cC%YQU{A z7XR~iFZjFrp`G;CY%J8<)NhN;X6q(B8^y`f-Gxh3)oupEN1oN{v+A~P+Qk3-71mSZ zXZ&Z~?de*qq$@0^>Zgc5I^_GGN1)eU-T43eAEmXTg{H!J`S$G-^2}DyT{b4ddI&Wf z2qq#5Ar5Shy;^VZH)A!`b6e2LY<^x^n>kXP9yL+E#qb%YK(O$bXl`oIMegL8nIl^3Xb0cbYNctc|zw9xyg$sqL1@*<{(9lElq`0RF zgC%GZ_&uSAf-jul)~u~;60nLAZcsoYSB77N2ZbHd@QTuCn+3lASr?w`fF+&)_i}%Z z<+qv2Jr4d}^;9)}&iwh)Xs`}NUxzJ2BTCPoKpQl}*x0$F9P|O-R)xN$_r}}}%Kv?5 zW|x4)XduMkN~Br9@#FeTjxfU)937pLIlCXcNq4%>k|!sIi+T-!9ECABblleJl+1L$ z9!iG1{I2puBFhfg6?sE=gf1BF*6?(>imGbx-MhJ&=|2YwSBQ;njeb94G2i)_6&^yw z`e&JYsy^q7E1rd?{)a2{3K_kN+7C;CD*d0s#Z-`4)!W?HG;CXkZS3zR6^Ukfy z$;=cpCFtO2G;%W2tJ+J+uo{ykd4s1fF$;fWz@(YXlLq@~Toi*eS?imI1y9&9Qg9Ce zwDyc9Baihh9zL0Bk5G={FfUW*q@L$uLzc6zkmF>n|2s4^EpzsnO;BsX`GaTW5<7IX z-F-=~+FECnGQoB|nH_vcB>Y%^z`IbVBZN~xbGIIPX&PoSN;7B8y7^{9L>WYBFw>O7 zECRX?dUtnJzIIw>o9)xN%lEsQ{jS zDZQL_r=gFMU|yG8XG~|NvZ?D!uz#-u2Z}%~1MCiTwLDp0q;Ql1HQhR?&>ls`k#y@p zsO+I*uC6V(Lp{Ig!=r9M@nV6{YWbyWUbOAH94(Qh)1wwXWO@3pLM#iM&I*?n-cavx z@*etVPZ+!qli{9S18YFj!gtnb@{;h;KmhVi>kBX=+f=4|_xK+#i)*pkfw~2IFUxP@ z&SaF5Z*OWV>Q@0bWvw5quD;2_SV|BacF`f>qbE-E!ygJt@N4Pi^g9i5O6Xj;K?c3I z#}2hT0$VzG*s!vY28n6ytJawfP?8W%3fvZ%4nRtE#{qRl&CowleP3DHN{!&H7302V z_!x?pJ0p!iT2t+!%}h)XQp(OwYASNmx1yr1u4&_ z@%<8>OLk;rP~z3AS2v(87ld1+!xZ(FlS?SHzvA7gOP3O&gC4)$d|OlTNln z%EO0OFI>2D$0dy&pLy|XzD%!PkIn0AR+e&da)Khz#**j3}AIc zebAmn=|_~;@K!aD^ywjj*>F*%`?fRjvR z>BhvIpz_3teSIG%6Z~LzKk=OCtt>VD9;c;!!14|~43xNV7h_K^$bIwf-67*(zuLN_ z$Vs7Hi^?^x>$cuE+3Nsn$vR>a&q#HL7M2)9<`nA)I{cQ$)!IFj#Dy}VxcJ_-vA`Cl zz5JDxl&%DhvMsppf9_mA3BDjGe?afvkBRA7HvK1Ub4*q?GY+oy&1nrrU({F`zb5nY zYzMo*76Kc0R#9qw=%CH|Ove9-kJMTS4O5!x8P{`K#rWx>POZ?>1ink%5x0N1oV3n9 z$-|Ei?V&GZ3O~rM3xnn(%=|vJ%CVcBIYDoZsp;+@6+uYsf`$Ob8LFZpR3nfibV7@x z?fSz!F=kQp@#8}>#fwr$*OsQbgy+WVU~I212ysrNynO?~EJ0DLL|;VNPg%%2(GSkN z2?<7NsO5hoE>0WFR8R=;18fOm=T?|6gNsM;fy@qHo&m^0{aZwZURI8%W?D$rwdE80 zb*74BYp(YJz->*Ox{QX)+JXo3=sIPb;)>Xh3@z8202<8k^{v zM=ifkZ0N!UlNhn&wkNyzn4jM)f-$R(`yjI%>^X8ED+*>wx}C>c-H1gau*y50mS$8# zO6nvRjfre=H;bd*~?z%-hb(I5&WXI|6dNG9bc-25^aK(5!DQ5DH6(S95VV#+CIYxNKmxzyV+PG597%czZciG;^#tf2&kko^GAU#wk) z6b({-N&lnuwYeLmI(2%uLXGAi4bdHJui%=R>IwigE?Ur7x-|V9+vl)#vaE&NlJH9^ z9RQ9QJtUt|F`QN(Eu&=4oEo{oJl1htVn${`LHO4+0@#l?)swm@@tkw7%v!zr@1OEN z+4Aou&t7uVvMm#1to{Ypm;9Y?f!`o6O-)v5*zn=gwKIalPQahO*{~p3`Z#7j%wpEW z!C?0su_)o#;8}u~pfAD4=8g}T2_zKrYO5Z&w*|9{G|=N+BT6GEfKruy{c4TUia2mm zb-6KxDh(gcWTmFxgiS?legm|0FdOeT>|>nJR6d6i?o+y-c&>|Xta#SO3uw|Za!KCg zo4jJ>L4&5|#g1S)bGv57zXPq#vSTGd_n}r@p-y&tGj_mH5`fE8oxt>`a>7$^& z2^$q%Q(1D~?h7X09X4;Ku?0M!Fln0reDQ+DT#O`MW&&)QDf7XFsGn61xDzxR8?m2S z8m(}1!yAog1q7u+&-Lo)uK6JgP}QeWaG1)nhrvDgeEM2e5S6Oycc()=mBcz?PiBb; zEMfTCyI;RcRJ~%ihAj(e=fx&GaNxkJE@HHfPdWI|9f~A3cj>*_ZwlxRJ%UugE3Ve* zZ2RF!wg1#}PuIkY)?*A4p1iei-ANXsm`5K0ABA|m~7wP=&4ZXsKx%Lo3OmOh$#b*8f3~g*`cl2d=mGY zYL^l>@EB#&+`X3?w%nScX5s$tlI_~jt_tz&{v^iwFI~Or1@?ibhjDOs7Cx*GB&Y~) z$oh{0boT2k3HQJl`YD`pZ_xYm1Gcl3>svmpX%z;jL&;-kyVk*Zt*rB5gft#@8sPBXQoN1uW=+VC9YbX80n#ow?Zf4w+ zvCOQTZ>X9cKh}kFwjcjW!F2g@Z#IqmDGJ2s=sM4W$bfuA5 zQ~L7nvZl$CC;zGmcY9b+P*4$xluGDxC`gOLIFo-m=5Cj>!av&DMX zY(l41ref^s`sm;dRekGr=*)QY48NmC)eDcx3^i2NwbUS3y)tQW z(&|sop71<=5EBzYD(v}PILDf_^CBJ9^npVL3_eC6;<1G{rjIK))LZG~h{f(ng$m40 zOfzNeinT^sk5&-&%;1$Y8)Y1fU`v4lmD0{*HlU8?ajZ zc|{km(vF7>ilb~BpaP(pKM9LGt}%W*&FoK41w`J#WlQU3h$i^` z1%+0^QjfXZiJ#^TUbyKl-W5jmKLbv`y}dqb7@Z!5xCwJo$HS}1P3S*#%$VaufxB(J zU}x}8yRdUo{?*xGHvW^gpM=o5cCo8S|DfXglXUSj(i+Hgnm$qy(g5JjSaYh6Vk!s{ z1Fx7^N4T5@e+-zM3;6_a@+Hgc+_-u>y;gXw7V%$T#+u& zJ;6^y`IM>pdQWUAEGz`60}mHvNU%ANCYTwPDR~$4|ATPV;G#<80xT3#nBwB8TeghD zNj5PjXJIE9lAp(eDd;VC<*EpwJ1YekO|ZiWq;hH+7%(tVLxz?*_8hU|MUcSq0q#jt z$xoji>@j@7ja83jD5*#?yJPJJ4j*pwDmBJ=t-4>j+~Jf75vwb67Wj-BHf&~xmx>?) zy+rf;&E7|%uEKoQ%$d6`w4A5%S+Zx1hW6s=K{s!1N)EOj{W#LI!HT!e>tQS@<=lbH z7cbN;n`SOtnB!?uX>s4R)H#6OajC2H)Q4v#-_K6N?EObwg0wQo6&u4v%A; zUC)keTWYRknARm?q#=2o7rr>cx)UvA!t?!dpezAV>kO?*D=O&v4xn(MYz)sH)u&%S z$B;@o;jI3QE8pGLhy-ctuX^81RviX-+g!!_kk81i?KXma(4tBk! z;6HrBogr4fam3P>@#|>lz zPhb8$iw#aW1w~KO5W485JFJ)K@Wr^Bci;=$+~T(_h0+&Y)x?%jVYYY6%gcFo2V;l# zm;x$}0s*#^F)~~ssYXAYc z=i2QBdP~*~1RvM8-jfz$^1VyR9)S=$I=_zVASfgdF5RLd@>UTo_3VokDZY`7X z9nAZZIKS5;KIZuGF`pvyz%hleWLRtVxYx*fg(Te#M*l ze=jkW#Q}Z>@_kw{f8$`|-rs9#2E$KbSeryk`KnKk_+Q_jUa5#7{8~r6sfePnL7AKN z%nd}~As2o#CxhGt+DX+hXS$NP)9!P|%JhO(7Ue>R+^;j2k_zQaeTt~cC}Z3T z_fJdLEa6I})PZ5OSKOAkIcU)7M~Au|$vg7lM}}lc*3HInn|&4~kt=GPp4ChtZn-C$ z!>JnJ`=DGIIJ>_)l4>MB*fQhqh3-YFOP7Tn`rIz#2%B@m&}c?%t+vK6kDx-FLS?uFgn{$8W`le)-ekrXC7+c=-Hs5<$h`xLB zD@m`*N8WVX7Oy#}DKD^TYv4?7_BCIHLoz8Ejdi~m7qNOot&z^}3ytn)c{`vY; zWOL=pbt*=8ENp&yT?=ekr!i2493NYsy>j&H9)a=h`phf#TI(o~i zXYs$=$o<3|ifv?NWS7XfkzfknVy93B@e=WDLy9TJZ2i(h5%_84|D4e~B=b#XL3%}S z?|ygtPe{7{^Y*QMdd4lT-+ayX4>mMf=5xj7s(6);w`*;`T(ra1xYvO9Wzsb_?~gYW zjWC#E(1KmTS+8EDy;UgG(4O<aKG293!(u07jy=Adrk*oVZ8=0f@A2P5aCr6?$V+E?iR%=@q8=RdMW4zMdUln;N? zcYunLq0wY!6S{^?H5jD5(WS*k>#v8sB&Foa=51NA{Y;AM zeZqclQG=`knbKvl;T^5iFs7aLp7kz1*YYc{CE{mY%lGBY9S;P33vc-uPGXtnzUTA} zc#dPPUypC!C8zn_^}_g*altv(c{f#UCQS_M6qz-)`}=u?J9ln%UFx3P#v@tPdt;Z5 zT_KlrJl?EcKDE|&p0`_geC?p72ai{l+Xt)8F#_Q<3kkhoZy_ypa;)sLyAp*1brY*~ z6Xz#9bvmSZOl^J%)e)x%JZP~|?;iKwdwtf&7tc2Jk0?FY@^g7hx#yIsg63SeV)Vfp z0~LpSq`gR9_BIv5$}RAIpQ5PzjHb$nlGp0LGFCn)d7bY6Wx{PU_vG;&$pKR*PH=Vz zZrt8jSLcvAqAB0aHg$jH{_>KYpUdwpE@$024Kff6K^rNVfG#otYQ2IrW6IR-9BFAT zXt^63^4#_8+uMl6HErMObKZ#r$Aqi^T0XY)4yOCV4^bhfOYiZN;%6|vs;FDKFhK!56 z8us_G%)OEZZ6X`yb+HM1vVB_xrr8(Ve6!OwKHk2~_KD7wlM0H-N`{1X^@YExby`OA z*2J8ET}`JIJKIX-e4P#Q@YkFzwzS%Pa*AuVHSL+$09#8n3}5& z6>IC^hA&2!bqwd`e(E6h()|rLYf^VxYPNeEKEuRq-1gBqCN{h5kqrFvrM(E{Rd)-T z?DTippV_FZ7m{^r^T(OAg>>}fXv}*M8Ue1ME=?t&4;?nF^}eVrW6u~Nv!}ymVyT$& zFLZhQ074N1WN5Yt{ET=qscpaA_~1i5Omn}j+@8babF`i_mF%3{albNRzMA-qWA%M4+Gu zQ-5QVAq%bx%6FQn5dJbqh^xY8<^ahb1v~2c{}F@$+~fshTHd^2`LF}aOK5QfQh{S% zY^2R?os!rLC#_|>Eu&-y54Tb2=defQke-)9Y_9fm8gQRl0N{=UCH&a9iis1KUJou> z5<&~6POSkQNC>~N`e{$*aUN4T0-E5Pb9LGPB-~kImMzWZ@8IC>UAslBJ zsBSArxfmCBZr83|rj_NNKg&6cakxXWb^yW72-p%mWKBx;+nF$bGD?PqKBv2UTIwV2 z@IVl!M4b^F5CL(%_iY|P39k6VOUzh~m1FgIUd47;qN zRuD(8xxqUj54*%KpkrceKL}?dxjrYpl$CAX{_FmZM%<$WV4I!2fY}HmbYl#L`uNcTA@HMb^;K4U-@YJ~ zrUyRWR2DOKMSZ~1(i88pTHS{RPfQ5e=i*^ zXV!XHzQR;_H=+AMCIjXUS{t&++Ki}5~-FO?_I!pj#gbeWjsqt-Nh|A?=HfM*_BGRV>j3;|GKI*{jqAdYi+ z0yd8TXt3yESEyl9`;;9dB`rPABboM>#*=9ZdnWS1g1%@Dyy;xA1f(4BUSJ}LiF!_B z00etnUoc!(&$9tZfhbscQn1RbWhe+7z-D%m311#>{qR5!77wy3&>ab=={d`0ap6M$ zICkt<9=Yr^1{kQkb zuF{9w)4yEvcgqQ`3oG^Pyx$5~EF)tL-&N*`8f%_6yFxv@24^ooK8mekA_nr8=oiU= zhz=;oYsz=;ZUw`j)wGV@1eYe98%!vOzpZR@>-O#AaMhauZ~)~cUJX4Srf?K9M5bAC za!&D-tTg@59nmN;-Xy?)ii+8k-xT?Lq?l3xH&cgbCk@D_1uLizKy6f}y6TjOnpb-! zWi-jio18kmk@7hfIdn+YO!hLn!fF6k<#?#L*2cv+AF6XYas0SZS<7Jn8lWFE2Q00j3H;ZS)<-^3 zKVlV^JIrkaYF`9#;L9u2n6jCP&+-+2H!jxLxbxDe4LGJ+s>vZy#wN5={SC9TU~OHN zFE8M2Vqt*I0AEKR3T)j``tZO3>^dmH1)+;j4>OstpJ)MZDI}y|glv<)4)sxD%;i-# zsfF7?_h>eti*L9#Y-Zbwl%D$GQ#k98`C16$b3$?Xd|C5OI|5Tn7&rO2J_TvOFE98?W7{78Aj8lj%~lgh*{ zI~|HHU{C#AZiDQYl$m~7{h#tkM(LBs<2F)v^Nd*mB&6OE#t+}WR{-!H)N@d;o;|-q z-I!eQXUheSeElP`XYAQw#UmX*4g^hVB|bt{N(Qj{*Xx%0-SJK3f!0|^f42Mzmul<9 z#VTdMww&e=Do=~Q*Zr}_rRmwViknXm+KgQ;guJp;y83@fq; zMp(qj$-Nz(Pgu~k1S`Y9@_ysjiGYkTI5_fN0UR8zC(g8mL31o?)iz1qX(!$MN z>FGIKMddw-o&50P`Z9+L&$HG}y9PA5IM#6}(46+x?|TU8#QH_Cj#p!Ss*8&j)zl2? zxPU{I1fv8d60@UGlbu75vD7&8`qt`>0BIK|f1TA~nUGZN5-)`tY~v-RQ>(KR=l$6S z&whS&8R$H+D1y8Waf;@HLBNpg>F7xo);TYfZOak|2{{mCk21r_qz<@;WPsd93`kBh zrP+`!3XB88`Fn-*~$Smw03cBEdj*q?tS`b zt=|;-qQe3VU??wzw9yWT>97qdk@XRwHMEfc-{C3TD7 zYXr0%7?@$A(?p`s^LY>D_w+2-3MIzqn7Lh^h9=-?H=Au*%Xar4djJY~nJYcSC zi=e(9ie9!@_2anw6Gd{<4EXxacX%`I-m@%%v)x$WrSWJ2{*L(K= z{fY)MopF7^EM)3vS%}~5{sw)l=Mjz z!%YdWfW1tIZPyrm>g37I5W>;U>3j7STJ^Zv8il}uy4!dlD{;ihtD;30dxQcC^^y=2 zU3iWFYym`g^Pv{^&eC!jZ@n;FAg@3-$(Me|_rla?9tvV)1!R~an|<=9ofk}(eW9L{ zmQT5_-NKt!3QJ<$+h7znf4b7npX)x~1CB0h|M2Ws8wtU*@|Sn>p5guQs`Z?t^x8ky zMx$y)HA86*<&IG4V$ak**H>imTXgUw|C`LOf{KeGY5Dzr2-Atp!Ew(|ur%8Uk=NYc zrQSEIH7k8^VB$15^lsW0QnB3+5PVg1Jb(YHdVjCYKiRl>?g^NQZEb13m)pSoqCh-! zCn~8>E@D|k%=0Xl1M^yO)`xSlt$eRUE@D1Ht)&mH!N0DzvC(O~^SEa$I3{eURibxy zy)@PTge2n0VmT!V&`j&lsqA9b1oYb$gMf{iXH2GD8`5{m-mcZ&|A}x?I?gBCdGVlS z^+cu=VGUtHFK)tz$S;3$2#HvO$CP)c+Ju=ua`s6WS!6kfOS} zslkwBWpr2}I)ea|1MH3Z%$gyJQ^mt4krZi}n9uJ)c1JQ-Q*&$7ADyP+kOzhK3=qd~&IDSwVx+BryVUqw zg;H&@q|_-mehd|6{TABI?>s>WmL^VY4C%kSho0;2WI(aX#muy@2;c*(&?Y=2j84G1 z1+h|q-LpfDecY4niCqI)=9zcS=nwLR#!Xf)ZL`p4tu0%7KDfYP+6C*GEN$3&Xri?8 z<*~fzE5ww~fWBKorzSF)~b?TU?`wt;;bOad}I zH|}CjZa`=$`hO*k!^a7ySL5JJ#EU*?59Tad^hof`6J8Qe7`JwbT{qyr>d4azX7}&( zrJ=kf^~j+W&gLD5Xbd{tKL;6k!^n;K_cl-50_j9V?h@&^9$9L2d3gM>fPiOk3p{;z z!$1U0zCigP1#oq|-1*E;zsk;@%a>9Lijn5fef!OVjW;2)LJs${R-y2@GM@|&-J86(*ob(B}<{C4V$=7z3U>%6N^9dp@fJM7_v z_j1)*Bt&Gp>2pHXSU^t^X^@{nVQPFGso4cRZn||bnn&tVVIcscCD^nH%n<{n(2Pa6 zwKQi~4A;^+D;^H*0{x?SZy+mX$iW=?D$K^@nur&M!zPrp}nYv_oo&i^?V zDTG|u?Y>9*AY>s7q_Q6d?=I~4U*YqRO9gLIeHYfnxVDVnK01D|hSl18Q_Gr)we3b+ za67-zJV(?Nbf3s@_WzL&@ajL5loVG~OiB;e^L$K}{O6gvE(nY`HU02I^AQsg1Jaz} zCG6jC>>IPh`M|YL-!qICZbn>knKwpROcy`|;BNfb5)m#{mNnWJUp!)YT`vFFUVWWN z!GC1%rZrmmX3a%wM#-k$0xw5-uBf%W#p9gv)wm5It=d7yLNB8|SrESyIXASAfP+MjjT%t`@q%a>;j_EY zQL|5>6UFR5x$*(k@yCzToSkb!-wvFgHW{E>h*i8>>b-v}SC~qf-B=C&AZ8#=A5$>tYH-9?%ZV)N6s_&m7!|oe@&H8xMT=t>yb`{{*&r7;vMhAMYykAG>SC)d zPf_GtnJ;E|2vhWqg2tgF5yC5mpHNZZJN5uU5mEEpvr{UcV%T7s8(w2IeLc5=%i@D( zmpc5jI8e}{hWV7eJLA!#^AQo5gE$v6)l*H;pzO=K;!|h+&JEL^vMMFM`w@hZ-($zv zx@W^q4ZXjadJh^$rzulM)G)^WS~U;bcKPQGF`vvQGb#$jEt(D_hXkqj?>}d*xI7Cy z0bvaXG9wLlfI5cT9Q}6Ro6x4FrAXvFe7Dp#-y8of-r<_P4)XzP-CCRzbMT)^!228J z6Fm`DCJK<0RP0u2a$+uxa&Qg3uLLJFT~s;HMpE4h+?@-+FbtX#X8N!y_TDi zR!LqD&@mrBE*sr>JF~i)ezpd|AxO*j5vq~=(630)uspkqSg2sqh3chu3+#in4t8(2 z%Yt5cYDcFvl>MqjbFCL-?J%lKH-0H>qJj3&g7HUsFVFJ6n$J4Ipv*; zD38d%I~yy;A=hBW5%2VaRFk)MHf?iqyg-yj;YmNv4YQWSE5v72r=q}3@jmmkKDq7{ zo%(7X+IpXwh5-|gby6~F^RDJtM_@BL&h}t{8y-(iKde?H?3nnnK~{#Ik*jkP1>=zui^07A960w{plY#s%sAL3@{K<>OPwaV8a#z`!Er&Og4ba zL8!U|Kx}c;hTh;{;$+DiTPiJ#11G1G;Ig`PRQz-dd9Bz7zyzL+`&3^8REwCVW~hZC`67S-+(&i^YDI;sNLcH(ja ztxoTyYT3h5lR26By*9MqoKX}!c&B;pot<-Yj)`ltyniMeO>*1EC@m`+W%JEzX+u^wQfrcunK zEM9yjw*j#qQDengTWp61qDLnULrQ}_bUH|y%%QR$42EpwY)ru92Or;W=*g$_va zs@go)&Dg-i#JkT-aK%*ZEn$*gg@d||@ZP z32Zf1JOL`N;-+=PV*hLS6ijEF3@PC!Vue}rw!No29$OgRM?G@r$dR_@9ix87gVP(W zl??BYjFG(GaV&kgConiZetF)Rt5-{_BRhZE>{G8M(=F}Dp$k@5p8Uw(RPk;epdJwi z^QkGOvO=RwNsmZ;1#bhy`5W3nq0Fr-a5G^Dm5i6TLGv$0RAw?b!jz4|sIS!>{aY;AsbYBAaU^G%i{l=Al&nji<;a*si(& z^l{H30|-X2062t=w&^X+{nKtbBL5E5#uK%kA_}Hs&+xchW$}8+eN!S&e4H7F7 zH2b@%ILbf`JhWU1s>eJaa_M!pwgTc~1|1^{;YUI69^D|b_q4;1yEd8ezn%(_h}$on z<}JkWvFm4d6bm=33hOx`P(mQm-{IP^kUM$cvO`>(Vg@Ls(1WFcw~HaVtcqdLDE!%> zDC2|~omdKVoRF0Vg6`NpjontWp2kT403r%P&d~k3e9mgGzQst>kd#{G*0Q`sxh1zC zc*>42i@n5X^|L4`49Pi@}M(gICYZ3Gf!+yZ}-eTg9H4rHC9}SmKC5>fc}+ z{$WE@RZaRo0RDCCXvIX75UI=8z0y!j_PKgx5c0%{mZr{J83c*&e!O+!7Uj!?q>H+( z-$gr&96GesdjJE`if{hDDs8_|j80|Vs$gg1NJ3~(e0(4Zr#pMy1;YJetxIF=NQzF` zo;$CMRUbe85gJVN>%i4u))u?eOyBr%%^%pAI{aD9V^K&6G-(+0`!pkseLAvMTPi zY$H3c@_r+-s|0!r3IGBAAv3hXbc!=KCypbD0aIIASpn>KWZ!>z7t@_zMnJ)kC|Jj~ z&nMceyBTd+UCxpd+$R3Ijok)yzj%1nq5X&K#f0jph#`g&;yhrO?!itSKFt*~mh_27 z=@zPg*4Ri8+)^M;vK-hI+iSmE0CYvYS3BT32@0tp%^j_M#2X_$6djU^mtQ+ddfjBg zuNAYfnSsp${!$)eiGSIct0iAbOUW)4N5X$_I7Gm!jKin3DC)5@dC06vk+1c9pVKoP z1{?m-YKLTR#=%(}zsZ^GCr2%t2`@(gqOd>=&77olamr!d1`sC?R!G>&`=^*$O>M6} zAwXIttYS5HT5wUNLwrk9WopxFk?V^)cL#!I)`-@dqq0v-?Mz8M9VZAHN#9?kJ-XvF zl1k~oB*YL!McqlTV5#@o-tRgZ6ikwq8RFud^SlF|o(Lg#{Xw?Cj5xJ269F|`qt}rm zmoHsv^IK*P*y4$_z03U$iuVU_K}lKJ*+}vy7S-}X?Os)^rnEx4+|r=aQgYhNpRb}x zWA+4)R(I2;o2fcIP9-^{@($2ki*?7a6|7fhZj_ z9DVoJUrXn3?xFIkubuZEhBT_FeUZI9;knk>kgQ>W+6PVlpa!>=)&Bsu88l(S84e&h zr1kwdPYoR!8Wo>XW6NwGg+g5@k(3o`hwdy3$@)%Phsp|a$FAK z2{s&8^+Y<#$kW7^@1v>yXYAoxG!Ock%>b6go7?NU)8-7=@ORh5)uY;Yy||M-|IQJ& z`u%RyiwE&(WSS}KDAhVvko$p3SqvpirA?%|Dt=pgpN9%(y;xPzdarPf4>PAMO^+?I z23?MrY$!op*bZqokrkI(W(W>x)h%vcxYYqS;T4P%W?Ab?-bpHhnGy^Lj9C=;g7gHzIz%|6as`q21aB>nZB4&ciexoja@3#|fgG2X!Z}4hq^)EARM@f|Sh|zDIZKS@lu)?y+Mh z-yQWbQe~;<8-Zbp;cTzRF`PMvegxBqZgmVN&|9MG$G+bW%Tx$0;B7AFCUri#0S*A? zLvDdMYm6RENE2uhm1lPunbuR_%~4(mdmm&CoJ6Z7BW4A-Dd-%t%6Kf>`A5>)F_b1K z-nkj12QwBg?k(o7%w{9Fz#wBK#tF=f?nhE1bQC}jzyO$%MXxiFXsO-UVn7ea>PfPT z=6(wpP_Eh}>BVe`IMCWTo#|ef3}LQ@fKSLKuP@ol(A~`%BrTT%Y_?^x$#k|pqrX|b zW6$aNV(LIluAe@KSp2-of!NW<(bt2{Fwh2Jf<7+c`I1)5sx6~X z;mslTY>V6WI{EHd_ujX(tZE5NZ+2>KFmm>7u+luE^T~V05XCb_Uz0^wdn&wFBBHR5 z9yv?iMxL5~X}HSzgYh*BZZDZLxqZmUnrPYZgFW0Va+h2w@}AY!?fk&GJ-YR8CF!!W zWkrcfP0sm*UH1n(p0Qfn;hCRD%$9(-3~yaU^-V>|x{3k{yI)__k-|WN23E|lit1Oc z-Q{(b5pUF_Jt(3|u=8Ng}^ znwlp;&4()ivuLP@aosnuPQ$50lhmXMD+YjXJAXr`6kU>g#prWi%?&O?lsXU4URx1Q zl3;}{9xYR7CHgO@Xq3^}bzM6P`w&=pqVEc|wR`n}FX>tZ`Yc@7p{gW8eI#DTZ5^%Y zH!{d)Ew&$p_IOB3NSlAFupfVi6M;tuYtr>r#AFQVBh|s{r~@)-og}xDVTqDoN9?QH(OJC zKoO(P*R9QD&Rz`vdnxZBhXAwTuua$Op86Jh)KJ{U} zZVp#6$OmO>z*R{6-=J7OU&Ar~UiD~TdGou@hhf{-G>>{Y@%bJ-#qx2@Ch^%>Sz|v` zA%@uaT(Y(h(MMHcWi>(HX%m!@dUe{6y~&?B|KfJ*d8yNleSOwHiKr`1PEHp^F|c>AZV)`S^gHChP%*Ph3l*7{Pi28&_Id>iV-ZnDLGy zKh<5jac9rR%4lQ{{qtt!CJ6j(-^y<+MjW4sBX=upddwbbS zK>40|cEPc+XW^}+lf+OgNg2}?&y{jOR~l=c_4|}|CTUB-j_DkGAN1f0I#PNFxi2bb0&G~SvsmHtqT43t!@_XVC&9i zS1YjXg>)QZKyKdy;=WK#>|3eE*Ml(_0^yonHoAa;caqxc7zbsxuo5UE{(vmw3M8e9;D9n&I2-a21ULJX(2@n}I<)Uc1XZ3v34@ z0;I%gY;hK8cAg{mY|9l_S>`CVLCScix z?+3TACUA!)Ts`H)lK8C3va&=qY3ZaJtydv55M>0jgn4$+#=ZYAntw^~>87SfYtG+g z;H)sFxS=#S@LPkcheZf}LN71P$ll+=EsGn1{sWWUqjzDAo>=G<@5DYfaQ4fBf|i%I z%Q!7~j!H5sQF!)WUbmT$bF`tA^wIzG!HbSywlA{C&EycHOeJOI)DFVH;?L|tbbfJl z!|6%eHNjj~)7pyD9Zp|6b6J zqw+WtAkVG@1}a;!nea{|uT3j8lt7loYa+6(P%~tbgZJGn@a>NuQPj%ypZqpTxG;r- zz=w3Xazg&bvSl5-d`p(ZxHMVJtG({ybJO^reDX*r^--y1kC+OFiG&n#Y311=1HAWx z67uO}I-5@`d2)1CqgOwXl}<`1t|Ap8{n@(Ckdq+}5Lro-*ywqrAPw1z!$qkh?o4OY zt=h2yGMc_s)@;MLUjgyQgE6x~;76T02)-#%{%QzucO<2Qa@A7}5+@Uz+l;At+9GG)|vipPg4mp+O<;D5; z9>{-XD~#Sy%%svsG$pFdZCZsbFI*PNeY_-IkEPuFf< zK6-kq67KetDO~K;Wx;@=q^yg*{`cEFr7>M8Njf2Uz2Ieh$ET7eO?4h8Oe|U)9%j6Y z(a{)f@IP1f`*+toPRVm^%o$#p_C0X#^oA856(=uRoU}iri*L-eee;*+9Qho&M^7(B zYwsRUL$-j`ED!;XDnM@6br<>S=!p~4aE{WiETAApJT&#~8pdN`Ed(&gipJSYzG^$I zsWDz+KDFZ(+_MSR^w-YOTI+0i>92uC{}bSD4rV|#Ne!OhF9oa?_?MBsa8j7RWIw$ZOd1GkzmI`wVAqMES1`gSjtPamt0DEmKwd?^b} zZFcv}F|EXl8#8;jaAc`+(IyI%Ppu^U8REiFX#d{%uGhhXA_)EwwtREZ&W`9KCRWjZ zPFNertPC>a{GVFbesHh{$lU`N+|W%_42SVYZM|E92O&U9VhWK7kxZM`tq4Ywic;5J zikqdl>AU%Gn_&hm+2KGRt~FT;#V{8KeVFD&8(#W%v5dr#0ZNnuU&n9x+Hc=VpR;FY zT3QCxob!YtivYPJ4MAlP0}5-ly}4<%zb5WLWqiwkBsJmaPwkidadRs*QVyH?&o$RK z?VSAU+f)J@Hzc5PoW~eZkYF{lfePyyCm`UV%UVU)fS4w>Llldn<%~dbUwOZ=-+D^V z^3teFJ5K;$&^8KdBk(udc_|9thjozZcC`IarIeFc5kMW!Ap89MA;N2Ur0V_p)=6sA zDdO}_q9aagL@|Jn&bwfHNL7n&ukE&P<%+XIH2pdysae$beUEq*c3udIK=mUm+_aR2 z7@eL1iYye<70$#bN{>rXQ76e=67+8~X3p$WIB3k~{J(uX_fZE4<0WQ*&g&dJH1S&tT7{5L-B(zA4#R2l2p>qU)>#>szb z&aS=B`^ZgJjU81qzr1{#RPT=eF8LkcV~`OX8usgD4U>yc80RrqH*cf--sEpv{d>0) zm%R6%OTM|0o|06(Qm&21gI;0LOd*^zE>Su1r6R~jZotOP=^kOHd~QzpCu6+sg&01k zoC1NljLr#UNJekyC4EnyHgs@^=pnEl-acr?bQM|4vR79#v?nH3*7=@2dmy(W%C6H= z?6efRCX`g;=hxgTYzsvci+odzITA@LS?}aWa}DESE|~YK8W?M#7xZu6^E&(GOUp?n zi6zyGO^)W%*FbtP0*NjiRhTuAo~46akJm&#RFNG>2gj<^%^fT40YNCv%b=P>v3~05 zG)C4yFI?{BmxM_Yo=`0m{;p(d*Vs*)XfaeNKW$FC5TnbJLp6RoyNYEoCIS+ zrl*afT$aw<(wB@pZuGg)QsORW{QGP_Rk?t7i^>x!TsldN$BaOy2V@99a7NR%5Aibl zkgy?OH&MwT1fNCd;C?)!`UWi9wR;M2hFzVRlQUgSF1gfER5NMYh>a|MD)ds|PF2;) zGm_bd7k2Q<87GsY`FWd^SI^&xC_lLY`>C(QEEw(bV zdn`YE@I1CT;Z9+d2k8yI?{%Y&n}lHVw3gGb<*rq2X+V=SqsS2UWXqMg0guXE`cB0b z;2_s~b6w`;wOyvA>_(Hj7fY+1Gzt9w`Y957liPx?`)pn5L9Cj7)+9wuI+|0Zxv!0O%=S$56s2Wl z2WstC9TJGcqR90pM&0V!?Vz6Jo=NL%idEg}??oQUIxjqM>59e5qL*y_JU`zU2V+7Y z)c&B<~lvokQ5)yXgKt6gV6|SIXykkC! zX=g#ZaZ!*FMVn(zvqX7Bk$MB=q>-n3vphKV#;jc9T7I^VMCiocf@@*vwB5T%=&HGR z;lvUbXWpV|!NDFvIPY;}S^e$A3+=-Tva=0PlHWLD`oZbYU7zt0)`NAP1&#|lIkhyi z{;B*iNlVM3*V0G}9EFAyrYKZ$q%DSwprKql@*B(@y6sPTK=z>Fb@}9$EQq$J9}~yS zzHIFO1Pp%n?lT7n7gMJuomWl(96ko9Wp#GQgB3not3TajniaJdf2ttcLER>M1g|~R zha7VwZTWN_OqCJ2dS=Hd0#xl9o#oP0vZ5OW_93#o< zozCl8Gf8#I)4MSY_8($dPm74(c!fhQ0y9#N4>jAhd(8~KbGVo?Xy2wa{{(>VBAxaf zprOe5?)8$3MyJ(wqbTJ;Q04xOl!5gvM@V2PGDLETnJADMV)Di6^d}QnzuTMaV&(sK zOuN4ZhGOy9d$e^|>2^}>Jkn09d!XA;+nBFi9xac%%ikPsM2G}Qq!mM#XVVu>iq%PY z>5#zd;JML)^JaxvWNhr)4qo3p#Bdt^Ev?$=6`^}n*3k)M-AH7IE}m*^e58jAxLhH& z0&Ut2ym;-}ddJSk+E*GIdOrF#OjT7`Sy_^6R+C#*Q8 z1}M?>mJI3r(_IWKA;g8(5wB^%f}TM9R(Y{bq)769_sGhH`TdDfJ@3EXHlm&W>pfE4 z$M5`gr=~_A8)O=Rvp4N*y?a;9(&OL8ij$r||2n^Y#;d?;4lbs!$^}{cQ0O5+e4jgK zM03ryrhMHw$B<3g`Y)4RoiYqhU-n8Z#$Z`YmIz%zujQ_~$f9;nNjRIwF>T-1_8Ztu zWccu4UIMFMI3++MF+Ng=#+u|xPIH>6E`9F%t9S3TKi>}i7Uc z+*%xRu=-57G5Nv*B^e64t^EsqINz``rhd}DS^!ykQj{-%AiQKY+&jB*+;kfcU0O!GUA=bt*u^O0c; z)Fa-x`lmO92s8P&*z4A9tKR?bA8qiQV~>I+wdvNIHulf;47x%)D0pTZ z!1TAPbhBBzR!~2?AjqoUtO-QsW=zA%)Az`3LXM8kY1>z40!)m^(nqedSK;Q$%USQn z#?Q`w#O)^|R9VAZ+^D!2gV&Wi>n0MvD(k!to~IYEe0zHxQ%}T*H(*le<`gGS4xnn# zEw_7n+w04hFC}&*$f)j^hpp_t6!HS@J3Hq$yresS{t4l5pdAtL$X?XX;u*@VgxQqa zw{oP_B$Y%(P=ej-&Vom_#(W|XIjrRMzMX|96tuGug=l4+aFT@-+qd!tvN8BTLViN2 zqlR0vEBDm!`Jy-KK4nm2v^as}bgCURF|7pHd&j<2V8;HO^1+B5U3MD2-Vn9s{{C;< z5KbuLSRS35PFG2O6tLGWky02!l_0#X+qiKJkP00g7$(j%b)Nl}_b&MJc*t}l>Xk5a z*)&iZpK=;X+<=q6RyKD|QVVT4l6AiQ{x%6T&tB`!5_2$moV|2u?s=yqMRXqrlM3JS zpaf#fg?2i76f9*3M;IW$?%KZ1cr4SX0kS1)k-Q2}6mFDF1yMumE+i)05<4APWydv2 z=A%wU3iJHzZH!K`4#a7Qv`WZK!FOIV!9h4FsTNcLWCm}TJtEU#uOI>#XzO#&V^r}z z*0}x$uT8yV3eEN=KINU?RgZrrbOLLhYZZPBi7M0KQ#(>VE9+nUy2Q+c#ebiXIyzwoK@$zH(RtW)ZhJTFbNy5Mq z`JV?59H2G1923(!Fk||fHOhi@PARUE-d2#QsJ`|4s`m|wj439Alhn-WSgP58q(liWXQaPn^C{4Dh9OfNj zCUS^G$zh@+GdhioY%_>a4l~+jBP0u{s1!m>c2O#+geDa_Nv-$uteC@lUGMwXzTWlE z%vD-z{eHji_xV20ec#Xh%qwdMv^9d^*$<2YLQCt&k>kg+ES=;TvT#8s1))VtOSc1} zW>eNGT~=k*na_7h%<0$@oPY|Cfmw4WTCcqJ{ZQ5(<*|VAYT&HYxl)5A7t2la(YM?IbLS`?H$_}@v`jo^5kXe(*tNi{xo{D_F2VNIjWokj z6BZP1yI}D9yH1}v_3GK~-L)g5R15R}Ex@WMExlkmcMl#ou!#cRD1{ z`1yBlLHs~+NSamN=kdX>SWp6dWH$#+Wc-$q>cMlK(==mN)7Q%HPL(x}3W_0^>NGsw1U+#{}xcX(5K z?sdg~9?|t}!>oD3NQXd6rT_*1=Bkec5%42!rh`0ck$ zh2{uuZ>d_QS$=E`glR|3#JAA?{aY}ofmC?`Jjv_249&m(xh-@PEB?wd|6!94R{NS2 z+NovT9BaRL@emWYI2Celzu0nXb@W;P_)Wj29e?n2U)VCuzv*?4-1OXsgf{nfrYz5-Q7NR-pDZPzP3vE!sZAU!fDyoK#|ju6 zNqWoJy@JB*RcwA}OM%#aK5$^r!*Y|sE>zha9E&W2Z@A&4BwGYRwFp_%md10R>;FAcfTr+ZgQ#QeO^W>fqZ z)tI84VsQAQ+_>CLUAisv^d!an_1?XyOP1hyb3Dmb7Re6On0ZeYQ!sK*^KRXeJgVM5 zuGA$n73IlR%aPE#w9!KPD(st>djE+{j zi1uj?R;FfJL6`PBCmO&{FAX)cHF;TAHmqQlV^nXFu&Ben+B&&iGfOo3J*Gg(XxGLN*`UFTt8#4e&8u7mP3ADjn~8NtJ}O@w>QPoAx)-Rs<5vdMQuvC`}BKLrD`Q){2h|XOP=TJp7wV0;~g@qk%?s3=DE8>A>hwOpHHR|Ew@6XYL2qgKrlX zul%U2R{C;xANIVl0PBZSOw5sNjviVvE{bQt8)ACs2ltGuO7?v`{w-SM#vJg-+r-rE zw=Eyp8UOY&sn5|PH`_}!ABUKPl&;frqCvZMC zE-?|-8yH>9#rYnD!9%dVix;O9Q!8awH~f14{-^9*s!)$Ztu9j>ha`J%3_da$@$r!ipCy zXPF5hKZxBwfAR<+Z()r^Xc8y;s2jFw#iNo$TvTVya*W|!!MCOBT>uP#^qZ5FqBdR z#1=`*M$bKnO`2m=?Wg0$>9ZJ1Ypb{^YJrXSmgfH!3X=v{rq{eN5WU#{%$X$ngBS(h zwc3a9N5*!MQX%zL6fvYCkg(S)ysO3+5jK#lYQs!o6)6xmqYtIB$YH-DQ3JA zIyv`#-w*XP2=bmmUlQKfT+8TZd$LwYkBHLN!;7G;`=3rhfg(DjW%BHpGd&Ic=cNU! z9Lu-fA(floT4n6>%*oF9^`zEgUzpl9lt#L^O=$Mnadz%Gzq@t|F$RIWfCjzHVGBU7 ze`e)`TojuWnEll7fLLEM{_0lCj^J#d;~t-|PtT%Y!0D2-#?QNE&Pl>A3Sxj84KX&N zFk{2d^`qT3B#r42sra(3aDryfEl;m{S_Y@wuuAbfYnKiOMgtd}T*6LQomknrf!hwr z)a9a?21TgD+K_~4sO=cu72hstNAu1YROLyhHj z2DT-nN+WT=hgBoQdZvtF)*P{jKrFptAa@CF?_8wMOW4!0RoDhQP*SUyJ6xwwY(dho z;J|(aCj}(A_K8u@X3ssSNqrf2igE;6waWs)B>ssSp@ORg;y*(p%Zlu-q*LbS-{e%* z^SS0Vb)}{@V+N)>7@Lr#lY?f$;wRqguKo?YPqFt>6zI%q;U^O4C=W1Lq6Q zDYko*SHEdHhch6vi&WawA1GqE$9&~$Edh+)NOWth zIf9$z%FZCCw#wIQfYnBMX~&*iZ)HOAb8PR>&4-}1sI4C$`~3i?RmE2pGk1*HfEm-k;<*E#TFAk*gf9 z$4xLwD9B-Lh@hLYoMugJXiKcgM~=LGepm@@9J%;vba1%0?%c6lk^R>%vcaK&LEQojzOT+9K{Zv6(m3Ow2UCSW*T+S_ zQr2@a#O`_WmW>@>N4z0?rya!b+zc@(#?MNpFgHz>wwtpm&zuzmoDv$ z#$u$E?6N(w%bq(tiyhyZzOnn;l+l#X6cIJ{&)0G#;{W{RJa~JSj1*r!bbyI}=pUtx z`9m3_bL2>*)J;Z#b)CqC-~Kk^@eIEqg?3un`a7_4x>OYxVp!vT;onqm`1ov5)nBl< z4%^P#>8@zmUYo0M2pD2Vcd!+YZ%I%J14^jjOP1}W10pcb-9}6-w2vw12`nhd_9>kQ zG={>Dw%>d&U|vZ73yrp6(_l1(soBZ7=@9s%y+aPi_HIkhnNk`O?~Q-~sPQQM_-K#P z_J*D@|60C#_R9j_zPf0V$i(N(4{0t7r$4%3>zOBOtxDI<$t9r}oQmQ_0iVDYQuxF* z%mn~H?3v0Z=CCbDJrEeV<`4|utM}E>cOqq5fWAWR*%9i0vaW4NFNI>GXpMAl{O5T4 z{KCN^^_e}X{-6q0BeapeK-^BIJqiR7)CyiDW~{HVPy-Zn&|!#OA&)NZ(5wI%fqi{> zu4uWBPZjOj9N_=JgN_Z1Yf2?~B^6jWionj|U?x#MXre6`8=yIv3s0jHxL7y7HmS-y zqvwbxDSHUu+oPk+$b=Y?AmT9dB|X(>f&q1Mt+w$Akm(k{&ES-45R?J+Apd|Y2a)P3 zE^>z3xke=UN2*_-D?=g(^uz6bsnyqMZSaZ96>2;RYQ{>%Pr1 zOAc%?XRx)>d|Ccy)WcCN(f{;0e?t+HVt#z*sQJoIAN}$kawWS*zu1)<4{_5KE<_Yy zAhhPtU;yjQfWG3zs|IZ((@|-X`d?T#5UbjAMTIBayKqXt6w8rLL4=XE%rl^xK^a3Z z>Ex~0i@G>y;#bMVxS2>(2z{;MW`ze0ElJ!B&Z6lnWyuuxDXj2+!Z)*ZM z%fBlF&D`~Pc7UIKjX1oid=MSMs{tRgPRK+qmDh`hG3|OEW7Nsq2~yYZZlw+1zEtMP z$h77qust%yKsfZdVP{407uZznCmzgz8&ft%*k9hQ8L4cKQ?FLEmV3CmE(*=;_WATX zbtuKgZSY=^C4xYjHczEGyST(7_Z;Pwe*zFr4Ycnf9yBzUO7w<5Vxf{%nhqU`;IeR1 zk)=kLJWE%q*^6rwxRJ@^Vl=wF54*(Oq=W~i%c^<-t>Xj?wqNrK#FAdvBtDALlf(f& z325~nNj`a&(1e5WLfHrh{ApfPbBfEbk3U|DH8L4M;+iUFGAmJa_fq%))QSalvA6V& z@O)RbJYOP&qt!m~xY@WQXD704`YJD1uAX9DLJ~b^(Rf-*uU@?h3d=YHCs5yYKLP^K z=yfK4Yh>gBz|w8m84L-M)B>CV^mkU%zFjl?sFp>-PFas3`Rn>^A$QC-zdvl4(CohS zFDz3InNl%e^dq~4)^pp|74P4F@i;5!Oa~x@3qQ}rjH}_3Lhr|MqLVAy>3VL9v8B&o`=G-K+K!Ys$UJ79B*tqR9 zmzdRcI>-D=>MTBrw*zF8RE-5BR6$xWS>4f?DCCN?pCXgo!kz6Eo{32c^(aIhCE-@j zJ-NjtG{y*w4)B3(1K^2_s-}bz6aUrcAjNbKdp@T(1zZ4XS9{wSUH|8`kC()cUblAb zoT6Up-|#fa-L4|av2-Fvg_qVUQvGN_+J0cVqekMEf?CeD@<9XBaW9div43xk4Ei;? zJ&4g%#)Y(xlP>z2ull_EuJ}E$&}8X1wIct>PwHCcB`QCHU(mBNA}1z3Q-}}&#dlLp zx;N?Y=}78`T$?_qYRAirjl+_l!tHEEp~LMk+?KuPznXc%dy{1mKw-T2FvEE zfuaDQnJT_5sl-Q7P+ICTQ0v6;NUzjTDKZ^iR}t z>17np)}-EIe`DCwN#^j8aq?Wm5|O*!yE2Xj$cXL;sJ`<0@M&|dudY_wI+_zulMX9L z4pFAAbN0qMso_;#h~c+vwB9W?T-u3rq@~uuul1yFlB^Lq8M^TI(n?WehA$e9Tc4q2 zfL6~?iXwy4&dt@wkB?w5n&nUR4o_ibzR4N{2WH-1S104~s|3xgl@l)<6R%42RF$i@ zZdntVU9^4*tV!|DFoE*S1_?OK>q@kWb+ETzd_^)2d6UJY$nw%Rmu!QVMhrQOyuwg` zs{W1L;s5a4Z1HzQ)0F6$XG%tL`ocn)J%!TjcKh~7`3CkAWN7_O)H;X??oQhUEFzi9 z(W8Nt=gU3yuQerM!N9^j)Hb+}eD%m+M{@RAZf@<@8X(?8KymKe23?0y{$bC9ruZLK zU5&$@$;QZXRM%`c)a2lsi!0y$*B=ir9HgzG&ayv5U++ey$ezS%xVrYmix=}Z#Q&2D zWEI-LciY?8UI^C2Lf^%y^Dx;iuZ)kKTu zEgPXT1ve~7{#`I93e(f;0??vcnEsQ$9xF(9*meb|;FWP}oB$G_mDEnjOT03e4q z_ewO__q`Q;b%@^}v_j{572(kE4q#ya(`g)(vv@OPp}dD0ERaR1%J$Z(XYnUbZbt@5 zeT*tuHJ^(eXzCJsorb0P8?MXa${HVP|NYib5z44~Oz6O2Y?lZpsima?1WA{Cfz1hD zc;(u)h19Up$qT^_mLLTq1^&g#)i!Is%|r7+WoY=a5S>bddWC_ST9@{z_LDxNYeEcD z^Gi{MM$xb6_R1KmA zaB^Nkc4lTB?gcWxVA=TGA^}MV&zGO9J%u&lf`O|@qd<@#s>^=J{*=eF_`)pJvg@I? zD<{-*4>%~X3LSTRUXO`*l&xES8)1_W&`4AeW*fqdh{rLONY!JM?t<^p-gWox6&O=F z**Q%qxM^0&5e*%-Y-9pF502LiDC!;;x#1uR->y%Iw+ULFm&dnJBU})}G0Xyd;HZ*R zHm4FZ;a%_uV0dxKal8SW0K?7f%GHqCIU>-&Z&NkLG3qZ*;C&OY1NF8rTbvRwYS7Y) z5K~D+;J1U_zJmslHulGfjE zV+WOel}zu?V9<23C}4(l#hFC^zig2oq9bI&+5FZNsw-lPj)uC}%&jYQS?};bf%n*? zA-bnhzgwGf;5%#(z^ez3_{~v} zHJG1woE(d-Xa4cyLnF0y074F)JJ<4a4u>J`S2DLZ;R`?OkL+4fMIl7ac<_%fErKs_ zhL5yI_!LD&Nvw}@nW5Q3@9YmAvuE$xIeU0o%KJy;$Bw6?BHtW3^#WeV>pyi2H7dJ! Y4A}T_;Xq}88n3BSob8WKUhwsQ17^`8IRF3v literal 0 HcmV?d00001 diff --git a/jetstream/clustering/administration.md b/jetstream/clustering/administration.md new file mode 100644 index 0000000..941ee69 --- /dev/null +++ b/jetstream/clustering/administration.md @@ -0,0 +1,85 @@ +# Cluster Administration + +Once a JetStream cluster is operating interactions with the CLI and with `nats` CLI is the same as before. For these examples, lets assume we have a 5 server cluster, n1-n5 in a cluster named C1. + +## Creating clustered streams + +When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. + +```nohighlight +$ nats str add ORDERS --replicas 3 +.... +Information for Stream ORDERS_4 created 2021-02-05T12:07:34+01:00 +.... +Configuration: +.... + Replicas: 3 + +Cluster Information: + + Name: C1 + Leader: n1-c1 + Replica: n4-c1, current, seen 0.07s ago + Replica: n3-c1, current, seen 0.07s ago + +``` + +Above you can see that the cluster information will be reported in all cases where Stream info is shown such as after add or using `nats stream info`. + +Here we have a stream in the NATS cluster `C1`, its current leader is a node `n1-c1` and it has 2 followers - `n4-c1` and `n3-c1`. + +The `current` indicates that followers are up to date and have all the messages, here both cluster peers were seen very recently. + +The replica count cannot be edited once configured. + +## Forcing leader election + +Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time. + +Moving leadership away from a node does not remove it from the cluster and does not prevent it from becoming a leader again, this is merely a triggered leader election. + +```nohighlight +$ nats stream cluster step-down ORDERS +14:32:17 Requesting leader step down of "n1-c1" in a 3 peer RAFT group +14:32:18 New leader elected "n4-c1" + +Information for Stream ORDERS created 2021-02-05T12:07:34+01:00 +... +Cluster Information: + + Name: c1 + Leader: n4-c1 + Replica: n1-c1, current, seen 0.12s ago + Replica: n3-c1, current, seen 0.12s ago +``` + +## Evicting a peer + +Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down. + +There might be a case though where you know a machine will never return, and you want to signal to JetStream that the machine will not return. This will remove it from the Stream in question and all it's Consumers. + +After the node is removed the cluster will notice that the replica count is not honored anymore and will immediately pick a new node and start replicating data to it. The new node will be selected using the same placement rules as the existing stream. + +```nohighlight +$ nats s cluster peer-remove ORDERS +? Select a Peer n4-c1 +14:38:50 Removing peer "n4-c1" +14:38:50 Requested removal of peer "n4-c1" +``` + +At this point the stream and all consumers will have removed `n4-c1` from the group, they will all start new peer selection and data replication. + +```nohighlight +$ nats stream info ORDERS +.... +Cluster Information: + + Name: c1 + Leader: n3-c1 + Replica: n1-c1, current, seen 0.02s ago + Replica: n2-c1, outdated, seen 0.42s ago +``` + +We can see a new replica was picked, the stream is back to replication level of 3 and `n4-c1` is not active any more in this Stream or any of its Consumers. + From d3ebabbeb0e7e4738f878e5c4d9dd6dded5ee16c Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Fri, 5 Feb 2021 13:14:22 -0700 Subject: [PATCH 10/84] updates Signed-off-by: Colin Sullivan --- SUMMARY.md | 5 +- jetstream/about_jetstream/jetstream.md | 2 +- jetstream/clustering/clustering.md | 94 ++------------------------ jetstream/concepts/concepts.md | 2 +- 4 files changed, 10 insertions(+), 93 deletions(-) diff --git a/SUMMARY.md b/SUMMARY.md index c0c55b0..4252c67 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -206,6 +206,7 @@ ## JetStream +* [About Jetstream](jetstream/about_jetstream/jetstream.md) * [Concepts](jetstream/concepts/concepts.md) * [Streams](jetstream/concepts/streams.md) * [Consumes](jetstream/concepts/consumers.md) @@ -219,8 +220,10 @@ * [Streams](jetstream/administration/streams.md) * [Consumers](jetstream/administration/consumers.md) * [Monitoring](jetstream/monitoring/monitoring.md) +* [Clustering](jetstream/clustering/clustering.md) + * [Administration](jetstream/clustering/administration.md) * [Configuration Management](jetstream/configuration_mgmt/configuration_mgmt.md) - * [nats Admin CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) + * [NATS Admin CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) * [Terraform](jetstream/configuration_mgmt/configuration_mgmt.md#terraform) * [GitHub Actions](jetstream/configuration_mgmt/github_actions.md) * [Kubernetes Controller](jetstream/configuration_mgmt/kubernetes_controller.md) diff --git a/jetstream/about_jetstream/jetstream.md b/jetstream/about_jetstream/jetstream.md index 29261b8..dc87dcf 100644 --- a/jetstream/about_jetstream/jetstream.md +++ b/jetstream/about_jetstream/jetstream.md @@ -1,4 +1,4 @@ -# Jetstream +# JetStream JetStream was created to solve the problems identified with streaming in technology today - complexity, fragility, and a lack of scalability. Some technologies address these better than others, but no current streaming technology is truly multi-tenant, horizontally scalable, and supports multiple deployment models. No technology we are aware of can scale from edge to cloud under the same security context while having complete deployment observability for operations. diff --git a/jetstream/clustering/clustering.md b/jetstream/clustering/clustering.md index 6e6ed08..51c8d73 100644 --- a/jetstream/clustering/clustering.md +++ b/jetstream/clustering/clustering.md @@ -20,15 +20,15 @@ In order to ensure data consistency across complete restarts, a quorum of server **Meta Group** - all servers join the Meta Group and the JetStream API is managed by this group. A leader is elected and this owns the API and takes care of server placement. -![](images/meta-group.png) FIXME +![Meta Group](../../assets/images/meta-group.png) **Stream Group** - each Stream creates a RAFT group, this group synchronizes state and data between its members. The elected leader handles ACKs and so forth, if there is no leader the stream will not accept messages. -![](images/stream-groups.png) FIXME +![Stream Groups](../../assets/images/stream-groups.png) **Consumer Group** - each Consumer creates a RAFT group, this group synchronizes consumer state between its members. The group will live on the machines where the Stream Group is and handle consumption ACKs etc. Each Consumer will have its own group. -![](images/consumer-groups.png) FIXME +![Consumer Groups](../../assets/images/consumer-groups.png) ### Cluster Size Generally we recommend 3 or 5 Jetstream enabled servers in a NATS cluster. This balances scalability with a tolerance for failure. For example, if 5 servers are Jetstream enabled You would want two servers is one “zone”, two servers in another, and the remaining server in a third. This means you can lose any one “zone” at any time and continue operating. @@ -103,90 +103,4 @@ cluster { } ``` -Add nodes as necessary. Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use Jetstream. - -## Administration - -Once a JetStream cluster is operating interactions with the CLI and with `nats` CLI is the same as before. For these examples, lets assume we have a 5 server cluster, n1-n5 in a cluster named C1. - -### Creating clustered streams - -When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. - -```nohighlight -$ nats str add ORDERS --replicas 3 -.... -Information for Stream ORDERS_4 created 2021-02-05T12:07:34+01:00 -.... -Configuration: -.... - Replicas: 3 - -Cluster Information: - - Name: C1 - Leader: n1-c1 - Replica: n4-c1, current, seen 0.07s ago - Replica: n3-c1, current, seen 0.07s ago - -``` - -Above you can see that the cluster information will be reported in all cases where Stream info is shown such as after add or using `nats stream info`. - -Here we have a stream in the NATS cluster `C1`, its current leader is a node `n1-c1` and it has 2 followers - `n4-c1` and `n3-c1`. - -The `current` indicates that followers are up to date and have all the messages, here both cluster peers were seen very recently. - -The replica count cannot be edited once configured. - -### Forcing leader election - -Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time. - -Moving leadership away from a node does not remove it from the cluster and does not prevent it from becoming a leader again, this is merely a triggered leader election. - -```nohighlight -$ nats stream cluster step-down ORDERS -14:32:17 Requesting leader step down of "n1-c1" in a 3 peer RAFT group -14:32:18 New leader elected "n4-c1" - -Information for Stream ORDERS created 2021-02-05T12:07:34+01:00 -... -Cluster Information: - - Name: c1 - Leader: n4-c1 - Replica: n1-c1, current, seen 0.12s ago - Replica: n3-c1, current, seen 0.12s ago -``` - -### Evicting a peer - -Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down. - -There might be a case though where you know a machine will never return, and you want to signal to JetStream that the machine will not return. This will remove it from the Stream in question and all it's Consumers. - -After the node is removed the cluster will notice that the replica count is not honored anymore and will immediately pick a new node and start replicating data to it. The new node will be selected using the same placement rules as the existing stream. - -```nohighlight -$ nats s cluster peer-remove ORDERS -? Select a Peer n4-c1 -14:38:50 Removing peer "n4-c1" -14:38:50 Requested removal of peer "n4-c1" -``` - -At this point the stream and all consumers will have removed `n4-c1` from the group, they will all start new peer selection and data replication. - -```nohighlight -$ nats stream info ORDERS -.... -Cluster Information: - - Name: c1 - Leader: n3-c1 - Replica: n1-c1, current, seen 0.02s ago - Replica: n2-c1, outdated, seen 0.42s ago -``` - -We can see a new replica was picked, the stream is back to replication level of 3 and `n4-c1` is not active any more in this Stream or any of its Consumers. - +Add nodes as necessary. Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use Jetstream. \ No newline at end of file diff --git a/jetstream/concepts/concepts.md b/jetstream/concepts/concepts.md index 00785d4..4a533c6 100644 --- a/jetstream/concepts/concepts.md +++ b/jetstream/concepts/concepts.md @@ -4,7 +4,7 @@ In JetStream the configuration for storing messages is defined separately from h We'll discuss these 2 subjects in the context of this architecture. -![Orders](../.gitbook/assets/images/streams-and-consumers-75p.png) +![Orders](../../assets/images/streams-and-consumers-75p.png) While this is an incomplete architecture it does show a number of key points: From 6875a3fab9dccea3b9a833d6991d3fdbf13945bf Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 14:30:37 -0600 Subject: [PATCH 11/84] updates based on RI changes to README --- jetstream/monitoring/monitoring.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/jetstream/monitoring/monitoring.md b/jetstream/monitoring/monitoring.md index 13b5185..39add46 100644 --- a/jetstream/monitoring/monitoring.md +++ b/jetstream/monitoring/monitoring.md @@ -24,7 +24,11 @@ All these events have JSON Schemas that describe them, schemas can be viewed on |Consumer maximum delivery reached|`$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES..`|`io.nats.jetstream.advisory.v1.max_deliver`| |Message delivery terminated using AckTerm|`$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED..`|`io.nats.jetstream.advisory.v1.terminated`| |Message acknowledged in a sampled Consumer|`$JS.EVENT.METRIC.CONSUMER.ACK..`|`io.nats.jetstream.metric.v1.consumer_ack`| +|Clustered Stream elected a new leader|`$JS.EVENT.ADVISORY.STREAM.LEADER_ELECTED.`|`io.nats.jetstream.advisory.v1.stream_leader_elected`| +|Clustered Stream lost quorum|`$JS.EVENT.ADVISORY.STREAM.QUORUM_LOST.`|`io.nats.jetstream.advisory.v1.stream_quorum_lost` +|Clustered Consumer elected a new leader|`$JS.EVENT.ADVISORY.CONSUMER.LEADER_ELECTED..`|`io.nats.jetstream.advisory.v1.consumer_leader_elected`| +|Clustered Consumer lost quorum|`$JS.EVENT.ADVISORY.CONSUMER.QUORUM_LOST..`|`io.nats.jetstream.advisory.v1.consumer_quorum_lost`| ### Dashboards -The [NATS Surveyor](https://github.com/nats-io/nats-surveyor) system has initial support for passing JetStream metrics to Prometheus, dashboards and more will be added towards final release. \ No newline at end of file +The [NATS Surveyor](https://github.com/nats-io/nats-surveyor) system has initial support for passing JetStream metrics to Prometheus, dashboards and more will be added towards final release. From 501ecd83dc3118455199a72133eb709f785dbb22 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 14:35:01 -0600 Subject: [PATCH 12/84] updates based on RI changes to README --- jetstream/administration/consumers.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/jetstream/administration/consumers.md b/jetstream/administration/consumers.md index e3e21ac..3506676 100644 --- a/jetstream/administration/consumers.md +++ b/jetstream/administration/consumers.md @@ -184,11 +184,11 @@ Acknowledged message You can prevent ACKs by supplying `--no-ack`. -To do this from code you'd send a `Request()` to `$JS.NEXT.ORDERS.DISPATCH`: +To do this from code you'd send a `Request()` to `$JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH`: ``` -$ nats req '$JS.NEXT.ORDERS.DISPATCH' '' -Published [$JS.NEXT.ORDERS.DISPATCH] : '' +$ nats req '$JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH' '' +Published [$JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH] : '' Received [ORDERS.processed] : 'order 3' ``` @@ -216,4 +216,4 @@ Listening on [monitor.ORDERS] Note the subject here of the received message is reported as `ORDERS.processed` this helps you distinguish what you're seeing in a Stream covering a wildcard, or multiple subject, subject space. -This Consumer needs no ack, so any new message into the ORDERS system will show up here in real time. \ No newline at end of file +This Consumer needs no ack, so any new message into the ORDERS system will show up here in real time. From deb7f26a01a3f6d4497501beb4aa33f2de3e0f47 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 14:39:47 -0600 Subject: [PATCH 13/84] updates based on RI changes to README --- .../nats_api_reference/nats_api_reference.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/jetstream/nats_api_reference/nats_api_reference.md b/jetstream/nats_api_reference/nats_api_reference.md index 1e994ed..afb8466 100644 --- a/jetstream/nats_api_reference/nats_api_reference.md +++ b/jetstream/nats_api_reference/nats_api_reference.md @@ -143,9 +143,24 @@ Events and Advisories: ``` $JS.EVENT.METRIC.CONSUMER_ACK.. $JS.EVENT.ADVISORY.MAX_DELIVERIES.. +$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.. +$JS.EVENT.ADVISORY.STREAM.CREATED. +$JS.EVENT.ADVISORY.STREAM.DELETED. +$JS.EVENT.ADVISORY.STREAM.UPDATED. +$JS.EVENT.ADVISORY.CONSUMER.CREATED.. +$JS.EVENT.ADVISORY.CONSUMER.DELETED.. +$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_CREATE. +$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_COMPLETE. +$JS.EVENT.ADVISORY.STREAM.RESTORE_CREATE. +$JS.EVENT.ADVISORY.STREAM.RESTORE_COMPLETE. +$JS.EVENT.ADVISORY.STREAM.LEADER_ELECTED. +$JS.EVENT.ADVISORY.STREAM.QUORUM_LOST. +$JS.EVENT.ADVISORY.CONSUMER.LEADER_ELECTED.. +$JS.EVENT.ADVISORY.CONSUMER.QUORUM_LOST.. +$JS.EVENT.ADVISORY.API ``` -This allow you to easily create ACL rules that limit users to a specific Stream or Consumer and to specific verbs for administration purposes. For ensuring only the receiver of a message can Ack it we have response permissions ensuring you can only Publish to Response subject for messages you received. +This design allows you to easily create ACL rules that limit users to a specific Stream or Consumer and to specific verbs for administration purposes. For ensuring only the receiver of a message can Ack it we have response permissions ensuring you can only Publish to Response subject for messages you received. ### Acknowledging Messages From 5f727f432a267577394a587b99ce446eee79083b Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 14:47:50 -0600 Subject: [PATCH 14/84] Updates based on RI changes to README --- jetstream/administration/account.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/jetstream/administration/account.md b/jetstream/administration/account.md index 33761ac..2053ef0 100644 --- a/jetstream/administration/account.md +++ b/jetstream/administration/account.md @@ -26,6 +26,7 @@ $ nats str add ORDERS ? Message size limit -1 ? Maximum message age limit 1y ? Maximum individual message size [? for help] (-1) -1 +? Number of replicas to store 3 Stream ORDERS was created Information for Stream ORDERS @@ -35,7 +36,7 @@ Configuration: Subjects: ORDERS.* Acknowledgements: true Retention: File - Limits - Replicas: 1 + Replicas: 3 Maximum Messages: -1 Maximum Bytes: -1 Maximum Age: 8760h0m0s @@ -54,10 +55,9 @@ Statistics: You can get prompted interactively for missing information as above, or do it all on one command. Pressing `?` in the CLI will help you map prompts to CLI options: ``` -$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old -``` +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --replicas 3``` -Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: +Additionally, one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: ``` $ nats str add ORDERS --config orders.json @@ -132,6 +132,20 @@ $ nats str info ORDERS -j This is the general pattern for the entire `nats` utility as it relates to JetStream - prompting for needed information but every action can be run non-interactively making it usable as a cli api. All information output like seen above can be turned into JSON using `-j`. +In clustered mode additional information will be included: + +```nohighlight +$ nats str info ORDERS +... +Cluster Information: + Name: JSC + Leader: S1 + Replica: S3, current, seen 0.04s ago + Replica: S2, current, seen 0.04s ago +``` + +Here the cluster name is configured as `JSC`, there is a server `S1` that's the current leader with `S3` and `S2` are replicas. Both replicas are current and have been seen recently. + #### Copying A stream can be copied into another, which also allows the configuration of the new one to be adjusted via CLI flags: From 7487129f1ea9c0be38fa8b52dba8d0ac05a68ade Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 14:52:16 -0600 Subject: [PATCH 15/84] Updates based on RI changes to README --- jetstream/concepts/streams.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jetstream/concepts/streams.md b/jetstream/concepts/streams.md index 861ec64..8b33f64 100644 --- a/jetstream/concepts/streams.md +++ b/jetstream/concepts/streams.md @@ -23,7 +23,7 @@ When defining Streams the items below make up the entire configuration of the se |MaxConsumers|How many Consumers can be defined for a given Stream, `-1` for unlimited| |Name|A name for the Stream that may not have spaces, tabs or `.`| |NoAck|Disables acknowledging messages that are received by the Stream| -|Replicas|How many replicas to keep for each message (not implemented as of January 2020)| +|Replicas|How many replicas to keep for each message in a clustered JetStream, maximum 5| |Retention|How message retention is considered, `LimitsPolicy` (default), `InterestPolicy` or `WorkQueuePolicy`| |Discard|When a Stream reached it's limits either, `DiscardNew` refuses new messages while `DiscardOld` (default) deletes old messages| |Storage|The type of storage backend, `file` and `memory` as of January 2020| From 56a90151bd6fd04ddb58d7a20f35e9dce6cb9087 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 14:54:13 -0600 Subject: [PATCH 16/84] update based on RI changes to README --- jetstream/monitoring/monitoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jetstream/monitoring/monitoring.md b/jetstream/monitoring/monitoring.md index 39add46..f95e510 100644 --- a/jetstream/monitoring/monitoring.md +++ b/jetstream/monitoring/monitoring.md @@ -8,7 +8,7 @@ Typically, NATS is monitored via HTTP endpoints like `/varz`, we do not at this JetStream publish a number of advisories that can inform operations about health and state of the Streams. These advisories are published to normal NATS subjects below `$JS.EVENT.ADVISORY.>` and one can store these advisories in JetStream Streams if desired. -The command `nats event --js-advsiroy` can view all these events on your console. The Golang package [jsm.go](https://github.com/nats-io/jsm.go) can consume and render these events and have data types for each of these events. +The command `nats event --js-advisory` can view all these events on your console. The Golang package [jsm.go](https://github.com/nats-io/jsm.go) can consume and render these events and have data types for each of these events. All these events have JSON Schemas that describe them, schemas can be viewed on the CLI using the `nats schema show ` command. From df06f3387162c48f50fe91aa39fef1c0774f46ea Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 14:59:41 -0600 Subject: [PATCH 17/84] updates based on RI changes to README --- jetstream/administration/administration.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/jetstream/administration/administration.md b/jetstream/administration/administration.md index 8dcd564..1abfecb 100644 --- a/jetstream/administration/administration.md +++ b/jetstream/administration/administration.md @@ -1,11 +1,13 @@ ## Administration and Usage from the CLI -Once the server is running it's time to use the management tool. This can be downloaded from the [GitHub Release Page](https://github.com/nats-io/jetstream/releases/) or you can use the `synadia/jsm:latest` docker image. +Once the server is running it's time to use the management tool. This can be downloaded from the [GitHub Release Page](https://github.com/nats-io/natscli/releases/) or you can use the `synadia/jsm:latest` docker image. On OS X homebrew can be used to install the latest version: -``` +```nohighlight +$ brew tap nats-io/nats-tools +$ brew install nats-io/nats-tools/nats $ nats --help usage: nats [] [ ...] -NATS Management Utility +NATS Utility Flags: --help Show context-sensitive help (also try --help-long and --help-man). @@ -27,4 +29,4 @@ We'll walk through the above scenario and introduce features of the CLI and of J Throughout this example, we'll show other commands like `nats pub` and `nats sub` to interact with the system. These are normal existing core NATS commands and JetStream is fully usable by only using core NATS. -We'll touch on some additional features but please review the section on the design model to understand all possible permutations. \ No newline at end of file +We'll touch on some additional features but please review the section on the design model to understand all possible permutations. From 8892475f8de93dcb1f70a22fca69dc9e1e37effc Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 15:01:52 -0600 Subject: [PATCH 18/84] updates based on RI changes to README --- jetstream/administration/account.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/jetstream/administration/account.md b/jetstream/administration/account.md index 2053ef0..7c3eecd 100644 --- a/jetstream/administration/account.md +++ b/jetstream/administration/account.md @@ -4,10 +4,20 @@ JetStream is multi-tenant so you will need to check that your account is enabled ```nohighlight $ nats account info - - Memory: 0 B of 6.4 GB - Storage: 0 B of 1.1 TB - Streams: 1 of Unlimited +Connection Information: + Client ID: 8 + Client IP: 127.0.0.1 + RTT: 178.545µs + Headers Supported: true + Maximum Payload: 1.0 MiB + Connected URL: nats://localhost:4222 + Connected Address: 127.0.0.1:4222 + Connected Server ID: NCCOHA6ONXJOGAEZP4WPU4UJ3IQP2VVXEPRKTQCGBCW4IL4YYW4V4KKL +JetStream Account Information: + Memory: 0 B of 5.7 GiB + Storage: 0 B of 11 GiB + Streams: 0 of Unlimited + Max Consumers: unlimited ``` ### Streams From 2b6d217e372b32fa40eafbfc402420a76607c790 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 15:07:16 -0600 Subject: [PATCH 19/84] updates from RI changes to README --- jetstream/concepts/streams.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jetstream/concepts/streams.md b/jetstream/concepts/streams.md index 8b33f64..87c8205 100644 --- a/jetstream/concepts/streams.md +++ b/jetstream/concepts/streams.md @@ -10,7 +10,7 @@ Streams can consume many subjects. Here we have `ORDERS.*` but we could also con Streams support various retention policies - they can be kept based on limits like max count, size or age but also more novel methods like keeping them as long as any Consumers have them unacknowledged, or work queue like behavior where a message is removed after first ack. -Streams support deduplication using a `Msg-Id` header and a sliding window within which to track duplicate messages. See the [Message Deduplication](#message-deduplication) section. +Streams support deduplication using a `Nats-Msg-Id` header and a sliding window within which to track duplicate messages. See the [Message Deduplication](#message-deduplication) section. When defining Streams the items below make up the entire configuration of the set. From 9d1e79110cd6b941152c79d3744020342c83716c Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 15:12:28 -0600 Subject: [PATCH 20/84] updates based on RI changes to README --- jetstream/model_deep_dive/model_deep_dive.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/jetstream/model_deep_dive/model_deep_dive.md b/jetstream/model_deep_dive/model_deep_dive.md index aa87c20..63f263f 100644 --- a/jetstream/model_deep_dive/model_deep_dive.md +++ b/jetstream/model_deep_dive/model_deep_dive.md @@ -30,16 +30,16 @@ The `WorkQueuePolicy` mode is a specialized mode where a message, once consumed ### Message Deduplication -JetStream support idempotent message writes by ignoring duplicate messages as indicated by the `Msg-Id` header. +JetStream support idempotent message writes by ignoring duplicate messages as indicated by the `Nats-Msg-Id` header. ```nohighlight -% nats req -H Msg-Id:1 ORDERS.new hello1 -% nats req -H Msg-Id:1 ORDERS.new hello2 -% nats req -H Msg-Id:1 ORDERS.new hello3 -% nats req -H Msg-Id:1 ORDERS.new hello4 +% nats req -H Nats-Msg-Id:1 ORDERS.new hello1 +% nats req -H Nats-Msg-Id:1 ORDERS.new hello2 +% nats req -H Nats-Msg-Id:1 ORDERS.new hello3 +% nats req -H Nats-Msg-Id:1 ORDERS.new hello4 ``` -Here we set a `Msg-Id:1` header which tells JetStream to ensure we do not have duplicates of this message - we only consult the message ID not the body. +Here we set a `Nats-Msg-Id:1` header which tells JetStream to ensure we do not have duplicates of this message - we only consult the message ID not the body. ```nohighlight $ nats str info ORDERS From b9d6095d36940ea5fd94b3e1041591452d54389d Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 15:13:36 -0600 Subject: [PATCH 21/84] updates based on RI changes to README --- jetstream/nats_api_reference/nats_api_reference.md | 1 - 1 file changed, 1 deletion(-) diff --git a/jetstream/nats_api_reference/nats_api_reference.md b/jetstream/nats_api_reference/nats_api_reference.md index afb8466..4d35174 100644 --- a/jetstream/nats_api_reference/nats_api_reference.md +++ b/jetstream/nats_api_reference/nats_api_reference.md @@ -61,7 +61,6 @@ The API uses JSON for inputs and outputs, all the responses are typed using a `t |-------|--------|-----------|---------------|----------------| |`$JS.API.STREAM.LIST`|`api.JSApiStreamList`|Paged list known Streams including all their current information|`api.JSApiStreamListRequest`|`api.JSApiStreamListResponse`| |`$JS.API.STREAM.NAMES`|`api.JSApiStreamNames`|Paged list of Streams|`api.JSApiStreamNamesRequest`|`api.JSApiStreamNamesResponse`| -|`$JS.API.STREAM.LOOKUP`|`api.JSApiStreamLookup`|Finds a stream with interest on a certain subject|`api.JSApiStreamLookupRequest`|`api.JSApiStreamLookupResponse`| |`$JS.API.STREAM.CREATE.*`|`api.JSApiStreamCreateT`|Creates a new Stream|`api.StreamConfig`|`api.JSApiStreamCreateResponse`| |`$JS.API.STREAM.UPDATE.*`|`api.JSApiStreamUpdateT`|Updates an existing Stream with new config|`api.StreamConfig`|`api.JSApiStreamUpdateResponse`| |`$JS.API.STREAM.INFO.*`|`api.JSApiStreamInfoT`|Information about config and state of a Stream|empty payload, Stream name in subject|`api.JSApiStreamInfoResponse`| From 70b13dbd5b969f086269c6db8ef5082f109eeda9 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 15:23:38 -0600 Subject: [PATCH 22/84] Move JetStream outline under NATS Server --- SUMMARY.md | 76 +++++++++++++++++++++++++++--------------------------- 1 file changed, 38 insertions(+), 38 deletions(-) diff --git a/SUMMARY.md b/SUMMARY.md index 4252c67..45e3ab7 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -105,6 +105,44 @@ * [Tutorial](nats-server/nats_docker/nats-docker-tutorial.md) * [Docker Swarm](nats-server/nats_docker/docker_swarm.md) * [Python and NGS Running in Docker](nats-server/nats_docker/ngs-docker-python.md) + +## JetStream + +* [About Jetstream](jetstream/about_jetstream/jetstream.md) +* [Concepts](jetstream/concepts/concepts.md) + * [Streams](jetstream/concepts/streams.md) + * [Consumes](jetstream/concepts/consumers.md) + * [Configuration](jetstream/concepts/configuration.md) +* [Getting Started](jetstream/getting_started/getting_started.md) + * [Using Docker](jetstream/getting_started/using_docker.md) + * [Using Docker with NGS](jetstream/getting_started/using_docker.md#using-docker-with-ngs) + * [Using Source](jetstream/getting_started/using_source.md) +* [Administration & Usage from CLI](jetstream/administration/administration.md) + * [Account Information](jetstream/administration/account.md) + * [Streams](jetstream/administration/streams.md) + * [Consumers](jetstream/administration/consumers.md) +* [Monitoring](jetstream/monitoring/monitoring.md) +* [Clustering](jetstream/clustering/clustering.md) + * [Administration](jetstream/clustering/administration.md) +* [Configuration Management](jetstream/configuration_mgmt/configuration_mgmt.md) + * [NATS Admin CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) + * [Terraform](jetstream/configuration_mgmt/configuration_mgmt.md#terraform) + * [GitHub Actions](jetstream/configuration_mgmt/github_actions.md) + * [Kubernetes Controller](jetstream/configuration_mgmt/kubernetes_controller.md) +* [Disaser Recovery](jetstream/disaster_recovery/disaster_recovery.md) +* [Model Deep Dive](jetstream/model_deep_dive/model_deep_dive.md) + * [Stream Limits, Retention Modes and Discard Policy](jetstream/model_deep_dive/model_deep_dive.md#stream-limits-retention-modes-and-discard-policy) + * [Message Deduplication](jetstream/model_deep_dive/model_deep_dive.md#message-deduplication) + * [Acknowledgment Models](jetstream/model_deep_dive/model_deep_dive.md#acknowledgement-models) + * [Exactly Once Delivery](jetstream/model_deep_dive/model_deep_dive.md#exactly-once-delivery) + * [Consumer Starting Position](jetstream/model_deep_dive/model_deep_dive.md#consumer-starting-position) + * [Ephemeral Consumers](jetstream/model_deep_dive/model_deep_dive.md#ephemeral-consumers) + * [Consumer Message Rates](jetstream/model_deep_dive/model_deep_dive.md#consumer-message-rates) + * [Stream Templates](jetstream/model_deep_dive/model_deep_dive.md#stream-templates) + * [Ack Sampling](jetstream/model_deep_dive/model_deep_dive.md#ack-sampling) + * [Storage Overhead](jetstream/model_deep_dive/model_deep_dive.md#storage-overhead) +* [NATS API Reference](jetstream/nats_api_reference/nats_api_reference.md) +* [Multi-tenancy & Resource Mgmt](jetstream/multi-tenancy/resource_management.md) ## NATS Tools @@ -203,41 +241,3 @@ * [Using a Load Balancer for External Access to NATS](nats-on-kubernetes/nats-external-nlb.md) * [Creating a NATS Super Cluster in Digital Ocean with Helm](nats-on-kubernetes/super-cluster-on-digital-ocean.md) * [From Zero to K8S to Leafnodes using Helm](nats-on-kubernetes/from-zero-to-leafnodes.md) - -## JetStream - -* [About Jetstream](jetstream/about_jetstream/jetstream.md) -* [Concepts](jetstream/concepts/concepts.md) - * [Streams](jetstream/concepts/streams.md) - * [Consumes](jetstream/concepts/consumers.md) - * [Configuration](jetstream/concepts/configuration.md) -* [Getting Started](jetstream/getting_started/getting_started.md) - * [Using Docker](jetstream/getting_started/using_docker.md) - * [Using Docker with NGS](jetstream/getting_started/using_docker.md#using-docker-with-ngs) - * [Using Source](jetstream/getting_started/using_source.md) -* [Administration & Usage from CLI](jetstream/administration/administration.md) - * [Account Information](jetstream/administration/account.md) - * [Streams](jetstream/administration/streams.md) - * [Consumers](jetstream/administration/consumers.md) -* [Monitoring](jetstream/monitoring/monitoring.md) -* [Clustering](jetstream/clustering/clustering.md) - * [Administration](jetstream/clustering/administration.md) -* [Configuration Management](jetstream/configuration_mgmt/configuration_mgmt.md) - * [NATS Admin CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) - * [Terraform](jetstream/configuration_mgmt/configuration_mgmt.md#terraform) - * [GitHub Actions](jetstream/configuration_mgmt/github_actions.md) - * [Kubernetes Controller](jetstream/configuration_mgmt/kubernetes_controller.md) -* [Disaser Recovery](jetstream/disaster_recovery/disaster_recovery.md) -* [Model Deep Dive](jetstream/model_deep_dive/model_deep_dive.md) - * [Stream Limits, Retention Modes and Discard Policy](jetstream/model_deep_dive/model_deep_dive.md#stream-limits-retention-modes-and-discard-policy) - * [Message Deduplication](jetstream/model_deep_dive/model_deep_dive.md#message-deduplication) - * [Acknowledgment Models](jetstream/model_deep_dive/model_deep_dive.md#acknowledgement-models) - * [Exactly Once Delivery](jetstream/model_deep_dive/model_deep_dive.md#exactly-once-delivery) - * [Consumer Starting Position](jetstream/model_deep_dive/model_deep_dive.md#consumer-starting-position) - * [Ephemeral Consumers](jetstream/model_deep_dive/model_deep_dive.md#ephemeral-consumers) - * [Consumer Message Rates](jetstream/model_deep_dive/model_deep_dive.md#consumer-message-rates) - * [Stream Templates](jetstream/model_deep_dive/model_deep_dive.md#stream-templates) - * [Ack Sampling](jetstream/model_deep_dive/model_deep_dive.md#ack-sampling) - * [Storage Overhead](jetstream/model_deep_dive/model_deep_dive.md#storage-overhead) -* [NATS API Reference](jetstream/nats_api_reference/nats_api_reference.md) -* [Multi-tenancy & Resource Mgmt](jetstream/multi-tenancy/resource_management.md) From 648be33d81a5f91684ba2a1ed9827ec36012bf2c Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 15:34:57 -0600 Subject: [PATCH 23/84] Jetstream ->JetStream --- jetstream/getting_started/getting_started.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/jetstream/getting_started/getting_started.md b/jetstream/getting_started/getting_started.md index 91b2364..6e5ff6c 100644 --- a/jetstream/getting_started/getting_started.md +++ b/jetstream/getting_started/getting_started.md @@ -1,16 +1,16 @@ # Getting Started -Getting started with Jetstream is straightforward. While we speak of Jetstream as if it is a seperate component, it's actually a subsystem built into the NATS server that needs to be enabled. +Getting started with JetStream is straightforward. While we speak of Jetstream as if it is a seperate component, it's actually a subsystem built into the NATS server that needs to be enabled. ## Command line -Enable Jetstream by specifying the `-js` flag when starting the NATS server. +Enable JetStream by specifying the `-js` flag when starting the NATS server. `$ nats-server -js` ## Configuration File -Enable Jetstream through a configuration file. By default, the Jetstream subsytem will store data in the /tmp directory. Here's a minimal file that will store data in a local "nats" directory, suitable for development and local testing. +Enable JetStream through a configuration file. By default, the JetStream subsytem will store data in the /tmp directory. Here's a minimal file that will store data in a local "nats" directory, suitable for development and local testing. `$ nats-server -c js.conf` @@ -21,6 +21,6 @@ jetstream { } ``` -Normally Jetstream will be run in clustered mode and will replicate data, so the best place to store Jetstream data would be locally on a fast SSD. One should specifically avoid NAS or NFS storage for Jetstream. +Normally JetStream will be run in clustered mode and will replicate data, so the best place to store JetStream data would be locally on a fast SSD. One should specifically avoid NAS or NFS storage for JetStream. -See [Using Docker](./using_docker.md) and [Using Source](./using_source.md) for more information. \ No newline at end of file +See [Using Docker](./using_docker.md) and [Using Source](./using_source.md) for more information. From 6e33b1cc7567c9c8eccca30724297137289d3b43 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Sat, 6 Feb 2021 11:15:16 -0700 Subject: [PATCH 24/84] Add lame duck mode documentation. Signed-off-by: Colin Sullivan --- SUMMARY.md | 1 + nats-server/nats_admin/README.md | 1 + nats-server/nats_admin/lame_duck_mode.md | 28 ++++++++++++++++++++++++ 3 files changed, 30 insertions(+) create mode 100644 nats-server/nats_admin/lame_duck_mode.md diff --git a/SUMMARY.md b/SUMMARY.md index 39a8193..ac9fbae 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -102,6 +102,7 @@ * [Upgrading a Cluster](nats-server/nats_admin/upgrading_cluster.md) * [Slow Consumers](nats-server/nats_admin/slow_consumers.md) * [Signals](nats-server/nats_admin/signals.md) + * [Lame Duck Mode](nats-server/nats_admin/lame_duck_mode.md) * [NATS and Docker](nats-server/nats_docker/README.md) * [Tutorial](nats-server/nats_docker/nats-docker-tutorial.md) * [Docker Swarm](nats-server/nats_docker/docker_swarm.md) diff --git a/nats-server/nats_admin/README.md b/nats-server/nats_admin/README.md index 4fc1a05..ef96055 100644 --- a/nats-server/nats_admin/README.md +++ b/nats-server/nats_admin/README.md @@ -8,4 +8,5 @@ Managing a NATS server is simple, typical lifecycle operations include: * Monitoring the server via: * The monitoring [endpoint](../configuration/monitoring.md) and tools like [nats-top](../../nats-tools/nats_top/) * By subscribing to [system events](../configuration/sys_accounts/) +* Gracefully shut down a server with [Lame Duck Mode](lame_duck_mode.md) diff --git a/nats-server/nats_admin/lame_duck_mode.md b/nats-server/nats_admin/lame_duck_mode.md new file mode 100644 index 0000000..dd9438b --- /dev/null +++ b/nats-server/nats_admin/lame_duck_mode.md @@ -0,0 +1,28 @@ +# Lame Duck Mode + +In production we recommend that a server is shut down with ​lame duck mode​ +as a graceful way to slowly evict clients. With large deployments this +mitigates the "thundering herd" situation that will place CPU pressure on +servers as TLS enabled clients reconnect. + +## Server + +Lame duck mode is initiated by signaling the server: + +```text +nats-server --signal ldm +``` + +After entering lame duck mode, the server will stop accepting new connections, +wait for a 10 second grace period, then begin to evict clients over a period of time +configurable by the [lame_duck_duration](https://docs.nats.io/nats-server/configuration#runtime-configuration) +configuration option. This period defaults to 2 minutes. + +## Clients + +When entering lame duck mode, the server will send a message to clients. Some +maintainer supported clients will invoke an optional callback indicating that +a server is entering lame duck mode. This is used for cases where an application +can benefit from preparing for the short outage between the time it is evicted and +automatically reconnected to another server. + From a4fb219bb1a115259383d138b4e613b0ea1c10fe Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Sat, 6 Feb 2021 15:54:08 -0700 Subject: [PATCH 25/84] add websocket configuration Signed-off-by: Colin Sullivan --- SUMMARY.md | 1 + nats-server/configuration/websockets.md | 60 +++++++++++++++++++++++++ 2 files changed, 61 insertions(+) create mode 100644 nats-server/configuration/websockets.md diff --git a/SUMMARY.md b/SUMMARY.md index 39a8193..f0eb82d 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -98,6 +98,7 @@ * [Monitoring](nats-server/configuration/monitoring.md) * [System Events](nats-server/configuration/sys_accounts/README.md) * [System Events & Decentralized JWT Tutorial](nats-server/configuration/sys_accounts/sys_accounts.md) + * [Websockets](nats-server/configuration/websockets.md) * [Managing A NATS Server](nats-server/nats_admin/README.md) * [Upgrading a Cluster](nats-server/nats_admin/upgrading_cluster.md) * [Slow Consumers](nats-server/nats_admin/slow_consumers.md) diff --git a/nats-server/configuration/websockets.md b/nats-server/configuration/websockets.md new file mode 100644 index 0000000..571f037 --- /dev/null +++ b/nats-server/configuration/websockets.md @@ -0,0 +1,60 @@ +# Websocket Support + +*Supported since NATS server version 2.2* + +Websocket support can be enabled in the server and may be used alongside the +traditional TCP socket connections. TLS, compression and +Origin Header checking are supported. + +To enable websocket support in the server, add a `websockets` configuration +block in the server's configuration file like the following: + +``` +websocket { + # Specify a host and port to listen for websocket connections + # listen: "host:port" + + # It can also be configured with individual parameters, + # namely host and port. + # host: "hostname" + # port: 4443 + + # This will optionally specify what host:port for websocket + # connections to be advertised in the cluster + # advertise: "host:port" + + # TLS configuration is required + tls { + cert_file: "/path/to/cert.pem" + key_file: "/path/to/key.pem" + } + + # If same_origin is true, then the Origin header of the + # client request must match the request's Host. + # same_origin: true + + # This list specifies the only accepted values for + # the client's request Origin header. The scheme, + # host and port must match. By convention, the + # absence of port for an http:// scheme will be 80, + # and for https:// will be 443. + # allowed_origins [ + # "http://www.example.com" + # "https://www.other-example.com" + # ] + + # This enables support for compressed websocket frames + # in the server. For compression to be used, both server + # and client have to support it. + # compression: true + + # This is the total time allowed for the server to + # read the client request and write the response back + # to the client. This includes the time needed for the + # TLS handshake. + # handshake_timeout: "2s" +} +``` + +Leaf nodes support outbound websocket connections by specifying the `ws` as the +scheme component of the remote server URL, for example `ws://hostname:4443`. \ No newline at end of file From 33acdbe454fb1d2c626f09cf4ce83f003c7c1e7a Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 8 Feb 2021 09:18:37 -0600 Subject: [PATCH 26/84] Update websockets.md --- nats-server/configuration/websockets.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nats-server/configuration/websockets.md b/nats-server/configuration/websockets.md index 571f037..64844fb 100644 --- a/nats-server/configuration/websockets.md +++ b/nats-server/configuration/websockets.md @@ -1,6 +1,6 @@ # Websocket Support -*Supported since NATS server version 2.2* +*Supported since NATS Server version 2.2* Websocket support can be enabled in the server and may be used alongside the traditional TCP socket connections. TLS, compression and @@ -57,4 +57,4 @@ websocket { ``` Leaf nodes support outbound websocket connections by specifying the `ws` as the -scheme component of the remote server URL, for example `ws://hostname:4443`. \ No newline at end of file +scheme component of the remote server URL, for example `ws://hostname:4443`. From 4f428bce64850fa97b0bc726075747d6c5cfc959 Mon Sep 17 00:00:00 2001 From: Ivan Kozlovic Date: Mon, 8 Feb 2021 09:59:02 -0700 Subject: [PATCH 27/84] Updates Signed-off-by: Ivan Kozlovic --- .../configuration/leafnodes/leafnode_conf.md | 26 +++++ nats-server/configuration/websockets.md | 103 +++++++++++++++++- 2 files changed, 123 insertions(+), 6 deletions(-) diff --git a/nats-server/configuration/leafnodes/leafnode_conf.md b/nats-server/configuration/leafnodes/leafnode_conf.md index a3df027..6f7e5aa 100644 --- a/nats-server/configuration/leafnodes/leafnode_conf.md +++ b/nats-server/configuration/leafnodes/leafnode_conf.md @@ -93,6 +93,32 @@ If other form of credentials are used \(jwt, nkey or other\), then the server wi | `account` | [Account](../securing_nats/accounts.md) name or jwt public key identifying the local account to bind to this remote server. Any traffic locally on this account will be forwarded to the remote server. | | `credentials` | Credential file for connecting to the leafnode server. | | `tls` | A [TLS configuration](leafnode_conf.md#tls-configuration-block) block. Leafnode client will use specified TLS certificates when connecting/authenticating. | +| `ws_compression` | If connecting with [Websocket](leafnode_conf#connecting-using-websocket-protocol) protocol, this boolean (`true` or `false`) indicates to the remote server that it wishes to use compression. The default is `false`. | +| `ws_no_masking` | If connecting with [Websocket](leafnode_conf#connecting-using-websocket-protocol) protocol, this boolean indicates to the remote server that it wishes not to mask outbound websocket frames. The default is `false`, which means that outbound frames will be masked. | + +### Connecting using Websocket protocol + +Since NATS 2.2.0, Leaf nodes support outbound websocket connections by specifying `ws` as the scheme component of the remote server URLs: +``` +leafnodes { + remotes [ + {urls: ["ws://hostname1:443", "ws://hostname2:443"]} + ] +} +``` + +Note that if a URL has the `ws` scheme, all URLs the list must be `ws`. You cannot mix and match. +Therefore this would be considered an invalid configuration: +``` + remotes [ + # Invalid configuration that will prevent the server from starting + {urls: ["ws://hostname1:443", "nats-leaf://hostname2:7422"]} + ] +``` + +Note that the decision to make a TLS connection is not based on `wss://` (as opposed to `ws://`) but instead in the presence of a TLS configuration in the `leafnodes{}` or the specific remote configuration block. + +To configure Websocket in the remote server, check the [Websocket](../websockets.md) secion. ### `tls` Configuration Block diff --git a/nats-server/configuration/websockets.md b/nats-server/configuration/websockets.md index 64844fb..4b0c1e4 100644 --- a/nats-server/configuration/websockets.md +++ b/nats-server/configuration/websockets.md @@ -17,18 +17,22 @@ websocket { # It can also be configured with individual parameters, # namely host and port. # host: "hostname" - # port: 4443 + port: 443 # This will optionally specify what host:port for websocket # connections to be advertised in the cluster # advertise: "host:port" - # TLS configuration is required + # TLS configuration is required by default tls { cert_file: "/path/to/cert.pem" key_file: "/path/to/key.pem" } + # For test environments, you can disable the need for TLS + # by explicitly setting this option to `true` + # no_tls: true + # If same_origin is true, then the Origin header of the # client request must match the request's Host. # same_origin: true @@ -36,8 +40,8 @@ websocket { # This list specifies the only accepted values for # the client's request Origin header. The scheme, # host and port must match. By convention, the - # absence of port for an http:// scheme will be 80, - # and for https:// will be 443. + # absence of TCP port in the URL will be port 80 + # for an "http://" scheme, and 443 for "https://". # allowed_origins [ # "http://www.example.com" # "https://www.other-example.com" @@ -53,8 +57,95 @@ websocket { # to the client. This includes the time needed for the # TLS handshake. # handshake_timeout: "2s" + + # Name of the cookie, which if present in WebSocket upgrade headers, + # will be treated as the JWT during CONNECT phase as long as + # the "jwt" field specified in the CONNECT protocol is missing or empty. + # Note that the server needs to be running in operator mode for this + # option to be used. + # jwt_cookie: "my_jwt_cookie_name" + + # If no user name is provided when a websocket client connects, will default + # this user name in the authentication phase. If specified, this will + # override, for websocket clients, any `no_auth_user` value defined in the + # main configuration file. + # Note that this is not compatible with running the server in operator mode. + # no_auth_user: "my_username_for_apps_not_providing_credentials" + + # See below to know what is the normal way of limiting websocket clients + # to specific users. + # If there are no users specified in the configuration, this simple authorization + # block allows you to override the values that would be configured in the + # equivalent block in the main section. + # authorization { + # # If this is specified, the client has to provide the same username + # # and password to be able to connect. + # # username: "my_user_name" + # # password: "my_password" + # + # # If this is specified, the password field in the CONNECT has to + # # match this token. + # # token: "my_token" + # + # # This overrides the main's authorization timeout. For consistency + # # with the main's authorization configuration block, this is expressed + # # as a number of seconds. + # # timeout: 2.0 + #} } ``` -Leaf nodes support outbound websocket connections by specifying the `ws` as the -scheme component of the remote server URL, for example `ws://hostname:4443`. +## Authorization of Websocket Users + +A new field when configuring users allows you to restrict which type of connections are allowed for a specific user. + +Consider this configuration: + +``` +authorization { + users [ + {user: foo password: foopwd, permission: {...}} + {user: bar password: barpwd, permission: {...}} + ] +} +``` + +If a websocket client were to connect and use the username `foo` and password `foopwd`, it would be accepted. +Now suppose that you would want websocket client to only be accepted if it connected using the username `bar` +and password `barpwd`, then you would use the option `allowed_connection_types` to restrict which type +of connections can bind to this user. + +``` +authorization { + users [ + {user: foo password: foopwd, permission: {...}} + {user: bar password: barpwd, permission: {...}, allowed_connection_types: ["WEBSOCKET"]} + ] +} +``` + +The option `allowed_connection_types` (also can be named `connection_types` or `clients`) as you can see +is a list, and you can allow several type of clients. Suppose you want the user `bar` to accept both +standard NATS clients and websocket clients, you would configure the user like this: + +``` +authorization { + users [ + {user: foo password: foopwd, permission: {...}} + {user: bar password: barpwd, permission: {...}, allowed_connection_types: ["STANDARD", "WEBSOCKET"]} + ] +} +``` + +The absence of `allowed_connection_types` means that all type of connections are allowed (the default behavior). + +The possible values are currently: +* `STANDARD` +* `WEBSOCKET` +* `LEAFNODE` +* `MQTT` + +## Leaf nodes connections + +You can configure remote Leaf node connections so that they connect to the Websocket port instead of the Leaf node port. +See [Leafnode](leafnodes/leafnode_conf.md#connecting-using-websocket-protocol) section. From 2e36c79791342d8f9f6888746131071c2c141dca Mon Sep 17 00:00:00 2001 From: Ivan Kozlovic Date: Mon, 8 Feb 2021 11:49:32 -0700 Subject: [PATCH 28/84] Some updates taken from Alberto's closed PR. Signed-off-by: Ivan Kozlovic --- nats-server/configuration/websockets.md | 37 +++++++++++++++++-------- 1 file changed, 25 insertions(+), 12 deletions(-) diff --git a/nats-server/configuration/websockets.md b/nats-server/configuration/websockets.md index 4b0c1e4..2cce11c 100644 --- a/nats-server/configuration/websockets.md +++ b/nats-server/configuration/websockets.md @@ -12,18 +12,22 @@ block in the server's configuration file like the following: ``` websocket { # Specify a host and port to listen for websocket connections + # # listen: "host:port" # It can also be configured with individual parameters, # namely host and port. + # # host: "hostname" port: 443 # This will optionally specify what host:port for websocket - # connections to be advertised in the cluster + # connections to be advertised in the cluster. + # # advertise: "host:port" # TLS configuration is required by default + # tls { cert_file: "/path/to/cert.pem" key_file: "/path/to/key.pem" @@ -31,17 +35,19 @@ websocket { # For test environments, you can disable the need for TLS # by explicitly setting this option to `true` + # # no_tls: true - # If same_origin is true, then the Origin header of the - # client request must match the request's Host. + # [Cross-origin resource sharing option](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). When set to `true`, the HTTP origin header must match the request’s hostname. + # The default is `false`. + # # same_origin: true - # This list specifies the only accepted values for - # the client's request Origin header. The scheme, - # host and port must match. By convention, the - # absence of TCP port in the URL will be port 80 + # [Cross-origin resource sharing option](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). List of accepted origins. When empty, and `same_origin` is `false`, clients from any origin are allowed to connect. + # This list specifies the only accepted values for the client's request Origin header. The scheme, + # host and port must match. By convention, the absence of TCP port in the URL will be port 80 # for an "http://" scheme, and 443 for "https://". + # # allowed_origins [ # "http://www.example.com" # "https://www.other-example.com" @@ -50,19 +56,24 @@ websocket { # This enables support for compressed websocket frames # in the server. For compression to be used, both server # and client have to support it. + # # compression: true # This is the total time allowed for the server to # read the client request and write the response back # to the client. This includes the time needed for the # TLS handshake. + # # handshake_timeout: "2s" - # Name of the cookie, which if present in WebSocket upgrade headers, - # will be treated as the JWT during CONNECT phase as long as - # the "jwt" field specified in the CONNECT protocol is missing or empty. - # Note that the server needs to be running in operator mode for this - # option to be used. + # Name for an HTTP cookie, that if present will be used as a client JWT. + # If the client specifies a JWT in the CONNECT protocol, this option is ignored. + # The cookie should be set by the HTTP server as described [here](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies). + # This setting is useful when generating NATS `Bearer` client JWTs as the + # result of some authentication mechanism. The HTTP server after correct + # authentication can issue a JWT for the user, that is set securely preventing + # access by unintended scripts. Note these JWTs must be [NATS JWTs](https://docs.nats.io/nats-server/configuration/securing_nats/jwt). + # # jwt_cookie: "my_jwt_cookie_name" # If no user name is provided when a websocket client connects, will default @@ -70,6 +81,7 @@ websocket { # override, for websocket clients, any `no_auth_user` value defined in the # main configuration file. # Note that this is not compatible with running the server in operator mode. + # # no_auth_user: "my_username_for_apps_not_providing_credentials" # See below to know what is the normal way of limiting websocket clients @@ -77,6 +89,7 @@ websocket { # If there are no users specified in the configuration, this simple authorization # block allows you to override the values that would be configured in the # equivalent block in the main section. + # # authorization { # # If this is specified, the client has to provide the same username # # and password to be able to connect. From 093b6459c673251f3280db583d3f09d666cd592f Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 8 Feb 2021 13:41:32 -0600 Subject: [PATCH 29/84] update link to JWT --- nats-server/configuration/websockets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-server/configuration/websockets.md b/nats-server/configuration/websockets.md index 2cce11c..235a8cd 100644 --- a/nats-server/configuration/websockets.md +++ b/nats-server/configuration/websockets.md @@ -72,7 +72,7 @@ websocket { # This setting is useful when generating NATS `Bearer` client JWTs as the # result of some authentication mechanism. The HTTP server after correct # authentication can issue a JWT for the user, that is set securely preventing - # access by unintended scripts. Note these JWTs must be [NATS JWTs](https://docs.nats.io/nats-server/configuration/securing_nats/jwt). + # access by unintended scripts. Note these JWTs must be [NATS JWTs](securing_nats/jwt). # # jwt_cookie: "my_jwt_cookie_name" From 6d00a5523b3f0c7b4bf3b9479eac07ee5b03fd61 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 8 Feb 2021 13:46:16 -0600 Subject: [PATCH 30/84] Update websockets.md --- nats-server/configuration/websockets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-server/configuration/websockets.md b/nats-server/configuration/websockets.md index 235a8cd..2cce11c 100644 --- a/nats-server/configuration/websockets.md +++ b/nats-server/configuration/websockets.md @@ -72,7 +72,7 @@ websocket { # This setting is useful when generating NATS `Bearer` client JWTs as the # result of some authentication mechanism. The HTTP server after correct # authentication can issue a JWT for the user, that is set securely preventing - # access by unintended scripts. Note these JWTs must be [NATS JWTs](securing_nats/jwt). + # access by unintended scripts. Note these JWTs must be [NATS JWTs](https://docs.nats.io/nats-server/configuration/securing_nats/jwt). # # jwt_cookie: "my_jwt_cookie_name" From 5cb7c796a1958d8435fcd96cdb7ab8abc7830c86 Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Tue, 9 Feb 2021 12:15:06 -0500 Subject: [PATCH 31/84] [added] description for jsz endpoint Signed-off-by: Matthias Hanel --- nats-server/configuration/monitoring.md | 142 ++++++++++++++++++++++++ 1 file changed, 142 insertions(+) diff --git a/nats-server/configuration/monitoring.md b/nats-server/configuration/monitoring.md index 0416bad..4ab0854 100644 --- a/nats-server/configuration/monitoring.md +++ b/nats-server/configuration/monitoring.md @@ -10,6 +10,7 @@ To monitor the NATS messaging system, `nats-server` provides a lightweight HTTP * [Gateways](monitoring.md#gateway-information) * [Leaf Nodes](monitoring.md#leaf-nodes-information) * [Subscription Routing](monitoring.md#subscription-routing-information) +* [Jetstream Information](monitoring.md#jetstream-information) All endpoints return a JSON object. @@ -520,6 +521,147 @@ The `/subsz` endpoint reports detailed information about the current subscriptio } ``` +## Jetstream Information + +The `/jsz` endpoint reports more detailed information on jetstream. For accounts it uses a paging mechanism which defaults to 1024 connections. + +**Endpoint:** `http://server:port/connz` + +| Result | Return Code | +| :--- | :--- | +| Success | 200 \(OK\) | +| Error | 400 \(Bad Request\) | + +#### Arguments + +| Argument | Values | Description | +| :--- | :--- | :--- | +| acc | account name | Include metrics for the specified account. Default is unset. | +| accounts | true, 1, false, 0 | Include account specific jetstream information. Default is false. | +| streams | true, 1, false, 0 | Include streams. When set, implies `accounts=true`. Default is false. | +| consumers | true, 1, false, 0 | Include consumer. When set, implies `streams=true`. Default is false. | +| config | true, 1, false, 0 | When stream or consumer are requested, include their respective configuration. Default is false. | +| leader-only | true, 1, false, 0 | Only the leader responds. Default is false.| +| offset | number > 0 | Pagination offset. Default is 0. | +| limit | number > 0 | Number of results to return. Default is 1024. | + +#### Examples + +Get basic jetstream information: [http://demo.nats.io:8222/jsz](http://demo.nats.io:8222/jsz) + +Request accounts and control limit and offset: [http://demo.nats.io:8222/jsz?accounts=true&limit=16&offset=128](http://demo.nats.io:8222/jsz?accounts=true&limit=16&offset=128). + +You can also report detailed consumer information on a per connection basis using consumer=true. For example: [http://demo.nats.io:8222/jsz?consumers=true](http://demo.nats.io:8222/jsz/consumer=true). + +#### Response + +```javascript +{ + "server_id": "NCVIDODSZ45C5OD67ZD7EJUIJPQDP6CM74SJX6TJIF2G7NLYS5LCVYHS", + "now": "2021-02-08T19:08:30.555533-05:00", + "config": { + "max_memory": 10485760, + "max_storage": 10485760, + "store_dir": "/var/folders/9h/6g_c9l6n6bb8gp331d_9y0_w0000gn/T/srv_7500251552558" + }, + "memory": 0, + "storage": 66, + "api": { + "total": 5, + "errors": 0 + }, + "total_streams": 1, + "total_consumers": 1, + "total_messages": 1, + "total_message_bytes": 33, + "meta_cluster": { + "name": "cluster_name", + "replicas": [ + { + "name": "server_5500", + "current": false, + "active": 2932926000 + } + ] + }, + "account_details": [ + { + "name": "BCC_TO_HAVE_ONE_EXTRA", + "id": "BCC_TO_HAVE_ONE_EXTRA", + "memory": 0, + "storage": 0, + "api": { + "total": 0, + "errors": 0 + } + }, + { + "name": "ACC", + "id": "ACC", + "memory": 0, + "storage": 66, + "api": { + "total": 5, + "errors": 0 + }, + "stream_detail": [ + { + "name": "my-stream-replicated", + "cluster": { + "name": "cluster_name", + "replicas": [ + { + "name": "server_5500", + "current": false, + "active": 2931517000 + } + ] + }, + "state": { + "messages": 1, + "bytes": 33, + "first_seq": 1, + "first_ts": "2021-02-09T00:08:27.623735Z", + "last_seq": 1, + "last_ts": "2021-02-09T00:08:27.623735Z", + "consumer_count": 1 + }, + "consumer_detail": [ + { + "stream_name": "my-stream-replicated", + "name": "my-consumer-replicated", + "created": "2021-02-09T00:08:27.427631Z", + "delivered": { + "consumer_seq": 0, + "stream_seq": 0 + }, + "ack_floor": { + "consumer_seq": 0, + "stream_seq": 0 + }, + "num_ack_pending": 0, + "num_redelivered": 0, + "num_waiting": 0, + "num_pending": 1, + "cluster": { + "name": "cluster_name", + "replicas": [ + { + "name": "server_5500", + "current": false, + "active": 2933232000 + } + ] + } + } + ] + } + ] + } + ] +} +``` + ## Creating Monitoring Applications NATS monitoring endpoints support [JSONP](https://en.wikipedia.org/wiki/JSONP) and [CORS](https://en.wikipedia.org/wiki/Cross-origin_resource_sharing#How_CORS_works). You can easily create single page web applications for monitoring. To do this you simply pass the `callback` query parameter to any endpoint. From d3dd6913fc557caad1e833f0277ab5f960def487 Mon Sep 17 00:00:00 2001 From: Karan Kumar <54062421+karankumarshreds@users.noreply.github.com> Date: Wed, 10 Feb 2021 00:22:01 +0530 Subject: [PATCH 32/84] Fixed apiVersion for eks-efs-provisioner (#212) * Fixed apiVersion for eks-efs-provisioner * Update stan-ft-k8s-aws.md --- nats-on-kubernetes/stan-ft-k8s-aws.md | 165 +++++++++++++------------- 1 file changed, 84 insertions(+), 81 deletions(-) diff --git a/nats-on-kubernetes/stan-ft-k8s-aws.md b/nats-on-kubernetes/stan-ft-k8s-aws.md index 3142343..6e72ee6 100644 --- a/nats-on-kubernetes/stan-ft-k8s-aws.md +++ b/nats-on-kubernetes/stan-ft-k8s-aws.md @@ -60,8 +60,9 @@ metadata: name: run-efs-provisioner subjects: - kind: ServiceAccount - name: efs-provisioner - # replace with namespace where provisioner is deployed + name: + efs-provisioner + # replace with namespace where provisioner is deployed namespace: default roleRef: kind: ClusterRole @@ -102,13 +103,16 @@ data: dns.name: "" --- kind: Deployment -apiVersion: extensions/v1beta1 +apiVersion: apps/v1 metadata: name: efs-provisioner spec: replicas: 1 + selector: + matchLabels: + app: efs-provisioner strategy: - type: Recreate + type: Recreate template: metadata: labels: @@ -172,14 +176,14 @@ spec: Result of deploying the manifest: ```bash -serviceaccount/efs-provisioner created -clusterrole.rbac.authorization.k8s.io/efs-provisioner-runner created -clusterrolebinding.rbac.authorization.k8s.io/run-efs-provisioner created -role.rbac.authorization.k8s.io/leader-locking-efs-provisioner created -rolebinding.rbac.authorization.k8s.io/leader-locking-efs-provisioner created -configmap/efs-provisioner created -deployment.extensions/efs-provisioner created -storageclass.storage.k8s.io/aws-efs created +serviceaccount/efs-provisioner created +clusterrole.rbac.authorization.k8s.io/efs-provisioner-runner created +clusterrolebinding.rbac.authorization.k8s.io/run-efs-provisioner created +role.rbac.authorization.k8s.io/leader-locking-efs-provisioner created +rolebinding.rbac.authorization.k8s.io/leader-locking-efs-provisioner created +configmap/efs-provisioner created +deployment.extensions/efs-provisioner created +storageclass.storage.k8s.io/aws-efs created persistentvolumeclaim/efs created ``` @@ -200,14 +204,14 @@ spec: app: stan clusterIP: None ports: - - name: client - port: 4222 - - name: cluster - port: 6222 - - name: monitor - port: 8222 - - name: metrics - port: 7777 + - name: client + port: 4222 + - name: cluster + port: 6222 + - name: monitor + port: 8222 + - name: metrics + port: 7777 --- apiVersion: v1 kind: ConfigMap @@ -270,69 +274,69 @@ spec: terminationGracePeriodSeconds: 30 containers: - - name: stan - image: nats-streaming:alpine + - name: stan + image: nats-streaming:alpine - ports: - # In case of NATS embedded mode expose these ports - - containerPort: 4222 - name: client - - containerPort: 6222 - name: cluster - - containerPort: 8222 - name: monitor - args: - - "-sc" - - "/etc/stan-config/stan.conf" + ports: + # In case of NATS embedded mode expose these ports + - containerPort: 4222 + name: client + - containerPort: 6222 + name: cluster + - containerPort: 8222 + name: monitor + args: + - "-sc" + - "/etc/stan-config/stan.conf" - # Required to be able to define an environment variable - # that refers to other environment variables. This env var - # is later used as part of the configuration file. - env: - - name: POD_NAME - valueFrom: - fieldRef: - fieldPath: metadata.name - - name: POD_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace - - name: CLUSTER_ADVERTISE - value: $(POD_NAME).stan.$(POD_NAMESPACE).svc - volumeMounts: - - name: config-volume - mountPath: /etc/stan-config - - name: efs - mountPath: /data/stan - resources: - requests: - cpu: 0 - livenessProbe: - httpGet: - path: / - port: 8222 - initialDelaySeconds: 10 - timeoutSeconds: 5 - - name: metrics - image: synadia/prometheus-nats-exporter:0.5.0 - args: - - -connz - - -routez - - -subz - - -varz - - -channelz - - -serverz - - http://localhost:8222 - ports: - - containerPort: 7777 - name: metrics + # Required to be able to define an environment variable + # that refers to other environment variables. This env var + # is later used as part of the configuration file. + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: CLUSTER_ADVERTISE + value: $(POD_NAME).stan.$(POD_NAMESPACE).svc + volumeMounts: + - name: config-volume + mountPath: /etc/stan-config + - name: efs + mountPath: /data/stan + resources: + requests: + cpu: 0 + livenessProbe: + httpGet: + path: / + port: 8222 + initialDelaySeconds: 10 + timeoutSeconds: 5 + - name: metrics + image: synadia/prometheus-nats-exporter:0.5.0 + args: + - -connz + - -routez + - -subz + - -varz + - -channelz + - -serverz + - http://localhost:8222 + ports: + - containerPort: 7777 + name: metrics volumes: - - name: config-volume - configMap: - name: stan-config - - name: efs - persistentVolumeClaim: - claimName: efs + - name: config-volume + configMap: + name: stan-config + - name: efs + persistentVolumeClaim: + claimName: efs ``` Your cluster now will look something like this: @@ -457,4 +461,3 @@ Subscribe to get all the messages: ```bash stan-sub -c stan -all foo ``` - From dad3450866b2eeba791009ecdb402b35f2a9ac5a Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Tue, 9 Feb 2021 16:42:21 -0500 Subject: [PATCH 33/84] [added] description of nats based account resolver Signed-off-by: Matthias Hanel --- .../securing_nats/jwt/resolver.md | 72 ++++++++++++++++++- 1 file changed, 69 insertions(+), 3 deletions(-) diff --git a/nats-server/configuration/securing_nats/jwt/resolver.md b/nats-server/configuration/securing_nats/jwt/resolver.md index f733a34..f8b5c6a 100644 --- a/nats-server/configuration/securing_nats/jwt/resolver.md +++ b/nats-server/configuration/securing_nats/jwt/resolver.md @@ -1,9 +1,10 @@ # Account lookup using Resolver -The `resolver` configuration option is used in conjunction with [NATS JWT Authentication](./) and [nsc](../../../../nats-tools/nsc/). The `resolver` option specifies a URL where the nats-server can retrieve an account JWT. There are two built-in resolver implementations: +The `resolver` configuration option is used in conjunction with [NATS JWT Authentication](./) and [nsc](../../../../nats-tools/nsc/). The `resolver` option specifies a URL where the nats-server can retrieve an account JWT. There are three built-in resolver implementations: -* `URL` -* `MEMORY` +* [`URL`](resolver.md#URL-Resolver) +* [`MEMORY`](resolver.md#Memory) +* [nats based resolver](resolver.md#nats-based-resolver) > If the operator JWT specified in `operator` contains an account resolver URL, `resolver` only needs to be specified in order to overwrite that default. @@ -34,3 +35,68 @@ The `MEMORY` resolver is recommended when the server has a small number of accou For more information on how to configure a memory resolver, see [this tutorial](mem_resolver.md). +## nats based resolver + +Nats based resolver embed the functionality of the [account server](https://github.com/nats-io/nats-account-server) inside the nats-server. +To not have to store all account jwt on every server, this resolver has two sub types `full` and `cache`. +Their commonalities are that they exchange/lookup account jwt via nats and the system account and store them in a local (not shared) directory. + +### full + +This resolver stores all jwt and exchanges them in an eventually consistent way with other resolver of the same type. +[`nsc`](../../../../nats-tools/nsc/README.md) supports push/pull/purge with this resolver type. +Jwt, uploaded this way, are stored in a directory the server has exclusive access to. + +```yaml +resolver: { + type: full + # Directory in which account jwt will be stored + dir: './jwt' + # In order to support jwt deletion, set to true + # If the resolver type is full delete will rename the jwt. + # This is to allow manual restoration in case of inadvertent deletion. + # To restore a jwt, remove the added suffix .delete and restart or send a reload signal. + # To free up storage you must manually delete files with the suffix .delete. + allow_delete: false + # Interval at which a nats-server with a nats based account resolver will compare + # it's state with one random nats based account resolver in the cluster and if needed, + # exchange jwt and converge on the same set of jwt. + interval: "2m" + # limit on the number of jwt stored, will reject new jwt once limit is hit. + limit: 1000 +} +``` + +This resolver type also supports `resolver_preload`. When present jwt listed are stored in the resolver. +There, they may be subject to updates. Restarts of the `nats-server` will hold on to these more recent versions. + +Not every server in a cluster needs to be set to `full`. +You need enough to still serve your workload adequately, while some server are offline. + +### cache + +This resolver only stores a subset of jwt and evicts extra ones based on an LRU scheme. +Missing jwt are downloaded from `full` nats based resolver. +This resolver is essentially the URL Resolver in nats. + +```yaml +resolver: { + type: cache + # Directory in which account jwt will be store + dir: "./" + # limit on the number of jwt stored, will evict old jwt once limit is hit. + limit: 1000 + # How long to hold on to a jwt before discarding it. + ttl: "2m" +} +``` + +### nats based resolver - integration + +nats based resolver utilize the system account for lookup and upload of account jwt. +If your application requires tighter integration you can make use of these subjects for tighter integration. + +To upload or update a possibly on the fly generated account jwt without `nsc`, send it as request to `$SYS.REQ.CLAIMS.UPDATE`. +Each participating `full` nats based account resolver will respond with a message detailing success or failure. + +To serve a requested account jwt yourself, subscribe to `$SYS.REQ.ACCOUNT.*.CLAIMS.LOOKUP` and respond with the account jwt corresponding to the requested account id (wildcard). From 354355f85bc19d7afdcadfe32102288c84f7cf5b Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Tue, 9 Feb 2021 17:08:18 -0500 Subject: [PATCH 34/84] mention nats based resolver in nats account server Signed-off-by: Matthias Hanel --- .../configuration/securing_nats/jwt/resolver.md | 10 +++++----- nats-tools/nas/README.md | 11 +++++++---- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/nats-server/configuration/securing_nats/jwt/resolver.md b/nats-server/configuration/securing_nats/jwt/resolver.md index f8b5c6a..4626609 100644 --- a/nats-server/configuration/securing_nats/jwt/resolver.md +++ b/nats-server/configuration/securing_nats/jwt/resolver.md @@ -45,7 +45,7 @@ Their commonalities are that they exchange/lookup account jwt via nats and the s This resolver stores all jwt and exchanges them in an eventually consistent way with other resolver of the same type. [`nsc`](../../../../nats-tools/nsc/README.md) supports push/pull/purge with this resolver type. -Jwt, uploaded this way, are stored in a directory the server has exclusive access to. +[JWTs](../../nats-server/configuration/securing_nats/jwt/), uploaded this way, are stored in a directory the server has exclusive access to. ```yaml resolver: { @@ -75,7 +75,7 @@ You need enough to still serve your workload adequately, while some server are o ### cache -This resolver only stores a subset of jwt and evicts extra ones based on an LRU scheme. +This resolver only stores a subset of [JWT](../../nats-server/configuration/securing_nats/jwt/) and evicts extra ones based on an LRU scheme. Missing jwt are downloaded from `full` nats based resolver. This resolver is essentially the URL Resolver in nats. @@ -93,10 +93,10 @@ resolver: { ### nats based resolver - integration -nats based resolver utilize the system account for lookup and upload of account jwt. +nats based resolver utilize the system account for lookup and upload of account [JWTs](../../nats-server/configuration/securing_nats/jwt/) . If your application requires tighter integration you can make use of these subjects for tighter integration. -To upload or update a possibly on the fly generated account jwt without `nsc`, send it as request to `$SYS.REQ.CLAIMS.UPDATE`. +To upload or update a possibly on the fly generated account jwt without [`nsc`](../../../../nats-tools/nsc/README.md), send it as request to `$SYS.REQ.CLAIMS.UPDATE`. Each participating `full` nats based account resolver will respond with a message detailing success or failure. -To serve a requested account jwt yourself, subscribe to `$SYS.REQ.ACCOUNT.*.CLAIMS.LOOKUP` and respond with the account jwt corresponding to the requested account id (wildcard). +To serve a requested account [JWT](../../nats-server/configuration/securing_nats/jwt/) yourself and essentially implement an account server, subscribe to `$SYS.REQ.ACCOUNT.*.CLAIMS.LOOKUP` and respond with the account jwt corresponding to the requested account id (wildcard). diff --git a/nats-tools/nas/README.md b/nats-tools/nas/README.md index 9ad7f48..97d3813 100644 --- a/nats-tools/nas/README.md +++ b/nats-tools/nas/README.md @@ -1,11 +1,14 @@ # nats-account-server -The [NATS Account Server](https://github.com/nats-io/nats-account-server) is an HTTP server that hosts and vends [JWTs](../../nats-server/configuration/securing_nats/jwt/) for nats-server 2.0 account authentication. The server supports an number of stores which enable it to serve account [JWTs](../../nats-server/configuration/securing_nats/jwt/) from: - -* a [directory](nas_conf.md#directory-configuration) -* an [NSC](../nsc/nsc.md) [directory](nas_conf.md#nsc-configuration) +The [NATS Account Server](https://github.com/nats-io/nats-account-server) is an HTTP server that hosts and vends [JWTs](../../nats-server/configuration/securing_nats/jwt/) for nats-server 2.0 account authentication. The server supports an number of stores which enable it to serve account [JWTs](../../nats-server/configuration/securing_nats/jwt/) from a [directory](nas_conf.md#directory-configuration) > The nats server can be configured with a [memory resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#memory) as well. This avoids usage of the account server. +> The nats server can be configured with a [nats based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) for the same purpose as well. +> +> Usage of [full nats based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) over [NATS Account Server](https://github.com/nats-io/nats-account-server) is recommended. +> +> The [NATS Account Server](https://github.com/nats-io/nats-account-server) also speaks the [full nats based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) protocol and +> can be used as such. The server can operate in a _READ ONLY_ mode where it serves content from a directory, or in [notification mode](notifications.md), where it can notify a NATS server that a JWT in the store has been modified, updating the NATS server with the updated JWT. From 8491031e1765d5459e07d6f88b76e9f04cb2502e Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Tue, 9 Feb 2021 17:35:39 -0500 Subject: [PATCH 35/84] [removed] -nsc option and config from account server doc Signed-off-by: Matthias Hanel --- nats-tools/nas/nas_conf.md | 59 +------------------------------------- 1 file changed, 1 insertion(+), 58 deletions(-) diff --git a/nats-tools/nas/nas_conf.md b/nats-tools/nas/nas_conf.md index 05f95d9..8a594b6 100644 --- a/nats-tools/nas/nas_conf.md +++ b/nats-tools/nas/nas_conf.md @@ -9,62 +9,6 @@ Basic configuration revolves around 4 settings: For complete information on please refer to the project's [Github](https://github.com/nats-io/nats-account-server). -## `nsc` Configuration - -For a basic usage of the server you can specify the `-nsc` flag, and specify the path to an operator in your environment. - -> If you have not yet created an operator or accounts, you'll need to do so before continuing. See [NSC](../nsc/) - -You can easily locate the path by running `nsc env` to print your `nsc` settings: - -```text -> nsc env -╭──────────────────────────────────────────╮ -│ NSC Environment │ -├──────────────────┬─────┬─────────────────┤ -│ Setting │ Set │ Effective Value │ -├──────────────────┼─────┼─────────────────┤ -│ $NKEYS_PATH │ No │ ~/.nkeys │ -│ $NSC_HOME │ No │ ~/.nsc │ -│ Config │ │ ~/.nsc/nsc.json │ -├──────────────────┼─────┼─────────────────┤ -│ Stores Dir │ │ ~/.nsc/nats │ -│ Default Operator │ │ Test │ -│ Default Account │ │ TestAccount │ -│ Default Cluster │ │ │ -╰──────────────────┴─────┴─────────────────╯ -``` - -The path you are interested in the `Stores Dir`. This is the root of all operators, you'll also need the name of your operator. If your current operator is not listed, you can list all your available operators by doing: - -```text -> nsc list operators -``` - -To start the `nats-account-server` with the operator `Test`: - -```text -> nats-account-server -nsc ~/.nsc/nats/Test -2019/05/10 11:22:41.845148 [INF] starting NATS Account server, version 0.0-dev -2019/05/10 11:22:41.845241 [INF] server time is Fri May 10 11:22:41 CDT 2019 -2019/05/10 11:22:41.845245 [INF] loading operator from /Users/synadia/.nsc/nats/Test/Test.jwt -2019/05/10 11:22:41.846367 [INF] creating a read-only store for the NSC folder at /Users/synadia/.nsc/nats/Test -2019/05/10 11:22:41.846459 [INF] NATS is not configured, server will not fire notifications on update -2019/05/10 11:22:41.855291 [INF] http listening on port 9090 -2019/05/10 11:22:41.855301 [INF] nats-account-server is running -2019/05/10 11:22:41.855303 [INF] configure the nats-server with: -2019/05/10 11:22:41.855306 [INF] resolver: URL(http://localhost:9090/jwt/v1/accounts/) -``` - -By default the server will serve JWTs on the localhost at port 9090. The last line in the shown in the printout is important, that is the resolver URL you'll have to provide on your NATS server configuration. You'll also need the matching operator JWT which is on `~/.nsc/nats/Test/Test.jwt` if you are following the example above. On the server configuration you'll need to expand the `~` as necessary. Here's what my NATS server configuration looks like: - -```text -operator: /Users/synadia/.nsc/nats/Test/Test.jwt -resolver: URL(http://localhost:9090/jwt/v1/accounts/) -``` - -Note that servers you create with the `-nsc` option \(or store option\) are read-only. This means that the server will not accept POST requests to update the JWT store. - ## Directory Configuration You can start a server using a plain directory. In this case you'll be responsible for adding any JWT that you want resolved. @@ -95,7 +39,7 @@ A step by step tutorial using directory configuration can be found [here](dir_st ## Configuration File -While the `-nsc` and `-dir` store flags are sufficient for some very simple developer setups, any production or non-read-only server will require a configuration file. +While the `-dir` store flag is sufficient for some very simple developer setups, any production or non-read-only server will require a configuration file. Let's take a look at the configuration options: @@ -117,7 +61,6 @@ Let's take a look at the configuration options: | Option | Description | | :--- | :--- | | `dir` | Configures a directory as a store. | -| `nsc` | Configures an nsc read-only store. The value should be the path to an operator _directory_. Option is mutually exclusive with `dir`. | | `readonly` | If `true`, the store will not accept POST requests. Note that to receive requests, the store must also have `operatorjwtpath` specified as a root option. | | `shard` | If `true`, JWTs will be stored in multiple sub directories of the store directory. | From a3224f0245cd7ce0972f57a83ca1218d77f4e2ec Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Tue, 9 Feb 2021 16:50:24 -0600 Subject: [PATCH 36/84] updates based on RI updates to README 020921 --- jetstream/clustering/administration.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/jetstream/clustering/administration.md b/jetstream/clustering/administration.md index 941ee69..92efe7f 100644 --- a/jetstream/clustering/administration.md +++ b/jetstream/clustering/administration.md @@ -32,9 +32,9 @@ The `current` indicates that followers are up to date and have all the messages, The replica count cannot be edited once configured. -## Forcing leader election +## Forcing Stream and Consumer leader election -Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time. +Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process, but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time. Moving leadership away from a node does not remove it from the cluster and does not prevent it from becoming a leader again, this is merely a triggered leader election. @@ -53,6 +53,18 @@ Cluster Information: Replica: n3-c1, current, seen 0.12s ago ``` +The same is true for consumers, `nats consumer cluster step-down ORDERS NEW`. + +#### Forcing Meta Group leader election + +Similar to Streams and Consumers above the Meta Group allows leader stand down. The Meta Group is cluster wide and spans all accounts, therefore to manage the meta group you have to use a `SYSTEM` user. + +```nohighlight +$ nats server raft step-down --user system +17:44:24 Current leader: n2-c2 +17:44:24 New leader: n1-c2 +``` + ## Evicting a peer Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down. From b7aecc69769157c79547a3d632c73991249a6fce Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Wed, 10 Feb 2021 13:34:54 -0500 Subject: [PATCH 37/84] Capitalize NATS Signed-off-by: Matthias Hanel --- nats-server/configuration/securing_nats/jwt/resolver.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/nats-server/configuration/securing_nats/jwt/resolver.md b/nats-server/configuration/securing_nats/jwt/resolver.md index 4626609..80cd0f5 100644 --- a/nats-server/configuration/securing_nats/jwt/resolver.md +++ b/nats-server/configuration/securing_nats/jwt/resolver.md @@ -4,7 +4,7 @@ The `resolver` configuration option is used in conjunction with [NATS JWT Authen * [`URL`](resolver.md#URL-Resolver) * [`MEMORY`](resolver.md#Memory) -* [nats based resolver](resolver.md#nats-based-resolver) +* [NATS Based Resolver](resolver.md#nats-based-resolver) > If the operator JWT specified in `operator` contains an account resolver URL, `resolver` only needs to be specified in order to overwrite that default. @@ -35,11 +35,11 @@ The `MEMORY` resolver is recommended when the server has a small number of accou For more information on how to configure a memory resolver, see [this tutorial](mem_resolver.md). -## nats based resolver +## NATS Based Resolver -Nats based resolver embed the functionality of the [account server](https://github.com/nats-io/nats-account-server) inside the nats-server. +NATS based resolver embed the functionality of the [account server](https://github.com/nats-io/nats-account-server) inside the nats-server. To not have to store all account jwt on every server, this resolver has two sub types `full` and `cache`. -Their commonalities are that they exchange/lookup account jwt via nats and the system account and store them in a local (not shared) directory. +Their commonalities are that they exchange/lookup account jwt via NATS and the system account and store them in a local (not shared) directory. ### full From 44fa12e358906c5c56abba0b0e8c64f67631c8b1 Mon Sep 17 00:00:00 2001 From: Ivan Kozlovic Date: Wed, 10 Feb 2021 13:18:37 -0700 Subject: [PATCH 38/84] Add note about frame types and content of a frame wrt NATS protocols Signed-off-by: Ivan Kozlovic --- nats-server/configuration/websockets.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/nats-server/configuration/websockets.md b/nats-server/configuration/websockets.md index 2cce11c..7e742d2 100644 --- a/nats-server/configuration/websockets.md +++ b/nats-server/configuration/websockets.md @@ -6,6 +6,11 @@ Websocket support can be enabled in the server and may be used alongside the traditional TCP socket connections. TLS, compression and Origin Header checking are supported. +**Important** + +- NATS Supports only Websocket data frames in Binary, not Text format (https://tools.ietf.org/html/rfc6455#section-5.6). The server will always send in Binary and your clients MUST send in Binary too. +- For writers of client libraries: a Websocket frame is not guaranteed to contain a full NATS protocol (actually will generally not). Any data from a frame must be going through a parser that can handle partial protocols. See the protocol description [here](../../nats-protocol/nats-protocol/README.md). + To enable websocket support in the server, add a `websockets` configuration block in the server's configuration file like the following: From 9edf22f7b8b0aebd1c085c8992affce3b6f21219 Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Wed, 10 Feb 2021 15:48:45 -0500 Subject: [PATCH 39/84] Update nats-server/configuration/securing_nats/jwt/resolver.md Co-authored-by: Colin Sullivan --- nats-server/configuration/securing_nats/jwt/resolver.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-server/configuration/securing_nats/jwt/resolver.md b/nats-server/configuration/securing_nats/jwt/resolver.md index 80cd0f5..f145800 100644 --- a/nats-server/configuration/securing_nats/jwt/resolver.md +++ b/nats-server/configuration/securing_nats/jwt/resolver.md @@ -41,7 +41,7 @@ NATS based resolver embed the functionality of the [account server](https://gith To not have to store all account jwt on every server, this resolver has two sub types `full` and `cache`. Their commonalities are that they exchange/lookup account jwt via NATS and the system account and store them in a local (not shared) directory. -### full +### Full This resolver stores all jwt and exchanges them in an eventually consistent way with other resolver of the same type. [`nsc`](../../../../nats-tools/nsc/README.md) supports push/pull/purge with this resolver type. From 81f67989f613ae6f789ee7de3aa37fba69c3a7cc Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Wed, 10 Feb 2021 15:49:10 -0500 Subject: [PATCH 40/84] Update nats-server/configuration/securing_nats/jwt/resolver.md Co-authored-by: Colin Sullivan --- nats-server/configuration/securing_nats/jwt/resolver.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-server/configuration/securing_nats/jwt/resolver.md b/nats-server/configuration/securing_nats/jwt/resolver.md index f145800..15eb685 100644 --- a/nats-server/configuration/securing_nats/jwt/resolver.md +++ b/nats-server/configuration/securing_nats/jwt/resolver.md @@ -67,7 +67,7 @@ resolver: { } ``` -This resolver type also supports `resolver_preload`. When present jwt listed are stored in the resolver. +This resolver type also supports `resolver_preload`. When present, JWTs are listed are stored in the resolver. There, they may be subject to updates. Restarts of the `nats-server` will hold on to these more recent versions. Not every server in a cluster needs to be set to `full`. From 65eb601578dfb386cee21c0d4d103a5862625572 Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Wed, 10 Feb 2021 15:49:17 -0500 Subject: [PATCH 41/84] Update nats-server/configuration/securing_nats/jwt/resolver.md Co-authored-by: Colin Sullivan --- nats-server/configuration/securing_nats/jwt/resolver.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-server/configuration/securing_nats/jwt/resolver.md b/nats-server/configuration/securing_nats/jwt/resolver.md index 15eb685..054b9fe 100644 --- a/nats-server/configuration/securing_nats/jwt/resolver.md +++ b/nats-server/configuration/securing_nats/jwt/resolver.md @@ -73,7 +73,7 @@ There, they may be subject to updates. Restarts of the `nats-server` will hold o Not every server in a cluster needs to be set to `full`. You need enough to still serve your workload adequately, while some server are offline. -### cache +### Cache This resolver only stores a subset of [JWT](../../nats-server/configuration/securing_nats/jwt/) and evicts extra ones based on an LRU scheme. Missing jwt are downloaded from `full` nats based resolver. From 897fbe1c3b6900151e39750f5977a7d685e62cdf Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Wed, 10 Feb 2021 15:49:26 -0500 Subject: [PATCH 42/84] Update nats-server/configuration/securing_nats/jwt/resolver.md Co-authored-by: Colin Sullivan --- nats-server/configuration/securing_nats/jwt/resolver.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-server/configuration/securing_nats/jwt/resolver.md b/nats-server/configuration/securing_nats/jwt/resolver.md index 054b9fe..78d614e 100644 --- a/nats-server/configuration/securing_nats/jwt/resolver.md +++ b/nats-server/configuration/securing_nats/jwt/resolver.md @@ -91,7 +91,7 @@ resolver: { } ``` -### nats based resolver - integration +### NATS Based Resolver - Integration nats based resolver utilize the system account for lookup and upload of account [JWTs](../../nats-server/configuration/securing_nats/jwt/) . If your application requires tighter integration you can make use of these subjects for tighter integration. From 03fc69af676cb56542710978500d575473e47385 Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Wed, 10 Feb 2021 15:49:36 -0500 Subject: [PATCH 43/84] Update nats-server/configuration/securing_nats/jwt/resolver.md Co-authored-by: Colin Sullivan --- nats-server/configuration/securing_nats/jwt/resolver.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-server/configuration/securing_nats/jwt/resolver.md b/nats-server/configuration/securing_nats/jwt/resolver.md index 78d614e..4375211 100644 --- a/nats-server/configuration/securing_nats/jwt/resolver.md +++ b/nats-server/configuration/securing_nats/jwt/resolver.md @@ -93,7 +93,7 @@ resolver: { ### NATS Based Resolver - Integration -nats based resolver utilize the system account for lookup and upload of account [JWTs](../../nats-server/configuration/securing_nats/jwt/) . +The NATS based resolver utilizes the system account for lookup and upload of account [JWTs](../../nats-server/configuration/securing_nats/jwt/) . If your application requires tighter integration you can make use of these subjects for tighter integration. To upload or update a possibly on the fly generated account jwt without [`nsc`](../../../../nats-tools/nsc/README.md), send it as request to `$SYS.REQ.CLAIMS.UPDATE`. From 1fb41bd6d9d3fdcc4d483f4b3597ac1a7f96c06a Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Wed, 10 Feb 2021 15:49:53 -0500 Subject: [PATCH 44/84] Update nats-server/configuration/securing_nats/jwt/resolver.md Co-authored-by: Colin Sullivan --- nats-server/configuration/securing_nats/jwt/resolver.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-server/configuration/securing_nats/jwt/resolver.md b/nats-server/configuration/securing_nats/jwt/resolver.md index 4375211..cfc0958 100644 --- a/nats-server/configuration/securing_nats/jwt/resolver.md +++ b/nats-server/configuration/securing_nats/jwt/resolver.md @@ -96,7 +96,7 @@ resolver: { The NATS based resolver utilizes the system account for lookup and upload of account [JWTs](../../nats-server/configuration/securing_nats/jwt/) . If your application requires tighter integration you can make use of these subjects for tighter integration. -To upload or update a possibly on the fly generated account jwt without [`nsc`](../../../../nats-tools/nsc/README.md), send it as request to `$SYS.REQ.CLAIMS.UPDATE`. +To upload or update any generated account jwt without [`nsc`](../../../../nats-tools/nsc/README.md), send it as request to `$SYS.REQ.CLAIMS.UPDATE`. Each participating `full` nats based account resolver will respond with a message detailing success or failure. To serve a requested account [JWT](../../nats-server/configuration/securing_nats/jwt/) yourself and essentially implement an account server, subscribe to `$SYS.REQ.ACCOUNT.*.CLAIMS.LOOKUP` and respond with the account jwt corresponding to the requested account id (wildcard). From 300cd6511460d4504a5052381c4a60385f6127af Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Wed, 10 Feb 2021 15:49:59 -0500 Subject: [PATCH 45/84] Update nats-tools/nas/README.md Co-authored-by: Colin Sullivan --- nats-tools/nas/README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/nats-tools/nas/README.md b/nats-tools/nas/README.md index 97d3813..88767f1 100644 --- a/nats-tools/nas/README.md +++ b/nats-tools/nas/README.md @@ -3,7 +3,7 @@ The [NATS Account Server](https://github.com/nats-io/nats-account-server) is an HTTP server that hosts and vends [JWTs](../../nats-server/configuration/securing_nats/jwt/) for nats-server 2.0 account authentication. The server supports an number of stores which enable it to serve account [JWTs](../../nats-server/configuration/securing_nats/jwt/) from a [directory](nas_conf.md#directory-configuration) > The nats server can be configured with a [memory resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#memory) as well. This avoids usage of the account server. -> The nats server can be configured with a [nats based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) for the same purpose as well. +> The NATS server can be configured with a [NATS based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) for the same purpose as well. > > Usage of [full nats based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) over [NATS Account Server](https://github.com/nats-io/nats-account-server) is recommended. > @@ -17,4 +17,3 @@ The server supports replica mode, which allows load balancing, fault tolerance a The account server can host activation tokens as well as account JWTs. These tokens are used when one account needs to give permission to another account to access a private export. Tokens can be configured as full tokens, or URLs. By hosting them in the account server you can avoid the copy/paste process of embedding tokens. They can also be updated more easily on expiration. The account serer furthermore allows for jwt inspection. All account server configuration options can be found [here](nas_conf.md#configuration-file). It futhermore allows [inspection](inspecting_jwts.md) of JWT. - From c60a0bb5357199771b1fafe9dcc77468e7d6151a Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Wed, 10 Feb 2021 15:50:12 -0500 Subject: [PATCH 46/84] Update nats-tools/nas/README.md Co-authored-by: Colin Sullivan --- nats-tools/nas/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-tools/nas/README.md b/nats-tools/nas/README.md index 88767f1..3cac2be 100644 --- a/nats-tools/nas/README.md +++ b/nats-tools/nas/README.md @@ -5,7 +5,7 @@ The [NATS Account Server](https://github.com/nats-io/nats-account-server) is an > The nats server can be configured with a [memory resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#memory) as well. This avoids usage of the account server. > The NATS server can be configured with a [NATS based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) for the same purpose as well. > -> Usage of [full nats based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) over [NATS Account Server](https://github.com/nats-io/nats-account-server) is recommended. +> Usage of [full NATS based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) over [NATS Account Server](https://github.com/nats-io/nats-account-server) is recommended. > > The [NATS Account Server](https://github.com/nats-io/nats-account-server) also speaks the [full nats based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) protocol and > can be used as such. From e3da8cfe6a7b2398a9b69c2a7bc7588377ed41d8 Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Wed, 10 Feb 2021 15:50:22 -0500 Subject: [PATCH 47/84] Update nats-server/configuration/securing_nats/jwt/resolver.md Co-authored-by: Colin Sullivan --- nats-server/configuration/securing_nats/jwt/resolver.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-server/configuration/securing_nats/jwt/resolver.md b/nats-server/configuration/securing_nats/jwt/resolver.md index cfc0958..eee71dc 100644 --- a/nats-server/configuration/securing_nats/jwt/resolver.md +++ b/nats-server/configuration/securing_nats/jwt/resolver.md @@ -75,7 +75,7 @@ You need enough to still serve your workload adequately, while some server are o ### Cache -This resolver only stores a subset of [JWT](../../nats-server/configuration/securing_nats/jwt/) and evicts extra ones based on an LRU scheme. +This resolver only stores a subset of [JWT](../../nats-server/configuration/securing_nats/jwt/) and evicts others based on an LRU scheme. Missing jwt are downloaded from `full` nats based resolver. This resolver is essentially the URL Resolver in nats. From ab5b1a4565d932554cb960e1c4b19d224fb88c87 Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Wed, 10 Feb 2021 15:50:31 -0500 Subject: [PATCH 48/84] Update nats-server/configuration/securing_nats/jwt/resolver.md Co-authored-by: Colin Sullivan --- nats-server/configuration/securing_nats/jwt/resolver.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-server/configuration/securing_nats/jwt/resolver.md b/nats-server/configuration/securing_nats/jwt/resolver.md index eee71dc..fe9bee8 100644 --- a/nats-server/configuration/securing_nats/jwt/resolver.md +++ b/nats-server/configuration/securing_nats/jwt/resolver.md @@ -1,4 +1,4 @@ -# Account lookup using Resolver +# Account Lookup Using a Resolver The `resolver` configuration option is used in conjunction with [NATS JWT Authentication](./) and [nsc](../../../../nats-tools/nsc/). The `resolver` option specifies a URL where the nats-server can retrieve an account JWT. There are three built-in resolver implementations: From 20046aa2ebdb332b9d7cf0bc9ac3a0b85d6e54c5 Mon Sep 17 00:00:00 2001 From: Bill Hass Date: Wed, 10 Feb 2021 16:34:11 -0500 Subject: [PATCH 49/84] Clarify Example Gateway Connection Logic The current description reads like each external gateway, B1 & B2, should have three connections from the local cluster because the local cluster has three gateway nodes. However, my understanding is that the external gateway _cluster_ will have three connections from the local cluster. Added cluster at the end of sentence and clarified there will be three outbound connections from the local cluster. --- nats-server/configuration/gateways/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-server/configuration/gateways/README.md b/nats-server/configuration/gateways/README.md index 89ce668..0bd811e 100644 --- a/nats-server/configuration/gateways/README.md +++ b/nats-server/configuration/gateways/README.md @@ -27,7 +27,7 @@ If gateways are to be used in a cluster, **all** servers of this cluster need to A nats-server in a gateway role will specify a port where it will accept gateway connections. If the configuration specifies other _external_ `gateways`, the gateway will create one outbound gateway connection for each gateway in its configuration. It will also gossip other gateways it knows or discovers. Fewer _external_ `gateways` mean less configuration. Yet, the ability to discover more gateways and gateway nodes depends on these servers running. This is similar to _seed server_ in cluster. It is recommended to have all _seed server_ of a cluster listed in the `gateways` section. -If the local cluster has three gateway nodes, this means there will be three outbound connections to each external gateway. +If the local cluster has three gateway nodes, this means there will be three outbound connections from the local cluster to each external gateway cluster. ![Gateway Connections](../../../.gitbook/assets/simple.svg) From 2b9d45ad7eeb067a20206e1505e9e379c23db4e4 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 11 Feb 2021 15:56:18 -0600 Subject: [PATCH 50/84] updates based on RI changes to README --- jetstream/clustering/administration.md | 92 +++++++++++++++++++++++++- 1 file changed, 90 insertions(+), 2 deletions(-) diff --git a/jetstream/clustering/administration.md b/jetstream/clustering/administration.md index 92efe7f..7c05e83 100644 --- a/jetstream/clustering/administration.md +++ b/jetstream/clustering/administration.md @@ -2,6 +2,10 @@ Once a JetStream cluster is operating interactions with the CLI and with `nats` CLI is the same as before. For these examples, lets assume we have a 5 server cluster, n1-n5 in a cluster named C1. +## Account Level + +Within an account there are operations and reports that show where users data is placed and which allow them some basic interactions with the RAFT system. + ## Creating clustered streams When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. @@ -32,7 +36,26 @@ The `current` indicates that followers are up to date and have all the messages, The replica count cannot be edited once configured. -## Forcing Stream and Consumer leader election +### Viewing Stream Placement and Stats + +Users can get overall statistics about their streams and also where these streams are placed: + +``` +$ nats stream report +Obtaining Stream stats ++----------+-----------+----------+--------+---------+------+---------+----------------------+----------+ +| Stream | Consumers | Messages | Bytes | Storage | Lost | Deleted | Cluster | Template | ++----------+-----------+----------+--------+---------+------+---------+----------------------+----------+ +| ORDERS | 4 | 0 | 0 B | File | 0 | 0 | n1-c1*, n2-c1, n3-c1 | | +| ORDERS_3 | 4 | 0 | 0 B | File | 0 | 0 | n1-c1*, n2-c1, n3-c1 | | +| ORDERS_4 | 4 | 0 | 0 B | File | 0 | 0 | n1-c1*, n2-c1, n3-c1 | | +| ORDERS_5 | 4 | 0 | 0 B | File | 0 | 0 | n1-c1, n2-c1, n3-c1* | | +| ORDERS_2 | 4 | 1,385 | 13 MiB | File | 0 | 1 | n1-c1, n2-c1, n3-c1* | | +| ORDERS_0 | 4 | 1,561 | 14 MiB | File | 0 | 0 | n1-c1, n2-c1*, n3-c1 | | ++----------+-----------+----------+--------+---------+------+---------+----------------------+----------+ +``` + +#### Forcing Stream and Consumer leader election Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process, but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time. @@ -55,6 +78,71 @@ Cluster Information: The same is true for consumers, `nats consumer cluster step-down ORDERS NEW`. +## System Level + +Systems users can view state of the Meta Group - but not individual Stream or Consumers. + +### Viewing the cluster state + +We have a high level report of cluster state: + +```nohighlight +$ nats server report jetstream --user system ++--------------------------------------------------------------------------------------------------+ +| JetStream Summary | ++--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+ +| Server | Cluster | Streams | Consumers | Messages | Bytes | Memory | File | API Req | API Err | ++--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+ +| n3-c2 | c2 | 0 | 0 | 0 | 0 B | 0 B | 0 B | 1 | 0 | +| n3-c1 | c1 | 6 | 24 | 2,946 | 27 MiB | 0 B | 27 MiB | 3 | 0 | +| n2-c2 | c2 | 0 | 0 | 0 | 0 B | 0 B | 0 B | 3 | 0 | +| n1-c2 | c2 | 0 | 0 | 0 | 0 B | 0 B | 0 B | 14 | 2 | +| n2-c1 | c1 | 6 | 24 | 2,946 | 27 MiB | 0 B | 27 MiB | 15 | 0 | +| n1-c1* | c1 | 6 | 24 | 2,946 | 27 MiB | 0 B | 27 MiB | 31 | 0 | ++--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+ +| | | 18 | 72 | 8,838 | 80 MiB | 0 B | 80 MiB | 67 | 2 | ++--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+ ++---------------------------------------------------+ +| RAFT Meta Group Information | ++-------+--------+---------+---------+--------+-----+ +| Name | Leader | Current | Offline | Active | Lag | ++-------+--------+---------+---------+--------+-----+ +| n1-c1 | yes | true | false | 0.00s | 0 | +| n1-c2 | | true | false | 0.05s | 0 | +| n2-c1 | | true | false | 0.05s | 0 | +| n2-c2 | | true | false | 0.05s | 0 | +| n3-c1 | | true | false | 0.05s | 0 | +| n3-c2 | | true | false | 0.05s | 0 | ++-------+--------+---------+---------+--------+-----+ +``` + +This is a full cluster wide report, the report can be limited to a specific account using `--account`. + +Here we see the distribution of streams, messages, api calls etc by across 2 super clusters and an overview of the RAFT meta group. + +In the Meta Group report the server `n2-c1` is not current and has not been seen for 9 seconds, it's also behind by 2 raft operations. + +This report is built using raw data that can be obtained from the monitor port on the `/jsz` url, or over nats using: + +```nohightlight +$ nats server req jetstream --help +... + --name=NAME Limit to servers matching a server name + --host=HOST Limit to servers matching a server host name + --cluster=CLUSTER Limit to servers matching a cluster name + --tags=TAGS ... Limit to servers with these configured tags + --account=ACCOUNT Show statistics scoped to a specific account + --accounts Include details about accounts + --streams Include details about Streams + --consumer Include details about Consumers + --config Include details about configuration + --leader Request a response from the Meta-group leader only + --all Include accounts, streams, consumers and configuration +$ nats server req jetstream --leader +``` + +This will produce a wealth of raw information about the current state of your cluster - here requesting it from the leader only. + #### Forcing Meta Group leader election Similar to Streams and Consumers above the Meta Group allows leader stand down. The Meta Group is cluster wide and spans all accounts, therefore to manage the meta group you have to use a `SYSTEM` user. @@ -65,7 +153,7 @@ $ nats server raft step-down --user system 17:44:24 New leader: n1-c2 ``` -## Evicting a peer +### Evicting a peer Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down. From 6bacb945b4d00aed938befdb63146085d262e2ba Mon Sep 17 00:00:00 2001 From: Ivan Kozlovic Date: Tue, 16 Feb 2021 10:29:41 -0700 Subject: [PATCH 51/84] Restructe layout, add README and config page, etc.. Resolves #205 Signed-off-by: Ivan Kozlovic --- SUMMARY.md | 2 +- nats-server/configuration/README.md | 1 + .../configuration/leafnodes/leafnode_conf.md | 2 +- nats-server/configuration/websocket/README.md | 12 ++++++++++++ .../websocket_conf.md} | 17 +++-------------- 5 files changed, 18 insertions(+), 16 deletions(-) create mode 100644 nats-server/configuration/websocket/README.md rename nats-server/configuration/{websockets.md => websocket/websocket_conf.md} (86%) diff --git a/SUMMARY.md b/SUMMARY.md index f0eb82d..eb680a1 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -98,7 +98,7 @@ * [Monitoring](nats-server/configuration/monitoring.md) * [System Events](nats-server/configuration/sys_accounts/README.md) * [System Events & Decentralized JWT Tutorial](nats-server/configuration/sys_accounts/sys_accounts.md) - * [Websockets](nats-server/configuration/websockets.md) + * [Websocket](nats-server/configuration/websocket/README.md) * [Managing A NATS Server](nats-server/nats_admin/README.md) * [Upgrading a Cluster](nats-server/nats_admin/upgrading_cluster.md) * [Slow Consumers](nats-server/nats_admin/slow_consumers.md) diff --git a/nats-server/configuration/README.md b/nats-server/configuration/README.md index bbe1b84..214582c 100644 --- a/nats-server/configuration/README.md +++ b/nats-server/configuration/README.md @@ -117,6 +117,7 @@ authorization: { | [`cluster`](clustering/cluster_config.md) | Configuration map for [cluster](clustering/). | | | [`gateway`](gateways/gateway.md#gateway-configuration-block) | Configuration map for [gateway](gateways/). | | | [`leafnode`](leafnodes/leafnode_conf.md) | Configuration map for a [leafnode](leafnodes/). | | +| [`websocket`](websocket/websocket_conf.md) | Configuration map for [websocket](websocket/). | | ### Connection Timeouts diff --git a/nats-server/configuration/leafnodes/leafnode_conf.md b/nats-server/configuration/leafnodes/leafnode_conf.md index 6f7e5aa..f7ff7e2 100644 --- a/nats-server/configuration/leafnodes/leafnode_conf.md +++ b/nats-server/configuration/leafnodes/leafnode_conf.md @@ -118,7 +118,7 @@ Therefore this would be considered an invalid configuration: Note that the decision to make a TLS connection is not based on `wss://` (as opposed to `ws://`) but instead in the presence of a TLS configuration in the `leafnodes{}` or the specific remote configuration block. -To configure Websocket in the remote server, check the [Websocket](../websockets.md) secion. +To configure Websocket in the remote server, check the [Websocket](../websocket/websocket_conf.md) secion. ### `tls` Configuration Block diff --git a/nats-server/configuration/websocket/README.md b/nats-server/configuration/websocket/README.md new file mode 100644 index 0000000..2c17afa --- /dev/null +++ b/nats-server/configuration/websocket/README.md @@ -0,0 +1,12 @@ +# Websocket + +*Supported since NATS Server version 2.2* + +Websocket support can be enabled in the server and may be used alongside the +traditional TCP socket connections. TLS, compression and +Origin Header checking are supported. + +**Important** + +- NATS Supports only Websocket data frames in Binary, not Text format (https://tools.ietf.org/html/rfc6455#section-5.6). The server will always send in Binary and your clients MUST send in Binary too. +- For writers of client libraries: a Websocket frame is not guaranteed to contain a full NATS protocol (actually will generally not). Any data from a frame must be going through a parser that can handle partial protocols. See the protocol description [here](../../../nats-protocol/nats-protocol/README.md). diff --git a/nats-server/configuration/websockets.md b/nats-server/configuration/websocket/websocket_conf.md similarity index 86% rename from nats-server/configuration/websockets.md rename to nats-server/configuration/websocket/websocket_conf.md index 7e742d2..889fc00 100644 --- a/nats-server/configuration/websockets.md +++ b/nats-server/configuration/websocket/websocket_conf.md @@ -1,17 +1,6 @@ -# Websocket Support +# Configuration -*Supported since NATS Server version 2.2* - -Websocket support can be enabled in the server and may be used alongside the -traditional TCP socket connections. TLS, compression and -Origin Header checking are supported. - -**Important** - -- NATS Supports only Websocket data frames in Binary, not Text format (https://tools.ietf.org/html/rfc6455#section-5.6). The server will always send in Binary and your clients MUST send in Binary too. -- For writers of client libraries: a Websocket frame is not guaranteed to contain a full NATS protocol (actually will generally not). Any data from a frame must be going through a parser that can handle partial protocols. See the protocol description [here](../../nats-protocol/nats-protocol/README.md). - -To enable websocket support in the server, add a `websockets` configuration +To enable websocket support in the server, add a `websocket` configuration block in the server's configuration file like the following: ``` @@ -166,4 +155,4 @@ The possible values are currently: ## Leaf nodes connections You can configure remote Leaf node connections so that they connect to the Websocket port instead of the Leaf node port. -See [Leafnode](leafnodes/leafnode_conf.md#connecting-using-websocket-protocol) section. +See [Leafnode](../leafnodes/leafnode_conf.md#connecting-using-websocket-protocol) section. From d956645089b6cd3863ec2a1546c8615db7f6b896 Mon Sep 17 00:00:00 2001 From: Ivan Kozlovic Date: Tue, 16 Feb 2021 13:11:13 -0700 Subject: [PATCH 52/84] [ADDED] MQTT documentation Resolves #204 Signed-off-by: Ivan Kozlovic --- SUMMARY.md | 1 + nats-server/configuration/README.md | 1 + nats-server/configuration/mqtt/README.md | 157 +++++++++++++++++ nats-server/configuration/mqtt/mqtt_conf.md | 179 ++++++++++++++++++++ 4 files changed, 338 insertions(+) create mode 100644 nats-server/configuration/mqtt/README.md create mode 100644 nats-server/configuration/mqtt/mqtt_conf.md diff --git a/SUMMARY.md b/SUMMARY.md index ac9fbae..2e5dc4a 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -96,6 +96,7 @@ * [Configuration](nats-server/configuration/leafnodes/leafnode_conf.md) * [Logging](nats-server/configuration/logging.md) * [Monitoring](nats-server/configuration/monitoring.md) + * [MQTT](nats-server/configuration/mqtt/README.md) * [System Events](nats-server/configuration/sys_accounts/README.md) * [System Events & Decentralized JWT Tutorial](nats-server/configuration/sys_accounts/sys_accounts.md) * [Managing A NATS Server](nats-server/nats_admin/README.md) diff --git a/nats-server/configuration/README.md b/nats-server/configuration/README.md index bbe1b84..ccb3eaf 100644 --- a/nats-server/configuration/README.md +++ b/nats-server/configuration/README.md @@ -117,6 +117,7 @@ authorization: { | [`cluster`](clustering/cluster_config.md) | Configuration map for [cluster](clustering/). | | | [`gateway`](gateways/gateway.md#gateway-configuration-block) | Configuration map for [gateway](gateways/). | | | [`leafnode`](leafnodes/leafnode_conf.md) | Configuration map for a [leafnode](leafnodes/). | | +| [`mqtt`](mqtt/mqtt_conf.md) | Configuration map for a [mqtt](mqtt/). | | ### Connection Timeouts diff --git a/nats-server/configuration/mqtt/README.md b/nats-server/configuration/mqtt/README.md new file mode 100644 index 0000000..950f7ec --- /dev/null +++ b/nats-server/configuration/mqtt/README.md @@ -0,0 +1,157 @@ +# MQTT + +*Supported since NATS Server version 2.2* + +NATS follows as close as possible the MQTT v3.1.1 [specification](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html). + +## JetStream Requirements + +In order for an MQTT client to connect to the NATS Server, the user's account must be JetStream enabled. +This is because persistence is needed for the sessions and retained messages (since even retained messages +of QoS 0 are persisted). + +## MQTT Topics and NATS Subjects + +MQTT Topics are similar to NATS Subjects, but have distinctive differences. + +MQTT topic uses "`/`" as a level separator. For instance `foo/bar` would translate to NATS subject `foo.bar`. +But in MQTT, `/foo/bar/` is a valid subject, which, if simply translated, would become `.foo.bar.`, which +is NOT a valid NATS Subject. + +NATS Server will convert an MQTT topic following those rules: + +| MQTT character | NATS character(s) | Topic (MQTT) | Subject (NATS) | +| :---: | :---: | :---: | :---: | +| `/` between two levels | `.` | `foo/bar` | `foo.bar` | +| `/` as first level | `/.` | `/foo/bar` | `/.foo.bar` | +| `/` as last level | `./` | `foo/bar/` | `foo.bar./` | +| `/` next to another | `./` | `foo//bar` | `foo./.bar` | +| `/` next to another | `/.` | `//foo/bar` | `/./.foo.bar` | +| `.` | Not Support | `foo.bar` | Not Supported | +| ` ` | Not Support | `foo bar` | Not Supported | + +As indicated above, if an MQTT topic contains the character ` ` or `.`, NATS will reject it, +causing the connection to be closed for published messages, and returning a failure code in +the SUBACK packet for a subscriptions. + +### MQTT Wildcards + +As in NATS, MQTT wildcards represent either multi or single levels. As in NATS, they are +allowed only for subscriptions, not for published messages. + +| MQTT Wildcard | NATS Wildcard | +| :---: | :---: | +| `#` | `>` | +| `+` | `*` | + +The wilcard `#` matches any number of levels within a topic, which means that a subscription +on `foo/#` would receive messages on `foo/bar`, or `foo/bar/baz`, but also on `foo`. +This is not the case in NATS where a subscription on `foo.>` can receive messages on `foo/bar` +or `foo/bar/baz`, but not on `foo`. To solve this, NATS Server will create two subscriptions, +one on `foo.>` and one on `foo`. If the MQTT subscription is simply on `#`, then a single +NATS subscription on `>` is enough. + +The wildcard `+` matches a single level, which means `foo/+` can receive message on `foo/bar` or +`foo/baz`, but not on `foo/bar/baz` nor `foo`. This is the same with NATS subscriptions using +the wildcard `*`. Therefore `foo/+` would translate to `foo.*`. + +## Communication between MQTT and NATS + +When an MQTT client creates a subscription on a topic, the NATS Server creates the similar +NATS subscription (with conversion from MQTT topic to NATS subject) so that the interest +is registered in the cluster and known to any NATS publishers. + +That is, say an MQTT client connects to server "A" and creates a subscription of `foo/bar`, +server "A" creates a subscription on `foo.bar`, which interest is propagated as any other +NATS subscription. A publisher connecting anywhere in the cluster and publishing on `foo.bar` +would cause server "A" to deliver a QoS 0 message to the MQTT subscription. + +This works the same way for MQTT publishers. When the server receives an MQTT publish +message, it is converted to the NATS subject and published, which means that any matching NATS +subscription will receive the MQTT message. + +If the MQTT subscription is QoS1 and an MQTT publisher publishes an MQTT QoS1 message on +the same or any other server in the cluster, the message will be persisted in the cluster +and routed and delivered as QoS 1 to the MQTT subscription. + +## QoS 1 Redeliveries + +When the server delivers a QoS 1 message to a QoS 1 subscription, it will keep the message +until it receives the PUBACK for the corresponding packet identifier. If it does not receive +it within the "ack_wait" interval, that message will be resent. + +## Max Ack Pending + +This is the amount of QoS 1 messages the server can send to a subscription without receiving +any PUBACK for those messages. The maximum value is 65535. + +The total of subscriptions' `max_ack_pending` on a given session cannot exceed 65535. Attempting +to create a subscription that would bring the total above the limit would result in the server +returning a failure code in the SUBACK for this subscription. + +Due to how the NATS Server handles the MQTT "`#`" wildcard, each subscription ending with "`#`" +will use 2 times the `max_ack_pending` value. + +## Sessions + +NATS Server will persist all sessions, even if they are created with the "clean session" flag, which means +that session last only for the duration of the network connection between the client and the server. + +A session is identified by a client identifier. If two connections try to use the same client identifier, +the server, per specification, will close the existing connection and accept the new one. + +If the user incorrectly starts two applications that use the same client identifier, this would result +in a very quick flapping if the MQTT client has a reconnect feature and quickly reconnects. + +To prevent this, the NATS Server will accept the new session and will delay the closing of the +old connection to reduce the flapping rate. + +Detection of the concurrent use of sessions also works in cluster mode. + +## Retained Messages + +When a server receives an MQTT publish packet with the RETAIN flag set (regardless of its QoS), it stores the application message and its QoS, so that it can be delivered to future subscribers whose subscriptions match its topic name. + +When a new subscription is established, the last retained message, if any, on each matching topic name will be sent to the subscriber. + +A PUBLISH Packet with a RETAIN flag set to 1 and a payload containing zero bytes will be processed as normal and sent to clients with a subscription matching the topic name. Additionally any existing retained message with the same topic name will be removed and any future subscribers for the topic will not receive a retained message. + +## Clustering + +NATS supports MQTT in a NATS cluster. The replication factor is automatically set based on the size +of the cluster. + +### Connections with same client ID + +If a client is connected to a server "A" in the cluster and another client connects to a server "B" and +uses the same client identifier, server "A" will close its client upon discovering the use of +an active client identifier. + +Of course, this is not as easy and immediate than if the two applications are connected to the same server. +So users should avoid as much as possible this situation. + +There may be cases where the server will reject the new connection if there is no safe way to +close the existing connection if it is in the middle of processing some MQTT packets. + +### Retained Messages + +Retained messages are stored in the cluster and available to any server in the cluster. However, +this is not immediate and if a producer connects to a server and produces a retained message +and another connection connects to another server and starts a matching subscription, it +may not receive the retained message if the server it connects to has not yet been made +aware of this retained message. + +In other words, retained messages in clustering mode is best-effort, and applications that rely on the +presence of a retained message should connect on the server that produced them. + +## Limitations + +- NATS does not support QoS 2 messages. If it receives a published message with QoS greater than 1, +it will close the connection. +- NATS messages published to MQTT subscriptions are always delivered as QoS 0 messages. +- MQTT published messages on topic names containing "` `" or "`.`" characters will cause the +connection to be closed. Presence of those characters in MQTT subscriptions will result in error +code in the SUBACK packet. +- MQTT wildcard `#` may cause the NATS Server to create two subscriptions. +- MQTT concurrent sessions may result in the new connection to be evicted instead of the existing one. +- MQTT retained messages in clustering mode is best effort. diff --git a/nats-server/configuration/mqtt/mqtt_conf.md b/nats-server/configuration/mqtt/mqtt_conf.md new file mode 100644 index 0000000..fe7dcdd --- /dev/null +++ b/nats-server/configuration/mqtt/mqtt_conf.md @@ -0,0 +1,179 @@ +# Configuration + +To enable MQTT support in the server, add a `mqtt` configuration +block in the server's configuration file like the following: + +``` +mqtt { + # Specify a host and port to listen for websocket connections + # + # listen: "host:port" + # It can also be configured with individual parameters, + # namely host and port. + # + # host: "hostname" + port: 1883 + + # TLS configuration. + # + tls { + cert_file: "/path/to/cert.pem" + key_file: "/path/to/key.pem" + + # Root CA file + # + # ca_file: "/path/to/ca.pem" + + # If true, require and verify client certificates. + # + # verify: true + + # TLS handshake timeout in fractional seconds. + # + # timeout: 2.0 + + # If true, require and verify client certificates and map certificate + # values for authentication purposes. + # + # verify_and_map: true + } + + # If no user name is provided when an MQTT client connects, will default + # this user name in the authentication phase. If specified, this will + # override, for MQTT clients, any `no_auth_user` value defined in the + # main configuration file. + # Note that this is not compatible with running the server in operator mode. + # + # no_auth_user: "my_username_for_apps_not_providing_credentials" + + # See below to know what is the normal way of limiting MQTT clients + # to specific users. + # If there are no users specified in the configuration, this simple authorization + # block allows you to override the values that would be configured in the + # equivalent block in the main section. + # + # authorization { + # # If this is specified, the client has to provide the same username + # # and password to be able to connect. + # # username: "my_user_name" + # # password: "my_password" + # + # # If this is specified, the password field in the CONNECT packet has to + # # match this token. + # # token: "my_token" + # + # # This overrides the main's authorization timeout. For consistency + # # with the main's authorization configuration block, this is expressed + # # as a number of seconds. + # # timeout: 2.0 + #} + + # This is the amount of time after which a QoS 1 message sent to + # a client is redelivered as a DUPLICATE if the server has not + # received the PUBACK packet on the original Packet Identifier. + # The value has to be positive. + # Zero will cause the server to use the default value (30 seconds). + # Note that changes to this option is applied only to new MQTT subscriptions. + # + # Expressed as a time duration, with "s", "m", "h" indicating seconds, + # minutes and hours respectively. For instance "10s" for 10 seconds, + # "1m" for 1 minute, etc... + # + # ack_wait: "1m" + + # This is the amount of QoS 1 messages the server can send to + # a subscription without receiving any PUBACK for those messages. + # The valid range is [0..65535]. + # + # The total of subscriptions' max_ack_pending on a given session cannot + # exceed 65535. Attempting to create a subscription that would bring + # the total above the limit would result in the server returning 0x80 + # in the SUBACK for this subscription. + # Due to how the NATS Server handles the MQTT "#" wildcard, each + # subscription ending with "#" will use 2 times the max_ack_pending value. + # Note that changes to this option is applied only to new subscriptions. + # + # max_ack_pending: 100 +} +``` + +## Authorization of MQTT Users + +A new field when configuring users allows you to restrict which type of connections are allowed for a specific user. + +Consider this configuration: + +``` +authorization { + users [ + {user: foo password: foopwd, permission: {...}} + {user: bar password: barpwd, permission: {...}} + ] +} +``` + +If an MQTT client were to connect and use the username `foo` and password `foopwd`, it would be accepted. +Now suppose that you would want an MQTT client to only be accepted if it connected using the username `bar` +and password `barpwd`, then you would use the option `allowed_connection_types` to restrict which type +of connections can bind to this user. + +``` +authorization { + users [ + {user: foo password: foopwd, permission: {...}} + {user: bar password: barpwd, permission: {...}, allowed_connection_types: ["MQTT"]} + ] +} +``` + +The option `allowed_connection_types` (also can be named `connection_types` or `clients`) as you can see +is a list, and you can allow several type of clients. Suppose you want the user `bar` to accept both +standard NATS clients and MQTT clients, you would configure the user like this: + +``` +authorization { + users [ + {user: foo password: foopwd, permission: {...}} + {user: bar password: barpwd, permission: {...}, allowed_connection_types: ["STANDARD", "MQTT"]} + ] +} +``` + +The absence of `allowed_connection_types` means that all type of connections are allowed (the default behavior). + +The possible values are currently: +* `STANDARD` +* `WEBSOCKET` +* `LEAFNODE` +* `MQTT` + +### Special permissions + +When an MQTT client creates a QoS 1 subscription, this translates to the creation +of a JetStream durable subscription. To receive messages for this durable, the NATS Server +creates a subscription with a subject such as `$MQTT.sub.` and sets it as the +JetStream durable's delivery subject. + +Therefore, if you have set some permissions for the MQTT user, you need to allow +subscribe permissions on `$MQTT.sub.>`. + +Here is an example of a basic configuration that sets some permissions to a user named "mqtt". +As you can see, the subscribe permission `$MQTT.sub.>` is added to allow this client to +create QoS 1 subscriptions. + +``` + listen: 127.0.0.1:4222 + jetstream: enabled + authorization { + mqtt_perms = { + publish = ["baz"] + subscribe = ["foo", "bar", "$MQTT.sub.>"] + } + users = [ + {user: mqtt, password: pass, permissions: $mqtt_perms, allowed_connection_types: ["MQTT"]} + ] + } + mqtt { + listen: 127.0.0.1:1883 + } +``` From 3f01c728c179f39f2104d2d485a9ef440980c4c2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jaime=20Pi=C3=B1a?= Date: Wed, 17 Feb 2021 13:59:12 -0800 Subject: [PATCH 53/84] Replace mkpasswd with natscli (#223) With mkpasswd now gone from nats-server, this change replaces instances of mkpasswd with the new official natscli tool. --- SUMMARY.md | 2 +- developing-with-nats/security/userpass.md | 10 ++-- docs/nats_tools/README.md | 2 +- .../securing_nats/auth_intro/tokens.md | 10 ++-- .../auth_intro/username_password.md | 12 ++-- nats-tools/mkpasswd.md | 52 ---------------- nats-tools/nats-tools.md | 2 +- nats-tools/natscli.md | 59 +++++++++++++++++++ 8 files changed, 81 insertions(+), 68 deletions(-) delete mode 100644 nats-tools/mkpasswd.md create mode 100644 nats-tools/natscli.md diff --git a/SUMMARY.md b/SUMMARY.md index ac9fbae..09da07f 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -111,7 +111,7 @@ ## NATS Tools * [Introduction](nats-tools/nats-tools.md) -* [mkpasswd](nats-tools/mkpasswd.md) +* [nats](nats-tools/natscli.md) * [nk](nats-tools/nk.md) * [nsc](nats-tools/nsc/README.md) * [Basics](nats-tools/nsc/nsc.md) diff --git a/developing-with-nats/security/userpass.md b/developing-with-nats/security/userpass.md index a36789b..8609dff 100644 --- a/developing-with-nats/security/userpass.md +++ b/developing-with-nats/security/userpass.md @@ -6,12 +6,14 @@ For this example, start the server using: > nats-server --user myname --pass password ``` -You can encrypt passwords to pass to `nats-server` using a simple [tool](../../nats-tools/mkpasswd.md) provided by the server: +You can encrypt passwords to pass to `nats-server` using a simple [tool](../../nats-tools/natscli.md): ```bash -> go run mkpasswd.go -p -> password: password -> bcrypt hash: $2a$11$1oJy/wZYNTxr9jNwMNwS3eUGhBpHT3On8CL9o7ey89mpgo88VG6ba +> nats server passwd +? Enter password [? for help] ********************** +? Reenter password [? for help] ********************** + +$2a$11$qbtrnb0mSG2eV55xoyPqHOZx/lLBlryHRhU3LK2oOPFRwGF/5rtGK ``` and use the hashed password in the server config. The client still uses the plain text version. diff --git a/docs/nats_tools/README.md b/docs/nats_tools/README.md index 4de99cb..2da3bb1 100644 --- a/docs/nats_tools/README.md +++ b/docs/nats_tools/README.md @@ -2,7 +2,7 @@ The NATS Ecosystem has many tools to support server configuration, enhance monitoring or tune performance: -- [mkpasswd](nats_tools/mkpasswd.md) - Generates or bcrypts passwords +- [nats](nats_tools/natscli.md) - Interact with and manage NATS - [nk](nats_tools/nk.md) - Generate NKeys - [nsc](nats_tools/nsc/README.md) - Configure Operators, Accounts and Users - [nats account server](nats_tools/nas/README.md) - Serve Account JWTs diff --git a/nats-server/configuration/securing_nats/auth_intro/tokens.md b/nats-server/configuration/securing_nats/auth_intro/tokens.md index 2500735..f786590 100644 --- a/nats-server/configuration/securing_nats/auth_intro/tokens.md +++ b/nats-server/configuration/securing_nats/auth_intro/tokens.md @@ -29,12 +29,14 @@ Listening on [>] Tokens can be bcrypted enabling an additional layer of security, as the clear-text version of the token would not be persisted on the server configuration file. -You can generate bcrypted tokens and passwords using the [`mkpasswd`](../../../../nats-tools/mkpasswd.md) tool: +You can generate bcrypted tokens and passwords using the [`nats`](../../../../nats-tools/natscli.md) tool: ```text -> mkpasswd -pass: dag0HTXl4RGg7dXdaJwbC8 -bcrypt hash: $2a$11$PWIFAL8RsWyGI3jVZtO9Nu8.6jOxzxfZo7c/W0eLk017hjgUKWrhy +> nats server passwd +? Enter password [? for help] ********************** +? Reenter password [? for help] ********************** + +$2a$11$PWIFAL8RsWyGI3jVZtO9Nu8.6jOxzxfZo7c/W0eLk017hjgUKWrhy ``` Here's a simple configuration file: diff --git a/nats-server/configuration/securing_nats/auth_intro/username_password.md b/nats-server/configuration/securing_nats/auth_intro/username_password.md index f1b3d13..ee52e4e 100644 --- a/nats-server/configuration/securing_nats/auth_intro/username_password.md +++ b/nats-server/configuration/securing_nats/auth_intro/username_password.md @@ -30,12 +30,14 @@ authorization: { ## Bcrypted Passwords -Username/password also supports bcrypted passwords using the [`mkpasswd`](../../../../nats-tools/mkpasswd.md) tool. Simply replace the clear text password with the bcrypted entries: +Username/password also supports bcrypted passwords using the [`nats`](../../../../nats-tools/natscli.md) tool. Simply replace the clear text password with the bcrypted entries: ```text -> mkpasswd -pass: (Uffs#rG42PAu#Oxi^BNng -bcrypt hash: $2a$11$V1qrpBt8/SLfEBr4NJq4T.2mg8chx8.MTblUiTBOLV3MKDeAy.f7u +> nats server passwd +? Enter password [? for help] ********************** +? Reenter password [? for help] ********************** + +$2a$11$V1qrpBt8/SLfEBr4NJq4T.2mg8chx8.MTblUiTBOLV3MKDeAy.f7u ``` And on the configuration file: @@ -44,7 +46,7 @@ And on the configuration file: authorization: { users: [ {user: a, password: "$2a$11$V1qrpBt8/SLfEBr4NJq4T.2mg8chx8.MTblUiTBOLV3MKDeAy.f7u"}, - ... + ... ] } ``` diff --git a/nats-tools/mkpasswd.md b/nats-tools/mkpasswd.md deleted file mode 100644 index 0da545f..0000000 --- a/nats-tools/mkpasswd.md +++ /dev/null @@ -1,52 +0,0 @@ -# mkpasswd - -The server supports hashing of passwords and authentication tokens using `bcrypt`. To take advantage of this, simply replace the plaintext password in the configuration with its `bcrypt` hash, and the server will automatically utilize `bcrypt` as needed. - -A utility for creating `bcrypt` hashes is included with the nats-server distribution \(`util/mkpasswd.go`\). Running it with no arguments will generate a new secure password along with the associated hash. This can be used for a password or a token in the configuration. - -## Installing `mkpasswd` - -If you have [go installed](https://golang.org/doc/install), you can easily install the `mkpasswd` tool by doing: - -```text -go get github.com/nats-io/nats-server/util/mkpasswd -``` - -Alternatively, you can: - -```text -git clone git@github.com:nats-io/nats-server -cd nats-server/util/mkpasswd -go install mkpasswd.go -``` - -## Generating bcrypted passwords - -With `mkpasswd` installed: - -```text -> mkpasswd -pass: #IclkRPHUpsTmACWzmIGXr -bcrypt hash: $2a$11$3kIDaCxw.Glsl1.u5nKa6eUnNDLV5HV9tIuUp7EHhMt6Nm9myW1aS -``` - -If you already have a password selected, you can supply the `-p` flag on the command line, enter your desired password, and a `bcrypt` hash will be generated for it: - -```text -> mkpasswd -p -Enter Password: ******* -Reenter Password: ****** -bcrypt hash: $2a$11$3kIDaCxw.Glsl1.u5nKa6eUnNDLV5HV9tIuUp7EHhMt6Nm9myW1aS -``` - -To use the password on the server, add the hash into the server configuration file's authorization section. - -```text - authorization { - user: derek - password: $2a$11$3kIDaCxw.Glsl1.u5nKa6eUnNDLV5HV9tIuUp7EHhMt6Nm9myW1aS - } -``` - -Note the client will still have to provide the plain text version of the password, the server however will only store the hash to verify that the password is correct when supplied. - diff --git a/nats-tools/nats-tools.md b/nats-tools/nats-tools.md index 6468420..6b61f00 100644 --- a/nats-tools/nats-tools.md +++ b/nats-tools/nats-tools.md @@ -2,7 +2,7 @@ The NATS Ecosystem has many tools to support server configuration, enhance monitoring or tune performance: -* [mkpasswd](mkpasswd.md) - Generates or bcrypts passwords +* [nats](natscli.md) - Interact with and manage NATS * [nk](nk.md) - Generate NKeys * [nsc](nsc/) - Configure Operators, Accounts and Users * [nats account server](nas/) - Serve Account JWTs diff --git a/nats-tools/natscli.md b/nats-tools/natscli.md new file mode 100644 index 0000000..ed4b303 --- /dev/null +++ b/nats-tools/natscli.md @@ -0,0 +1,59 @@ +# natscli + +A command line utility to interact with and manage NATS. + +This utility replaces various past tools that were named in the form `nats-sub` and `nats-pub`, adds several new capabilities and support full JetStream management. + +Check out the repo for more details: [github.com/nats-io/natscli](https://github.com/nats-io/natscli). + +## Installing `nats` + +For macOS: + +``` +> brew tap nats-io/nats-tools +> brew install nats-io/nats-tools/nats +``` + +For Arch Linux: + +``` +> yay natscli +``` + +For Docker: + +``` +docker pull synadia/nats-box:latest + +docker run -ti synadia/nats-box +``` + +Binaries are also available as [GitHub Releases](https://github.com/nats-io/natscli/releases). + +## Generating bcrypted passwords + +The server supports hashing of passwords and authentication tokens using `bcrypt`. To take advantage of this, simply replace the plaintext password in the configuration with its `bcrypt` hash, and the server will automatically utilize `bcrypt` as needed. + +The `nats` utility has a command for creating `bcrypt` hashes. This can be used for a password or a token in the configuration. + +With `nats` installed: + +```plain +> nats server passwd +? Enter password [? for help] ********************** +? Reenter password [? for help] ********************** + +$2a$11$3kIDaCxw.Glsl1.u5nKa6eUnNDLV5HV9tIuUp7EHhMt6Nm9myW1aS +``` + +To use the password on the server, add the hash into the server configuration file's authorization section. + +``` + authorization { + user: derek + password: $2a$11$3kIDaCxw.Glsl1.u5nKa6eUnNDLV5HV9tIuUp7EHhMt6Nm9myW1aS + } +``` + +Note the client will still have to provide the plain text version of the password, the server however will only store the hash to verify that the password is correct when supplied. From 2dc6e3d5d028853ffc8aab647e7f6e25e300533c Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Wed, 17 Feb 2021 15:37:03 -0700 Subject: [PATCH 54/84] add when to use mqtt Signed-off-by: Colin Sullivan --- nats-server/configuration/mqtt/README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/nats-server/configuration/mqtt/README.md b/nats-server/configuration/mqtt/README.md index 950f7ec..2f5f928 100644 --- a/nats-server/configuration/mqtt/README.md +++ b/nats-server/configuration/mqtt/README.md @@ -4,6 +4,23 @@ NATS follows as close as possible the MQTT v3.1.1 [specification](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html). +## When to use MQTT + +MQTT support in NATS is intended to be an enabling technology allowing users to leverage existing +investments in their IoT deployments. Updating software on the edge or endpoints can be onerous +and risky, especially when embedded applications are involved. + +In greenfield IoT deployments, when possible, we prefer NATS extended out to endpoints and devices +for a few reasons. There are significant advantages with security and observability when using a +single technology end to end. Compared to MQTT, NATS is nearly as lightweight in terms of protocol +bandwidth and maintainer supported clients efficiently utilize resources so we consider NATS to be a +good choice to use end to end, including use on resource constrained devices. + +In existing MQTT deployments or in situations when endpoints can only support MQTT, using a NATS server +as a drop-in MQTT server replacement to securely connect to a remote NATS cluster or supercluster is +compelling. You can keep your existing IoT investment and use NATS for secure, resilient, and +scalable access to your streams and services. + ## JetStream Requirements In order for an MQTT client to connect to the NATS Server, the user's account must be JetStream enabled. From 770b37eab9f94d168631974fb51378c59d5d6f0e Mon Sep 17 00:00:00 2001 From: Matthias Hanel Date: Tue, 23 Feb 2021 17:03:18 -0500 Subject: [PATCH 55/84] [added] explanation of authentication timeout while using no_auth_user Signed-off-by: Matthias Hanel --- nats-server/configuration/securing_nats/accounts.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/nats-server/configuration/securing_nats/accounts.md b/nats-server/configuration/securing_nats/accounts.md index 5b2576d..c2c8299 100644 --- a/nats-server/configuration/securing_nats/accounts.md +++ b/nats-server/configuration/securing_nats/accounts.md @@ -190,3 +190,6 @@ The above example shows how clients without authentication can be associated wit > Please note that the `no_auth_user` will not work with nkeys. The user referenced can also be part of the [authorization](authorization.md) block. +> Despite `no_auth_user` being set, clients still need to communicate that they will not be using credentials. +> The [authentication timeout](auth_intro/auth_timeout.md) applies to this process as well. +> When your connection is slow, you may run into this timeout and the resulting `Authentication Timeout` error, despite not providing credentials. From 57fe4240bf16b670932a2e2981459bf61b2346e8 Mon Sep 17 00:00:00 2001 From: Andy Garfield Date: Wed, 24 Feb 2021 08:04:05 -0500 Subject: [PATCH 56/84] Fixed typo --- whats_new_20.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/whats_new_20.md b/whats_new_20.md index abd0e9d..4b2d95d 100644 --- a/whats_new_20.md +++ b/whats_new_20.md @@ -54,12 +54,10 @@ Services and streams are mechanisms to share messages between accounts. Think of a service as an RPC endpoint into an account. Behind that account there might be many microservices working in concert to handle requests, but from outside the account there is simply one subject exposed. -**Services** definition to share an endpoint: +**Service** definitions share an endpoint: * Export a service to allow other accounts to import -* Import a service to allow requests to be sent securely and seamlessly to - - another account +* Import a service to allow requests to be sent securely and seamlessly to another account Use cases include most applications - anything that accepts a request and returns a response. From 132bf6b19fc36d62c250e8300d1766754955fe65 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Wed, 24 Feb 2021 11:19:50 -0600 Subject: [PATCH 57/84] fixing JetStream capitalization --- nats-server/configuration/monitoring.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/nats-server/configuration/monitoring.md b/nats-server/configuration/monitoring.md index 4ab0854..dfe69a0 100644 --- a/nats-server/configuration/monitoring.md +++ b/nats-server/configuration/monitoring.md @@ -10,7 +10,7 @@ To monitor the NATS messaging system, `nats-server` provides a lightweight HTTP * [Gateways](monitoring.md#gateway-information) * [Leaf Nodes](monitoring.md#leaf-nodes-information) * [Subscription Routing](monitoring.md#subscription-routing-information) -* [Jetstream Information](monitoring.md#jetstream-information) +* [JetStream Information](monitoring.md#jetstream-information) All endpoints return a JSON object. @@ -57,7 +57,7 @@ http: localhost:8222 For example, to monitor this server locally, the endpoint would be [http://localhost:8222/varz](http://localhost:8222/varz). It reports various general statistics. -## Monitoring endpoints +## Monitoring Endpoints The following sections describe each supported monitoring endpoint: `varz`, `connz`, `routez`, `subsz`, `gatewayz`, and `leafz`. There are not any required arguments, however use of arguments can let you tailor monitoring to your environment and tooling. @@ -433,7 +433,7 @@ The `/gatewayz` endpoint reports information about gateways used to create a NAT } ``` -### Leaf Nodes Information +### Leaf Node Information The `/leafz` endpoint reports detailed information about the leaf node connections. @@ -521,9 +521,9 @@ The `/subsz` endpoint reports detailed information about the current subscriptio } ``` -## Jetstream Information +## JetStream Information -The `/jsz` endpoint reports more detailed information on jetstream. For accounts it uses a paging mechanism which defaults to 1024 connections. +The `/jsz` endpoint reports more detailed information on JetStream. For accounts it uses a paging mechanism which defaults to 1024 connections. **Endpoint:** `http://server:port/connz` @@ -537,7 +537,7 @@ The `/jsz` endpoint reports more detailed information on jetstream. For accounts | Argument | Values | Description | | :--- | :--- | :--- | | acc | account name | Include metrics for the specified account. Default is unset. | -| accounts | true, 1, false, 0 | Include account specific jetstream information. Default is false. | +| accounts | true, 1, false, 0 | Include account specific JetStream information. Default is false. | | streams | true, 1, false, 0 | Include streams. When set, implies `accounts=true`. Default is false. | | consumers | true, 1, false, 0 | Include consumer. When set, implies `streams=true`. Default is false. | | config | true, 1, false, 0 | When stream or consumer are requested, include their respective configuration. Default is false. | @@ -547,7 +547,7 @@ The `/jsz` endpoint reports more detailed information on jetstream. For accounts #### Examples -Get basic jetstream information: [http://demo.nats.io:8222/jsz](http://demo.nats.io:8222/jsz) +Get basic JetStream information: [http://demo.nats.io:8222/jsz](http://demo.nats.io:8222/jsz) Request accounts and control limit and offset: [http://demo.nats.io:8222/jsz?accounts=true&limit=16&offset=128](http://demo.nats.io:8222/jsz?accounts=true&limit=16&offset=128). From 5c1827430ae522015e22bf5d8d75715009a04d14 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Wed, 24 Feb 2021 11:50:34 -0600 Subject: [PATCH 58/84] Syntax/grammar review updates --- .../securing_nats/jwt/resolver.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/nats-server/configuration/securing_nats/jwt/resolver.md b/nats-server/configuration/securing_nats/jwt/resolver.md index fe9bee8..3a52ef9 100644 --- a/nats-server/configuration/securing_nats/jwt/resolver.md +++ b/nats-server/configuration/securing_nats/jwt/resolver.md @@ -37,13 +37,13 @@ For more information on how to configure a memory resolver, see [this tutorial]( ## NATS Based Resolver -NATS based resolver embed the functionality of the [account server](https://github.com/nats-io/nats-account-server) inside the nats-server. -To not have to store all account jwt on every server, this resolver has two sub types `full` and `cache`. -Their commonalities are that they exchange/lookup account jwt via NATS and the system account and store them in a local (not shared) directory. +The NATS based resolver embeds the functionality of the [account server](https://github.com/nats-io/nats-account-server) inside the nats-server. +In order to avoid having to store all account JWT on every server, this resolver has two sub types `full` and `cache`. +Their commonalities are that they exchange/lookup account JWT via NATS and the system account, and store them in a local (not shared) directory. ### Full -This resolver stores all jwt and exchanges them in an eventually consistent way with other resolver of the same type. +The Full resolver stores all JWTs and exchanges them in an eventually consistent way with other resolvers of the same type. [`nsc`](../../../../nats-tools/nsc/README.md) supports push/pull/purge with this resolver type. [JWTs](../../nats-server/configuration/securing_nats/jwt/), uploaded this way, are stored in a directory the server has exclusive access to. @@ -67,17 +67,17 @@ resolver: { } ``` -This resolver type also supports `resolver_preload`. When present, JWTs are listed are stored in the resolver. +This resolver type also supports `resolver_preload`. When present, JWTs are listed and stored in the resolver. There, they may be subject to updates. Restarts of the `nats-server` will hold on to these more recent versions. Not every server in a cluster needs to be set to `full`. -You need enough to still serve your workload adequately, while some server are offline. +You need enough to still serve your workload adequately, while some servers are offline. ### Cache -This resolver only stores a subset of [JWT](../../nats-server/configuration/securing_nats/jwt/) and evicts others based on an LRU scheme. -Missing jwt are downloaded from `full` nats based resolver. -This resolver is essentially the URL Resolver in nats. +The Cache resolver only stores a subset of [JWT](../../nats-server/configuration/securing_nats/jwt/) and evicts others based on an LRU scheme. +Missing JWTs are downloaded from `full` nats based resolver. +This resolver is essentially the URL Resolver in NATS. ```yaml resolver: { @@ -96,7 +96,7 @@ resolver: { The NATS based resolver utilizes the system account for lookup and upload of account [JWTs](../../nats-server/configuration/securing_nats/jwt/) . If your application requires tighter integration you can make use of these subjects for tighter integration. -To upload or update any generated account jwt without [`nsc`](../../../../nats-tools/nsc/README.md), send it as request to `$SYS.REQ.CLAIMS.UPDATE`. -Each participating `full` nats based account resolver will respond with a message detailing success or failure. +To upload or update any generated account JWT without [`nsc`](../../../../nats-tools/nsc/README.md), send it as a request to `$SYS.REQ.CLAIMS.UPDATE`. +Each participating `full` NATS based account resolver will respond with a message detailing success or failure. -To serve a requested account [JWT](../../nats-server/configuration/securing_nats/jwt/) yourself and essentially implement an account server, subscribe to `$SYS.REQ.ACCOUNT.*.CLAIMS.LOOKUP` and respond with the account jwt corresponding to the requested account id (wildcard). +To serve a requested account [JWT](../../nats-server/configuration/securing_nats/jwt/) yourself and essentially implement an account server, subscribe to `$SYS.REQ.ACCOUNT.*.CLAIMS.LOOKUP` and respond with the account JWT corresponding to the requested account id (wildcard). From 14b3eac59ec0a9c86c1b461769a8be1204981154 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Wed, 24 Feb 2021 12:04:12 -0600 Subject: [PATCH 59/84] small grammar change --- nats-tools/nas/nas_conf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-tools/nas/nas_conf.md b/nats-tools/nas/nas_conf.md index 8a594b6..bf46b05 100644 --- a/nats-tools/nas/nas_conf.md +++ b/nats-tools/nas/nas_conf.md @@ -7,7 +7,7 @@ Basic configuration revolves around 4 settings: * NATS \(for cases where updates are enabled\) * Logging -For complete information on please refer to the project's [Github](https://github.com/nats-io/nats-account-server). +For complete information, please refer to the project's [Github](https://github.com/nats-io/nats-account-server). ## Directory Configuration From 442badf74fadbd982930325a57cecdb9d0def864 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 25 Feb 2021 10:12:48 -0600 Subject: [PATCH 60/84] review of syntax and grammar --- nats-server/configuration/mqtt/README.md | 35 ++++++++++++------------ 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/nats-server/configuration/mqtt/README.md b/nats-server/configuration/mqtt/README.md index 2f5f928..501cb72 100644 --- a/nats-server/configuration/mqtt/README.md +++ b/nats-server/configuration/mqtt/README.md @@ -2,9 +2,9 @@ *Supported since NATS Server version 2.2* -NATS follows as close as possible the MQTT v3.1.1 [specification](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html). +NATS follows as closely as possible to the MQTT v3.1.1 [specification](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html). -## When to use MQTT +## When to Use MQTT MQTT support in NATS is intended to be an enabling technology allowing users to leverage existing investments in their IoT deployments. Updating software on the edge or endpoints can be onerous @@ -18,14 +18,14 @@ good choice to use end to end, including use on resource constrained devices. In existing MQTT deployments or in situations when endpoints can only support MQTT, using a NATS server as a drop-in MQTT server replacement to securely connect to a remote NATS cluster or supercluster is -compelling. You can keep your existing IoT investment and use NATS for secure, resilient, and +compelling. You can keep your existing IoT investment and use NATS for secure, resilient, and scalable access to your streams and services. ## JetStream Requirements -In order for an MQTT client to connect to the NATS Server, the user's account must be JetStream enabled. -This is because persistence is needed for the sessions and retained messages (since even retained messages -of QoS 0 are persisted). +For an MQTT client to connect to the NATS server, the user's account must be JetStream enabled. +This is because persistence is needed for the sessions and retained messages since even retained messages +of QoS 0 are persisted. ## MQTT Topics and NATS Subjects @@ -72,9 +72,9 @@ The wildcard `+` matches a single level, which means `foo/+` can receive message `foo/baz`, but not on `foo/bar/baz` nor `foo`. This is the same with NATS subscriptions using the wildcard `*`. Therefore `foo/+` would translate to `foo.*`. -## Communication between MQTT and NATS +## Communication Between MQTT and NATS -When an MQTT client creates a subscription on a topic, the NATS Server creates the similar +When an MQTT client creates a subscription on a topic, the NATS server creates the similar NATS subscription (with conversion from MQTT topic to NATS subject) so that the interest is registered in the cluster and known to any NATS publishers. @@ -106,13 +106,13 @@ The total of subscriptions' `max_ack_pending` on a given session cannot exceed 6 to create a subscription that would bring the total above the limit would result in the server returning a failure code in the SUBACK for this subscription. -Due to how the NATS Server handles the MQTT "`#`" wildcard, each subscription ending with "`#`" +Due to how the NATS server handles the MQTT "`#`" wildcard, each subscription ending with "`#`" will use 2 times the `max_ack_pending` value. ## Sessions -NATS Server will persist all sessions, even if they are created with the "clean session" flag, which means -that session last only for the duration of the network connection between the client and the server. +NATS Server will persist all sessions, even if they are created with the "clean session" flag, meaning +that sessions only last for the duration of the network connection between the client and the server. A session is identified by a client identifier. If two connections try to use the same client identifier, the server, per specification, will close the existing connection and accept the new one. @@ -120,7 +120,7 @@ the server, per specification, will close the existing connection and accept the If the user incorrectly starts two applications that use the same client identifier, this would result in a very quick flapping if the MQTT client has a reconnect feature and quickly reconnects. -To prevent this, the NATS Server will accept the new session and will delay the closing of the +To prevent this, the NATS server will accept the new session and will delay the closing of the old connection to reduce the flapping rate. Detection of the concurrent use of sessions also works in cluster mode. @@ -138,17 +138,16 @@ A PUBLISH Packet with a RETAIN flag set to 1 and a payload containing zero bytes NATS supports MQTT in a NATS cluster. The replication factor is automatically set based on the size of the cluster. -### Connections with same client ID +### Connections with Same Client ID If a client is connected to a server "A" in the cluster and another client connects to a server "B" and -uses the same client identifier, server "A" will close its client upon discovering the use of +uses the same client identifier, server "A" will close its client connection upon discovering the use of an active client identifier. -Of course, this is not as easy and immediate than if the two applications are connected to the same server. -So users should avoid as much as possible this situation. +Users should avoid this situation as this is not as easy and immediate as if the two applications are connected to the same server. There may be cases where the server will reject the new connection if there is no safe way to -close the existing connection if it is in the middle of processing some MQTT packets. +close the existing connection, such as when it is in the middle of processing some MQTT packets. ### Retained Messages @@ -169,6 +168,6 @@ it will close the connection. - MQTT published messages on topic names containing "` `" or "`.`" characters will cause the connection to be closed. Presence of those characters in MQTT subscriptions will result in error code in the SUBACK packet. -- MQTT wildcard `#` may cause the NATS Server to create two subscriptions. +- MQTT wildcard `#` may cause the NATS server to create two subscriptions. - MQTT concurrent sessions may result in the new connection to be evicted instead of the existing one. - MQTT retained messages in clustering mode is best effort. From 4dc2d7f0b7d61192f53fee3173429e374144059f Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Sat, 27 Feb 2021 17:30:51 -0700 Subject: [PATCH 61/84] update docs Signed-off-by: Colin Sullivan --- jetstream/administration/administration.md | 2 +- jetstream/administration/consumers.md | 2 +- jetstream/administration/streams.md | 68 +++++++++++++++------- jetstream/clustering/administration.md | 4 +- jetstream/getting_started/using_docker.md | 23 ++------ jetstream/getting_started/using_source.md | 4 +- 6 files changed, 58 insertions(+), 45 deletions(-) diff --git a/jetstream/administration/administration.md b/jetstream/administration/administration.md index 1abfecb..4585bbe 100644 --- a/jetstream/administration/administration.md +++ b/jetstream/administration/administration.md @@ -1,6 +1,6 @@ ## Administration and Usage from the CLI -Once the server is running it's time to use the management tool. This can be downloaded from the [GitHub Release Page](https://github.com/nats-io/natscli/releases/) or you can use the `synadia/jsm:latest` docker image. On OS X homebrew can be used to install the latest version: +Once the server is running it's time to use the management tool. This can be downloaded from the [GitHub Release Page](https://github.com/nats-io/natscli/releases/) or you can use the `natsio/nats-box:latest` docker image. On OS X homebrew can be used to install the latest version: ```nohighlight $ brew tap nats-io/nats-tools diff --git a/jetstream/administration/consumers.md b/jetstream/administration/consumers.md index 3506676..9526c4d 100644 --- a/jetstream/administration/consumers.md +++ b/jetstream/administration/consumers.md @@ -156,7 +156,7 @@ More details about the `State` section will be shown later when discussing the a #### Consuming Pull-Based Consumers -Pull-based Consumers require you to specifically ask for messages and ack them, typically you would do this with the client library `Request()` feature, but the `jsm` utility has a helper: +Pull-based Consumers require you to specifically ask for messages and ack them, typically you would do this with the client library `Request()` feature, but the `nats` utility has a helper: First we ensure we have a message: diff --git a/jetstream/administration/streams.md b/jetstream/administration/streams.md index 72e8689..1fcaf4c 100644 --- a/jetstream/administration/streams.md +++ b/jetstream/administration/streams.md @@ -42,7 +42,7 @@ Statistics: You can get prompted interactively for missing information as above, or do it all on one command. Pressing `?` in the CLI will help you map prompts to CLI options: ``` -$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old +nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --dupe-window="0s" --replicas 1 ``` Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: @@ -68,26 +68,29 @@ Information about the configuration of the Stream can be seen, and if you did no ```nohighlight $ nats str info ORDERS -Information for Stream ORDERS +Information for Stream ORDERS created 2021-02-27T16:49:36-07:00 Configuration: Subjects: ORDERS.* - No Acknowledgements: false + Acknowledgements: true Retention: File - Limits Replicas: 1 - Maximum Messages: -1 - Maximum Bytes: -1 - Maximum Age: 8760h0m0s - Maximum Consumers: -1 + Discard Policy: Old + Duplicate Window: 2m0s + Maximum Messages: unlimited + Maximum Bytes: unlimited + Maximum Age: 1y0d0h0m0s + Maximum Message Size: unlimited + Maximum Consumers: unlimited -Statistics: +State: - Messages: 0 - Bytes: 0 B - FirstSeq: 0 - LastSeq: 0 - Active Consumers: 0 + Messages: 0 + Bytes: 0 B + FirstSeq: 0 + LastSeq: 0 + Active Consumers: 0 ``` Most commands that show data as above support `-j` to show the results as JSON: @@ -105,14 +108,20 @@ $ nats str info ORDERS -j "max_msgs": -1, "max_bytes": -1, "max_age": 31536000000000000, + "max_msg_size": -1, "storage": "file", - "num_replicas": 1 + "discard": "old", + "num_replicas": 1, + "duplicate_window": 120000000000 }, - "stats": { + "created": "2021-02-27T23:49:36.700424Z", + "state": { "messages": 0, "bytes": 0, "first_seq": 0, + "first_ts": "0001-01-01T00:00:00Z", "last_seq": 0, + "last_ts": "0001-01-01T00:00:00Z", "consumer_count": 0 } } @@ -128,14 +137,29 @@ A stream can be copied into another, which also allows the configuration of the $ nats str cp ORDERS ARCHIVE --subjects "ORDERS_ARCVHIVE.*" --max-age 2y Stream ORDERS was created -Information for Stream ARCHIVE +Information for Stream ORDERS created 2021-02-27T16:52:46-07:00 Configuration: - Subjects: ORDERS_ARCVHIVE.* -... - Maximum Age: 17520h0m0s -... + Subjects: ORDERS_ARCHIVE.* + Acknowledgements: true + Retention: File - Limits + Replicas: 1 + Discard Policy: Old + Duplicate Window: 2m0s + Maximum Messages: unlimited + Maximum Bytes: unlimited + Maximum Age: 2y0d0h0m0s + Maximum Message Size: unlimited + Maximum Consumers: unlimited + +State: + + Messages: 0 + Bytes: 0 B + FirstSeq: 0 + LastSeq: 0 + Active Consumers: 0 ``` #### Editing @@ -208,7 +232,7 @@ To delete all data in a stream use `purge`: ```nohighlight $ nats str purge ORDERS -f ... -Statistics: +State: Messages: 0 Bytes: 0 B @@ -231,5 +255,5 @@ Finally for demonstration purposes, you can also delete the whole Stream and rec ``` $ nats str rm ORDERS -f -$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --dupe-window="0s" --replicas 1 ``` diff --git a/jetstream/clustering/administration.md b/jetstream/clustering/administration.md index 7c05e83..ef213fd 100644 --- a/jetstream/clustering/administration.md +++ b/jetstream/clustering/administration.md @@ -8,12 +8,12 @@ Within an account there are operations and reports that show where users data is ## Creating clustered streams -When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. +When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. The replica count must be less than the maximum number of servers. ```nohighlight $ nats str add ORDERS --replicas 3 .... -Information for Stream ORDERS_4 created 2021-02-05T12:07:34+01:00 +Information for Stream ORDERS created 2021-02-05T12:07:34+01:00 .... Configuration: .... diff --git a/jetstream/getting_started/using_docker.md b/jetstream/getting_started/using_docker.md index a16549e..f46a7fa 100644 --- a/jetstream/getting_started/using_docker.md +++ b/jetstream/getting_started/using_docker.md @@ -1,29 +1,17 @@ # Using Docker -The `synadia/jsm:latest` docker image contains both the JetStream enabled NATS Server and the `nats` utility this guide covers. +The `synadia/nats-box:latest` docker image contains the `nats` utility this guide covers. -In one window start JetStream: +In one window start a JetStream enabled nats server: ``` -$ docker run -ti -p 4222:4222 --name jetstream synadia/jsm:latest server -[1] 2020/01/20 12:44:11.752465 [INF] Starting nats-server version 2.2.0-beta -[1] 2020/01/20 12:44:11.752694 [INF] Git commit [19dc3eb] -[1] 2020/01/20 12:44:11.752875 [INF] Starting JetStream -[1] 2020/01/20 12:44:11.753692 [INF] ----------- JETSTREAM (Beta) ----------- -[1] 2020/01/20 12:44:11.753794 [INF] Max Memory: 1.46 GB -[1] 2020/01/20 12:44:11.753822 [INF] Max Storage: 1.00 TB -[1] 2020/01/20 12:44:11.753860 [INF] Store Directory: "/tmp/jetstream" -[1] 2020/01/20 12:44:11.753893 [INF] ---------------------------------------- -[1] 2020/01/20 12:44:11.753988 [INF] JetStream state for account "$G" recovered -[1] 2020/01/20 12:44:11.754148 [INF] Listening for client connections on 0.0.0.0:4222 -[1] 2020/01/20 12:44:11.754279 [INF] Server id is NDYX5IMGF2YLX6RC4WLZA7T3JGHPZR2RNCCIFUQBT6C4TP27Z6ZIC73V -[1] 2020/01/20 12:44:11.754308 [INF] Server is ready +$ docker run --network host -p 4222:4222 nats -js ``` And in another log into the utilities: ``` -$ docker run -ti --link jetstream synadia/jsm:latest +$ docker run -ti --network host synadia/nats-box ``` This shell has the `nats` utility and all other NATS cli tools used in the rest of this guide. @@ -34,6 +22,7 @@ Now skip to the `Administer JetStream` section. You can join a JetStream instance to your [NGS](https://synadia.com/ngs/pricing) account, first we need a credential for testing JetStream: +You'll want to do this outside of docker to keep the credentials that are generated. ``` $ nsc add user -a YourAccount --name leafnode --expiry 1M ``` @@ -42,7 +31,7 @@ You'll get a credential file somewhere like `~/.nkeys/creds/synadia/YourAccount/ ``` $ docker run -ti -v ~/.nkeys/creds/synadia/YourAccount/leafnode.creds:/leafnode.creds --name jetstream synadia/jsm:latest server -[1] 2020/01/20 12:44:11.752465 [INF] Starting nats-server version 2.2.0-beta +[1] 2020/01/20 12:44:11.752465 [INF] Starting nats-server version 2.2.0 ... [1] 2020/01/20 12:55:01.849033 [INF] Connected leafnode to "connect.ngs.global" ``` diff --git a/jetstream/getting_started/using_source.md b/jetstream/getting_started/using_source.md index b21eb8a..7b7e821 100644 --- a/jetstream/getting_started/using_source.md +++ b/jetstream/getting_started/using_source.md @@ -17,7 +17,7 @@ Starting the server you can use the `-js` flag. This will setup the server to re ``` $ ./nats-server -js -[16928] 2019/12/04 19:16:29.596968 [INF] Starting nats-server version 2.2.0-beta +[16928] 2019/12/04 19:16:29.596968 [INF] Starting nats-server version 2.2.0 [16928] 2019/12/04 19:16:29.597056 [INF] Git commit [not set] [16928] 2019/12/04 19:16:29.597072 [INF] Starting JetStream [16928] 2019/12/04 19:16:29.597444 [INF] ----------- JETSTREAM (Beta) ----------- @@ -35,7 +35,7 @@ You can override the storage directory if you want. ``` $ ./nats-server -js -sd /tmp/test -[16943] 2019/12/04 19:20:00.874148 [INF] Starting nats-server version 2.2.0-beta +[16943] 2019/12/04 19:20:00.874148 [INF] Starting nats-server version 2.2.0 [16943] 2019/12/04 19:20:00.874247 [INF] Git commit [not set] [16943] 2019/12/04 19:20:00.874273 [INF] Starting JetStream [16943] 2019/12/04 19:20:00.874605 [INF] ----------- JETSTREAM (Beta) ----------- From ce3996faabba9196572c7ac1ffb868962b7b1214 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Sun, 28 Feb 2021 14:43:16 -0700 Subject: [PATCH 62/84] add replication Signed-off-by: Colin Sullivan --- assets/images/replication-setup.png | Bin 0 -> 18016 bytes assets/images/replication.png | Bin 0 -> 15453 bytes jetstream/data_replication/replication.md | 169 ++++++++++++++++++++++ 3 files changed, 169 insertions(+) create mode 100644 assets/images/replication-setup.png create mode 100644 assets/images/replication.png create mode 100644 jetstream/data_replication/replication.md diff --git a/assets/images/replication-setup.png b/assets/images/replication-setup.png new file mode 100644 index 0000000000000000000000000000000000000000..7d14c7d0e31828d5f2e59d0b6f98b18e32d9851b GIT binary patch literal 18016 zcmd74cR1F6_y_vP-kZmW;xR&a5Ry^$ij0uGS7wPPd+(8zog$@#Y@uw5BrC}#D_J3% zb3c85zjMxYoxjdM=Q`i(`d(M@JfF{d-uLT%y?_E9g44*6o4XQc#@)V}O_BI&5R>*hPY<$Xf<8jfw zgu7;I)*#c7_D8TQkWdhPo7YAE}B{KrT z&*@nNgK?#&)dix^kGiTJcsFb>qa;HwR5H-8-k`dp@r^`~j^rX8N$b*f>-<)9=d4GI zb4`~S(gq!;^-^P>DWjLGx3H)P|Jf~~Sr4PN_UpxpC>rVq!VPX1X|{UPvhdwY<}$YN zt&U={yH)Mm+Xwyy0x76h0gk%r$|P=>&R507!6VKQ1CIJ{=osRZJUtsHJu|LfK$MTN z_`Dxl(VuY8(n_>uitK$`Q9%taNmf-{YgdauriCm;aPFFIu`Mqo1SOfsVlD-*Op1e&Rp$CorEOWBCfJhWcJyQsSP^U%qdFcY1L1< z?VkkVh20n_7k>5ZuIwZxS~G;>0M+l$mq zUBP+}Nix_P4zW@L(#A=O=4owuJ{(iufwq$3;_EgxoagIaUQnYkHa4E|kW}<~G2(1h zHKwSqEX~ryAE}Wvdb4UwfJ6F6&M^XTO5m`f$e!uEkk zWj=~&^G5M<8Aa+Ryt?t5yMi&QDdDhtFNZF1Natxq!B}tqau~#8BqNyhkhD$jUELj| z6eh+yhx2{STjyE4GN-?+h4B@7-H^eEVYfO?Dx&yCn`A*n4r)*IH|Lsyzy{N@v})-t zeUB6roWL0S;7_)p(;F{)(&gJ`g>$uD7Hd*Oah7m0Of68GEj5`7T^S#uGnYH)%&e16 zdGMi25&OOQt?Ms`Vq~wx4i9&4?8OCA7Y;;sbt%`xiP?*Zi^nd)38R>BlTf?dPDBv) zc6?3kJyI@Xh9f!JG4hYP@__Lnb&R5A-MFZ^*Rgz0;g!b8CnEzZm^OG6j#$Zvl>LL9 zEc|ZHT3Dq0lT-91vSiZ&B?s{wQDusD6C%1&J0a=U^NjP?FKn6~vDD0jwqA%=E-ZFb zT!hCe4voxuJby4e`TaX@iTjGFf%Wg`4@i^JZw?=jN$MVg8Km*(?=IpEq4qc-*AZ8* zUj4e_)K7{x!##tiHw z;cjthsgjx+@e8fIST2pnGsg$pm1~LYa*2_Vk(qX_q4B&*1 zT#zicR&Mm4KlcR8n~;8gf47>rzrP|*=#A&`si=WJwD}Sttrga~nwrkv8-Esh5y7XYfyaOAnj`w` z!$zDtC*Iw!aOh{#%n^4b>;L=_=_HfDFIhKEUTTo@r#1W*Z}`N7>GH0xX9@+YWbmg{ zo-lMIJ-@-z=4N)p*YWWbRx z{AS`F)Bnc&J@e{lc{jW-IU|Ed*|p5{9f$1D*jUU+iD6v7fLQ}wUtgc2n_Ji8%O-C< zX5}5lm<238u)?xP1{_v@H?*?4DDL_LE@47J0S8+3{?T98vAy+`l^f>fY&*U(vIjqD z;nrzpi75T%OoyPi?A4H#=Qky-WykojD);5Ua2SvFlVEgYQ(Hy-C zItxtcHxg)i!u?tzHZ)F=!!o{gb(g!;Lp@h6h%Fus3vJpQXjWU++-$LfT{7cZK{+}0 zc=YedM#JZm^_~j3Kegpf^lMMH=DS;wVakQcMSHuuG5lfX<=wCJUUoEm`J&}15|u%Y zkUibE5iqU4b=u){#jF#Y3-z; zMF4&m^2d)KbiFw)0t}>|?e9yjnlTD_Z3ny8ZY?%g{PYTOpM;C76^PmcCrl*h;ZWMpI= zKMTLvBx8gb%3rugX(ah$`s0_nligasWsZ@8{c zDIj*`ip=iMb5tjUdM~x;oz!_S?UCeY9i#(_{#98SMLfD`tBVEkll}-6W%<`Hc|$|S zALHXX1_qJ0OAJ2AmhZGXxw_K#5fqu++TXK`w`vO~-d}Irpka%%ZMS%nDuD3`&ieJdy*@MbKzQtLT2)LoZhYXcWfGKpA~F5VQg~*&UoOeyi#`nz-kcQw3-4FAR~RlE zKgV>kQZ!dCE-l501%{Ry@VkiT6z7WPkn=$J9P3u)8&qKrK=tx8Sg-E-R%}+!o`fDp zrDkMA!BvS3Oo2G!|98wGmh%-QyR4$CYw4RH+xOeCp<8m35OD1Wk#$km?#5I34cUJ9~SD4GmIm zc-=4SH~XJl^I3n@J@@sQg0)u?ISP+$GHMMjoF&ICsy_w1&+ElW%myzi4P+jwheY z=!&5jXl!WcxQ+QW{3@C!;w_C*qshyJ#k)Q60^%oJugY7@ z(Q9Ed=WJXY%LYB3aM!a3?%zmTfEgg3$odi>49m?sJL4E}I!-;xwr)Icc)&|GN$sZc{U^Z#qhXHltdI~K&ku!5Rut3Sn<4T4 zhKbVxVaBd+&9OB-BV)cRfd!pE8`6??Ne?0u2~EO&cV@hW(v!Wbn*D7rZsxQg73}Pa z=6q$Tbc~JpxR1_J4QW7p;~bPd+0pYXC>RRq2=&{X2|>yxePFLFFW-1Y-}vT@=FiTY z!GZga1x@QO@NI6s)AW7McFp^D^w?5I3~eh?LsBY30F$p(IdgwzxT*VL^^#Nd?gi$l z!)PAK(>q;A?~SSRRqM0uQD0Yg(f7Z=jlL!&HK!b-lE|t+^oSBU zjaf9k!j+u9$HTGblVHL+#J^To?F{<^j{e>#9B6@y^lPM~vY&~JK)7qOm5}}8VBv;Q z^ty|ylN0rdnVA`qm9-=9n$OqaVbqH#fa_U`{ZC~0Wru$J;2S(TJoH%n%(V5JZ;BCZ z(?v#&uejYfeWLZ7r=cL@@nZ-tAG0MP>J!)~~eyG?i2MPF|<5$HKXQn2cI2Rf%y)g6r)}bokvWx+4vh>)$IBCd%xp#%|rZg@9-W z!M8_BD~U~pkB_{ZZs?^pWrzUTk_I3LK7RhKFV{9c@(Bui zN$PoOBX0f->%V zb4nPa&vby{w}dT$-TY~`E}`5l<$$HnS)8<{dU~{f{`|S6uaEcMThTb}PpkV<yn3uUVb6G_$h&M{~Fb>f4!3=t1=WKpY26uWU_CesED9w zNx)b-E7DFK2g+3&Uw&i2Z)R3SW$rXM`>7Ag8tV%9L{Z;04x-&;6fr$Ly=8pDFB~e) zCj%@LQC1>ZPoa9w;#7&3NkPRdJ>%Mbb+P|kb77Zrjjou)7bh1Nx(!RVwfWyF4|+pU zYL!0pgC)44jOI?&#;!li-?q)Z{e4oZKiviOG?jhT9nF%5kH2+_PNve65x}!skYU~P zI`8i&#OGQH6P1kc71U}<`oKq_L_sxp%4=xE6&Lm&Q7kSstPDe;#7L;uso`GGXzAmd&SBF4T#36__d$=k^&L0iUAHF1 z`cK>qZps!OYu?w+g#ETVh6!w)5}w-B7?*myGj()vVcpY#dVP-1NJrGk?Q(%eJdL6cB`4-8EO9pZ0tysEbrYEoNN8l+rICeSkeyA4yyjV9?MU;>;o~M`^v((@!Js77q@wBCszG$c#p{u3D^h8XtEbol&2;51gf% zMeE%mjFRBI%n;|uw8A^;hI*ok=5klEV1Jvk3m$G5uE|lg ztRxf|VKFPTl~I3gVH830>qe+v$i|k8*Vk4O{=;WecSoa_me{wFFvLQ$M2W>_bn-Yf zx$3N4R!!jc5xTIMqWi$RS&*O^6{3Ix6~}~JG45*3JwL;8KaOmsCuzk zlCl|?&f7M1TBV^h=HVH;rg&kICHw6y3Sz+o$xu!VlTdrijxVt&&YM27Re0=GtMDz+W3n^GGedYSi?}r> ztmOBENoU3gjFzegQ;Xa0E>cX0vVmTU6>FFFCheORlA1XAcvu87D>TE9&BGM3p!gBF zId<$$C_kDJiH1QBh0&)fGB#|I!Wy=fe~mMTp~O?eSLV=c^X{7xa9q?Re3LHXB3sJ0 z?jo%^WG3`r@saYOy~1v@3}nJz3w^sDUCmV3_=IJOnLaY*w(?^l*lHyST-wEb*(?|o zLVIuhwWLHMWm|ci8M$n(Mu%Bi8YPUtxK(N3MfO|#sp(qF15b{Th>0SomG@UkrZ#fQ-JIgvzF4j&}xTsEQ90K|7+-g z@EA8CGm%r(j5Kc{i8-`1fRV>%lZD}c%t@&uC3%Ael z@q3#Va*6Nxr>ECW8+(fUynnx)FOoZbl4Zw z?6SQmamsi@acxe9wtR+u>-;JCDS7qRIXGM}z!L9)Xt5AH39UyLW~b+;;R42Jy2*8F z&xY%*%_T5TbtInpF$vAnH-!du$(5K%bKJbpntyW=vMag_TQW4{QoGq_d+sVZ45QtspUzh=aJ9^z44iGjAQ@^oy-_ zX#cb=ryzKtPJ(ky+%@Z_P1)PukNe5%espr_!Lp0;o(*FI}A4E9BZmbt?w8_op zx3pL*>shprGNF0Om`C8McSk3L%x-M@FkM@SP^EwsQ!TK++dw+kzqN$66K}cRrz1~I zGa+e@sk$<`)S=WLfL#coO~zqXE*kkzmL)|A6DI4W4#=zF@)ySjzker6>ZbMad}j(A zP$dWAM0P-1eaC z?fe!b%Yb!-(UZGOPO=AQ)3;I&r_E36Jj5tIRYIcD3CC-NCA$hT2eAK&x7GguRVy!f zwq11=bG9{d7CgEN)QqAqG92}a_R`kz2|x`TmYzTh?w+jF)WUr;O!Y@j`I;WB=Y*(w z0YD7Vj&BMI&M6n7l`XYi-dCLKseWNmO$4+n&`QCDBULir;N;cH(J$=1NyTrkjdzcc>I{5CQcni3>TOxMsa5Q z01KP)v>?J|g%(g^7Y|B_ry|W|7xJ z^}=RhSn9KsBx28ylRyg(J)XOC+`CY47)5fxU@AQp+2VGXOmN)d%bc{EP$;C{M} zX+EEy~-1E!OT< z7lu|&(LlpDIH-t%(aQHY=jL{O)$Uz_TLlY}CZ25K<0HYExGyAaUSW;T%*<@t+w^c7 z_7a4!fuW&Pk26VB0?&EV6X&aH=F)PBlG5uV4#PyM~9`FI@8am-LRGsQ0Gx8+I+WBvAp4-i3T% z@tlZe0%=)e3Pk>o6ItvgBd4Ge&n1(rHj z`k~0;{gacE{mpjDs3&DP`ANW5%x})Nf3J0AfjYWVCTDVTvi`v;!y;6UgMiC{(8EFC z;^Inr&fx(SjkMwATjtNMM7NEz`AV`FFU&=BPV94xStS%NBqaa?t zI(u`-tDb|Bf`a28Ukx-o*SdFpz+Wa910P>Xk&tPyC z3wGzbcUzI|@CpP1ODEr3SzV1Ao8DO&YDFq2D1-^TfA$O~9<7-#Yxel`U@1TD1jZ?3 z9|T;5W&CWfP&1vA-`;w{Q2^p8uv^b=sd}1;K(T0C%3xkDvG6cSLe--`2JT&_wN4`v z5y=+n1i^)9?xxseb5L`^jG%cFuk85Ll;9vxfC7U+tR?OS04@w%8Ik!Lg zN#6+dqd=*hvk`FR-qOkv!DkV7q3UY~a)Qr|QQM=mv{c?!!hMQv#m>$SxKY?d;_YeJ zTDOUM9LwMMgk*jE^L?XJQ|4n0gM+#*KdRLpKUVeVeSEontjd8lFy9smkU{Q0{q``& z8^TCUixx!wWE$=@G_urIR&*v@J5G* zIC}P>q8&YPhRrE9y_gjvp z2SX0Fmu!IlnhRokP-@mFZf_ze3B1P2(96Q-&lTtJnXdh>7r<+#-1Hp{tT5g9RKo)e z#Ro@!*H}Cs*3V!%uK3TcDsxIAj4Ex3kv5f!%Ru*R{kU7+>B2HuZO~)l+PW2$Me~VM zCNPgly@4oEMSME&=rREzAzD&08e$;zd*DucYnMWM`}&mb->=$lb)WtqV?W)r9aue} z?1cx*$5y3pXegIDQxL&I5&=p9BCItSOC#BjV{Q(^On)vez7~`Onj}<1ZEfbw-w+a+a;#`!+_>a%gBhsi3N3yJNH^Q{uO_Dq5L`Psxg{CpAToV`APP0 zHe(Av0rBkU>Z%0lNNijj?P4m~esTGa%d%0Npv*8B+9{&S&A;&?I0Ir{|#B#bsL746sykqYw`$5MC{LA;0Y}t;9lF)>*5_;0<7;BN*#wVf zbg?d%K7+ymqWpf9zQ+O~DlSb|)vYxEMP1?ss%FJ2_C-ni?9(qxWGMLSKMR6R$K$R? zGy3u=D9n5FpFe*FqX|G$Bq2q}1ocq@aa21XnzultEPB8&lDZ3fR@W=jfx67^=TW}SWFwrV2@1d1 zoonrYNr?cCEGO=buXsrTu7deIEh>sQ+)#$W+~B1P?Vb+-u00H|X+=%P*!u>Qfxx(d z9Rn06DmDvXE=>g){bi@?6L*07g0aosdj}am;`2Ttq|-WNbhYTi7XiRpO5nqRwE>Or zPSxYRg)PUVI3YR`JOApoW)(24NV8G$@$q%=@>l@g`t5xUUJ%-9*%jnq{qxaGWv(hJ z6s#wJDuCxo{q%f)Mg_1CmYZyz<vax&MdqY_q?(cwb=Mw|Sb~rx_(^EpVZgTCK2_Vc+6RV@;Q8o{5GvSxnU<{0DsR z`HfJ{;Yd6c6gds+GjWt8hts1OqMD2Fg)lsoVshd{Rc4f=ysN5*;vhW5Ss|1&g@8Em z7R&#CACd6Gc!sfgv*6jY8v=62n+k3HKwXag1;U*Gfz@8{@$u(^LXpJSxek9jXK+_y zD(Eagxa%K@9MgCLR6kEGB&bf%9kB@r0a73z(nd1q)ckATjl*baid5voA+NPsfEL4!h`Kr0d`~u#CS^J_RT^*x^{}%r1*RQ`041|EXPMMyb&T-iI zaGM(>SUfn2QSBjEU<&3Cj&cZcZy?v-9W7JhrIUv@fi)#Fv``6!iycuU>H5@rzXIY0 zn{JAVNq~V*M_0ENGA#H=+*SsQz+~_rrt1IzeM?uDBfAM!*7^j%pb{I1njic6@NLMN z&rUWx=Q@ZX>kW^N&O~iVpUuqQ;8@L5U;~Oe#yNWw!ZF`d(DReA?!J)DBSfHbX))! z^FKeOf^~qHgoNZ#etr^|6e^~{{{wP5fa`MzV-pjl`-hOGmzS5XtEi~_p`c0D$DZ$7 zG!x)%iO4+NwK$_kfTj};Rv^R{ex28fZq^Lqdif})kdP4fG?S=(Fq)fo2+jxW;mw-~ zF5^&MM5|IEU?cFlkiUNY>Y5><`gd0%Q)U3AL6v|lA@1Z)IN~{gZh&uywVPzR4A($7 z1X=#{aJAAkU-}_OxZl#3^XR#2fZEYl3Bnt&7nEAt15i}yBXtnK3`&i;zHw=IfWe7j zdo{BCXcCqJcn`Res?4Sy=g*%9wCLe+>kYq<5aZ2u@RuY{ef^5)LN0#E;~yX0CK2=v zo)-HA{r!LgD%!?~az1rY@<;su_U=9;r0 zh8n&1tueCD)eYr0EJGjw(QvzU&uQGb!vuef1f&}s0Km?=FZ&8FKR@kw`s2qS=FuXN zNUG4EebdV;yig_?y}QSSw889-7@~dFO+nK8JzhtNcMNX|6%I=jg@nYb_c=C%mSTaTIF< zfC>x|TM{8#Qq*%*XQ{j{qv!sbU{al9L6cYT4#nHOwdprxQpD5l^UkcG?( z{LqA3-A9X(5!u_JC^>oHfgaOdu5wHUppn%C)i&@ZZBEAmh5zy@QS&q|N$V&H&I{+LT7NI3mV zy*(RR9YD9v0w#oMU98@^P?Zd29$gXmEdf&uN*Qx?5zv?t&;a8E5JQX^%485#0V|wS zKW`om37!b7so6Eadx4P=2s&rR1l4oEOk-gZ`~(cas(mY7x^#b7;S~TyGBvq{yFf_= zo9sV`I1r5v#mneYO~Fg*=H$5YPe8m@mXr4hMi2U2n|&4W*gI1}sc%fcUy~CPgn3A^ z7HGpG2PrQ{fT#z+ZXmuCJh;M0{YIk>(d)oy!9u+;D9H?4eCd~~&jWgzbp@m(AgKy^ zg+YP>Vc#Q+#E1_N(iQSlp0_=j%?0BN>GMo3k&IL>?i}9*?|}U zf=bZH2@qlg92sDpe%T(46KB@ROIXO9m zE95wCDIy~mWK>`Cj-iXKTuf>y*zNL?(tErF$p!8DLku=>!11kBJHZ>HigS>iE#of= z2~mRGk#5|_+uM5g*T~$Kyher~6v(34IXM+}y$oOtq+(7W060L1sIIAbR$NTx%b&%JA)q%n!jXpE)twUB9&^ZtJOFO3JBPpyMEopR`Td(t_Z_6>&!_)m za}>6h!%{N=z;pB})A#4gQFbABd;>lQ9vvNZKJ5k5i`{B{+TZm_{y|_@E~55UUgn=e z#Ky*Yz;;z6Om6_U1H3K-Vt;z0(r>+k~8XKZLAW~+ee zj-OwnDmxOX%3FW5x6zv&B|R6%z>4tS8Bl>7XekW-SPLR|Fbu~oG?iX`Z;O zrsg)$0uXN;GRZUGEKPD9ul+PQN9+bqhm8i06}XCNkA4GcH2DUeVb~eWmUS|Nw#PqO zX|_*dT7SY&y7d*#%{c>h8dDl*K!D2U_0Q6O%n-FsUzVs!Z9w%57?gtlhxmE_{ycoy zsKz&2F#kB0Y#}Z z|7w@UADzfiSi0y*w9>?plFar=VSq8hcD_$PNw;I1(~OoogTn#Z*GAmn-SXvHoFD&EvR8601q1 z5CyH}F?+NwEG`0Jc_-Q#c&6C#?z;tl|JPra0q6}F2mp>YfDHZ_i@6)#Wn+DMor*c8 z9^i}Yf;mO-Ha0fWuJh;Nbx-5pC?+Hpj48g;8YQ7q=Q75D=l%P8MdljqZfhRA|Aml% zIlHMCFhF!inqq_r+H27t@5?6Xn1*`Fm@bPBIe*eP%?BY?ME|U zoVDHFnM>9R70u}rn>Dn9!vZDv^BsyvN8o;VxaaP%$@qo?hXh6sz(;_u;vRsFS(_fa z#6Z~L&UEt>e9pZpYIPMH)u_XTv>9A#@DW&73L((Qwr$*Ctj9PFEhUqkBzG!#8Lzuv zMn+u?dnmhXMP7-d)IL}>xEKb+WHK8#{Ph<+B*XQ=1jS$L4rb4BFeJ}@&Ju;-0k<3| zLb}9_Shyzy4m%*q3MvBC7)XWI7?gt;^U6c3J6H+?Fah*BT)OB)M687my$`@6XuZU`l0dQ+ zo!dTuEMO^;wQQaK0#akg>TaxKqEwM+6!04*BP3l&IA#wdo}TN06gzsX({lH1@KuVoFFz7{=3$fliVEI<5*k zZukWkWMH>Cg_03@P-kX6Zl&_#1B|FQ6@-Nn7T<{!j(s7@NMo7}`W7H4h~F(p+RD35 zPCAW_cNBm|>0JmNaHJamSI`T_E+~BK$^Hjc*oqL3LF>>hQSUU+v4GfxMWlWO?m?hR zVpFQfK!XDqG_!ZqNhH9Y3kur~=WS4}3ZZ8TjA3&a_#}Xt*x^c`lo+7#2gIu7syK!x zy$fKlny*h`%g6#lR2ZIr#h42;_|R=7(n67=`meQ!(fvW}X^2D44r~fE4AI$viR(Xk zuWGK5rZ5TwT2+*cpsb)A_^M$wfO5Uv3A7b8^k@CkOgT~FrnXJ*oa)8_Nvhe01F4q? z>=QKGfNu{ts5zN3f3U6LMG3)JK`|{fW~ES>IJvaoDw5R^mj-erEEep<&Djf}%W~4R zakX(F-+Nr-yXGEq;xu&_}4<_+Bnv|s>bDgsR}*=hcTYIf!Wrl23y)YR}^x^(0AZTfFzW^q^o zAXQsd&FL!0eEfKIuH)hmef0FgfC04x^E2FaDE`0!>CCwC4*DQuu?&f)LrW+=>0A3g zW{}-N(b!^ead8pdzc=pQ+?m3q0Y>@fYN?98AQJ(6fW`^}0)j5-X-C~G#+Rj~3UC_&)aJ&E6_Rs8L)B(vip2%7@^IR=~#MxaHh|u z{iIL&SFT4EswBl$0~G+JctJyhac!yY!_)-khxU9-LiJEh$FgHoRH9t+ppgVHPUjgZ zg8;%tfD@nbM5Yj*5CPyT9_=#yA>i|Nz@MLLns^xkANJ=+!Iler2)9hRB5WA&3ucYp zp9Y0Y>k&Znof;N4t=x+jiHK)pMWm&t11Q4EgmOXV&wYG+=Apbr1cN)8^ajLsTG&y1 zM6T2W68HDJ4A_Pv0F^d>etrT@9Un2#=tGI!a&S6h_Xw!?o(_0dpD2o!H#V{~dat#C=J|DMDsgW*h#m3{WD3W-cWIA+ zErHq^+l>TBvQvx&?5PPrLSdU$004AjbxCZ0i_3T|`IReIu!k=oKn8r^M3%*rR}P=0 z0RmiJUr)>#1!C!~+oTj0kM^Wpg3b#xHm+Bk!bX0txyuMtmdDBA1DB~LCP)m>nd9r6 zzW_Xv(ohhFvHO`q6wTwh!a`i&6*_*@+=W{XYeJmg9|geSb0!)WFh64RX!?u|yBzUM zqSk_%0JS@Euin1}qzou3v4_QJOBoh{F)(=S7qE;K4gw4hzmN8}<`Gzs1si`JhFkBo zLgVc0e9O=<3b;5#%ZGp?>}tY^vi_991w^1a2DbYRmFZ55L%h{OPx|uOTDt4~-`~+d zPyqec9z{71k2I>XPuSPIaRUchlmL`L>qHz3{sPnhQ1O6KoSJ5`!iF&TnSClqc`WWt zIEeo!GYAHVU})5c`S=>p_|S`+*uE5Eus7VluLH;6lHQA_Aa2D&-wc7ySQ=~(9Qn<| z)$%y$0{pLFg$8P7ey#4G>ljO*T=YR;dwsrJK&=1_JyxNnJc1QOSmXNux=mi%bi@F~ zhOGw5%-_edNVr`u8~_fi5KaXG=u0fM1|%hvBj8oX2CJ4^UixNcprL;&w@AXy6;N>O zL$wqvw!xkXhcCIUqZUErVXauxH_$~m2>|Gm;K zu2>t)?OK>zk&ri3tin6?DD!uiq7MLgTv(=bSoDU?BtGwnXy*IdFYH@4f4A0PBLH-Ou6SVF(HU7-OqP&HW$(PBc9fUEhmin1Cw< z$B&!O36ycoZGUKH_}JS^dxq@}>)-aF@^FfhzlczfiUIl7#xp3Vpal&(E#mEL=%mujz>{M)eLB#c zZIKqtesniq8LFX=8F5w^ymI|1U?rg*1zvZ5%Lck0pB5G>Y)L+WAjK^y_|2~Oag6J} z=fZ~V?H*AiSr729>_NvBze8C9PW%5a_H{a7&+K!>PsT~8b!2uWi>}?_l(gg}3p!e> z>x>y*-QE8??%wIk)#^IgKp&4D53*Sp4VCAA!vUmq!@T(N?o< z5ti|vUg$xMEUy@!ODVg5;YBFKg(!61kMB^bGS8v{g1I2PLM)omQl`*Mvdj{k=b}Ce2QD>NDSl;@F znSyHqD4`@p$7unvNDJfO)VT|a*6#M z^^g(gCVi$8)Z_w>>px}$sTBQ@O`5?B`R!ZMBhWXMFZC}`=T5poQOYP}9{*{(C!L=} zg}3S>ruEYA=c^Do@ZB_V+?knW@S8vL@T5b1XPaX z>9Hh#y2O(i3~fVwz0vbL89#>q)B;50;Y}u%Uvi6+7g}z`g=-5N^rZGjVyLp$w1CQDq*w-x|^ zmyb_ea?di+(EL^h(a4&2w~GYpRGp}4B7{_iWG6^toy@8a(6s0CzOFjAlgeSl5N+?KKHp%d=14a1cl zYvXb>dr`Wc2&Uswp!85sAhH_fiu|^nN zJFK)7n*zowC}rmt4}c~_Tfc|G8U$?5kSF4<%+S>nZh3_{!NqF_SSjmq;CA45*vJh+ z*iFgh^h>r)Gcdbzko&-&1N~Rv#)zf;dwR6dTc!>$i$?A>@EWtw4-^hNz;pV3Z;E#O z@9yrivNA1~ex+9aZmiA*9aCMAFK|Hm&{nMh4;949^qYFv+U3BX<1)}oVB+Y)u160x zKSLu~r++tu_{|RCiu;$@nP}XDhMnI`C z8)z_e9B(BB#pjFyLW10ajq#C@*bb~&@MSUnS=2#KpHbj#f<}Mwkxc|_DwF%Z%RKL zN>{OlTXFcS1nabeqXE{eu=jp@v8QVPS~l%;*5GBfORzc;+_7J+qWUGW!njYw?ux)Z}Es#)l&<@sun6K?Bdn@zeq})41#I%}#s>V8O=11Ht|taPHXN zeB-!2*~rtlOeSXzt!uF>CUGiYKm@wS0|4L7&W^NSNR%gJ3xvXuJ#^W6LUhk;RC!X= z)N(89;PL|~(>zyu!5%lSrJ9)G04Xtg$Jf_a`N|^^@Jd0J0uarUo0R1Q$0piAH0~w5o4>hOKM#Shj<|0*L(IrFtqJ8`fEk4*l_*-R$KnR qEJgqOI5POc6{Md7Re3tI+ zTsDrD_8yk5BrZ2wGF*cDt^^xLJ4aiaKW+GU_;@e#@LlE;(BaTHcZY&pJkY@fHA`zpH!^|baPdzITe7A7pWZaR)Uf9KcwZS$2Ywq-RUb8y zwb!3EUbY@2M}pg*X1oGid|aY`ZjgQ5ZU0=gvGsDawneV;%fp2J9yvVF@$a!3s<;UG zSSh+_lSQ0)Wt53HYajbRow!-L{@sZt4EHkcpH77Jom8#K1aAdR9!phG4IgE^pPHJ3 zqos|KArDT^QP)V%$WVcoq-(G3VC17N=j9=w?PAZz?<*vy?XF1TQ8~vaYwJX`u@%v}1<<4KBGA76Pj4FV2YlC14ry;XhmaE{vASY;wfo5$PG z>KvbftUXasL0`Z}PT!uar{#uKaB3qG~?MIJ}RhGZ8QE=;p2I?X0G#&8wuO zZ>8%hYz2(ZfgB%T?G4 zYp9?{QWfSWDhMhNlnreivCaff8+iwJTR|ZgHw_O@XCWhdeqRG`lB}GNzA&E_fuyY> zNOX2}b5kVp2^w00Ey>~pU4>Kyy)=Ccv5vNOI32u-H-YTSFYL|h=&Pt|WTogWLbekS z6tQ!4MV3NDRFEL-z=y1{t+I%nrmHp{FK8)CQugvwR1t&*4#KLc27(S+hVZ*Oj?bDP z=%}lTSMYYT7BLiX^42o2cJS0tF;wt2a#r$Hw$>vHYwF2cYdPuSZS-}$+;j*64n_na zK^Kym3W;CY#n9Km(N=&Ur{pVRXFwo|>T2-n3wT1`G8%#^Sb1ePIR`#{FntAYf|jn4 zf{uVZ509<8ySj|B5bUC>pR*^{)m>2-EJs$r$#_!#J^+432R2oTf_Gr_5JiUob-$|-Q?7TJ^1;wk-_o%`N9uXC1p=nC1|T9f-}-{Fm$p~6;RaE5#k3& zA#3IB2zR_RaCWxN8p29mh)H^BC^|Sh%L*eK;wfZk!(-(oYpbn?*OXE5BPiIqxw?tS z;hf#{)HUQSmE5reWdj=l4|}4XF!E4-6%`jjIW6c$*g@9~D^FI^bX9iKQWC|3EvmW+ z@ar4uBF!}Uy!4fH!CDmDg|Xg#B8Y$CmxDk5;ZwF94E+7K)8v=eXq}osp-!Qcurj(n z@zZ_5y1GjVTXRJ+HEGGTpDr9Dy*X>~*4uiX%6UFKK84kLgQ=s`E-&NA3quXgOp%>V zs>!)br`nn6ZL3{^+`0wZbT1Iwusdn^7)@QP*s}&6zsJh1U%hw3`MQp zQM%V=tExPjp2=jLK7E>Hso#J0l}zZK0$LWah529KYq_ylZ2Ya;x7SIks$UkxXUq9D zutDsY=y1i9mWtCD8joOZjI$cM&d2JjJL7PXOw^8*-=p{!sj0$MbMH@HxOlN9<_yk# z{)bX)M@MpQE;=hK3l(zn=AL-zVAcl@9*FoYTsXukLAiVPh)d_|9Unh_q|!3gx*l()N*n;&l_vu&i!gwjblxDg2WBR`*ZPJ7wd z+Y8?a3o}1NN4Iqa%bv!1938`@5XHRmvzR!{_2B;f{PZXy;r_#ig;6JkXIj#Tip}4F|oe#n@ET<9XBqGGYpUHkGSyGz%b-S9wehRa6RbE-C1Z|3}Ya*maOmey6+>{Fba zv?>odWndQZ>Z#mf{@;v@j5IYhQ4w|5;Dpk1S1cU4#KGZDt$YjXbEjlnai%kOXP$kd zf`h|_r$MWZHgkzW!h+kKJVcl;aC77A?AU)c#D%?n&GRKeoKf6+rhGj59I_;_ zQoCF2@=kMb98gkH+DAuMZ|dE*JFn~Z?c20JMn;|&7atF!V^ffq-#z6e&cwvjYd@%- zN>8OyU-wL>rKMHFLDn=ex!v1qVq;@d?IL8|TTDcH=`A6}Ja}*eHbu%cho=R5)MoW) z#iX8|p5;JgU}wG~{XEO+EUuZVi`bx)V zvCH0)FSe<@{U&MjYmEiZDN)fAZEbC+5S>(Y%kF$fa0(6%4$C#u4o*(8o}Pl~85uFL zvEh#&A74u3HZ6Vk_mh`q2WICMr>0!6IGjJW^nCWHxR_Wk0bK5J?5_GKmRc3=itA%z z4g$i${^jPbu9va9W-Hg+atdqD2@5mM&CN-=(ZR0Nq}MAF1}a$!3JQvS<}gT@C@vOB zdGW#*>(f){%1Wp9P2wrR!QqOv^_j2nqM`Zu0*_VM^NWhCR~N<&@*QG2?x@G|?N)ij zHQnX~8{axWJ?=<|CRA;-pLt7vUVXs_tj0%TfXZw6c2~ecGY%CJPJiBN8SKBt&-ny_$MX3R zgDb5_{5F1_VlmxaP|zTlfg27`P`lon#=U!s^s87iFobq`ZY)TK^GQ_Z5 z=$flL_xcsEaDlN>@qM!GI?TW4=Xqdk(fioMy)t`}U%eXNd&U%x-wXEP`0eHMH*bzW zIE4_JR?l*_qU#^eG1b@qIk%W)TqKKihm|guSrT8+~*)8wR*cibEQux0oX%BmZHObN%@fI<3Y=++; z7AtJ~j)ERFt+IcOgoA6%4w>7L%$WDw>C-jO#ZopG21DuE!#@nC?^VN+RUkT-OgySu zw~d6@C+Fq0D4*P1EK9u|PFb~SNl`ih?(K4V@`MWM=NCsMRn`2ork}-yDjy ze@X;n&6sgc6c7@^h>CXc2CXkMZpY$O81)u$u==Vi*rc-_OOJ7`=Vbds1Ve#Wlzfgv z^dM^iCiCd1D6gZ(jum7aA`k-RKc3HmaJ~rO$W0po1{Lp4M!wHV#zhxaZEdbKT)uqO z^JD3&SNr3H32x!HZ#^n2T#J9i4@BNzmaGBCUyCtuo6Qc*cT zLql`t?p+UG7)}h4+BM-|+-em#Hd$9d^jpTB&;`ud78AKL15nsyh_ zG&76(@qd~zDv_HkqKbm^{9E;Okz#C+kket$IV=1n05EVlLhxpN6>Dk{!j zKYx}3xOIq4>h`T$GysVN1O$p3s}t1EPfSi)7WKgFuF8cSLIOldNeM($=b+W`)GB_a zNb2+F&m-CIIp$DoZheSuo8dxhxtFP6U*1x@S!3a2zrH-zQ-Aw}4q#a=dN#H#H3x?@ z{U52L!AOLkmIc_qRon&7gxQv{^{8p?r|0d=PozFNYxPY z=2Ui`Lr19nH!OBX(xhR&c+2)x+L^quXq_f%u&4NC<~btG zo2n|-NSCK-^ypV3(?$zaI^@@jL%ulNr`Si*>RnyFYJQ`i%XGf}Qm4aUK z*Izt<#kS9=Hw5^4UqXH!Mfx2dANrXsZx z=2+hUF2A7Q7`P^wTulEbcX)xP9`}UR-pSfwhes=h1roCLEAX=%}h>w^#QW- zeJi_^$maV49WU;E==aZtP>My}tz(Y`of)vZZrr%hXU2iia$1BPRxHLOBqT&4F{-cB z6A3fOS+7-OlL|PF>iuq76(osz_WU`lfGFuF3iag4lT#SzM!%@zQm-o}X>jn?LS^9E zDU70`V)dxS0Ml4wxVX6XppeLhxpw0K<6o%m?rsQfJihZk+;a4ad$8D=nq!V- zeKVCubDcY~*={d9;J`~@DhLBoOLV(ZH4+a&G=g9=RL5w3oiYgPJ-uk;=?`Xi)+A=n z;Ho7npoG4d8NUN&fOT>(8(sE^H8y-D$^t8A9b8J zX>oIHHiucjLDTum^<4ZYXTnd)#(;MLV1<8W=Kspf|CO2lD>MIBX8wOWeSc4oadB~Z zhc^$r(cNuS6&MxwhZe};HLeU0#qLTBT#T*nL9%C>gteGIpkT=vsE?mtVu$CRAX!;y zDH94IV?t}1rb%C)2p#}{vJ;h(k}{Yoy>StBik*EAclobt_09m50XUciQuU?=s;YW~ zN?5ZSKSH+odV=9|d%ON|+CA{SzCJ3wr0m z?^v|t;_)N&^hDrTQPltr?mT`>!Lj=8LfuJB+zO2`D|Sb?t{teZt}aSc)cdv!d|%Gx z%a?CZ273rqo?Bax+k5&$*31tiOlxV~o3ylj?DzK1RxGgsbOA6Rfk}vs9iQ`4`*sQ= zLi$Pj>GS8)7!l&o^?GL{n@mYf9scxyLzrber*f4Bb4^2fqjjSja&hkk)0mlqmMP&E zK9CP|LAv=mV8z3vl)N8v?adp$Xhm-GN@hsy)3UPaa;@um{1;u`y?cj3^#`rFK308s z@+P0OG#lm`b+W@>VrXcnT!9IJaTEX_|9~jJ7JCT(Siiri0Km93eynJc2s3L(62OwE z`}dDVD@HtWu3A}HL51MWe0KwnRB>woNFzzsJ)zI*6-P%#O1OQQMch%CYp|-(iW>1E z&BHMPEFu&KIXHSRS)zHwey#BlKG^R7xI;`F~=y%)z zVnQq|EJa5hJ4S=%>KK8u>9wChXanz=kOVPL*ab-8JR?S@y7K&QFC=a*oTu#LGA-SO znsIK$MKQ~eNc_O2>;vEiSQjN6>>c(Q$iKcvc)-AA@~iNqKGy8u9Z24v*}H?Z@oUBp z7|N@Z!TFKTkNIqA-n@Au@zlGn;@LA4%KiJ>-HZ)STv=>FO zsAynJYJ9Tl`#wN#GB1re14mk3s4UNa=5Frn94l%h=55oZ$)$0nM-Wc*_JrGnGO~|+ z>1D4twk;=Z=4JMO)aWtf=uh=o)FtbXp9o=Kc9PHL!wBR7X>O@p06%%jW13;x9JG;* z&5xQ2|E2VqRIbITuF&RWoO8OpI+o1;6YO#Q?nKChCGd!zTh1-g6|Y6`{)*HlV7pxr zrXgqRM%IR!p5C$B3f9yllWDgIz0j`o1ffxfxCZ^6%oim_#S=L@(9MymN{Wi^X@_1( zOD+$FdSd2_DU>=Vjv@m=9|@2A1cr=48?9`wqHgZp=orQ`xF{3qXYvBy7fxhu_fg;_!EXUE$Rz^`uS(s z*^ROXM!94v7pp{>zVu&9uok!e*v3A&dE<0a58giI;p43Ctvr|3r&FsYE-q8tPkn!$ zz>!FK6P1ffK%RknU>UBD4zB5;{h~j;!SzVY;_JAVP=zxws_@Dq?tSPO3dJ^g(Pp#& z?|r+xf{TxDo^9_T=FUQ(JoMXgD~=f#xw)P|fU2%;E4^^M{Ag@`$>$ezzZH6NRzk6u zmWCmd>C>dBht9c9hqyb;y!*xiBNc`xo12@Xl9H0V+P+UmYDlf52wuAMNy_8L2epdW zNYQ$)_rF_iw%e5~KAwylr$_sKCyLe%AKGI)Dn03%ec|$SU08%LQ=~C2n$u6ooQ~(zznn8!FuGvULmSc7ZdK@nZz$<*bXz%a z1^c6+#pH6YOPiNxSw-d=U9Cl~Q}M^1h9Lms8Bh>&jbOO!bb*ue`nPf2BR?NLWPoDL zsdMK>=Ap=7d=T1TJZ2lde)eo6kDk%MzPp>DR`{bc+aykBB|>$7}|-t~5!< zLZ?~pwbAlXL+WVYvVcYui+DT3ks~de_Y&7rxcK?$M@B~6Mb=3f1yIHUZibJapBI>@ zQs4v&qglkgswiJ3Cd)ZR3mY%%A2baPDfO}Ty|y;588t1vwkk@Y_~kECcW&9wDa7@C zp0@RsT0MGC{w(RsW8ON1Z2_i)3jCJ+x0aSCKq?}Pd~FzAa#GS?US%u?4<4N1cTfEm zzPjoMJdUoe?qJh%xeHue3T|#(E%1p%U~Xo?=mUZLP5x-p&4G!<41M!|#5n43;EQNs zmD$#!{K@<(!O>19v8s}hMlMd;xslOxn7g*^1GvXoh1}vf(uwnw{H>XdivO7Bkp!lY z3^}U5fx%;bDXGmk3Wf3n7^wK;(n0HCFzjGsEFbm<8`dU1u}}wFS%v*EU9!;eX@+;y z!Cri{$lt4dPBYl;0J=|sq3YE^{!xcSF9k#6c>L3*ktgIL{`#J-$n1vHs%-hVEF)X{ zG}l+Nn?y=foC0=3YjslIk|E=z=gpCCbp)qc=zMckuBbmIb)kb!!6>uvOh@I%uBeo3 zZkFeMagdQWz*OO}WjS@7iz9rKvu8{iV@dfh7&7*3J|YWqQ~RV3!R^AY!I1$mo23j{ z$cH*nJ7F7L6Rt|dKJn*CpZu0B$gynR+V`Z(uwJfj07c5r8_#*rmyEQf7-V+$BosyJ z84EElpUlbm*!+gvpBZH9Wob$2N+0ny>&Y?9msvAzgEU>HIzBmTR-!gTW59>UjJmks z{gAmQ*vo9yowMKI=`%ln;D}1H1>;EhcG*`m*;JwmpZ_-J!WYM1MElwnIw?J5HV!+7 zb5?0AXwa;XEQ+hscl8}LWxE_NjrLW9i^YXbaapx4ZFhAEjnkjvGd8}sv}GH4E3v*|^nUiKfGeUnHid$lU}M`yZt z=fJxNe?~yUDPTb6+?` z$i_0!2l&xcX~`@HT+R1x8ozGF0q!qk9&M@+Iwrc;L0+7O_0^;buK1ytpmBk6_A^{7 z+j{jaucIcVT%MnfEl^gk_GiLJZ4tvl?$Q8xa@5NjrRIcnn{Q?%o zvg?hli2EQ4g&cP{a3PdaNGY<&Z8r*dYH)r#f|qOAP$(#D!rd$W{^IR9=Fw1W%QP-J zw15A8R7g?3-vKDsP{PBn)>+8}21<7K^Z@&<4*>}yCdR~p&&C=XpFE>RK4a26vp6Jn zj{yz0tlHA`5JEzSxmNP9Y#;}6w#qxTh#nT*@sY4 zxdy##{7{b?2wH=B5rhtdavxsoE(B?TyTtA~z`%LM$cPEMYqBE;X6*Oo%j(*`j$2+o znO%kP`IB2Hi=~-90|3KG$;o|0N%Gho@ub=E-%)Fsz!I)b<}6)m0Sb8!${_f_^(bAj8F+-K_hzny{lzr{lrV$V-Ww4sMy1P)zkAhboxy5AXd*d*g(xd!pPtTBCSC2^}O@Fi!5{3>Ad?P8)F7J=G7 zMn~naugT-uneZgcwGK1?gKU8dXs9W^cIy+{WJ%hrJ*Lnt~~K-%(cX=^-)v^1^mK4_oo*Y5 z1c}2iSZ$v=JJZRNbHFW@3}pMUW2I`59`0|*m+WZcKRtf}1{(O{S&g(JRh1HObE#PS z)N0Tx9XP5*q9O2?g^^3!!?>(&y*#}#r>|{6FToeyAC$~K`hiol+e9U*-7eS30Ged< zvts(@Sn)LE@sh$53>jH3fS=#f)+3j~0Z*u@{zr6Q>w9b@_2}fYo|}_=duNF+t`s4` zxGr6^ck-XhZ@%hhPnvvD1!S&OTjkigrIqcfv+Yt^KRc!>&eY(z)z@Y(^R)=Vnnx^k z37uXPo^~E_b&p8`Jq7eM#tlgay9d&t@z#z#;Dma_b_VhZE?L~4P{XXA&dZAmD^=&@ z)NjZ|<~QMrnLiMr=U>L(^oaB1q4mg#>C+QgT{mu=w##@pu;MS6W8hKn5gzd&l~dG6 ze^hYHFG6ok9F~vF0ReAos(do}oMeZP97KYZcSFq61L8aT;vs4nzy<1Vd{c5z9Yhys zdI+}UJwM$9H-mA(NP>5u$;eyPqxmf)SN`k-*G{o7`j?fEKJJR**J8WUO_96zTe1q- zkkAIv2I(4tON=zxbfWdH^_m6@p_Ky(JqgmMuGgh!>4M|kBu|^)Hj#VD3g< zUfv#XSzjg4E3p&_c1+NgJp(@f%!J{@Y73enV_GpJ^BE&KaR09VGfh_WM%gAPH{3x8 z01jwqtT|c4=hrEaQM|US-E;Q*`3~t{)jJ>EJjBus*DcoI>L+=d5{kswx2(}yURdc1 z2+Sc~_Dugx<5$zicw-ZA{6uxgj-A<3D?jdeLeZcOYP?NNO@oij=>g;d`lQY4G6{y_ zNzYz7)?S!jE)M&2Vlt{&KR*w`d`S|ER-?p2v4&jJx#9VJ8!x&2B7~3Ap4>}&yea14 zLoZ*@5SRs97qb5Q$}QnCIdJ+Fh>MQgJG{ZepY#VX`R|q-;QG=!yPA}`Um=-y3n#s* z)wh=|psBaNKikG#GPh^?!`H8FG<0-T%&e@NS1GDv4eExO-O5~|%4BB~jodN;)%e8dwmVopmH5q=;e(kHx*+^dQ^1)KU4OYBM;jY?Fis6q06B zjotXtkol}5aV;q#^{JmzA85Ja^ysP|*)=@zT=@<%vG#)K@so7#xeJ{bO!um^Hoh3` zwll~qWLTo{Gh?tTWmz475DrUH8iBTG3Cjj9~xWC zIm+t6878XcZ3?fo;Mnht4+2}FJDbSmHQjyVdu`aGHhuN z#KyO}64EG0hhD2lv0>1@b(PB@hz^Z$A;(^TiRB;H_P9NoC4`Q*ashsp_F^D`NMib3ql2z0dljZUglm78X*uR2nrqp&hBY_ z&xRltGyiFL+1fmS2a5umtQ)Ke1%YEuhtygtA$ZN0_5|N9Ae{Q1{_Z#&{vpGy{?yt~ z=;_i-gWSIW%t79_I4ZTKs-=aVDVvolUml8(tbA`ScNUZ5C&AFg$VQ7sMX^aofh_J_ zU7aP6n+}fqGtE`nTU)PRHZB|l0cjlolV8*h^MPN~O(s}#&)VUbz`2^krk5^VdTrOh zbp6H+%a3=@{Yk4A>X_Y({zw-9+CI&IZ#>N7=4Xgd0SfI;%XfN=Rh!_cEoTot_2bikvYl1jo)q2GlHndHREg4z3 z&Ib*4a{BrVckkXk%*q;7HdA`QF7E_PS@O^)3}7w4VtdAQm1(R)R8Xb7hBB#3cm94vuIJ|`#;*3{I%t?A z=HAgC7jii+D<{Y6J8uJ`wzn2M7NFIkL`2A0Sz-A2_z-M{eDK`4b0YqWnHoin+iMZv zx89!}I5)1aM{L_3v;#j(Djy%O+}=u#wy<(w%Vxy?%ihAyDteR2gCKkZoo72yn9$di zXCM5F(Qp;n#0ToQs!D{%EbHr^o=(E208GAx5W9OqYp6<=JjZ{16`w6O|1G`sTY6^H zBpM`UpcebSjeJ3j89h2YG;{|{1juz%M-crim93TvMu_$I_b)0sNxKJVOnRt4lL?fKQ@LfDT4!8X4VsrIT7JtpL00HQS#RoPy6WI|u}9Uzv9@=o)~K z2dLqe8%Nkjx7bKmf|S-sI&0@B53ewMamP zynnAi+2=LW8xCZ?uSB*<$qC6<|iF7LC?J0r+K3=E_@-<&z3LtwflEj%05J) zfqEK~otb%^QVpuffhy2bfj0K;+TGV<=yB=w?{tUJIaSY^3zA_?XSRZ2^sVjfLm*8B z4*ooL7ldR~Ij*#%6t;lpA(}VSv7Ns=FYg$J4R-ObcXtS1HdbD=Uo$L2xxCbzspe5{Es^#x@(3 zQlwbsWdn$4MzJ2Yc<5dDOxUFMjUs2m?66637E}PF)=25-Fbm6cWj{4;VEtwcyLQgs z!{H$gA==qxftC*G(#225#zBbrwJ$9@W={yv=Yz<;Uf)mO%VrH+!PwYX6L$;ntv4vo zKtgXkh?K4<`#^$RFZeWqksDdKnaTESeoI13c2*V?nBoYP450b&65JX6PzV(^A>Rr4jsn>oTqO5j>Sk#(FNpe%5ZW>tQLkp2mN8(iMN6%I$O)Y5%!PLWDEopoel!1WXYQ>fAMSg02AhiLB&n^_;++pBEs6LxeA39vJsPlbtJ8^KFwY0Nc59s{M5w2X{e&=PHvGF9Kz8zE5#r9X3sbBq}%+|}bnu7W`PudO+@ ztsUqZD9#-mTq_eVbnCkX)f`_5ur&Ymp`RlomT#{E_hP;=fj?a0<&DGP_JaKTPC`Q8 z@VS%!JQlYWs4OC0Q;%`haB_j1A(I6vmxm4r0DDxmwN@qh^{)T74A^E=T*J@G6ynOf zEwgkGGv*o;ghUe5$z*cztk6Xcj&H&(uCA^#nq{o8^lOXLq^kD?-%Pm$QPtp^5bZUf zL$5N8)fgNrF}Qw*TR`Fh;yX8oO7{%D0VasDPd!m85@gMsX62p3flw_QBhFgkw}WI` z#_+Pjk`icJr}m!d1Eu{5emkG?0dUuZbbJSnsn|=1%QRc5Or6FQapmRnpE!bMBRU@M z=ulq);z2qY;%j}!E}F8wD9}y%ZRCWM&LggxBZ02$tOP44sl&)!;2=FZK{P1T!y{1ntY8fySc z5j?pQ-VrDXfMQwO+RF&=t?+_P>-XQtg8`R-#Z9ykap{bO>xyXw^&mC$L zpU|L_TDC}_NX#_;c@Q_`EFNv#!7HC{EJ0Y14ObN_zp zZvye4>_3FCV|-UI6X>;}Gcc_xpP#Yz%pKfOEH1#v#Kgx>A*}p?(nEJBe(x%m6`{N) zeTZ2Eyj-DcR(cqb1I)LZ&!OpAJG=~V{yy!WW_Ow$0wl%6y0svK$@b%^n1gqD@0>df=#|{=zH=BBHHO8y(U~JLuP}E3;q;$? zwEOovMn^_UvOqzJm4V^s?da$&b_^!T4uZ=0_t9)y-(f5R y3^BCmmA&W_sbm7>!kyR1n_d5_uh!`G;KA=Vr_W^>JHTthC?$DyY__b$wf_duoh%Rl literal 0 HcmV?d00001 diff --git a/jetstream/data_replication/replication.md b/jetstream/data_replication/replication.md new file mode 100644 index 0000000..632f22f --- /dev/null +++ b/jetstream/data_replication/replication.md @@ -0,0 +1,169 @@ +## Data Replication + +Replication allow you to move data between streams in either a 1:1 mirror style or by multiplexing multiple source streams into a new stream. In future builds this will allow data to be replicated between accounts as well, ideal for sending data from a Leafnode into a central store. + +![](../../assets/images/replication.png) + +Here we have 2 main streams - _ORDERS_ and _RETURNS_ - these streams are clustered across 3 nodes. These Streams have short retention periods and are memory based. + +We create a _ARCHIVE_ stream that has 2 _sources_ set, the _ARCHIVE_ will pull data from the sources into itself. This stream has a very long retention period and is file based and replicated across 3 nodes. Additional messages can be added to the ARCHIVE by sending to it directly. + +Finally, we create a _REPORT_ stream mirrored from _ARCHIVE_ that is not clustered and retains data for a month. The _REPORT_ Stream does not listen for any incoming messages, it can only consume data from _ARCHIVE_. + +### Mirrors + +A *mirror* copies data from 1 other stream, as far as possible IDs and ordering will match exactly the source. A *mirror* does not listen on a subject for any data to be added. The Start Sequence and Start Time can be set, but no subject filter. A stream can only have 1 *mirror* and if it is a mirror it cannot also have any *source*. + +### Sources + +A *source* is a stream where data is copied from, one stream can have multiple sources and will read data in from them all. The stream will also listen for messages on it's own subject. We can therefore not maintain absolute ordering, but data from 1 single source will be in the correct order but mixed in with other streams. You might also find the timestamps of streams can be older and newer mixed in together as a result. + +A Stream with sources may also listen on subjects, but could have no listening subject. When using the `nats` CLI to create sourced streams use `--subjects` to supply subjects to listen on. + +A source can have start time or start sequence and can filter by a subject. + +### Configuration + +The ORDERS and RETURNS streams as normal, I will not show how to create them. + +```nohighlight +$ nats s report +Obtaining Stream stats + ++---------+---------+----------+-----------+----------+-------+------+---------+----------------------+ +| Stream | Storage | Template | Consumers | Messages | Bytes | Lost | Deleted | Cluster | ++---------+---------+----------+-----------+----------+-------+------+---------+----------------------+ +| ORDERS | Memory | | 0 | 0 | 0 B | 0 | 0 | n1-c2, n2-c2*, n3-c2 | +| RETURNS | Memory | | 0 | 0 | 0 B | 0 | 0 | n1-c2*, n2-c2, n3-c2 | ++---------+---------+----------+-----------+----------+-------+------+---------+----------------------+ +``` + +We now add the ARCHIVE: + +```nohighlight +$ nats s add ARCHIVE --source ORDERS --source RETURNS +? Storage backend file +? Retention Policy Limits +? Discard Policy Old +? Message count limit -1 +? Message size limit -1 +? Maximum message age limit -1 +? Maximum individual message size -1 +? Duplicate tracking time window 2m +? Number of replicas to store 3 +? ORDERS Source Start Sequence 0 +? ORDERS Source UTC Time Stamp (YYYY:MM:DD HH:MM:SS) +? ORDERS Source Filter source by subject +? RETURNS Source Start Sequence 0 +? RETURNS Source UTC Time Stamp (YYYY:MM:DD HH:MM:SS) +? RETURNS Source Filter source by subject +``` + +And we add the REPORT: + +```nohighlight +$ nats s add REPORT --mirror ARCHIVE +? Storage backend file +? Retention Policy Limits +? Discard Policy Old +? Message count limit -1 +? Message size limit -1 +? Maximum message age limit 1M +? Maximum individual message size -1 +? Duplicate tracking time window 2m +? Number of replicas to store 1 +? Mirror Start Sequence 0 +? Mirror Start Time (YYYY:MM:DD HH:MM:SS) +? Mirror subject filter +``` + +When configured we'll see some additional information in a `nats stream info` output: + +```nohighlight +$ nats stream info ARCHIVE +... +Source Information: + + Stream Name: ORDERS + Lag: 0 + Last Seen: 2m23s + + Stream Name: RETURNS + Lag: 0 + Last Seen: 2m15s +... + +$ nats stream info REPORT +... +Mirror Information: + + Stream Name: ARCHIVE + Lag: 0 + Last Seen: 2m35s +... +``` + +Here the `Lag` is how far behind we were reported as being last time we saw a message. + +We can confirm all our setup using a `nats stream report`: + +```nohighlight +$ nats s report ++-------------------------------------------------------------------------------------------------------------------+ +| Stream Report | ++---------+---------+----------+-------------+-----------+----------+-------+------+---------+----------------------+ +| Stream | Storage | Template | Replication | Consumers | Messages | Bytes | Lost | Deleted | Cluster | ++---------+---------+----------+-------------+-----------+----------+-------+------+---------+----------------------+ +| ARCHIVE | File | | Sourced | 1 | 0 | 0 B | 0 | 0 | n1-c2*, n2-c2, n3-c2 | +| ORDERS | Memory | | | 1 | 0 | 0 B | 0 | 0 | n1-c2, n2-c2*, n3-c2 | +| REPORT | File | | Mirror | 0 | 0 | 0 B | 0 | 0 | n1-c2* | +| RETURNS | Memory | | | 1 | 0 | 0 B | 0 | 0 | n1-c2, n2-c2, n3-c2* | ++---------+---------+----------+-------------+-----------+----------+-------+------+---------+----------------------+ + ++---------------------------------------------------------+ +| Replication Report | ++---------+--------+---------------+--------+-----+-------+ +| Stream | Kind | Source Stream | Active | Lag | Error | ++---------+--------+---------------+--------+-----+-------+ +| ARCHIVE | Source | ORDERS | never | 0 | | +| ARCHIVE | Source | RETURNS | never | 0 | | +| REPORT | Mirror | ARCHIVE | never | 0 | | ++---------+--------+---------------+--------+-----+-------+ +``` + +We then create some data in both ORDERS and RETURNS: + +```nohighlight +$ nats req ORDERS.new "ORDER {{Count}}" --count 100 +$ nats req RETURNS.new "RETURN {{Count}}" --count 100 +``` + +We can now see from a Stream Report that the data has been replicated: + +```nohighlight +$ nats s report --dot replication.dot +Obtaining Stream stats + ++---------+---------+----------+-----------+----------+---------+------+---------+----------------------+ +| Stream | Storage | Template | Consumers | Messages | Bytes | Lost | Deleted | Cluster | ++---------+---------+----------+-----------+----------+---------+------+---------+----------------------+ +| ORDERS | Memory | | 1 | 100 | 3.3 KiB | 0 | 0 | n1-c2, n2-c2*, n3-c2 | +| RETURNS | Memory | | 1 | 100 | 3.5 KiB | 0 | 0 | n1-c2*, n2-c2, n3-c2 | +| ARCHIVE | File | | 1 | 200 | 27 KiB | 0 | 0 | n1-c2, n2-c2, n3-c2* | +| REPORT | File | | 0 | 200 | 27 KiB | 0 | 0 | n1-c2* | ++---------+---------+----------+-----------+----------+---------+------+---------+----------------------+ + ++---------------------------------------------------------+ +| Replication Report | ++---------+--------+---------------+--------+-----+-------+ +| Stream | Kind | Source Stream | Active | Lag | Error | ++---------+--------+---------------+--------+-----+-------+ +| ARCHIVE | Source | ORDERS | 14.48s | 0 | | +| ARCHIVE | Source | RETURNS | 9.83s | 0 | | +| REPORT | Mirror | ARCHIVE | 9.82s | 0 | | ++---------+--------+---------------+--------+-----+-------+ +``` + +Here we also pass the `--dot replication.dot` argument that writes a GraphViz format map of the replication setup. + +![](../../assets/images/replication-setup.png) \ No newline at end of file From 615153208434ba2ae0dcf78e219b87029ee79eb7 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Sun, 28 Feb 2021 14:44:14 -0700 Subject: [PATCH 63/84] add replication Signed-off-by: Colin Sullivan --- SUMMARY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/SUMMARY.md b/SUMMARY.md index 45e3ab7..1e59b38 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -122,6 +122,7 @@ * [Streams](jetstream/administration/streams.md) * [Consumers](jetstream/administration/consumers.md) * [Monitoring](jetstream/monitoring/monitoring.md) +* [Data Replication](jetstream/data_replication/replication.md) * [Clustering](jetstream/clustering/clustering.md) * [Administration](jetstream/clustering/administration.md) * [Configuration Management](jetstream/configuration_mgmt/configuration_mgmt.md) From 19de4f5841dff5dd003aa9ef1b877d306fcfaaa8 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Sun, 28 Feb 2021 16:58:09 -0700 Subject: [PATCH 64/84] use natsio/nats-box Signed-off-by: Colin Sullivan --- jetstream/getting_started/using_docker.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/jetstream/getting_started/using_docker.md b/jetstream/getting_started/using_docker.md index f46a7fa..f36feef 100644 --- a/jetstream/getting_started/using_docker.md +++ b/jetstream/getting_started/using_docker.md @@ -1,6 +1,6 @@ # Using Docker -The `synadia/nats-box:latest` docker image contains the `nats` utility this guide covers. +The `natsio/nats-box:latest` docker image contains the `nats` utility this guide covers. In one window start a JetStream enabled nats server: @@ -11,7 +11,7 @@ $ docker run --network host -p 4222:4222 nats -js And in another log into the utilities: ``` -$ docker run -ti --network host synadia/nats-box +$ docker run -ti --network host natsio/nats-box ``` This shell has the `nats` utility and all other NATS cli tools used in the rest of this guide. From 6d4e66410a2577fe7473542e866ef0c0f73d1de8 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 1 Mar 2021 09:31:13 -0600 Subject: [PATCH 65/84] Grammar/syntax updates --- jetstream/data_replication/replication.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/jetstream/data_replication/replication.md b/jetstream/data_replication/replication.md index 632f22f..7561645 100644 --- a/jetstream/data_replication/replication.md +++ b/jetstream/data_replication/replication.md @@ -1,6 +1,6 @@ ## Data Replication -Replication allow you to move data between streams in either a 1:1 mirror style or by multiplexing multiple source streams into a new stream. In future builds this will allow data to be replicated between accounts as well, ideal for sending data from a Leafnode into a central store. +Replication allows you to move data between streams in either a 1:1 mirror style or by multiplexing multiple source streams into a new stream. In future builds this will allow data to be replicated between accounts as well, ideal for sending data from a Leafnode into a central store. ![](../../assets/images/replication.png) @@ -20,7 +20,7 @@ A *source* is a stream where data is copied from, one stream can have multiple s A Stream with sources may also listen on subjects, but could have no listening subject. When using the `nats` CLI to create sourced streams use `--subjects` to supply subjects to listen on. -A source can have start time or start sequence and can filter by a subject. +A source can have Start Time or Start Sequence and can filter by a subject. ### Configuration @@ -166,4 +166,4 @@ Obtaining Stream stats Here we also pass the `--dot replication.dot` argument that writes a GraphViz format map of the replication setup. -![](../../assets/images/replication-setup.png) \ No newline at end of file +![](../../assets/images/replication-setup.png) From a3f6f9c7aacf4551d5f4c96d3012688100aa12d3 Mon Sep 17 00:00:00 2001 From: Ivan Kozlovic Date: Mon, 1 Mar 2021 11:02:42 -0700 Subject: [PATCH 66/84] Updates following NATS Streaming server release v0.21.0 Signed-off-by: Ivan Kozlovic --- nats-streaming-concepts/monitoring/endpoints.md | 12 ++++++++++-- nats-streaming-server/configuring/cfgfile.md | 3 +++ 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/nats-streaming-concepts/monitoring/endpoints.md b/nats-streaming-concepts/monitoring/endpoints.md index 9ff9684..31f06d4 100644 --- a/nats-streaming-concepts/monitoring/endpoints.md +++ b/nats-streaming-concepts/monitoring/endpoints.md @@ -98,7 +98,8 @@ You can control these via URL arguments \(limit and offset\). For example: [http "clients": [ { "id": "benchmark-sub-0", - "hb_inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayQK" + "hb_inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayQK", + "subs_count": 1 } ] } @@ -119,6 +120,7 @@ You can also report detailed subscription information on a per client basis usin { "id": "benchmark-sub-0", "hb_inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayQK", + "subs_count": 1, "subscriptions": { "foo": [ { @@ -146,6 +148,7 @@ You can select a specific client based on its client ID with `client=`, and { "id": "me", "hb_inbox": "_INBOX.HG0uDuNtAPxJQ1lVjIC2sr", + "subs_count": 1, "subscriptions": { "foo": [ { @@ -222,6 +225,7 @@ You can also get the list of subscriptions with `subs=1`. For example: [http://l "bytes": 0, "first_seq": 0, "last_seq": 0, + "subs_count": 1, "subscriptions": [ { "client_id": "me", @@ -249,7 +253,8 @@ You can select a specific channel based on its name with `channel=name`. For exa "msgs": 649234, "bytes": 97368590, "first_seq": 1, - "last_seq": 649234 + "last_seq": 649234, + "subs_count": 1 } ``` @@ -262,6 +267,7 @@ And again, you can get detailed subscriptions with `subs=1`. For example: [http: "bytes": 105698990, "first_seq": 1, "last_seq": 704770, + "subs_count": 10, "subscriptions": [ { "client_id": "me", @@ -301,6 +307,7 @@ For durables that are currently running, the `is_offline` field is set to `false "bytes": 0, "first_seq": 0, "last_seq": 0, + "subs_count": 1, "subscriptions": [ { "client_id": "me", @@ -328,6 +335,7 @@ When that same durable goes offline, `is_offline` is be set to `true`. Although "bytes": 0, "first_seq": 0, "last_seq": 0, + "subs_count": 1, "subscriptions": [ { "client_id": "me", diff --git a/nats-streaming-server/configuring/cfgfile.md b/nats-streaming-server/configuring/cfgfile.md index 746ba28..3648f35 100644 --- a/nats-streaming-server/configuring/cfgfile.md +++ b/nats-streaming-server/configuring/cfgfile.md @@ -197,6 +197,9 @@ For a given channel, the possible parameters are: | raft\_commit\_timeout | Specifies the time without an Apply\(\) operation before sending an heartbeat to ensure timely commit. Due to random staggering, may be delayed as much as 2x this value | Duration | `raft_commit_timeout: "100ms"` | `100ms` | v0.11.2 | | proceed\_on\_restore\_failure | Allow a non leader node to proceed with restore failures, do not use unless you understand the risks! | `true` or `false` | `proceed_on_restore_failure: true` | `false` | v0.17.0 | | allow_add_remove_node| Enable the ability to send NATS requests to the leader to add/remove cluster nodes | `true` or `false` | `allow_add_remove_node: true` | `false` | v0.19.0 | +| bolt_free_list_sync | Causes the RAFT log to synchronize the free list on write operations. Reduces performance at runtime, but makes the recovery faster | `true` or `false` | `bolt_free_list_sync: true` | `false` | v0.21.0 | +| bolt_free_list_map | Sets the backend freelist type to use a map instead of the default array type. Improves performance for large RAFT logs, with fragmented free list | `true` or `false` | `bolt_free_list_map: true` | `false` | v0.21.0 | +| nodes_connections | Enable creation of dedicated NATS connections to communicate with other nodes | `true` or `false` | `nodes_connections: true` | `false` | v0.21.0 | ## SQL Options Configuration From bc3b2b9cd7b09577f9f21846ed910c3f03f0d820 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Tue, 2 Mar 2021 21:39:26 -0700 Subject: [PATCH 67/84] add jsz http endpoint Signed-off-by: Colin Sullivan --- nats-server/configuration/monitoring.md | 55 +++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/nats-server/configuration/monitoring.md b/nats-server/configuration/monitoring.md index 0416bad..15ec707 100644 --- a/nats-server/configuration/monitoring.md +++ b/nats-server/configuration/monitoring.md @@ -246,6 +246,61 @@ You can also report detailed subscription information on a per connection basis } ``` +### JetStream Information + +The `/jsz` endpoint reports information about the JetStream subsystem. + +**Endpoint:** `http://server:port/jsm` + +| Result | Return Code | +| :--- | :--- | +| Success | 200 \(OK\) | +| Error | 400 \(Bad Request\) | + +#### Arguments + +| Argument | Values | Description | +| :--- | :--- | :--- | +| acc | account name | Provide information for a specfic account | +| accounts | true, 1, false, 0 | Provide information for all accounts. The default is false. | +| streams | true, 1, false, 0 | Include stream information. The default is false. | +| consumers | true, 1, false, 0 | Include consumer information. The default is false. | +| config | true, false | Include configuration with streams and consumers. The default is false. | +| offset | integer > 0 | Pagination offset. Default is 0. | +| limit | integer > 0 | Number of results to return. Default is 1024. | +| leader-only | true, false | TODO | + +As noted above, the `routez` endpoint does support the `subs` argument from the `/connz` endpoint. For example: [http://demo.nats.io:8222/routez?subs=1](http://demo.nats.io:8222/jsz?accounts=1&streams=1&consumers=1&config=1) + +#### Example + +* Get Jetstream information: http://host:port/jsz?accounts=1&streams=1&consumers=1&config=1 + +#### Response + +```javascript +{ + "server_id": "NACDVKFBUW4C4XA24OOT6L4MDP56MW76J5RJDFXG7HLABSB46DCMWCOW", + "now": "2019-06-24T14:29:16.046656-07:00", + "num_routes": 1, + "routes": [ + { + "rid": 1, + "remote_id": "de475c0041418afc799bccf0fdd61b47", + "did_solicit": true, + "ip": "127.0.0.1", + "port": 61791, + "pending_size": 0, + "in_msgs": 0, + "out_msgs": 0, + "in_bytes": 0, + "out_bytes": 0, + "subscriptions": 0 + } + ] +} +``` + ### Route Information The `/routez` endpoint reports information on active routes for a cluster. Routes are expected to be low, so there is no paging mechanism with this endpoint. From 042264f33b07206dfae7a1b778a40d762ac103de Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Wed, 3 Mar 2021 09:30:05 -0600 Subject: [PATCH 68/84] Grammar/syntax review --- nats-server/configuration/monitoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-server/configuration/monitoring.md b/nats-server/configuration/monitoring.md index 15ec707..a93734c 100644 --- a/nats-server/configuration/monitoring.md +++ b/nats-server/configuration/monitoring.md @@ -274,7 +274,7 @@ As noted above, the `routez` endpoint does support the `subs` argument from the #### Example -* Get Jetstream information: http://host:port/jsz?accounts=1&streams=1&consumers=1&config=1 +* Get JetStream information: http://host:port/jsz?accounts=1&streams=1&consumers=1&config=1 #### Response From 71172b43166c4341ef3084312efdd997c4bcfe47 Mon Sep 17 00:00:00 2001 From: Ivan Kozlovic Date: Fri, 5 Mar 2021 09:06:35 -0700 Subject: [PATCH 69/84] Fix typo Co-authored-by: Colin Sullivan --- nats-server/configuration/mqtt/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-server/configuration/mqtt/README.md b/nats-server/configuration/mqtt/README.md index 501cb72..29f21a8 100644 --- a/nats-server/configuration/mqtt/README.md +++ b/nats-server/configuration/mqtt/README.md @@ -61,7 +61,7 @@ allowed only for subscriptions, not for published messages. | `#` | `>` | | `+` | `*` | -The wilcard `#` matches any number of levels within a topic, which means that a subscription +The wildcard `#` matches any number of levels within a topic, which means that a subscription on `foo/#` would receive messages on `foo/bar`, or `foo/bar/baz`, but also on `foo`. This is not the case in NATS where a subscription on `foo.>` can receive messages on `foo/bar` or `foo/bar/baz`, but not on `foo`. To solve this, NATS Server will create two subscriptions, From 0f1d9e01a810564da9b282f45815710bdf89cfd8 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 14:08:37 +0000 Subject: [PATCH 70/84] GitBook: [master] 60 pages and 12 assets modified --- .../assets}/consumer-groups.png | Bin .../images => .gitbook/assets}/meta-group.png | Bin .../assets}/replication-setup.png | Bin .../assets}/replication.png | Bin .../assets}/stream-groups.png | Bin .../assets}/streams-and-consumers-75p.png | Bin SUMMARY.md | 75 ++++---- developing-with-nats-streaming/connecting.md | 2 +- faq.md | 4 +- jetstream/about_jetstream/jetstream.md | 31 ---- .../{administration.md => README.md} | 5 +- jetstream/administration/account.md | 64 +++---- jetstream/administration/consumers.md | 49 +++--- jetstream/administration/streams.md | 55 +++--- jetstream/clustering/README.md | 111 ++++++++++++ jetstream/clustering/administration.md | 31 ++-- jetstream/clustering/clustering.md | 106 ----------- jetstream/concepts/README.md | 20 +++ jetstream/concepts/concepts.md | 19 -- jetstream/concepts/configuration.md | 7 +- jetstream/concepts/consumers.md | 41 ++--- jetstream/concepts/streams.md | 39 ++-- .../{configuration_mgmt.md => README.md} | 38 ++-- .../configuration_mgmt/github_actions.md | 3 +- .../kubernetes_controller.md | 7 +- .../configuration_mgmt/nats-admin-cli.md | 2 + jetstream/configuration_mgmt/terraform.md | 2 + .../disaster_recovery.md | 39 ++-- jetstream/getting_started/README.md | 27 +++ jetstream/getting_started/getting_started.md | 26 --- .../getting_started/using-docker-with-ngs.md | 2 + jetstream/getting_started/using_docker.md | 14 +- jetstream/getting_started/using_source.md | 13 +- jetstream/jetstream.md | 35 ++++ .../{model_deep_dive.md => README.md} | 166 +++++++++--------- jetstream/model_deep_dive/ack-sampling.md | 2 + .../model_deep_dive/acknowledgment-models.md | 2 + .../model_deep_dive/consumer-message-rates.md | 2 + .../consumer-starting-position.md | 2 + .../model_deep_dive/ephemeral-consumers.md | 2 + .../model_deep_dive/exactly-once-delivery.md | 2 + .../model_deep_dive/message-deduplication.md | 2 + jetstream/model_deep_dive/storage-overhead.md | 2 + ...mits-retention-modes-and-discard-policy.md | 2 + jetstream/model_deep_dive/stream-templates.md | 2 + jetstream/monitoring.md | 35 ++++ jetstream/monitoring/monitoring.md | 34 ---- .../nats_api_reference.md | 117 ++++++------ .../{data_replication => }/replication.md | 41 ++--- .../resource_management.md | 27 +-- nats-on-kubernetes/nats-external-nlb.md | 12 +- nats-on-kubernetes/stan-ft-k8s-aws.md | 1 + nats-server/configuration/monitoring.md | 2 +- nats-server/nats_admin/lame_duck_mode.md | 16 +- nats-server/nats_admin/upgrading_cluster.md | 2 +- .../channels/subscriptions/queue-group.md | 2 +- nats-streaming-server/configuring/cfgfile.md | 18 +- nats-tools/nas/dir_store.md | 2 +- nats-tools/natscli.md | 13 +- 59 files changed, 714 insertions(+), 661 deletions(-) rename {assets/images => .gitbook/assets}/consumer-groups.png (100%) rename {assets/images => .gitbook/assets}/meta-group.png (100%) rename {assets/images => .gitbook/assets}/replication-setup.png (100%) rename {assets/images => .gitbook/assets}/replication.png (100%) rename {assets/images => .gitbook/assets}/stream-groups.png (100%) rename {assets/images => .gitbook/assets}/streams-and-consumers-75p.png (100%) delete mode 100644 jetstream/about_jetstream/jetstream.md rename jetstream/administration/{administration.md => README.md} (96%) create mode 100644 jetstream/clustering/README.md delete mode 100644 jetstream/clustering/clustering.md create mode 100644 jetstream/concepts/README.md delete mode 100644 jetstream/concepts/concepts.md rename jetstream/configuration_mgmt/{configuration_mgmt.md => README.md} (87%) create mode 100644 jetstream/configuration_mgmt/nats-admin-cli.md create mode 100644 jetstream/configuration_mgmt/terraform.md rename jetstream/{disaster_recovery => }/disaster_recovery.md (76%) create mode 100644 jetstream/getting_started/README.md delete mode 100644 jetstream/getting_started/getting_started.md create mode 100644 jetstream/getting_started/using-docker-with-ngs.md create mode 100644 jetstream/jetstream.md rename jetstream/model_deep_dive/{model_deep_dive.md => README.md} (77%) create mode 100644 jetstream/model_deep_dive/ack-sampling.md create mode 100644 jetstream/model_deep_dive/acknowledgment-models.md create mode 100644 jetstream/model_deep_dive/consumer-message-rates.md create mode 100644 jetstream/model_deep_dive/consumer-starting-position.md create mode 100644 jetstream/model_deep_dive/ephemeral-consumers.md create mode 100644 jetstream/model_deep_dive/exactly-once-delivery.md create mode 100644 jetstream/model_deep_dive/message-deduplication.md create mode 100644 jetstream/model_deep_dive/storage-overhead.md create mode 100644 jetstream/model_deep_dive/stream-limits-retention-modes-and-discard-policy.md create mode 100644 jetstream/model_deep_dive/stream-templates.md create mode 100644 jetstream/monitoring.md delete mode 100644 jetstream/monitoring/monitoring.md rename jetstream/{nats_api_reference => }/nats_api_reference.md (61%) rename jetstream/{data_replication => }/replication.md (85%) rename jetstream/{multi-tenancy => }/resource_management.md (93%) diff --git a/assets/images/consumer-groups.png b/.gitbook/assets/consumer-groups.png similarity index 100% rename from assets/images/consumer-groups.png rename to .gitbook/assets/consumer-groups.png diff --git a/assets/images/meta-group.png b/.gitbook/assets/meta-group.png similarity index 100% rename from assets/images/meta-group.png rename to .gitbook/assets/meta-group.png diff --git a/assets/images/replication-setup.png b/.gitbook/assets/replication-setup.png similarity index 100% rename from assets/images/replication-setup.png rename to .gitbook/assets/replication-setup.png diff --git a/assets/images/replication.png b/.gitbook/assets/replication.png similarity index 100% rename from assets/images/replication.png rename to .gitbook/assets/replication.png diff --git a/assets/images/stream-groups.png b/.gitbook/assets/stream-groups.png similarity index 100% rename from assets/images/stream-groups.png rename to .gitbook/assets/stream-groups.png diff --git a/assets/images/streams-and-consumers-75p.png b/.gitbook/assets/streams-and-consumers-75p.png similarity index 100% rename from assets/images/streams-and-consumers-75p.png rename to .gitbook/assets/streams-and-consumers-75p.png diff --git a/SUMMARY.md b/SUMMARY.md index eebc7a2..8b84a30 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -107,45 +107,45 @@ * [Tutorial](nats-server/nats_docker/nats-docker-tutorial.md) * [Docker Swarm](nats-server/nats_docker/docker_swarm.md) * [Python and NGS Running in Docker](nats-server/nats_docker/ngs-docker-python.md) - + ## JetStream -* [About Jetstream](jetstream/about_jetstream/jetstream.md) -* [Concepts](jetstream/concepts/concepts.md) - * [Streams](jetstream/concepts/streams.md) - * [Consumes](jetstream/concepts/consumers.md) - * [Configuration](jetstream/concepts/configuration.md) -* [Getting Started](jetstream/getting_started/getting_started.md) - * [Using Docker](jetstream/getting_started/using_docker.md) - * [Using Docker with NGS](jetstream/getting_started/using_docker.md#using-docker-with-ngs) - * [Using Source](jetstream/getting_started/using_source.md) -* [Administration & Usage from CLI](jetstream/administration/administration.md) - * [Account Information](jetstream/administration/account.md) - * [Streams](jetstream/administration/streams.md) - * [Consumers](jetstream/administration/consumers.md) -* [Monitoring](jetstream/monitoring/monitoring.md) -* [Data Replication](jetstream/data_replication/replication.md) -* [Clustering](jetstream/clustering/clustering.md) - * [Administration](jetstream/clustering/administration.md) -* [Configuration Management](jetstream/configuration_mgmt/configuration_mgmt.md) - * [NATS Admin CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) - * [Terraform](jetstream/configuration_mgmt/configuration_mgmt.md#terraform) - * [GitHub Actions](jetstream/configuration_mgmt/github_actions.md) - * [Kubernetes Controller](jetstream/configuration_mgmt/kubernetes_controller.md) -* [Disaser Recovery](jetstream/disaster_recovery/disaster_recovery.md) -* [Model Deep Dive](jetstream/model_deep_dive/model_deep_dive.md) - * [Stream Limits, Retention Modes and Discard Policy](jetstream/model_deep_dive/model_deep_dive.md#stream-limits-retention-modes-and-discard-policy) - * [Message Deduplication](jetstream/model_deep_dive/model_deep_dive.md#message-deduplication) - * [Acknowledgment Models](jetstream/model_deep_dive/model_deep_dive.md#acknowledgement-models) - * [Exactly Once Delivery](jetstream/model_deep_dive/model_deep_dive.md#exactly-once-delivery) - * [Consumer Starting Position](jetstream/model_deep_dive/model_deep_dive.md#consumer-starting-position) - * [Ephemeral Consumers](jetstream/model_deep_dive/model_deep_dive.md#ephemeral-consumers) - * [Consumer Message Rates](jetstream/model_deep_dive/model_deep_dive.md#consumer-message-rates) - * [Stream Templates](jetstream/model_deep_dive/model_deep_dive.md#stream-templates) - * [Ack Sampling](jetstream/model_deep_dive/model_deep_dive.md#ack-sampling) - * [Storage Overhead](jetstream/model_deep_dive/model_deep_dive.md#storage-overhead) -* [NATS API Reference](jetstream/nats_api_reference/nats_api_reference.md) -* [Multi-tenancy & Resource Mgmt](jetstream/multi-tenancy/resource_management.md) +* [About Jetstream](jetstream/jetstream.md) +* [Concepts](jetstream/concepts/README.md) + * [Streams](jetstream/concepts/streams.md) + * [Consumes](jetstream/concepts/consumers.md) + * [Configuration](jetstream/concepts/configuration.md) +* [Getting Started](jetstream/getting_started/README.md) + * [Using Docker](jetstream/getting_started/using_docker.md) + * [Using Docker with NGS](jetstream/getting_started/using-docker-with-ngs.md) + * [Using Source](jetstream/getting_started/using_source.md) +* [Administration & Usage from CLI](jetstream/administration/README.md) + * [Account Information](jetstream/administration/account.md) + * [Streams](jetstream/administration/streams.md) + * [Consumers](jetstream/administration/consumers.md) +* [Monitoring](jetstream/monitoring.md) +* [Data Replication](jetstream/replication.md) +* [Clustering](jetstream/clustering/README.md) + * [Administration](jetstream/clustering/administration.md) +* [Configuration Management](jetstream/configuration_mgmt/README.md) + * [NATS Admin CLI](jetstream/configuration_mgmt/nats-admin-cli.md) + * [Terraform](jetstream/configuration_mgmt/terraform.md) + * [GitHub Actions](jetstream/configuration_mgmt/github_actions.md) + * [Kubernetes Controller](jetstream/configuration_mgmt/kubernetes_controller.md) +* [Disaser Recovery](jetstream/disaster_recovery.md) +* [Model Deep Dive](jetstream/model_deep_dive/README.md) + * [Stream Limits, Retention Modes and Discard Policy](jetstream/model_deep_dive/stream-limits-retention-modes-and-discard-policy.md) + * [Message Deduplication](jetstream/model_deep_dive/message-deduplication.md) + * [Acknowledgment Models](jetstream/model_deep_dive/acknowledgment-models.md) + * [Exactly Once Delivery](jetstream/model_deep_dive/exactly-once-delivery.md) + * [Consumer Starting Position](jetstream/model_deep_dive/consumer-starting-position.md) + * [Ephemeral Consumers](jetstream/model_deep_dive/ephemeral-consumers.md) + * [Consumer Message Rates](jetstream/model_deep_dive/consumer-message-rates.md) + * [Stream Templates](jetstream/model_deep_dive/stream-templates.md) + * [Ack Sampling](jetstream/model_deep_dive/ack-sampling.md) + * [Storage Overhead](jetstream/model_deep_dive/storage-overhead.md) +* [NATS API Reference](jetstream/nats_api_reference.md) +* [Multi-tenancy & Resource Mgmt](jetstream/resource_management.md) ## NATS Tools @@ -244,3 +244,4 @@ * [Using a Load Balancer for External Access to NATS](nats-on-kubernetes/nats-external-nlb.md) * [Creating a NATS Super Cluster in Digital Ocean with Helm](nats-on-kubernetes/super-cluster-on-digital-ocean.md) * [From Zero to K8S to Leafnodes using Helm](nats-on-kubernetes/from-zero-to-leafnodes.md) + diff --git a/developing-with-nats-streaming/connecting.md b/developing-with-nats-streaming/connecting.md index 638ce03..e1920f3 100644 --- a/developing-with-nats-streaming/connecting.md +++ b/developing-with-nats-streaming/connecting.md @@ -6,7 +6,7 @@ NATS Streaming is a service on top of NATS. To connect to the service you first Connecting to a streaming server requires a cluster id, defined by the server configuration, and a client ID defined by the client. -_Client ID should contain only alphanumeric characters, `-` or `_`_ +_Client ID should contain only alphanumeric characters, `-` or \`_\`\_ Connecting to a server running locally on the default port is as simple as this: diff --git a/faq.md b/faq.md index 11c6ae4..f4e52a6 100644 --- a/faq.md +++ b/faq.md @@ -159,9 +159,9 @@ NATS 2.0 is completely backwards compatible with NATS < 2.x configure files a The default setting for a single server is 65,536. Although there is no specified limit to the number of connections supported by NATS, there are some environmental factors that will influence your decision as to how many connections to allow per server. -Most systems can handle several thousand NATS connections per server without any changes although some have a very low default such as OS X. You'll want to look at kernel/OS settings to increase that limit. You'll also want to look at default TCP buffer sizes to best optimize your machine for your traffic characteristics. +Most systems can handle several thousand NATS connections per server without any changes although some have a very low default such as OS X. You'll want to look at kernel/OS settings to increase that limit. You'll also want to look at default TCP buffer sizes to best optimize your machine for your traffic characteristics. -If you are using TLS you'll want to be sure the hardware can handle the CPU load created by TLS negotiation when there is the thundering herd of inbound connections after an outage or network partition event. This often overlooked factor is usually the constraint limiting the number of connections a single server should support. Choosing a cipher suite that is supported by TLS acceleration can mitigate this (e.g. AES with x86). Thinking of the entire system, you'll also want to look at a range of reconnect delay times or add reconnect jitter to the NATS clients to even out the distribution of connection attempts over time and reduce CPU spikes. +If you are using TLS you'll want to be sure the hardware can handle the CPU load created by TLS negotiation when there is the thundering herd of inbound connections after an outage or network partition event. This often overlooked factor is usually the constraint limiting the number of connections a single server should support. Choosing a cipher suite that is supported by TLS acceleration can mitigate this \(e.g. AES with x86\). Thinking of the entire system, you'll also want to look at a range of reconnect delay times or add reconnect jitter to the NATS clients to even out the distribution of connection attempts over time and reduce CPU spikes. All said, each server can be tuned to handle a large number of clients, and given the flexibility and scalability of NATS with clusters, superclusters, and leaf nodes one can build a NATS deployment supporting many millions of connections. diff --git a/jetstream/about_jetstream/jetstream.md b/jetstream/about_jetstream/jetstream.md deleted file mode 100644 index dc87dcf..0000000 --- a/jetstream/about_jetstream/jetstream.md +++ /dev/null @@ -1,31 +0,0 @@ -# JetStream - -JetStream was created to solve the problems identified with streaming in technology today - complexity, fragility, and a lack of scalability. Some technologies address these better than others, but no current streaming technology is truly multi-tenant, horizontally scalable, and supports multiple deployment models. No technology we are aware of can scale from edge to cloud under the same security context while having complete deployment observability for operations. - -## Goals -JetStream was developed with the following goals in mind: - -- The system must be easy to configure and operate and be observable. -- The system must be secure and operate well with NATS 2.0 security models. -- The system must scale horizontally and be applicable to a high ingestion rate. -- The system must support multiple use cases. -- The system must self heal and always be available. -- The system must have an API that is closer to core NATS. -- The system must allow NATS messages to be part of a stream as desired. -- The system must display payload agnostic behavior. -- The system must not have third party dependencies. - -## High-Level Design and Features -In terms of deployment, a JetStream server is simply a NATS server with the JetStream subsystem enabled, launched with the `-js` flag with a configured server name and cluster name. From a client perspective, it does not matter which servers are running JetStream so long as there is some route to a JetStream enabled server or servers. This allows for a flexible deployment which to optimize resources for particular servers that will store streams versus very low overhead stateless servers, reducing OpEx and ultimately creating a scalable and manageable system. - -## Feature List -- At-least-once delivery; exactly once within a window -- Store messages and replay by time or sequence -- Wildcard support -- Account aware -- Data at rest encryption -- Cleanse specific messages (GDPR) -- Horizontal scalability -- Persist Streams and replay via Consumers - -JetStream is designed to bifurcate ingestion and consumption of messages to provide multiple ways to consume data from the same stream. To that end, JetStream functionality is composed of server streams and server consumers. diff --git a/jetstream/administration/administration.md b/jetstream/administration/README.md similarity index 96% rename from jetstream/administration/administration.md rename to jetstream/administration/README.md index 4585bbe..fa3f2c8 100644 --- a/jetstream/administration/administration.md +++ b/jetstream/administration/README.md @@ -1,8 +1,8 @@ -## Administration and Usage from the CLI +# Administration & Usage from CLI Once the server is running it's time to use the management tool. This can be downloaded from the [GitHub Release Page](https://github.com/nats-io/natscli/releases/) or you can use the `natsio/nats-box:latest` docker image. On OS X homebrew can be used to install the latest version: -```nohighlight +```text $ brew tap nats-io/nats-tools $ brew install nats-io/nats-tools/nats $ nats --help @@ -30,3 +30,4 @@ We'll walk through the above scenario and introduce features of the CLI and of J Throughout this example, we'll show other commands like `nats pub` and `nats sub` to interact with the system. These are normal existing core NATS commands and JetStream is fully usable by only using core NATS. We'll touch on some additional features but please review the section on the design model to understand all possible permutations. + diff --git a/jetstream/administration/account.md b/jetstream/administration/account.md index 7c3eecd..84b9583 100644 --- a/jetstream/administration/account.md +++ b/jetstream/administration/account.md @@ -1,8 +1,10 @@ -### Account Information +# Account Information + +## Account Information JetStream is multi-tenant so you will need to check that your account is enabled for JetStream and is not limited. You can view your limits as follows: -```nohighlight +```text $ nats account info Connection Information: Client ID: 8 @@ -20,13 +22,13 @@ JetStream Account Information: Max Consumers: unlimited ``` -### Streams +## Streams The first step is to set up storage for our `ORDERS` related messages, these arrive on a wildcard of subjects all flowing into the same Stream and they are kept for 1 year. -#### Creating +### Creating -```nohighlight +```text $ nats str add ORDERS ? Subjects to consume ORDERS.* ? Storage backend file @@ -64,31 +66,32 @@ Statistics: You can get prompted interactively for missing information as above, or do it all on one command. Pressing `?` in the CLI will help you map prompts to CLI options: +```text +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --replicas 3 ``` -$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --replicas 3``` Additionally, one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: -``` +```text $ nats str add ORDERS --config orders.json ``` -#### Listing +### Listing We can confirm our Stream was created: -```nohighlight +```text $ nats str ls Streams: - ORDERS + ORDERS ``` -#### Querying +### Querying Information about the configuration of the Stream can be seen, and if you did not specify the Stream like below, it will prompt you based on all known ones: -```nohighlight +```text $ nats str info ORDERS Information for Stream ORDERS @@ -114,7 +117,7 @@ Statistics: Most commands that show data as above support `-j` to show the results as JSON: -```nohighlight +```text $ nats str info ORDERS -j { "config": { @@ -144,7 +147,7 @@ This is the general pattern for the entire `nats` utility as it relates to JetSt In clustered mode additional information will be included: -```nohighlight +```text $ nats str info ORDERS ... Cluster Information: @@ -156,11 +159,11 @@ Cluster Information: Here the cluster name is configured as `JSC`, there is a server `S1` that's the current leader with `S3` and `S2` are replicas. Both replicas are current and have been seen recently. -#### Copying +### Copying A stream can be copied into another, which also allows the configuration of the new one to be adjusted via CLI flags: -```nohighlight +```text $ nats str cp ORDERS ARCHIVE --subjects "ORDERS_ARCVHIVE.*" --max-age 2y Stream ORDERS was created @@ -174,11 +177,11 @@ Configuration: ... ``` -#### Editing +### Editing -A stream configuration can be edited, which allows the configuration to be adjusted via CLI flags. Here I have a incorrectly created ORDERS stream that I fix: +A stream configuration can be edited, which allows the configuration to be adjusted via CLI flags. Here I have a incorrectly created ORDERS stream that I fix: -```nohighlight +```text $ nats str info ORDERS -j | jq .config.subjects [ "ORDERS.new" @@ -197,24 +200,24 @@ Configuration: Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: -``` +```text $ nats str edit ORDERS --config orders.json ``` -#### Publishing Into a Stream +### Publishing Into a Stream Now let's add in some messages to our Stream. You can use `nats pub` to add messages, pass the `--wait` flag to see the publish ack being returned. You can publish without waiting for acknowledgement: -```nohighlight +```text $ nats pub ORDERS.scratch hello Published [sub1] : 'hello' ``` But if you want to be sure your messages got to JetStream and were persisted you can make a request: -```nohighlight +```text $ nats req ORDERS.scratch hello 13:45:03 Sending request on [ORDERS.scratch] 13:45:03 Received on [_INBOX.M8drJkd8O5otORAo0sMNkg.scHnSafY]: '+OK' @@ -222,7 +225,7 @@ $ nats req ORDERS.scratch hello Keep checking the status of the Stream while doing this and you'll see it's stored messages increase. -```nohighlight +```text $ nats str info ORDERS Information for Stream ORDERS ... @@ -237,11 +240,11 @@ Statistics: After putting some throw away data into the Stream, we can purge all the data out - while keeping the Stream active: -#### Deleting All Data +### Deleting All Data To delete all data in a stream use `purge`: -```nohighlight +```text $ nats str purge ORDERS -f ... Statistics: @@ -253,19 +256,20 @@ Statistics: Active Consumers: 0 ``` -#### Deleting A Message +### Deleting A Message A single message can be securely removed from the stream: -```nohighlight +```text $ nats str rmm ORDERS 1 -f ``` -#### Deleting Sets +### Deleting Sets Finally for demonstration purposes, you can also delete the whole Stream and recreate it so then we're ready for creating the Consumers: -``` +```text $ nats str rm ORDERS -f $ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 ``` + diff --git a/jetstream/administration/consumers.md b/jetstream/administration/consumers.md index 9526c4d..da1c727 100644 --- a/jetstream/administration/consumers.md +++ b/jetstream/administration/consumers.md @@ -1,14 +1,14 @@ -### Consumers +# Consumers Consumers is how messages are read or consumed from the Stream. We support pull and push-based Consumers and the example scenario has both, lets walk through that. -#### Creating Pull-Based Consumers +## Creating Pull-Based Consumers -The `NEW` and `DISPATCH` Consumers are pull-based, meaning the services consuming data from them have to ask the system for the next available message. This means you can easily scale your services up by adding more workers and the messages will get spread across the workers based on their availability. +The `NEW` and `DISPATCH` Consumers are pull-based, meaning the services consuming data from them have to ask the system for the next available message. This means you can easily scale your services up by adding more workers and the messages will get spread across the workers based on their availability. Pull-based Consumers are created the same as push-based Consumers, just don't specify a delivery target. -``` +```text $ nats con ls ORDERS No Consumers defined ``` @@ -17,7 +17,7 @@ We have no Consumers, lets add the `NEW` one: I supply the `--sample` options on the CLI as this is not prompted for at present, everything else is prompted. The help in the CLI explains each: -``` +```text $ nats con add --sample 100 ? Select a Stream ORDERS ? Consumer name NEW @@ -48,29 +48,29 @@ State: Redelivered Messages: 0 ``` -This is a pull-based Consumer (empty Delivery Target), it gets messages from the first available message and requires specific acknowledgement of each and every message. +This is a pull-based Consumer \(empty Delivery Target\), it gets messages from the first available message and requires specific acknowledgement of each and every message. It only received messages that originally entered the Stream on `ORDERS.received`. Remember the Stream subscribes to `ORDERS.*`, this lets us select a subset of messages from the Stream. - + A Maximum Delivery limit of 20 is set, this means if the message is not acknowledged it will be retried but only up to this maximum total deliveries. Again this can all be done in a single CLI call, lets make the `DISPATCH` Consumer: -``` +```text $ nats con add ORDERS DISPATCH --filter ORDERS.processed --ack explicit --pull --deliver all --sample 100 --max-deliver 20 ``` Additionally, one can store the configuration in a JSON file, the format of this is the same as `$ nats con info ORDERS DISPATCH -j | jq .config`: -``` +```text $ nats con add ORDERS MONITOR --config monitor.json ``` -#### Creating Push-Based Consumers +## Creating Push-Based Consumers Our `MONITOR` Consumer is push-based, has no ack and will only get new messages and is not sampled: -``` +```text $ nats con add ? Select a Stream ORDERS ? Consumer name MONITOR @@ -101,21 +101,21 @@ State: Again you can do this with a single non interactive command: -``` +```text $ nats con add ORDERS MONITOR --ack none --target monitor.ORDERS --deliver last --replay instant --filter '' ``` Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats con info ORDERS MONITOR -j | jq .config`: -``` +```text $ nats con add ORDERS --config monitor.json ``` -#### Listing +## Listing You can get a quick list of all the Consumers for a specific Stream: -``` +```text $ nats con ls ORDERS Consumers for Stream ORDERS: @@ -124,11 +124,11 @@ Consumers for Stream ORDERS: NEW ``` -#### Querying +## Querying All details for an Consumer can be queried, lets first look at a pull-based Consumer: -``` +```text $ nats con info ORDERS DISPATCH Information for Consumer ORDERS > DISPATCH @@ -154,13 +154,13 @@ State: More details about the `State` section will be shown later when discussing the ack models in depth. -#### Consuming Pull-Based Consumers +## Consuming Pull-Based Consumers Pull-based Consumers require you to specifically ask for messages and ack them, typically you would do this with the client library `Request()` feature, but the `nats` utility has a helper: First we ensure we have a message: -``` +```text $ nats pub ORDERS.processed "order 1" $ nats pub ORDERS.processed "order 2" $ nats pub ORDERS.processed "order 3" @@ -168,7 +168,7 @@ $ nats pub ORDERS.processed "order 3" We can now read them using `nats`: -``` +```text $ nats con next ORDERS DISPATCH --- received on ORDERS.processed order 1 @@ -186,7 +186,7 @@ You can prevent ACKs by supplying `--no-ack`. To do this from code you'd send a `Request()` to `$JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH`: -``` +```text $ nats req '$JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH' '' Published [$JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH] : '' Received [ORDERS.processed] : 'order 3' @@ -194,11 +194,11 @@ Received [ORDERS.processed] : 'order 3' Here `nats req` cannot ack, but in your code you'd respond to the received message with a nil payload as an Ack to JetStream. -#### Consuming Push-Based Consumers +## Consuming Push-Based Consumers Push-based Consumers will publish messages to a subject and anyone who subscribes to the subject will get them, they support different Acknowledgement models covered later, but here on the `MONITOR` Consumer we have no Acknowledgement. -``` +```text $ nats con info ORDERS MONITOR ... Delivery Subject: monitor.ORDERS @@ -207,7 +207,7 @@ $ nats con info ORDERS MONITOR The Consumer is publishing to that subject, so lets listen there: -``` +```text $ nats sub monitor.ORDERS Listening on [monitor.ORDERS] [#3] Received on [ORDERS.processed]: 'order 3' @@ -217,3 +217,4 @@ Listening on [monitor.ORDERS] Note the subject here of the received message is reported as `ORDERS.processed` this helps you distinguish what you're seeing in a Stream covering a wildcard, or multiple subject, subject space. This Consumer needs no ack, so any new message into the ORDERS system will show up here in real time. + diff --git a/jetstream/administration/streams.md b/jetstream/administration/streams.md index 1fcaf4c..56ad55f 100644 --- a/jetstream/administration/streams.md +++ b/jetstream/administration/streams.md @@ -1,10 +1,10 @@ -### Streams +# Streams The first step is to set up storage for our `ORDERS` related messages, these arrive on a wildcard of subjects all flowing into the same Stream and they are kept for 1 year. -#### Creating +## Creating -```nohighlight +```text $ nats str add ORDERS ? Subjects to consume ORDERS.* ? Storage backend file @@ -41,32 +41,32 @@ Statistics: You can get prompted interactively for missing information as above, or do it all on one command. Pressing `?` in the CLI will help you map prompts to CLI options: -``` +```text nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --dupe-window="0s" --replicas 1 ``` Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: -``` +```text $ nats str add ORDERS --config orders.json ``` -#### Listing +## Listing We can confirm our Stream was created: -```nohighlight +```text $ nats str ls Streams: - ORDERS + ORDERS ``` -#### Querying +## Querying Information about the configuration of the Stream can be seen, and if you did not specify the Stream like below, it will prompt you based on all known ones: -```nohighlight +```text $ nats str info ORDERS Information for Stream ORDERS created 2021-02-27T16:49:36-07:00 @@ -95,7 +95,7 @@ State: Most commands that show data as above support `-j` to show the results as JSON: -```nohighlight +```text $ nats str info ORDERS -j { "config": { @@ -129,11 +129,11 @@ $ nats str info ORDERS -j This is the general pattern for the entire `nats` utility as it relates to JetStream - prompting for needed information but every action can be run non-interactively making it usable as a cli api. All information output like seen above can be turned into JSON using `-j`. -#### Copying +## Copying A stream can be copied into another, which also allows the configuration of the new one to be adjusted via CLI flags: -```nohighlight +```text $ nats str cp ORDERS ARCHIVE --subjects "ORDERS_ARCVHIVE.*" --max-age 2y Stream ORDERS was created @@ -162,11 +162,11 @@ State: Active Consumers: 0 ``` -#### Editing +## Editing -A stream configuration can be edited, which allows the configuration to be adjusted via CLI flags. Here I have a incorrectly created ORDERS stream that I fix: +A stream configuration can be edited, which allows the configuration to be adjusted via CLI flags. Here I have a incorrectly created ORDERS stream that I fix: -```nohighlight +```text $ nats str info ORDERS -j | jq .config.subjects [ "ORDERS.new" @@ -185,24 +185,24 @@ Configuration: Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: -``` +```text $ nats str edit ORDERS --config orders.json ``` -#### Publishing Into a Stream +## Publishing Into a Stream Now let's add in some messages to our Stream. You can use `nats pub` to add messages, pass the `--wait` flag to see the publish ack being returned. You can publish without waiting for acknowledgement: -```nohighlight +```text $ nats pub ORDERS.scratch hello Published [sub1] : 'hello' ``` But if you want to be sure your messages got to JetStream and were persisted you can make a request: -```nohighlight +```text $ nats req ORDERS.scratch hello 13:45:03 Sending request on [ORDERS.scratch] 13:45:03 Received on [_INBOX.M8drJkd8O5otORAo0sMNkg.scHnSafY]: '+OK' @@ -210,7 +210,7 @@ $ nats req ORDERS.scratch hello Keep checking the status of the Stream while doing this and you'll see it's stored messages increase. -```nohighlight +```text $ nats str info ORDERS Information for Stream ORDERS ... @@ -225,11 +225,11 @@ Statistics: After putting some throw away data into the Stream, we can purge all the data out - while keeping the Stream active: -#### Deleting All Data +## Deleting All Data To delete all data in a stream use `purge`: -```nohighlight +```text $ nats str purge ORDERS -f ... State: @@ -241,19 +241,20 @@ State: Active Consumers: 0 ``` -#### Deleting A Message +## Deleting A Message A single message can be securely removed from the stream: -```nohighlight +```text $ nats str rmm ORDERS 1 -f ``` -#### Deleting Sets +## Deleting Sets Finally for demonstration purposes, you can also delete the whole Stream and recreate it so then we're ready for creating the Consumers: -``` +```text $ nats str rm ORDERS -f $ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --dupe-window="0s" --replicas 1 ``` + diff --git a/jetstream/clustering/README.md b/jetstream/clustering/README.md new file mode 100644 index 0000000..d97925b --- /dev/null +++ b/jetstream/clustering/README.md @@ -0,0 +1,111 @@ +# Clustering + +Clustering in Jetstream is required for a highly available and scalable system. Behind clustering is RAFT. There's no need to understand RAFT in depth to use clustering, but knowing a little explains some of the requirements behind setting up Jetstream clusters. + +## RAFT + +JetStream uses a NATS optimized RAFT algorithm for clustering. Typically raft generates a lot of traffic, but the NATS server optimizes this by combining the data plane for replicating messages with the messages RAFT would normally use to ensure consensus. + +### Raft groups + +The RAFT groups include API handlers sstreams, consumers, and an internal algorithm designates which servers handle which streams and consumers. + +The raft algorithm has a few requirements: + +* A log to persist state +* A quorum for consensus. + +### The Quorum + +In order to ensure data consistency across complete restarts, a quorum of servers is required. A quorum is ½ cluster size + 1. This is the minimum number of nodes to ensure at least one node has the most recent data and state after a catastrophic failure. So for a cluster size of 3, you’ll need at least two Jetstream enabled NATS servers available to store new messages. For a cluster size of 5, you’ll need at least 3 NATS servers, and so forth. + +### RAFT Groups + +**Meta Group** - all servers join the Meta Group and the JetStream API is managed by this group. A leader is elected and this owns the API and takes care of server placement. + +![Meta Group](../../.gitbook/assets/meta-group.png) + +**Stream Group** - each Stream creates a RAFT group, this group synchronizes state and data between its members. The elected leader handles ACKs and so forth, if there is no leader the stream will not accept messages. + +![Stream Groups](../../.gitbook/assets/stream-groups.png) + +**Consumer Group** - each Consumer creates a RAFT group, this group synchronizes consumer state between its members. The group will live on the machines where the Stream Group is and handle consumption ACKs etc. Each Consumer will have its own group. + +![Consumer Groups](../../.gitbook/assets/consumer-groups.png) + +### Cluster Size + +Generally we recommend 3 or 5 Jetstream enabled servers in a NATS cluster. This balances scalability with a tolerance for failure. For example, if 5 servers are Jetstream enabled You would want two servers is one “zone”, two servers in another, and the remaining server in a third. This means you can lose any one “zone” at any time and continue operating. + +### Mixing Jetstream enabled servers with standard NATS servers + +This is possible, and even recommended in some cases. By mixing server types you can dedicate certain machines optimized for storage for Jetstream and others optimized solely for compute for standard NATS servers, reducing operational expense. With the right configuration, the standard servers would handle non-persistent NATS traffic and the Jetstream enabled servers would handle Jetstream traffic. + +## Configuration + +To configure Jetstream clusters, just configure clusters as you normally would by specifying a cluster block in the configuration. Any Jetstream enabled servers in the list of clusters will automatically chatter and set themselves up. Unlike core NATS clustering though, each Jetstream node **must specify** a server name and cluster name. + +Below are explicity listed server configuration for a three node cluster across three machines, `n1-c1`, `n2-c1`, and `n3-c1`. + +### Server 1 \(host\_a\) + +```text +server_name=n1-c1 +listen=4222 + +jetstream { + store_dir=/nats/storage +} + +cluster { + name: C1 + listen: localhost:6222 + routes: [ + nats-route://host_b:6222 + nats-route://host_c:6222 + ] +} +``` + +### Server 2 \(host\_b\) + +```text +server_name=n2-c1 +listen=4222 + +jetstream { + store_dir=/nats/storage +} + +cluster { + name: C1 + listen: localhost:6222 + routes: [ + nats-route://host_a:6222 + nats-route://host_c:6222 + ] +} +``` + +### Server 3 \(host\_c\) + +```text +server_name=n3-c1 +listen=4222 + +jetstream { + store_dir=/nats/storage +} + +cluster { + name: C1 + listen: localhost:6222 + routes: [ + nats-route://host_a:6222 + nats-route://host_b:6222 + ] +} +``` + +Add nodes as necessary. Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use Jetstream. + diff --git a/jetstream/clustering/administration.md b/jetstream/clustering/administration.md index ef213fd..c4d554f 100644 --- a/jetstream/clustering/administration.md +++ b/jetstream/clustering/administration.md @@ -1,6 +1,6 @@ -# Cluster Administration +# Administration -Once a JetStream cluster is operating interactions with the CLI and with `nats` CLI is the same as before. For these examples, lets assume we have a 5 server cluster, n1-n5 in a cluster named C1. +Once a JetStream cluster is operating interactions with the CLI and with `nats` CLI is the same as before. For these examples, lets assume we have a 5 server cluster, n1-n5 in a cluster named C1. ## Account Level @@ -8,9 +8,9 @@ Within an account there are operations and reports that show where users data is ## Creating clustered streams -When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. The replica count must be less than the maximum number of servers. +When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, \(we suggest 1, 3 or 5\), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. The replica count must be less than the maximum number of servers. -```nohighlight +```text $ nats str add ORDERS --replicas 3 .... Information for Stream ORDERS created 2021-02-05T12:07:34+01:00 @@ -18,14 +18,13 @@ Information for Stream ORDERS created 2021-02-05T12:07:34+01:00 Configuration: .... Replicas: 3 - + Cluster Information: Name: C1 Leader: n1-c1 Replica: n4-c1, current, seen 0.07s ago Replica: n3-c1, current, seen 0.07s ago - ``` Above you can see that the cluster information will be reported in all cases where Stream info is shown such as after add or using `nats stream info`. @@ -40,7 +39,7 @@ The replica count cannot be edited once configured. Users can get overall statistics about their streams and also where these streams are placed: -``` +```text $ nats stream report Obtaining Stream stats +----------+-----------+----------+--------+---------+------+---------+----------------------+----------+ @@ -57,11 +56,11 @@ Obtaining Stream stats #### Forcing Stream and Consumer leader election -Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process, but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time. +Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process, but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time. Moving leadership away from a node does not remove it from the cluster and does not prevent it from becoming a leader again, this is merely a triggered leader election. -```nohighlight +```text $ nats stream cluster step-down ORDERS 14:32:17 Requesting leader step down of "n1-c1" in a 3 peer RAFT group 14:32:18 New leader elected "n4-c1" @@ -86,7 +85,7 @@ Systems users can view state of the Meta Group - but not individual Stream or Co We have a high level report of cluster state: -```nohighlight +```text $ nats server report jetstream --user system +--------------------------------------------------------------------------------------------------+ | JetStream Summary | @@ -124,7 +123,7 @@ In the Meta Group report the server `n2-c1` is not current and has not been seen This report is built using raw data that can be obtained from the monitor port on the `/jsz` url, or over nats using: -```nohightlight +```text $ nats server req jetstream --help ... --name=NAME Limit to servers matching a server name @@ -147,7 +146,7 @@ This will produce a wealth of raw information about the current state of your cl Similar to Streams and Consumers above the Meta Group allows leader stand down. The Meta Group is cluster wide and spans all accounts, therefore to manage the meta group you have to use a `SYSTEM` user. -```nohighlight +```text $ nats server raft step-down --user system 17:44:24 Current leader: n2-c2 17:44:24 New leader: n1-c2 @@ -157,11 +156,11 @@ $ nats server raft step-down --user system Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down. -There might be a case though where you know a machine will never return, and you want to signal to JetStream that the machine will not return. This will remove it from the Stream in question and all it's Consumers. +There might be a case though where you know a machine will never return, and you want to signal to JetStream that the machine will not return. This will remove it from the Stream in question and all it's Consumers. -After the node is removed the cluster will notice that the replica count is not honored anymore and will immediately pick a new node and start replicating data to it. The new node will be selected using the same placement rules as the existing stream. +After the node is removed the cluster will notice that the replica count is not honored anymore and will immediately pick a new node and start replicating data to it. The new node will be selected using the same placement rules as the existing stream. -```nohighlight +```text $ nats s cluster peer-remove ORDERS ? Select a Peer n4-c1 14:38:50 Removing peer "n4-c1" @@ -170,7 +169,7 @@ $ nats s cluster peer-remove ORDERS At this point the stream and all consumers will have removed `n4-c1` from the group, they will all start new peer selection and data replication. -```nohighlight +```text $ nats stream info ORDERS .... Cluster Information: diff --git a/jetstream/clustering/clustering.md b/jetstream/clustering/clustering.md deleted file mode 100644 index 51c8d73..0000000 --- a/jetstream/clustering/clustering.md +++ /dev/null @@ -1,106 +0,0 @@ -# Clustering - -Clustering in Jetstream is required for a highly available and scalable system. Behind -clustering is RAFT. There's no need to understand RAFT in depth to use clustering, but knowing a little explains some of the requirements behind setting up Jetstream clusters. - -## RAFT -JetStream uses a NATS optimized RAFT algorithm for clustering. Typically raft generates a lot of traffic, but the NATS server optimizes this by combining the data plane for replicating messages with the messages RAFT would normally use to ensure consensus. - -### Raft groups -The RAFT groups include API handlers sstreams, consumers, and an internal algorithm designates which servers handle which streams and consumers. - -The raft algorithm has a few requirements: -- A log to persist state -- A quorum for consensus. - -### The Quorum -In order to ensure data consistency across complete restarts, a quorum of servers is required. A quorum is ½ cluster size + 1. This is the minimum number of nodes to ensure at least one node has the most recent data and state after a catastrophic failure. So for a cluster size of 3, you’ll need at least two Jetstream enabled NATS servers available to store new messages. For a cluster size of 5, you’ll need at least 3 NATS servers, and so forth. - -### RAFT Groups - -**Meta Group** - all servers join the Meta Group and the JetStream API is managed by this group. A leader is elected and this owns the API and takes care of server placement. - -![Meta Group](../../assets/images/meta-group.png) - -**Stream Group** - each Stream creates a RAFT group, this group synchronizes state and data between its members. The elected leader handles ACKs and so forth, if there is no leader the stream will not accept messages. - -![Stream Groups](../../assets/images/stream-groups.png) - -**Consumer Group** - each Consumer creates a RAFT group, this group synchronizes consumer state between its members. The group will live on the machines where the Stream Group is and handle consumption ACKs etc. Each Consumer will have its own group. - -![Consumer Groups](../../assets/images/consumer-groups.png) - -### Cluster Size -Generally we recommend 3 or 5 Jetstream enabled servers in a NATS cluster. This balances scalability with a tolerance for failure. For example, if 5 servers are Jetstream enabled You would want two servers is one “zone”, two servers in another, and the remaining server in a third. This means you can lose any one “zone” at any time and continue operating. - -### Mixing Jetstream enabled servers with standard NATS servers - -This is possible, and even recommended in some cases. By mixing server types you can dedicate certain machines optimized for storage for Jetstream and others optimized solely for compute for standard NATS servers, reducing operational expense. With the right configuration, the standard servers would handle non-persistent NATS traffic and the Jetstream enabled servers would handle Jetstream traffic. - -## Configuration - -To configure Jetstream clusters, just configure clusters as you normally would by specifying a cluster block in the configuration. Any Jetstream enabled servers in the list of clusters will automatically chatter and set themselves up. Unlike core NATS clustering though, each Jetstream node **must specify** a server name and cluster name. - -Below are explicity listed server configuration for a three node cluster across three machines, `n1-c1`, `n2-c1`, and `n3-c1`. - -### Server 1 (host_a) - -```nohighlight -server_name=n1-c1 -listen=4222 - -jetstream { - store_dir=/nats/storage -} - -cluster { - name: C1 - listen: localhost:6222 - routes: [ - nats-route://host_b:6222 - nats-route://host_c:6222 - ] -} -``` - -### Server 2 (host_b) - -```nohighlight -server_name=n2-c1 -listen=4222 - -jetstream { - store_dir=/nats/storage -} - -cluster { - name: C1 - listen: localhost:6222 - routes: [ - nats-route://host_a:6222 - nats-route://host_c:6222 - ] -} -``` - -### Server 3 (host_c) - -```nohighlight -server_name=n3-c1 -listen=4222 - -jetstream { - store_dir=/nats/storage -} - -cluster { - name: C1 - listen: localhost:6222 - routes: [ - nats-route://host_a:6222 - nats-route://host_b:6222 - ] -} -``` - -Add nodes as necessary. Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use Jetstream. \ No newline at end of file diff --git a/jetstream/concepts/README.md b/jetstream/concepts/README.md new file mode 100644 index 0000000..0536bcc --- /dev/null +++ b/jetstream/concepts/README.md @@ -0,0 +1,20 @@ +# Concepts + +In JetStream the configuration for storing messages is defined separately from how they are consumed. Storage is defined in a _Stream_ and consuming messages is defined by multiple _Consumers_. + +We'll discuss these 2 subjects in the context of this architecture. + +![Orders](../../.gitbook/assets/streams-and-consumers-75p.png) + +While this is an incomplete architecture it does show a number of key points: + +* Many related subjects are stored in a Stream +* Consumers can have different modes of operation and receive just subsets of the messages +* Multiple Acknowledgement modes are supported + +A new order arrives on `ORDERS.received`, gets sent to the `NEW` Consumer who, on success, will create a new message on `ORDERS.processed`. The `ORDERS.processed` message again enters the Stream where a `DISPATCH` Consumer receives it and once processed it will create an `ORDERS.completed` message which will again enter the Stream. These operations are all `pull` based meaning they are work queues and can scale horizontally. All require acknowledged delivery ensuring no order is missed. + +All messages are delivered to a `MONITOR` Consumer without any acknowledgement and using Pub/Sub semantics - they are pushed to the monitor. + +As messages are acknowledged to the `NEW` and `DISPATCH` Consumers, a percentage of them are Sampled and messages indicating redelivery counts, ack delays and more, are delivered to the monitoring system. + diff --git a/jetstream/concepts/concepts.md b/jetstream/concepts/concepts.md deleted file mode 100644 index 4a533c6..0000000 --- a/jetstream/concepts/concepts.md +++ /dev/null @@ -1,19 +0,0 @@ -## Concepts - -In JetStream the configuration for storing messages is defined separately from how they are consumed. Storage is defined in a *Stream* and consuming messages is defined by multiple *Consumers*. - -We'll discuss these 2 subjects in the context of this architecture. - -![Orders](../../assets/images/streams-and-consumers-75p.png) - -While this is an incomplete architecture it does show a number of key points: - - * Many related subjects are stored in a Stream - * Consumers can have different modes of operation and receive just subsets of the messages - * Multiple Acknowledgement modes are supported - -A new order arrives on `ORDERS.received`, gets sent to the `NEW` Consumer who, on success, will create a new message on `ORDERS.processed`. The `ORDERS.processed` message again enters the Stream where a `DISPATCH` Consumer receives it and once processed it will create an `ORDERS.completed` message which will again enter the Stream. These operations are all `pull` based meaning they are work queues and can scale horizontally. All require acknowledged delivery ensuring no order is missed. - -All messages are delivered to a `MONITOR` Consumer without any acknowledgement and using Pub/Sub semantics - they are pushed to the monitor. - -As messages are acknowledged to the `NEW` and `DISPATCH` Consumers, a percentage of them are Sampled and messages indicating redelivery counts, ack delays and more, are delivered to the monitoring system. \ No newline at end of file diff --git a/jetstream/concepts/configuration.md b/jetstream/concepts/configuration.md index aa4824c..74c831a 100644 --- a/jetstream/concepts/configuration.md +++ b/jetstream/concepts/configuration.md @@ -1,10 +1,11 @@ -### Configuration +# Configuration -The rest of this document introduces the `nats` utility, but for completeness and reference this is how you'd create the ORDERS scenario. We'll configure a 1 year retention for order related messages: +The rest of this document introduces the `nats` utility, but for completeness and reference this is how you'd create the ORDERS scenario. We'll configure a 1 year retention for order related messages: ```bash $ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard=old $ nats con add ORDERS NEW --filter ORDERS.received --ack explicit --pull --deliver all --max-deliver=-1 --sample 100 $ nats con add ORDERS DISPATCH --filter ORDERS.processed --ack explicit --pull --deliver all --max-deliver=-1 --sample 100 $ nats con add ORDERS MONITOR --filter '' --ack none --target monitor.ORDERS --deliver last --replay instant -``` \ No newline at end of file +``` + diff --git a/jetstream/concepts/consumers.md b/jetstream/concepts/consumers.md index 7ce7fde..288c27c 100644 --- a/jetstream/concepts/consumers.md +++ b/jetstream/concepts/consumers.md @@ -1,33 +1,34 @@ -### Consumers +# Consumes -Each Consumer, or related group of Consumers, of a Stream will need an Consumer defined. It's ok to define thousands of these pointing at the same Stream. +Each Consumer, or related group of Consumers, of a Stream will need an Consumer defined. It's ok to define thousands of these pointing at the same Stream. -Consumers can either be `push` based where JetStream will deliver the messages as fast as possible to a subject of your choice or `pull` based for typical work queue like behavior. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging. +Consumers can either be `push` based where JetStream will deliver the messages as fast as possible to a subject of your choice or `pull` based for typical work queue like behavior. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging. In the orders example above we have 3 Consumers. The first two select a subset of the messages from the Stream by specifying a specific subject like `ORDERS.processed`. The Stream consumes `ORDERS.*` and this allows you to receive just what you need. The final Consumer receives all messages in a `push` fashion. -Consumers track their progress, they know what messages were delivered, acknowledged, etc., and will redeliver messages they sent that were not acknowledged. When first created, the Consumer has to know what message to send as the first one. You can configure either a specific message in the set (`StreamSeq`), specific time (`StartTime`), all (`DeliverAll`) or last (`DeliverLast`). This is the starting point and from there, they all behave the same - delivering all of the following messages with optional Acknowledgement. +Consumers track their progress, they know what messages were delivered, acknowledged, etc., and will redeliver messages they sent that were not acknowledged. When first created, the Consumer has to know what message to send as the first one. You can configure either a specific message in the set \(`StreamSeq`\), specific time \(`StartTime`\), all \(`DeliverAll`\) or last \(`DeliverLast`\). This is the starting point and from there, they all behave the same - delivering all of the following messages with optional Acknowledgement. Acknowledgements default to `AckExplicit` - the only supported mode for pull-based Consumers - meaning every message requires a distinct acknowledgement. But for push-based Consumers, you can set `AckNone` that does not require any acknowledgement, or `AckAll` which quite interestingly allows you to acknowledge a specific message, like message `100`, which will also acknowledge messages `1` through `99`. The `AckAll` mode can be a great performance boost. Some messages may cause your applications to crash and cause a never ending loop forever poisoning your system. The `MaxDeliver` setting allow you to set a upper bound to how many times a message may be delivered. -To assist with creating monitoring applications, one can set a `SampleFrequency` which is a percentage of messages for which the system should sample and create events. These events will include delivery counts and ack waits. +To assist with creating monitoring applications, one can set a `SampleFrequency` which is a percentage of messages for which the system should sample and create events. These events will include delivery counts and ack waits. When defining Consumers the items below make up the entire configuration of the Consumer: -|Item|Description| -|----|-----------| -|AckPolicy|How messages should be acknowledged, `AckNone`, `AckAll` or `AckExplicit`| -|AckWait|How long to allow messages to remain un-acknowledged before attempting redelivery| -|DeliverPolicy|The initial starting mode of the consumer, `DeliverAll`, `DeliverLast`, `DeliverNew`, `DeliverByStartSequence` or `DeliverByStartTime`| -|DeliverySubject|The subject to deliver observed messages, when not set, a pull-based Consumer is created| -|Durable|The name of the Consumer| -|FilterSubject|When consuming from a Stream with many subjects, or wildcards, select only a specific incoming subjects, supports wildcards| -|MaxDeliver|Maximum amount times a specific message will be delivered. Use this to avoid poison pills crashing all your services forever| -|OptStartSeq|When first consuming messages from the Stream start at this particular message in the set| -|ReplayPolicy|How messages are sent `ReplayInstant` or `ReplayOriginal`| -|SampleFrequency|What percentage of acknowledgements should be samples for observability, 0-100| -|OptStartTime|When first consuming messages from the Stream start with messages on or after this time| -|RateLimit|The rate of message delivery in bits per second| -|MaxAckPending|The maximum number of messages without acknowledgement that can be outstanding, once this limit is reached message delivery will be suspended| +| Item | Description | +| :--- | :--- | +| AckPolicy | How messages should be acknowledged, `AckNone`, `AckAll` or `AckExplicit` | +| AckWait | How long to allow messages to remain un-acknowledged before attempting redelivery | +| DeliverPolicy | The initial starting mode of the consumer, `DeliverAll`, `DeliverLast`, `DeliverNew`, `DeliverByStartSequence` or `DeliverByStartTime` | +| DeliverySubject | The subject to deliver observed messages, when not set, a pull-based Consumer is created | +| Durable | The name of the Consumer | +| FilterSubject | When consuming from a Stream with many subjects, or wildcards, select only a specific incoming subjects, supports wildcards | +| MaxDeliver | Maximum amount times a specific message will be delivered. Use this to avoid poison pills crashing all your services forever | +| OptStartSeq | When first consuming messages from the Stream start at this particular message in the set | +| ReplayPolicy | How messages are sent `ReplayInstant` or `ReplayOriginal` | +| SampleFrequency | What percentage of acknowledgements should be samples for observability, 0-100 | +| OptStartTime | When first consuming messages from the Stream start with messages on or after this time | +| RateLimit | The rate of message delivery in bits per second | +| MaxAckPending | The maximum number of messages without acknowledgement that can be outstanding, once this limit is reached message delivery will be suspended | + diff --git a/jetstream/concepts/streams.md b/jetstream/concepts/streams.md index 87c8205..6b31291 100644 --- a/jetstream/concepts/streams.md +++ b/jetstream/concepts/streams.md @@ -1,31 +1,32 @@ -### Streams +# Streams -Streams define how messages are stored and retention duration. Streams consume normal NATS subjects, any message found on those subjects will be delivered to the defined storage system. You can do a normal publish to the subject for unacknowledged delivery, else if you send a Request to the subject the JetStream server will reply with an acknowledgement that it was stored. +Streams define how messages are stored and retention duration. Streams consume normal NATS subjects, any message found on those subjects will be delivered to the defined storage system. You can do a normal publish to the subject for unacknowledged delivery, else if you send a Request to the subject the JetStream server will reply with an acknowledgement that it was stored. As of January 2020, in the tech preview we have `file` and `memory` based storage systems, we do not yet support clustering. In the diagram above we show the concept of storing all `ORDERS.*` in the Stream even though there are many types of order related messages. We'll show how you can selectively consume subsets of messages later. Relatively speaking the Stream is the most resource consuming component so being able to combine related data in this manner is important to consider. -Streams can consume many subjects. Here we have `ORDERS.*` but we could also consume `SHIPPING.state` into the same Stream should that make sense (not shown here). +Streams can consume many subjects. Here we have `ORDERS.*` but we could also consume `SHIPPING.state` into the same Stream should that make sense \(not shown here\). Streams support various retention policies - they can be kept based on limits like max count, size or age but also more novel methods like keeping them as long as any Consumers have them unacknowledged, or work queue like behavior where a message is removed after first ack. -Streams support deduplication using a `Nats-Msg-Id` header and a sliding window within which to track duplicate messages. See the [Message Deduplication](#message-deduplication) section. +Streams support deduplication using a `Nats-Msg-Id` header and a sliding window within which to track duplicate messages. See the [Message Deduplication](../model_deep_dive/#message-deduplication) section. When defining Streams the items below make up the entire configuration of the set. -|Item|Description| -|----|-----------| -|MaxAge|Maximum age of any message in the stream, expressed in microseconds| -|MaxBytes|How big the Stream may be, when the combined stream size exceeds this old messages are removed| -|MaxMsgSize|The largest message that will be accepted by the Stream| -|MaxMsgs|How many messages may be in a Stream, oldest messages will be removed if the Stream exceeds this size| -|MaxConsumers|How many Consumers can be defined for a given Stream, `-1` for unlimited| -|Name|A name for the Stream that may not have spaces, tabs or `.`| -|NoAck|Disables acknowledging messages that are received by the Stream| -|Replicas|How many replicas to keep for each message in a clustered JetStream, maximum 5| -|Retention|How message retention is considered, `LimitsPolicy` (default), `InterestPolicy` or `WorkQueuePolicy`| -|Discard|When a Stream reached it's limits either, `DiscardNew` refuses new messages while `DiscardOld` (default) deletes old messages| -|Storage|The type of storage backend, `file` and `memory` as of January 2020| -|Subjects|A list of subjects to consume, supports wildcards| -|Duplicates|The window within which to track duplicate messages| +| Item | Description | +| :--- | :--- | +| MaxAge | Maximum age of any message in the stream, expressed in microseconds | +| MaxBytes | How big the Stream may be, when the combined stream size exceeds this old messages are removed | +| MaxMsgSize | The largest message that will be accepted by the Stream | +| MaxMsgs | How many messages may be in a Stream, oldest messages will be removed if the Stream exceeds this size | +| MaxConsumers | How many Consumers can be defined for a given Stream, `-1` for unlimited | +| Name | A name for the Stream that may not have spaces, tabs or `.` | +| NoAck | Disables acknowledging messages that are received by the Stream | +| Replicas | How many replicas to keep for each message in a clustered JetStream, maximum 5 | +| Retention | How message retention is considered, `LimitsPolicy` \(default\), `InterestPolicy` or `WorkQueuePolicy` | +| Discard | When a Stream reached it's limits either, `DiscardNew` refuses new messages while `DiscardOld` \(default\) deletes old messages | +| Storage | The type of storage backend, `file` and `memory` as of January 2020 | +| Subjects | A list of subjects to consume, supports wildcards | +| Duplicates | The window within which to track duplicate messages | + diff --git a/jetstream/configuration_mgmt/configuration_mgmt.md b/jetstream/configuration_mgmt/README.md similarity index 87% rename from jetstream/configuration_mgmt/configuration_mgmt.md rename to jetstream/configuration_mgmt/README.md index f654c60..7869bbf 100644 --- a/jetstream/configuration_mgmt/configuration_mgmt.md +++ b/jetstream/configuration_mgmt/README.md @@ -1,50 +1,51 @@ -## Configuration Management +# Configuration Management -In many cases managing the configuration in your application code is the best model, many teams though wish to pre-create Streams and Consumers. +In many cases managing the configuration in your application code is the best model, many teams though wish to pre-create Streams and Consumers. We support a number of tools to assist with this: - * `nats` CLI with configuration files - * [Terraform](https://www.terraform.io/) - * [GitHub Actions](https://github.com/features/actions) - * [Kubernetes JetStream Controller](https://github.com/nats-io/nack#jetstream-controller) +* `nats` CLI with configuration files +* [Terraform](https://www.terraform.io/) +* [GitHub Actions](https://github.com/features/actions) +* [Kubernetes JetStream Controller](https://github.com/nats-io/nack#jetstream-controller) -### nats Admin CLI +## nats Admin CLI The `nats` CLI can be used to manage Streams and Consumers easily using it's `--config` flag, for example: -### Add a new Stream +## Add a new Stream This creates a new Stream based on `orders.json`. The `orders.json` file can be extracted from an existing stream using `nats stream info ORDERS -j | jq .config` -``` +```text $ nats str add ORDERS --config orders.json ``` -### Edit an existing Stream +## Edit an existing Stream This edits an existing stream ensuring it complies with the configuration in `orders.json` -``` + +```text $ nats str edit ORDERS --config orders.json ``` -### Add a New Consumer +## Add a New Consumer This creates a new Consumer based on `orders_new.json`. The `orders_new.json` file can be extracted from an existing stream using `nats con info ORDERS NEW -j | jq .config` -``` +```text $ nats con add ORDERS NEW --config orders_new.json ``` -### Terraform +## Terraform Terraform is a Cloud configuration tool from Hashicorp found at [terraform.io](https://www.terraform.io/), we maintain a Provider for Terraform called [terraform-provider-jetstream](https://github.com/nats-io/terraform-provider-jetstream/) that can maintain JetStream using Terraform. -#### Setup +### Setup Our provider is not hosted by Hashicorp so installation is a bit more complex than typical. Browse to the [Release Page](https://github.com/nats-io/terraform-provider-jetstream/releases) and download the release for your platform and extract it into your Terraform plugins directory. -``` +```text $ unzip -l terraform-provider-jetstream_0.0.2_darwin_amd64.zip Archive: terraform-provider-jetstream_0.0.2_darwin_amd64.zip Length Date Time Name @@ -58,7 +59,7 @@ Place the `terraform-provider-jetstream_v0.0.2` file in `~/.terraform.d/plugins/ In your project you can configure the Provider like this: -```terraform +```text provider "jetstream" { servers = "connect.ngs.global" credentials = "ngs_jetstream_admin.creds" @@ -67,7 +68,7 @@ provider "jetstream" { And start using it, here's an example that create the `ORDERS` example. Review the [Project README](https://github.com/nats-io/terraform-provider-jetstream#readme) for full details. -```terraform +```text resource "jetstream_stream" "ORDERS" { name = "ORDERS" subjects = ["ORDERS.*"] @@ -103,3 +104,4 @@ output "ORDERS_SUBJECTS" { value = jetstream_stream.ORDERS.subjects } ``` + diff --git a/jetstream/configuration_mgmt/github_actions.md b/jetstream/configuration_mgmt/github_actions.md index 81975bd..dd271fa 100644 --- a/jetstream/configuration_mgmt/github_actions.md +++ b/jetstream/configuration_mgmt/github_actions.md @@ -1,4 +1,4 @@ -### GitHub Actions +# GitHub Actions We have a pack of GitHub Actions that let you manage an already running JetStream Server, useful for managing releases or standing up test infrastructure. @@ -53,3 +53,4 @@ jobs: message: Published new deployment via "${{ github.event_name }}" in "${{ github.repository }}" server: js.example.net ``` + diff --git a/jetstream/configuration_mgmt/kubernetes_controller.md b/jetstream/configuration_mgmt/kubernetes_controller.md index 9501fe6..3ea6392 100644 --- a/jetstream/configuration_mgmt/kubernetes_controller.md +++ b/jetstream/configuration_mgmt/kubernetes_controller.md @@ -1,6 +1,6 @@ -### Kubernetes JetStream Controller +# Kubernetes Controller -The JetStream controllers allows you to manage NATS JetStream Streams and Consumers via K8S CRDs. You can find more info on how to deploy and usage [here](https://github.com/nats-io/nack#getting-started). Below you can find an example on how to create a stream and a couple of consumers: +The JetStream controllers allows you to manage NATS JetStream Streams and Consumers via K8S CRDs. You can find more info on how to deploy and usage [here](https://github.com/nats-io/nack#getting-started). Below you can find an example on how to create a stream and a couple of consumers: ```yaml --- @@ -41,7 +41,7 @@ spec: Once the CRDs are installed you can use `kubectl` to manage the streams and consumers as follows: -```sh +```bash $ kubectl get streams NAME STATE STREAM NAME SUBJECTS mystream Created mystream [orders.*] @@ -55,3 +55,4 @@ my-push-consumer Created mystream my-push-consumer none # kubectl describe streams mystream # kubectl describe consumers my-pull-consumer ``` + diff --git a/jetstream/configuration_mgmt/nats-admin-cli.md b/jetstream/configuration_mgmt/nats-admin-cli.md new file mode 100644 index 0000000..58f1a8b --- /dev/null +++ b/jetstream/configuration_mgmt/nats-admin-cli.md @@ -0,0 +1,2 @@ +# NATS Admin CLI + diff --git a/jetstream/configuration_mgmt/terraform.md b/jetstream/configuration_mgmt/terraform.md new file mode 100644 index 0000000..6719963 --- /dev/null +++ b/jetstream/configuration_mgmt/terraform.md @@ -0,0 +1,2 @@ +# Terraform + diff --git a/jetstream/disaster_recovery/disaster_recovery.md b/jetstream/disaster_recovery.md similarity index 76% rename from jetstream/disaster_recovery/disaster_recovery.md rename to jetstream/disaster_recovery.md index 92b851b..fe0751f 100644 --- a/jetstream/disaster_recovery/disaster_recovery.md +++ b/jetstream/disaster_recovery.md @@ -1,40 +1,40 @@ -## Disaster Recovery +# Disaser Recovery Disaster Recovery of the JetStream system is a topic we are still exploring and fleshing out and that will be impacted by the clustering work. For example replication will extend the options available to you. Today we have a few approaches to consider: - * `nats` CLI + Configuration Backups + Data Snapshots - * Configuration Management + Data Snapshots +* `nats` CLI + Configuration Backups + Data Snapshots +* Configuration Management + Data Snapshots -### Data Backup +## Data Backup -In all scenarios you can perform data snapshots and restores over the NATS protocol. This is good if you do not manage the NATS servers hosting your data, and you wish to do a backup of your data. +In all scenarios you can perform data snapshots and restores over the NATS protocol. This is good if you do not manage the NATS servers hosting your data, and you wish to do a backup of your data. The backup includes: - * Stream configuration and state - * Stream Consumer configuration and state - * All data including metadata like timestamps and headers +* Stream configuration and state +* Stream Consumer configuration and state +* All data including metadata like timestamps and headers -```nohighlight +```text $ nats stream backup ORDERS /data/js-backup/ORDERS.tgz Starting backup of Stream "ORDERS" with 13 data blocks 2.4 MiB/s [====================================================================] 100% -Received 13 MiB bytes of compressed data in 3368 chunks for stream "ORDERS" in 1.223428188s, 813 MiB uncompressed +Received 13 MiB bytes of compressed data in 3368 chunks for stream "ORDERS" in 1.223428188s, 813 MiB uncompressed ``` During the backup the Stream is in a state where it's configuration cannot change and no data will be expired from it based on Limits or Retention Policies. Progress using the terminal bar can be disabled using `--no-progress`, it will then issue log lines instead. -### Restoring Data +## Restoring Data -The backup made above can be restored into another server - but into the same Stream name. +The backup made above can be restored into another server - but into the same Stream name. -```nohighlight +```text $ nats str restore ORDERS /data/js-backup/ORDERS.tgz Starting restore of Stream "ORDERS" from file "/data/js-backup/ORDERS.tgz" @@ -54,13 +54,13 @@ The `/data/js-backup/ORDERS.tgz` file can also be extracted into the data dir of Progress using the terminal bar can be disabled using `--no-progress`, it will then issue log lines instead. -### Interactive CLI +## Interactive CLI -In environments where the `nats` CLI is used interactively to configure the server you do not have a desired state to recreate the server from. This is not the ideal way to administer the server, we recommend Configuration Management, but many will use this approach. +In environments where the `nats` CLI is used interactively to configure the server you do not have a desired state to recreate the server from. This is not the ideal way to administer the server, we recommend Configuration Management, but many will use this approach. -Here you can back up the configuration into a directory from where you can recover the configuration later. The data for File backed stores can also be backed up. +Here you can back up the configuration into a directory from where you can recover the configuration later. The data for File backed stores can also be backed up. -```nohighlight +```text $ nats backup /data/js-backup 15:56:11 Creating JetStream backup into /data/js-backup 15:56:11 Stream ORDERS to /data/js-backup/stream_ORDERS.json @@ -74,7 +74,7 @@ During the same process the data can also be backed up by passing `--data`, this Later the data can be restored, for Streams we support editing the Stream configuration in place to match what was in the backup. -``` +```text $ nats restore /tmp/backup --update-streams 15:57:42 Reading file /tmp/backup/stream_ORDERS.json 15:57:42 Reading file /tmp/backup/stream_ORDERS_consumer_NEW.json @@ -82,4 +82,5 @@ $ nats restore /tmp/backup --update-streams 15:57:42 Restoring Consumer ORDERS > NEW ``` -The `nats restore` tool does not support restoring data, the same process using `nats stream restore`, as outlined earlier, can be used which will also restore Stream and Consumer configurations and state. \ No newline at end of file +The `nats restore` tool does not support restoring data, the same process using `nats stream restore`, as outlined earlier, can be used which will also restore Stream and Consumer configurations and state. + diff --git a/jetstream/getting_started/README.md b/jetstream/getting_started/README.md new file mode 100644 index 0000000..1b342e0 --- /dev/null +++ b/jetstream/getting_started/README.md @@ -0,0 +1,27 @@ +# Getting Started + +Getting started with JetStream is straightforward. While we speak of Jetstream as if it is a seperate component, it's actually a subsystem built into the NATS server that needs to be enabled. + +## Command line + +Enable JetStream by specifying the `-js` flag when starting the NATS server. + +`$ nats-server -js` + +## Configuration File + +Enable JetStream through a configuration file. By default, the JetStream subsytem will store data in the /tmp directory. Here's a minimal file that will store data in a local "nats" directory, suitable for development and local testing. + +`$ nats-server -c js.conf` + +```text +# js.conf +jetstream { + store_dir=nats +} +``` + +Normally JetStream will be run in clustered mode and will replicate data, so the best place to store JetStream data would be locally on a fast SSD. One should specifically avoid NAS or NFS storage for JetStream. + +See [Using Docker](using_docker.md) and [Using Source](using_source.md) for more information. + diff --git a/jetstream/getting_started/getting_started.md b/jetstream/getting_started/getting_started.md deleted file mode 100644 index 6e5ff6c..0000000 --- a/jetstream/getting_started/getting_started.md +++ /dev/null @@ -1,26 +0,0 @@ -# Getting Started - -Getting started with JetStream is straightforward. While we speak of Jetstream as if it is a seperate component, it's actually a subsystem built into the NATS server that needs to be enabled. - -## Command line - -Enable JetStream by specifying the `-js` flag when starting the NATS server. - -`$ nats-server -js` - -## Configuration File - -Enable JetStream through a configuration file. By default, the JetStream subsytem will store data in the /tmp directory. Here's a minimal file that will store data in a local "nats" directory, suitable for development and local testing. - -`$ nats-server -c js.conf` - -```text -# js.conf -jetstream { - store_dir=nats -} -``` - -Normally JetStream will be run in clustered mode and will replicate data, so the best place to store JetStream data would be locally on a fast SSD. One should specifically avoid NAS or NFS storage for JetStream. - -See [Using Docker](./using_docker.md) and [Using Source](./using_source.md) for more information. diff --git a/jetstream/getting_started/using-docker-with-ngs.md b/jetstream/getting_started/using-docker-with-ngs.md new file mode 100644 index 0000000..5dd8335 --- /dev/null +++ b/jetstream/getting_started/using-docker-with-ngs.md @@ -0,0 +1,2 @@ +# Using Docker with NGS + diff --git a/jetstream/getting_started/using_docker.md b/jetstream/getting_started/using_docker.md index f36feef..ad44991 100644 --- a/jetstream/getting_started/using_docker.md +++ b/jetstream/getting_started/using_docker.md @@ -4,13 +4,13 @@ The `natsio/nats-box:latest` docker image contains the `nats` utility this guide In one window start a JetStream enabled nats server: -``` +```text $ docker run --network host -p 4222:4222 nats -js ``` And in another log into the utilities: -``` +```text $ docker run -ti --network host natsio/nats-box ``` @@ -18,22 +18,24 @@ This shell has the `nats` utility and all other NATS cli tools used in the rest Now skip to the `Administer JetStream` section. -### Using Docker with NGS +## Using Docker with NGS You can join a JetStream instance to your [NGS](https://synadia.com/ngs/pricing) account, first we need a credential for testing JetStream: You'll want to do this outside of docker to keep the credentials that are generated. -``` + +```text $ nsc add user -a YourAccount --name leafnode --expiry 1M ``` You'll get a credential file somewhere like `~/.nkeys/creds/synadia/YourAccount/leafnode.creds`, mount this file into the docker container for JetStream using `-v ~/.nkeys/creds/synadia/YourAccount/leafnode.creds:/leafnode.creds`. -``` +```text $ docker run -ti -v ~/.nkeys/creds/synadia/YourAccount/leafnode.creds:/leafnode.creds --name jetstream synadia/jsm:latest server [1] 2020/01/20 12:44:11.752465 [INF] Starting nats-server version 2.2.0 ... [1] 2020/01/20 12:55:01.849033 [INF] Connected leafnode to "connect.ngs.global" ``` -Your JSM shell will still connect locally, other connections in your NGS account can use JetStream at this point. \ No newline at end of file +Your JSM shell will still connect locally, other connections in your NGS account can use JetStream at this point. + diff --git a/jetstream/getting_started/using_source.md b/jetstream/getting_started/using_source.md index 7b7e821..d75d72f 100644 --- a/jetstream/getting_started/using_source.md +++ b/jetstream/getting_started/using_source.md @@ -1,10 +1,10 @@ -# Using Source or a Binary +# Using Source You will also want to have installed from the nats.go repo the examples/tools such as nats-pub, nats-sub, nats-req and possibly nats-bench. One of the design goals of JetStream was to be native to core NATS, so even though we will most certainly add in syntactic sugar to clients to make them more appealing, for this tech preview we will be using plain old NATS. You will need a copy of the nats-server source locally and will need to be in the jetstream branch. -``` +```text $ git clone https://github.com/nats-io/nats-server.git $ cd nats-server $ git checkout master @@ -14,7 +14,7 @@ $ ls -l nats-server Starting the server you can use the `-js` flag. This will setup the server to reasonably use memory and disk. This is a sample run on my machine. JetStream will default to 1TB of disk and 75% of available memory for now. -``` +```text $ ./nats-server -js [16928] 2019/12/04 19:16:29.596968 [INF] Starting nats-server version 2.2.0 @@ -32,7 +32,7 @@ $ ./nats-server -js You can override the storage directory if you want. -``` +```text $ ./nats-server -js -sd /tmp/test [16943] 2019/12/04 19:20:00.874148 [INF] Starting nats-server version 2.2.0 @@ -50,7 +50,7 @@ $ ./nats-server -js -sd /tmp/test These options can also be set in your configuration file: -``` +```text // enables jetstream, an empty block will enable and use defaults jetstream { // jetstream data will be in /data/nats-server/jetstream @@ -62,4 +62,5 @@ jetstream { // 10GB max_file_store: 10737418240 } -``` \ No newline at end of file +``` + diff --git a/jetstream/jetstream.md b/jetstream/jetstream.md new file mode 100644 index 0000000..cd085c5 --- /dev/null +++ b/jetstream/jetstream.md @@ -0,0 +1,35 @@ +# About Jetstream + +JetStream was created to solve the problems identified with streaming in technology today - complexity, fragility, and a lack of scalability. Some technologies address these better than others, but no current streaming technology is truly multi-tenant, horizontally scalable, and supports multiple deployment models. No technology we are aware of can scale from edge to cloud under the same security context while having complete deployment observability for operations. + +## Goals + +JetStream was developed with the following goals in mind: + +* The system must be easy to configure and operate and be observable. +* The system must be secure and operate well with NATS 2.0 security models. +* The system must scale horizontally and be applicable to a high ingestion rate. +* The system must support multiple use cases. +* The system must self heal and always be available. +* The system must have an API that is closer to core NATS. +* The system must allow NATS messages to be part of a stream as desired. +* The system must display payload agnostic behavior. +* The system must not have third party dependencies. + +## High-Level Design and Features + +In terms of deployment, a JetStream server is simply a NATS server with the JetStream subsystem enabled, launched with the `-js` flag with a configured server name and cluster name. From a client perspective, it does not matter which servers are running JetStream so long as there is some route to a JetStream enabled server or servers. This allows for a flexible deployment which to optimize resources for particular servers that will store streams versus very low overhead stateless servers, reducing OpEx and ultimately creating a scalable and manageable system. + +## Feature List + +* At-least-once delivery; exactly once within a window +* Store messages and replay by time or sequence +* Wildcard support +* Account aware +* Data at rest encryption +* Cleanse specific messages \(GDPR\) +* Horizontal scalability +* Persist Streams and replay via Consumers + +JetStream is designed to bifurcate ingestion and consumption of messages to provide multiple ways to consume data from the same stream. To that end, JetStream functionality is composed of server streams and server consumers. + diff --git a/jetstream/model_deep_dive/model_deep_dive.md b/jetstream/model_deep_dive/README.md similarity index 77% rename from jetstream/model_deep_dive/model_deep_dive.md rename to jetstream/model_deep_dive/README.md index 63f263f..c6670c7 100644 --- a/jetstream/model_deep_dive/model_deep_dive.md +++ b/jetstream/model_deep_dive/README.md @@ -1,8 +1,8 @@ -## Model Deep Dive +# Model Deep Dive The Orders example touched on a lot of features, but some like different Ack models and message limits, need a bit more detail. This section will expand on the above and fill in some blanks. -### Stream Limits, Retention Modes and Discard Policy +## Stream Limits, Retention Modes and Discard Policy Streams store data on disk, but we cannot store all data forever so we need ways to control their size automatically. @@ -10,38 +10,38 @@ There are 3 features that come into play when Streams decide how long they store The `Retention Policy` describes based on what criteria a set will evict messages from its storage: -|Retention Policy|Description| -|----------------|-----------| -|`LimitsPolicy` |Limits are set for how many messages, how big the storage and how old messages may be| -|`WorkQueuePolicy`|Messages are kept until they were consumed by any one single observer and then removed| -|`InterestPolicy`|Messages are kept as long as there are Consumers active for them| +| Retention Policy | Description | +| :--- | :--- | +| `LimitsPolicy` | Limits are set for how many messages, how big the storage and how old messages may be | +| `WorkQueuePolicy` | Messages are kept until they were consumed by any one single observer and then removed | +| `InterestPolicy` | Messages are kept as long as there are Consumers active for them | In all Retention Policies the basic limits apply as upper bounds, these are `MaxMsgs` for how many messages are kept in total, `MaxBytes` for how big the set can be in total and `MaxAge` for what is the oldest message that will be kept. These are the only limits in play with `LimitsPolicy` retention. -One can then define additional ways a message may be removed from the Stream earlier than these limits. In `WorkQueuePolicy` the messages will be removed as soon as any Consumer received an Acknowledgement. In `InterestPolicy` messages will be removed as soon as there are no more Consumers. +One can then define additional ways a message may be removed from the Stream earlier than these limits. In `WorkQueuePolicy` the messages will be removed as soon as any Consumer received an Acknowledgement. In `InterestPolicy` messages will be removed as soon as there are no more Consumers. In both `WorkQueuePolicy` and `InterestPolicy` the age, size and count limits will still apply as upper bounds. -A final control is the Maximum Size any single message may have. NATS have it's own limit for maximum size (1 MiB by default), but you can say a Stream will only accept messages up to 1024 bytes using `MaxMsgSize`. +A final control is the Maximum Size any single message may have. NATS have it's own limit for maximum size \(1 MiB by default\), but you can say a Stream will only accept messages up to 1024 bytes using `MaxMsgSize`. The `Discard Policy` sets how messages are discard when limits set by `LimitsPolicy` are reached. The `DiscardOld` option removes old messages making space for new, while `DiscardNew` refuses any new messages. The `WorkQueuePolicy` mode is a specialized mode where a message, once consumed and acknowledged, is discarded from the Stream. In this mode there are a few limits on consumers. Inherently it's about 1 message to one consumer, this means you cannot have overlapping consumers defined on the Stream - needs unique filter subjects. -### Message Deduplication +## Message Deduplication JetStream support idempotent message writes by ignoring duplicate messages as indicated by the `Nats-Msg-Id` header. -```nohighlight +```text % nats req -H Nats-Msg-Id:1 ORDERS.new hello1 % nats req -H Nats-Msg-Id:1 ORDERS.new hello2 % nats req -H Nats-Msg-Id:1 ORDERS.new hello3 % nats req -H Nats-Msg-Id:1 ORDERS.new hello4 ``` - + Here we set a `Nats-Msg-Id:1` header which tells JetStream to ensure we do not have duplicates of this message - we only consult the message ID not the body. -```nohighlight +```text $ nats str info ORDERS .... State: @@ -52,21 +52,21 @@ State: The default window to track duplicates in is 2 minutes, this can be set on the command line using `--dupe-window` when creating a stream, though we would caution against large windows. -### Acknowledgement Models +## Acknowledgement Models -Streams support acknowledging receiving a message, if you send a `Request()` to a subject covered by the configuration of the Stream the service will reply to you once it stored the message. If you just publish, it will not. A Stream can be set to disable Acknowledgements by setting `NoAck` to `true` in it's configuration. +Streams support acknowledging receiving a message, if you send a `Request()` to a subject covered by the configuration of the Stream the service will reply to you once it stored the message. If you just publish, it will not. A Stream can be set to disable Acknowledgements by setting `NoAck` to `true` in it's configuration. Consumers have 3 acknowledgement modes: -|Mode|Description| -|----|-----------| -|`AckExplicit`|This requires every message to be specifically acknowledged, it's the only supported option for pull-based Consumers| -|`AckAll`|In this mode if you acknowledge message `100` it will also acknowledge message `1`-`99`, this is good for processing batches and to reduce ack overhead| -|`AckNone`|No acknowledgements are supported| +| Mode | Description | +| :--- | :--- | +| `AckExplicit` | This requires every message to be specifically acknowledged, it's the only supported option for pull-based Consumers | +| `AckAll` | In this mode if you acknowledge message `100` it will also acknowledge message `1`-`99`, this is good for processing batches and to reduce ack overhead | +| `AckNone` | No acknowledgements are supported | To understand how Consumers track messages we will start with a clean `ORDERS` Stream and `DISPATCH` Consumer. -``` +```text $ nats str info ORDERS ... Statistics: @@ -80,7 +80,7 @@ Statistics: The Set is entirely empty -``` +```text $ nats con info ORDERS DISPATCH ... State: @@ -91,11 +91,11 @@ State: Redelivered Messages: 0 ``` -The Consumer has no messages oustanding and has never had any (Consumer sequence is 1). +The Consumer has no messages oustanding and has never had any \(Consumer sequence is 1\). We publish one message to the Stream and see that the Stream received it: -``` +```text $ nats pub ORDERS.processed "order 4" Published 7 bytes to ORDERS.processed $ nats str info ORDERS @@ -111,7 +111,7 @@ Statistics: As the Consumer is pull-based, we can fetch the message, ack it, and check the Consumer state: -``` +```text $ nats con next ORDERS DISPATCH --- received on ORDERS.processed order 4 @@ -132,7 +132,7 @@ The message got delivered and acknowledged - `Acknowledgement floor` is `1` and We'll publish another message, fetch it but not Ack it this time and see the status: -``` +```text $ nats pub ORDERS.processed "order 5" Published 7 bytes to ORDERS.processed @@ -149,11 +149,11 @@ State: Redelivered Messages: 0 ``` -Now we can see the Consumer have processed 2 messages (obs sequence is 3, next message will be 3) but the Ack floor is still 1 - thus 1 message is pending acknowledgement. Indeed this is confirmed in the `Pending messages`. +Now we can see the Consumer have processed 2 messages \(obs sequence is 3, next message will be 3\) but the Ack floor is still 1 - thus 1 message is pending acknowledgement. Indeed this is confirmed in the `Pending messages`. If I fetch it again and again do not ack it: -``` +```text $ nats con next ORDERS DISPATCH --no-ack --- received on ORDERS.processed order 5 @@ -171,7 +171,7 @@ The Consumer sequence increases - each delivery attempt increase the sequence - Finally if I then fetch it again and ack it this time: -``` +```text $ nats con next ORDERS DISPATCH --- received on ORDERS.processed order 5 @@ -190,39 +190,39 @@ Having now Acked the message there are no more pending. Additionally there are a few types of acknowledgements: -|Type|Bytes|Description| -|----|-----|-----------| -|`AckAck`|nil, `+ACK`|Acknowledges a message was completely handled| -|`AckNak`|`-NAK`|Signals that the message will not be processed now and processing can move onto the next message, NAK'd message will be retried| -|`AckProgress`|`+WPI`|When sent before the AckWait period indicates that work is ongoing and the period should be extended by another equal to `AckWait`| -|`AckNext`|`+NXT`|Acknowledges the message was handled and requests delivery of the next message to the reply subject. Only applies to Pull-mode.| -|`AckTerm`|`+TERM`|Instructs the server to stop redelivery of a message without acknowledging it as successfully processed| +| Type | Bytes | Description | +| :--- | :--- | :--- | +| `AckAck` | nil, `+ACK` | Acknowledges a message was completely handled | +| `AckNak` | `-NAK` | Signals that the message will not be processed now and processing can move onto the next message, NAK'd message will be retried | +| `AckProgress` | `+WPI` | When sent before the AckWait period indicates that work is ongoing and the period should be extended by another equal to `AckWait` | +| `AckNext` | `+NXT` | Acknowledges the message was handled and requests delivery of the next message to the reply subject. Only applies to Pull-mode. | +| `AckTerm` | `+TERM` | Instructs the server to stop redelivery of a message without acknowledging it as successfully processed | So far all the examples was the `AckAck` type of acknowledgement, by replying to the Ack with the body as indicated in `Bytes` you can pick what mode of acknowledgement you want. All of these acknowledgement modes, except `AckNext`, support double acknowledgement - if you set a reply subject when acknowledging the server will in turn acknowledge having received your ACK. The `+NXT` acknowledgement can have a few formats: `+NXT 10` requests 10 messages and `+NXT {"no_wait": true}` which is the same data that can be sent in a Pull request. - -### Exactly Once Delivery + +## Exactly Once Delivery JetStream supports Exactly Once delivery by combining Message Deduplication and double acks. -On the publishing side you can avoid duplicate message ingestion using the [Message Deduplication](#message-deduplication) feature. +On the publishing side you can avoid duplicate message ingestion using the [Message Deduplication](./#message-deduplication) feature. Consumers can be 100% sure a message was correctly processed by requesting the server Acknowledge having received your acknowledgement by setting a reply subject on the Ack. If you receive this response you will never receive that message again. -### Consumer Starting Position +## Consumer Starting Position When setting up an Consumer you can decide where to start, the system supports the following for the `DeliverPolicy`: -|Policy|Description| -|------|-----------| -|`all`|Delivers all messages that are available| -|`last`|Delivers the latest message, like a `tail -n 1 -f`| -|`new`|Delivers only new messages that arrive after subscribe time| -|`by_start_time`|Delivers from a specific time onward. Requires `OptStartTime` to be set| -|`by_start_sequence`|Delivers from a specific stream sequence. Requires `OptStartSeq` to be set| +| Policy | Description | +| :--- | :--- | +| `all` | Delivers all messages that are available | +| `last` | Delivers the latest message, like a `tail -n 1 -f` | +| `new` | Delivers only new messages that arrive after subscribe time | +| `by_start_time` | Delivers from a specific time onward. Requires `OptStartTime` to be set | +| `by_start_sequence` | Delivers from a specific stream sequence. Requires `OptStartSeq` to be set | Regardless of what mode you set, this is only the starting point. Once started it will always give you what you have not seen or acknowledged. So this is merely how it picks the very first message. @@ -230,7 +230,7 @@ Lets look at each of these, first we make a new Stream `ORDERS` and add 100 mess Now create a `DeliverAll` pull-based Consumer: -``` +```text $ nats con add ORDERS ALL --pull --filter ORDERS.processed --ack none --replay instant --deliver all $ nats con next ORDERS ALL --- received on ORDERS.processed @@ -241,18 +241,18 @@ Acknowledged message Now create a `DeliverLast` pull-based Consumer: -``` +```text $ nats con add ORDERS LAST --pull --filter ORDERS.processed --ack none --replay instant --deliver last $ nats con next ORDERS LAST --- received on ORDERS.processed order 100 Acknowledged message -``` +``` Now create a `MsgSetSeq` pull-based Consumer: -``` +```text $ nats con add ORDERS TEN --pull --filter ORDERS.processed --ack none --replay instant --deliver 10 $ nats con next ORDERS TEN --- received on ORDERS.processed @@ -263,7 +263,7 @@ Acknowledged message And finally a time-based Consumer. Let's add some messages a minute apart: -``` +```text $ nats str purge ORDERS $ for i in 1 2 3 do @@ -274,7 +274,7 @@ done Then create an Consumer that starts 2 minutes ago: -``` +```text $ nats con add ORDERS 2MIN --pull --filter ORDERS.processed --ack none --replay instant --deliver 2m $ nats con next ORDERS 2MIN --- received on ORDERS.processed @@ -283,29 +283,29 @@ order 2 Acknowledged message ``` -### Ephemeral Consumers +## Ephemeral Consumers So far, all the Consumers you have seen were Durable, meaning they exist even after you disconnect from JetStream. In our Orders scenario, though the `MONITOR` Consumer could very well be a short-lived thing there just while an operator is debugging the system, there is no need to remember the last seen position if all you are doing is wanting to observe the real-time state. -In this case, we can make an Ephemeral Consumer by first subscribing to the delivery subject, then creating a durable and giving it no durable name. An Ephemeral Consumer exists as long as any subscription is active on its delivery subject. It is automatically be removed, after a short grace period to handle restarts, when there are no subscribers. +In this case, we can make an Ephemeral Consumer by first subscribing to the delivery subject, then creating a durable and giving it no durable name. An Ephemeral Consumer exists as long as any subscription is active on its delivery subject. It is automatically be removed, after a short grace period to handle restarts, when there are no subscribers. Ephemeral Consumers can only be push-based. Terminal 1: -``` +```text $ nats sub my.monitor ``` Terminal 2: -``` +```text $ nats con add ORDERS --filter '' --ack none --target 'my.monitor' --deliver last --replay instant --ephemeral ``` The `--ephemeral` switch tells the system to make an Ephemeral Consumer. -### Consumer Message Rates +## Consumer Message Rates Typically what you want is if a new Consumer is made the selected messages are delivered to you as quickly as possible. You might want to replay messages at the rate they arrived though, meaning if messages first arrived 1 minute apart and you make a new Consumer it will get the messages a minute apart. @@ -313,7 +313,7 @@ This is useful in load testing scenarios etc. This is called the `ReplayPolicy` You can only set `ReplayPolicy` on push-based Consumers. -``` +```text $ nats con add ORDERS REPLAY --target out.original --filter ORDERS.processed --ack none --deliver all --sample 100 --replay original ... Replay Policy: original @@ -322,7 +322,7 @@ $ nats con add ORDERS REPLAY --target out.original --filter ORDERS.processed --a Now lets publish messages into the Set 10 seconds apart: -``` +```text $ for i in 1 2 3 <15:15:35 do nats pub ORDERS.processed "order ${i}" @@ -335,7 +335,7 @@ Published [ORDERS.processed] : 'order 3' And when we consume them they will come to us 10 seconds apart: -``` +```text $ nats sub -t out.original Listening on [out.original] 2020/01/03 15:17:26 [#1] Received on [ORDERS.processed]: 'order 1' @@ -344,11 +344,11 @@ Listening on [out.original] ^C ``` -### Stream Templates +## Stream Templates When you have many similar streams it can be helpful to auto create them, lets say you have a service by client and they are on subjects `CLIENT.*`, you can construct a template that will auto generate streams for any matching traffic. -``` +```text $ nats str template add CLIENTS --subjects "CLIENT.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size 2048 --max-streams 1024 --discard old Stream Template CLIENTS was created @@ -374,13 +374,13 @@ Managed Streams: You can see no streams currently exist, let's publish some data: -``` +```text $ nats pub CLIENT.acme hello ``` And we'll have 1 new Stream: -``` +```text $ nats str ls Streams: @@ -389,7 +389,7 @@ Streams: When the template is deleted all the streams it created will be deleted too. -### Ack Sampling +## Ack Sampling In the earlier sections we saw that samples are being sent to a monitoring system. Let's look at that in depth; how the monitoring system works and what it contains. @@ -397,24 +397,24 @@ As messages pass through an Consumer you'd be interested in knowing how many are Consumers can sample Ack'ed messages for you and publish samples so your monitoring system can observe the health of an Consumer. We will add support for this to [NATS Surveyor](https://github.com/nats-io/nats-surveyor). -#### Configuration +### Configuration You can configure an Consumer for sampling by passing the `--sample 80` option to `nats consumer add`, this tells the system to sample 80% of Acknowledgements. When viewing info of an Consumer you can tell if it's sampled or not: -``` +```text $ nats con info ORDERS NEW ... Sampling Rate: 100 ... ``` -#### Consuming +### Consuming Samples are published to `$JS.EVENT.METRIC.CONSUMER_ACK..` in JSON format containing `api.ConsumerAckMetric`. Use the `nats con events` command to view samples: -```nohighlight +```text $ nats con events ORDERS NEW Listening for Advisories on $JS.EVENT.ADVISORY.*.ORDERS.NEW Listening for Metrics on $JS.EVENT.METRIC.*.ORDERS.NEW @@ -427,7 +427,7 @@ Listening for Metrics on $JS.EVENT.METRIC.*.ORDERS.NEW Delay: 1.009ms ``` -```nohighlight +```text $ nats con events ORDERS NEW --json { "stream": "ORDERS", @@ -447,7 +447,7 @@ $ nats con events ORDERS NEW --json } ``` -### Storage Overhead +## Storage Overhead JetStream file storage is very efficient storing as little extra information about the message as possible. @@ -455,17 +455,17 @@ JetStream file storage is very efficient storing as little extra information abo We do store some message data with each message, namely: - * Message headers - * The subject it was received on - * The time it was received - * The message payload - * A hash of the message - * The message sequence - * A few other bits like length of the subject and lengh of headers +* Message headers +* The subject it was received on +* The time it was received +* The message payload +* A hash of the message +* The message sequence +* A few other bits like length of the subject and lengh of headers Without any headers the size is: -``` +```text length of the message record (4bytes) + seq(8) + ts(8) + subj_len(2) + subj + msg + hash(8) ``` @@ -473,9 +473,9 @@ A 5 byte `hello` message without headers will take 39 bytes. With headers: -``` +```text length of the message record (4bytes) + seq(8) + ts(8) + subj_len(2) + subj + hdr_len(4) + hdr + msg + hash(8) ``` -So if you are publishing many small messages the overhead will be, relatively speaking, quite large, but for larger messages -the overhead is very small. If you publish many small messages it's worth trying to optimise the subject length. +So if you are publishing many small messages the overhead will be, relatively speaking, quite large, but for larger messages the overhead is very small. If you publish many small messages it's worth trying to optimise the subject length. + diff --git a/jetstream/model_deep_dive/ack-sampling.md b/jetstream/model_deep_dive/ack-sampling.md new file mode 100644 index 0000000..eedc01d --- /dev/null +++ b/jetstream/model_deep_dive/ack-sampling.md @@ -0,0 +1,2 @@ +# Ack Sampling + diff --git a/jetstream/model_deep_dive/acknowledgment-models.md b/jetstream/model_deep_dive/acknowledgment-models.md new file mode 100644 index 0000000..1b1561d --- /dev/null +++ b/jetstream/model_deep_dive/acknowledgment-models.md @@ -0,0 +1,2 @@ +# Acknowledgment Models + diff --git a/jetstream/model_deep_dive/consumer-message-rates.md b/jetstream/model_deep_dive/consumer-message-rates.md new file mode 100644 index 0000000..f3401a1 --- /dev/null +++ b/jetstream/model_deep_dive/consumer-message-rates.md @@ -0,0 +1,2 @@ +# Consumer Message Rates + diff --git a/jetstream/model_deep_dive/consumer-starting-position.md b/jetstream/model_deep_dive/consumer-starting-position.md new file mode 100644 index 0000000..2e2b6b8 --- /dev/null +++ b/jetstream/model_deep_dive/consumer-starting-position.md @@ -0,0 +1,2 @@ +# Consumer Starting Position + diff --git a/jetstream/model_deep_dive/ephemeral-consumers.md b/jetstream/model_deep_dive/ephemeral-consumers.md new file mode 100644 index 0000000..f12004a --- /dev/null +++ b/jetstream/model_deep_dive/ephemeral-consumers.md @@ -0,0 +1,2 @@ +# Ephemeral Consumers + diff --git a/jetstream/model_deep_dive/exactly-once-delivery.md b/jetstream/model_deep_dive/exactly-once-delivery.md new file mode 100644 index 0000000..2bc6b24 --- /dev/null +++ b/jetstream/model_deep_dive/exactly-once-delivery.md @@ -0,0 +1,2 @@ +# Exactly Once Delivery + diff --git a/jetstream/model_deep_dive/message-deduplication.md b/jetstream/model_deep_dive/message-deduplication.md new file mode 100644 index 0000000..98ab7f5 --- /dev/null +++ b/jetstream/model_deep_dive/message-deduplication.md @@ -0,0 +1,2 @@ +# Message Deduplication + diff --git a/jetstream/model_deep_dive/storage-overhead.md b/jetstream/model_deep_dive/storage-overhead.md new file mode 100644 index 0000000..bc9a5e4 --- /dev/null +++ b/jetstream/model_deep_dive/storage-overhead.md @@ -0,0 +1,2 @@ +# Storage Overhead + diff --git a/jetstream/model_deep_dive/stream-limits-retention-modes-and-discard-policy.md b/jetstream/model_deep_dive/stream-limits-retention-modes-and-discard-policy.md new file mode 100644 index 0000000..00be723 --- /dev/null +++ b/jetstream/model_deep_dive/stream-limits-retention-modes-and-discard-policy.md @@ -0,0 +1,2 @@ +# Stream Limits, Retention Modes and Discard Policy + diff --git a/jetstream/model_deep_dive/stream-templates.md b/jetstream/model_deep_dive/stream-templates.md new file mode 100644 index 0000000..95ed438 --- /dev/null +++ b/jetstream/model_deep_dive/stream-templates.md @@ -0,0 +1,2 @@ +# Stream Templates + diff --git a/jetstream/monitoring.md b/jetstream/monitoring.md new file mode 100644 index 0000000..f647bc1 --- /dev/null +++ b/jetstream/monitoring.md @@ -0,0 +1,35 @@ +# Monitoring + +## Server Metrics + +Typically, NATS is monitored via HTTP endpoints like `/varz`, we do not at this moment have a JetStream equivelant, but it's planned that server and account level metrics will be made available. + +## Advisories + +JetStream publish a number of advisories that can inform operations about health and state of the Streams. These advisories are published to normal NATS subjects below `$JS.EVENT.ADVISORY.>` and one can store these advisories in JetStream Streams if desired. + +The command `nats event --js-advisory` can view all these events on your console. The Golang package [jsm.go](https://github.com/nats-io/jsm.go) can consume and render these events and have data types for each of these events. + +All these events have JSON Schemas that describe them, schemas can be viewed on the CLI using the `nats schema show ` command. + +| Description | Subject | Kind | +| :--- | :--- | :--- | +| API interactions | `$JS.EVENT.ADVISORY.API` | `io.nats.jetstream.advisory.v1.api_audit` | +| Stream CRUD operations | `$JS.EVENT.ADVISORY.STREAM.CREATED.` | `io.nats.jetstream.advisory.v1.stream_action` | +| Consumer CRUD operations | `$JS.EVENT.ADVISORY.CONSUMER.CREATED..` | `io.nats.jetstream.advisory.v1.consumer_action` | +| Snapshot started using `nats stream backup` | `$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_CREATE.` | `io.nats.jetstream.advisory.v1.snapshot_create` | +| Snapshot completed | `$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_COMPLETE.` | `io.nats.jetstream.advisory.v1.snapshot_complete` | +| Restore started using `nats stream restore` | `$JS.EVENT.ADVISORY.STREAM.RESTORE_CREATE.` | `io.nats.jetstream.advisory.v1.restore_create` | +| Restore completed | `$JS.EVENT.ADVISORY.STREAM.RESTORE_COMPLETE.` | `io.nats.jetstream.advisory.v1.restore_complete` | +| Consumer maximum delivery reached | `$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES..` | `io.nats.jetstream.advisory.v1.max_deliver` | +| Message delivery terminated using AckTerm | `$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED..` | `io.nats.jetstream.advisory.v1.terminated` | +| Message acknowledged in a sampled Consumer | `$JS.EVENT.METRIC.CONSUMER.ACK..` | `io.nats.jetstream.metric.v1.consumer_ack` | +| Clustered Stream elected a new leader | `$JS.EVENT.ADVISORY.STREAM.LEADER_ELECTED.` | `io.nats.jetstream.advisory.v1.stream_leader_elected` | +| Clustered Stream lost quorum | `$JS.EVENT.ADVISORY.STREAM.QUORUM_LOST.` | `io.nats.jetstream.advisory.v1.stream_quorum_lost` | +| Clustered Consumer elected a new leader | `$JS.EVENT.ADVISORY.CONSUMER.LEADER_ELECTED..` | `io.nats.jetstream.advisory.v1.consumer_leader_elected` | +| Clustered Consumer lost quorum | `$JS.EVENT.ADVISORY.CONSUMER.QUORUM_LOST..` | `io.nats.jetstream.advisory.v1.consumer_quorum_lost` | + +## Dashboards + +The [NATS Surveyor](https://github.com/nats-io/nats-surveyor) system has initial support for passing JetStream metrics to Prometheus, dashboards and more will be added towards final release. + diff --git a/jetstream/monitoring/monitoring.md b/jetstream/monitoring/monitoring.md deleted file mode 100644 index f95e510..0000000 --- a/jetstream/monitoring/monitoring.md +++ /dev/null @@ -1,34 +0,0 @@ -## Monitoring - -### Server Metrics - -Typically, NATS is monitored via HTTP endpoints like `/varz`, we do not at this moment have a JetStream equivelant, but it's planned that server and account level metrics will be made available. - -### Advisories - -JetStream publish a number of advisories that can inform operations about health and state of the Streams. These advisories are published to normal NATS subjects below `$JS.EVENT.ADVISORY.>` and one can store these advisories in JetStream Streams if desired. - -The command `nats event --js-advisory` can view all these events on your console. The Golang package [jsm.go](https://github.com/nats-io/jsm.go) can consume and render these events and have data types for each of these events. - -All these events have JSON Schemas that describe them, schemas can be viewed on the CLI using the `nats schema show ` command. - -|Description|Subject|Kind| -|-----------|-------|----| -|API interactions|`$JS.EVENT.ADVISORY.API`|`io.nats.jetstream.advisory.v1.api_audit`| -|Stream CRUD operations|`$JS.EVENT.ADVISORY.STREAM.CREATED.`|`io.nats.jetstream.advisory.v1.stream_action`| -|Consumer CRUD operations|`$JS.EVENT.ADVISORY.CONSUMER.CREATED..`|`io.nats.jetstream.advisory.v1.consumer_action`| -|Snapshot started using `nats stream backup`|`$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_CREATE.`|`io.nats.jetstream.advisory.v1.snapshot_create`| -|Snapshot completed|`$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_COMPLETE.`|`io.nats.jetstream.advisory.v1.snapshot_complete`| -|Restore started using `nats stream restore`|`$JS.EVENT.ADVISORY.STREAM.RESTORE_CREATE.`|`io.nats.jetstream.advisory.v1.restore_create`| -|Restore completed|`$JS.EVENT.ADVISORY.STREAM.RESTORE_COMPLETE.`|`io.nats.jetstream.advisory.v1.restore_complete`| -|Consumer maximum delivery reached|`$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES..`|`io.nats.jetstream.advisory.v1.max_deliver`| -|Message delivery terminated using AckTerm|`$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED..`|`io.nats.jetstream.advisory.v1.terminated`| -|Message acknowledged in a sampled Consumer|`$JS.EVENT.METRIC.CONSUMER.ACK..`|`io.nats.jetstream.metric.v1.consumer_ack`| -|Clustered Stream elected a new leader|`$JS.EVENT.ADVISORY.STREAM.LEADER_ELECTED.`|`io.nats.jetstream.advisory.v1.stream_leader_elected`| -|Clustered Stream lost quorum|`$JS.EVENT.ADVISORY.STREAM.QUORUM_LOST.`|`io.nats.jetstream.advisory.v1.stream_quorum_lost` -|Clustered Consumer elected a new leader|`$JS.EVENT.ADVISORY.CONSUMER.LEADER_ELECTED..`|`io.nats.jetstream.advisory.v1.consumer_leader_elected`| -|Clustered Consumer lost quorum|`$JS.EVENT.ADVISORY.CONSUMER.QUORUM_LOST..`|`io.nats.jetstream.advisory.v1.consumer_quorum_lost`| - -### Dashboards - -The [NATS Surveyor](https://github.com/nats-io/nats-surveyor) system has initial support for passing JetStream metrics to Prometheus, dashboards and more will be added towards final release. diff --git a/jetstream/nats_api_reference/nats_api_reference.md b/jetstream/nats_api_reference.md similarity index 61% rename from jetstream/nats_api_reference/nats_api_reference.md rename to jetstream/nats_api_reference.md index 4d35174..610e552 100644 --- a/jetstream/nats_api_reference/nats_api_reference.md +++ b/jetstream/nats_api_reference.md @@ -1,18 +1,18 @@ -## NATS API Reference +# NATS API Reference Thus far we saw a lot of CLI interactions. The CLI works by sending and receiving specially crafted messages over core NATS to configure the JetStream system. In time we will look to add file based configuration but for now the only method is the NATS API. **NOTE:** Some NATS client libraries may need to enable an option to use old style requests when interacting withe JetStream server. Consult the libraries README's for more information. -### Reference +## Reference All of these subjects are found as constants in the NATS Server source, so for example the `$JS.API.STREAM.LIST` is a constant in the nats-server source `api.JetStreamListStreams` tables below will reference these constants and likewise data structures in the server for payloads. -### Error Handling +## Error Handling The APIs used for administrative tools all respond with standardised JSON and these include errors. -```nohighlight +```text $ nats req '$JS.API.STREAM.INFO.nonexisting' '' Published 11 bytes to $JS.API.STREAM.INFO.nonexisting Received [_INBOX.lcWgjX2WgJLxqepU0K9pNf.mpBW9tHK] : { @@ -24,7 +24,7 @@ Received [_INBOX.lcWgjX2WgJLxqepU0K9pNf.mpBW9tHK] : { } ``` -```nohighlight +```text $ nats req '$JS.STREAM.INFO.ORDERS' '' Published 6 bytes to $JS.STREAM.INFO.ORDERS Received [_INBOX.fwqdpoWtG8XFXHKfqhQDVA.vBecyWmF] : '{ @@ -39,9 +39,9 @@ Here the responses include a `type` which can be used to find the JSON Schema fo Non admin APIs - like those for adding a message to the stream will respond with `-ERR` or `+OK` with an optional reason after. -### Admin API +## Admin API -All the admin actions the `nats` CLI can do falls in the sections below. The API structure are kept in the `api` package in the `jsm.go` repository. +All the admin actions the `nats` CLI can do falls in the sections below. The API structure are kept in the `api` package in the `jsm.go` repository. Subjects that and in `T` like `api.JSApiConsumerCreateT` are formats and would need to have the Stream Name and in some cases also the Consumer name interpolated into them. In this case `t := fmt.Sprintf(api.JSApiConsumerCreateT, streamName)` to get the final subject. @@ -49,61 +49,61 @@ The command `nats events` will show you an audit log of all API access events wh The API uses JSON for inputs and outputs, all the responses are typed using a `type` field which indicates their Schema. A JSON Schema repository can be found in `nats-io/jetstream/schemas`. -#### General Info +### General Info -|Subject|Constant|Description|Request Payload|Response Payload| -|-------|--------|-----------|---------------|----------------| -|`$JS.API.INFO`|`api.JSApiAccountInfo`|Retrieves stats and limits about your account|empty payload|`api.JetStreamAccountStats`| +| Subject | Constant | Description | Request Payload | Response Payload | +| :--- | :--- | :--- | :--- | :--- | +| `$JS.API.INFO` | `api.JSApiAccountInfo` | Retrieves stats and limits about your account | empty payload | `api.JetStreamAccountStats` | -#### Streams +### Streams -|Subject|Constant|Description|Request Payload|Response Payload| -|-------|--------|-----------|---------------|----------------| -|`$JS.API.STREAM.LIST`|`api.JSApiStreamList`|Paged list known Streams including all their current information|`api.JSApiStreamListRequest`|`api.JSApiStreamListResponse`| -|`$JS.API.STREAM.NAMES`|`api.JSApiStreamNames`|Paged list of Streams|`api.JSApiStreamNamesRequest`|`api.JSApiStreamNamesResponse`| -|`$JS.API.STREAM.CREATE.*`|`api.JSApiStreamCreateT`|Creates a new Stream|`api.StreamConfig`|`api.JSApiStreamCreateResponse`| -|`$JS.API.STREAM.UPDATE.*`|`api.JSApiStreamUpdateT`|Updates an existing Stream with new config|`api.StreamConfig`|`api.JSApiStreamUpdateResponse`| -|`$JS.API.STREAM.INFO.*`|`api.JSApiStreamInfoT`|Information about config and state of a Stream|empty payload, Stream name in subject|`api.JSApiStreamInfoResponse`| -|`$JS.API.STREAM.DELETE.*`|`api.JSApiStreamDeleteT`|Deletes a Stream and all its data|empty payload, Stream name in subject|`api.JSApiStreamDeleteResponse`| -|`$JS.API.STREAM.PURGE.*`|`api.JSApiStreamPurgeT`|Purges all of the data in a Stream, leaves the Stream|empty payload, Stream name in subject|`api.JSApiStreamPurgeResponse`| -|`$JS.API.STREAM.MSG.DELETE.*`|`api.JSApiMsgDeleteT`|Deletes a specific message in the Stream by sequence, useful for GDPR compliance|`api.JSApiMsgDeleteRequest`|`api.JSApiMsgDeleteResponse`| -|`$JS.API.STREAM.MSG.GET.*`|`api.JSApiMsgGetT`|Retrieves a specific message from the stream|`api.JSApiMsgGetRequest`|`api.JSApiMsgGetResponse`| -|`$JS.API.STREAM.SNAPSHOT.*`|`api.JSApiStreamSnapshotT`|Initiates a streaming backup of a streams data|`api.JSApiStreamSnapshotRequest`|`api.JSApiStreamSnapshotResponse`| -|`$JS.API.STREAM.RESTORE.*`|`api.JSApiStreamRestoreT`|Initiates a streaming restore of a stream|`{}`|`api.JSApiStreamRestoreResponse`| +| Subject | Constant | Description | Request Payload | Response Payload | +| :--- | :--- | :--- | :--- | :--- | +| `$JS.API.STREAM.LIST` | `api.JSApiStreamList` | Paged list known Streams including all their current information | `api.JSApiStreamListRequest` | `api.JSApiStreamListResponse` | +| `$JS.API.STREAM.NAMES` | `api.JSApiStreamNames` | Paged list of Streams | `api.JSApiStreamNamesRequest` | `api.JSApiStreamNamesResponse` | +| `$JS.API.STREAM.CREATE.*` | `api.JSApiStreamCreateT` | Creates a new Stream | `api.StreamConfig` | `api.JSApiStreamCreateResponse` | +| `$JS.API.STREAM.UPDATE.*` | `api.JSApiStreamUpdateT` | Updates an existing Stream with new config | `api.StreamConfig` | `api.JSApiStreamUpdateResponse` | +| `$JS.API.STREAM.INFO.*` | `api.JSApiStreamInfoT` | Information about config and state of a Stream | empty payload, Stream name in subject | `api.JSApiStreamInfoResponse` | +| `$JS.API.STREAM.DELETE.*` | `api.JSApiStreamDeleteT` | Deletes a Stream and all its data | empty payload, Stream name in subject | `api.JSApiStreamDeleteResponse` | +| `$JS.API.STREAM.PURGE.*` | `api.JSApiStreamPurgeT` | Purges all of the data in a Stream, leaves the Stream | empty payload, Stream name in subject | `api.JSApiStreamPurgeResponse` | +| `$JS.API.STREAM.MSG.DELETE.*` | `api.JSApiMsgDeleteT` | Deletes a specific message in the Stream by sequence, useful for GDPR compliance | `api.JSApiMsgDeleteRequest` | `api.JSApiMsgDeleteResponse` | +| `$JS.API.STREAM.MSG.GET.*` | `api.JSApiMsgGetT` | Retrieves a specific message from the stream | `api.JSApiMsgGetRequest` | `api.JSApiMsgGetResponse` | +| `$JS.API.STREAM.SNAPSHOT.*` | `api.JSApiStreamSnapshotT` | Initiates a streaming backup of a streams data | `api.JSApiStreamSnapshotRequest` | `api.JSApiStreamSnapshotResponse` | +| `$JS.API.STREAM.RESTORE.*` | `api.JSApiStreamRestoreT` | Initiates a streaming restore of a stream | `{}` | `api.JSApiStreamRestoreResponse` | -#### Stream Templates +### Stream Templates -|Subject|Constant|Description|Request Payload|Response Payload| -|-------|--------|-----------|---------------|----------------| -|`$JS.API.STREAM.TEMPLATE.CREATE.*`|`api.JSApiTemplateCreateT`|Creates a Stream Template|`api.StreamTemplateConfig`|`api.JSApiStreamTemplateCreateResponse`| -|`$JS.API.STREAM.TEMPLATE.NAMES`|`api.JSApiTemplateNames`|Paged list all known templates|`api.JSApiStreamTemplateNamesRequest`|`api.JSApiStreamTemplateNamesResponse`| -|`$JS.API.STREAM.TEMPLATE.INFO.*`|`api.JSApiTemplateInfoT`|Information about the config and state of a Stream Template|empty payload, Template name in subject|`api.JSApiStreamTemplateInfoResponse`| -|`$JS.API.STREAM.TEMPLATE.DELETE.*`|`api.JSApiTemplateDeleteT`|Delete a specific Stream Template **and all streams created by this template**|empty payload, Template name in subject|`api.JSApiStreamTemplateDeleteResponse`| +| Subject | Constant | Description | Request Payload | Response Payload | +| :--- | :--- | :--- | :--- | :--- | +| `$JS.API.STREAM.TEMPLATE.CREATE.*` | `api.JSApiTemplateCreateT` | Creates a Stream Template | `api.StreamTemplateConfig` | `api.JSApiStreamTemplateCreateResponse` | +| `$JS.API.STREAM.TEMPLATE.NAMES` | `api.JSApiTemplateNames` | Paged list all known templates | `api.JSApiStreamTemplateNamesRequest` | `api.JSApiStreamTemplateNamesResponse` | +| `$JS.API.STREAM.TEMPLATE.INFO.*` | `api.JSApiTemplateInfoT` | Information about the config and state of a Stream Template | empty payload, Template name in subject | `api.JSApiStreamTemplateInfoResponse` | +| `$JS.API.STREAM.TEMPLATE.DELETE.*` | `api.JSApiTemplateDeleteT` | Delete a specific Stream Template **and all streams created by this template** | empty payload, Template name in subject | `api.JSApiStreamTemplateDeleteResponse` | -#### Consumers +### Consumers -|Subject|Constant|Description|Request Payload|Response Payload| -|-------|--------|-----------|---------------|----------------| -|`$JS.API.CONSUMER.CREATE.*`|`api.JSApiConsumerCreateT`|Create an ephemeral Consumer|`api.ConsumerConfig`, Stream name in subject|`api.JSApiConsumerCreateResponse`| -|`$JS.API.CONSUMER.DURABLE.CREATE.*`|`api.JSApiDurableCreateT`|Create an Consumer|`api.ConsumerConfig`, Stream name in subject|`api.JSApiConsumerCreateResponse`| -|`$JS.API.CONSUMER.LIST.*`|`api.JSApiConsumerListT`|Paged list of known Consumers including their current info|`api.JSApiConsumerListRequest`|`api.JSApiConsumerListResponse`| -|`$JS.API.CONSUMER.NAMES.*`|`api.JSApiConsumerNamesT`|Paged list of known Consumer names|`api.JSApiConsumerNamesRequest`|`api.JSApiConsumerNamesResponse`| -|`$JS.API.CONSUMER.INFO.*.*`|`api.JSApiConsumerInfoT`|Information about an Consumer|empty payload, Stream and Consumer names in subject|`api.JSApiConsumerInfoResponse`| -|`$JS.API.CONSUMER.DELETE.*.*`|`api.JSApiConsumerDeleteT`|Deletes an Consumer|empty payload, Stream and Consumer names in subject|`api.JSApiConsumerDeleteResponse`| +| Subject | Constant | Description | Request Payload | Response Payload | +| :--- | :--- | :--- | :--- | :--- | +| `$JS.API.CONSUMER.CREATE.*` | `api.JSApiConsumerCreateT` | Create an ephemeral Consumer | `api.ConsumerConfig`, Stream name in subject | `api.JSApiConsumerCreateResponse` | +| `$JS.API.CONSUMER.DURABLE.CREATE.*` | `api.JSApiDurableCreateT` | Create an Consumer | `api.ConsumerConfig`, Stream name in subject | `api.JSApiConsumerCreateResponse` | +| `$JS.API.CONSUMER.LIST.*` | `api.JSApiConsumerListT` | Paged list of known Consumers including their current info | `api.JSApiConsumerListRequest` | `api.JSApiConsumerListResponse` | +| `$JS.API.CONSUMER.NAMES.*` | `api.JSApiConsumerNamesT` | Paged list of known Consumer names | `api.JSApiConsumerNamesRequest` | `api.JSApiConsumerNamesResponse` | +| `$JS.API.CONSUMER.INFO.*.*` | `api.JSApiConsumerInfoT` | Information about an Consumer | empty payload, Stream and Consumer names in subject | `api.JSApiConsumerInfoResponse` | +| `$JS.API.CONSUMER.DELETE.*.*` | `api.JSApiConsumerDeleteT` | Deletes an Consumer | empty payload, Stream and Consumer names in subject | `api.JSApiConsumerDeleteResponse` | -#### ACLs +### ACLs It's hard to notice here but there is a clear pattern in these subjects, lets look at the various JetStream related subjects: General information -``` +```text $JS.API.INFO ``` Stream and Consumer Admin -``` +```text $JS.API.STREAM.CREATE. $JS.API.STREAM.UPDATE. $JS.API.STREAM.DELETE. @@ -130,7 +130,7 @@ $JS.API.STREAM.TEMPLATE.NAMES Stream and Consumer Use -``` +```text $JS.API.CONSUMER.MSG.NEXT.. $JS.ACK...x.x.x $JS.SNAPSHOT.ACK.. @@ -139,7 +139,7 @@ $JS.SNAPSHOT.RESTORE.. Events and Advisories: -``` +```text $JS.EVENT.METRIC.CONSUMER_ACK.. $JS.EVENT.ADVISORY.MAX_DELIVERIES.. $JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.. @@ -161,17 +161,17 @@ $JS.EVENT.ADVISORY.API This design allows you to easily create ACL rules that limit users to a specific Stream or Consumer and to specific verbs for administration purposes. For ensuring only the receiver of a message can Ack it we have response permissions ensuring you can only Publish to Response subject for messages you received. -### Acknowledging Messages +## Acknowledging Messages Messages that need acknowledgement will have a Reply subject set, something like `$JS.ACK.ORDERS.test.1.2.2`, this is the prefix defined in `api.JetStreamAckPre` followed by `......`. -In all the Synadia maintained API's you can simply do `msg.Respond(nil)` (or language equivalent) which will send nil to the reply subject. +In all the Synadia maintained API's you can simply do `msg.Respond(nil)` \(or language equivalent\) which will send nil to the reply subject. -### Fetching The Next Message From a Pull-based Consumer +## Fetching The Next Message From a Pull-based Consumer If you have a pull-based Consumer you can send a standard NATS Request to `$JS.API.CONSUMER.MSG.NEXT..`, here the format is defined in `api.JetStreamRequestNextT` and requires populating using `fmt.Sprintf()`. -```nohighlight +```text $ nats req '$JS.API.CONSUMER.MSG.NEXT.ORDERS.test' '1' Published 1 bytes to $JS.API.CONSUMER.MSG.NEXT.ORDERS.test Received [js.1] : 'message 1' @@ -183,7 +183,7 @@ The above request for the next message will stay in the server for as long as th This is often not desired, pull consumers support a mode where a JSON document is sent describing the pull request. -```json +```javascript { "expires": "2020-11-10T12:41:00.075933464Z", "batch": 10, @@ -192,7 +192,7 @@ This is often not desired, pull consumers support a mode where a JSON document i This requests 10 messages and asks the server to keep this request until the specific `expires` time, this is useful when you poll the server frequently and do not want the pull requests to accumulate on the server. Set the expire time to now + your poll frequency. -```json +```javascript { "batch": 10, "no_wait": true @@ -201,7 +201,7 @@ This requests 10 messages and asks the server to keep this request until the spe Here we see a second format of the Pull request that will not store the request on the queue at all but when there are no messages to deliver will send a nil bytes message with a `Status` header of `404`, this way you can know when you reached the end of the stream for example. A `409` is returned if the Consumer has reached `MaxAckPending` limits. -``` +```text [rip@dev1]% nats req '$JS.API.CONSUMER.MSG.NEXT.ORDERS.NEW' '{"no_wait": true, "batch": 10}' test --password test 13:45:30 Sending request on "$JS.API.CONSUMER.MSG.NEXT.ORDERS.NEW" @@ -210,11 +210,11 @@ Here we see a second format of the Pull request that will not store the request 13:45:30 Description: No Messages ``` -### Fetching From a Stream By Sequence +## Fetching From a Stream By Sequence -If you know the Stream sequence of a message you can fetch it directly, this does not support acks. Do a Request() to `$JS.API.STREAM.MSG.GET.ORDERS` sending it the message sequence as payload. Here the prefix is defined in `api.JetStreamMsgBySeqT` which also requires populating using `fmt.Sprintf()`. +If you know the Stream sequence of a message you can fetch it directly, this does not support acks. Do a Request\(\) to `$JS.API.STREAM.MSG.GET.ORDERS` sending it the message sequence as payload. Here the prefix is defined in `api.JetStreamMsgBySeqT` which also requires populating using `fmt.Sprintf()`. -```nohighlight +```text $ nats req '$JS.API.STREAM.MSG.GET.ORDERS' '{"seq": 1}' Published 1 bytes to $JS.STREAM.ORDERS.MSG.BYSEQ Received [_INBOX.cJrbzPJfZrq8NrFm1DsZuH.k91Gb4xM] : '{ @@ -230,6 +230,7 @@ Received [_INBOX.cJrbzPJfZrq8NrFm1DsZuH.k91Gb4xM] : '{ The Subject shows where the message was received, Data is base64 encoded and Time is when it was received. -### Consumer Samples +## Consumer Samples + +Samples are published to a specific subject per Consumer, something like `$JS.EVENT.METRIC.CONSUMER_ACK..` you can just subscribe to that and get `api.ConsumerAckMetric` messages in JSON format. The prefix is defined in `api.JetStreamMetricConsumerAckPre`. -Samples are published to a specific subject per Consumer, something like `$JS.EVENT.METRIC.CONSUMER_ACK..` you can just subscribe to that and get `api.ConsumerAckMetric` messages in JSON format. The prefix is defined in `api.JetStreamMetricConsumerAckPre`. diff --git a/jetstream/data_replication/replication.md b/jetstream/replication.md similarity index 85% rename from jetstream/data_replication/replication.md rename to jetstream/replication.md index 7561645..96f6768 100644 --- a/jetstream/data_replication/replication.md +++ b/jetstream/replication.md @@ -1,32 +1,32 @@ -## Data Replication +# Data Replication -Replication allows you to move data between streams in either a 1:1 mirror style or by multiplexing multiple source streams into a new stream. In future builds this will allow data to be replicated between accounts as well, ideal for sending data from a Leafnode into a central store. +Replication allows you to move data between streams in either a 1:1 mirror style or by multiplexing multiple source streams into a new stream. In future builds this will allow data to be replicated between accounts as well, ideal for sending data from a Leafnode into a central store. -![](../../assets/images/replication.png) +![](../.gitbook/assets/replication.png) Here we have 2 main streams - _ORDERS_ and _RETURNS_ - these streams are clustered across 3 nodes. These Streams have short retention periods and are memory based. -We create a _ARCHIVE_ stream that has 2 _sources_ set, the _ARCHIVE_ will pull data from the sources into itself. This stream has a very long retention period and is file based and replicated across 3 nodes. Additional messages can be added to the ARCHIVE by sending to it directly. +We create a _ARCHIVE_ stream that has 2 _sources_ set, the _ARCHIVE_ will pull data from the sources into itself. This stream has a very long retention period and is file based and replicated across 3 nodes. Additional messages can be added to the ARCHIVE by sending to it directly. -Finally, we create a _REPORT_ stream mirrored from _ARCHIVE_ that is not clustered and retains data for a month. The _REPORT_ Stream does not listen for any incoming messages, it can only consume data from _ARCHIVE_. +Finally, we create a _REPORT_ stream mirrored from _ARCHIVE_ that is not clustered and retains data for a month. The _REPORT_ Stream does not listen for any incoming messages, it can only consume data from _ARCHIVE_. -### Mirrors +## Mirrors -A *mirror* copies data from 1 other stream, as far as possible IDs and ordering will match exactly the source. A *mirror* does not listen on a subject for any data to be added. The Start Sequence and Start Time can be set, but no subject filter. A stream can only have 1 *mirror* and if it is a mirror it cannot also have any *source*. +A _mirror_ copies data from 1 other stream, as far as possible IDs and ordering will match exactly the source. A _mirror_ does not listen on a subject for any data to be added. The Start Sequence and Start Time can be set, but no subject filter. A stream can only have 1 _mirror_ and if it is a mirror it cannot also have any _source_. -### Sources +## Sources -A *source* is a stream where data is copied from, one stream can have multiple sources and will read data in from them all. The stream will also listen for messages on it's own subject. We can therefore not maintain absolute ordering, but data from 1 single source will be in the correct order but mixed in with other streams. You might also find the timestamps of streams can be older and newer mixed in together as a result. +A _source_ is a stream where data is copied from, one stream can have multiple sources and will read data in from them all. The stream will also listen for messages on it's own subject. We can therefore not maintain absolute ordering, but data from 1 single source will be in the correct order but mixed in with other streams. You might also find the timestamps of streams can be older and newer mixed in together as a result. -A Stream with sources may also listen on subjects, but could have no listening subject. When using the `nats` CLI to create sourced streams use `--subjects` to supply subjects to listen on. +A Stream with sources may also listen on subjects, but could have no listening subject. When using the `nats` CLI to create sourced streams use `--subjects` to supply subjects to listen on. -A source can have Start Time or Start Sequence and can filter by a subject. +A source can have Start Time or Start Sequence and can filter by a subject. -### Configuration +## Configuration The ORDERS and RETURNS streams as normal, I will not show how to create them. -```nohighlight +```text $ nats s report Obtaining Stream stats @@ -40,7 +40,7 @@ Obtaining Stream stats We now add the ARCHIVE: -```nohighlight +```text $ nats s add ARCHIVE --source ORDERS --source RETURNS ? Storage backend file ? Retention Policy Limits @@ -61,7 +61,7 @@ $ nats s add ARCHIVE --source ORDERS --source RETURNS And we add the REPORT: -```nohighlight +```text $ nats s add REPORT --mirror ARCHIVE ? Storage backend file ? Retention Policy Limits @@ -79,7 +79,7 @@ $ nats s add REPORT --mirror ARCHIVE When configured we'll see some additional information in a `nats stream info` output: -```nohighlight +```text $ nats stream info ARCHIVE ... Source Information: @@ -107,7 +107,7 @@ Here the `Lag` is how far behind we were reported as being last time we saw a me We can confirm all our setup using a `nats stream report`: -```nohighlight +```text $ nats s report +-------------------------------------------------------------------------------------------------------------------+ | Stream Report | @@ -133,14 +133,14 @@ $ nats s report We then create some data in both ORDERS and RETURNS: -```nohighlight +```text $ nats req ORDERS.new "ORDER {{Count}}" --count 100 $ nats req RETURNS.new "RETURN {{Count}}" --count 100 ``` We can now see from a Stream Report that the data has been replicated: -```nohighlight +```text $ nats s report --dot replication.dot Obtaining Stream stats @@ -166,4 +166,5 @@ Obtaining Stream stats Here we also pass the `--dot replication.dot` argument that writes a GraphViz format map of the replication setup. -![](../../assets/images/replication-setup.png) +![](../.gitbook/assets/replication-setup.png) + diff --git a/jetstream/multi-tenancy/resource_management.md b/jetstream/resource_management.md similarity index 93% rename from jetstream/multi-tenancy/resource_management.md rename to jetstream/resource_management.md index 20d9838..59f1fcc 100644 --- a/jetstream/multi-tenancy/resource_management.md +++ b/jetstream/resource_management.md @@ -1,16 +1,18 @@ +# Multi-tenancy & Resource Mgmt + ## Multi Tenancy and Resource Management JetStream is compatible with NATS 2.0 Multi Tenancy using Accounts. A JetStream enabled server supports creating fully isolated JetStream environments for different accounts. To enable JetStream in a server we have to configure it at the top level first: -``` +```text jetstream: enabled ``` This will dynamically determine the available resources. It's recommended that you set specific limits though: -``` +```text jetstream { store_dir: /data/jetstream max_mem: 1G @@ -20,7 +22,7 @@ jetstream { At this point JetStream will be enabled and if you have a server that does not have accounts enabled all users in the server would have access to JetStream -``` +```text jetstream { store_dir: /data/jetstream max_mem: 1G @@ -36,7 +38,7 @@ accounts { Here the `HR` account would have access to all the resources configured on the server, we can restrict it: -``` +```text jetstream { store_dir: /data/jetstream max_mem: 1G @@ -63,14 +65,13 @@ If you try to configure JetStream for an account without enabling it globally yo As part of the JetStream efforts a new `nats` CLI is being developed to act as a single point of access to the NATS eco system. -This CLI has been seen throughout the guide, it's available in the Docker containers today and downloadable on the [Releases](https://github.com/nats-io/jetstream/releases) -page. +This CLI has been seen throughout the guide, it's available in the Docker containers today and downloadable on the [Releases](https://github.com/nats-io/jetstream/releases) page. ### Configuration Contexts The CLI has a number of environment configuration settings - where your NATS server is, credentials, TLS keys and more: -```nohighlight +```text $ nats --help ... -s, --server=NATS_URL NATS servers @@ -86,14 +87,13 @@ $ nats --help ... ``` -You can set these using the CLI flag, the environmet variable - like **NATS_URL** - or using our context feature. +You can set these using the CLI flag, the environmet variable - like **NATS\_URL** - or using our context feature. -A context is a named configuration that stores all these settings, you can switch between access configurations and -designate a default. +A context is a named configuration that stores all these settings, you can switch between access configurations and designate a default. Creating one is easy, just specify the same settings to the `nats context save` -```nohighlight +```text $ nats context save example --server nats://nats.example.net:4222 --description 'Example.Net Server' $ nats context save local --server nats://localhost:4222 --description 'Local Host' --select $ nats context ls @@ -105,7 +105,7 @@ Known contexts: We passed `--select` to the `local` one meaning it will be the default when nothing is set. -```nohighlight +```text $ nats rtt nats://localhost:4222: @@ -126,8 +126,9 @@ All `nats` commands are context aware and the `nats context` command has various Server URLs and Credential paths can be resolved via the `nsc` command by specifying an url, for example to find user `new` within the `orders` account of the `acme` operator you can use this: -```nohighlight +```text $ nats context save example --description 'Example.Net Server' --nsc nsc://acme/orders/new ``` The server list and credentials path will now be resolved via `nsc`, if these are specifically set in the context, the specific context configuration will take precedence. + diff --git a/nats-on-kubernetes/nats-external-nlb.md b/nats-on-kubernetes/nats-external-nlb.md index dc29dcc..1a8f839 100644 --- a/nats-on-kubernetes/nats-external-nlb.md +++ b/nats-on-kubernetes/nats-external-nlb.md @@ -1,3 +1,5 @@ +# Using a Load Balancer for External Access to NATS + ## Using a Load Balancer for External Access to NATS In the example below, you can find how to use an [AWS Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) to connect externally to a cluster that has TLS setup. @@ -42,7 +44,7 @@ Also, it would be recommended to set [no\_advertise](../nats-server/configuratio With the following, you can create a 3-node NATS Server cluster: -```sh +```bash kubectl apply -f https://raw.githubusercontent.com/nats-io/k8s/b55687a97a5fd55485e1af302fbdbe43d2d3b968/nats-server/leafnodes/nats-cluster.yaml ``` @@ -85,13 +87,13 @@ data: Now let's expose the NATS Server by creating an L4 load balancer on Azure: -```sh +```bash kubectl apply -f https://raw.githubusercontent.com/nats-io/k8s/b55687a97a5fd55485e1af302fbdbe43d2d3b968/nats-server/leafnodes/lb.yaml ``` Confirm the public IP that was allocated to the `nats-lb` service that was created, in this case it is `52.155.49.45`: -``` +```text $ kubectl get svc -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR kubernetes ClusterIP 10.0.0.1 443/TCP 81d @@ -113,13 +115,13 @@ leaf { You can also add a NATS Streaming cluster into the cluster connecting to the port 4222: -```sh +```bash kubectl apply -f https://raw.githubusercontent.com/nats-io/k8s/b55687a97a5fd55485e1af302fbdbe43d2d3b968/nats-server/leafnodes/stan-server.yaml ``` Now if you create two NATS Servers that connect to the same leafnode port, they will be able to receive messages to each other: -```sh +```bash nats-server -c leafnodes/leaf.conf -p 4222 & nats-server -c leafnodes/leaf.conf -p 4223 & diff --git a/nats-on-kubernetes/stan-ft-k8s-aws.md b/nats-on-kubernetes/stan-ft-k8s-aws.md index 6e72ee6..76e3d95 100644 --- a/nats-on-kubernetes/stan-ft-k8s-aws.md +++ b/nats-on-kubernetes/stan-ft-k8s-aws.md @@ -461,3 +461,4 @@ Subscribe to get all the messages: ```bash stan-sub -c stan -all foo ``` + diff --git a/nats-server/configuration/monitoring.md b/nats-server/configuration/monitoring.md index a93734c..7d27a91 100644 --- a/nats-server/configuration/monitoring.md +++ b/nats-server/configuration/monitoring.md @@ -274,7 +274,7 @@ As noted above, the `routez` endpoint does support the `subs` argument from the #### Example -* Get JetStream information: http://host:port/jsz?accounts=1&streams=1&consumers=1&config=1 +* Get JetStream information: [http://host:port/jsz?accounts=1&streams=1&consumers=1&config=1](http://host:port/jsz?accounts=1&streams=1&consumers=1&config=1) #### Response diff --git a/nats-server/nats_admin/lame_duck_mode.md b/nats-server/nats_admin/lame_duck_mode.md index dd9438b..3c9d0c1 100644 --- a/nats-server/nats_admin/lame_duck_mode.md +++ b/nats-server/nats_admin/lame_duck_mode.md @@ -1,9 +1,6 @@ # Lame Duck Mode -In production we recommend that a server is shut down with ​lame duck mode​ -as a graceful way to slowly evict clients. With large deployments this -mitigates the "thundering herd" situation that will place CPU pressure on -servers as TLS enabled clients reconnect. +In production we recommend that a server is shut down with ​lame duck mode​ as a graceful way to slowly evict clients. With large deployments this mitigates the "thundering herd" situation that will place CPU pressure on servers as TLS enabled clients reconnect. ## Server @@ -13,16 +10,9 @@ Lame duck mode is initiated by signaling the server: nats-server --signal ldm ``` -After entering lame duck mode, the server will stop accepting new connections, -wait for a 10 second grace period, then begin to evict clients over a period of time -configurable by the [lame_duck_duration](https://docs.nats.io/nats-server/configuration#runtime-configuration) -configuration option. This period defaults to 2 minutes. +After entering lame duck mode, the server will stop accepting new connections, wait for a 10 second grace period, then begin to evict clients over a period of time configurable by the [lame\_duck\_duration](https://docs.nats.io/nats-server/configuration#runtime-configuration) configuration option. This period defaults to 2 minutes. ## Clients -When entering lame duck mode, the server will send a message to clients. Some -maintainer supported clients will invoke an optional callback indicating that -a server is entering lame duck mode. This is used for cases where an application -can benefit from preparing for the short outage between the time it is evicted and -automatically reconnected to another server. +When entering lame duck mode, the server will send a message to clients. Some maintainer supported clients will invoke an optional callback indicating that a server is entering lame duck mode. This is used for cases where an application can benefit from preparing for the short outage between the time it is evicted and automatically reconnected to another server. diff --git a/nats-server/nats_admin/upgrading_cluster.md b/nats-server/nats_admin/upgrading_cluster.md index 3339f97..0517b7e 100644 --- a/nats-server/nats_admin/upgrading_cluster.md +++ b/nats-server/nats_admin/upgrading_cluster.md @@ -38,7 +38,7 @@ nats-sub -s nats://localhost:4222 ">" `nats-sub` is a subscriber sample included with all NATS clients. `nats-sub` subscribes to a subject and prints out any messages received. You can find the source code to the go version of `nats-sub` in [GitHub](https://github.com/nats-io/nats.go/tree/master/examples). After starting the subscriber you should see a message on 'A' that a new client connected. -We have two servers and a client. Time to simulate our rolling upgrade. But wait, before we upgrade 'A', let's introduce a new server 'C'. Server 'C' will join the existing cluster while we perform the upgrade. Its sole purpose is to provide an additional place where clients can go other than 'A' and ensure we don't end up with a single server serving all the clients after the upgrade procedure. Clients will randomly select a server when connecting unless a special option is provided that disables that functionality \(usually called 'DontRandomize' or 'noRandomize'\). You can read more about ["Avoiding the Thundering Herd"](). Suffice it to say that clients redistribute themselves about evenly between all servers in the cluster. In our case 1/2 of the clients on 'A' will jump over to 'B' and the remaining half to 'C'. +We have two servers and a client. Time to simulate our rolling upgrade. But wait, before we upgrade 'A', let's introduce a new server 'C'. Server 'C' will join the existing cluster while we perform the upgrade. Its sole purpose is to provide an additional place where clients can go other than 'A' and ensure we don't end up with a single server serving all the clients after the upgrade procedure. Clients will randomly select a server when connecting unless a special option is provided that disables that functionality \(usually called 'DontRandomize' or 'noRandomize'\). You can read more about ["Avoiding the Thundering Herd"](upgrading_cluster.md). Suffice it to say that clients redistribute themselves about evenly between all servers in the cluster. In our case 1/2 of the clients on 'A' will jump over to 'B' and the remaining half to 'C'. Let's start our temporary server: diff --git a/nats-streaming-concepts/channels/subscriptions/queue-group.md b/nats-streaming-concepts/channels/subscriptions/queue-group.md index 90a6220..df5b5eb 100644 --- a/nats-streaming-concepts/channels/subscriptions/queue-group.md +++ b/nats-streaming-concepts/channels/subscriptions/queue-group.md @@ -8,5 +8,5 @@ When the last member of the group leaves \(subscription unsubscribed/closed/or c A queue subscription can also be durable. For that, the client needs to provide a queue and durable name. The behavior is, as you would expect, a combination of queue and durable subscription. Though unlike a durable subscription, the client ID is not part of the queue group name since the client ID must be unique, and would prevent more than one connection to participate in the queue group. The main difference between a queue subscription and a durable one, is that when the last member leaves the group, the state of the group will be maintained by the server. Later, when a member rejoins the group, the delivery will resume. -_**Note: For a durable queue subscription, the last member to**_ **unsubscribe** _**\(not simply close\) causes the group to be removed from the server.**_ +_**Note: For a durable queue subscription, the last member to**_ **unsubscribe** _**\(not simply close\) causes the group to be removed from the server.**_ diff --git a/nats-streaming-server/configuring/cfgfile.md b/nats-streaming-server/configuring/cfgfile.md index 3648f35..3e4689c 100644 --- a/nats-streaming-server/configuring/cfgfile.md +++ b/nats-streaming-server/configuring/cfgfile.md @@ -68,20 +68,20 @@ Note: You may need to scroll horizontally to see all columns. | hb\_interval | Interval at which the server sends an heartbeat to a client | Duration | `hb_interval: "10s"` | `30s` | v0.3.6 | | hb\_timeout | How long the server waits for a heartbeat response from the client before considering it a failed heartbeat | Duration | `hb_timeout: "10s"` | `10s` | v0.3.6 | | hb\_fail\_count | Count of failed heartbeats before server closes the client connection. The actual total wait is: \(fail count + 1\) \* \(hb interval + hb timeout\) | Number | `hb_fail_count: 2` | `10` | v0.3.6 | -| ft\_group | In Fault Tolerance mode, you can start a group of streaming servers with only one server being active while others are running in standby mode. This is the name of this FT group | String | `ft_group: "my_ft_group"` | N/A | v0.4.0| +| ft\_group | In Fault Tolerance mode, you can start a group of streaming servers with only one server being active while others are running in standby mode. This is the name of this FT group | String | `ft_group: "my_ft_group"` | N/A | v0.4.0 | | partitioning | If set to true, a list of channels must be defined in store\_limits/channels section. This section then serves two purposes, overriding limits for a given channel or adding it to the partition | `true` or `false` | `partitioning: true` | `false` | v0.5.0 | | sql\_options | SQL Store specific options | Map: `sql_options: { ... }` | [**See details below**](cfgfile.md#sql-options-configuration) | | v0.7.0 | | cluster | Cluster Configuration | Map: `cluster: { ... }` | [**See details below**](cfgfile.md#cluster-configuration) | | v0.9.0 | -| syslog_name | On Windows, when running several servers as a service, use this name for the event source | String | | v0.11.0 | +| syslog\_name | On Windows, when running several servers as a service, use this name for the event source | String | | v0.11.0 | | | encrypt | Specify if server should encrypt messages \(only the payload\) when storing them | `true` or `false` | `encrypt: true` | `false` | v0.12.0 | | encryption\_cipher | Cipher to use for encryption. Currently support AES and CHAHA \(ChaChaPoly\). Defaults to AES | `AES` or `CHACHA` | `encryption_cipher: "AES"` | Depends on platform | v0.12.0 | | encryption\_key | Encryption key. It is recommended to specify the key through the `NATS_STREAMING_ENCRYPTION_KEY` environment variable instead | String | `encryption_key: "mykey"` | N/A | v0.12.0 | | credentials | Credentials file to connect to external NATS 2.0+ Server | String | `credentials: "streaming_server.creds"` | N/A | v0.16.2 | | username | Username is used to connect to a NATS Server when authentication with multiple users is enabled | String | `username: "streaming_server"` | N/A | v0.19.0 | -| password | Password used with above `username` | String | `password: "password"` | N/A | v0.19.0 | +| password | Password used with above `username` | String | `password: "password"` | N/A | v0.19.0 | | token | Authentication token if the NATS Server requires a token | String | `token: "some_token"` | N/A | v0.19.0 | | nkey\_seed\_file | Path to an NKey seed file \(1\) if NKey authentication is used | File Path | `nkey_seed_file: "/path/to/some/seedfile"` | N/A | v0.19.0 | -| replace_durable | Replace the existing durable subscription instead of reporting a duplicate durable error | `true` or `false` | `replace_durable: true` | `false` | v0.20.0 | +| replace\_durable | Replace the existing durable subscription instead of reporting a duplicate durable error | `true` or `false` | `replace_durable: true` | `false` | v0.20.0 | Notes: @@ -196,10 +196,10 @@ For a given channel, the possible parameters are: | raft\_lease\_timeout | Specifies how long a leader waits without being able to contact a quorum of nodes before stepping down as leader | Duration | `raft_lease_timeout: "1s"` | `1s` | v0.11.2 | | raft\_commit\_timeout | Specifies the time without an Apply\(\) operation before sending an heartbeat to ensure timely commit. Due to random staggering, may be delayed as much as 2x this value | Duration | `raft_commit_timeout: "100ms"` | `100ms` | v0.11.2 | | proceed\_on\_restore\_failure | Allow a non leader node to proceed with restore failures, do not use unless you understand the risks! | `true` or `false` | `proceed_on_restore_failure: true` | `false` | v0.17.0 | -| allow_add_remove_node| Enable the ability to send NATS requests to the leader to add/remove cluster nodes | `true` or `false` | `allow_add_remove_node: true` | `false` | v0.19.0 | -| bolt_free_list_sync | Causes the RAFT log to synchronize the free list on write operations. Reduces performance at runtime, but makes the recovery faster | `true` or `false` | `bolt_free_list_sync: true` | `false` | v0.21.0 | -| bolt_free_list_map | Sets the backend freelist type to use a map instead of the default array type. Improves performance for large RAFT logs, with fragmented free list | `true` or `false` | `bolt_free_list_map: true` | `false` | v0.21.0 | -| nodes_connections | Enable creation of dedicated NATS connections to communicate with other nodes | `true` or `false` | `nodes_connections: true` | `false` | v0.21.0 | +| allow\_add\_remove\_node | Enable the ability to send NATS requests to the leader to add/remove cluster nodes | `true` or `false` | `allow_add_remove_node: true` | `false` | v0.19.0 | +| bolt\_free\_list\_sync | Causes the RAFT log to synchronize the free list on write operations. Reduces performance at runtime, but makes the recovery faster | `true` or `false` | `bolt_free_list_sync: true` | `false` | v0.21.0 | +| bolt\_free\_list\_map | Sets the backend freelist type to use a map instead of the default array type. Improves performance for large RAFT logs, with fragmented free list | `true` or `false` | `bolt_free_list_map: true` | `false` | v0.21.0 | +| nodes\_connections | Enable creation of dedicated NATS connections to communicate with other nodes | `true` or `false` | `nodes_connections: true` | `false` | v0.21.0 | ## SQL Options Configuration @@ -209,5 +209,5 @@ For a given channel, the possible parameters are: | source | How to connect to the database. This is driver specific | String | `source: "ivan:pwd@/nss_db"` | N/A | v0.7.0 | | no\_caching | Enable/Disable caching for messages and subscriptions operations. | `true` or `false` | `no_caching: false` | `false` \(caching enabled\) | v0.7.0 | | max\_open\_conns | Maximum number of opened connections to the database. Value <= 0 means no limit. | Number | `max_open_conns: 5` | unlimited | v0.7.0 | -| bulk\_insert\_limit | Maximum number of messages stored with a single SQL "INSERT" statement. The default behavior is to send individual insert commands as part of a SQL transaction. | Number | `bulk_insert_limit: 1000` | `0` (not enabled) | v0.20.0 | +| bulk\_insert\_limit | Maximum number of messages stored with a single SQL "INSERT" statement. The default behavior is to send individual insert commands as part of a SQL transaction. | Number | `bulk_insert_limit: 1000` | `0` \(not enabled\) | v0.20.0 | diff --git a/nats-tools/nas/dir_store.md b/nats-tools/nas/dir_store.md index 6d4ccd4..4bd59f3 100644 --- a/nats-tools/nas/dir_store.md +++ b/nats-tools/nas/dir_store.md @@ -60,7 +60,7 @@ Generated account key - private key stored "~/.nkeys/AAA/accounts/B/B.nk" Success! - added account "B" ``` -With the account and a couple of users in place, let's push all the accounts to the nats-account-server. If the account JWT server URL is not set on the operator, you may want to set it. Note that account servers typically require the path `/jwt/v1` in addition to the protocol and hostport (or you can specify the `--account-jwt-server-url` flag to nsc's `push` command). +With the account and a couple of users in place, let's push all the accounts to the nats-account-server. If the account JWT server URL is not set on the operator, you may want to set it. Note that account servers typically require the path `/jwt/v1` in addition to the protocol and hostport \(or you can specify the `--account-jwt-server-url` flag to nsc's `push` command\). ```text ❯ nsc edit operator --account-jwt-server-url http://localhost:9090/jwt/v1 diff --git a/nats-tools/natscli.md b/nats-tools/natscli.md index ed4b303..7ff1b66 100644 --- a/nats-tools/natscli.md +++ b/nats-tools/natscli.md @@ -1,4 +1,4 @@ -# natscli +# nats A command line utility to interact with and manage NATS. @@ -10,20 +10,20 @@ Check out the repo for more details: [github.com/nats-io/natscli](https://github For macOS: -``` +```text > brew tap nats-io/nats-tools > brew install nats-io/nats-tools/nats ``` For Arch Linux: -``` +```text > yay natscli ``` For Docker: -``` +```text docker pull synadia/nats-box:latest docker run -ti synadia/nats-box @@ -39,7 +39,7 @@ The `nats` utility has a command for creating `bcrypt` hashes. This can be used With `nats` installed: -```plain +```text > nats server passwd ? Enter password [? for help] ********************** ? Reenter password [? for help] ********************** @@ -49,7 +49,7 @@ $2a$11$3kIDaCxw.Glsl1.u5nKa6eUnNDLV5HV9tIuUp7EHhMt6Nm9myW1aS To use the password on the server, add the hash into the server configuration file's authorization section. -``` +```text authorization { user: derek password: $2a$11$3kIDaCxw.Glsl1.u5nKa6eUnNDLV5HV9tIuUp7EHhMt6Nm9myW1aS @@ -57,3 +57,4 @@ To use the password on the server, add the hash into the server configuration fi ``` Note the client will still have to provide the plain text version of the password, the server however will only store the hash to verify that the password is correct when supplied. + From ea8b4087f0e557133f7f7a47758323198ad26cf6 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 14:10:40 +0000 Subject: [PATCH 71/84] GitBook: [master] 2 pages modified --- SUMMARY.md | 2 +- jetstream/concepts/consumers.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/SUMMARY.md b/SUMMARY.md index 8b84a30..ab0a84e 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -113,7 +113,7 @@ * [About Jetstream](jetstream/jetstream.md) * [Concepts](jetstream/concepts/README.md) * [Streams](jetstream/concepts/streams.md) - * [Consumes](jetstream/concepts/consumers.md) + * [Consumers](jetstream/concepts/consumers.md) * [Configuration](jetstream/concepts/configuration.md) * [Getting Started](jetstream/getting_started/README.md) * [Using Docker](jetstream/getting_started/using_docker.md) diff --git a/jetstream/concepts/consumers.md b/jetstream/concepts/consumers.md index 288c27c..a411233 100644 --- a/jetstream/concepts/consumers.md +++ b/jetstream/concepts/consumers.md @@ -1,4 +1,4 @@ -# Consumes +# Consumers Each Consumer, or related group of Consumers, of a Stream will need an Consumer defined. It's ok to define thousands of these pointing at the same Stream. From ab3029f361dfc4d15db6def4fdab82be33604c1e Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 14:15:31 +0000 Subject: [PATCH 72/84] GitBook: [master] 2 pages modified --- SUMMARY.md | 2 +- jetstream/concepts/configuration.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/SUMMARY.md b/SUMMARY.md index ab0a84e..33dc577 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -114,7 +114,7 @@ * [Concepts](jetstream/concepts/README.md) * [Streams](jetstream/concepts/streams.md) * [Consumers](jetstream/concepts/consumers.md) - * [Configuration](jetstream/concepts/configuration.md) + * [Example Configuration](jetstream/concepts/configuration.md) * [Getting Started](jetstream/getting_started/README.md) * [Using Docker](jetstream/getting_started/using_docker.md) * [Using Docker with NGS](jetstream/getting_started/using-docker-with-ngs.md) diff --git a/jetstream/concepts/configuration.md b/jetstream/concepts/configuration.md index 74c831a..529a119 100644 --- a/jetstream/concepts/configuration.md +++ b/jetstream/concepts/configuration.md @@ -1,6 +1,6 @@ -# Configuration +# Example Configuration -The rest of this document introduces the `nats` utility, but for completeness and reference this is how you'd create the ORDERS scenario. We'll configure a 1 year retention for order related messages: +[Addition documentation](../clustering/administration.md) introduces the `nats` utility, but for completeness and reference this is how you'd create the ORDERS scenario. We'll configure a 1 year retention for order related messages: ```bash $ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard=old From 1ca6ed34a55cb9a9ca29a9f31f9aa4ade0a2406d Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 14:15:54 +0000 Subject: [PATCH 73/84] GitBook: [master] one page modified --- jetstream/concepts/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jetstream/concepts/configuration.md b/jetstream/concepts/configuration.md index 529a119..d15c09e 100644 --- a/jetstream/concepts/configuration.md +++ b/jetstream/concepts/configuration.md @@ -1,6 +1,6 @@ # Example Configuration -[Addition documentation](../clustering/administration.md) introduces the `nats` utility, but for completeness and reference this is how you'd create the ORDERS scenario. We'll configure a 1 year retention for order related messages: +[Additional documentation](../clustering/administration.md) introduces the `nats` utility, but for completeness and reference this is how you'd create the ORDERS scenario. We'll configure a 1 year retention for order related messages: ```bash $ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard=old From dc27c6adad0c7d786f30e0c42b235850a1edb3d5 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 14:18:07 +0000 Subject: [PATCH 74/84] GitBook: [master] one page modified --- jetstream/getting_started/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/jetstream/getting_started/README.md b/jetstream/getting_started/README.md index 1b342e0..367268c 100644 --- a/jetstream/getting_started/README.md +++ b/jetstream/getting_started/README.md @@ -1,8 +1,8 @@ # Getting Started -Getting started with JetStream is straightforward. While we speak of Jetstream as if it is a seperate component, it's actually a subsystem built into the NATS server that needs to be enabled. +Getting started with JetStream is straightforward. While we speak of JetStream as if it is a separate component, it's actually a subsystem built into the NATS server that needs to be enabled. -## Command line +## Command Line Enable JetStream by specifying the `-js` flag when starting the NATS server. @@ -10,7 +10,7 @@ Enable JetStream by specifying the `-js` flag when starting the NATS server. ## Configuration File -Enable JetStream through a configuration file. By default, the JetStream subsytem will store data in the /tmp directory. Here's a minimal file that will store data in a local "nats" directory, suitable for development and local testing. +You can also enable JetStream through a configuration file. By default, the JetStream subsytem will store data in the /tmp directory. Here's a minimal file that will store data in a local "nats" directory, suitable for development and local testing. `$ nats-server -c js.conf` From 5257c83a48a0a78b6fd73bb910d1164d3f554fe8 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 14:22:51 +0000 Subject: [PATCH 75/84] GitBook: [master] one page modified --- .../getting_started/using-docker-with-ngs.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/jetstream/getting_started/using-docker-with-ngs.md b/jetstream/getting_started/using-docker-with-ngs.md index 5dd8335..a09242d 100644 --- a/jetstream/getting_started/using-docker-with-ngs.md +++ b/jetstream/getting_started/using-docker-with-ngs.md @@ -1,2 +1,19 @@ # Using Docker with NGS +You can join a JetStream instance to your [NGS](https://synadia.com/ngs/pricing) account, first we need a credential for testing JetStream: + +```text +$ nsc add user -a YourAccount --name leafnode --expiry 1M +``` + +You'll get a credential file somewhere like `~/.nkeys/creds/synadia/YourAccount/leafnode.creds`, mount this file into the docker container for JetStream using `-v ~/.nkeys/creds/synadia/YourAccount/leafnode.creds:/leafnode.creds`. + +```text +$ docker run -ti -v ~/.nkeys/creds/synadia/YourAccount/leafnode.creds:/leafnode.creds --name jetstream synadia/jsm:latest server +[1] 2020/01/20 12:44:11.752465 [INF] Starting nats-server version 2.2.0-beta +... +[1] 2020/01/20 12:55:01.849033 [INF] Connected leafnode to "connect.ngs.global" +``` + +Your JSM shell will still connect locally, other connections in your NGS account can use JetStream at this point. + From aa207d6a14b6cf1d7b4c7c479cd6a20e827347bd Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 14:29:20 +0000 Subject: [PATCH 76/84] GitBook: [master] 2 pages modified --- SUMMARY.md | 1 - .../getting_started/using-docker-with-ngs.md | 19 ------------------- 2 files changed, 20 deletions(-) delete mode 100644 jetstream/getting_started/using-docker-with-ngs.md diff --git a/SUMMARY.md b/SUMMARY.md index 33dc577..fb3d627 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -117,7 +117,6 @@ * [Example Configuration](jetstream/concepts/configuration.md) * [Getting Started](jetstream/getting_started/README.md) * [Using Docker](jetstream/getting_started/using_docker.md) - * [Using Docker with NGS](jetstream/getting_started/using-docker-with-ngs.md) * [Using Source](jetstream/getting_started/using_source.md) * [Administration & Usage from CLI](jetstream/administration/README.md) * [Account Information](jetstream/administration/account.md) diff --git a/jetstream/getting_started/using-docker-with-ngs.md b/jetstream/getting_started/using-docker-with-ngs.md deleted file mode 100644 index a09242d..0000000 --- a/jetstream/getting_started/using-docker-with-ngs.md +++ /dev/null @@ -1,19 +0,0 @@ -# Using Docker with NGS - -You can join a JetStream instance to your [NGS](https://synadia.com/ngs/pricing) account, first we need a credential for testing JetStream: - -```text -$ nsc add user -a YourAccount --name leafnode --expiry 1M -``` - -You'll get a credential file somewhere like `~/.nkeys/creds/synadia/YourAccount/leafnode.creds`, mount this file into the docker container for JetStream using `-v ~/.nkeys/creds/synadia/YourAccount/leafnode.creds:/leafnode.creds`. - -```text -$ docker run -ti -v ~/.nkeys/creds/synadia/YourAccount/leafnode.creds:/leafnode.creds --name jetstream synadia/jsm:latest server -[1] 2020/01/20 12:44:11.752465 [INF] Starting nats-server version 2.2.0-beta -... -[1] 2020/01/20 12:55:01.849033 [INF] Connected leafnode to "connect.ngs.global" -``` - -Your JSM shell will still connect locally, other connections in your NGS account can use JetStream at this point. - From c3854f2c5f5c1c997eb07af7e1a26659ed102d05 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 14:38:19 +0000 Subject: [PATCH 77/84] GitBook: [master] one page modified --- jetstream/clustering/README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/jetstream/clustering/README.md b/jetstream/clustering/README.md index d97925b..1847297 100644 --- a/jetstream/clustering/README.md +++ b/jetstream/clustering/README.md @@ -1,6 +1,6 @@ # Clustering -Clustering in Jetstream is required for a highly available and scalable system. Behind clustering is RAFT. There's no need to understand RAFT in depth to use clustering, but knowing a little explains some of the requirements behind setting up Jetstream clusters. +Clustering in JetStream is required for a highly available and scalable system. Behind clustering is RAFT. There's no need to understand RAFT in depth to use clustering, but knowing a little explains some of the requirements behind setting up JetStream clusters. ## RAFT @@ -8,7 +8,7 @@ JetStream uses a NATS optimized RAFT algorithm for clustering. Typically raft ge ### Raft groups -The RAFT groups include API handlers sstreams, consumers, and an internal algorithm designates which servers handle which streams and consumers. +The RAFT groups include API handlers, streams, consumers, and an internal algorithm designates which servers handle which streams and consumers. The raft algorithm has a few requirements: @@ -29,23 +29,23 @@ In order to ensure data consistency across complete restarts, a quorum of server ![Stream Groups](../../.gitbook/assets/stream-groups.png) -**Consumer Group** - each Consumer creates a RAFT group, this group synchronizes consumer state between its members. The group will live on the machines where the Stream Group is and handle consumption ACKs etc. Each Consumer will have its own group. +**Consumer Group** - each Consumer creates a RAFT group, this group synchronizes consumer state between its members. The group will live on the machines where the Stream Group is and handle consumption ACKs etc. Each Consumer will have their own group. ![Consumer Groups](../../.gitbook/assets/consumer-groups.png) ### Cluster Size -Generally we recommend 3 or 5 Jetstream enabled servers in a NATS cluster. This balances scalability with a tolerance for failure. For example, if 5 servers are Jetstream enabled You would want two servers is one “zone”, two servers in another, and the remaining server in a third. This means you can lose any one “zone” at any time and continue operating. +Generally, we recommend 3 or 5 JetStream enabled servers in a NATS cluster. This balances scalability with a tolerance for failure. For example, if 5 servers are JetStream enabled You would want two servers is one “zone”, two servers in another, and the remaining server in a third. This means you can lose any one “zone” at any time and continue operating. -### Mixing Jetstream enabled servers with standard NATS servers +### Mixing JetStream enabled servers with standard NATS servers -This is possible, and even recommended in some cases. By mixing server types you can dedicate certain machines optimized for storage for Jetstream and others optimized solely for compute for standard NATS servers, reducing operational expense. With the right configuration, the standard servers would handle non-persistent NATS traffic and the Jetstream enabled servers would handle Jetstream traffic. +This is possible and even recommended in some cases. By mixing server types you can dedicate certain machines optimized for storage for Jetstream and others optimized solely for compute for standard NATS servers, reducing operational expense. With the right configuration, the standard servers would handle non-persistent NATS traffic and the JetStream enabled servers would handle JetStream traffic. ## Configuration -To configure Jetstream clusters, just configure clusters as you normally would by specifying a cluster block in the configuration. Any Jetstream enabled servers in the list of clusters will automatically chatter and set themselves up. Unlike core NATS clustering though, each Jetstream node **must specify** a server name and cluster name. +To configure JetStream clusters, just configure clusters as you normally would by specifying a cluster block in the configuration. Any JetStream enabled servers in the list of clusters will automatically chatter and set themselves up. Unlike core NATS clustering though, each JetStream node **must specify** a server name and cluster name. -Below are explicity listed server configuration for a three node cluster across three machines, `n1-c1`, `n2-c1`, and `n3-c1`. +Below are explicitly listed server configuration for a three-node cluster across three machines, `n1-c1`, `n2-c1`, and `n3-c1`. ### Server 1 \(host\_a\) @@ -107,5 +107,5 @@ cluster { } ``` -Add nodes as necessary. Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use Jetstream. +Add nodes as necessary. Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use JetStream. From 2f8dc99d0d76a3d2cf8caa4e704041ef2bc9aded Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 14:47:03 +0000 Subject: [PATCH 78/84] GitBook: [master] 213 pages modified --- jetstream/configuration_mgmt/README.md | 74 +------------------ .../kubernetes_controller.md | 2 +- jetstream/configuration_mgmt/terraform.md | 66 +++++++++++++++++ 3 files changed, 70 insertions(+), 72 deletions(-) diff --git a/jetstream/configuration_mgmt/README.md b/jetstream/configuration_mgmt/README.md index 7869bbf..59a6459 100644 --- a/jetstream/configuration_mgmt/README.md +++ b/jetstream/configuration_mgmt/README.md @@ -5,9 +5,9 @@ In many cases managing the configuration in your application code is the best mo We support a number of tools to assist with this: * `nats` CLI with configuration files -* [Terraform](https://www.terraform.io/) -* [GitHub Actions](https://github.com/features/actions) -* [Kubernetes JetStream Controller](https://github.com/nats-io/nack#jetstream-controller) +* [Terraform](terraform.md) +* [GitHub Actions](github_actions.md) +* [Kubernetes JetStream Controller](kubernetes_controller.md) ## nats Admin CLI @@ -37,71 +37,3 @@ This creates a new Consumer based on `orders_new.json`. The `orders_new.json` fi $ nats con add ORDERS NEW --config orders_new.json ``` -## Terraform - -Terraform is a Cloud configuration tool from Hashicorp found at [terraform.io](https://www.terraform.io/), we maintain a Provider for Terraform called [terraform-provider-jetstream](https://github.com/nats-io/terraform-provider-jetstream/) that can maintain JetStream using Terraform. - -### Setup - -Our provider is not hosted by Hashicorp so installation is a bit more complex than typical. Browse to the [Release Page](https://github.com/nats-io/terraform-provider-jetstream/releases) and download the release for your platform and extract it into your Terraform plugins directory. - -```text -$ unzip -l terraform-provider-jetstream_0.0.2_darwin_amd64.zip -Archive: terraform-provider-jetstream_0.0.2_darwin_amd64.zip - Length Date Time Name ---------- ---------- ----- ---- - 11357 03-09-2020 10:48 LICENSE - 1830 03-09-2020 12:53 README.md - 24574336 03-09-2020 12:54 terraform-provider-jetstream_v0.0.2 -``` - -Place the `terraform-provider-jetstream_v0.0.2` file in `~/.terraform.d/plugins/terraform-provider-jetstream_v0.0.2` - -In your project you can configure the Provider like this: - -```text -provider "jetstream" { - servers = "connect.ngs.global" - credentials = "ngs_jetstream_admin.creds" -} -``` - -And start using it, here's an example that create the `ORDERS` example. Review the [Project README](https://github.com/nats-io/terraform-provider-jetstream#readme) for full details. - -```text -resource "jetstream_stream" "ORDERS" { - name = "ORDERS" - subjects = ["ORDERS.*"] - storage = "file" - max_age = 60 * 60 * 24 * 365 -} - -resource "jetstream_consumer" "ORDERS_NEW" { - stream_id = jetstream_stream.ORDERS.id - durable_name = "NEW" - deliver_all = true - filter_subject = "ORDERS.received" - sample_freq = 100 -} - -resource "jetstream_consumer" "ORDERS_DISPATCH" { - stream_id = jetstream_stream.ORDERS.id - durable_name = "DISPATCH" - deliver_all = true - filter_subject = "ORDERS.processed" - sample_freq = 100 -} - -resource "jetstream_consumer" "ORDERS_MONITOR" { - stream_id = jetstream_stream.ORDERS.id - durable_name = "MONITOR" - deliver_last = true - ack_policy = "none" - delivery_subject = "monitor.ORDERS" -} - -output "ORDERS_SUBJECTS" { - value = jetstream_stream.ORDERS.subjects -} -``` - diff --git a/jetstream/configuration_mgmt/kubernetes_controller.md b/jetstream/configuration_mgmt/kubernetes_controller.md index 3ea6392..df4d813 100644 --- a/jetstream/configuration_mgmt/kubernetes_controller.md +++ b/jetstream/configuration_mgmt/kubernetes_controller.md @@ -1,6 +1,6 @@ # Kubernetes Controller -The JetStream controllers allows you to manage NATS JetStream Streams and Consumers via K8S CRDs. You can find more info on how to deploy and usage [here](https://github.com/nats-io/nack#getting-started). Below you can find an example on how to create a stream and a couple of consumers: +The JetStream controllers allow you to manage NATS JetStream Streams and Consumers via K8S CRDs. You can find more info on how to deploy and usage [here](https://github.com/nats-io/nack#getting-started). Below you can find an example of how to create a stream and a couple of consumers: ```yaml --- diff --git a/jetstream/configuration_mgmt/terraform.md b/jetstream/configuration_mgmt/terraform.md index 6719963..574b59b 100644 --- a/jetstream/configuration_mgmt/terraform.md +++ b/jetstream/configuration_mgmt/terraform.md @@ -1,2 +1,68 @@ # Terraform +Terraform is a Cloud configuration tool from Hashicorp found at [terraform.io](https://www.terraform.io/), we maintain a Provider for Terraform called [terraform-provider-jetstream](https://github.com/nats-io/terraform-provider-jetstream/) that can maintain JetStream using Terraform. + +### Setup + +Our provider is not hosted by Hashicorp so installation is a bit more complex than typical. Browse to the [Release Page](https://github.com/nats-io/terraform-provider-jetstream/releases) and download the release for your platform and extract it into your Terraform plugins directory. + +```text +$ unzip -l terraform-provider-jetstream_0.0.2_darwin_amd64.zip +Archive: terraform-provider-jetstream_0.0.2_darwin_amd64.zip + Length Date Time Name +--------- ---------- ----- ---- + 11357 03-09-2020 10:48 LICENSE + 1830 03-09-2020 12:53 README.md + 24574336 03-09-2020 12:54 terraform-provider-jetstream_v0.0.2 +``` + +Place the `terraform-provider-jetstream_v0.0.2` file in `~/.terraform.d/plugins/terraform-provider-jetstream_v0.0.2` + +In your project you can configure the Provider like this: + +```text +provider "jetstream" { + servers = "connect.ngs.global" + credentials = "ngs_jetstream_admin.creds" +} +``` + +And start using it, here's an example that create the `ORDERS` example. Review the [Project README](https://github.com/nats-io/terraform-provider-jetstream#readme) for full details. + +```text +resource "jetstream_stream" "ORDERS" { + name = "ORDERS" + subjects = ["ORDERS.*"] + storage = "file" + max_age = 60 * 60 * 24 * 365 +} + +resource "jetstream_consumer" "ORDERS_NEW" { + stream_id = jetstream_stream.ORDERS.id + durable_name = "NEW" + deliver_all = true + filter_subject = "ORDERS.received" + sample_freq = 100 +} + +resource "jetstream_consumer" "ORDERS_DISPATCH" { + stream_id = jetstream_stream.ORDERS.id + durable_name = "DISPATCH" + deliver_all = true + filter_subject = "ORDERS.processed" + sample_freq = 100 +} + +resource "jetstream_consumer" "ORDERS_MONITOR" { + stream_id = jetstream_stream.ORDERS.id + durable_name = "MONITOR" + deliver_last = true + ack_policy = "none" + delivery_subject = "monitor.ORDERS" +} + +output "ORDERS_SUBJECTS" { + value = jetstream_stream.ORDERS.subjects +} +``` + From 45ab567ab3a44a4347f97999f9690c97c041f94f Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 14:53:27 +0000 Subject: [PATCH 79/84] GitBook: [master] one page modified --- jetstream/model_deep_dive/README.md | 40 ++++++++++++++--------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/jetstream/model_deep_dive/README.md b/jetstream/model_deep_dive/README.md index c6670c7..7c2055b 100644 --- a/jetstream/model_deep_dive/README.md +++ b/jetstream/model_deep_dive/README.md @@ -91,7 +91,7 @@ State: Redelivered Messages: 0 ``` -The Consumer has no messages oustanding and has never had any \(Consumer sequence is 1\). +The Consumer has no messages outstanding and has never had any \(Consumer sequence is 1\). We publish one message to the Stream and see that the Stream received it: @@ -149,7 +149,7 @@ State: Redelivered Messages: 0 ``` -Now we can see the Consumer have processed 2 messages \(obs sequence is 3, next message will be 3\) but the Ack floor is still 1 - thus 1 message is pending acknowledgement. Indeed this is confirmed in the `Pending messages`. +Now we can see the Consumer has processed 2 messages \(obs sequence is 3, next message will be 3\) but the Ack floor is still 1 - thus 1 message is pending acknowledgement. Indeed this is confirmed in the `Pending messages`. If I fetch it again and again do not ack it: @@ -167,9 +167,9 @@ State: Redelivered Messages: 1 ``` -The Consumer sequence increases - each delivery attempt increase the sequence - and our redelivered count also goes up. +The Consumer sequence increases - each delivery attempt increases the sequence - and our redelivered count also goes up. -Finally if I then fetch it again and ack it this time: +Finally, if I then fetch it again and ack it this time: ```text $ nats con next ORDERS DISPATCH @@ -188,7 +188,7 @@ State: Having now Acked the message there are no more pending. -Additionally there are a few types of acknowledgements: +Additionally, there are a few types of acknowledgements: | Type | Bytes | Description | | :--- | :--- | :--- | @@ -198,11 +198,11 @@ Additionally there are a few types of acknowledgements: | `AckNext` | `+NXT` | Acknowledges the message was handled and requests delivery of the next message to the reply subject. Only applies to Pull-mode. | | `AckTerm` | `+TERM` | Instructs the server to stop redelivery of a message without acknowledging it as successfully processed | -So far all the examples was the `AckAck` type of acknowledgement, by replying to the Ack with the body as indicated in `Bytes` you can pick what mode of acknowledgement you want. +So far all of the examples were the `AckAck` type of acknowledgement, by replying to the Ack with the body as indicated in `Bytes` you can pick what mode of acknowledgement you want. All of these acknowledgement modes, except `AckNext`, support double acknowledgement - if you set a reply subject when acknowledging the server will in turn acknowledge having received your ACK. -The `+NXT` acknowledgement can have a few formats: `+NXT 10` requests 10 messages and `+NXT {"no_wait": true}` which is the same data that can be sent in a Pull request. +The `+NXT` acknowledgement can have a few formats: `+NXT 10` requests 10 messages and `+NXT {"no_wait": true}` which is the same data that can be sent in a Pull Request. ## Exactly Once Delivery @@ -214,7 +214,7 @@ Consumers can be 100% sure a message was correctly processed by requesting the s ## Consumer Starting Position -When setting up an Consumer you can decide where to start, the system supports the following for the `DeliverPolicy`: +When setting up a Consumer you can decide where to start, the system supports the following for the `DeliverPolicy`: | Policy | Description | | :--- | :--- | @@ -226,7 +226,7 @@ When setting up an Consumer you can decide where to start, the system supports t Regardless of what mode you set, this is only the starting point. Once started it will always give you what you have not seen or acknowledged. So this is merely how it picks the very first message. -Lets look at each of these, first we make a new Stream `ORDERS` and add 100 messages to it. +Let's look at each of these, first we make a new Stream `ORDERS` and add 100 messages to it. Now create a `DeliverAll` pull-based Consumer: @@ -272,7 +272,7 @@ do done ``` -Then create an Consumer that starts 2 minutes ago: +Then create a Consumer that starts 2 minutes ago: ```text $ nats con add ORDERS 2MIN --pull --filter ORDERS.processed --ack none --replay instant --deliver 2m @@ -285,7 +285,7 @@ Acknowledged message ## Ephemeral Consumers -So far, all the Consumers you have seen were Durable, meaning they exist even after you disconnect from JetStream. In our Orders scenario, though the `MONITOR` Consumer could very well be a short-lived thing there just while an operator is debugging the system, there is no need to remember the last seen position if all you are doing is wanting to observe the real-time state. +So far, all the Consumers you have seen were Durable, meaning they exist even after you disconnect from JetStream. In our Orders scenario, though the `MONITOR` a Consumer could very well be a short-lived thing there just while an operator is debugging the system, there is no need to remember the last seen position if all you are doing is wanting to observe the real-time state. In this case, we can make an Ephemeral Consumer by first subscribing to the delivery subject, then creating a durable and giving it no durable name. An Ephemeral Consumer exists as long as any subscription is active on its delivery subject. It is automatically be removed, after a short grace period to handle restarts, when there are no subscribers. @@ -320,7 +320,7 @@ $ nats con add ORDERS REPLAY --target out.original --filter ORDERS.processed --a ... ``` -Now lets publish messages into the Set 10 seconds apart: +Now let's publish messages into the Set 10 seconds apart: ```text $ for i in 1 2 3 <15:15:35 @@ -346,7 +346,7 @@ Listening on [out.original] ## Stream Templates -When you have many similar streams it can be helpful to auto create them, lets say you have a service by client and they are on subjects `CLIENT.*`, you can construct a template that will auto generate streams for any matching traffic. +When you have many similar streams it can be helpful to auto-create them, let's say you have a service by client and they are on subjects `CLIENT.*`, you can construct a template that will auto-generate streams for any matching traffic. ```text $ nats str template add CLIENTS --subjects "CLIENT.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size 2048 --max-streams 1024 --discard old @@ -393,15 +393,15 @@ When the template is deleted all the streams it created will be deleted too. In the earlier sections we saw that samples are being sent to a monitoring system. Let's look at that in depth; how the monitoring system works and what it contains. -As messages pass through an Consumer you'd be interested in knowing how many are being redelivered and how many times but also how long it takes for messages to be acknowledged. +As messages pass through a Consumer you'd be interested in knowing how many are being redelivered and how many times but also how long it takes for messages to be acknowledged. -Consumers can sample Ack'ed messages for you and publish samples so your monitoring system can observe the health of an Consumer. We will add support for this to [NATS Surveyor](https://github.com/nats-io/nats-surveyor). +Consumers can sample Ack'ed messages for you and publish samples so your monitoring system can observe the health of a Consumer. We will add support for this to [NATS Surveyor](https://github.com/nats-io/nats-surveyor). ### Configuration -You can configure an Consumer for sampling by passing the `--sample 80` option to `nats consumer add`, this tells the system to sample 80% of Acknowledgements. +You can configure a Consumer for sampling bypassing the `--sample 80` option to `nats consumer add`, this tells the system to sample 80% of Acknowledgements. -When viewing info of an Consumer you can tell if it's sampled or not: +When viewing info of a Consumer you can tell if it's sampled or not: ```text $ nats con info ORDERS NEW @@ -449,7 +449,7 @@ $ nats con events ORDERS NEW --json ## Storage Overhead -JetStream file storage is very efficient storing as little extra information about the message as possible. +JetStream file storage is very efficient, storing as little extra information about the message as possible. **NOTE:** This might change once clustering is supported. @@ -461,7 +461,7 @@ We do store some message data with each message, namely: * The message payload * A hash of the message * The message sequence -* A few other bits like length of the subject and lengh of headers +* A few other bits like the length of the subject and the length of headers Without any headers the size is: @@ -477,5 +477,5 @@ With headers: length of the message record (4bytes) + seq(8) + ts(8) + subj_len(2) + subj + hdr_len(4) + hdr + msg + hash(8) ``` -So if you are publishing many small messages the overhead will be, relatively speaking, quite large, but for larger messages the overhead is very small. If you publish many small messages it's worth trying to optimise the subject length. +So if you are publishing many small messages the overhead will be, relatively speaking, quite large, but for larger messages the overhead is very small. If you publish many small messages it's worth trying to optimize the subject length. From 9791a4d12cb65998c9a7991c221a512f0f5ca0ee Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 14:58:32 +0000 Subject: [PATCH 80/84] GitBook: [master] 14 pages modified --- SUMMARY.md | 12 +----------- jetstream/concepts/streams.md | 2 +- .../README.md => model_deep_dive.md} | 6 +++--- jetstream/model_deep_dive/ack-sampling.md | 2 -- jetstream/model_deep_dive/acknowledgment-models.md | 2 -- jetstream/model_deep_dive/consumer-message-rates.md | 2 -- .../model_deep_dive/consumer-starting-position.md | 2 -- jetstream/model_deep_dive/ephemeral-consumers.md | 2 -- jetstream/model_deep_dive/exactly-once-delivery.md | 2 -- jetstream/model_deep_dive/message-deduplication.md | 2 -- jetstream/model_deep_dive/storage-overhead.md | 2 -- ...ream-limits-retention-modes-and-discard-policy.md | 2 -- jetstream/model_deep_dive/stream-templates.md | 2 -- 13 files changed, 5 insertions(+), 35 deletions(-) rename jetstream/{model_deep_dive/README.md => model_deep_dive.md} (97%) delete mode 100644 jetstream/model_deep_dive/ack-sampling.md delete mode 100644 jetstream/model_deep_dive/acknowledgment-models.md delete mode 100644 jetstream/model_deep_dive/consumer-message-rates.md delete mode 100644 jetstream/model_deep_dive/consumer-starting-position.md delete mode 100644 jetstream/model_deep_dive/ephemeral-consumers.md delete mode 100644 jetstream/model_deep_dive/exactly-once-delivery.md delete mode 100644 jetstream/model_deep_dive/message-deduplication.md delete mode 100644 jetstream/model_deep_dive/storage-overhead.md delete mode 100644 jetstream/model_deep_dive/stream-limits-retention-modes-and-discard-policy.md delete mode 100644 jetstream/model_deep_dive/stream-templates.md diff --git a/SUMMARY.md b/SUMMARY.md index fb3d627..a170656 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -132,17 +132,7 @@ * [GitHub Actions](jetstream/configuration_mgmt/github_actions.md) * [Kubernetes Controller](jetstream/configuration_mgmt/kubernetes_controller.md) * [Disaser Recovery](jetstream/disaster_recovery.md) -* [Model Deep Dive](jetstream/model_deep_dive/README.md) - * [Stream Limits, Retention Modes and Discard Policy](jetstream/model_deep_dive/stream-limits-retention-modes-and-discard-policy.md) - * [Message Deduplication](jetstream/model_deep_dive/message-deduplication.md) - * [Acknowledgment Models](jetstream/model_deep_dive/acknowledgment-models.md) - * [Exactly Once Delivery](jetstream/model_deep_dive/exactly-once-delivery.md) - * [Consumer Starting Position](jetstream/model_deep_dive/consumer-starting-position.md) - * [Ephemeral Consumers](jetstream/model_deep_dive/ephemeral-consumers.md) - * [Consumer Message Rates](jetstream/model_deep_dive/consumer-message-rates.md) - * [Stream Templates](jetstream/model_deep_dive/stream-templates.md) - * [Ack Sampling](jetstream/model_deep_dive/ack-sampling.md) - * [Storage Overhead](jetstream/model_deep_dive/storage-overhead.md) +* [Model Deep Dive](jetstream/model_deep_dive.md) * [NATS API Reference](jetstream/nats_api_reference.md) * [Multi-tenancy & Resource Mgmt](jetstream/resource_management.md) diff --git a/jetstream/concepts/streams.md b/jetstream/concepts/streams.md index 6b31291..fd776d6 100644 --- a/jetstream/concepts/streams.md +++ b/jetstream/concepts/streams.md @@ -10,7 +10,7 @@ Streams can consume many subjects. Here we have `ORDERS.*` but we could also con Streams support various retention policies - they can be kept based on limits like max count, size or age but also more novel methods like keeping them as long as any Consumers have them unacknowledged, or work queue like behavior where a message is removed after first ack. -Streams support deduplication using a `Nats-Msg-Id` header and a sliding window within which to track duplicate messages. See the [Message Deduplication](../model_deep_dive/#message-deduplication) section. +Streams support deduplication using a `Nats-Msg-Id` header and a sliding window within which to track duplicate messages. See the [Message Deduplication](../model_deep_dive.md#message-deduplication) section. When defining Streams the items below make up the entire configuration of the set. diff --git a/jetstream/model_deep_dive/README.md b/jetstream/model_deep_dive.md similarity index 97% rename from jetstream/model_deep_dive/README.md rename to jetstream/model_deep_dive.md index 7c2055b..5b51081 100644 --- a/jetstream/model_deep_dive/README.md +++ b/jetstream/model_deep_dive.md @@ -24,9 +24,9 @@ In both `WorkQueuePolicy` and `InterestPolicy` the age, size and count limits wi A final control is the Maximum Size any single message may have. NATS have it's own limit for maximum size \(1 MiB by default\), but you can say a Stream will only accept messages up to 1024 bytes using `MaxMsgSize`. -The `Discard Policy` sets how messages are discard when limits set by `LimitsPolicy` are reached. The `DiscardOld` option removes old messages making space for new, while `DiscardNew` refuses any new messages. +The `Discard Policy` sets how messages are discarded when limits set by `LimitsPolicy` are reached. The `DiscardOld` option removes old messages making space for new, while `DiscardNew` refuses any new messages. -The `WorkQueuePolicy` mode is a specialized mode where a message, once consumed and acknowledged, is discarded from the Stream. In this mode there are a few limits on consumers. Inherently it's about 1 message to one consumer, this means you cannot have overlapping consumers defined on the Stream - needs unique filter subjects. +The `WorkQueuePolicy` mode is a specialized mode where a message, once consumed and acknowledged, is discarded from the Stream. In this mode, there are a few limits on consumers. Inherently it's about 1 message to one consumer, this means you cannot have overlapping consumers defined on the Stream - needs unique filter subjects. ## Message Deduplication @@ -208,7 +208,7 @@ The `+NXT` acknowledgement can have a few formats: `+NXT 10` requests 10 message JetStream supports Exactly Once delivery by combining Message Deduplication and double acks. -On the publishing side you can avoid duplicate message ingestion using the [Message Deduplication](./#message-deduplication) feature. +On the publishing side you can avoid duplicate message ingestion using the [Message Deduplication](model_deep_dive.md#message-deduplication) feature. Consumers can be 100% sure a message was correctly processed by requesting the server Acknowledge having received your acknowledgement by setting a reply subject on the Ack. If you receive this response you will never receive that message again. diff --git a/jetstream/model_deep_dive/ack-sampling.md b/jetstream/model_deep_dive/ack-sampling.md deleted file mode 100644 index eedc01d..0000000 --- a/jetstream/model_deep_dive/ack-sampling.md +++ /dev/null @@ -1,2 +0,0 @@ -# Ack Sampling - diff --git a/jetstream/model_deep_dive/acknowledgment-models.md b/jetstream/model_deep_dive/acknowledgment-models.md deleted file mode 100644 index 1b1561d..0000000 --- a/jetstream/model_deep_dive/acknowledgment-models.md +++ /dev/null @@ -1,2 +0,0 @@ -# Acknowledgment Models - diff --git a/jetstream/model_deep_dive/consumer-message-rates.md b/jetstream/model_deep_dive/consumer-message-rates.md deleted file mode 100644 index f3401a1..0000000 --- a/jetstream/model_deep_dive/consumer-message-rates.md +++ /dev/null @@ -1,2 +0,0 @@ -# Consumer Message Rates - diff --git a/jetstream/model_deep_dive/consumer-starting-position.md b/jetstream/model_deep_dive/consumer-starting-position.md deleted file mode 100644 index 2e2b6b8..0000000 --- a/jetstream/model_deep_dive/consumer-starting-position.md +++ /dev/null @@ -1,2 +0,0 @@ -# Consumer Starting Position - diff --git a/jetstream/model_deep_dive/ephemeral-consumers.md b/jetstream/model_deep_dive/ephemeral-consumers.md deleted file mode 100644 index f12004a..0000000 --- a/jetstream/model_deep_dive/ephemeral-consumers.md +++ /dev/null @@ -1,2 +0,0 @@ -# Ephemeral Consumers - diff --git a/jetstream/model_deep_dive/exactly-once-delivery.md b/jetstream/model_deep_dive/exactly-once-delivery.md deleted file mode 100644 index 2bc6b24..0000000 --- a/jetstream/model_deep_dive/exactly-once-delivery.md +++ /dev/null @@ -1,2 +0,0 @@ -# Exactly Once Delivery - diff --git a/jetstream/model_deep_dive/message-deduplication.md b/jetstream/model_deep_dive/message-deduplication.md deleted file mode 100644 index 98ab7f5..0000000 --- a/jetstream/model_deep_dive/message-deduplication.md +++ /dev/null @@ -1,2 +0,0 @@ -# Message Deduplication - diff --git a/jetstream/model_deep_dive/storage-overhead.md b/jetstream/model_deep_dive/storage-overhead.md deleted file mode 100644 index bc9a5e4..0000000 --- a/jetstream/model_deep_dive/storage-overhead.md +++ /dev/null @@ -1,2 +0,0 @@ -# Storage Overhead - diff --git a/jetstream/model_deep_dive/stream-limits-retention-modes-and-discard-policy.md b/jetstream/model_deep_dive/stream-limits-retention-modes-and-discard-policy.md deleted file mode 100644 index 00be723..0000000 --- a/jetstream/model_deep_dive/stream-limits-retention-modes-and-discard-policy.md +++ /dev/null @@ -1,2 +0,0 @@ -# Stream Limits, Retention Modes and Discard Policy - diff --git a/jetstream/model_deep_dive/stream-templates.md b/jetstream/model_deep_dive/stream-templates.md deleted file mode 100644 index 95ed438..0000000 --- a/jetstream/model_deep_dive/stream-templates.md +++ /dev/null @@ -1,2 +0,0 @@ -# Stream Templates - From c4c49544046d66504a258e896079bd3d795db069 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 14:59:32 +0000 Subject: [PATCH 81/84] GitBook: [master] one page modified --- jetstream/nats_api_reference.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jetstream/nats_api_reference.md b/jetstream/nats_api_reference.md index 610e552..14a0e90 100644 --- a/jetstream/nats_api_reference.md +++ b/jetstream/nats_api_reference.md @@ -165,7 +165,7 @@ This design allows you to easily create ACL rules that limit users to a specific Messages that need acknowledgement will have a Reply subject set, something like `$JS.ACK.ORDERS.test.1.2.2`, this is the prefix defined in `api.JetStreamAckPre` followed by `......`. -In all the Synadia maintained API's you can simply do `msg.Respond(nil)` \(or language equivalent\) which will send nil to the reply subject. +In all of the Synadia maintained API's you can simply do `msg.Respond(nil)` \(or language equivalent\) which will send nil to the reply subject. ## Fetching The Next Message From a Pull-based Consumer From 05a96ec4b19e67d20048d37392724d0e50e76fa0 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 15:00:05 +0000 Subject: [PATCH 82/84] GitBook: [master] one page modified --- jetstream/nats_api_reference.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jetstream/nats_api_reference.md b/jetstream/nats_api_reference.md index 14a0e90..4c7fcfe 100644 --- a/jetstream/nats_api_reference.md +++ b/jetstream/nats_api_reference.md @@ -2,7 +2,7 @@ Thus far we saw a lot of CLI interactions. The CLI works by sending and receiving specially crafted messages over core NATS to configure the JetStream system. In time we will look to add file based configuration but for now the only method is the NATS API. -**NOTE:** Some NATS client libraries may need to enable an option to use old style requests when interacting withe JetStream server. Consult the libraries README's for more information. +**NOTE:** Some NATS client libraries may need to enable an option to use old style requests when interacting with the JetStream server. Consult the libraries README's for more information. ## Reference From ec57cc1c4e7dc1a45756acd6b8fc4c4331ca831f Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 16:33:31 +0000 Subject: [PATCH 83/84] GitBook: [master] 12 pages modified --- SUMMARY.md | 4 +- jetstream/model_deep_dive.md | 2 +- nats-server/configuration/README.md | 5 +- .../configuration/leafnodes/leafnode_conf.md | 17 +- nats-server/configuration/monitoring.md | 2 +- nats-server/configuration/mqtt.md | 128 +++++++++++++ nats-server/configuration/mqtt/README.md | 173 ------------------ .../configuration/securing_nats/accounts.md | 5 +- .../securing_nats/jwt/resolver.md | 29 +-- nats-server/configuration/websocket.md | 11 ++ nats-server/configuration/websocket/README.md | 12 -- nats-tools/nas/README.md | 11 +- 12 files changed, 171 insertions(+), 228 deletions(-) create mode 100644 nats-server/configuration/mqtt.md delete mode 100644 nats-server/configuration/mqtt/README.md create mode 100644 nats-server/configuration/websocket.md delete mode 100644 nats-server/configuration/websocket/README.md diff --git a/SUMMARY.md b/SUMMARY.md index dbc8dcc..20f3c5d 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -96,10 +96,10 @@ * [Configuration](nats-server/configuration/leafnodes/leafnode_conf.md) * [Logging](nats-server/configuration/logging.md) * [Monitoring](nats-server/configuration/monitoring.md) - * [MQTT](nats-server/configuration/mqtt/README.md) + * [MQTT](nats-server/configuration/mqtt.md) * [System Events](nats-server/configuration/sys_accounts/README.md) * [System Events & Decentralized JWT Tutorial](nats-server/configuration/sys_accounts/sys_accounts.md) - * [Websocket](nats-server/configuration/websocket/README.md) + * [Websocket](nats-server/configuration/websocket.md) * [Managing A NATS Server](nats-server/nats_admin/README.md) * [Upgrading a Cluster](nats-server/nats_admin/upgrading_cluster.md) * [Slow Consumers](nats-server/nats_admin/slow_consumers.md) diff --git a/jetstream/model_deep_dive.md b/jetstream/model_deep_dive.md index 5b51081..01fb3d3 100644 --- a/jetstream/model_deep_dive.md +++ b/jetstream/model_deep_dive.md @@ -2,7 +2,7 @@ The Orders example touched on a lot of features, but some like different Ack models and message limits, need a bit more detail. This section will expand on the above and fill in some blanks. -## Stream Limits, Retention Modes and Discard Policy +## Stream Limits, Retention, and Policy Streams store data on disk, but we cannot store all data forever so we need ways to control their size automatically. diff --git a/nats-server/configuration/README.md b/nats-server/configuration/README.md index 66481a8..1236971 100644 --- a/nats-server/configuration/README.md +++ b/nats-server/configuration/README.md @@ -117,9 +117,8 @@ authorization: { | [`cluster`](clustering/cluster_config.md) | Configuration map for [cluster](clustering/). | | | [`gateway`](gateways/gateway.md#gateway-configuration-block) | Configuration map for [gateway](gateways/). | | | [`leafnode`](leafnodes/leafnode_conf.md) | Configuration map for a [leafnode](leafnodes/). | | -| [`mqtt`](mqtt/mqtt_conf.md) | Configuration map for a [mqtt](mqtt/). | | -| [`websocket`](websocket/websocket_conf.md) | Configuration map for [websocket](websocket/). | | - +| [`mqtt`](https://github.com/nats-io/nats.docs/tree/53202d44215a11c4c4ad7caea03a703d302bc954/nats-server/configuration/mqtt/mqtt_conf.md) | Configuration map for a [mqtt](mqtt.md). | | +| [`websocket`](https://github.com/nats-io/nats.docs/tree/53202d44215a11c4c4ad7caea03a703d302bc954/nats-server/configuration/websocket/websocket_conf.md) | Configuration map for [websocket](websocket.md). | | ### Connection Timeouts diff --git a/nats-server/configuration/leafnodes/leafnode_conf.md b/nats-server/configuration/leafnodes/leafnode_conf.md index f7ff7e2..936fc13 100644 --- a/nats-server/configuration/leafnodes/leafnode_conf.md +++ b/nats-server/configuration/leafnodes/leafnode_conf.md @@ -93,13 +93,14 @@ If other form of credentials are used \(jwt, nkey or other\), then the server wi | `account` | [Account](../securing_nats/accounts.md) name or jwt public key identifying the local account to bind to this remote server. Any traffic locally on this account will be forwarded to the remote server. | | `credentials` | Credential file for connecting to the leafnode server. | | `tls` | A [TLS configuration](leafnode_conf.md#tls-configuration-block) block. Leafnode client will use specified TLS certificates when connecting/authenticating. | -| `ws_compression` | If connecting with [Websocket](leafnode_conf#connecting-using-websocket-protocol) protocol, this boolean (`true` or `false`) indicates to the remote server that it wishes to use compression. The default is `false`. | -| `ws_no_masking` | If connecting with [Websocket](leafnode_conf#connecting-using-websocket-protocol) protocol, this boolean indicates to the remote server that it wishes not to mask outbound websocket frames. The default is `false`, which means that outbound frames will be masked. | +| `ws_compression` | If connecting with [Websocket](https://github.com/nats-io/nats.docs/tree/0ab2bb72b305dcf4817b8f1fc4f37ab9b0c8a2db/nats-server/configuration/leafnodes/leafnode_conf/README.md#connecting-using-websocket-protocol) protocol, this boolean \(`true` or `false`\) indicates to the remote server that it wishes to use compression. The default is `false`. | +| `ws_no_masking` | If connecting with [Websocket](https://github.com/nats-io/nats.docs/tree/0ab2bb72b305dcf4817b8f1fc4f37ab9b0c8a2db/nats-server/configuration/leafnodes/leafnode_conf/README.md#connecting-using-websocket-protocol) protocol, this boolean indicates to the remote server that it wishes not to mask outbound websocket frames. The default is `false`, which means that outbound frames will be masked. | ### Connecting using Websocket protocol Since NATS 2.2.0, Leaf nodes support outbound websocket connections by specifying `ws` as the scheme component of the remote server URLs: -``` + +```text leafnodes { remotes [ {urls: ["ws://hostname1:443", "ws://hostname2:443"]} @@ -107,18 +108,18 @@ leafnodes { } ``` -Note that if a URL has the `ws` scheme, all URLs the list must be `ws`. You cannot mix and match. -Therefore this would be considered an invalid configuration: -``` +Note that if a URL has the `ws` scheme, all URLs the list must be `ws`. You cannot mix and match. Therefore this would be considered an invalid configuration: + +```text remotes [ # Invalid configuration that will prevent the server from starting {urls: ["ws://hostname1:443", "nats-leaf://hostname2:7422"]} ] ``` -Note that the decision to make a TLS connection is not based on `wss://` (as opposed to `ws://`) but instead in the presence of a TLS configuration in the `leafnodes{}` or the specific remote configuration block. +Note that the decision to make a TLS connection is not based on `wss://` \(as opposed to `ws://`\) but instead in the presence of a TLS configuration in the `leafnodes{}` or the specific remote configuration block. -To configure Websocket in the remote server, check the [Websocket](../websocket/websocket_conf.md) secion. +To configure Websocket in the remote server, check the [Websocket](https://github.com/nats-io/nats.docs/tree/0ab2bb72b305dcf4817b8f1fc4f37ab9b0c8a2db/nats-server/configuration/websocket/websocket_conf.md) secion. ### `tls` Configuration Block diff --git a/nats-server/configuration/monitoring.md b/nats-server/configuration/monitoring.md index 9e00aa5..82d4f44 100644 --- a/nats-server/configuration/monitoring.md +++ b/nats-server/configuration/monitoring.md @@ -596,7 +596,7 @@ The `/jsz` endpoint reports more detailed information on JetStream. For accounts | streams | true, 1, false, 0 | Include streams. When set, implies `accounts=true`. Default is false. | | consumers | true, 1, false, 0 | Include consumer. When set, implies `streams=true`. Default is false. | | config | true, 1, false, 0 | When stream or consumer are requested, include their respective configuration. Default is false. | -| leader-only | true, 1, false, 0 | Only the leader responds. Default is false.| +| leader-only | true, 1, false, 0 | Only the leader responds. Default is false. | | offset | number > 0 | Pagination offset. Default is 0. | | limit | number > 0 | Number of results to return. Default is 1024. | diff --git a/nats-server/configuration/mqtt.md b/nats-server/configuration/mqtt.md new file mode 100644 index 0000000..f452d9c --- /dev/null +++ b/nats-server/configuration/mqtt.md @@ -0,0 +1,128 @@ +# MQTT + +_Supported since NATS Server version 2.2_ + +NATS follows as closely as possible to the MQTT v3.1.1 [specification](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html). + +## When to Use MQTT + +MQTT support in NATS is intended to be an enabling technology allowing users to leverage existing investments in their IoT deployments. Updating software on the edge or endpoints can be onerous and risky, especially when embedded applications are involved. + +In greenfield IoT deployments, when possible, we prefer NATS extended out to endpoints and devices for a few reasons. There are significant advantages with security and observability when using a single technology end to end. Compared to MQTT, NATS is nearly as lightweight in terms of protocol bandwidth and maintainer supported clients efficiently utilize resources so we consider NATS to be a good choice to use end to end, including use on resource constrained devices. + +In existing MQTT deployments or in situations when endpoints can only support MQTT, using a NATS server as a drop-in MQTT server replacement to securely connect to a remote NATS cluster or supercluster is compelling. You can keep your existing IoT investment and use NATS for secure, resilient, and scalable access to your streams and services. + +## JetStream Requirements + +For an MQTT client to connect to the NATS server, the user's account must be JetStream enabled. This is because persistence is needed for the sessions and retained messages since even retained messages of QoS 0 are persisted. + +## MQTT Topics and NATS Subjects + +MQTT Topics are similar to NATS Subjects, but have distinctive differences. + +MQTT topic uses "`/`" as a level separator. For instance `foo/bar` would translate to NATS subject `foo.bar`. But in MQTT, `/foo/bar/` is a valid subject, which, if simply translated, would become `.foo.bar.`, which is NOT a valid NATS Subject. + +NATS Server will convert an MQTT topic following those rules: + +| MQTT character | NATS character\(s\) | Topic \(MQTT\) | Subject \(NATS\) | +| :---: | :---: | :---: | :---: | +| `/` between two levels | `.` | `foo/bar` | `foo.bar` | +| `/` as first level | `/.` | `/foo/bar` | `/.foo.bar` | +| `/` as last level | `./` | `foo/bar/` | `foo.bar./` | +| `/` next to another | `./` | `foo//bar` | `foo./.bar` | +| `/` next to another | `/.` | `//foo/bar` | `/./.foo.bar` | +| `.` | Not Support | `foo.bar` | Not Supported | +| | Not Support | `foo bar` | Not Supported | + +As indicated above, if an MQTT topic contains the character ``` or``.\`, NATS will reject it, causing the connection to be closed for published messages, and returning a failure code in the SUBACK packet for a subscriptions. + +### MQTT Wildcards + +As in NATS, MQTT wildcards represent either multi or single levels. As in NATS, they are allowed only for subscriptions, not for published messages. + +| MQTT Wildcard | NATS Wildcard | +| :---: | :---: | +| `#` | `>` | +| `+` | `*` | + +The wildcard `#` matches any number of levels within a topic, which means that a subscription on `foo/#` would receive messages on `foo/bar`, or `foo/bar/baz`, but also on `foo`. This is not the case in NATS where a subscription on `foo.>` can receive messages on `foo/bar` or `foo/bar/baz`, but not on `foo`. To solve this, NATS Server will create two subscriptions, one on `foo.>` and one on `foo`. If the MQTT subscription is simply on `#`, then a single NATS subscription on `>` is enough. + +The wildcard `+` matches a single level, which means `foo/+` can receive message on `foo/bar` or `foo/baz`, but not on `foo/bar/baz` nor `foo`. This is the same with NATS subscriptions using the wildcard `*`. Therefore `foo/+` would translate to `foo.*`. + +## Communication Between MQTT and NATS + +When an MQTT client creates a subscription on a topic, the NATS server creates the similar NATS subscription \(with conversion from MQTT topic to NATS subject\) so that the interest is registered in the cluster and known to any NATS publishers. + +That is, say an MQTT client connects to server "A" and creates a subscription of `foo/bar`, server "A" creates a subscription on `foo.bar`, which interest is propagated as any other NATS subscription. A publisher connecting anywhere in the cluster and publishing on `foo.bar` would cause server "A" to deliver a QoS 0 message to the MQTT subscription. + +This works the same way for MQTT publishers. When the server receives an MQTT publish message, it is converted to the NATS subject and published, which means that any matching NATS subscription will receive the MQTT message. + +If the MQTT subscription is QoS1 and an MQTT publisher publishes an MQTT QoS1 message on the same or any other server in the cluster, the message will be persisted in the cluster and routed and delivered as QoS 1 to the MQTT subscription. + +## QoS 1 Redeliveries + +When the server delivers a QoS 1 message to a QoS 1 subscription, it will keep the message until it receives the PUBACK for the corresponding packet identifier. If it does not receive it within the "ack\_wait" interval, that message will be resent. + +## Max Ack Pending + +This is the amount of QoS 1 messages the server can send to a subscription without receiving any PUBACK for those messages. The maximum value is 65535. + +The total of subscriptions' `max_ack_pending` on a given session cannot exceed 65535. Attempting to create a subscription that would bring the total above the limit would result in the server returning a failure code in the SUBACK for this subscription. + +Due to how the NATS server handles the MQTT "`#`" wildcard, each subscription ending with "`#`" will use 2 times the `max_ack_pending` value. + +## Sessions + +NATS Server will persist all sessions, even if they are created with the "clean session" flag, meaning that sessions only last for the duration of the network connection between the client and the server. + +A session is identified by a client identifier. If two connections try to use the same client identifier, the server, per specification, will close the existing connection and accept the new one. + +If the user incorrectly starts two applications that use the same client identifier, this would result in a very quick flapping if the MQTT client has a reconnect feature and quickly reconnects. + +To prevent this, the NATS server will accept the new session and will delay the closing of the old connection to reduce the flapping rate. + +Detection of the concurrent use of sessions also works in cluster mode. + +## Retained Messages + +When a server receives an MQTT publish packet with the RETAIN flag set \(regardless of its QoS\), it stores the application message and its QoS, so that it can be delivered to future subscribers whose subscriptions match its topic name. + +When a new subscription is established, the last retained message, if any, on each matching topic name will be sent to the subscriber. + +A PUBLISH Packet with a RETAIN flag set to 1 and a payload containing zero bytes will be processed as normal and sent to clients with a subscription matching the topic name. Additionally any existing retained message with the same topic name will be removed and any future subscribers for the topic will not receive a retained message. + +## Clustering + +NATS supports MQTT in a NATS cluster. The replication factor is automatically set based on the size of the cluster. + +### Connections with Same Client ID + +If a client is connected to a server "A" in the cluster and another client connects to a server "B" and uses the same client identifier, server "A" will close its client connection upon discovering the use of an active client identifier. + +Users should avoid this situation as this is not as easy and immediate as if the two applications are connected to the same server. + +There may be cases where the server will reject the new connection if there is no safe way to close the existing connection, such as when it is in the middle of processing some MQTT packets. + +### Retained Messages + +Retained messages are stored in the cluster and available to any server in the cluster. However, this is not immediate and if a producer connects to a server and produces a retained message and another connection connects to another server and starts a matching subscription, it may not receive the retained message if the server it connects to has not yet been made aware of this retained message. + +In other words, retained messages in clustering mode is best-effort, and applications that rely on the presence of a retained message should connect on the server that produced them. + +## Limitations + +* NATS does not support QoS 2 messages. If it receives a published message with QoS greater than 1, + + it will close the connection. + +* NATS messages published to MQTT subscriptions are always delivered as QoS 0 messages. +* MQTT published messages on topic names containing "```" or "``.\`" characters will cause the + + connection to be closed. Presence of those characters in MQTT subscriptions will result in error + + code in the SUBACK packet. + +* MQTT wildcard `#` may cause the NATS server to create two subscriptions. +* MQTT concurrent sessions may result in the new connection to be evicted instead of the existing one. +* MQTT retained messages in clustering mode is best effort. + diff --git a/nats-server/configuration/mqtt/README.md b/nats-server/configuration/mqtt/README.md deleted file mode 100644 index 29f21a8..0000000 --- a/nats-server/configuration/mqtt/README.md +++ /dev/null @@ -1,173 +0,0 @@ -# MQTT - -*Supported since NATS Server version 2.2* - -NATS follows as closely as possible to the MQTT v3.1.1 [specification](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html). - -## When to Use MQTT - -MQTT support in NATS is intended to be an enabling technology allowing users to leverage existing -investments in their IoT deployments. Updating software on the edge or endpoints can be onerous -and risky, especially when embedded applications are involved. - -In greenfield IoT deployments, when possible, we prefer NATS extended out to endpoints and devices -for a few reasons. There are significant advantages with security and observability when using a -single technology end to end. Compared to MQTT, NATS is nearly as lightweight in terms of protocol -bandwidth and maintainer supported clients efficiently utilize resources so we consider NATS to be a -good choice to use end to end, including use on resource constrained devices. - -In existing MQTT deployments or in situations when endpoints can only support MQTT, using a NATS server -as a drop-in MQTT server replacement to securely connect to a remote NATS cluster or supercluster is -compelling. You can keep your existing IoT investment and use NATS for secure, resilient, and -scalable access to your streams and services. - -## JetStream Requirements - -For an MQTT client to connect to the NATS server, the user's account must be JetStream enabled. -This is because persistence is needed for the sessions and retained messages since even retained messages -of QoS 0 are persisted. - -## MQTT Topics and NATS Subjects - -MQTT Topics are similar to NATS Subjects, but have distinctive differences. - -MQTT topic uses "`/`" as a level separator. For instance `foo/bar` would translate to NATS subject `foo.bar`. -But in MQTT, `/foo/bar/` is a valid subject, which, if simply translated, would become `.foo.bar.`, which -is NOT a valid NATS Subject. - -NATS Server will convert an MQTT topic following those rules: - -| MQTT character | NATS character(s) | Topic (MQTT) | Subject (NATS) | -| :---: | :---: | :---: | :---: | -| `/` between two levels | `.` | `foo/bar` | `foo.bar` | -| `/` as first level | `/.` | `/foo/bar` | `/.foo.bar` | -| `/` as last level | `./` | `foo/bar/` | `foo.bar./` | -| `/` next to another | `./` | `foo//bar` | `foo./.bar` | -| `/` next to another | `/.` | `//foo/bar` | `/./.foo.bar` | -| `.` | Not Support | `foo.bar` | Not Supported | -| ` ` | Not Support | `foo bar` | Not Supported | - -As indicated above, if an MQTT topic contains the character ` ` or `.`, NATS will reject it, -causing the connection to be closed for published messages, and returning a failure code in -the SUBACK packet for a subscriptions. - -### MQTT Wildcards - -As in NATS, MQTT wildcards represent either multi or single levels. As in NATS, they are -allowed only for subscriptions, not for published messages. - -| MQTT Wildcard | NATS Wildcard | -| :---: | :---: | -| `#` | `>` | -| `+` | `*` | - -The wildcard `#` matches any number of levels within a topic, which means that a subscription -on `foo/#` would receive messages on `foo/bar`, or `foo/bar/baz`, but also on `foo`. -This is not the case in NATS where a subscription on `foo.>` can receive messages on `foo/bar` -or `foo/bar/baz`, but not on `foo`. To solve this, NATS Server will create two subscriptions, -one on `foo.>` and one on `foo`. If the MQTT subscription is simply on `#`, then a single -NATS subscription on `>` is enough. - -The wildcard `+` matches a single level, which means `foo/+` can receive message on `foo/bar` or -`foo/baz`, but not on `foo/bar/baz` nor `foo`. This is the same with NATS subscriptions using -the wildcard `*`. Therefore `foo/+` would translate to `foo.*`. - -## Communication Between MQTT and NATS - -When an MQTT client creates a subscription on a topic, the NATS server creates the similar -NATS subscription (with conversion from MQTT topic to NATS subject) so that the interest -is registered in the cluster and known to any NATS publishers. - -That is, say an MQTT client connects to server "A" and creates a subscription of `foo/bar`, -server "A" creates a subscription on `foo.bar`, which interest is propagated as any other -NATS subscription. A publisher connecting anywhere in the cluster and publishing on `foo.bar` -would cause server "A" to deliver a QoS 0 message to the MQTT subscription. - -This works the same way for MQTT publishers. When the server receives an MQTT publish -message, it is converted to the NATS subject and published, which means that any matching NATS -subscription will receive the MQTT message. - -If the MQTT subscription is QoS1 and an MQTT publisher publishes an MQTT QoS1 message on -the same or any other server in the cluster, the message will be persisted in the cluster -and routed and delivered as QoS 1 to the MQTT subscription. - -## QoS 1 Redeliveries - -When the server delivers a QoS 1 message to a QoS 1 subscription, it will keep the message -until it receives the PUBACK for the corresponding packet identifier. If it does not receive -it within the "ack_wait" interval, that message will be resent. - -## Max Ack Pending - -This is the amount of QoS 1 messages the server can send to a subscription without receiving -any PUBACK for those messages. The maximum value is 65535. - -The total of subscriptions' `max_ack_pending` on a given session cannot exceed 65535. Attempting -to create a subscription that would bring the total above the limit would result in the server -returning a failure code in the SUBACK for this subscription. - -Due to how the NATS server handles the MQTT "`#`" wildcard, each subscription ending with "`#`" -will use 2 times the `max_ack_pending` value. - -## Sessions - -NATS Server will persist all sessions, even if they are created with the "clean session" flag, meaning -that sessions only last for the duration of the network connection between the client and the server. - -A session is identified by a client identifier. If two connections try to use the same client identifier, -the server, per specification, will close the existing connection and accept the new one. - -If the user incorrectly starts two applications that use the same client identifier, this would result -in a very quick flapping if the MQTT client has a reconnect feature and quickly reconnects. - -To prevent this, the NATS server will accept the new session and will delay the closing of the -old connection to reduce the flapping rate. - -Detection of the concurrent use of sessions also works in cluster mode. - -## Retained Messages - -When a server receives an MQTT publish packet with the RETAIN flag set (regardless of its QoS), it stores the application message and its QoS, so that it can be delivered to future subscribers whose subscriptions match its topic name. - -When a new subscription is established, the last retained message, if any, on each matching topic name will be sent to the subscriber. - -A PUBLISH Packet with a RETAIN flag set to 1 and a payload containing zero bytes will be processed as normal and sent to clients with a subscription matching the topic name. Additionally any existing retained message with the same topic name will be removed and any future subscribers for the topic will not receive a retained message. - -## Clustering - -NATS supports MQTT in a NATS cluster. The replication factor is automatically set based on the size -of the cluster. - -### Connections with Same Client ID - -If a client is connected to a server "A" in the cluster and another client connects to a server "B" and -uses the same client identifier, server "A" will close its client connection upon discovering the use of -an active client identifier. - -Users should avoid this situation as this is not as easy and immediate as if the two applications are connected to the same server. - -There may be cases where the server will reject the new connection if there is no safe way to -close the existing connection, such as when it is in the middle of processing some MQTT packets. - -### Retained Messages - -Retained messages are stored in the cluster and available to any server in the cluster. However, -this is not immediate and if a producer connects to a server and produces a retained message -and another connection connects to another server and starts a matching subscription, it -may not receive the retained message if the server it connects to has not yet been made -aware of this retained message. - -In other words, retained messages in clustering mode is best-effort, and applications that rely on the -presence of a retained message should connect on the server that produced them. - -## Limitations - -- NATS does not support QoS 2 messages. If it receives a published message with QoS greater than 1, -it will close the connection. -- NATS messages published to MQTT subscriptions are always delivered as QoS 0 messages. -- MQTT published messages on topic names containing "` `" or "`.`" characters will cause the -connection to be closed. Presence of those characters in MQTT subscriptions will result in error -code in the SUBACK packet. -- MQTT wildcard `#` may cause the NATS server to create two subscriptions. -- MQTT concurrent sessions may result in the new connection to be evicted instead of the existing one. -- MQTT retained messages in clustering mode is best effort. diff --git a/nats-server/configuration/securing_nats/accounts.md b/nats-server/configuration/securing_nats/accounts.md index c2c8299..572bef4 100644 --- a/nats-server/configuration/securing_nats/accounts.md +++ b/nats-server/configuration/securing_nats/accounts.md @@ -189,7 +189,6 @@ no_auth_user: a The above example shows how clients without authentication can be associated with the user `a` within account `A`. > Please note that the `no_auth_user` will not work with nkeys. The user referenced can also be part of the [authorization](authorization.md) block. +> +> Despite `no_auth_user` being set, clients still need to communicate that they will not be using credentials. The [authentication timeout](auth_intro/auth_timeout.md) applies to this process as well. When your connection is slow, you may run into this timeout and the resulting `Authentication Timeout` error, despite not providing credentials. -> Despite `no_auth_user` being set, clients still need to communicate that they will not be using credentials. -> The [authentication timeout](auth_intro/auth_timeout.md) applies to this process as well. -> When your connection is slow, you may run into this timeout and the resulting `Authentication Timeout` error, despite not providing credentials. diff --git a/nats-server/configuration/securing_nats/jwt/resolver.md b/nats-server/configuration/securing_nats/jwt/resolver.md index 3a52ef9..afedd47 100644 --- a/nats-server/configuration/securing_nats/jwt/resolver.md +++ b/nats-server/configuration/securing_nats/jwt/resolver.md @@ -1,4 +1,4 @@ -# Account Lookup Using a Resolver +# Account lookup using Resolver The `resolver` configuration option is used in conjunction with [NATS JWT Authentication](./) and [nsc](../../../../nats-tools/nsc/). The `resolver` option specifies a URL where the nats-server can retrieve an account JWT. There are three built-in resolver implementations: @@ -37,15 +37,11 @@ For more information on how to configure a memory resolver, see [this tutorial]( ## NATS Based Resolver -The NATS based resolver embeds the functionality of the [account server](https://github.com/nats-io/nats-account-server) inside the nats-server. -In order to avoid having to store all account JWT on every server, this resolver has two sub types `full` and `cache`. -Their commonalities are that they exchange/lookup account JWT via NATS and the system account, and store them in a local (not shared) directory. +The NATS based resolver embeds the functionality of the [account server](https://github.com/nats-io/nats-account-server) inside the nats-server. In order to avoid having to store all account JWT on every server, this resolver has two sub types `full` and `cache`. Their commonalities are that they exchange/lookup account JWT via NATS and the system account, and store them in a local \(not shared\) directory. ### Full -The Full resolver stores all JWTs and exchanges them in an eventually consistent way with other resolvers of the same type. -[`nsc`](../../../../nats-tools/nsc/README.md) supports push/pull/purge with this resolver type. -[JWTs](../../nats-server/configuration/securing_nats/jwt/), uploaded this way, are stored in a directory the server has exclusive access to. +The Full resolver stores all JWTs and exchanges them in an eventually consistent way with other resolvers of the same type. [`nsc`](../../../../nats-tools/nsc/) supports push/pull/purge with this resolver type. [JWTs](https://github.com/nats-io/nats.docs/tree/8c85d9c047d2203c7867b62a8415cdfa4d117f04/nats-server/configuration/nats-server/configuration/securing_nats/jwt/README.md), uploaded this way, are stored in a directory the server has exclusive access to. ```yaml resolver: { @@ -67,17 +63,13 @@ resolver: { } ``` -This resolver type also supports `resolver_preload`. When present, JWTs are listed and stored in the resolver. -There, they may be subject to updates. Restarts of the `nats-server` will hold on to these more recent versions. +This resolver type also supports `resolver_preload`. When present, JWTs are listed and stored in the resolver. There, they may be subject to updates. Restarts of the `nats-server` will hold on to these more recent versions. -Not every server in a cluster needs to be set to `full`. -You need enough to still serve your workload adequately, while some servers are offline. +Not every server in a cluster needs to be set to `full`. You need enough to still serve your workload adequately, while some servers are offline. ### Cache -The Cache resolver only stores a subset of [JWT](../../nats-server/configuration/securing_nats/jwt/) and evicts others based on an LRU scheme. -Missing JWTs are downloaded from `full` nats based resolver. -This resolver is essentially the URL Resolver in NATS. +The Cache resolver only stores a subset of [JWT](https://github.com/nats-io/nats.docs/tree/8c85d9c047d2203c7867b62a8415cdfa4d117f04/nats-server/configuration/nats-server/configuration/securing_nats/jwt/README.md) and evicts others based on an LRU scheme. Missing JWTs are downloaded from `full` nats based resolver. This resolver is essentially the URL Resolver in NATS. ```yaml resolver: { @@ -93,10 +85,9 @@ resolver: { ### NATS Based Resolver - Integration -The NATS based resolver utilizes the system account for lookup and upload of account [JWTs](../../nats-server/configuration/securing_nats/jwt/) . -If your application requires tighter integration you can make use of these subjects for tighter integration. +The NATS based resolver utilizes the system account for lookup and upload of account [JWTs](https://github.com/nats-io/nats.docs/tree/8c85d9c047d2203c7867b62a8415cdfa4d117f04/nats-server/configuration/nats-server/configuration/securing_nats/jwt/README.md) . If your application requires tighter integration you can make use of these subjects for tighter integration. -To upload or update any generated account JWT without [`nsc`](../../../../nats-tools/nsc/README.md), send it as a request to `$SYS.REQ.CLAIMS.UPDATE`. -Each participating `full` NATS based account resolver will respond with a message detailing success or failure. +To upload or update any generated account JWT without [`nsc`](../../../../nats-tools/nsc/), send it as a request to `$SYS.REQ.CLAIMS.UPDATE`. Each participating `full` NATS based account resolver will respond with a message detailing success or failure. + +To serve a requested account [JWT](https://github.com/nats-io/nats.docs/tree/8c85d9c047d2203c7867b62a8415cdfa4d117f04/nats-server/configuration/nats-server/configuration/securing_nats/jwt/README.md) yourself and essentially implement an account server, subscribe to `$SYS.REQ.ACCOUNT.*.CLAIMS.LOOKUP` and respond with the account JWT corresponding to the requested account id \(wildcard\). -To serve a requested account [JWT](../../nats-server/configuration/securing_nats/jwt/) yourself and essentially implement an account server, subscribe to `$SYS.REQ.ACCOUNT.*.CLAIMS.LOOKUP` and respond with the account JWT corresponding to the requested account id (wildcard). diff --git a/nats-server/configuration/websocket.md b/nats-server/configuration/websocket.md new file mode 100644 index 0000000..8f133c4 --- /dev/null +++ b/nats-server/configuration/websocket.md @@ -0,0 +1,11 @@ +# Websocket + +_Supported since NATS Server version 2.2_ + +Websocket support can be enabled in the server and may be used alongside the traditional TCP socket connections. TLS, compression and Origin Header checking are supported. + +**Important** + +* NATS Supports only Websocket data frames in Binary, not Text format \([https://tools.ietf.org/html/rfc6455\#section-5.6](https://tools.ietf.org/html/rfc6455#section-5.6)\). The server will always send in Binary and your clients MUST send in Binary too. +* For writers of client libraries: a Websocket frame is not guaranteed to contain a full NATS protocol \(actually will generally not\). Any data from a frame must be going through a parser that can handle partial protocols. See the protocol description [here](../../nats-protocol/nats-protocol/). + diff --git a/nats-server/configuration/websocket/README.md b/nats-server/configuration/websocket/README.md deleted file mode 100644 index 2c17afa..0000000 --- a/nats-server/configuration/websocket/README.md +++ /dev/null @@ -1,12 +0,0 @@ -# Websocket - -*Supported since NATS Server version 2.2* - -Websocket support can be enabled in the server and may be used alongside the -traditional TCP socket connections. TLS, compression and -Origin Header checking are supported. - -**Important** - -- NATS Supports only Websocket data frames in Binary, not Text format (https://tools.ietf.org/html/rfc6455#section-5.6). The server will always send in Binary and your clients MUST send in Binary too. -- For writers of client libraries: a Websocket frame is not guaranteed to contain a full NATS protocol (actually will generally not). Any data from a frame must be going through a parser that can handle partial protocols. See the protocol description [here](../../../nats-protocol/nats-protocol/README.md). diff --git a/nats-tools/nas/README.md b/nats-tools/nas/README.md index 3cac2be..709c835 100644 --- a/nats-tools/nas/README.md +++ b/nats-tools/nas/README.md @@ -2,13 +2,11 @@ The [NATS Account Server](https://github.com/nats-io/nats-account-server) is an HTTP server that hosts and vends [JWTs](../../nats-server/configuration/securing_nats/jwt/) for nats-server 2.0 account authentication. The server supports an number of stores which enable it to serve account [JWTs](../../nats-server/configuration/securing_nats/jwt/) from a [directory](nas_conf.md#directory-configuration) -> The nats server can be configured with a [memory resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#memory) as well. This avoids usage of the account server. -> The NATS server can be configured with a [NATS based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) for the same purpose as well. -> +> The nats server can be configured with a [memory resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#memory) as well. This avoids usage of the account server. The NATS server can be configured with a [NATS based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) for the same purpose as well. +> > Usage of [full NATS based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) over [NATS Account Server](https://github.com/nats-io/nats-account-server) is recommended. -> -> The [NATS Account Server](https://github.com/nats-io/nats-account-server) also speaks the [full nats based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) protocol and -> can be used as such. +> +> The [NATS Account Server](https://github.com/nats-io/nats-account-server) also speaks the [full nats based resolver](../../nats-server/configuration/securing_nats/jwt/resolver.md#nats-based-resolver) protocol and can be used as such. The server can operate in a _READ ONLY_ mode where it serves content from a directory, or in [notification mode](notifications.md), where it can notify a NATS server that a JWT in the store has been modified, updating the NATS server with the updated JWT. @@ -17,3 +15,4 @@ The server supports replica mode, which allows load balancing, fault tolerance a The account server can host activation tokens as well as account JWTs. These tokens are used when one account needs to give permission to another account to access a private export. Tokens can be configured as full tokens, or URLs. By hosting them in the account server you can avoid the copy/paste process of embedding tokens. They can also be updated more easily on expiration. The account serer furthermore allows for jwt inspection. All account server configuration options can be found [here](nas_conf.md#configuration-file). It futhermore allows [inspection](inspecting_jwts.md) of JWT. + From bd4a6dc5e10a23c666c6c2249ec8dd4182ec2f78 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 15 Mar 2021 16:43:35 +0000 Subject: [PATCH 84/84] GitBook: [master] 2 pages modified --- jetstream/configuration_mgmt/README.md | 30 +------------------ .../configuration_mgmt/nats-admin-cli.md | 28 +++++++++++++++++ 2 files changed, 29 insertions(+), 29 deletions(-) diff --git a/jetstream/configuration_mgmt/README.md b/jetstream/configuration_mgmt/README.md index 59a6459..30ca1a4 100644 --- a/jetstream/configuration_mgmt/README.md +++ b/jetstream/configuration_mgmt/README.md @@ -4,36 +4,8 @@ In many cases managing the configuration in your application code is the best mo We support a number of tools to assist with this: -* `nats` CLI with configuration files +* [CLI with Configuration Files](nats-admin-cli.md) * [Terraform](terraform.md) * [GitHub Actions](github_actions.md) * [Kubernetes JetStream Controller](kubernetes_controller.md) -## nats Admin CLI - -The `nats` CLI can be used to manage Streams and Consumers easily using it's `--config` flag, for example: - -## Add a new Stream - -This creates a new Stream based on `orders.json`. The `orders.json` file can be extracted from an existing stream using `nats stream info ORDERS -j | jq .config` - -```text -$ nats str add ORDERS --config orders.json -``` - -## Edit an existing Stream - -This edits an existing stream ensuring it complies with the configuration in `orders.json` - -```text -$ nats str edit ORDERS --config orders.json -``` - -## Add a New Consumer - -This creates a new Consumer based on `orders_new.json`. The `orders_new.json` file can be extracted from an existing stream using `nats con info ORDERS NEW -j | jq .config` - -```text -$ nats con add ORDERS NEW --config orders_new.json -``` - diff --git a/jetstream/configuration_mgmt/nats-admin-cli.md b/jetstream/configuration_mgmt/nats-admin-cli.md index 58f1a8b..fb16025 100644 --- a/jetstream/configuration_mgmt/nats-admin-cli.md +++ b/jetstream/configuration_mgmt/nats-admin-cli.md @@ -1,2 +1,30 @@ # NATS Admin CLI +## nats Admin CLI + +The `nats` CLI can be used to manage Streams and Consumers easily using it's `--config` flag, for example: + +## Add a new Stream + +This creates a new Stream based on `orders.json`. The `orders.json` file can be extracted from an existing stream using `nats stream info ORDERS -j | jq .config` + +```text +$ nats str add ORDERS --config orders.json +``` + +## Edit an existing Stream + +This edits an existing stream ensuring it complies with the configuration in `orders.json` + +```text +$ nats str edit ORDERS --config orders.json +``` + +## Add a New Consumer + +This creates a new Consumer based on `orders_new.json`. The `orders_new.json` file can be extracted from an existing stream using `nats con info ORDERS NEW -j | jq .config` + +```text +$ nats con add ORDERS NEW --config orders_new.json +``` +