From 1fd9cab1a5cc8f8e54417352b16bf210681469d7 Mon Sep 17 00:00:00 2001 From: ainsley Date: Tue, 24 Nov 2020 15:12:44 -0600 Subject: [PATCH 01/31] adding JetStream Docs --- jetstream/administration/account.md | 247 +++++++++ jetstream/administration/administration.md | 30 ++ jetstream/administration/consumers.md | 219 ++++++++ jetstream/administration/streams.md | 235 +++++++++ jetstream/concepts/concepts.md | 19 + jetstream/concepts/configuration.md | 10 + jetstream/concepts/consumers.md | 33 ++ jetstream/concepts/streams.md | 31 ++ .../configuration_mgmt/configuration_mgmt.md | 105 ++++ .../configuration_mgmt/github_actions.md | 55 ++ .../kubernetes_controller.md | 57 +++ .../disaster_recovery/disaster_recovery.md | 85 ++++ jetstream/getting_started/getting_started.md | 3 + jetstream/getting_started/using_docker.md | 50 ++ jetstream/getting_started/using_source.md | 65 +++ jetstream/model_deep_dive/model_deep_dive.md | 481 ++++++++++++++++++ jetstream/monitoring/monitoring.md | 30 ++ .../multi-tenancy/resource_management.md | 133 +++++ .../nats_api_reference/nats_api_reference.md | 221 ++++++++ 19 files changed, 2109 insertions(+) create mode 100644 jetstream/administration/account.md create mode 100644 jetstream/administration/administration.md create mode 100644 jetstream/administration/consumers.md create mode 100644 jetstream/administration/streams.md create mode 100644 jetstream/concepts/concepts.md create mode 100644 jetstream/concepts/configuration.md create mode 100644 jetstream/concepts/consumers.md create mode 100644 jetstream/concepts/streams.md create mode 100644 jetstream/configuration_mgmt/configuration_mgmt.md create mode 100644 jetstream/configuration_mgmt/github_actions.md create mode 100644 jetstream/configuration_mgmt/kubernetes_controller.md create mode 100644 jetstream/disaster_recovery/disaster_recovery.md create mode 100644 jetstream/getting_started/getting_started.md create mode 100644 jetstream/getting_started/using_docker.md create mode 100644 jetstream/getting_started/using_source.md create mode 100644 jetstream/model_deep_dive/model_deep_dive.md create mode 100644 jetstream/monitoring/monitoring.md create mode 100644 jetstream/multi-tenancy/resource_management.md create mode 100644 jetstream/nats_api_reference/nats_api_reference.md diff --git a/jetstream/administration/account.md b/jetstream/administration/account.md new file mode 100644 index 0000000..33761ac --- /dev/null +++ b/jetstream/administration/account.md @@ -0,0 +1,247 @@ +### Account Information + +JetStream is multi-tenant so you will need to check that your account is enabled for JetStream and is not limited. You can view your limits as follows: + +```nohighlight +$ nats account info + + Memory: 0 B of 6.4 GB + Storage: 0 B of 1.1 TB + Streams: 1 of Unlimited +``` + +### Streams + +The first step is to set up storage for our `ORDERS` related messages, these arrive on a wildcard of subjects all flowing into the same Stream and they are kept for 1 year. + +#### Creating + +```nohighlight +$ nats str add ORDERS +? Subjects to consume ORDERS.* +? Storage backend file +? Retention Policy Limits +? Discard Policy Old +? Message count limit -1 +? Message size limit -1 +? Maximum message age limit 1y +? Maximum individual message size [? for help] (-1) -1 +Stream ORDERS was created + +Information for Stream ORDERS + +Configuration: + + Subjects: ORDERS.* + Acknowledgements: true + Retention: File - Limits + Replicas: 1 + Maximum Messages: -1 + Maximum Bytes: -1 + Maximum Age: 8760h0m0s + Maximum Message Size: -1 + Maximum Consumers: -1 + +Statistics: + + Messages: 0 + Bytes: 0 B + FirstSeq: 0 + LastSeq: 0 + Active Consumers: 0 +``` + +You can get prompted interactively for missing information as above, or do it all on one command. Pressing `?` in the CLI will help you map prompts to CLI options: + +``` +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old +``` + +Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: + +``` +$ nats str add ORDERS --config orders.json +``` + +#### Listing + +We can confirm our Stream was created: + +```nohighlight +$ nats str ls +Streams: + + ORDERS +``` + +#### Querying + +Information about the configuration of the Stream can be seen, and if you did not specify the Stream like below, it will prompt you based on all known ones: + +```nohighlight +$ nats str info ORDERS +Information for Stream ORDERS + +Configuration: + + Subjects: ORDERS.* + No Acknowledgements: false + Retention: File - Limits + Replicas: 1 + Maximum Messages: -1 + Maximum Bytes: -1 + Maximum Age: 8760h0m0s + Maximum Consumers: -1 + +Statistics: + + Messages: 0 + Bytes: 0 B + FirstSeq: 0 + LastSeq: 0 + Active Consumers: 0 +``` + +Most commands that show data as above support `-j` to show the results as JSON: + +```nohighlight +$ nats str info ORDERS -j +{ + "config": { + "name": "ORDERS", + "subjects": [ + "ORDERS.*" + ], + "retention": "limits", + "max_consumers": -1, + "max_msgs": -1, + "max_bytes": -1, + "max_age": 31536000000000000, + "storage": "file", + "num_replicas": 1 + }, + "stats": { + "messages": 0, + "bytes": 0, + "first_seq": 0, + "last_seq": 0, + "consumer_count": 0 + } +} +``` + +This is the general pattern for the entire `nats` utility as it relates to JetStream - prompting for needed information but every action can be run non-interactively making it usable as a cli api. All information output like seen above can be turned into JSON using `-j`. + +#### Copying + +A stream can be copied into another, which also allows the configuration of the new one to be adjusted via CLI flags: + +```nohighlight +$ nats str cp ORDERS ARCHIVE --subjects "ORDERS_ARCVHIVE.*" --max-age 2y +Stream ORDERS was created + +Information for Stream ARCHIVE + +Configuration: + + Subjects: ORDERS_ARCVHIVE.* +... + Maximum Age: 17520h0m0s +... +``` + +#### Editing + +A stream configuration can be edited, which allows the configuration to be adjusted via CLI flags. Here I have a incorrectly created ORDERS stream that I fix: + +```nohighlight +$ nats str info ORDERS -j | jq .config.subjects +[ + "ORDERS.new" +] + +$ nats str edit ORDERS --subjects "ORDERS.*" +Stream ORDERS was updated + +Information for Stream ORDERS + +Configuration: + + Subjects: ORDERS.* +.... +``` + +Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: + +``` +$ nats str edit ORDERS --config orders.json +``` + +#### Publishing Into a Stream + +Now let's add in some messages to our Stream. You can use `nats pub` to add messages, pass the `--wait` flag to see the publish ack being returned. + +You can publish without waiting for acknowledgement: + +```nohighlight +$ nats pub ORDERS.scratch hello +Published [sub1] : 'hello' +``` + +But if you want to be sure your messages got to JetStream and were persisted you can make a request: + +```nohighlight +$ nats req ORDERS.scratch hello +13:45:03 Sending request on [ORDERS.scratch] +13:45:03 Received on [_INBOX.M8drJkd8O5otORAo0sMNkg.scHnSafY]: '+OK' +``` + +Keep checking the status of the Stream while doing this and you'll see it's stored messages increase. + +```nohighlight +$ nats str info ORDERS +Information for Stream ORDERS +... +Statistics: + + Messages: 3 + Bytes: 147 B + FirstSeq: 1 + LastSeq: 3 + Active Consumers: 0 +``` + +After putting some throw away data into the Stream, we can purge all the data out - while keeping the Stream active: + +#### Deleting All Data + +To delete all data in a stream use `purge`: + +```nohighlight +$ nats str purge ORDERS -f +... +Statistics: + + Messages: 0 + Bytes: 0 B + FirstSeq: 1,000,001 + LastSeq: 1,000,000 + Active Consumers: 0 +``` + +#### Deleting A Message + +A single message can be securely removed from the stream: + +```nohighlight +$ nats str rmm ORDERS 1 -f +``` + +#### Deleting Sets + +Finally for demonstration purposes, you can also delete the whole Stream and recreate it so then we're ready for creating the Consumers: + +``` +$ nats str rm ORDERS -f +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 +``` diff --git a/jetstream/administration/administration.md b/jetstream/administration/administration.md new file mode 100644 index 0000000..8dcd564 --- /dev/null +++ b/jetstream/administration/administration.md @@ -0,0 +1,30 @@ +## Administration and Usage from the CLI + +Once the server is running it's time to use the management tool. This can be downloaded from the [GitHub Release Page](https://github.com/nats-io/jetstream/releases/) or you can use the `synadia/jsm:latest` docker image. + +``` +$ nats --help +usage: nats [] [ ...] +NATS Management Utility + +Flags: + --help Show context-sensitive help (also try --help-long and --help-man). + --version Show application version. + -s, --server="localhost:4222" NATS servers + --creds=CREDS User credentials + --tlscert=TLSCERT TLS public certifcate + --tlskey=TLSKEY TLS private key + --tlsca=TLSCA TLS certifcate authority chain + --timeout=2s Time to give JetStream to respond to queries + +Commands: + help [...] + Show help. + ... +``` + +We'll walk through the above scenario and introduce features of the CLI and of JetStream as we recreate the setup above. + +Throughout this example, we'll show other commands like `nats pub` and `nats sub` to interact with the system. These are normal existing core NATS commands and JetStream is fully usable by only using core NATS. + +We'll touch on some additional features but please review the section on the design model to understand all possible permutations. \ No newline at end of file diff --git a/jetstream/administration/consumers.md b/jetstream/administration/consumers.md new file mode 100644 index 0000000..e3e21ac --- /dev/null +++ b/jetstream/administration/consumers.md @@ -0,0 +1,219 @@ +### Consumers + +Consumers is how messages are read or consumed from the Stream. We support pull and push-based Consumers and the example scenario has both, lets walk through that. + +#### Creating Pull-Based Consumers + +The `NEW` and `DISPATCH` Consumers are pull-based, meaning the services consuming data from them have to ask the system for the next available message. This means you can easily scale your services up by adding more workers and the messages will get spread across the workers based on their availability. + +Pull-based Consumers are created the same as push-based Consumers, just don't specify a delivery target. + +``` +$ nats con ls ORDERS +No Consumers defined +``` + +We have no Consumers, lets add the `NEW` one: + +I supply the `--sample` options on the CLI as this is not prompted for at present, everything else is prompted. The help in the CLI explains each: + +``` +$ nats con add --sample 100 +? Select a Stream ORDERS +? Consumer name NEW +? Delivery target +? Start policy (all, last, 1h, msg sequence) all +? Filter Stream by subject (blank for all) ORDERS.received +? Maximum Allowed Deliveries 20 +Information for Consumer ORDERS > NEW + +Configuration: + + Durable Name: NEW + Pull Mode: true + Subject: ORDERS.received + Deliver All: true + Deliver Last: false + Ack Policy: explicit + Ack Wait: 30s + Replay Policy: instant + Maximum Deliveries: 20 + Sampling Rate: 100 + +State: + + Last Delivered Message: Consumer sequence: 1 Stream sequence: 1 + Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0 + Pending Messages: 0 + Redelivered Messages: 0 +``` + +This is a pull-based Consumer (empty Delivery Target), it gets messages from the first available message and requires specific acknowledgement of each and every message. + +It only received messages that originally entered the Stream on `ORDERS.received`. Remember the Stream subscribes to `ORDERS.*`, this lets us select a subset of messages from the Stream. + +A Maximum Delivery limit of 20 is set, this means if the message is not acknowledged it will be retried but only up to this maximum total deliveries. + +Again this can all be done in a single CLI call, lets make the `DISPATCH` Consumer: + +``` +$ nats con add ORDERS DISPATCH --filter ORDERS.processed --ack explicit --pull --deliver all --sample 100 --max-deliver 20 +``` + +Additionally, one can store the configuration in a JSON file, the format of this is the same as `$ nats con info ORDERS DISPATCH -j | jq .config`: + +``` +$ nats con add ORDERS MONITOR --config monitor.json +``` + +#### Creating Push-Based Consumers + +Our `MONITOR` Consumer is push-based, has no ack and will only get new messages and is not sampled: + +``` +$ nats con add +? Select a Stream ORDERS +? Consumer name MONITOR +? Delivery target monitor.ORDERS +? Start policy (all, last, 1h, msg sequence) last +? Acknowledgement policy none +? Replay policy instant +? Filter Stream by subject (blank for all) +? Maximum Allowed Deliveries -1 +Information for Consumer ORDERS > MONITOR + +Configuration: + + Durable Name: MONITOR + Delivery Subject: monitor.ORDERS + Deliver All: false + Deliver Last: true + Ack Policy: none + Replay Policy: instant + +State: + + Last Delivered Message: Consumer sequence: 1 Stream sequence: 3 + Acknowledgment floor: Consumer sequence: 0 Stream sequence: 2 + Pending Messages: 0 + Redelivered Messages: 0 +``` + +Again you can do this with a single non interactive command: + +``` +$ nats con add ORDERS MONITOR --ack none --target monitor.ORDERS --deliver last --replay instant --filter '' +``` + +Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats con info ORDERS MONITOR -j | jq .config`: + +``` +$ nats con add ORDERS --config monitor.json +``` + +#### Listing + +You can get a quick list of all the Consumers for a specific Stream: + +``` +$ nats con ls ORDERS +Consumers for Stream ORDERS: + + DISPATCH + MONITOR + NEW +``` + +#### Querying + +All details for an Consumer can be queried, lets first look at a pull-based Consumer: + +``` +$ nats con info ORDERS DISPATCH +Information for Consumer ORDERS > DISPATCH + +Configuration: + + Durable Name: DISPATCH + Pull Mode: true + Subject: ORDERS.processed + Deliver All: true + Deliver Last: false + Ack Policy: explicit + Ack Wait: 30s + Replay Policy: instant + Sampling Rate: 100 + +State: + + Last Delivered Message: Consumer sequence: 1 Stream sequence: 1 + Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0 + Pending Messages: 0 + Redelivered Messages: 0 +``` + +More details about the `State` section will be shown later when discussing the ack models in depth. + +#### Consuming Pull-Based Consumers + +Pull-based Consumers require you to specifically ask for messages and ack them, typically you would do this with the client library `Request()` feature, but the `jsm` utility has a helper: + +First we ensure we have a message: + +``` +$ nats pub ORDERS.processed "order 1" +$ nats pub ORDERS.processed "order 2" +$ nats pub ORDERS.processed "order 3" +``` + +We can now read them using `nats`: + +``` +$ nats con next ORDERS DISPATCH +--- received on ORDERS.processed +order 1 + +Acknowledged message + +$ nats con next ORDERS DISPATCH +--- received on ORDERS.processed +order 2 + +Acknowledged message +``` + +You can prevent ACKs by supplying `--no-ack`. + +To do this from code you'd send a `Request()` to `$JS.NEXT.ORDERS.DISPATCH`: + +``` +$ nats req '$JS.NEXT.ORDERS.DISPATCH' '' +Published [$JS.NEXT.ORDERS.DISPATCH] : '' +Received [ORDERS.processed] : 'order 3' +``` + +Here `nats req` cannot ack, but in your code you'd respond to the received message with a nil payload as an Ack to JetStream. + +#### Consuming Push-Based Consumers + +Push-based Consumers will publish messages to a subject and anyone who subscribes to the subject will get them, they support different Acknowledgement models covered later, but here on the `MONITOR` Consumer we have no Acknowledgement. + +``` +$ nats con info ORDERS MONITOR +... + Delivery Subject: monitor.ORDERS +... +``` + +The Consumer is publishing to that subject, so lets listen there: + +``` +$ nats sub monitor.ORDERS +Listening on [monitor.ORDERS] +[#3] Received on [ORDERS.processed]: 'order 3' +[#4] Received on [ORDERS.processed]: 'order 4' +``` + +Note the subject here of the received message is reported as `ORDERS.processed` this helps you distinguish what you're seeing in a Stream covering a wildcard, or multiple subject, subject space. + +This Consumer needs no ack, so any new message into the ORDERS system will show up here in real time. \ No newline at end of file diff --git a/jetstream/administration/streams.md b/jetstream/administration/streams.md new file mode 100644 index 0000000..72e8689 --- /dev/null +++ b/jetstream/administration/streams.md @@ -0,0 +1,235 @@ +### Streams + +The first step is to set up storage for our `ORDERS` related messages, these arrive on a wildcard of subjects all flowing into the same Stream and they are kept for 1 year. + +#### Creating + +```nohighlight +$ nats str add ORDERS +? Subjects to consume ORDERS.* +? Storage backend file +? Retention Policy Limits +? Discard Policy Old +? Message count limit -1 +? Message size limit -1 +? Maximum message age limit 1y +? Maximum individual message size [? for help] (-1) -1 +Stream ORDERS was created + +Information for Stream ORDERS + +Configuration: + + Subjects: ORDERS.* + Acknowledgements: true + Retention: File - Limits + Replicas: 1 + Maximum Messages: -1 + Maximum Bytes: -1 + Maximum Age: 8760h0m0s + Maximum Message Size: -1 + Maximum Consumers: -1 + +Statistics: + + Messages: 0 + Bytes: 0 B + FirstSeq: 0 + LastSeq: 0 + Active Consumers: 0 +``` + +You can get prompted interactively for missing information as above, or do it all on one command. Pressing `?` in the CLI will help you map prompts to CLI options: + +``` +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old +``` + +Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: + +``` +$ nats str add ORDERS --config orders.json +``` + +#### Listing + +We can confirm our Stream was created: + +```nohighlight +$ nats str ls +Streams: + + ORDERS +``` + +#### Querying + +Information about the configuration of the Stream can be seen, and if you did not specify the Stream like below, it will prompt you based on all known ones: + +```nohighlight +$ nats str info ORDERS +Information for Stream ORDERS + +Configuration: + + Subjects: ORDERS.* + No Acknowledgements: false + Retention: File - Limits + Replicas: 1 + Maximum Messages: -1 + Maximum Bytes: -1 + Maximum Age: 8760h0m0s + Maximum Consumers: -1 + +Statistics: + + Messages: 0 + Bytes: 0 B + FirstSeq: 0 + LastSeq: 0 + Active Consumers: 0 +``` + +Most commands that show data as above support `-j` to show the results as JSON: + +```nohighlight +$ nats str info ORDERS -j +{ + "config": { + "name": "ORDERS", + "subjects": [ + "ORDERS.*" + ], + "retention": "limits", + "max_consumers": -1, + "max_msgs": -1, + "max_bytes": -1, + "max_age": 31536000000000000, + "storage": "file", + "num_replicas": 1 + }, + "stats": { + "messages": 0, + "bytes": 0, + "first_seq": 0, + "last_seq": 0, + "consumer_count": 0 + } +} +``` + +This is the general pattern for the entire `nats` utility as it relates to JetStream - prompting for needed information but every action can be run non-interactively making it usable as a cli api. All information output like seen above can be turned into JSON using `-j`. + +#### Copying + +A stream can be copied into another, which also allows the configuration of the new one to be adjusted via CLI flags: + +```nohighlight +$ nats str cp ORDERS ARCHIVE --subjects "ORDERS_ARCVHIVE.*" --max-age 2y +Stream ORDERS was created + +Information for Stream ARCHIVE + +Configuration: + + Subjects: ORDERS_ARCVHIVE.* +... + Maximum Age: 17520h0m0s +... +``` + +#### Editing + +A stream configuration can be edited, which allows the configuration to be adjusted via CLI flags. Here I have a incorrectly created ORDERS stream that I fix: + +```nohighlight +$ nats str info ORDERS -j | jq .config.subjects +[ + "ORDERS.new" +] + +$ nats str edit ORDERS --subjects "ORDERS.*" +Stream ORDERS was updated + +Information for Stream ORDERS + +Configuration: + + Subjects: ORDERS.* +.... +``` + +Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: + +``` +$ nats str edit ORDERS --config orders.json +``` + +#### Publishing Into a Stream + +Now let's add in some messages to our Stream. You can use `nats pub` to add messages, pass the `--wait` flag to see the publish ack being returned. + +You can publish without waiting for acknowledgement: + +```nohighlight +$ nats pub ORDERS.scratch hello +Published [sub1] : 'hello' +``` + +But if you want to be sure your messages got to JetStream and were persisted you can make a request: + +```nohighlight +$ nats req ORDERS.scratch hello +13:45:03 Sending request on [ORDERS.scratch] +13:45:03 Received on [_INBOX.M8drJkd8O5otORAo0sMNkg.scHnSafY]: '+OK' +``` + +Keep checking the status of the Stream while doing this and you'll see it's stored messages increase. + +```nohighlight +$ nats str info ORDERS +Information for Stream ORDERS +... +Statistics: + + Messages: 3 + Bytes: 147 B + FirstSeq: 1 + LastSeq: 3 + Active Consumers: 0 +``` + +After putting some throw away data into the Stream, we can purge all the data out - while keeping the Stream active: + +#### Deleting All Data + +To delete all data in a stream use `purge`: + +```nohighlight +$ nats str purge ORDERS -f +... +Statistics: + + Messages: 0 + Bytes: 0 B + FirstSeq: 1,000,001 + LastSeq: 1,000,000 + Active Consumers: 0 +``` + +#### Deleting A Message + +A single message can be securely removed from the stream: + +```nohighlight +$ nats str rmm ORDERS 1 -f +``` + +#### Deleting Sets + +Finally for demonstration purposes, you can also delete the whole Stream and recreate it so then we're ready for creating the Consumers: + +``` +$ nats str rm ORDERS -f +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 +``` diff --git a/jetstream/concepts/concepts.md b/jetstream/concepts/concepts.md new file mode 100644 index 0000000..00785d4 --- /dev/null +++ b/jetstream/concepts/concepts.md @@ -0,0 +1,19 @@ +## Concepts + +In JetStream the configuration for storing messages is defined separately from how they are consumed. Storage is defined in a *Stream* and consuming messages is defined by multiple *Consumers*. + +We'll discuss these 2 subjects in the context of this architecture. + +![Orders](../.gitbook/assets/images/streams-and-consumers-75p.png) + +While this is an incomplete architecture it does show a number of key points: + + * Many related subjects are stored in a Stream + * Consumers can have different modes of operation and receive just subsets of the messages + * Multiple Acknowledgement modes are supported + +A new order arrives on `ORDERS.received`, gets sent to the `NEW` Consumer who, on success, will create a new message on `ORDERS.processed`. The `ORDERS.processed` message again enters the Stream where a `DISPATCH` Consumer receives it and once processed it will create an `ORDERS.completed` message which will again enter the Stream. These operations are all `pull` based meaning they are work queues and can scale horizontally. All require acknowledged delivery ensuring no order is missed. + +All messages are delivered to a `MONITOR` Consumer without any acknowledgement and using Pub/Sub semantics - they are pushed to the monitor. + +As messages are acknowledged to the `NEW` and `DISPATCH` Consumers, a percentage of them are Sampled and messages indicating redelivery counts, ack delays and more, are delivered to the monitoring system. \ No newline at end of file diff --git a/jetstream/concepts/configuration.md b/jetstream/concepts/configuration.md new file mode 100644 index 0000000..aa4824c --- /dev/null +++ b/jetstream/concepts/configuration.md @@ -0,0 +1,10 @@ +### Configuration + +The rest of this document introduces the `nats` utility, but for completeness and reference this is how you'd create the ORDERS scenario. We'll configure a 1 year retention for order related messages: + +```bash +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard=old +$ nats con add ORDERS NEW --filter ORDERS.received --ack explicit --pull --deliver all --max-deliver=-1 --sample 100 +$ nats con add ORDERS DISPATCH --filter ORDERS.processed --ack explicit --pull --deliver all --max-deliver=-1 --sample 100 +$ nats con add ORDERS MONITOR --filter '' --ack none --target monitor.ORDERS --deliver last --replay instant +``` \ No newline at end of file diff --git a/jetstream/concepts/consumers.md b/jetstream/concepts/consumers.md new file mode 100644 index 0000000..7ce7fde --- /dev/null +++ b/jetstream/concepts/consumers.md @@ -0,0 +1,33 @@ +### Consumers + +Each Consumer, or related group of Consumers, of a Stream will need an Consumer defined. It's ok to define thousands of these pointing at the same Stream. + +Consumers can either be `push` based where JetStream will deliver the messages as fast as possible to a subject of your choice or `pull` based for typical work queue like behavior. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging. + +In the orders example above we have 3 Consumers. The first two select a subset of the messages from the Stream by specifying a specific subject like `ORDERS.processed`. The Stream consumes `ORDERS.*` and this allows you to receive just what you need. The final Consumer receives all messages in a `push` fashion. + +Consumers track their progress, they know what messages were delivered, acknowledged, etc., and will redeliver messages they sent that were not acknowledged. When first created, the Consumer has to know what message to send as the first one. You can configure either a specific message in the set (`StreamSeq`), specific time (`StartTime`), all (`DeliverAll`) or last (`DeliverLast`). This is the starting point and from there, they all behave the same - delivering all of the following messages with optional Acknowledgement. + +Acknowledgements default to `AckExplicit` - the only supported mode for pull-based Consumers - meaning every message requires a distinct acknowledgement. But for push-based Consumers, you can set `AckNone` that does not require any acknowledgement, or `AckAll` which quite interestingly allows you to acknowledge a specific message, like message `100`, which will also acknowledge messages `1` through `99`. The `AckAll` mode can be a great performance boost. + +Some messages may cause your applications to crash and cause a never ending loop forever poisoning your system. The `MaxDeliver` setting allow you to set a upper bound to how many times a message may be delivered. + +To assist with creating monitoring applications, one can set a `SampleFrequency` which is a percentage of messages for which the system should sample and create events. These events will include delivery counts and ack waits. + +When defining Consumers the items below make up the entire configuration of the Consumer: + +|Item|Description| +|----|-----------| +|AckPolicy|How messages should be acknowledged, `AckNone`, `AckAll` or `AckExplicit`| +|AckWait|How long to allow messages to remain un-acknowledged before attempting redelivery| +|DeliverPolicy|The initial starting mode of the consumer, `DeliverAll`, `DeliverLast`, `DeliverNew`, `DeliverByStartSequence` or `DeliverByStartTime`| +|DeliverySubject|The subject to deliver observed messages, when not set, a pull-based Consumer is created| +|Durable|The name of the Consumer| +|FilterSubject|When consuming from a Stream with many subjects, or wildcards, select only a specific incoming subjects, supports wildcards| +|MaxDeliver|Maximum amount times a specific message will be delivered. Use this to avoid poison pills crashing all your services forever| +|OptStartSeq|When first consuming messages from the Stream start at this particular message in the set| +|ReplayPolicy|How messages are sent `ReplayInstant` or `ReplayOriginal`| +|SampleFrequency|What percentage of acknowledgements should be samples for observability, 0-100| +|OptStartTime|When first consuming messages from the Stream start with messages on or after this time| +|RateLimit|The rate of message delivery in bits per second| +|MaxAckPending|The maximum number of messages without acknowledgement that can be outstanding, once this limit is reached message delivery will be suspended| diff --git a/jetstream/concepts/streams.md b/jetstream/concepts/streams.md new file mode 100644 index 0000000..861ec64 --- /dev/null +++ b/jetstream/concepts/streams.md @@ -0,0 +1,31 @@ +### Streams + +Streams define how messages are stored and retention duration. Streams consume normal NATS subjects, any message found on those subjects will be delivered to the defined storage system. You can do a normal publish to the subject for unacknowledged delivery, else if you send a Request to the subject the JetStream server will reply with an acknowledgement that it was stored. + +As of January 2020, in the tech preview we have `file` and `memory` based storage systems, we do not yet support clustering. + +In the diagram above we show the concept of storing all `ORDERS.*` in the Stream even though there are many types of order related messages. We'll show how you can selectively consume subsets of messages later. Relatively speaking the Stream is the most resource consuming component so being able to combine related data in this manner is important to consider. + +Streams can consume many subjects. Here we have `ORDERS.*` but we could also consume `SHIPPING.state` into the same Stream should that make sense (not shown here). + +Streams support various retention policies - they can be kept based on limits like max count, size or age but also more novel methods like keeping them as long as any Consumers have them unacknowledged, or work queue like behavior where a message is removed after first ack. + +Streams support deduplication using a `Msg-Id` header and a sliding window within which to track duplicate messages. See the [Message Deduplication](#message-deduplication) section. + +When defining Streams the items below make up the entire configuration of the set. + +|Item|Description| +|----|-----------| +|MaxAge|Maximum age of any message in the stream, expressed in microseconds| +|MaxBytes|How big the Stream may be, when the combined stream size exceeds this old messages are removed| +|MaxMsgSize|The largest message that will be accepted by the Stream| +|MaxMsgs|How many messages may be in a Stream, oldest messages will be removed if the Stream exceeds this size| +|MaxConsumers|How many Consumers can be defined for a given Stream, `-1` for unlimited| +|Name|A name for the Stream that may not have spaces, tabs or `.`| +|NoAck|Disables acknowledging messages that are received by the Stream| +|Replicas|How many replicas to keep for each message (not implemented as of January 2020)| +|Retention|How message retention is considered, `LimitsPolicy` (default), `InterestPolicy` or `WorkQueuePolicy`| +|Discard|When a Stream reached it's limits either, `DiscardNew` refuses new messages while `DiscardOld` (default) deletes old messages| +|Storage|The type of storage backend, `file` and `memory` as of January 2020| +|Subjects|A list of subjects to consume, supports wildcards| +|Duplicates|The window within which to track duplicate messages| diff --git a/jetstream/configuration_mgmt/configuration_mgmt.md b/jetstream/configuration_mgmt/configuration_mgmt.md new file mode 100644 index 0000000..f654c60 --- /dev/null +++ b/jetstream/configuration_mgmt/configuration_mgmt.md @@ -0,0 +1,105 @@ +## Configuration Management + +In many cases managing the configuration in your application code is the best model, many teams though wish to pre-create Streams and Consumers. + +We support a number of tools to assist with this: + + * `nats` CLI with configuration files + * [Terraform](https://www.terraform.io/) + * [GitHub Actions](https://github.com/features/actions) + * [Kubernetes JetStream Controller](https://github.com/nats-io/nack#jetstream-controller) + +### nats Admin CLI + +The `nats` CLI can be used to manage Streams and Consumers easily using it's `--config` flag, for example: + +### Add a new Stream + +This creates a new Stream based on `orders.json`. The `orders.json` file can be extracted from an existing stream using `nats stream info ORDERS -j | jq .config` + +``` +$ nats str add ORDERS --config orders.json +``` + +### Edit an existing Stream + +This edits an existing stream ensuring it complies with the configuration in `orders.json` +``` +$ nats str edit ORDERS --config orders.json +``` + +### Add a New Consumer + +This creates a new Consumer based on `orders_new.json`. The `orders_new.json` file can be extracted from an existing stream using `nats con info ORDERS NEW -j | jq .config` + +``` +$ nats con add ORDERS NEW --config orders_new.json +``` + +### Terraform + +Terraform is a Cloud configuration tool from Hashicorp found at [terraform.io](https://www.terraform.io/), we maintain a Provider for Terraform called [terraform-provider-jetstream](https://github.com/nats-io/terraform-provider-jetstream/) that can maintain JetStream using Terraform. + +#### Setup + +Our provider is not hosted by Hashicorp so installation is a bit more complex than typical. Browse to the [Release Page](https://github.com/nats-io/terraform-provider-jetstream/releases) and download the release for your platform and extract it into your Terraform plugins directory. + +``` +$ unzip -l terraform-provider-jetstream_0.0.2_darwin_amd64.zip +Archive: terraform-provider-jetstream_0.0.2_darwin_amd64.zip + Length Date Time Name +--------- ---------- ----- ---- + 11357 03-09-2020 10:48 LICENSE + 1830 03-09-2020 12:53 README.md + 24574336 03-09-2020 12:54 terraform-provider-jetstream_v0.0.2 +``` + +Place the `terraform-provider-jetstream_v0.0.2` file in `~/.terraform.d/plugins/terraform-provider-jetstream_v0.0.2` + +In your project you can configure the Provider like this: + +```terraform +provider "jetstream" { + servers = "connect.ngs.global" + credentials = "ngs_jetstream_admin.creds" +} +``` + +And start using it, here's an example that create the `ORDERS` example. Review the [Project README](https://github.com/nats-io/terraform-provider-jetstream#readme) for full details. + +```terraform +resource "jetstream_stream" "ORDERS" { + name = "ORDERS" + subjects = ["ORDERS.*"] + storage = "file" + max_age = 60 * 60 * 24 * 365 +} + +resource "jetstream_consumer" "ORDERS_NEW" { + stream_id = jetstream_stream.ORDERS.id + durable_name = "NEW" + deliver_all = true + filter_subject = "ORDERS.received" + sample_freq = 100 +} + +resource "jetstream_consumer" "ORDERS_DISPATCH" { + stream_id = jetstream_stream.ORDERS.id + durable_name = "DISPATCH" + deliver_all = true + filter_subject = "ORDERS.processed" + sample_freq = 100 +} + +resource "jetstream_consumer" "ORDERS_MONITOR" { + stream_id = jetstream_stream.ORDERS.id + durable_name = "MONITOR" + deliver_last = true + ack_policy = "none" + delivery_subject = "monitor.ORDERS" +} + +output "ORDERS_SUBJECTS" { + value = jetstream_stream.ORDERS.subjects +} +``` diff --git a/jetstream/configuration_mgmt/github_actions.md b/jetstream/configuration_mgmt/github_actions.md new file mode 100644 index 0000000..81975bd --- /dev/null +++ b/jetstream/configuration_mgmt/github_actions.md @@ -0,0 +1,55 @@ +### GitHub Actions + +We have a pack of GitHub Actions that let you manage an already running JetStream Server, useful for managing releases or standing up test infrastructure. + +Full details and examples are in the [jetstream-gh-actions](https://github.com/nats-io/jetstream-gh-action) repository, here's an example. + +```yaml +on: push +name: orders +jobs: + + # First we delete the ORDERS stream and consumer if they already exist + clean_orders: + runs-on: ubuntu-latest + steps: + - name: orders_stream + uses: nats-io/jetstream-gh-action/delete/stream@master + with: + missing_ok: 1 + stream: ORDERS + server: js.example.net + + # Now we create the Stream and Consumers using the same configuration files the + # nats CLI utility would use as shown above + create_orders: + runs-on: ubuntu-latest + needs: clean_orders + steps: + - uses: actions/checkout@master + - name: orders_stream + uses: nats-io/jetstream-gh-action/create/stream@master + with: + config: ORDERS.json + server: js.example.net + - name: orders_new_consumer + uses: nats-io/jetstream-gh-action/create/consumer@master + with: + config: ORDERS_NEW.json + stream: ORDERS + server: js.example.net + + # We publish a message to a specific Subject, perhaps some consumer is + # waiting there for it to kick off tests + publish_message: + runs-on: ubuntu-latest + needs: create_orders + steps: + - uses: actions/checkout@master + - name: orders_new_consumer + uses: nats-io/jetstream-gh-action@master + with: + subject: ORDERS.deployment + message: Published new deployment via "${{ github.event_name }}" in "${{ github.repository }}" + server: js.example.net +``` diff --git a/jetstream/configuration_mgmt/kubernetes_controller.md b/jetstream/configuration_mgmt/kubernetes_controller.md new file mode 100644 index 0000000..9501fe6 --- /dev/null +++ b/jetstream/configuration_mgmt/kubernetes_controller.md @@ -0,0 +1,57 @@ +### Kubernetes JetStream Controller + +The JetStream controllers allows you to manage NATS JetStream Streams and Consumers via K8S CRDs. You can find more info on how to deploy and usage [here](https://github.com/nats-io/nack#getting-started). Below you can find an example on how to create a stream and a couple of consumers: + +```yaml +--- +apiVersion: jetstream.nats.io/v1beta1 +kind: Stream +metadata: + name: mystream +spec: + name: mystream + subjects: ["orders.*"] + storage: memory + maxAge: 1h +--- +apiVersion: jetstream.nats.io/v1beta1 +kind: Consumer +metadata: + name: my-push-consumer +spec: + streamName: mystream + durableName: my-push-consumer + deliverSubject: my-push-consumer.orders + deliverPolicy: last + ackPolicy: none + replayPolicy: instant +--- +apiVersion: jetstream.nats.io/v1beta1 +kind: Consumer +metadata: + name: my-pull-consumer +spec: + streamName: mystream + durableName: my-pull-consumer + deliverPolicy: all + filterSubject: orders.received + maxDeliver: 20 + ackPolicy: explicit +``` + +Once the CRDs are installed you can use `kubectl` to manage the streams and consumers as follows: + +```sh +$ kubectl get streams +NAME STATE STREAM NAME SUBJECTS +mystream Created mystream [orders.*] + +$ kubectl get consumers +NAME STATE STREAM CONSUMER ACK POLICY +my-pull-consumer Created mystream my-pull-consumer explicit +my-push-consumer Created mystream my-push-consumer none + +# If you end up in an Errored state, run kubectl describe for more info. +# kubectl describe streams mystream +# kubectl describe consumers my-pull-consumer +``` diff --git a/jetstream/disaster_recovery/disaster_recovery.md b/jetstream/disaster_recovery/disaster_recovery.md new file mode 100644 index 0000000..92b851b --- /dev/null +++ b/jetstream/disaster_recovery/disaster_recovery.md @@ -0,0 +1,85 @@ +## Disaster Recovery + +Disaster Recovery of the JetStream system is a topic we are still exploring and fleshing out and that will be impacted by the clustering work. For example replication will extend the options available to you. + +Today we have a few approaches to consider: + + * `nats` CLI + Configuration Backups + Data Snapshots + * Configuration Management + Data Snapshots + +### Data Backup + +In all scenarios you can perform data snapshots and restores over the NATS protocol. This is good if you do not manage the NATS servers hosting your data, and you wish to do a backup of your data. + +The backup includes: + + * Stream configuration and state + * Stream Consumer configuration and state + * All data including metadata like timestamps and headers + +```nohighlight +$ nats stream backup ORDERS /data/js-backup/ORDERS.tgz +Starting backup of Stream "ORDERS" with 13 data blocks + +2.4 MiB/s [====================================================================] 100% + +Received 13 MiB bytes of compressed data in 3368 chunks for stream "ORDERS" in 1.223428188s, 813 MiB uncompressed +``` + +During the backup the Stream is in a state where it's configuration cannot change and no data will be expired from it based on Limits or Retention Policies. + +Progress using the terminal bar can be disabled using `--no-progress`, it will then issue log lines instead. + +### Restoring Data + +The backup made above can be restored into another server - but into the same Stream name. + +```nohighlight +$ nats str restore ORDERS /data/js-backup/ORDERS.tgz +Starting restore of Stream "ORDERS" from file "/data/js-backup/ORDERS.tgz" + +13 MiB/s [====================================================================] 100% + +Restored stream "ORDERS" in 937.071149ms + +Information for Stream ORDERS + +Configuration: + + Subjects: ORDERS.> +... +``` + +The `/data/js-backup/ORDERS.tgz` file can also be extracted into the data dir of a stopped NATS Server. + +Progress using the terminal bar can be disabled using `--no-progress`, it will then issue log lines instead. + +### Interactive CLI + +In environments where the `nats` CLI is used interactively to configure the server you do not have a desired state to recreate the server from. This is not the ideal way to administer the server, we recommend Configuration Management, but many will use this approach. + +Here you can back up the configuration into a directory from where you can recover the configuration later. The data for File backed stores can also be backed up. + +```nohighlight +$ nats backup /data/js-backup +15:56:11 Creating JetStream backup into /data/js-backup +15:56:11 Stream ORDERS to /data/js-backup/stream_ORDERS.json +15:56:11 Consumer ORDERS > NEW to /data/js-backup/stream_ORDERS_consumer_NEW.json +15:56:11 Configuration backup complete +``` + +This backs up Stream, Consumer and Stream Template configuration. + +During the same process the data can also be backed up by passing `--data`, this will create files like `/data/js-backup/stream_ORDERS.tgz`. + +Later the data can be restored, for Streams we support editing the Stream configuration in place to match what was in the backup. + +``` +$ nats restore /tmp/backup --update-streams +15:57:42 Reading file /tmp/backup/stream_ORDERS.json +15:57:42 Reading file /tmp/backup/stream_ORDERS_consumer_NEW.json +15:57:42 Updating Stream ORDERS configuration +15:57:42 Restoring Consumer ORDERS > NEW +``` + +The `nats restore` tool does not support restoring data, the same process using `nats stream restore`, as outlined earlier, can be used which will also restore Stream and Consumer configurations and state. \ No newline at end of file diff --git a/jetstream/getting_started/getting_started.md b/jetstream/getting_started/getting_started.md new file mode 100644 index 0000000..0f95734 --- /dev/null +++ b/jetstream/getting_started/getting_started.md @@ -0,0 +1,3 @@ +## Getting Started + +This tech preview is limited to a single server and defaults to the global account. JetStream is NATS 2.0 aware and is scoped to accounts from a resource limit perspective. This is not the same as an individual server's resources, but may feel that way starting out. Don't worry, clustering is coming next but we wanted to get input early from the community. \ No newline at end of file diff --git a/jetstream/getting_started/using_docker.md b/jetstream/getting_started/using_docker.md new file mode 100644 index 0000000..d8923ea --- /dev/null +++ b/jetstream/getting_started/using_docker.md @@ -0,0 +1,50 @@ +### Using Docker + +The `synadia/jsm:latest` docker image contains both the JetStream enabled NATS Server and the `nats` utility this guide covers. + +In one window start JetStream: + +``` +$ docker run -ti -p 4222:4222 --name jetstream synadia/jsm:latest server +[1] 2020/01/20 12:44:11.752465 [INF] Starting nats-server version 2.2.0-beta +[1] 2020/01/20 12:44:11.752694 [INF] Git commit [19dc3eb] +[1] 2020/01/20 12:44:11.752875 [INF] Starting JetStream +[1] 2020/01/20 12:44:11.753692 [INF] ----------- JETSTREAM (Beta) ----------- +[1] 2020/01/20 12:44:11.753794 [INF] Max Memory: 1.46 GB +[1] 2020/01/20 12:44:11.753822 [INF] Max Storage: 1.00 TB +[1] 2020/01/20 12:44:11.753860 [INF] Store Directory: "/tmp/jetstream" +[1] 2020/01/20 12:44:11.753893 [INF] ---------------------------------------- +[1] 2020/01/20 12:44:11.753988 [INF] JetStream state for account "$G" recovered +[1] 2020/01/20 12:44:11.754148 [INF] Listening for client connections on 0.0.0.0:4222 +[1] 2020/01/20 12:44:11.754279 [INF] Server id is NDYX5IMGF2YLX6RC4WLZA7T3JGHPZR2RNCCIFUQBT6C4TP27Z6ZIC73V +[1] 2020/01/20 12:44:11.754308 [INF] Server is ready +``` + +And in another log into the utilities: + +``` +$ docker run -ti --link jetstream synadia/jsm:latest +``` + +This shell has the `nats` utility and all other NATS cli tools used in the rest of this guide. + +Now skip to the `Administer JetStream` section. + +### Using Docker with NGS + +You can join a JetStream instance to your [NGS](https://synadia.com/ngs/pricing) account, first we need a credential for testing JetStream: + +``` +$ nsc add user -a YourAccount --name leafnode --expiry 1M +``` + +You'll get a credential file somewhere like `~/.nkeys/creds/synadia/YourAccount/leafnode.creds`, mount this file into the docker container for JetStream using `-v ~/.nkeys/creds/synadia/YourAccount/leafnode.creds:/leafnode.creds`. + +``` +$ docker run -ti -v ~/.nkeys/creds/synadia/YourAccount/leafnode.creds:/leafnode.creds --name jetstream synadia/jsm:latest server +[1] 2020/01/20 12:44:11.752465 [INF] Starting nats-server version 2.2.0-beta +... +[1] 2020/01/20 12:55:01.849033 [INF] Connected leafnode to "connect.ngs.global" +``` + +Your JSM shell will still connect locally, other connections in your NGS account can use JetStream at this point. \ No newline at end of file diff --git a/jetstream/getting_started/using_source.md b/jetstream/getting_started/using_source.md new file mode 100644 index 0000000..4f6bee2 --- /dev/null +++ b/jetstream/getting_started/using_source.md @@ -0,0 +1,65 @@ +### Using Source + +You will also want to have installed from the nats.go repo the examples/tools such as nats-pub, nats-sub, nats-req and possibly nats-bench. One of the design goals of JetStream was to be native to core NATS, so even though we will most certainly add in syntactic sugar to clients to make them more appealing, for this tech preview we will be using plain old NATS. + +You will need a copy of the nats-server source locally and will need to be in the jetstream branch. + +``` +$ git clone https://github.com/nats-io/nats-server.git +$ cd nats-server +$ git checkout master +$ go build +$ ls -l nats-server +``` + +Starting the server you can use the `-js` flag. This will setup the server to reasonably use memory and disk. This is a sample run on my machine. JetStream will default to 1TB of disk and 75% of available memory for now. + +``` +$ ./nats-server -js + +[16928] 2019/12/04 19:16:29.596968 [INF] Starting nats-server version 2.2.0-beta +[16928] 2019/12/04 19:16:29.597056 [INF] Git commit [not set] +[16928] 2019/12/04 19:16:29.597072 [INF] Starting JetStream +[16928] 2019/12/04 19:16:29.597444 [INF] ----------- JETSTREAM (Beta) ----------- +[16928] 2019/12/04 19:16:29.597451 [INF] Max Memory: 96.00 GB +[16928] 2019/12/04 19:16:29.597454 [INF] Max Storage: 1.00 TB +[16928] 2019/12/04 19:16:29.597461 [INF] Store Directory: "/var/folders/m0/k03vs55n2b54kdg7jm66g27h0000gn/T/jetstream" +[16928] 2019/12/04 19:16:29.597469 [INF] ---------------------------------------- +[16928] 2019/12/04 19:16:29.597732 [INF] Listening for client connections on 0.0.0.0:4222 +[16928] 2019/12/04 19:16:29.597738 [INF] Server id is NAJ5GKP5OBVISP5MW3BFAD447LMTIOAHFEWMH2XYWLL5STVGN3MJHTXQ +[16928] 2019/12/04 19:16:29.597742 [INF] Server is ready +``` + +You can override the storage directory if you want. + +``` +$ ./nats-server -js -sd /tmp/test + +[16943] 2019/12/04 19:20:00.874148 [INF] Starting nats-server version 2.2.0-beta +[16943] 2019/12/04 19:20:00.874247 [INF] Git commit [not set] +[16943] 2019/12/04 19:20:00.874273 [INF] Starting JetStream +[16943] 2019/12/04 19:20:00.874605 [INF] ----------- JETSTREAM (Beta) ----------- +[16943] 2019/12/04 19:20:00.874613 [INF] Max Memory: 96.00 GB +[16943] 2019/12/04 19:20:00.874615 [INF] Max Storage: 1.00 TB +[16943] 2019/12/04 19:20:00.874620 [INF] Store Directory: "/tmp/test/jetstream" +[16943] 2019/12/04 19:20:00.874625 [INF] ---------------------------------------- +[16943] 2019/12/04 19:20:00.874868 [INF] Listening for client connections on 0.0.0.0:4222 +[16943] 2019/12/04 19:20:00.874874 [INF] Server id is NCR6KDDGWUU2FXO23WAXFY66VQE6JNWVMA24ALF2MO5GKAYFIMQULKUO +[16943] 2019/12/04 19:20:00.874877 [INF] Server is ready +``` + +These options can also be set in your configuration file: + +``` +// enables jetstream, an empty block will enable and use defaults +jetstream { + // jetstream data will be in /data/nats-server/jetstream + store_dir: "/data/nats-server" + + // 1GB + max_memory_store: 1073741824 + + // 10GB + max_file_store: 10737418240 +} +``` \ No newline at end of file diff --git a/jetstream/model_deep_dive/model_deep_dive.md b/jetstream/model_deep_dive/model_deep_dive.md new file mode 100644 index 0000000..aa87c20 --- /dev/null +++ b/jetstream/model_deep_dive/model_deep_dive.md @@ -0,0 +1,481 @@ +## Model Deep Dive + +The Orders example touched on a lot of features, but some like different Ack models and message limits, need a bit more detail. This section will expand on the above and fill in some blanks. + +### Stream Limits, Retention Modes and Discard Policy + +Streams store data on disk, but we cannot store all data forever so we need ways to control their size automatically. + +There are 3 features that come into play when Streams decide how long they store data. + +The `Retention Policy` describes based on what criteria a set will evict messages from its storage: + +|Retention Policy|Description| +|----------------|-----------| +|`LimitsPolicy` |Limits are set for how many messages, how big the storage and how old messages may be| +|`WorkQueuePolicy`|Messages are kept until they were consumed by any one single observer and then removed| +|`InterestPolicy`|Messages are kept as long as there are Consumers active for them| + +In all Retention Policies the basic limits apply as upper bounds, these are `MaxMsgs` for how many messages are kept in total, `MaxBytes` for how big the set can be in total and `MaxAge` for what is the oldest message that will be kept. These are the only limits in play with `LimitsPolicy` retention. + +One can then define additional ways a message may be removed from the Stream earlier than these limits. In `WorkQueuePolicy` the messages will be removed as soon as any Consumer received an Acknowledgement. In `InterestPolicy` messages will be removed as soon as there are no more Consumers. + +In both `WorkQueuePolicy` and `InterestPolicy` the age, size and count limits will still apply as upper bounds. + +A final control is the Maximum Size any single message may have. NATS have it's own limit for maximum size (1 MiB by default), but you can say a Stream will only accept messages up to 1024 bytes using `MaxMsgSize`. + +The `Discard Policy` sets how messages are discard when limits set by `LimitsPolicy` are reached. The `DiscardOld` option removes old messages making space for new, while `DiscardNew` refuses any new messages. + +The `WorkQueuePolicy` mode is a specialized mode where a message, once consumed and acknowledged, is discarded from the Stream. In this mode there are a few limits on consumers. Inherently it's about 1 message to one consumer, this means you cannot have overlapping consumers defined on the Stream - needs unique filter subjects. + +### Message Deduplication + +JetStream support idempotent message writes by ignoring duplicate messages as indicated by the `Msg-Id` header. + +```nohighlight +% nats req -H Msg-Id:1 ORDERS.new hello1 +% nats req -H Msg-Id:1 ORDERS.new hello2 +% nats req -H Msg-Id:1 ORDERS.new hello3 +% nats req -H Msg-Id:1 ORDERS.new hello4 +``` + +Here we set a `Msg-Id:1` header which tells JetStream to ensure we do not have duplicates of this message - we only consult the message ID not the body. + +```nohighlight +$ nats str info ORDERS +.... +State: + + Messages: 1 + Bytes: 67 B +``` + +The default window to track duplicates in is 2 minutes, this can be set on the command line using `--dupe-window` when creating a stream, though we would caution against large windows. + +### Acknowledgement Models + +Streams support acknowledging receiving a message, if you send a `Request()` to a subject covered by the configuration of the Stream the service will reply to you once it stored the message. If you just publish, it will not. A Stream can be set to disable Acknowledgements by setting `NoAck` to `true` in it's configuration. + +Consumers have 3 acknowledgement modes: + +|Mode|Description| +|----|-----------| +|`AckExplicit`|This requires every message to be specifically acknowledged, it's the only supported option for pull-based Consumers| +|`AckAll`|In this mode if you acknowledge message `100` it will also acknowledge message `1`-`99`, this is good for processing batches and to reduce ack overhead| +|`AckNone`|No acknowledgements are supported| + +To understand how Consumers track messages we will start with a clean `ORDERS` Stream and `DISPATCH` Consumer. + +``` +$ nats str info ORDERS +... +Statistics: + + Messages: 0 + Bytes: 0 B + FirstSeq: 0 + LastSeq: 0 + Active Consumers: 1 +``` + +The Set is entirely empty + +``` +$ nats con info ORDERS DISPATCH +... +State: + + Last Delivered Message: Consumer sequence: 1 Stream sequence: 1 + Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0 + Pending Messages: 0 + Redelivered Messages: 0 +``` + +The Consumer has no messages oustanding and has never had any (Consumer sequence is 1). + +We publish one message to the Stream and see that the Stream received it: + +``` +$ nats pub ORDERS.processed "order 4" +Published 7 bytes to ORDERS.processed +$ nats str info ORDERS +... +Statistics: + + Messages: 1 + Bytes: 53 B + FirstSeq: 1 + LastSeq: 1 + Active Consumers: 1 +``` + +As the Consumer is pull-based, we can fetch the message, ack it, and check the Consumer state: + +``` +$ nats con next ORDERS DISPATCH +--- received on ORDERS.processed +order 4 + +Acknowledged message + +$ nats con info ORDERS DISPATCH +... +State: + + Last Delivered Message: Consumer sequence: 2 Stream sequence: 2 + Acknowledgment floor: Consumer sequence: 1 Stream sequence: 1 + Pending Messages: 0 + Redelivered Messages: 0 +``` + +The message got delivered and acknowledged - `Acknowledgement floor` is `1` and `1`, the sequence of the Consumer is `2` which means its had only the one message through and got acked. Since it was acked, nothing is pending or redelivering. + +We'll publish another message, fetch it but not Ack it this time and see the status: + +``` +$ nats pub ORDERS.processed "order 5" +Published 7 bytes to ORDERS.processed + +$ nats con next ORDERS DISPATCH --no-ack +--- received on ORDERS.processed +order 5 + +$ nats con info ORDERS DISPATCH +State: + + Last Delivered Message: Consumer sequence: 3 Stream sequence: 3 + Acknowledgment floor: Consumer sequence: 1 Stream sequence: 1 + Pending Messages: 1 + Redelivered Messages: 0 +``` + +Now we can see the Consumer have processed 2 messages (obs sequence is 3, next message will be 3) but the Ack floor is still 1 - thus 1 message is pending acknowledgement. Indeed this is confirmed in the `Pending messages`. + +If I fetch it again and again do not ack it: + +``` +$ nats con next ORDERS DISPATCH --no-ack +--- received on ORDERS.processed +order 5 + +$ nats con info ORDERS DISPATCH +State: + + Last Delivered Message: Consumer sequence: 4 Stream sequence: 3 + Acknowledgment floor: Consumer sequence: 1 Stream sequence: 1 + Pending Messages: 1 + Redelivered Messages: 1 +``` + +The Consumer sequence increases - each delivery attempt increase the sequence - and our redelivered count also goes up. + +Finally if I then fetch it again and ack it this time: + +``` +$ nats con next ORDERS DISPATCH +--- received on ORDERS.processed +order 5 + +Acknowledged message +$ nats con info ORDERS DISPATCH +State: + + Last Delivered Message: Consumer sequence: 5 Stream sequence: 3 + Acknowledgment floor: Consumer sequence: 1 Stream sequence: 1 + Pending Messages: 0 + Redelivered Messages: 0 +``` + +Having now Acked the message there are no more pending. + +Additionally there are a few types of acknowledgements: + +|Type|Bytes|Description| +|----|-----|-----------| +|`AckAck`|nil, `+ACK`|Acknowledges a message was completely handled| +|`AckNak`|`-NAK`|Signals that the message will not be processed now and processing can move onto the next message, NAK'd message will be retried| +|`AckProgress`|`+WPI`|When sent before the AckWait period indicates that work is ongoing and the period should be extended by another equal to `AckWait`| +|`AckNext`|`+NXT`|Acknowledges the message was handled and requests delivery of the next message to the reply subject. Only applies to Pull-mode.| +|`AckTerm`|`+TERM`|Instructs the server to stop redelivery of a message without acknowledging it as successfully processed| + +So far all the examples was the `AckAck` type of acknowledgement, by replying to the Ack with the body as indicated in `Bytes` you can pick what mode of acknowledgement you want. + +All of these acknowledgement modes, except `AckNext`, support double acknowledgement - if you set a reply subject when acknowledging the server will in turn acknowledge having received your ACK. + +The `+NXT` acknowledgement can have a few formats: `+NXT 10` requests 10 messages and `+NXT {"no_wait": true}` which is the same data that can be sent in a Pull request. + +### Exactly Once Delivery + +JetStream supports Exactly Once delivery by combining Message Deduplication and double acks. + +On the publishing side you can avoid duplicate message ingestion using the [Message Deduplication](#message-deduplication) feature. + +Consumers can be 100% sure a message was correctly processed by requesting the server Acknowledge having received your acknowledgement by setting a reply subject on the Ack. If you receive this response you will never receive that message again. + +### Consumer Starting Position + +When setting up an Consumer you can decide where to start, the system supports the following for the `DeliverPolicy`: + +|Policy|Description| +|------|-----------| +|`all`|Delivers all messages that are available| +|`last`|Delivers the latest message, like a `tail -n 1 -f`| +|`new`|Delivers only new messages that arrive after subscribe time| +|`by_start_time`|Delivers from a specific time onward. Requires `OptStartTime` to be set| +|`by_start_sequence`|Delivers from a specific stream sequence. Requires `OptStartSeq` to be set| + +Regardless of what mode you set, this is only the starting point. Once started it will always give you what you have not seen or acknowledged. So this is merely how it picks the very first message. + +Lets look at each of these, first we make a new Stream `ORDERS` and add 100 messages to it. + +Now create a `DeliverAll` pull-based Consumer: + +``` +$ nats con add ORDERS ALL --pull --filter ORDERS.processed --ack none --replay instant --deliver all +$ nats con next ORDERS ALL +--- received on ORDERS.processed +order 1 + +Acknowledged message +``` + +Now create a `DeliverLast` pull-based Consumer: + +``` +$ nats con add ORDERS LAST --pull --filter ORDERS.processed --ack none --replay instant --deliver last +$ nats con next ORDERS LAST +--- received on ORDERS.processed +order 100 + +Acknowledged message +``` + +Now create a `MsgSetSeq` pull-based Consumer: + +``` +$ nats con add ORDERS TEN --pull --filter ORDERS.processed --ack none --replay instant --deliver 10 +$ nats con next ORDERS TEN +--- received on ORDERS.processed +order 10 + +Acknowledged message +``` + +And finally a time-based Consumer. Let's add some messages a minute apart: + +``` +$ nats str purge ORDERS +$ for i in 1 2 3 +do + nats pub ORDERS.processed "order ${i}" + sleep 60 +done +``` + +Then create an Consumer that starts 2 minutes ago: + +``` +$ nats con add ORDERS 2MIN --pull --filter ORDERS.processed --ack none --replay instant --deliver 2m +$ nats con next ORDERS 2MIN +--- received on ORDERS.processed +order 2 + +Acknowledged message +``` + +### Ephemeral Consumers + +So far, all the Consumers you have seen were Durable, meaning they exist even after you disconnect from JetStream. In our Orders scenario, though the `MONITOR` Consumer could very well be a short-lived thing there just while an operator is debugging the system, there is no need to remember the last seen position if all you are doing is wanting to observe the real-time state. + +In this case, we can make an Ephemeral Consumer by first subscribing to the delivery subject, then creating a durable and giving it no durable name. An Ephemeral Consumer exists as long as any subscription is active on its delivery subject. It is automatically be removed, after a short grace period to handle restarts, when there are no subscribers. + +Ephemeral Consumers can only be push-based. + +Terminal 1: + +``` +$ nats sub my.monitor +``` + +Terminal 2: + +``` +$ nats con add ORDERS --filter '' --ack none --target 'my.monitor' --deliver last --replay instant --ephemeral +``` + +The `--ephemeral` switch tells the system to make an Ephemeral Consumer. + +### Consumer Message Rates + +Typically what you want is if a new Consumer is made the selected messages are delivered to you as quickly as possible. You might want to replay messages at the rate they arrived though, meaning if messages first arrived 1 minute apart and you make a new Consumer it will get the messages a minute apart. + +This is useful in load testing scenarios etc. This is called the `ReplayPolicy` and have values of `ReplayInstant` and `ReplayOriginal`. + +You can only set `ReplayPolicy` on push-based Consumers. + +``` +$ nats con add ORDERS REPLAY --target out.original --filter ORDERS.processed --ack none --deliver all --sample 100 --replay original +... + Replay Policy: original +... +``` + +Now lets publish messages into the Set 10 seconds apart: + +``` +$ for i in 1 2 3 <15:15:35 +do + nats pub ORDERS.processed "order ${i}" + sleep 10 +done +Published [ORDERS.processed] : 'order 1' +Published [ORDERS.processed] : 'order 2' +Published [ORDERS.processed] : 'order 3' +``` + +And when we consume them they will come to us 10 seconds apart: + +``` +$ nats sub -t out.original +Listening on [out.original] +2020/01/03 15:17:26 [#1] Received on [ORDERS.processed]: 'order 1' +2020/01/03 15:17:36 [#2] Received on [ORDERS.processed]: 'order 2' +2020/01/03 15:17:46 [#3] Received on [ORDERS.processed]: 'order 3' +^C +``` + +### Stream Templates + +When you have many similar streams it can be helpful to auto create them, lets say you have a service by client and they are on subjects `CLIENT.*`, you can construct a template that will auto generate streams for any matching traffic. + +``` +$ nats str template add CLIENTS --subjects "CLIENT.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size 2048 --max-streams 1024 --discard old +Stream Template CLIENTS was created + +Information for Stream Template CLIENTS + +Configuration: + + Subjects: CLIENT.* + Acknowledgements: true + Retention: File - Limits + Replicas: 1 + Maximum Messages: -1 + Maximum Bytes: -1 + Maximum Age: 8760h0m0s + Maximum Message Size: 2048 + Maximum Consumers: -1 + Maximum Streams: 1024 + +Managed Streams: + + No Streams have been defined by this template +``` + +You can see no streams currently exist, let's publish some data: + +``` +$ nats pub CLIENT.acme hello +``` + +And we'll have 1 new Stream: + +``` +$ nats str ls +Streams: + + CLIENTS_acme +``` + +When the template is deleted all the streams it created will be deleted too. + +### Ack Sampling + +In the earlier sections we saw that samples are being sent to a monitoring system. Let's look at that in depth; how the monitoring system works and what it contains. + +As messages pass through an Consumer you'd be interested in knowing how many are being redelivered and how many times but also how long it takes for messages to be acknowledged. + +Consumers can sample Ack'ed messages for you and publish samples so your monitoring system can observe the health of an Consumer. We will add support for this to [NATS Surveyor](https://github.com/nats-io/nats-surveyor). + +#### Configuration + +You can configure an Consumer for sampling by passing the `--sample 80` option to `nats consumer add`, this tells the system to sample 80% of Acknowledgements. + +When viewing info of an Consumer you can tell if it's sampled or not: + +``` +$ nats con info ORDERS NEW +... + Sampling Rate: 100 +... +``` + +#### Consuming + +Samples are published to `$JS.EVENT.METRIC.CONSUMER_ACK..` in JSON format containing `api.ConsumerAckMetric`. Use the `nats con events` command to view samples: + +```nohighlight +$ nats con events ORDERS NEW +Listening for Advisories on $JS.EVENT.ADVISORY.*.ORDERS.NEW +Listening for Metrics on $JS.EVENT.METRIC.*.ORDERS.NEW + +15:08:31] [Ph0fsiOKRg1TS0c2k0mMz2] Acknowledgement Sample + Consumer: ORDERS > NEW + Stream Sequence: 40 + Consumer Sequence: 161 + Deliveries: 1 + Delay: 1.009ms +``` + +```nohighlight +$ nats con events ORDERS NEW --json +{ + "stream": "ORDERS", + "consumer": "NEW", + "consumer_seq": 155, + "stream_seq": 143, + "ack_time": 5387000, + "delivered": 1 +} +{ + "stream": "ORDERS", + "consumer": "NEW", + "consumer_seq": 156, + "stream_seq": 144, + "ack_time": 5807800, + "delivered": 1 +} +``` + +### Storage Overhead + +JetStream file storage is very efficient storing as little extra information about the message as possible. + +**NOTE:** This might change once clustering is supported. + +We do store some message data with each message, namely: + + * Message headers + * The subject it was received on + * The time it was received + * The message payload + * A hash of the message + * The message sequence + * A few other bits like length of the subject and lengh of headers + +Without any headers the size is: + +``` +length of the message record (4bytes) + seq(8) + ts(8) + subj_len(2) + subj + msg + hash(8) +``` + +A 5 byte `hello` message without headers will take 39 bytes. + +With headers: + +``` +length of the message record (4bytes) + seq(8) + ts(8) + subj_len(2) + subj + hdr_len(4) + hdr + msg + hash(8) +``` + +So if you are publishing many small messages the overhead will be, relatively speaking, quite large, but for larger messages +the overhead is very small. If you publish many small messages it's worth trying to optimise the subject length. diff --git a/jetstream/monitoring/monitoring.md b/jetstream/monitoring/monitoring.md new file mode 100644 index 0000000..13b5185 --- /dev/null +++ b/jetstream/monitoring/monitoring.md @@ -0,0 +1,30 @@ +## Monitoring + +### Server Metrics + +Typically, NATS is monitored via HTTP endpoints like `/varz`, we do not at this moment have a JetStream equivelant, but it's planned that server and account level metrics will be made available. + +### Advisories + +JetStream publish a number of advisories that can inform operations about health and state of the Streams. These advisories are published to normal NATS subjects below `$JS.EVENT.ADVISORY.>` and one can store these advisories in JetStream Streams if desired. + +The command `nats event --js-advsiroy` can view all these events on your console. The Golang package [jsm.go](https://github.com/nats-io/jsm.go) can consume and render these events and have data types for each of these events. + +All these events have JSON Schemas that describe them, schemas can be viewed on the CLI using the `nats schema show ` command. + +|Description|Subject|Kind| +|-----------|-------|----| +|API interactions|`$JS.EVENT.ADVISORY.API`|`io.nats.jetstream.advisory.v1.api_audit`| +|Stream CRUD operations|`$JS.EVENT.ADVISORY.STREAM.CREATED.`|`io.nats.jetstream.advisory.v1.stream_action`| +|Consumer CRUD operations|`$JS.EVENT.ADVISORY.CONSUMER.CREATED..`|`io.nats.jetstream.advisory.v1.consumer_action`| +|Snapshot started using `nats stream backup`|`$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_CREATE.`|`io.nats.jetstream.advisory.v1.snapshot_create`| +|Snapshot completed|`$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_COMPLETE.`|`io.nats.jetstream.advisory.v1.snapshot_complete`| +|Restore started using `nats stream restore`|`$JS.EVENT.ADVISORY.STREAM.RESTORE_CREATE.`|`io.nats.jetstream.advisory.v1.restore_create`| +|Restore completed|`$JS.EVENT.ADVISORY.STREAM.RESTORE_COMPLETE.`|`io.nats.jetstream.advisory.v1.restore_complete`| +|Consumer maximum delivery reached|`$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES..`|`io.nats.jetstream.advisory.v1.max_deliver`| +|Message delivery terminated using AckTerm|`$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED..`|`io.nats.jetstream.advisory.v1.terminated`| +|Message acknowledged in a sampled Consumer|`$JS.EVENT.METRIC.CONSUMER.ACK..`|`io.nats.jetstream.metric.v1.consumer_ack`| + +### Dashboards + +The [NATS Surveyor](https://github.com/nats-io/nats-surveyor) system has initial support for passing JetStream metrics to Prometheus, dashboards and more will be added towards final release. \ No newline at end of file diff --git a/jetstream/multi-tenancy/resource_management.md b/jetstream/multi-tenancy/resource_management.md new file mode 100644 index 0000000..20d9838 --- /dev/null +++ b/jetstream/multi-tenancy/resource_management.md @@ -0,0 +1,133 @@ +## Multi Tenancy and Resource Management + +JetStream is compatible with NATS 2.0 Multi Tenancy using Accounts. A JetStream enabled server supports creating fully isolated JetStream environments for different accounts. + +To enable JetStream in a server we have to configure it at the top level first: + +``` +jetstream: enabled +``` + +This will dynamically determine the available resources. It's recommended that you set specific limits though: + +``` +jetstream { + store_dir: /data/jetstream + max_mem: 1G + max_file: 100G +} +``` + +At this point JetStream will be enabled and if you have a server that does not have accounts enabled all users in the server would have access to JetStream + +``` +jetstream { + store_dir: /data/jetstream + max_mem: 1G + max_file: 100G +} + +accounts { + HR: { + jetstream: enabled + } +} +``` + +Here the `HR` account would have access to all the resources configured on the server, we can restrict it: + +``` +jetstream { + store_dir: /data/jetstream + max_mem: 1G + max_file: 100G +} + +accounts { + HR: { + jetstream { + max_mem: 512M + max_file: 1G + max_streams: 10 + max_consumers: 100 + } + } +} +``` + +Now the `HR` account it limited in various dimensions. + +If you try to configure JetStream for an account without enabling it globally you'll get a warning and the account designated as System cannot have JetStream enabled. + +## `nats` CLI + +As part of the JetStream efforts a new `nats` CLI is being developed to act as a single point of access to the NATS eco system. + +This CLI has been seen throughout the guide, it's available in the Docker containers today and downloadable on the [Releases](https://github.com/nats-io/jetstream/releases) +page. + +### Configuration Contexts + +The CLI has a number of environment configuration settings - where your NATS server is, credentials, TLS keys and more: + +```nohighlight +$ nats --help +... + -s, --server=NATS_URL NATS servers + --user=NATS_USER Username of Token + --password=NATS_PASSWORD Password + --creds=NATS_CREDS User credentials + --nkey=NATS_NKEY User NKEY + --tlscert=NATS_CERT TLS public certificate + --tlskey=NATS_KEY TLS private key + --tlsca=NATS_CA TLS certificate authority chain + --timeout=NATS_TIMEOUT Time to wait on responses from NATS + --context=CONTEXT NATS Configuration Context to use for access +... +``` + +You can set these using the CLI flag, the environmet variable - like **NATS_URL** - or using our context feature. + +A context is a named configuration that stores all these settings, you can switch between access configurations and +designate a default. + +Creating one is easy, just specify the same settings to the `nats context save` + +```nohighlight +$ nats context save example --server nats://nats.example.net:4222 --description 'Example.Net Server' +$ nats context save local --server nats://localhost:4222 --description 'Local Host' --select +$ nats context ls +Known contexts: + + example Example.Net Server + local* Local Host +``` + +We passed `--select` to the `local` one meaning it will be the default when nothing is set. + +```nohighlight +$ nats rtt +nats://localhost:4222: + + nats://127.0.0.1:4222: 245.115µs + nats://[::1]:4222: 390.239µs + +$ nats rtt --context example +nats://nats.example.net:4222: + + nats://192.0.2.10:4222: 41.560815ms + nats://192.0.2.11:4222: 41.486609ms + nats://192.0.2.12:4222: 41.178009ms +``` + +The `nats context select` command can be used to set the default context. + +All `nats` commands are context aware and the `nats context` command has various commands to view, edit and remove contexts. + +Server URLs and Credential paths can be resolved via the `nsc` command by specifying an url, for example to find user `new` within the `orders` account of the `acme` operator you can use this: + +```nohighlight +$ nats context save example --description 'Example.Net Server' --nsc nsc://acme/orders/new +``` + +The server list and credentials path will now be resolved via `nsc`, if these are specifically set in the context, the specific context configuration will take precedence. diff --git a/jetstream/nats_api_reference/nats_api_reference.md b/jetstream/nats_api_reference/nats_api_reference.md new file mode 100644 index 0000000..1e994ed --- /dev/null +++ b/jetstream/nats_api_reference/nats_api_reference.md @@ -0,0 +1,221 @@ +## NATS API Reference + +Thus far we saw a lot of CLI interactions. The CLI works by sending and receiving specially crafted messages over core NATS to configure the JetStream system. In time we will look to add file based configuration but for now the only method is the NATS API. + +**NOTE:** Some NATS client libraries may need to enable an option to use old style requests when interacting withe JetStream server. Consult the libraries README's for more information. + +### Reference + +All of these subjects are found as constants in the NATS Server source, so for example the `$JS.API.STREAM.LIST` is a constant in the nats-server source `api.JetStreamListStreams` tables below will reference these constants and likewise data structures in the server for payloads. + +### Error Handling + +The APIs used for administrative tools all respond with standardised JSON and these include errors. + +```nohighlight +$ nats req '$JS.API.STREAM.INFO.nonexisting' '' +Published 11 bytes to $JS.API.STREAM.INFO.nonexisting +Received [_INBOX.lcWgjX2WgJLxqepU0K9pNf.mpBW9tHK] : { + "type": "io.nats.jetstream.api.v1.stream_info_response", + "error": { + "code": 404, + "description": "stream not found" + } +} +``` + +```nohighlight +$ nats req '$JS.STREAM.INFO.ORDERS' '' +Published 6 bytes to $JS.STREAM.INFO.ORDERS +Received [_INBOX.fwqdpoWtG8XFXHKfqhQDVA.vBecyWmF] : '{ + "type": "io.nats.jetstream.api.v1.stream_info_response", + "config": { + "name": "ORDERS", + ... +} +``` + +Here the responses include a `type` which can be used to find the JSON Schema for each response. + +Non admin APIs - like those for adding a message to the stream will respond with `-ERR` or `+OK` with an optional reason after. + +### Admin API + +All the admin actions the `nats` CLI can do falls in the sections below. The API structure are kept in the `api` package in the `jsm.go` repository. + +Subjects that and in `T` like `api.JSApiConsumerCreateT` are formats and would need to have the Stream Name and in some cases also the Consumer name interpolated into them. In this case `t := fmt.Sprintf(api.JSApiConsumerCreateT, streamName)` to get the final subject. + +The command `nats events` will show you an audit log of all API access events which includes the full content of each admin request, use this to view the structure of messages the `nats` command sends. + +The API uses JSON for inputs and outputs, all the responses are typed using a `type` field which indicates their Schema. A JSON Schema repository can be found in `nats-io/jetstream/schemas`. + +#### General Info + +|Subject|Constant|Description|Request Payload|Response Payload| +|-------|--------|-----------|---------------|----------------| +|`$JS.API.INFO`|`api.JSApiAccountInfo`|Retrieves stats and limits about your account|empty payload|`api.JetStreamAccountStats`| + +#### Streams + +|Subject|Constant|Description|Request Payload|Response Payload| +|-------|--------|-----------|---------------|----------------| +|`$JS.API.STREAM.LIST`|`api.JSApiStreamList`|Paged list known Streams including all their current information|`api.JSApiStreamListRequest`|`api.JSApiStreamListResponse`| +|`$JS.API.STREAM.NAMES`|`api.JSApiStreamNames`|Paged list of Streams|`api.JSApiStreamNamesRequest`|`api.JSApiStreamNamesResponse`| +|`$JS.API.STREAM.LOOKUP`|`api.JSApiStreamLookup`|Finds a stream with interest on a certain subject|`api.JSApiStreamLookupRequest`|`api.JSApiStreamLookupResponse`| +|`$JS.API.STREAM.CREATE.*`|`api.JSApiStreamCreateT`|Creates a new Stream|`api.StreamConfig`|`api.JSApiStreamCreateResponse`| +|`$JS.API.STREAM.UPDATE.*`|`api.JSApiStreamUpdateT`|Updates an existing Stream with new config|`api.StreamConfig`|`api.JSApiStreamUpdateResponse`| +|`$JS.API.STREAM.INFO.*`|`api.JSApiStreamInfoT`|Information about config and state of a Stream|empty payload, Stream name in subject|`api.JSApiStreamInfoResponse`| +|`$JS.API.STREAM.DELETE.*`|`api.JSApiStreamDeleteT`|Deletes a Stream and all its data|empty payload, Stream name in subject|`api.JSApiStreamDeleteResponse`| +|`$JS.API.STREAM.PURGE.*`|`api.JSApiStreamPurgeT`|Purges all of the data in a Stream, leaves the Stream|empty payload, Stream name in subject|`api.JSApiStreamPurgeResponse`| +|`$JS.API.STREAM.MSG.DELETE.*`|`api.JSApiMsgDeleteT`|Deletes a specific message in the Stream by sequence, useful for GDPR compliance|`api.JSApiMsgDeleteRequest`|`api.JSApiMsgDeleteResponse`| +|`$JS.API.STREAM.MSG.GET.*`|`api.JSApiMsgGetT`|Retrieves a specific message from the stream|`api.JSApiMsgGetRequest`|`api.JSApiMsgGetResponse`| +|`$JS.API.STREAM.SNAPSHOT.*`|`api.JSApiStreamSnapshotT`|Initiates a streaming backup of a streams data|`api.JSApiStreamSnapshotRequest`|`api.JSApiStreamSnapshotResponse`| +|`$JS.API.STREAM.RESTORE.*`|`api.JSApiStreamRestoreT`|Initiates a streaming restore of a stream|`{}`|`api.JSApiStreamRestoreResponse`| + +#### Stream Templates + +|Subject|Constant|Description|Request Payload|Response Payload| +|-------|--------|-----------|---------------|----------------| +|`$JS.API.STREAM.TEMPLATE.CREATE.*`|`api.JSApiTemplateCreateT`|Creates a Stream Template|`api.StreamTemplateConfig`|`api.JSApiStreamTemplateCreateResponse`| +|`$JS.API.STREAM.TEMPLATE.NAMES`|`api.JSApiTemplateNames`|Paged list all known templates|`api.JSApiStreamTemplateNamesRequest`|`api.JSApiStreamTemplateNamesResponse`| +|`$JS.API.STREAM.TEMPLATE.INFO.*`|`api.JSApiTemplateInfoT`|Information about the config and state of a Stream Template|empty payload, Template name in subject|`api.JSApiStreamTemplateInfoResponse`| +|`$JS.API.STREAM.TEMPLATE.DELETE.*`|`api.JSApiTemplateDeleteT`|Delete a specific Stream Template **and all streams created by this template**|empty payload, Template name in subject|`api.JSApiStreamTemplateDeleteResponse`| + +#### Consumers + +|Subject|Constant|Description|Request Payload|Response Payload| +|-------|--------|-----------|---------------|----------------| +|`$JS.API.CONSUMER.CREATE.*`|`api.JSApiConsumerCreateT`|Create an ephemeral Consumer|`api.ConsumerConfig`, Stream name in subject|`api.JSApiConsumerCreateResponse`| +|`$JS.API.CONSUMER.DURABLE.CREATE.*`|`api.JSApiDurableCreateT`|Create an Consumer|`api.ConsumerConfig`, Stream name in subject|`api.JSApiConsumerCreateResponse`| +|`$JS.API.CONSUMER.LIST.*`|`api.JSApiConsumerListT`|Paged list of known Consumers including their current info|`api.JSApiConsumerListRequest`|`api.JSApiConsumerListResponse`| +|`$JS.API.CONSUMER.NAMES.*`|`api.JSApiConsumerNamesT`|Paged list of known Consumer names|`api.JSApiConsumerNamesRequest`|`api.JSApiConsumerNamesResponse`| +|`$JS.API.CONSUMER.INFO.*.*`|`api.JSApiConsumerInfoT`|Information about an Consumer|empty payload, Stream and Consumer names in subject|`api.JSApiConsumerInfoResponse`| +|`$JS.API.CONSUMER.DELETE.*.*`|`api.JSApiConsumerDeleteT`|Deletes an Consumer|empty payload, Stream and Consumer names in subject|`api.JSApiConsumerDeleteResponse`| + +#### ACLs + +It's hard to notice here but there is a clear pattern in these subjects, lets look at the various JetStream related subjects: + +General information + +``` +$JS.API.INFO +``` + +Stream and Consumer Admin + +``` +$JS.API.STREAM.CREATE. +$JS.API.STREAM.UPDATE. +$JS.API.STREAM.DELETE. +$JS.API.STREAM.INFO. +$JS.API.STREAM.PURGE. +$JS.API.STREAM.LIST +$JS.API.STREAM.NAMES +$JS.API.STREAM.MSG.DELETE. +$JS.API.STREAM.MSG.GET. +$JS.API.STREAM.SNAPSHOT. +$JS.API.STREAM.RESTORE. +$JS.API.CONSUMER.CREATE. +$JS.API.CONSUMER.DURABLE.CREATE.. +$JS.API.CONSUMER.DELETE.. +$JS.API.CONSUMER.INFO.. +$JS.API.CONSUMER.LIST. +$JS.API.CONSUMER.MSG.NEXT.. +$JS.API.CONSUMER.NAMES. +$JS.API.STREAM.TEMPLATE.CREATE. +$JS.API.STREAM.TEMPLATE.DELETE. +$JS.API.STREAM.TEMPLATE.INFO. +$JS.API.STREAM.TEMPLATE.NAMES +``` + +Stream and Consumer Use + +``` +$JS.API.CONSUMER.MSG.NEXT.. +$JS.ACK...x.x.x +$JS.SNAPSHOT.ACK.. +$JS.SNAPSHOT.RESTORE.. +``` + +Events and Advisories: + +``` +$JS.EVENT.METRIC.CONSUMER_ACK.. +$JS.EVENT.ADVISORY.MAX_DELIVERIES.. +``` + +This allow you to easily create ACL rules that limit users to a specific Stream or Consumer and to specific verbs for administration purposes. For ensuring only the receiver of a message can Ack it we have response permissions ensuring you can only Publish to Response subject for messages you received. + +### Acknowledging Messages + +Messages that need acknowledgement will have a Reply subject set, something like `$JS.ACK.ORDERS.test.1.2.2`, this is the prefix defined in `api.JetStreamAckPre` followed by `......`. + +In all the Synadia maintained API's you can simply do `msg.Respond(nil)` (or language equivalent) which will send nil to the reply subject. + +### Fetching The Next Message From a Pull-based Consumer + +If you have a pull-based Consumer you can send a standard NATS Request to `$JS.API.CONSUMER.MSG.NEXT..`, here the format is defined in `api.JetStreamRequestNextT` and requires populating using `fmt.Sprintf()`. + +```nohighlight +$ nats req '$JS.API.CONSUMER.MSG.NEXT.ORDERS.test' '1' +Published 1 bytes to $JS.API.CONSUMER.MSG.NEXT.ORDERS.test +Received [js.1] : 'message 1' +``` + +Here we ask for just 1 message - `nats req` only shows 1 - but you can fetch a batch of messages by varying the argument. This combines well with the `AckAll` Ack policy. + +The above request for the next message will stay in the server for as long as the client is connected and future pulls from the same client will accumulate on the server, meaning if you ask for 1 message 100 times and 1000 messages arrive you'll get sent 100 messages not 1. + +This is often not desired, pull consumers support a mode where a JSON document is sent describing the pull request. + +```json +{ + "expires": "2020-11-10T12:41:00.075933464Z", + "batch": 10, +} +``` + +This requests 10 messages and asks the server to keep this request until the specific `expires` time, this is useful when you poll the server frequently and do not want the pull requests to accumulate on the server. Set the expire time to now + your poll frequency. + +```json +{ + "batch": 10, + "no_wait": true +} +``` + +Here we see a second format of the Pull request that will not store the request on the queue at all but when there are no messages to deliver will send a nil bytes message with a `Status` header of `404`, this way you can know when you reached the end of the stream for example. A `409` is returned if the Consumer has reached `MaxAckPending` limits. + +``` +[rip@dev1]% nats req '$JS.API.CONSUMER.MSG.NEXT.ORDERS.NEW' '{"no_wait": true, "batch": 10}' + test --password test +13:45:30 Sending request on "$JS.API.CONSUMER.MSG.NEXT.ORDERS.NEW" +13:45:30 Received on "_INBOX.UKQGqq0W1EKl8inzXU1naH.XJiawTRM" rtt 594.908µs +13:45:30 Status: 404 +13:45:30 Description: No Messages +``` + +### Fetching From a Stream By Sequence + +If you know the Stream sequence of a message you can fetch it directly, this does not support acks. Do a Request() to `$JS.API.STREAM.MSG.GET.ORDERS` sending it the message sequence as payload. Here the prefix is defined in `api.JetStreamMsgBySeqT` which also requires populating using `fmt.Sprintf()`. + +```nohighlight +$ nats req '$JS.API.STREAM.MSG.GET.ORDERS' '{"seq": 1}' +Published 1 bytes to $JS.STREAM.ORDERS.MSG.BYSEQ +Received [_INBOX.cJrbzPJfZrq8NrFm1DsZuH.k91Gb4xM] : '{ + "type": "io.nats.jetstream.api.v1.stream_msg_get_response", + "message": { + "subject": "x", + "seq": 1, + "data": "aGVsbG8=", + "time": "2020-05-06T13:18:58.115424+02:00" + } +}' +``` + +The Subject shows where the message was received, Data is base64 encoded and Time is when it was received. + +### Consumer Samples + +Samples are published to a specific subject per Consumer, something like `$JS.EVENT.METRIC.CONSUMER_ACK..` you can just subscribe to that and get `api.ConsumerAckMetric` messages in JSON format. The prefix is defined in `api.JetStreamMetricConsumerAckPre`. From 5a1f9b9c6caa4d70a19eead01c6c20fd568eaa82 Mon Sep 17 00:00:00 2001 From: ainsley Date: Tue, 24 Nov 2020 16:56:10 -0600 Subject: [PATCH 02/31] adding JetStream to outline --- SUMMARY.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/SUMMARY.md b/SUMMARY.md index 0d6e9d0..d35f1ef 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -204,3 +204,37 @@ * [Creating a NATS Super Cluster in Digital Ocean with Helm](nats-on-kubernetes/super-cluster-on-digital-ocean.md) * [From Zero to K8S to Leafnodes using Helm](nats-on-kubernetes/from-zero-to-leafnodes.md) +## JetStream + +* [Concepts](jetstream/concepts/concepts.md) + * [Streams](jetstream/concepts/streams.md) + * [Consumes](jetstream/concepts/consumers.md) + * [Configuration](jetstream/concepts/configuration.md) +* [Getting Started](jetstream/getting_started/getting_started.md) + * [Using Docker](jetstream/getting_started/using_docker.md) + * [Using Docker with NGS](jetstream/getting_started/using_docker.md#using-docker-with-ngs) + * [Using Source](jetstream/getting_started/using_source.md) +* [Administration & Usage from CLI](jetstream/administration/administration.md) + * [Account Information](jetstream/administration/account.md) + * [Streams](jetstream/administration/streams.md) + * [Consumers](jetstream/administration/consumers.md) +* [Monitoring](jetstream/monitoring/monitoring.md) +* [Configuration Management](jetstream/configuration_mgmt/configuration_mgmt.md) + * [nats CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) + * [Terraform](jetstream/configuration_mgmt/configuration_mgmt.md#terraform) + * [GitHub Actions](jetstream/configuration_mgmt/github_actions.md) + * [Kubernetes Controller](jetstream/configuration_mgmt/kubernetes_controller.md) +* [Disaser Recovery](jetstream/disaster_recovery/disaster_recovery.md) +* [Model Deep Dive](jetstream/model_deep_dive/model_deep_dive.md) + * [Stream Limits & Retention Modes](jetstream/model_deep_dive/model_deep_dive.md#stream-limits-retention-modes-and-discard-policy) + * [Message Deduplication](jetstream/model_deep_dive/model_deep_dive.md#message-deduplication) + * [Acknowledgment Models](jetstream/model_deep_dive/model_deep_dive.md#acknowledgement-models) + * [Exactly Once Delivery](jetstream/model_deep_dive/model_deep_dive.md#exactly-once-delivery) + * [Consumer Starting Position](jetstream/model_deep_dive/model_deep_dive.md#consumer-starting-position) + * [Ephemeral Consumers](jetstream/model_deep_dive/model_deep_dive.md#ephemeral-consumers) + * [Consumer Message Rates](jetstream/model_deep_dive/model_deep_dive.md#consumer-message-rates) + * [Stream Templates](jetstream/model_deep_dive/model_deep_dive.md#stream-templates) + * [Ack Sampling](jetstream/model_deep_dive/model_deep_dive.md#ack-sampling) + * [Storage Overhead](jetstream/model_deep_dive/model_deep_dive.md#storage-overhead) +* [NATS API Reference](jetstream/nats_api_reference/nats_api_reference.md) +* [Multi-tenancy & Resource Mgmt](jetstream/multi-tenancy/resource_management.md) From fa388104238e1942df212cdf0f4cba04f3f2c0a1 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 11 Dec 2020 14:28:53 -0600 Subject: [PATCH 03/31] Update SUMMARY.md add updates from https://github.com/nats-io/jetstream/pull/392 --- SUMMARY.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SUMMARY.md b/SUMMARY.md index d35f1ef..c0c55b0 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -220,13 +220,13 @@ * [Consumers](jetstream/administration/consumers.md) * [Monitoring](jetstream/monitoring/monitoring.md) * [Configuration Management](jetstream/configuration_mgmt/configuration_mgmt.md) - * [nats CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) + * [nats Admin CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) * [Terraform](jetstream/configuration_mgmt/configuration_mgmt.md#terraform) * [GitHub Actions](jetstream/configuration_mgmt/github_actions.md) * [Kubernetes Controller](jetstream/configuration_mgmt/kubernetes_controller.md) * [Disaser Recovery](jetstream/disaster_recovery/disaster_recovery.md) * [Model Deep Dive](jetstream/model_deep_dive/model_deep_dive.md) - * [Stream Limits & Retention Modes](jetstream/model_deep_dive/model_deep_dive.md#stream-limits-retention-modes-and-discard-policy) + * [Stream Limits, Retention Modes and Discard Policy](jetstream/model_deep_dive/model_deep_dive.md#stream-limits-retention-modes-and-discard-policy) * [Message Deduplication](jetstream/model_deep_dive/model_deep_dive.md#message-deduplication) * [Acknowledgment Models](jetstream/model_deep_dive/model_deep_dive.md#acknowledgement-models) * [Exactly Once Delivery](jetstream/model_deep_dive/model_deep_dive.md#exactly-once-delivery) From f1a92bdcd6997fc6cac1973fef00c11480febd84 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Thu, 4 Feb 2021 17:48:40 -0700 Subject: [PATCH 04/31] add about jetstream Signed-off-by: Colin Sullivan --- jetstream/about_jetstream/jetstream.md | 31 ++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 jetstream/about_jetstream/jetstream.md diff --git a/jetstream/about_jetstream/jetstream.md b/jetstream/about_jetstream/jetstream.md new file mode 100644 index 0000000..29261b8 --- /dev/null +++ b/jetstream/about_jetstream/jetstream.md @@ -0,0 +1,31 @@ +# Jetstream + +JetStream was created to solve the problems identified with streaming in technology today - complexity, fragility, and a lack of scalability. Some technologies address these better than others, but no current streaming technology is truly multi-tenant, horizontally scalable, and supports multiple deployment models. No technology we are aware of can scale from edge to cloud under the same security context while having complete deployment observability for operations. + +## Goals +JetStream was developed with the following goals in mind: + +- The system must be easy to configure and operate and be observable. +- The system must be secure and operate well with NATS 2.0 security models. +- The system must scale horizontally and be applicable to a high ingestion rate. +- The system must support multiple use cases. +- The system must self heal and always be available. +- The system must have an API that is closer to core NATS. +- The system must allow NATS messages to be part of a stream as desired. +- The system must display payload agnostic behavior. +- The system must not have third party dependencies. + +## High-Level Design and Features +In terms of deployment, a JetStream server is simply a NATS server with the JetStream subsystem enabled, launched with the `-js` flag with a configured server name and cluster name. From a client perspective, it does not matter which servers are running JetStream so long as there is some route to a JetStream enabled server or servers. This allows for a flexible deployment which to optimize resources for particular servers that will store streams versus very low overhead stateless servers, reducing OpEx and ultimately creating a scalable and manageable system. + +## Feature List +- At-least-once delivery; exactly once within a window +- Store messages and replay by time or sequence +- Wildcard support +- Account aware +- Data at rest encryption +- Cleanse specific messages (GDPR) +- Horizontal scalability +- Persist Streams and replay via Consumers + +JetStream is designed to bifurcate ingestion and consumption of messages to provide multiple ways to consume data from the same stream. To that end, JetStream functionality is composed of server streams and server consumers. From d422ccb39435c2c0059ae67dbe7d1c24ae55e84b Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Thu, 4 Feb 2021 17:48:57 -0700 Subject: [PATCH 05/31] update getting started Signed-off-by: Colin Sullivan --- jetstream/getting_started/getting_started.md | 27 ++++++++++++++++++-- jetstream/getting_started/using_docker.md | 2 +- jetstream/getting_started/using_source.md | 2 +- 3 files changed, 27 insertions(+), 4 deletions(-) diff --git a/jetstream/getting_started/getting_started.md b/jetstream/getting_started/getting_started.md index 0f95734..91b2364 100644 --- a/jetstream/getting_started/getting_started.md +++ b/jetstream/getting_started/getting_started.md @@ -1,3 +1,26 @@ -## Getting Started +# Getting Started -This tech preview is limited to a single server and defaults to the global account. JetStream is NATS 2.0 aware and is scoped to accounts from a resource limit perspective. This is not the same as an individual server's resources, but may feel that way starting out. Don't worry, clustering is coming next but we wanted to get input early from the community. \ No newline at end of file +Getting started with Jetstream is straightforward. While we speak of Jetstream as if it is a seperate component, it's actually a subsystem built into the NATS server that needs to be enabled. + +## Command line + +Enable Jetstream by specifying the `-js` flag when starting the NATS server. + +`$ nats-server -js` + +## Configuration File + +Enable Jetstream through a configuration file. By default, the Jetstream subsytem will store data in the /tmp directory. Here's a minimal file that will store data in a local "nats" directory, suitable for development and local testing. + +`$ nats-server -c js.conf` + +```text +# js.conf +jetstream { + store_dir=nats +} +``` + +Normally Jetstream will be run in clustered mode and will replicate data, so the best place to store Jetstream data would be locally on a fast SSD. One should specifically avoid NAS or NFS storage for Jetstream. + +See [Using Docker](./using_docker.md) and [Using Source](./using_source.md) for more information. \ No newline at end of file diff --git a/jetstream/getting_started/using_docker.md b/jetstream/getting_started/using_docker.md index d8923ea..a16549e 100644 --- a/jetstream/getting_started/using_docker.md +++ b/jetstream/getting_started/using_docker.md @@ -1,4 +1,4 @@ -### Using Docker +# Using Docker The `synadia/jsm:latest` docker image contains both the JetStream enabled NATS Server and the `nats` utility this guide covers. diff --git a/jetstream/getting_started/using_source.md b/jetstream/getting_started/using_source.md index 4f6bee2..b21eb8a 100644 --- a/jetstream/getting_started/using_source.md +++ b/jetstream/getting_started/using_source.md @@ -1,4 +1,4 @@ -### Using Source +# Using Source or a Binary You will also want to have installed from the nats.go repo the examples/tools such as nats-pub, nats-sub, nats-req and possibly nats-bench. One of the design goals of JetStream was to be native to core NATS, so even though we will most certainly add in syntactic sugar to clients to make them more appealing, for this tech preview we will be using plain old NATS. From 7bf9e7ce655aa5336f9509149cf408012faa3102 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Thu, 4 Feb 2021 22:12:21 -0700 Subject: [PATCH 06/31] add clustering doc Signed-off-by: Colin Sullivan --- jetstream/clustering/clustering.md | 92 ++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 jetstream/clustering/clustering.md diff --git a/jetstream/clustering/clustering.md b/jetstream/clustering/clustering.md new file mode 100644 index 0000000..1cb9fce --- /dev/null +++ b/jetstream/clustering/clustering.md @@ -0,0 +1,92 @@ +# Clustering + +Clustering in Jetstream is required for a highly available and scalable system. Behind +clustering is RAFT. There's no need to understand RAFT in depth to use clustering, but knowing a little explains some of the requirements behind setting up Jetstream clusters. + +## RAFT +JetStream uses a NATS optimized RAFT algorithm for clustering. Typically raft generates a lot of traffic, but the NATS server optimizes this by combining the data plane for replicating messages with the messages RAFT would normally use to ensure consensus. + +### Raft groups +The RAFT groups include API handlers sstreams, consumers, and an internal algorithm designates which servers handle which streams and consumers. + +The raft algorithm has a few requirements: +- A log to persist state +- A quorum for consensus. + +### The Quorum +In order to ensure data consistency across complete restarts, a quorum of servers is required. A quorum is ½ cluster size + 1. This is the minimum number of nodes to ensure at least one node has the most recent data and state after a catastrophic failure. So for a cluster size of 3, you’ll need at least two Jetstream enabled NATS servers available to store new messages. For a cluster size of 5, you’ll need at least 3 NATS servers, and so forth. + +### Cluster Size +Generally we recommend 3 or 5 Jetstream enabled servers in a NATS cluster. This balances scalability with a tolerance for failure. For example, if 5 servers are Jetstream enabled You would want two servers is one “zone”, two servers in another, and the remaining server in a third. This means you can lose any one “zone” at any time and continue operating. + +### Mixing Jetstream enabled servers with standard NATS servers + +This is possible, and even recommended in some cases. By mixing server types you can dedicate certain machines optimized for storage for Jetstream and others optimized solely for compute for standard NATS servers, reducing operational expense. With the right configuration, the standard servers would handle non-persistent NATS traffic and the Jetstream enabled servers would handle Jetstream traffic. + +## Configuration + +To configure Jetstream clusters, just configure clusters as you normally would by specifying a cluster block in the configuration. Any Jetstream enabled servers in the list of clusters will automatically chatter and set themselves up. Unlike core NATS clustering though, each Jetstream node must specify a server name and cluster name. + +Below are explicity listed server configuration for a three node cluster across three machinges, `host_a`, `host_b`, and `host_c`. + +### Server 1 (host_a) + +```text +server_name=server1 +listen=4222 + +jetstream { + store_dir=/nats/storage +} + +cluster { + name: c1 + listen: localhost:6222 + routes: [ + nats-route://host_b:6222 + nats-route://host_c:6222 + ] +} +``` + +### Server 2 (host_b) + +```text +server_name=server2 +listen=4222 + +jetstream { + store_dir=/nats/storage +} + +cluster { + name: c1 + listen: localhost:6222 + routes: [ + nats-route://host_a:6222 + nats-route://host_c:6222 + ] +} +``` + +### Server 3 (host_c) + +```text +server_name=server3 +listen=4222 + +jetstream { + store_dir=/nats/storage +} + +cluster { + name: c1 + listen: localhost:6222 + routes: [ + nats-route://host_a:6222 + nats-route://host_b:6222 + ] +} +``` + +Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use Jetstream. \ No newline at end of file From 34db7d248bdaaf1d163e3860c4e3af2dd906fef8 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Fri, 5 Feb 2021 10:43:22 -0700 Subject: [PATCH 07/31] clustering updates Signed-off-by: Colin Sullivan --- jetstream/clustering/clustering.md | 124 ++++++++++++++++++++++++++--- 1 file changed, 112 insertions(+), 12 deletions(-) diff --git a/jetstream/clustering/clustering.md b/jetstream/clustering/clustering.md index 1cb9fce..6e6ed08 100644 --- a/jetstream/clustering/clustering.md +++ b/jetstream/clustering/clustering.md @@ -16,6 +16,20 @@ The raft algorithm has a few requirements: ### The Quorum In order to ensure data consistency across complete restarts, a quorum of servers is required. A quorum is ½ cluster size + 1. This is the minimum number of nodes to ensure at least one node has the most recent data and state after a catastrophic failure. So for a cluster size of 3, you’ll need at least two Jetstream enabled NATS servers available to store new messages. For a cluster size of 5, you’ll need at least 3 NATS servers, and so forth. +### RAFT Groups + +**Meta Group** - all servers join the Meta Group and the JetStream API is managed by this group. A leader is elected and this owns the API and takes care of server placement. + +![](images/meta-group.png) FIXME + +**Stream Group** - each Stream creates a RAFT group, this group synchronizes state and data between its members. The elected leader handles ACKs and so forth, if there is no leader the stream will not accept messages. + +![](images/stream-groups.png) FIXME + +**Consumer Group** - each Consumer creates a RAFT group, this group synchronizes consumer state between its members. The group will live on the machines where the Stream Group is and handle consumption ACKs etc. Each Consumer will have its own group. + +![](images/consumer-groups.png) FIXME + ### Cluster Size Generally we recommend 3 or 5 Jetstream enabled servers in a NATS cluster. This balances scalability with a tolerance for failure. For example, if 5 servers are Jetstream enabled You would want two servers is one “zone”, two servers in another, and the remaining server in a third. This means you can lose any one “zone” at any time and continue operating. @@ -25,14 +39,14 @@ This is possible, and even recommended in some cases. By mixing server types yo ## Configuration -To configure Jetstream clusters, just configure clusters as you normally would by specifying a cluster block in the configuration. Any Jetstream enabled servers in the list of clusters will automatically chatter and set themselves up. Unlike core NATS clustering though, each Jetstream node must specify a server name and cluster name. +To configure Jetstream clusters, just configure clusters as you normally would by specifying a cluster block in the configuration. Any Jetstream enabled servers in the list of clusters will automatically chatter and set themselves up. Unlike core NATS clustering though, each Jetstream node **must specify** a server name and cluster name. -Below are explicity listed server configuration for a three node cluster across three machinges, `host_a`, `host_b`, and `host_c`. +Below are explicity listed server configuration for a three node cluster across three machines, `n1-c1`, `n2-c1`, and `n3-c1`. ### Server 1 (host_a) -```text -server_name=server1 +```nohighlight +server_name=n1-c1 listen=4222 jetstream { @@ -40,7 +54,7 @@ jetstream { } cluster { - name: c1 + name: C1 listen: localhost:6222 routes: [ nats-route://host_b:6222 @@ -51,8 +65,8 @@ cluster { ### Server 2 (host_b) -```text -server_name=server2 +```nohighlight +server_name=n2-c1 listen=4222 jetstream { @@ -60,7 +74,7 @@ jetstream { } cluster { - name: c1 + name: C1 listen: localhost:6222 routes: [ nats-route://host_a:6222 @@ -71,8 +85,8 @@ cluster { ### Server 3 (host_c) -```text -server_name=server3 +```nohighlight +server_name=n3-c1 listen=4222 jetstream { @@ -80,7 +94,7 @@ jetstream { } cluster { - name: c1 + name: C1 listen: localhost:6222 routes: [ nats-route://host_a:6222 @@ -89,4 +103,90 @@ cluster { } ``` -Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use Jetstream. \ No newline at end of file +Add nodes as necessary. Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use Jetstream. + +## Administration + +Once a JetStream cluster is operating interactions with the CLI and with `nats` CLI is the same as before. For these examples, lets assume we have a 5 server cluster, n1-n5 in a cluster named C1. + +### Creating clustered streams + +When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. + +```nohighlight +$ nats str add ORDERS --replicas 3 +.... +Information for Stream ORDERS_4 created 2021-02-05T12:07:34+01:00 +.... +Configuration: +.... + Replicas: 3 + +Cluster Information: + + Name: C1 + Leader: n1-c1 + Replica: n4-c1, current, seen 0.07s ago + Replica: n3-c1, current, seen 0.07s ago + +``` + +Above you can see that the cluster information will be reported in all cases where Stream info is shown such as after add or using `nats stream info`. + +Here we have a stream in the NATS cluster `C1`, its current leader is a node `n1-c1` and it has 2 followers - `n4-c1` and `n3-c1`. + +The `current` indicates that followers are up to date and have all the messages, here both cluster peers were seen very recently. + +The replica count cannot be edited once configured. + +### Forcing leader election + +Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time. + +Moving leadership away from a node does not remove it from the cluster and does not prevent it from becoming a leader again, this is merely a triggered leader election. + +```nohighlight +$ nats stream cluster step-down ORDERS +14:32:17 Requesting leader step down of "n1-c1" in a 3 peer RAFT group +14:32:18 New leader elected "n4-c1" + +Information for Stream ORDERS created 2021-02-05T12:07:34+01:00 +... +Cluster Information: + + Name: c1 + Leader: n4-c1 + Replica: n1-c1, current, seen 0.12s ago + Replica: n3-c1, current, seen 0.12s ago +``` + +### Evicting a peer + +Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down. + +There might be a case though where you know a machine will never return, and you want to signal to JetStream that the machine will not return. This will remove it from the Stream in question and all it's Consumers. + +After the node is removed the cluster will notice that the replica count is not honored anymore and will immediately pick a new node and start replicating data to it. The new node will be selected using the same placement rules as the existing stream. + +```nohighlight +$ nats s cluster peer-remove ORDERS +? Select a Peer n4-c1 +14:38:50 Removing peer "n4-c1" +14:38:50 Requested removal of peer "n4-c1" +``` + +At this point the stream and all consumers will have removed `n4-c1` from the group, they will all start new peer selection and data replication. + +```nohighlight +$ nats stream info ORDERS +.... +Cluster Information: + + Name: c1 + Leader: n3-c1 + Replica: n1-c1, current, seen 0.02s ago + Replica: n2-c1, outdated, seen 0.42s ago +``` + +We can see a new replica was picked, the stream is back to replication level of 3 and `n4-c1` is not active any more in this Stream or any of its Consumers. + From 70a9b61e4ec75a1b51ec908956f0a4b5ec4b88cf Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Fri, 5 Feb 2021 13:13:30 -0700 Subject: [PATCH 08/31] clustering updates and pictures Signed-off-by: Colin Sullivan --- assets/images/consumer-groups.png | Bin 0 -> 16607 bytes assets/images/meta-group.png | Bin 0 -> 19301 bytes assets/images/stream-groups.png | Bin 0 -> 19445 bytes assets/images/streams-and-consumers-75p.png | Bin 0 -> 53849 bytes jetstream/clustering/administration.md | 85 ++++++++++++++++++++ 5 files changed, 85 insertions(+) create mode 100644 assets/images/consumer-groups.png create mode 100644 assets/images/meta-group.png create mode 100644 assets/images/stream-groups.png create mode 100644 assets/images/streams-and-consumers-75p.png create mode 100644 jetstream/clustering/administration.md diff --git a/assets/images/consumer-groups.png b/assets/images/consumer-groups.png new file mode 100644 index 0000000000000000000000000000000000000000..d3691663bd3ce8e4173c5649e95b2f11f7543cea GIT binary patch literal 16607 zcmch9cT|&2)Mr3J0Z~ATqJSbOAiY~F-9PpmHOVvc%-lP3?{97!qu^=^q{Otu5D0`+Nl{i40>RS; ze@!kEf+uImCW;UUp{9qNo`<8iwH?X|!ptXg{*9TJ+t%6LgPBj3nU~kh#f8(-&dkQm z%+a0G$;tyf0_~2@mUh;5R+i^|c)59bIJkK^xcLw~0?d5U+ezc~*=Pux>manX z9gt=!Fb{P-AAOhzpPsC@fR>gZuNe2sLRwuzO%G+q?e2*76&6u3*LPEJw6RdsQBYIkQI^#Nr0Uu$fq%gS zLQ1@RP#0@gb-0qQGQW$vlNQp(ms?MPS4hxF#1(WvTf&4b_@LmMxf{yW+1moGFDRtL zgVOSr)rUISXt_FQxblj)q210%EQjaMps?dQ_aKH2DHoSA<-U!R>C&As`CHR9{mKrR6E4rr~Yp*OXcXoF5baLWR2NDLg7BrW&kww6it%POO^vtCdoy?^bEbVNC zK%0-Shm(%9GWez`$iu7VB zb8jIYOEX2dCEQa(4rZh7T7G^f)X$j zG<90<8&g<*t1{bkbwiG}Mx@+;Oddv9oxPVb~JdnJa9&k51 zc^z{-GhKZiM|&PGn6$T!9*?b~mztWd8lMNZo2#aehOCmdf|nr5LfKrzi`N{jiga{? z+VOip&3)vY7479b^jzF5!BpN>9(Qq9Ia{B)PZnj%XREE@>}w&X?G3e3g*ijjoZV5ft{M&+X0|}Q#E7fD4J>FD7JM$J17DKgsQ7qAnYwg6dlYI+>jn3 z{2KbI2!yq>tAdb*m7bQYGr~gw>Fy?{q+zQ8pj2MpUdvHI)?G3)TBz$;3V5JUPzPC6@PJ3yli%H0-;!TT$4baS zP#z|aTSy%hdqISyn!Ki~4^&=A&_>(SOkZ2q z+gSj-V=ZH;r!M3rgy6T9bwIhh2RQfb7PjV^`sz3Vg(@rQ$)G^1 zf*V}G!HLHT*V9uK%I7I83!oMJ_z!gb3$@_)KgS23944M)4g$FYQIeHLc&Bbm`N0vo ztz28BU)iu^{@hwNUGqN@3tnq-Et=H*F^<*tZ9a(i!wQc&SO}R8JKoVC)S!jmPK;uC zM8Vf}ksvcF?t0X-Uqd^Qu9Qg&!Cf?5(qy+fUtM~}-J3#2`Nt1?$8S=7Y^tT{VC|q_ z0hi{o1zgfl#KBKWnm~;`#oZ9=;c}RFt)p)TanNt&@7t? zyL?wX3(g!$Mz4`i0lgg0f*`sZsz6yNOG$g{mYHBmh0PawS$TOzHa21s8qSVppItIa z%FvM_-Fqx7mv?{le{2sUA*Z4W+glsG$H75rU3F7cRTbJs0ENKwDIV*U=vZ1>nr=+g zT)uiW{DWz*THeRjfD^c$USw|W{py;U%%UO^85tQ!aAxLR4>UTnq=Ym!HkO5j<^6Pv zU*_;i#^@T4!*}-l5qnE(>vsDZ;?t{9ni}~D!7CdZokQ6Q6_z~|wlzqm`}eb&n{S7N zgf#yMGBt%*Sy^$@#tz!@>ot0?^78QwjgEHrVTfj*K$8xQl$A-$&CR*Jej73|GiR2T za^5L3Ffbs=9*Puk|8?j7eZrEG5IX(CyL&0vBSz1|{b;r>`g+6lCI!=bq zcgK@VU18J!Y+LB->yK4h#SHdu71h@ZFDxuv1})N{dLSSmpjTs`G}uqp5l!`QaB%Pr=qw95t62}EvL1Ek zHjm8A#E)0m;ASPKEVP6*NF$TJ{|aB1#!7v&Su!n6OUs$2vt+|2$26`I4I5xLI%6I&vaQE@rty zXO1F_&6VQ%^$RhbfL+;=j0{MMpd$t79|}fEo1QlQoF=BbY^?}&ap89EcDR0&0cZA% zqnC(&_l_wujq z)6;|fVUo)p%4lgZTDB%52U1Y{^yw8K;>1%|l*TKqNGhwU!kh@n$%(bwkxiZ)y?uRQ zPT}uTMcfH#W99#Oh5cA?()~~MMOUEW5D3tz!8ITND*ui0BK-APD{)7?*sr!21iDi z0kZ|!*%x+~zToze2?!444uBQ_3b>aDFI^(PefvMDS{*40b9(Y_D4oF!MngXBPYL$+bsYU$+qK#e%0(r zDgdwQYXG>NBZ7vA=lTr_3JM(MJU}Z8?1ppA{QV7q?zAs2vsizAg&W`#z-y(&#l^d< ztV*B-3Wef-wywghf9qs7+^dqp^o6)1}5?W4m|+G={30J{DU6LaM|{o zVQjPPl9DE_@%nP|@?TeZ=fG;sJ^%B^12tKzx@>(hbjViL+Bz4EOab&v-1GF%wWG5W zC|6r_Y;0#w;sYST&+mYDl2TJ|Gd*|!5TXt20A+rM1HcIfg}B?#jba6Oc@rM(Tu%pU zp1!cOWYHN-wY<4W)5!V=Z0m=jvR0T0j65BzNxA?(zv=$^I12{{RU@l$g&zB5BBIRh z+{;{meCPJ|c2j;@9zntT9$LaT>FDGD*Cj6)o53 zPu&|V#wp{YNAwyR8s&QIP@zsW4GnUju8uPbr(oF+Ro&hBs*xPj9LUfoPw=HD6Oxl7 z2uP^!Ffu+7e04t%$o=VneU15Wt|~ZdyS|qgxPs$3xcPVNEg>=nhWLz(I~`qJUteo7 z-C;o}y}xqxD&xb4Bz-SALgceQyyP%&`StncpZ)zEH|*QD^m^>n0737uvtQ{;=8EXC z%(8f=Bn5buuCA-QCnk1_HkQzRX&_83PsZC@*mH9_7U00Qk&(=r8Y((Ex*NA|U*sn5 z1>bBP9pwT7BzAUop1pY!y|v}C>>(N&8Y)#Z(r(E+3l1jyHkm90&e{N=1UEE%ia9xu zDH{cFeFq$}0234dr0SFz$Nv0j`Z-;aAu}^`tlaFv6~YTwZwm*{&6yV_QVapK{h!R2 z|4VD<|5Nwpme87t3UXj1`R%VuTUs(@Wo4DS%tAk>2wjtwmey~ zAn3IPN&q-A6-*iW_KoR}!5!I(7yN;8w+g{|b@S#;J%IjCLPE^AZ)6Ow;LsXaCs_pr zL;!J@S6ABy2gx9&wzijzjErm@9Aqpk?hz9cn4FBsR){z6NnjqduTfM{k@fKr0YL22fPMcS(%ah$1Uqxcw%lVy6BuP#Wo7Zn zC)cwq`2!zBG9nC+NV4nKLmJ$d7=a-JjyXJ^sfh{XjaYeL#a!ay0iKp4>8CXc#s~a? z8$`5)b_+QkEWGM%Wl6An189#p+6%Wr$C z#*vZPXx&c>^#0Nga{%w7M{(eI28Jat!y7gzoz$HKo1vh`7#L_5S!vByvD?1x= zM1hz0A2!$JZ86%m>1~62>vOZ~tg$8IKg}-aX?R!J!6`-IKbDuI69=uUtLvwp9`eSU zp<7^XXYXqxA0w{y?K=xA^z`%yyDi{Om{1}9z<_QC9Q(bcgnU#j#_A@<Y0O9Y_XK(jy9`1=C+}+)oYdreX0rKa!!kEIkHkJKj zP10qwb0|7r7H4q_3Wl!?=K*wu3TJPhP?*e*=1C5Lqi1MS=eq6vM568OZE$uyV;wp~ zcXc1`i6WeVksb2k=WtA45`YwUciiF7!9m;h`$#mf<*IfKlB1E-zO&uD0ssdvf}lUL zw%6X4&5str9qsHs0NH5?)L_=Dwtb$j?$6XKH}7TN`qZQhDgbPG>X^d* zIDW|HLc2D<`QL1#kFa-k=K3{|wl)&Os`}S*+ofo~MBDbmM=Vg{>Il#PAi+KJG@4o9 zSOi$93|?-}+U`rMn;)Hr8&z5=y&u?n*_a~aYz~Yvzz`XAtjX_Yx;FZ-!V!BpZ}0kU zt}IJ{lp7lxxK`RW<1On`U&=zDxIn4?EEAw&Nv~xeIFU@mkc=0B&o0@@=O33p@p;sG zW$&Bw`JrpQEIbxoj~@IuH~M@~%y*}^(Wh+Bh2}@I;YR1SpM)mzr`uC?ExDRb-*-6aFo9b`D{e}t(Z!>=&>}cNReTBgj zT#1r*OS75gDb#Y3#xN&tS~&`Qy2rIikpvR4b0W@NTuX3A6e`rXY{`HAgM!KYM4Tf^ z56SnH#~<_Xso*yeqmQF)PXz9zVa6TAYR$HWM(JC~)g^wf%MC>=jh8p4deL2@ZSAh} z3UXTaa6ck-u#~` zi;o{s89FWGu@7hl8m?${ul0nvuYcYHBm|~Z#KnnQT-ZY(iGCZlZ1Sqmm5J$YR`HEy zUHjtxwgTui@bt#GHp9p`#c#HCyv>_WcF+GkUUlhYe*=tMx4&+FVPs_ePgAe*;aN;|5c30w8L6Mqly^;?&y_xDLkPgiGXiOi^vGnxB5yxk{) zK#S77dGjXsRK)tFeZOKINpJhL=?GX(`;d^=-1dmQN1%RGh*H$pQP&!$O|eRB&FrA` zfO9yB^-Euo%|;`5pvRw0cmZOpZSBtQ@8VMPHBH`PRzkI1_Zw3ycm=-iGU%!wy#Zif z-A_wSIl8T!`j@>`5E~f!{#|h8%Tv;8ReUPl@fZ?gAA$;Gtd{rwO#QFgEAM~AlufJx zk@pN=Ijq25Ys22V^1MtW%*Np3y*ew%FJr9{9pIY-zoEC!?@q@p!!9n@{=W<@^`}IX zmX<07g&!@O?8#RfN$ndp5huaI+{g7(Tg%Nkb8CBbiGq)ogV;(sIPYwZJPL*7K&PYd z)lBqwXtWH{3NH>04Lv^EUE1BDzr0PDysd1w>v8IKbp#knS4c@!kNK`e69a|H{=Jdr zIqQ8;#hZ^{`$qXjf_S8{&MOOA)FrfZD6&$KJ9kxu)%t~{VI@iTD#>W3W`RrQyRX@B zj-L99TG2OEM-n%igoTA8fdykcec|r(Pu{>NhqrMi1DTD!d#kKL;Y8ZqzHEG25-CY5 z8u`5FkxOy}sWXF?ui)t7JYK#L^|%xXV(r{gq?W5q@93|cC56Z~ICeMoK~Nyx>jGYZ zksEu+JZ#IC{wKjw1Blme?BCt*ZOBDp1%O|)Fr+f#7noAui_n_7F;Dp{Z^jQ7ieOk* zl=ebXwYK<^Z!*zN1)gkvd1LQ{EYOmo7fZbW*08#6Kl{k~)@tGA&l1O`AAu+80@{qs z2(7)hGr-FVJi)pLaKz`b-CEat+>OCqVyy%2Z;IHRv@GlWdtg|4mR^$BPsx@gu&dV>3_d5)LH0~tK>}wGO>sMKu znVJUgeu)PPn*;G&0_&9TchYa-c@~tI_L_IJM52jXQ2c2eSKF&Uhg6T+zmKT!36)Uh zyiV459drSFjK5}W+6}7R9l)xNe2)rhn|ivwVQuLgbTdDhxOCPe=tL}jLM~F$TQrbp zaCkW6*)y{E!0v3gcF_n3pzSUbKz=PRGr|={dJVi=8;biW+LCSPX41 z($Lq%3;_&$MD=rL!b$%1%Feql*}PTLsEC_u3eG9yb$36k)snZd4}>mx?b?V$n7Kc>dnSTQ6uwKiJ=D)OJl7wA_97elKm3XXlWoY{PZoup+S1)QXzs zg^jLgfXA};*1er;Z*vkLb?)eFb+iAO~5 z*)J~_m$qB|swlFR2y8|t)(MWAJLF6@Tw0?c5Q4uoT5+?<*sfU0FKdpu?o6ej6nyda z%)IVE^VE3XJe(kS)FddaPNoBuT%$%^{24Xk={%Aic!)RZI zT*Di$JT4xz%Bcqolv(G^z-VA>)*axeBu<~#E#2AOeL&ZixXp$*`hkub4;h)7*HkI@ zoN0_AE}2-FJfpF&D1SR(6?A4_NiH)J(5E~ZoNV&QDsSx4D&j-qd5k2e_e2X2NW1Yy zP@r^vl&TC$1pz*G1n@P25$f5^ck95_6q>3CUzce7Z_nS zHu9qGADCA0Hn@7;IvdW&N(k!n11_mStoUQjVR#E=`^o9czrQm1k~llVIlfj%T7$H%@PI-ij2 zx2=y~z6a#@hs5^tDT+}b*B|CjgkM5*iuK3`Wk$eThh0mDNx=T)a?T`HDTe->=OGPx z0YPEB*Y2}r@8u5`x{jCAr;TXSV_F@Fexfn*A9V8S*@&*}VVM*SBL8d)1m-Y-P}aeA zpF=*rl&`NS@HKz!2Apc(mkR*hcFH{qOREvIdS`*M4Bg!f(%^P&?yU$2+EqTOcVj0? z^)9-+m*iEzB-PwLdYFU{k|Uo#t4we9`GGy&p6@27p;;Q1IMoJR%pJK`)6irwn$xjW z+asJ~WrF+w8mUGF^=)WDpg{a)Nf_pJ*FkXJb!S1T?Z+fnCNV@rL6; zlfKDDbG|&9AqqBqk7>AmoHY;GcJ#ERrMle>Cc~}8&$UA;zU-mpIw^wF8-);;g-_(^ z_WWm@cw=scbI2yi{zZz>PoEU`?_JQscc~$hQbbVP@!vw9MnB&;+@om=XXDm5HLfrx zI6gU%IGd7s0a;h6?5N4hyHwVCTxg_mGU;cO#fB%Gl>pgJ;+of0F^n|{+6Jsj$;e!= z{_-~V>sQVAK(1%Z5CfbLPt$gDx=J*8T?@n%nTSYuZl4~JXuO2n1_*Z8KDqsnnmUHy z?%lh1e(OAUGJ7wRkc0qsUR}C2b*b{AsRPRLV%@N0xxw7=ZsG$q$p_ylg6{#*H1XjJ z4C5CSjRx_tUcr(PUxJ=-@PtNgY#$ z#DN9!pItemBIXM`nvMyH365a9_ir3A@yswo86qbq2g3ZGlR>ekhZjmWRa|{gF$mcC zs~^y(gl~$rZ`nQxV}tMu3CR?Vm=9&ifgGI6iTh&~yoJR@vky-QA|oSR8l*R3DtMZh zOHG(>@%+JfLEeGomIWF?QaUe0mP_0An8`Muw^-+)pyNa+FnCc@jZ{L;6X{P^%P8Le zcX3EJ(Zt-QU(`(pryxHqDe(VeEGD9I-aWup6M>{zeo>K(gTq5nQBiJT z;mf9TlJ*J;mq6x11EdasLHRB>mr-2&wrJW_u)iQU#{F8w`*15$eZw7yAvs-0B9VT= z!@qJg8!~@5vf^?1+O_WAQ%!O_ff%Lg%+>tX<|_aXd;dHKSqYFdyi7{kd3v-9QXEfl zDOF+N8@QOKtu2cnY77$cA^5_js~Nxw9JQ9Tr}?VQAk}6~hM-j0T?3qZEezhp!SH2SnQj#(aN!9b{?;Vml#?s(uKi>c zRfJRAhD208{tgS?813B6)EPck5S|8k>(Bbjt<2%XcbUP4KqbTKkjQ51o9lmtOFh}K%2E~R4zVY2{U zK7}pFXvYBLy;>Ne9-H=b3gs zTgye$KG!H<{Tx^}b&QaM*PmHZ6+*zGusTK+;g0jnPn?KOWBg^*=2bOT|+R#Zq=>|*=0M?;S2^nBu2tgQI66Eyb z_SnnZr}v^oYCEaGo^*!d9xC*lFW91{NM>3j3I~WFmW?C5h$W93MXW<2`-nlJW z+bO|D^eKcM+sVdc`Jd- znIjOC)SQNqKD)o}f(&<&ZkaqBP6o{Jh%gcwT!GEnXqf^MN&DHF55(%$f7=+~my>=2 z*7xpLFJ8I?Qf5+l!)7zB0W9q7vRYc# zCnhF9dOZx()j-~ZL{MX6S2ZOSR+Y-K%$}ibLafO_@IU`81^9TzsyBmNw#oPiX ziJ0E2hki{M$-eCv5+7C~kX+9wF1{k4 zO%7t}xNNw!_5I`HV{hL}y&22!MI^qhot?Cb$`#y^4l+9+9eI=^@%zb0H7*2^4jgQG ztr}iyG7nUr^6z64?7Vbd!ux}RBR^P1Z7nqz)&19J-0MgKJ1ik5C0*hB1?FH-IwDh| zW3fHg$->EbRnZR>BH(wD_et%@!@Tym?k!rqb0+Xx_bHMs})P0=KyIA^i6a?*W zF8!BEAuSc`?#nru5hJ2I|K_pqoD(g|CkBu)k`fodLEoPubg9`0x&qXU(^b2-g5&Le z8*q*5(V0XH8|({zUPyt>073jY*CtS5L^+BT&pCvn+3^3_u11E6=gR6jUYp1{KfGk5YPf4l#P)!mJ+NFbUTx{nH{hB4Ez0# zh=~63ukzat#clzq(GMHAT#c&2+s=fvl`@t?q7?q@OdoyD<_#{{-gX6*Q_>|myI-vv zGT;R8z3C?4kZi|SPl2>`&y2g9*>M$S9s$JuBS`4-SnqcB11Z5}9#cipu#A_41BqRe z>n5RYdUHrEt&rDzHgoIaRn600l?lO)N-Oq0E$U@jIy&JX9F^HRU3c;+Y)(c1lu@l= zN3oO}#*l!3fWxvNy{Gr080L?c>Ri3CdU-(&?`EY!oPvN^|nS-1hFoU5m zy3=unFXxr(>^CZq`gvG{ZO9yh_xJX;$I*$61+^Y*D-CTBaEtC-S>{!$3ghLRMIfW5+PH+Y8LcMMsrW1i-T~9z0!tDK@T?w*w5ck$oO6}9Z%Ie<5=4#T z7-a-N*59CZEa=;&LsfL)#Vrss2%6FT?D-w!SGF@|{rvq7F6FIBe>|v9^};ODAQ|2& zXr6lF>OC-m`0_9S>#IR$-^Zt8o4p>Tc+~p1C5KJnZMfQSHO*Yyll;QMi2wnsb+_<4 zxfH=0glr&FPkbAEn$e(yPspU;XV4&C8QhmoBIKX5pI?#n@OV*< z%^*Tx`0VTfO*GNSn>Vz@$Qt{v>Etp*8C5MQ8$aUsLO&m<9`2O|k|x0jKvdkP4WpaN zyC{NQ|C?tKoouQ&PpZ}j^n=yTZy9@<*u-9WSzF3q*|6b{WKg~CF4+SweYjtRPw$e3 z*ksd?M`!PMy$d7Q1$mxKQ2;os3B~~+0Y#<($+LLTZ48SFweXIYdf1-fw z4q<^H8eMA9Js|@b2A>DS!U1OQ4OJXt^k>puJ$o82$=&qB7#>%JFo4oc&e>siX{>8S z_3MwDu*^(MELp8=CvfcoalaVoONc$&@Am+$mAaUl2J{1FMG3{{u5#>Q^ss;AQr@-Sc2kFZCfiglYXm$mUtp(7Ptk*D@Jsx;pI1rWUJ{_(m2n-@`yMT^ek174y@wkRW$M z{Gg5UM2_G*%Z*2;`}qDOyd~g7@CspwZreiy(fcWtqyjrR`>a$4j3OWHkW=v4`oGuekHt68srvNcaq-U657_Was4bn^fO}PWj?Q2W8afE(PB4X47R&lee>w)Icq=bKf8 z=jr~)Hrn{4EvP-JjdHwSZ{c_Yi#Rl%>s7$cCl)zl9Y4Mx&p*2aiJ|65*oZ2ryfX6Y z)eTU61V%=H?A@zhdZU#sAP<#MdVDp{_dC!4qgFL(X}me}`HfzDSXv2hV<`fN{$XWD zQv6x>gUKa?QCLBV)P9?4ir~>22MNscZ#@gBM>+Oq$W(^pQZ7+(7;<$}ar_2WlZA`R zGY*nZc5+E|x8off{O0b72OYoI-P|3)9<~?fzhP6k3H>A4@Z2b$9@sLVh`srTa(*7< zl{UxU91)?L*S|TUW^vZ{Y3G#!F&LYMV$4AjS9k5!IQGz`>{n)Ycg|&?_E68vMo_W5 zZ|Gs7s)K*y7JZjuomWJzsui#pn#SdvkgQ8LWn>m5YGr5wTmpMPcF0zd9SPl^`clb@ zB|NE;V@O@{5GRCyqXoxz;wcm;?5f#h8c?C#rQsKC8@7owt~nx}^= zF228LANC#lKoOTFmyG7o1ysTKxIk8k5PVeeUHHZJrS#A8{4|KmJWEU4==QoB5T|Td zG$+_-IJX*q_JCR>y%iMz~t@;(f`|Z`5XB!{@<%^x3?9*?^0r z`>!kfw6P3G4oTpzYEadYgWw2=KjXsCTXO^_o0uVNNAPjF4ep!B|zl_cTxL{&`pOZShd*5l`y8-8tr@nQBB=33{XPN%H+ z=I;Fv;Ph@+w=o2toQ)1M#*fH+!`44n%@-6DgqDq-_@{$Hh3uUg7SF-*vaI|IfgD8xzFnaUwHjEd7a=Sf z8$Q`fac|#<@2LZoZc3?M z8-bo53SF;iH~7E15Ioi7Wr01zfU3-d)Og47xb#48A5>pR){`iV!9149^kJu}6GmMy zllJD#sP=ZK@#YGzUQom7k+62g;?mNtrl>}xq=K#MUCloF$*=knuBbkc#T$Dx43cU1 zb(*OQ-?5h03qeQ=6z(_Aj2e~qDj6B2jZQKp5P^q#*rW9NR`(_l@mT@U`;IfdejTH8 zVmZ%idjlKlEMf(o#cL5jLF5c7BZDhO9^Y>}N&nSQ(mjkCdHv%!9R)>CH~WC1w4F(e zV~BQScfsj@R{bXY8pDF*7%@%IIAl@~VOP7Va?TLAX#o6}wpPOH#g0dLLw@VzA!G|JnX zQN>ZzXXMsUzMtQN;+reDkHYE5+LHqTyMlz*W}h5$$!tNiO>f^BOW?>%{o~}2$!TMJ z@mRzrrD(p7%W|fyKH=})9v9_`9xgk0wgiRgkUGeVhL;6}4TX#5IUFAAmRrATJB2!Q z*9J4VZ;&>OREPd7#Nw)h&NG#3l(k8ObgdIFS=ItOlm0H}2*b?Tq*I3Go;?3sL*b>$ zVKHNM&QuGV8>616>zL0_aBqS5L*khbNH`Z>7wl^VT-D*Y8u?ody|DN8bI2qp?P*Gt z^VjtunhA0p-&lWg6Y{ncLw}3@>TwKMON(WtRV{69ZogSe8lj^8cn( zg<6H97?i5im08`&1<(P~#XI|t>J?R0J0D-Tgn#_TO|wRX8Z->Z*$x^T3a@nUNbNr? zc=mTcb{dpNK+Tibb(d4`+Mh|fWJM#w=$*4$qdx)-Eq*TwkKO)=$QPJ%n`CAge3h2} zs%S@`rO$;F(Uw5_SIzB#lkJ?s%gr!^lZUlW5~HQK0z^Yk>O3y$S1|>Tc^F21gcXog zA*2FN^oeJ8x^CoQU&hU@ZU!=IyR8LP5@|zT2{=#_5)%5iUaK-Pn3@#Z5Z#{o(hDko z;99o(`MaZ_{t8gv%2`oQs`YDj>$r`+1L%FQk(ZmK{CWbGdT{i`yf$_;MD5yB<-_Og6su2U6!r4U^T90wGMXBRag`um6tPZiIcQxiG_meuPaaSk^%0aj zUH}`}A&zHyt5D^Y=m-Myt?Zp*97bISM2P>XidR2)wQewxzUJU6?kM!cX23H2fgy1J z4%hX<`{E==>aeBOrH_yIv0bsZ^jiIWq>PmZZnt6D^5d&XwB$36lMtV3usxiJBQ`w# z+iwnqm-MieEx7U+K+~s!-vT4?LUg8ua_zVME zUzosJ(i8%}^c9~?+?Rvbs6~>|z7#5Lm3l(+TG~hn+eIgHxsks3BdtBjyEsMqJ0IRp zP@BKBNG9bOyWe%APV90+6Al#$P2L*)`SUJVsu=U-%NGR{>SuaX(k<(awvvM5bT|g4 zvhV|Id9%=e!

zmji3hI^?Hl)nP?(=*HowW`dmlF0$Md@Apg_7r35iS9tbd37|&m zNjkgCzd7slO+FkJLAYzU2TqK0nPS{#`m(YZWo2bSg{Xc?A-1OiAjZMjgGu3&zcW%9n`qEB2m6gtuLZjTY9i&;)UG-o zqzEf7|9n?uO?#D&axeK3xNIZ?g+3YVhhHqks9(iIKV<$RsUN`xQArUbou8iv=Ewf4 zr~g@&5fezr12SSa`K$p|a=-;qsxfiRd2#mO1m54*C!?vE0Immwg@#^y)eo*)WwLz3 z8f(I^z%ju;b*yHza{EyaWy{C17GYT#5qM{{1TPpxoXYMCe8avN#!_C#6&C%MD+b<@ zl+V4xkJH_b)6Jn&uk<=Uj9TbulHa`9(+92^h^+?oX5@pag#Xob1HpW3Px>6F2#8~l zj05+jGE-)C!L2Z1pPhH0l9dyVO%wuQL|W4wD;n&+YR zLfmn>I6PnU)4vxH2L8E-AXl8DL$Mn4RUva#9^}?QeY$CdUZ_)a8W0XpH9qc~p6l@q z`z{8Grs(hQ2N^I>bxRJa$}c)98ynM8v+D&%Mqbu;-cmNsI(wi>S$G@V00f1%L?1qU z7;8j71oxe8J{xUuohJlWUdX}m4hl8K->%5qSHP$vSr8H+=@waB%nnT6DcEi zVnc(IlYM;{KVM%!3hpLDc{w+(u;qTD(#!5ciZ2@2L_%V1uA2Yjn+ZHDAiXy@F%b>2 z@gTW*S3rPjc6Junc;Nm9h*gI5ShfSuAqb3+{hW`Dq6uSD|9`!lphcbz$6mkoe{t>d gf4GRSjJ@zIf_kT2F-8jf#|022IW^f*sOgjc23u=O-2eap literal 0 HcmV?d00001 diff --git a/assets/images/meta-group.png b/assets/images/meta-group.png new file mode 100644 index 0000000000000000000000000000000000000000..1e802be26efb0851026ee1112c9d2ee17c73bf89 GIT binary patch literal 19301 zcmbrm1yq#X`!A_4*e0@7WA)F@JdG((4gfHX+w ze0%i$z32O%^PRJPYn`plH;O*jX>)>pI!NM>1_a_!U9(y-0Zx()e7Ct@;cXuvp2Maq- z3l}dgR~v6|3EX#avv#m`u(AHzhL4Ajmy?H&lZPL|E6Bnx!@~>y5#ZzE;T1Ca+up*? z#`T{L1-W=Y2M;tXtQ=gu-Ms7{{%zrFyk%KRE_-Cv-TKXPZo{BC| zMP)l{SzB2(PbH@454li z4Fv%wMLu3%VL@3xXFgp^du1af10H`ZsFPNJt+lADs=oqwX+;?yK36_1@VFKFlasrE zijjzXfVHEhfVQ88jF20+2yW^*@CyejsmeR>!C?Bj{sR8ah8C`F8sMg%y`8rqKaZxi zsxnl|!OdOAU*Ai}$;iOd+g45=0=0DH(-QPFu(k~F1hXW|E5mE+U}R~}XQOWq=10-Z zQAAbG7N+ZB=WffVE~f~_APX+L=vithI;;C@Yik)fz*GYR`2;D)jT|9 zA>P7HdWLT9wvJA2yu#X=iWU$#KbVE6zN?IeoszG%vLCNSpk|I)lFB!Ud4mo$IHjaPS#IeQC3OKNkEHV-9pYMP+3sX%05sj&|1UO&r@4f*Fnfo zTSmaqj?YR<-cX0vQ^{W)qGF?^i)jO=0Rioyo!a?V=*GQu7j3R)JP z8uE^wGV-!6a?1SHc7lq$)+#DO!geCo_WqVW0agM2wlbpDj;eN`3kOYCZw+60K6@8+ z4Ivfq-I{j%wq8P_+CF*~;FGqFpobctC!B^l%CAc2c%c zc9B(aa5c2G77*~TR#5i$cTw=yv=Q(Nuny3)HB_|JxAU~I;ZyMN0Gk9o`4BrdKNWs$ zJ5_#5H*duNMFT&5cW_xjLq}LcL0DK#2fdI40_F9*WWD+P0$^HvD)L%hHeN4 z5CIuJZ6&CHx4fUVuA+jMuc|0|b=&(}*o&&E^2r!@_*?3^2%@bcWU1;J0D86%wBR%H zRns@pbr$6ZpY{C(6=1r`dOmJ`F8<0ohF%t$E&{#|JdRp2wmOFH_F%I8EYTao#zWpK zKs6BjD9fuPU=?8RB=4=F6QHYQ1%;|;@+is(dVp58x>hjhBm} zGcS4w`VKzMf6UBl3x(N31vULWyYLMZo zP)j3Un1>Y<_&7yX5e=w_dLZb~2HkmpyNaltf`W^Uk+Zy}kAjwhuBW5DA-Lh}Z>TJ< z1Qxu7D%6P2&QOR)*cTe0i=IO(LsfNQTYYa}7Fiwl06k$DsIVNA#}}rc#Sc@E6;#mi zcaZhb(K3|b*S9mY1ii`H%3DDNb^U=qS9B2-6@qyRY6}Q?0ne%E%&(;F=%?t1cBtq- za^Tv3{OBL2ihktJkpX;cgRnyk1_mRBio6WO|J_zgh&H+1N0b^=H%S_cRC3jC-Dpm) zV0|?9A-fOUr}(wrHv`s(8*}p){D(Eqt`BkkjN=!kRiGY5oa>oA`YnD>niI zvq5HvnVOW&NMW+)JiYnex}y9tT^K&i<(#z8>^45l1o!Nv_pw=4arw7L`x1D^M&_CD zGealrr;IqyUd1ww-5|n}p}a)`ea47G#iYXmE)xIOA7n`(FxWj-)|*X!JDpQ?PF3*B zKn~-Nx7Ct4WZd0(^vg{mfBbmNVOWjTo6Pz3>~N#XbyjS1vgW3!sAyqPk;NrD5f%pc zHewA84P|xp_L~gBPhPzul2=gpV$EkU-x|{0+bgG`fuEM1K7>SuFDzJKeBRyVX>4p9 zoSBJPUw8buvSPV2*TR#k{xT+}y)7J%n3_83!v|&oK|$Ut{K!Y)@=4mUt*uhb%*;Q^ z>gQzr{Y5@HPd=-$8?dyuS5Dr$dGn@@fx*j$29dDv@P)1Vc|~1alC$%38>%cuiKpP< zcV}mT&J&f_qA56KgM%e{dV8U%^`)h^Jv==8x9Y|#Exz6m5D+Nj<0hVXRY(%01BHeU z4r(`_uB7v%_o=&l>+Sved%3sDwvT!$l!XjEoUmWN?7<6WS605Nd&u=pDuh}`M`xne zfz(M4=ecZ-YPYJQ;*Dp|p1E&M@Q&qwS1~f8HVa&1v9`6X77FK~WG)VKE;XvdHLiEH zaf#?s*VMc#B}Jc|on5NIbQ>#@Z9zCMFVCRz@wL|TZSk@E6L|{@CJHW-mr+r;WBEik zp2#QdM#se5Zbn_5Q&Le`xI~%XCLmaBIi8dA^%Vj`$dJoXJ^7w3_hNU$9FVdU)k@2jg{*HMLOGU~8MSblki-6$=z%#MkP>3@iXp~r+p0<|>L zFEh%UetzlcvGSd&)T{-JI*b53u)=)ro*b;GK()<-cB_J99~k$$iZ;=wmoH~syL!11 z-@(Y2wA+blNa*OwP960e6VuY7-oB*-!={dqPjXj-LdkP-atzY5Rn+g3uCK2fq|4Zo z0V8k%bEmQMl#&15-zTQ0M_V8lSb)El~GD6vyQ5f zQAmj5;lqdM2hb*KGj?kU|NYx@%=Xu_bYZu}48J)GmmAr<%8*-;`T6(d<>f1d0+nUlp)#ODh@p=0B~a zPJUp%lMG6(qq46$TAKJI=HAS+;j?K%XH5~%F6!|6?(bh^kD zwmW+l5-nKGpC}Adwx0-(FYX_Zr0lS!mAoxCS*yk3r6Yk}$Iv!4jRkXUyKl@uxb#>* zl{pzSe!ty{&pDgOU*32wWNq4bZ#Iahm7=`-XJ;Y))bv4m=3*?2$H&{V6ZLKk ziJwTnC%t~(|4}SstGLN6F-&mCq@(@Hw8_^3b#Rb2jtW!FSGsvUo*?Mj1dUcL@qUS5XW zPk05U2m=E+NI5UBnqsfGsm)>Bk<+SB8F!q!V<-yL1ve_H^XAe6Hm9%8eh`%!S*1ZC z8{5+#sz&FGUF_{Y1OEceNF3z)^Sa+&v;10L*Y(ld8$y8@xbxHm41P&;b&n8rqIJTy z&mq(q5;5Ny{Hz8pl&QhRIq?*56d3ROl7M}f zTh%3=dWBZ(DnlIdRPC>!SI)E~l5Q>8PyuvpXAS%V3Zoy6MXvgsd!+6E4ODl z+D8$puTMdD#mf9=%$brwm3}*B_r0PGq2j}wQp;C6%Jol{u$kYor%oWeRyAow+@E}# z2|GehTVv9++pIZ^Jm2n|2 z5n{$KzvU>+i2^4{|KN7q!{1FJ zBh%B-9np8#r3kYb(B1(YOJo!jJ$%Vc$U^@PdM;TGZbsjA)XrIB`?8&J>7_CcjOnQI z{0dU_g3j$U!(26QV0dKxu|8FpYKf&W=IQAv4k01T+F3RW7@tH%Mdf95^nTDgu=$hE zUhwN@4|T6hsG4%sf*g} zc&q2tPS2G##Id`w^`aWrSzZ0MuAZI^P5huVIOXK^^b$8q$}_+4lHrt19Qs5~Coy;7 zamBComq7_^llmwJL@hoW8e|~2qBfk_XTL|sR=eN>?@zt8wN)LXK#2()b$Iis7cT&pMgZ#jc!T=iL~zZ75hq z^n-48?G}931^djjKMQ!lxGjtVr4y}q3N>lr%RUtAw{fCgya(kWPxe{px|=6nf^GCt z+!Nc!FOaN??tLx9(p4D}X*dZYso~*aiB`nrmnU7v{XVr0jg?WyQCqO~7X7_E{05Ev`7InAgs@Q%YA3AZXY+kkZ7H^tyxHdne91n-F5#HeXz6R z?d|Q=z(aqniCn$)zys4n=rBHnTUQ>Fe;a-T#sIAgUhoXu+p>47%3Lj%2 zQ?l#hhZ1o6Yc8o-pRh$}#JvKxy|p3Bc$XCs0uVZz^~qs@)2+SvyT1Lh2deN5yzw{; zyG2UMQ#$mDHY?O;cFQ6ZV|e@>HuUE-9=>ud7swj+;4=nbg>k8PiUTX+$faZN4pJ?F%8V4D3w z?20`Xv?G~PQRhRz_z*$8bYusQb;%4okPgX1R(J$k1Y454@a!qjwRf{{4qm~l1z{YI zLzMC_Qm`Yfz;S6KZM%H&Ykq9)bnTjGRjfGrEXJO0*iida=QQ5)+EfvX0PKI*u0*`c z^v7>vwo2oS-xZu|5@+c4qN1baK!lFzgd+*uq`&v;Cp@%@;>ket<6PT z2y8VzmFr~?V%&+r-y6|qi`nJjWveQ=&5(raQjk@Ky~f7N0E}H~ZD+T%oMN6??>DmS zsg#~{;PpFly)42Nn69FKtUt)j)059EV3|DL^K?g=I$8gVY@W~DdE{phCu@g+n7Py( zK@W}ySH!?76daZ_jgtIX^RUdgNu9X^`+k>jy#h^;(gm2dR&-3oTdl6|Zc=eVBR_$6 z*!5O#{n*z(ibOI=NYJ`9@7?h9^2)1-kI@AEw9^sKEL2Ip~~h-2ce4iyKq(h6?9dpL1}v38;)MC?onh2+9KACj(ZPJ<& zw$>Y45tkfBEag=PMfEHEx(VLq=a)p{;J{i+K^HB9^c)<*%$UcL&q${BEzjlcGvGe``@TnHc?ad*AW}mNd z9~C05;^LgJ#h8n4D5$if7N+S4WMiuGG3J-`|wW{-l~$klZz$g=N()_G-Kk4Si>DxTGe+Ba`J z=sO-qB$WuQ4vh#ZOX}sGJj8hB9wCo>GPC`wV@$p)`)ZV5uu`kqylC0f^+p$7gZ~DX zlODuo=E`}@s5Xq3Nk@<}i8DU$o>b`hr^q@{67l@?*S^R7rs+uK3D&HI?jqIT*TMK> zN|G*dyE10^0FMja&6JuA`^FWZM8lxR&9erT|o{YJ)@oVejt;d;!|1dmPe08(%; zfn?0GO#A2KQQPsbA9}vz*(3L;}Lz7J?uR@@4VNXGQ(JvA-&`~PLsO*{c__t-i3b=||pY&-Yc(sR}*-S}NX5-9%b zeXm4_!@30q)aNeFJz!#{&BTEOcsT(WsahNPD0S_05yj)wYE?=a5@_t>bTDsPbTVhA zzXfZJZW^RpE2nKAu5mS~;@u((TYXjN7zTpkbHt}e=?|t|U+DvX!f;nN*(7Nj7-+KE zv-s5#F7@sK@P&{`2T9cO-0BwTa)U|X?A?Spu9^TFC9E5FC0P;TUdV; z;*FGs2jQjkY*z`{Xnxpa%YdA)Ic+M=ys4zXDJ$(`S^?vN{&Hbn|$}u zBx7Y*f3*#SyTGcujx%Y#s4KTXrYwS1K>e#AD`?`pQw;XWOrX*$6QO01z#!=6;)(7 z-krMw0%Ra-v3JxBl`eZ^pM(Y7IFFmtou0#tyR+LBV+`;!fhWXE<9Y%t#UUXf%F4>0 z(OApwuDkPOHMZSAX2;;*D~=^X0)m{%N+N*&;o#wYopwXS0=V(u__&wv%2^`l$Rto8 zPWE{p6p;Z#I{Nw%0D^2jnsS<`w7{gM ze&f({w0Y{qybQAd+d^QL-!*;ZWB;{-+k*)_byRhe9>>`dZOeFy@#bz8Y{gJ06bB!_ z10YYgY9_xAll@#?wmAL0Y*1qxd-q)UW2DN=hVAM}7%vrAYgcZri-|v-e))?OuFpIr zL(F|vhveqw5|fjMx3$UY-)~qxSQ`OgVOzdNxT)YMd9Ofq?`NvPKXM=3Z&9taV3 z>|7MG6_@CjwT+b-Bi{{k-)ChdW?+a1X=!|@t-tqmX`{mX`DP7}6?@GS;S6h7U-*Mk zgcfJ#=kA8hrJJ%BJx|$kjEsz;u<rxqd?>@cdp(k`3%~sM!H5oL zK>*Zr313~Z_$Q$PiEo4wE9B!Ve9I9rNkrH*|KSVm0mO4%EYM42PEO#jL8gndafH4n zE;2GE25VrS%iO7sKzULg#ABw2gWZVoOyZ)e5(=^Wv#67a;MNb%fycV5y$Ew9BSpS> zL*_i+YFf0ATJ16|^r4_Y4Q0OR#$cZz7g{@m-G$fPT)1PDMHrNo^ReZ{nS&^?@*cZz z)JsQIRa_yrxzhH*G>fIqSY8_AX8XaQtMdKTdv;eSAg)q2&RaJOZ(!N$xY@8podP(kqC zy|3;;Jp05{R7E{2^<$K@R9zj$etTX;Rk&X%6BdSUt%WZFsNuGxPpHxJMHsy`DROvN z7mX-;;uomg-yGO&Nq@$~u3!4d#)btm8mEmPWDsgymq(e+qD$VYjO{zl-E@fR$n|H~ zc;2C#mRulA+ye*q^Cj$eXH8BO??DjVe70T^!*TDY^~{y}_^Rt_PUysaI)<82Y46j_ zp(Q`ApK<1f@;=C7XrmVzrLS$mNM6Qwacm?OM}qI5 z`59%7xKaRHOVGT`R%nG(*{&7g(Y3@;G&%}tZ*MDo!W(%BTXkU7#i>yJsq3tbww)4|8bN18k}azg_tmL+8XjShS9 zf}KxD?VWf&b%0IHY6CWFYlWRl1bE)fZ%6OqsxX&Q8)W;P$yHtRCxOW)}omtKz*o*VS;QZg@0Uw<3;lPu#Kl7D;g_)DUNv(~2@ad)&CYx%0x z#ZO2PQjzNs_%yGwvUWzoj2QPV?}31%Kq)$};#QJA$TDi%j0{)MPP>aCl%vak=W?4##bCh9$?kEg?=CC6g#rm^9S)<~?t*W%p9}lqvxwts<|7UK~+?*m3@qUAQRD7O(lmT&O_IJd63)<=^LQ zMtK^+vE~&Df=o=;K*nyS^?@A3vn(>p{?|hLIly*2VP0c15?|h8pr@gU27A}A%eoI3 zdF(zB{D;Z~bpT?9=1&KQfMe1vtEeYoeoGfSXIUUGn0)6Cl+gIoZ6|1dMWcDYU%WbA zji@!}_ykI7zBJCR&!oZKWrCt$#(0q+C z-djEo$^)KoWZe1TtGKh*d)<5yJg)(Gvt&-==CnsDAPTHCI`{Z8%w$?Nc4at_73ISJ z4%?@(Axbjebq@ZZS9A;zG)upRa^dlEh4DkxB^QzNN+G!oX6KLegPLQ_DNaZbW|8Y? z8V*oofr3v-Y7NdkHce)4^Ln?&Wt7N8BYon%$H!Up*rC^E7E+~aB$ug&2asQMQQ9BL zo&Hh-)>P5jcX*q6Og||}E;?NRBWrZ?GQOzIVNvYgT49gvO|m)y~A{*rmJC zyKC9ws-(szs!Vb9t2JMCt63J6ScVoYou1!^`Nu9%>k^>^YcnfP`khL9h!Op5MpJ`Z z$bmB&1fVD2liP%IofTaRjJh=AOL5a}Gf&$(MtlHjLGQT$c%zmtd5h|LteOo zg-xC1p%7e>0y*2{X9eg`DXL}eMc%@J5jES(4WVhM-+aVLd&x`5P5(E$5jyFn$!8c% zzFTcqm*sEm zNe)J-y<)9BcHENsJ@z|)-mB4f%Ls%Lr`B(zUk04faw~@}P++}y3}^|q>9f4hi|->T z#DLp?I;rzZy)DyIcqC%BYro??YTJdac2i2yh$Fht*3OQ9R%|cnre6FN2ptwg1=W#) zo*fl=S4^Nm3|?}aC@c=x62N)CNDx`f=KBxf(T+haQRbia(C-ZeDREjFbym#pn|~RV zgZO4f5iGzWOnC7ao=tIwlCK}heFE05-pkV60*$3@@Pp=0V{6Wi#3E)GS*s6BS2fkG3bqrRl5jCHGBOMry^EZeh{Ni^;ma*w zqa2mNsZ{>sAW6SS{=*vsuLflj*qq4*^g_&hajtmE7~{_(pa$T{f`w1K;5+WMS5hz6 za?V`7umpcR=RTDB)?2O+T&BW5mC4_Z_Cpo=W*0Ynd1I`yuL= zMlO5ivjVD!XDqq`>^LWuwk}bh))CtNd4?YW;#{1L-Qyfh!ApQz3RMUsLqwPtRnO?n zCb*_TAsH2G3{fXc%?(jRy-n~RS@fLA)wb^{X1h46FX;oP_2>eNcQ%<=-22PZu5$DU z!TecrKAF_d=MJmInW=qiOn#ZCwC8jp2`y{6^gGU$zZ4o_vB($d?9DwNj?Sk6#SAe3e1gC$7mLdox9%^R#S;I`(M$DnXEb~`t>?2%O&Ms0SODQV@=b`5|a`nkRb#g3@KH|6gIhijq_HjS}-*iKYEH= z5vc#oxO@hcA?~;M`6(tmKKF)0xU<;}qMp033!`;O?zzm<+}!i>lb`#BMv-KRnNh`F z7yZ7LG7jKSzJwH|1=da^xbFFP7(u4lLl|%Js|^hazC06`q@>A$AmJ^bMWwVH0LI)U zeAET0ITrn#gGuVPB!B&)ZU>+b@rK<6b-5}OC~3|Ei#3CU3{$37i@HuM2@fCJ7L~8_ zRxj$zKRC*0j^xJIeUs{ct)pV+fxZ?w23k1sMdq|AF0iMtkIM!7<#?QTFJyVwcX!2T zjXT|+1#qJJ1UmrbUhIfY?c2c%TmS82HRNvL34<)kQ~;$A!#IXtGQ^&zMfQV@PQ39ZHE zA+4g5D9P0IT=?_I<>e(>B-EU#tQPFoGF!roVQ!%_*?Xsb>??GM1e*B{&gnH|}{LO%eLy}fHKmwLZ})s;_0 zuDwUF=dW!VMChc~EW36h`Of$nh`9#}S(<78gxtk*#atS8&NDifj=c%&Z-evI(+9`R zL+Lmt-lwwX&jGsdtK#a6D|NrXO|1$^B?yT%^p)hazV3iD0E>8NPlV~;MpzS$dfs*0+_k3Uzs zq?U_czQj3w;1rAnu%mIr8hcX@mFVKJ2kODui$mCq$@WQ-285lBZ5JR5-vr;W0ZhU>g6h4_SIZeH0aelKS$8rf074PAsu958g@QPyv)cBGK(v6mEH#~9e3_~?>8^OdbL z1~m#WTRV*=$bNOLMKztk76KWdz6@PWYPClee2n z_gLWENupPUKTTcryV(MC_~7KkPz-=hd<9rf61AC->+BI6Npa^+fw2s^cz!1xF=U5L z?bl7WrQ-=m&>Q3WI+T*a+6h!0a_t9@ho#M1g^UVEwXkxCz%~t2s{t#x{yV|^L-i>y zhzmTQwwjd^uPTlMhS zE6gbK9c=}Of_{ZrEbvVeTW5te_BE5AJ!oxIiLoBQB^99H3*$+-5f{}fLBcGotV@9Y z%{GCzu>(NE<4l!ZcVHB_2I2?sRe+U|M*fed^;NDMEfi0ioY&{O! zWYcgs)K)aXkyY;k%bf-9*c)1>l!FfJm(OShSX z9sxKS%NQZmB+X&t1Cboag3-w=8UzKM>S${}0r|LLCpZff;BYBGyP@%y=JV|)WetrE zpxIDT3KV+OkHAGGMsD(nN>1>kjI`SQrr zj~|7AE{)8>!or}|o){>mYc=5pRzaZ57Bsb^9 z$#DRvXuG$;*0U97#`Zpk51{h5;_kRt1ih?#81?&n)DpSPDL&ErYz;gOUZ%;1hgU;HIth|QoVLX z<6wg=^eir4%m7c}zQX3C4vo$a3ZbN=M?)ojO^f#dM3`M8)EVI(7*Q+y+^$NcDq zhv0%53)i!^_{F{}^dwN$9_psO*+XB6&5)icvPu7meKG)7DV^b?C7nwa1A5?ip|gpF zY<;H!B;I2{i%^RNf^(4Q8i4m@h{9q386MUqFt_z}P|pfcd)EvQ`9!#$zc03)beI40 z>8>p-#OGr&U}jWb6*eyXB&@GmZ3wTUf)5E>%SG2^)lprGe``vf^&_aujI$bz7lk(T z0i~mC&N`YT`X!a8x4B)EqUfw>JFlYT3%$&sAItBS1&C_TU^)=v2){eYrU#2HuncY- zcj4!X4fb-t$i1?lbB5e9b7A0o`Cy)f=QQk^;JjJ+cDd zz$5zcz*9F@z$DIu*l$|$wxWj(Kz%Df=784VBpzGw*YK37gT%!${X8BOFjEjUhVFu@fAsJFX+S(%EQsbdX_F6!0);sK3coRy@83R0O7ZSQM!F^V zdV(yS9n8kW9-|Ad&>^U8icCW;N9-nu(?Je$lsO3ZNQx+mD*{RjLjd{8l+-Z*bbJI_4_v$8I;93kTlWN(lh+FKu`WOOFpzhQhKauxDAC1I2IbfJjPU`I z5bQ*RTI-D|{7f1WWB^f3;`&rw%gy_~>RU9`RYKu!Uqr;2Nzm2Q2z_ODoT4?h_=b%v z)fkdsF5Y+lG7Q3w-OEGqQjOFkaIk`|LPs6SuFIakYZ)T|EC0*zmnpJ7Fn&Uf4^M%%9<40lmaCMr)uD zWTey`ISVU}q`D1ML3 zE~(3!(%yKhzwuGUe%uQq@=#oXjq;vSHvnd4q z`j_cKxXC7LO*esI@DI(#1<5tyU%fGp?QEKofN)FU4`qiAfaXNg?aYh**F2gr&Y92e z@q~Wk0wG|bvydWpfNT)poE>^r53ffS-vR6{*z!D~qRdifU92YA{BdEHwdzR^FZh~b z0DB3zXuNWW&!SSVsRra(u|tULxk*^g0siD}?lKp+Q7xnZLmNq^8>o)|;}CB6&qznb z;NhofQ+ltH^}a>=TWcULCNdf*zbo-v9;pV{ULa!>2~oK0 zs;%yHbyjOs?Yq1kHF4btuRu)G+#y`sAy0Zy>QT-*3r8YG>s5KCSU&aJ-}y>u-BVst z2UKL4y)ok;dhLf;3Q4{X7*NLXrAs{bz(K>PS<4sJ&l6M}f+FH2wU z=oQ8J40XUM+L%z}vh!s^WE!~uI0@h%esc-)*U449Dm;7SU?mKE%6cj_a^?FU72B@d zEgyB3H!O$h_}~V(4H)5Dpx=YrF~AXkoss6~wPNO38Yj*$l5Qbu2v6A*2_NL=>6{`n z1^(fl@OtBJp&JA#dQ+(rnh}qq4(Q3859QH7mJO8rj5xB2U}4JUz7cdR@F83f1$xhH znWJ}=+P+DeXgVMN_ZGw#2?VsjOPk5&6nl7b2`;da-y#rc=gJSwEkhL2dBxCu%!0wb5J^kw9-pm+uqi4Q%2PE$xc6hMSfVB{}n06CvfIp`L*{od7 zI;(6Rav>(1;^)twXf-xkVm(%29&gi|Yys5DI0OWpKqgEKa3&VD<2tD9}->5m8v&MePGU8#mAi z$;rxIp9|dJ1cZteP;rX{9dQ6jku_DQyE_rcet}F@&cJ|j4u0-}?h=t@1|j|#8u(Nm zi3#L8RiJlu^P#|eX&|> z|67_4?b-h)={uU-+$_Oi-b(8J3FWN%+M z;6*ZJ!G&Tku4H>na#?x&k9MWy%J8@J@vFk?4>&nH0C%_x6uPGpo%PaUs&%uw_inxS zEw*xe_G!2zi~V0>cwv7AOI%_P!Frhs7A4^MOvgUCjrxG?Z+-RaTiHpNCF@0I`x zk5b39^6}I8uKa_Z&z~{S3D%e`mVoD(xh}kpqwD?X-N6fP#y|8eoNhBE8j&)ewO0qk z;wR0xqaPnmJ8ykY4rrEd@bGYGNz{@+p);Iak0p@7k@%NL=8vpAA3_rn=qvwq9S$~BBxd2@q zTi{O@1JOHA7egZ`kU*1IQKAv>6_A%$UMzs%8m*O-8F;@GT>KV`5#p=*tPwygfj7E7 z@5Pb0-mK{v3`PtL@gCaqW|oZd0e42u%gbB*o&48>wb1$bhUbZ^ZcATk>23eeDI)w( zqu@`6Kr?K@Rj(>_3j<_YAczE22IOm+p}R?pL>B{Rz>UYuw_e1-bRhTA&q9os3Ji+6O`A!k80lQq#5PaV|ufT0M!R! zs_dZRiT@%O6!zRIuBi~cw_VaHX4(&jG1m4t?gUd#qHaGdrV(;}1c$e-??=TORB5)Y zH~V2@04UP7+!jg!!T5)sF6}XS8bNX;>{w$3sLu!VoKH^CkP^H(K&+h~ zbB&Mc4VXatKS-awm^{mPsdG8to|E?K)svGGf7|^guV4>p=tb+k6{T8NTx_g@s%q5w zJ~dU|<-jE319`y+gS(&Gl|p+6w9Z_;FXGR`XJf{0MPqkDgaC2R)h|BlaXBeRS2t-B zlPC7NWHe=y35c^PqoFcq}gm*n#=g0@q5U_CdRKwpAHOLq* zCOON6)&=IKrpYYbGAXGx#)}=04VqrW50tbE-wAg6%_^*TAT9SVhg~g%37}_MQbJ<} z-(y{wM^mY3qA>E24oFc=7?Ak__#)Bg`y+*0sC2G(BOS=o^F_UcXw;`Y$vD3csx|P7 zn=xyl5hTztH&Re^2DmMgcPF5%3c%l>QVny?BpOh$Caqb3#WoCSAl2KGD48jNJlS%% zm7Dzr6;x;#w9?iY8EDQO%#ztyzqJCeg2W1&wO8LGS*8D$s>D7+j z;!?g^-=ikOREkendcduvD1DmLF9Z9S>$uM^^k)Q0w)2xQ{rt|>$y0Yk4eFa@>w~na zk&8A%lL3nQB~WM(awMHxfB|3=5N4$IaSKNfBj3HF1^N+C+LU+Mp7f`Bu3zGB^&D;c zT@G&jg0|X6X#st{_dU+mo}X@;qfi~}VsOTpoy}0scxlHP#BQ_$irp*+b+R5hqJ~+l zA}^2i-IhP`Q_^RioW`JI*Ujnr_-`|oQYgTB*OU|3Hmz?@Gk49Q`Z!ZcQ}5Urc08Yj zf^r}xQ?Ih%cRT4jSKLQeR0!#}3wxJyaSS-e;^$d2ONj0^c2Cg;Zr58|sFi$x=4sXDy#;~hJ`dJ;-S2#{7sgO~@o>SJc}l0EGID%1NG@V}Mmxo@J&^kTr#0@k8w zJMvUVD-tOWr1~Rer1##xLB&pO@|)>LSvnSl7QRFk#SUEz@Z>lGzow^`d)BypY`Nz6^Hm<(L7{EL%&bMGz?uB%&li9@a8vSt?sTo zFS*VzYP8M%$_^Zbf7s-7XvRoDMek{q)Bk&2AF}9AU0;5MVnzov66q)>hY8Rou8p^7 zceC28n!%O7c#PbY&f`r(Q7J-FI1Hbamvcc^$rxIsw75My27whsGQH?gyYZ;73cdkL zs~Ni!m+!1EXNdSq^y3t)zpvzZFmw`{c^WZ;(+EwS&U6;P(Mw&!)b%Sb4TreFUS66DuGw$WXs=i3J9u_U^URuc^WTurhFI>i*=f{L)nMA zeeXX^;`)ALul?SUWya9!q$Hcduj3qysC~NJQ?Pk`Ky}VUlOK2Dr;gHOx;2U{&kDZ} z#w#2R*#hEz+wD&=Z@&HRLiX=B+nD7!7Dungc}I7#M}VdHE!)tyPQWOMkpjGF+@IBE zL|^H!fq{Yb9sPR&^z^>AW)@FmrrmkvQxuXJn7lRI0uGI$%)EZ4{|wA_y;YwQs#R4 zee;|?C~5yaY7NsZaQ)L z$TLuVjb{MS=M(!GnrQ_=@xLp){@D$_Pjg=b;}E@(b8>URFqFGdD_z~_pg_$SQ;F}a#8;>Shg+^_eKAyKO3{9r2dVV z|GjkW@kqV~ATod&J6~B-^G26PX&(+Slr0I)m7To~_7k+;6(la;y|9GR@hh5t<2vzQ z$Ni1CgB69MGK17qR}(QX3;>uE&<38Ko(Ac}+OJS8jM^6iyE(Y_1%udXF$N8?+^q04k*lW-6LT}7@MzINHa4vx?Ni_H zHd|A5Z)S6eyHVGlKOuo){0N|*{^$aR%+}t~!SwK9B%n@7e};Cl{s3Ir@ZVx7P`|JG zZ?RMxKDc|IQ32c?9!6(q;4WAICZEM>%@|SqT-Vr(G2DN46Mx=jt@Tt194R|TQU5Id&(_it>BS15sH?4)NR0_Iiye&RxJ0XA4LXW{Zb1b`$VqNk4& zp7X~9MZe%zGJt}cjS55@q4dSzAT~Vm&MVX!$m9v_|38@$kUGLv!I^xtGmrkY3P3#iM+@HQ{ZLj`wi0+^fMPse z8_qLm4Ho@hv-|Zz8K~ePa7hC2w2CeMu0Mb|vJ7K9o0JnlK&OcO{LlCLe=4oS_GGrdNZJwt}M4?HUg|Kb&JS9`c3?-W= zNkWl1GVgb7-S_i8?{WN(?|Z-RIPPwBUDr9Tb*^)*-})_4I$FxdjxZlVAP~n?RTOm* z2r@MMv7kHzPxhW$s2~uB$i0w8UKl?+M;BWJr;x(mzi|rkJ77J%IE55B1qChL+^*U< zTH1S9Vmz<9+Iqnwc#px_INCYd+Wh@SkY7;X3cuhLej$AU5l$gFej(ET#IA}7i<$iW z-qPOI^`8z!uJS_%T$+~Fj;>x?{UY7QMpL%#$ zd5X#*HLds!)r1tpoLwBR{4*Ude>dB|-`LptI9l71S_&ayLI2Dg26X&quA*|H&PW$e zYcGF{tCfzbzLT1;<=@Z63|y?NB(T29I{X4YNO&l3b&YrsbzD_<8?ux$ZIy#<)x<+2c zmWCKjJ!OCR+TB>$31jUiWU1_K=^vmT;2xl7jBz)H!9>wYIyQ2imJWJe9@u~YM}JRW zRRc$DXydC9;BFh>>8`74qNC(2pl9G@C!p@)q-lULKr8wBSS#vix@&r((QZb@68z2{ zT1Mh78vM2aPQEsZ-hTR;5?(I8qF7NK1y5&3*i|JhCvhKde`h1n0IaZ)i-3!_nun4B zR@_M4)8EcR&0kQ_&BsJb1t}n^>gMjP=cMZDsp}zTtEyvX=x<|a1%j8?2Bnf0HnK7b za15|?^YU_4*0Mv|IP1tGZ7sb}!rtB<3f|(j7#B2FBS6K)#YDtg6&k?Ms@8HwI(|YT z-qxD%K*ZP5Nn24@+eld0ir-b<)6v7zT1!mdSx8Aq$VW@uKs3OfU*5)0Sk=eDSI9@x zNDJDi>N{&F3t*rxjK7dNzn#3bvIa)NS=q(fMBPWi*wzwbpzk7P5TI(SC~TvQ@zN4? z5JjojS=o5_*?9R13uqd9s2gDnmE~2Ol$>mB?1lC8+=O*C3|0MAT?FMMTzoa4Z+Q=U z7{C&1&o654Jtv73XIi!gLyC#t1mtSiQECv1XI6%yAL^TK*4 z!9aR88YmAJM_n-iLkV$=JyO*jWvi#@po~>jS2l2$Gg1-sHxjbuw=^;qQ-jZ~jFq7? zIYmRPs;Y{kf{MDTsjE+~lAMJc(eC<+<#D_QHQ ziiqggBJGjN^77hpnrLHN4`po=Zx?SbUvDE#2|Zf}AuA(MVNE+77a=zze<2SSV?iGu zbr?@c3uWairlPEYLMn-g3n)3s85*PbvG#Iy&H~m(7#{-K z>YBz#5nJ&9q`ik=fU~ZWp@M>&0xZ{7&)H4E(^c8n1+A;%s&0$bR6{A)3uFAXEnOsB zC9GX^O)S+FT+k@eh6dQ^irFdKi>Z04_!%K}{JpePO(XpKA9F8+be;CQi$Jg;R2Aj){Zi%|g7x)%lNRd(dpPE3zTWVtSCF2`(f)Jf zYOJS$M_iY{{7=Qc9f8D84W?l?RmiHQW5c-aX-TIpR1|-_);>gs49|bVm`zT3Uijr5 zW+AJ=>$ERVJ4DN0ry#%h>LlOoR}ACZhZis?oxx9E);6UK9hHP(M z<71(`%|`kDtI?V3vXATjAx^iSNy;Khw+OKKOe@&-yN&>y>R(5^~A)) z((3A+;$ps64Gna(v`pTx zRV(t>FPC3q^{1F)4$;xkCCG-#+`fJLVM@w_h=@b&;}4hgVHMTyKYhBNr5tCqxiUq0 z>{#1S;SH3L(Y;%@$WRO8{?PxA4TCzis@$XZ#sj|ZZ7*d@HZ zy+4*%9$>s;5*{9YP~X6yZmPB#JFXNIB+Yo$^xm;EqAy;*{&J@+Drg8*&q(Y%k_Z~b@l#?TK*2{U6o*viPsp2wP z5mm_@uEvhyROGyH;Q&j=#9*FYfo9^9iVESLWU=EjeW})%BN>9Qp7%x#0U}kHPwk`B z^@#}y*X8BOVbMm(@;FL*bc-zy56{=Bsk@n(=ZA8{kub`1e|j7q&oh)m%M+#;?-upo z0rjih^-q4kf6zJWnPxUNvSJ(77%NP!IO|c~RmfBm@6Aw%ED6~S7_z&krl@!@mPMjv zlv=*|A&y?_(c{Or&r11}h$_%mVaH>iT+%KRy`#g$fe&we`SRt>)#=`X-iJ>U5>CI& z&Mt*FZ16^bMbb;9Qc7PP<{a_zCC88*?U6f*@g>PH5ki{|1_UP;X&9~m{rkY)+^QZA zNv>qlFsk+7T=-nu3PZD#UQJ9*Wp7k)+y9)KYrRcz)F@ew>maQ6V!mN9coU6Army_y znwy&odf84QqId)@F)=Yk^IVI6TvH<+T}&?uJ3Add*@^UP@90?W5$Z1yz4K6_!iq9q z5sB=2{po4Fs-`B=#pSZI5G50ALXVoax5)Vm7mSnH)rFalIXOEUC(Cu7q(@sBz-HSF z-!!@UeQJtbSomorQ@93nZl_~t7ztu86;&9zY*zR5VikS4JoIz3HR4$3+qc%3h=Q)Z zzV`N*Go+4ap(Eyoh6bYw2Rdh=legmH;v6O`sn)J%s%cFAc%P-BqH+@xasFeO?NP8m zu+~m$Qj53G`IK#KIiZD|zrTb*fl)K;jved{`LScijOu-^z#F8B3T0AK5~&U7gZ$Tz zrPd$oym_64I9i3B-smWE*WuTX48u>2ET!DVTK`qM6mfTXMP90GST-e`zs!=7hm}m3 zLWAzdFow5ZXwMXw!%zM}_uh#>iyAjFc6Rn}XEeDOq4VpQ+>spAg>ylS{F>5>PQmXD zDvp>9XOVvRR=)nQx6^D-Ml)~9edBfsSQP}~hnJ+Hq9SOIuu@(A`^Us2wrB9UWq{kU zAGj3v#B-(!!)p!`t_ z8lP(CIeIiX;_Li;OmnjwOjgm5kd|k-o{1KkeAz2jdQFlAX8cv0CPX%vsd_IcKnsQ8 zO*+E)d+=iK=4UXX8yg!nGwSqZk)#bfc<`g|z!iHxRON$Qle9g zLmtWQgd+Sk(e%QIMp=0l(iWwa>WL87dRw%c+*uaB@VrRnVdo(&-n&gdxq`2kwo}af zMud=R;8Mx7ph-9JLFYq@uaSJ0+dm$dlR`)5(f7fDLFf>F60wuxr@rNZ*}TNZmnf8@ zmP-sThjW1lyLXQgv<}v#@hSd#zB{1{i=C_|*4$onZY=R%SyV}LLZcbrN|=ejDsz#M zwqof^;3B(}6xg7v)`gkfkKVqI#`pASsO?=kxUWrV*8)p?7tKMN$Su%>m%&|5_V`-! zZJ4hnVeeFN*_^+W@4}nC-EGcYt#A)E6ej{#s!qtiT{2p4eHMR>o6<>rFMlvQ+U*Ew zG`QTd@HuRHyZRQvquP~&O0opq)ewBKt}MF65yrcwZP86ib`CnfaP6d}9` z-)>7D6WZV3AMQbhaR>QQlIiH`>be&fXOD@n(a_Xuh)5gd@IueaX=CUkB#S2G-V?oW z9~vb}OG{NWHRGF1XLWMS}*AP-M(|@{pZh@AK8Y}gC)#a-kK|(T%Z3` z(CfxKZ5j4#XLu>NE%JwRPt(Vy28qz3f&yyn>&=|N2OKBiiWN9k8Hu>e^s^|9&H^J{ zzwxu#U_I_Vs?O5jQMO^Im8gn-l;i0brOT!@V3l00mqMDKhNTIf7}6U$UeSXbS|qOM z2HEq;2f=qtuZVlw_|xPL$Ajvdn4Tw=4E3{DMU)?f-oE51T$N9SsS7Q=EK)z4JvZ|u zwXgIJrgL36eS&(f`1-zqH5e)|zO%CvG-Rg|YwI4iiN;QiH_EJf2G#rwEfr0D$jyO& zQ-CXUAuxZS!3l;^21`a_8>vHe`DumcT-WZ029kLb#Eg=)NEby5PqHXR-o%8NWO-lZ zd4tgX219JsOew8*|kuts3V(HLAN59rfZ+owY}Vw zRrU5kBL%w{Y=}fxPmdz_2QY<ZM78;Po5ys&e<71Tk@Hsxt_jv zC(rCxq4>>#?~L_S{$PPV-W?q!H))kV$iHL8gm{s)&LXqx_N5@5wZ@n3Sm;)Lo|Yl! zVv}&rUgHHTSJ3Wy93J0t1KZFxN?lEw(Uup@c<3rS;^tI$JgyVk z)-~(O$AZ{t+`o6yxs1O~tz>#EC?Rk}Awul;PZlj!V(OTzJ@cHhbC;CfVWZr1q{XBJ zRUrpyF_#MVJUDV5wl0)m3K z!G+3irO~v$^DaGvaqBnhO{0t^O!AwRi9tSmakL-MoH(#(YtR zhW3B@HK}D&oRU1YjGrqYrm1CLMYVhK={V*efwRPsn~_0wM$9!lF79}h%gF7O6_0A^ zYA{1!)t1)RmGt#bGM_nG4_vnxLd4{Py5CM303g9$?0bHI=#-0}uAxsGjT&q;gj70dsN`qmg8V*iQEm0>f zk(wj#Gs%GhhLN*m_bBXTP(UiHI=oMHV@ydS8DG06u7&xhmU z#q;OSlS2?iiaj24s(+oFyb~6NfUEju*JDo7!#eLdS|+Bb-R(aY#KeX(#Pfuf*u(ZN z{|!_R6^oddmXegqetI%3!7dIaXm_w@L||b%OOBByKp4}+OmWlC-eH^MiggSC9ROh0#?{Uwn2M6 zYaMy9UktndAt_j8d5PLS52rbzk=J@tJeoSp5SR>d9dKWqX96tv{!EyXg_foz$g5jgd-?BffY8E_EsKa5X)#IwKXQXHKBbgU6iy7 z&q))YJ1sKHq+!B9cb_zYS!tL6JAH4hdLleKu`kP6mi=ctFA1>&YtSjx->2fybZCg_ z3koj(`qFR)Oe5v-=zi=Iy`(TBqRi0bp$RpL|zMAXXE8PDk36M z&`6%)0Wyg`MwYN#ldm>7V4!W%SjIpAEPzGIhx*p7TT9E!ayB*`5YT~p5}A_1GB`Nc zbd&!@R@Q+#ck~=D-??+=$B!Rj%{Rleaus}hMAsL_NQV!6$ewgrRyjF2u+>Tm3I`^! zD!50Ww9&`OmTGiIXS?PM4Khdn|G^A!?m$HgH0dam)owWMYOxGzbTlXf3L`RR)u4Mi1AMuiu?z>SO}?502IIy+!%4+*exyk*kxRyqR$h9HVCT^$gWN4r0#{vba^Gj zHCL$@^d^zLU7IsNnf=8!ru&rZ-n@Bpke;5N_T!Xqri2T(dMvbJ4r;Q!Y`W9Ps?av?ExGQ?@S^&pd7Dx8~fsRn{ z`!!x?U`GTFEmjN+^%Ln%WCgocyp#D{8^XyzP>7_l+}&E!m1&uWAopwMqYHpVe2+Dz zyg7)o$HeG~mxUZQS;>&yt=3k%u`4h-e!0+H>s#y^Zz3)Su#=A1v$|tx+@Gqk9bHc@ zkw5|#nS<{NrJ6=hM!MVW(s+Nl4wq{B(<+&`jJ`AU#xu;i#TKfHBAlEQ$zs@N--gQA zFJ3(4=%|*Us#ucQ$Q{U&Tkb~TtjE$4za}+0{$N^~cqzLu^#1Gi^7A(?(dZUgyhpCR z%TU0NWtfyGQ6(M_ZW$B|lge?ZNzL21x z{wak@BmhHa&FKsrulOqqfJ{vrpX5+^eHlGZ>HO(bGk?tSC=aZ_Bdgu$i+BbG2EXn3 z>bAPn%>Tl4B2LeJ05^2$g&%{n-n1fq?b8ikmM!f)k_&j{6j7@ET;=^eYD2m}QWxgt z=F!^89?xFCmbq+H5(bb9HQmL+Zs~(;Q(I?U>JKWk?_AQmVbL;q^^dS=>8aA;Vy?`( zfHkMKi|hB>;OeAfVd(*+Bmyp;DQ0rFj;*slcX<~=w&>y3VBY&3+F4YA?pd2F!`a!` zaN&RJt9|?Sjq9~q&x>49)r3tBF%WRaxyr1mZ=ugIj`>D&!?vg?apMY`c#6r%Nx$ht z`+bCl{h-^HG28&g$wJ3KnLcdLj=rXRX1m`YyYW2^$2H=c>>&ns_`R4ITg)MLA-K#! z^+cWZ4t^{gCpkzLVTsC?nSS^sB|!uiq9GS>z%`xVTh&j^#|rgU2VYpg_{5d#EgP z16Y4q$UHY@9zS~La5_az??uDmV!ce0zu1+;u8#K2p99%tX&Wjv>C>OH!RvM%^X`rK zJhbQ#DwO20cb_mYbhs@a1P7r*KSs2+EF_hl03awEA76jo;eaC}BO@Pc+%J$&v}SD+ z+q2{kC%00Eevf8W#1k99*t?c@#IMbj;MUjIt=?xMwXXGi0HaKYk><+?g420f0pVfZ z_)N3-(`(B1uTS0!gEl|_Hx}S9^=r%9NI&k}wP=80BU!&kxr~%iaC39R8M9Gb^_K+? zSlWr#awUkfzyfX)U)ePVMRa;!M($AXu?Hz;jWqXXSEk;Mi4NcR`Skg7>$W4dmBANY z-X*yh_n=jF!FcmFg@gDm%kSJ8$IQ{EyVlhsUtvrY{WHwWB|h^ek*wdk00MrImls8X z{NO^289g<&=K|w>z|jTAFgbMZEei_}ay@aOE84#2eoK^xoTVpNcf?xG4HeQ{Yf9;` zR;U@rGv$`xpiEDEYafkMiyLTCf5ly=bhf*3-qu+dU z$RFPPlvgue`1<6_qo=O2ad3pg+#mdSMb~OGfG?jLK1=gCV13WV47gY1rx|Nt_YeaK9eV!IYIu70gwbR(%tv9i# zU?`LaxDS4`n9cvbQv@%>D?GjJc1N$^B~xht7WNshDQ{%BRD9`(RU(Xo|HQjI9O*4o zG`p_$eN5s?(E9f^MX&X0$=}HD4`(HqdiD{)?4yWPjkZAhW>!%_sq zR~4ok*xN=vW^oZen?+t^d9>RC@(2Lr-TtcYjnnVO>$B#DT~WQ=-M5vpO6SVsIwB6r zT??8z@Y$(Wlms$Xn-oz=4L=z^fiAZ;1GJTNyOLnmkO6$XJR-KhqSl&OBy)2|_lLIT zyoU1c^gf=PXT$gQ^eksa0qh);GEg2a9?Pr?{NwuTQ+0n)e_DgL3a$RzF-OgMnqe;@ zjhS;L!Hk!$0QS-4a3^*;0RR#xXpr%AxCv8j+9p*&Rlc|w64kWV3>+{wbJjdd+5 zrN!sQW0fl{PFPv8{Q;O686)7%Pfo8NA7gvfp1i2jSbHPZ0$JR;$|rq1b`rog`Ig}x zo`P&&KB_-DtrPJE`RL#Ir`XBt*u%g~taOajhfMW>=qE(J1VxTwx!S65ZaYhN+it(! zDZl7uU`fKUKDVZ|^6rm^fA0V=mi4OV&Q+1czMqcrwILq@{QLtacCuN5JgZr9S7n;{ zhIa;>xWeK}r}epH&*lNp>7Ohj9|GQw>lj&Ay6xgtywuLs%DI5w!Wlcy6DT+Q0w;XB zBz~V!b6*SMX}Lq}+HYNDMpdz?u|MkU%*^3C&HOX==i-i0?c<}w(v@QcG-en>m-yoc zmj37lxKzG=eQs}Wk7Oj2z|w-bUtmzP@V4)V(Evg&X{`vFKW-t3jyzznXL`2Pa$&UcQd-|)pxH}+lbKG@;T#l4;4gAeaGuVE4|e()=m z6nfQS{PNC00BaWt$u}4D&qn8&e-Th%Bkol;cCg0oo*Q;ncdz|MZxC73PEuI2Pro+t zC-pd`DfSd^bkl|ZKDpW9J>KT~|AHSy&&KO;}3D^sCLHl2J+t*02Rf8AP3ZF6%$dkr*W zE)fjRyPsa7vd{J0TbxWQi^>@-2xc8Ib;T}eJNew&l@ObQLAQqcTQ8ip>8HIV^OI?Y zu`g@9zN1A4O=9x3LITf>?{+W*S#00!eG;m?qJP$`(dYgwDO!PaWpaAC0cxr~+sCd0_zR#3O9DSG&Z`?p>GiISjssAg@!%+& zoZn{ntBWOp9VP*P=xHpz+zM_(iSJD+UBY7`o~>Yg z!6?#6`^asuVm#Yd(GYB1UDIpO5%HBJ97Sty=jE7vx^ux|2Rkv`QMAVh*6YDr@0q)F zm(^yg)pqr%0v-Cyct)r%!r-3)=w%EiUCjCafx5f1(s>Z10Bd)5*C%vu`Qt;{Bear4 zn3}fhxi@$5A2z;E%T1sye(Z$K(ZXSu8xC;pTjqUs`-T(OW7}ufwY84}szq|}$iTtN zK0+p57cv#=I4Eu{c>T#bX9WYri+o{b43ey+Pyfe`#jEa&K4M>iMWiBlNt6^6u|$jC zA-;DWGTfZ+&J!1y)QKMdxxgDOZ{l_^s9B}?;uSL97|#f`BLib!o)8*pQ5CFjv!bd6$K z$6C_o16TUTLmBUY8A{pSqTGrMM3y66y1ToV001Lx4{hssyd20EwVlcF=TzWsS-Yvu z+E11(YJ0`{YH&F$Q4HG*-*nCT9-Pjzb*U`b>Zy=j{1q5B=v@EEh$ZF@Cl$`Qn6~$v zK!#yx3lnoCpL}|s+Ax21MCi30Nx6z`srrz!jOeWQ_wF7&CAq`qEZs|U zkKQNqusVHCwgyvVy?Y6V+=dY1v{;PhiSI|zxi-E=*%5u$!?%CFqPsBi5?wmX#|v3V zOYwTQ2we1t{QD7-UrOGzz1YY%N-HoyNRz!+cD^9!hTL18{AWc09Kp-Z9E{2re%`le zUD7$X)@xG82&Qr~w+2yEhOqhF4F?2A#@l6x>xlbvqyv=5RVHte#qFVCy zK781P<+UuVE`Ow@dB*Vq*8X5y`rH}$M+*f@p2)iL)2XN3M`R5Brs56XeZ<8^^SAUS zXqv>IKltK%kz)>O^4INv=o7m5_CpJ`J1mAn#UILE+x0SkpP7-`oOLFp znv1$!GHZhEm7r)^(ur(+o*PhdF?5s8BrwUetlzjVT{g~lJotH6IhA*lHmP^Sz4DN; zUVZ;rX9Hi{&Xf}}DSvL*5yg=8D<(8stdS<@HHC$knUdui6Z;pZs|8ivdaNtod=j_LyEgWt0r*CgTd?%?koK-(f%iOm!_TXh` zV0%{2UFbg>@g_$z`QfwaplXj*dIPh$D;c-DSAI8^HxedHi1@=66HPPA+KT?NlH2FR zF}W95WJ6fo-Q8XN88fVU7%-HO@Q(5N}Kq#6@MV;InOE&kGWAQw5@5MJLh|U} z^Jc$h{n6Alx1Ih5Gl^RJ%8Odmj?tqm6^J<|AG&4Sb*Yf2^nLfotg1`B_S3C4kA4c> zz%?;i6bWWa$fgydhbGKaE%FJZ>)i zl9~+htRs2Y3)KcG1Pr| zT0-@$DOC$IoTHad3}#vq*LWHFUg}OwO<68~Yjaf(^tz8*Ie7T+1sR!h0CI@R&J_*f zYfm#V$prh&J@yxTdj8~UmSn{jrC(3@im-R`;luh)3 zL&YCIK2Z}H-ts=xqsV{&;tm}%b9A=)^Yc=H(|I{kdijDvLd)H}#b3vK$16u(teZbK z=Q=GOw0U4KI6v)CWB98H1w`Dano+Wq=WcYbDaf(FBF~ZsFAWqM60{K%6jWmSmJvXN z0}o{?@&BDgda5AgczMQZ$}qL1bED*kD1ErZj%(p+M9|~a!a@sy#4GXL@gU+dT67w? z?j*ugTQmbD6;(3{19-XQo0)B}_WL(LDaRo(MM*=`o{W7RH8`mMyj?|1rhsnRiWBaj z&vTD+)TQJQO^}c!WkpH&?^C9pst`WYny@eC!(xg-JOnW&?7g~>?CS=ZPFi|;C$v(; zZ{fzB9>V>4T=z@IXr*(h^YB%NN)zC-WkQGqAxZ)bPW zXW?^HrSq^nB&|RzAOVMTa^gu!PDYk{J9r^R7`Q)ln|3gDyrNtizy+hqRP(Kxi^m^6 zq`hoXLCGA`p1pQssaPG*A~&~B#4B4{3};57dHS2ICkLapExx(Sh#iiKin@32-gN-w zfyYv6(?t(CDmFGYgp7;~DX1bnbRCl1Q0|8F){>tmQ|zV(z0f9-wX;^1@9~H}YQT}Z=;gDti4Ss~q%KLCR>LJj z0!Z(PLH~BCXxo*0g!pbVE_r?9mj<=Br%Vg7oJop$1{G~r1Uto+m?wk`}D=Jf9iJ{N9zLi`yCg>Q?@FWKX_ZZj3_-JK;%p+X9 zmEDXNCR;njlJBEmO3e)l8@71gy0d$vR(o_fi<9_7r@ zUFOo#Cp!m_)Yrk7z71p)#) zw2-vE4wqv>`**^PL8hlkYHeBKD2l;&6WgPIMT6r{)2Qi&{4Z%u*vBCSQG;9}u%<_u z_;y%RLVug2(c`eub*xdcV&^@8ET#eUmd_*2`y{$D$6%zz<0>OeRvPRnza#o5J0hDj zTU~uZs)_BH)qROAL39$&LayV%!w`2K$)FSJ6n_QG1;fuitKVHuRa=B6ERz3yZ-peB z(9QqV47$GZoE4Rre^_Y1>`FM|#k*)}&Rpo(K>r#y2yU9BD1`EVZPqt_HSO$gr~({$ z+COX4pPo(pH^t(_n)fP-`S{ z8uhhpfdiA*L%0LB2G5KT%eK5E ztcDb5mY&&vXRZ&EmVriRFRNZ3tFAUqR#dqQ&Ukh>V(=l1>4y@b@zryRPlZvtizSh^ zMLgMAUJX|A2q#q-^cDE$QK93=JT%Qs8ItG;QMBZpX5FJFlX09NErv|BJanwJ1nCtz zf|PmTTI7+cgjwO_Nh2HQ^tY+)tz3W;RxTP%8wI&-5*3D&WO*uqQU|&4D8_>e*&1|p z2%@H-yhat6!HY)j8|zOD2v1>xM^F@8mEC9%8!USTj!8Fwe@Spc^e^C_nsCzW%LBa$ zk`8owVw5b%>!b$rj=||V#ryrUla?&ARv5;KgF9L}vFwP-Y?z#qK5Q9%a-dm2M$a0! zy|{oS23!x?7>>tN4CQqUs2QFf4WF>WmdXTCUUx12=%fxE$CEnt0`1HcG_-AwM3|_5 zvBY!>4?csb@}RN@c^peq?#EkrAAtj2(0ks6|9_u_5e*ZxHLMt~$nav2#~1dK7eX7o z#C%dpS{^b5gaqhBlGMqcn=G?10LA-4DYkro1vFsfR9sT70iCXbZIPPgAnw$~!ZI2% z9O=Y(XC9H(EX<@>I0s^0wk^u(0&JU(5q88l5-~_Rd20K8f|TY4*LL-%d4zzG+6ghv zkUq~jbTm)4K`HD%9H#E0tF(yIqCF^Ql3)krIzhAs)DF1rrEIfwh|_j=RAiN$V`)N#R&x>gA(Fd-At zg#0b{HEIMdvPWGn(Uqj04fcCJ1Fi;G<725}R?u!~$@!0~_#o+oIFZ^(2&&o^J(##u zSSNMx#X4yT_etZOn$&r|Z=J>-I6hFNygpBoAtT7}vDkihgy1E1gzr4_HFnrDLgo=2 zr+9k)^!uRk3%8Dw3C~>he!D77dbdW>>F%a|^P={^k@`P8p~td)CrFCN2a1<{`R_rj zs0d{`1pXu#DAn{yZqk^i&|$jyPb2n^V|)b3(bO@D%50tY9I*Zdw40`d@sqf5XY1Nl1<{sN=vgqN`7f=?QV6o z7&$pPut|>re7do@NqPm0|8wiVM(#jfe{fKTe8)856tzy`oJPu8V~^|}J*0R-d$+KU zKTfH_d|}+MjXX3sTjD;yPLm%vFag{ZV8}ZNGXrLR9QgFOyu4e@H?J!zA3k(Ax7`qG z0x}_gf9MVwBn?REr+QyLNbmyzhCg&K=pY3JyMzQ2beH)t?;@KXz!2vH=4oJ ztusG9goY8z-Y9z!S6`C0gbppBn;#Gu=>Gc$>axXamW+%H5_$T_(W3(V{A|%NfKYL9 zaa{-Uzr4Ks67U=^K<7#q8Wv=1;bMEq7ZMX_HjsO;i!5&^j+9zbr5__obH_(_SSu3Z z2k~`vlBDGQea0)$oE*qMoF#4-|EFpKG&zWc{(hEw-c{vF+ald<*2ZL)EB^MQi}kai zX>htH;pP@kGeMh~#k_dIcKJpXb!}}e{S7g(|1LP#ItWVd#Ovw(oEUT58*mi4&@HTh zfaG@3FcWdLg0x_Iw6bn~W@aXd3=_#sMFms`HZHDvmE2`;>kW}&=cy}(40_Yu1H?2fleH>l z@)Db?#ML){r+ssM5FZ^4V6}e+4tFoZwogw-Cyd0E#p9!)3pg*&KEE}Ej4d!=fra z#VN2OJTLF|p8&T)T1&rIFQ4YDWcb?LaJi_eKB$-*unLFlIuHFdE~m>@*9@G#Plryk zhjmfSn2%h5FNA-_ol69Px$oP>Eq-??aQ-Z}*XPCkS2&!7Y~K8?HHzd5ZF<#1O65!W z{JQDX>H;@5IH+3VlW~nl*E&fnK1q;u3eDXrLJJ58sHm^LGAynwNx9z!!TVnVfX}w) z{4MP}v#;nTSFT+ph7!3lZM&N%6-g=n4Y$zS%3g(|RhTo6A3p|$A}<_-+WH>SyRy$2 zE7YNe>oxvQ^8Z)5|Kv)zhd#9FDZ( zS9V6993n3=t)&NMq9rEc7px&Z08?#aTl8%2l09UAk2m_f(r$`(u748IKO9$N-VmT*f4?7PluRhu)72is?MC+u}x7e*^!21Etf#$Z5 z_KGawzLSzt1Yw|Yq|U&5z53ftRdT?cYu_qX(?L}0zI;VlApL&Z^IG!0vmX(j+vzFb ze@*x50H?RYj7lB|#edno8Yc{HnB_^g5y3{Y*zfY1_!NGlSo=sAuX!t7Ehr-sYoF#H zQOQ)+;c<1Sw(Nq|^V+_tP>+ax^H0IgiGw-_P3MOmV%xT-`>vi0Cm0y=XfMm!?wzWB z0%U@WPyH^%zo*Uey$Y+tkzVUV?2OZOcOhJS)81Ytsz3);R$On|X(3e2+QaH(1T~hF zBH!CPBS(T`HL~=2@0~kYOdnF*@7C^Q$g`?*>(WNL(>{3!h=VP~tZ~OCzOTLEe!sKD zzh@eJ2oS*bTa7MqR;Z_JI6$?yM7S>PO;JPT6;Lw^dUX!Mwf5_ZXXS5M386I;R#Kk# zMZObXfODJ`F6$)5;vfm^DjG7R)mCW#aNyEw!!WnhZ|~fu7vJf5o<5s>X6b7=f}N%5 zsKAR5qFy2H5CuiA!;u$N3XKSb3#m<88f^t3?@18d2Q16cxiqk>U(lOwWY{*?iBmeO-q8FT&isD zixnJ!sDB9$)(t6tJ9x?)Hn1i0;~z!+_^NWZyH{>hv2k+Vtz?RB^VpB1H>wBYD2S7x zjU`*uHxyEvMq>%Omsvwn`6nwwmtF_A9wE$z;!hB!-aeNSJ~bvC->nxwwzN;8h-Lu6 zOT$&Q&ZxwmD|~i*eJpv}@K1gGDcO`O?M|xc6C`k7D5Z3J&uL8-9pwbvIUuyoE%hfJ z&8JTtXMk()-QeJxQR+4!L74{+20~Bl{MdNjn0yfrNu-`p#a#YVLnA!d8$3M96!FS2 zd$~NQYv2}CxsJ^a_OBGX|EWJ#iQ`7a6jEs^yo`34k zxb6AOrqt%oU~};vrvfKhD1ZS1xYy-kW+pbGHDG=hM3=HSSLv- z=2z{dUnK5DHH^@|`TTj&wWI6Qbgx|U_XDpWuf19y;>{3sJq^inJ^!}G1pqhWau5p zxw4SLC=<96kcl@r;Iv<31!ZfGPDj>b76kt=C{tM?ULQ#Myo-HVA7IiD3M7q$bEXF* zdF)Md4iCTczBXDAJu6bLi}xFx4#m?Ez^?=EtX{_1yeMF!`*01x17l!c^~$x<^n?!$ zus85_9d$sU`995)`%*V{hg*!{Of@odP$yj7TS7c=?%~5+Cx0`-%-d07kCZMf%}SB@ zv%t#9Kjw!lgHI6{!I1`QbTTndN#A$zOHAcmy0vOjR`V)j}2T?p=AD8@G9JGjYzp!nwJN`6V=jw37>#{S)z{JK7?d`}7JS-x;>AKmYP8V(b2A ze#LD5Y1YZ^ypl8;tE)MBxs59>A!f$Lc?qK49^s+A#NqyLZeu#qGtRYAyZuZXW=_Lv zKOE&Z`2^^eme*V6AA9}~d-`p5Q$PGL@d-f(!D=nSif}kqxe5A2*e^!2?CXbY75(0E z?B?j2tikJBgpB<8Xv}i4k;Os$t*T<}SYQu)+^ffI z{IV-Gi|y^G+A!pIw&IyF<3UtZS6c&>9n6ogRG(tc2G*4uyyW!!rQ$sjH%4(q%>4|| zTxfz$Mn)80{m5Am_1kP|=p?Dd|ljIRS2Z##c6-J4ZfrC~xP@D#IlmNIg$^Z_2JLX)hMrxJEuM z_72=*EkALerFbGjZj|3t}L|H`ECv)p0{P6?ysRfeUNb$QKt*S?ye*WVm=mIr!L z^ZZ-H%+h>%_r`bQp67HMw*78=W3}8)T*`D}(yo-=E#BC!mJ0$ISZiG-N)>+7H-hsuH>|l%jDD*JLDwv-OE$^9$M7W@qbn7$HQW%P0XrN zqFTNjJGqo@v8V0XUg46u*Z;Y8;Af*r;{5M%Qw{&mQ=xr=!~at`AP*Z{G_XhkdPj_Yj{Bshudl` ze$gvqzq)#Pr=P9>tuNsr0ieUa;xCMp9}fJpz^~NTfYY~Cuy#-QFVJje?D}Xr#>xr zzjMj4?AaBXdS7Z1(~Cya3{UAf_&s#`f__p3A6V7@Do0fN30(M^LG3kp!sp$~eEKM; zKa_M@ZOjY7Ye(*R$B+I11^J@AhMvvEGm=Wm_x3{SWr=<+Dt-);#jVeMzdlJm?4B_z zEK}>R^O0$37`z<)as`U8w6t@V&0i%{GU4I$+5bP6Fk0;2@W)7ni?@P>I(ln?zn0Q% zQV$}-c_^X(C9`H68JrDOI6E9(fN(a1vd&yP{k-=?eNOy8)Qhc|+#F-k7q6yx)Q%y! zu(i?KW5@8Li6-=rOalLS$PPzdxt$JNL?F$=>614koP%;!OWb6Vf|Q>6UoQZP1_-+r z+D?CK8-ZMczP>jjzJHDLAYScl&D8)QpHzr#Y|Qi}XjKzBJ+dOZjmjtHY3Z8=0f zZp^kZ|L0MR(gpZ^0iC1N$eY^sd@TZ<#9zvVd%2R3sQWV>I$p#cmiE9P9|@&M&O-WB zo9URrU!iz_mc1Ju-8MA$r-+G(0iERh9$JMeC*}dI@5DEbW{H;K{hp@EypIrEmo!`mtq1wwQiFyqr0 zz;9>()@6d+UTz44eh-~(zkp;w3=9mE0J|M3I-%{W z4#^5#HliNiIbJ;C;nSxt>rNXQ86_yc_TZ*lk@f!e9+IwS&lNy;8XO!9btzGxygR?_ zWmjty$g&moa%H5=VZgprDWj5`WLM2!LFGj*lz_E{p<`~o@NDK;d^|gFi-2@^FFIOxUW3)j z6~3m3!XJS8Gl&W(cnu1!!!HmJ#eR7Jgy9#})rL>E|NKumNp$QoCrjjxHq%3#r0PE@ z#{VS#4ml4OwH2AwzaTiFjE$qVwgO1^_05}zrol`HELIuH*711!!^Fg>R4HF8D2Ib3 z8olxp0IK?Ak9S?=RhpyaJM03o}&OS^|zW?S8lvOCQL2U{!I!Vbv$i&jZ zuNPruWxZ@%c98V2+Vw0Hmz0JQL%vT>>!w7zQLV^sLWwx-LH6@daU@539!f9&KmTKe bc%Z(xjc!KoyD|JaFN7*mOR-4)=I#FlZ+tR< literal 0 HcmV?d00001 diff --git a/assets/images/streams-and-consumers-75p.png b/assets/images/streams-and-consumers-75p.png new file mode 100644 index 0000000000000000000000000000000000000000..a6a7c9a083588cb5306e5725379fa14f05a7bc6a GIT binary patch literal 53849 zcmeGEdo+~m8wZR}B_x$}(h)gSl4iyjrj%xenQ=ZdCa0M(2g4XM7-JHJ4kE-Zl_IrE zC4@=`$8xA7sU)f7luCu9D2MlYYJb;y-?iSgerx@HzxTg)uf2;oJkR~y_kCU0=lXm; z_fB+jpw3fSp@Kr8=2_D$T~Mf5>L}EVo63stO3UjO0lfY=+=?D9lyL)tIVe42^538I zuxP$0EL_jnQV)xbjEpo2kT4^S0!8o}yd@L`1abp80e|1dqOs^rXv`)I-VKA+GbS1v z!yi~vBQp~`72au=XGeNv_Nwp3E{(Xd8vL z=DNseXr_A*m26LQGV%11(`=Yrs>t3%LgWdZc!3f$-%KR3mSUL3fh@8qf#g7e+gZDq zN~yMX5s@TXu)xffWkck-;UZj#Q4%v#dos>iCilX7+R50FCJqui0>dsK0E3YSxS=D7 zflf>lYcFe0WKDwz0(XWi5`Ho!1QVR1I1-^PALr%iN^r8KMA=YeOv_N%NF)d&j^gZ2ceHm7;xfEUT#X4{LGZc@9~b53=w>UDk=cAFMg)&S z3JG!xkl=a2R0qD?0Y1WxaAI1zlFS16A}>ZD&&HZBFfkQ{SOpM?5-(bmi4)erjKU!? zXg0wdFPyQEY08u1FkV(Bo|o`>qjdxT7q=>`4$o6DD#galNP-L!bES@aFQDjsvB2UB#l+dWI;_$#IF5An@g&HPc zN}U2FEDXUmfX!#y(D_!}KuLrx+l%Jz?(8lK31&qEl7oeoLSe8-=4^#01i&bXQKn(! zK+_0mFowfrgxQn1&MsW4t(%9nmyIxzinb(1@`UydAsCU3m9uNG9fjy-MhLMdTfyj@ z-P{};BOD?1P>^5kj9nc}=}vOEs?3fh4W`PitRqQYw$3CGQ%JIxW0|fbCli{xy=^3e zPUbs!SQ$t0EM2hzhMkyX%)_HYY^_XfMGg|CB_+xhE*Rw=Vdg6Lbhb4mdIiX>MQE%m z+tI~D5*lpD!Fn=notPd0Tq}E~NFWI1^Jq~F8~Bn*uz*G|wPv7goj7bqS57EB5>2K! zM36a5ybV!kY0C-5+FRoSgmON{!^$cUS!;ongB|?s9_h+*u}7O&hni90N*s4Q0kTpS`2E&Bs8pxK}TZT|f za5jMyPe%#7DsXg>%IJ6(8y4C&B9v+@1&h%c1z2n0Ho zP8NHHv%)1zF4@g7M9PD5KoNzB>5d$9IDAwX$&_>CA+8uco8*DBvvhW$W9&psR})iW zs05Aiw4t%ECODzBYZ%L!6$W2$rMR0IvyG!*zHL2%q>iC*Hl4K@u zcfOEqOt)d#vzQ#2mqTb&kUfLI51=wlm|;vb-ksu!r8Dd~6ub=MYA*?pcsa4i;aHA| zV?-3r#M;gdZR{2hfDQ}5I-7)e(9swg)zTx3?!t-mK#&Lt>q51+_vCOqjKkbz3^YN; z4hqLeOqd)Nhu|njhZ!3Ok;y`PM^j5Xf~N~sYU)A{bP_q4x$tNtCpv*kpgI_Hsm1{T za;WJP5y6rGd7k7-<%cjab}~<;Bqs@lVHtr%i&<{is0jFUWK@I;0sfZ4 ze}Zs|gK0R0fb;O+xx9P>KNE9hzN^x$0D6kY#xTmeVF)xHq=Lj9$LWLY#4%!pv zNN}QIolHZ;_y{T%XX+jt=3r_a#CH(5axKy52s?)WG1Cdl6GQ}i2GbdIzBIzji4}o2 z;YtILTf?2)2)Ia37deMO;KK_NM>D*H;ep&Qf)GMQ9tw67#Y5s1LAG6;ND)iCBwU&$Odp9zBt?+9QX$LLAy^zJunG$`b|Yc%;jVTVYdXUc;|c?GbLBJ5 zyc}WrDXt7FTaia_a2Ojlsk^OBn9R<^F)TQO=4BFTj9>tkOvo?}qDruV*49B@NJs>5 z7(!8Kn3M=BM8aZ(4l>tJsRM=#?1Jn9q5|BFo%vBr2h&Kh zJ2Wz;uEr~61h~i4+&UBic znUgVz$l)+7u_p3xwihW3?V`MZ_CYC^9g@tf}MR?P2x|eY% zQeTiNEn$S(I)sUkfFoc%-AJCM9D6SERh%o=S&k!kQkdZ^Hyks<*_7etfDK?s&7@9v zp);2$5eMPKE>R)m08`fx^I1!c+LL<-}?aU&htmPypQ!hEe zi66{1rC_bOkyLjHHHw85*bxA7kqGuQPa#qRq;xh*!gaF?6A)SU!9iFroO1w`EtS(` z5w21+-Yb+yaFyUaL%jfPk!T3yvLSQAkqs4&4|ZXN;RtqQ5i2Ar$WtWZdI{-Ka=MFa z2#e)JG3B5=Xmm5W4aS*^5k`jL;k#y5LGC;c63&$4DwhYDncxK0?yO*`lu0vzP$x&= zoXjvXH)NLO3_ONs8*b&sCcD~jxy)dKE7?tmq4OkB;b>bSA59Hnkt0LJAuNWojiUf< zhC>Dz6lxkwQP(#K@>_0 zWo=1xlb!nf#?Q{RqtbNZNSH-4!Lvdjk;+?=^s{!8o+c#VvWE5_IG}b{TY)6gJ8t#v z%;wtx%V+Gi>`Ps`SS4>}xB6!f?`~$7pFH#qaYwa}l~sN*{LbGzM|uXkZf;1@3Zvx?JKdRC5(cxk9j#m z&m1jsD=H@Qt9+fE7fK`&f9#3MUqJ=+jhUNspM5;@>?7LLw5{)gK~ zYq+J_Teq%kZ*L#_Qr3w=H8nNefBcvd82Ir}v*wBwBnD#zi9~wW*SGJ<6Q@<%qcsLQ ztNg-F7%rU)*H6pHh|kTLJ} zUq@_=Zx!@$Zbpw|QCb@}&P~RwKl+f96NFol`MPX2aYtC#$jOJ#J31QMdN$;WjDI78 z{_?IS-anaG;IaA6siLCI8tj+1!q)96+Lip~P=kn6X`-SY@Qfamrj`@-<&_Dl`txVQ zoIICHq2GF~|H;g}*U)gFxk^52uxDRsaCiA;Le`xV^a2lEJw0as7mQDT|y?vBfKxBo6rez+$?0^2;4gT?mC5g`VHfqRo%z16JXm;Z0Xh9Xv zuR{=T856Tz`t^0a+mZWwjLLS?rc}@$%)nKfZJ!{t+F%NJOPlm7YF* zns)T)zNx90fPerkW8<*8ZwKF2>^hK~tZGT2tTixDkyotYmY%Tib#;yZ{yp5gT()rD zym>Sm8Y12Gz>2je^7`?6A{d7$Yal50cPk#H4 z_I9ri;aiYVU@!||^gA)MYh`6R@Nq`L(u#_T`wt&>cXZLUD`CCiF`Qgn?)H!V`0-7+ zkX-ZVk*dnvxeJ#r-Cte31ER?G@zbq}bKvpc+=cTo21Z$ z_W9NIB#WZTO5bmE4o5H6P)BE0_1(&2$BylX$$xz(a={(xm+rm0_Vt`iHSMK5W@-DFWZ%AhG+W#F2LYsr2on~I^`xU?F6zVJAl1f3W#-J8?*<1`dwT=k z;vPANg&F_&+#a8nrUs95F92I3xc;qy$K^s?t1Vo(_v%$NGRybw-Aj)^W=UInModi1 z!PHc$mDOwtg)-J7y|dc0(3=<@j+>a6xGL)0{`#hP*0X2NP^YSRC?^Kv5UeiF#AJBz zX`#=Z(BD6V*5kF!#(4bFg$ozn6!$0=df$qJeFht#rSJD#Lxs-nZdPz-M_Q zUW~`Jr^5lau$!)j4DUL5?AVQ7%mR9h#mW^c?(UtXq^7POcl@}o*SLH4u7}}*t4|(#UoqR9U2AijzC7H#)X<@(rp94pRqUhH z6;UIUr~lRs69m^^U0T&qCQ^F$x)&B`rsUSDWX$~ltew3*^Y@Q04nG0UrAud~DJIqxt;=|z z>$zd=+O=c9uVwdn2c-@D_@T&Fotm0*NNxWjU4PUDT@N!b^WD$Ef5yhK^-gYX37MPy zjBe#@QA~vSxD#!@ZQE+p(A4Cv?T*pHhQT78;oaivoeb!GxPD!BQe06(! z`-#ERTO-fjzI}39tr2F{&hE_l^ZPc3egEL>`*Cz+B=(8vfGs+!35_xq-miIen%s;-Mf#X=`gwOv@``+;i%> zg`WAX_nVtDCul|bw2N*Aix)2*QZZ&H=U{y7XHT}AqozNsFXc6>tyuA}Q)kBtT{nXz zs*1DDo;^EeUYzj(h9NsIlFQ{jjkk_z4UdoTrcOcH&upx%t)<;)9p0*^JQH@Klc(o_ zn>Wq2TVS#3P>$0dW;_VEdtia)n0ah$toJR+?Be3$=o5v7M%HQm$u%7vE9cCaW7POw z_QMRa8G%4ZM?ZW1T;bhMmF%0VTUuHmg&bfMyzdZq*UmX*9+ht4m+ORtQ2Jm3R5^%I z$SlqM6)}@5PMkO~F&Gy&Q$tm;at-Fm#L&G3(W{$tEU518OJX1Io4d^aI%E8&0fDd_ zc@fgO--W8G*mkWNjeEuH1}AUtN1g29u-M`5eZ7;01_tR0-_OS-^ekz6^vLeowQEB? zXKsEypCU||42c=*QaVKl+b_x?<)XCasl#Tuc=2M|=V{T~@)_{E?P|7Xn7P*_JKKVe zH(Fgcb1LMchpKog`(`d#vgH1Q2VISm6C=$=ox4btygwDtjJC=wbkFqUyP8>$@oZzv zzP`TIQS>KygD{`p5iROe!;_$&+W!Y)`Kl4`~G1etWvD^ zm46Rv6DRg6q7GAL)h{hPbFj7)ZuY>ab8~}DhU~=?jM~8^rAf;+e~TJ?Td}ib5#Pln zy?5JFKJIug zon5ygEqlY#jd8G8{f!^Pw^Tlct@&;I=U`{1+gRO~)oa$&r>x$3mEADkeD|+^l++JX z?q%II^WB?u@+65)*O=Z4h2!Yaqc?Z`G`fA~4zrl9yPFq)T`qd~?CskW82_V3j|^Ap zu4;a#Nsw56J8>Eoboxza=Ykz8)@G`xL-|2Ug+L$}>JU|ReQM30s93BpqxxwBURQfJZQb$Cb1%UL)#O#xeTgQ zW$eLqmQb;dy=j;;z6Q|pnvYYHzma&nrsKCtcdd?&&wf9BPt=z!>;EP$4ooU5Gu_i!<(iT^`Yg|J z;=1GMEp`2%OO1@{XUu_W$oOIGXk(MEFbDI<$ST^emg~88Sg_~iciQX0x~dsx3JX)J z9Cq%^YQ*~84!4&4psUs-ZHkZ(Px)5G9RBr7o|&q*ZXH`9QAg=|;%lw-Glnm0ZJeE0 zvFqozZ^&eb9j{zXIvVTj?|-IoF!O#$na;+IjhqbN?e7l1)Gyq+em(zEYHFsfxpd}^ zojX_hs9`@NKH;UEt(rn!mywc^()gCT zhetGYSN**H@nc}+>mxeUvFL`MbpcrG{Z5Y?8^eA*+PQP59&C!7xQ(ndWmVO1ucgJE z`)+n7#m=gZS6XoD?dBaT&`9h;ip(4Q3b$XgO`bqaUBiryKIQjuWOOv**5;($KECC$ zy;rUn=EYc~H0}KQwjv2&oAsjY!*|NO(*WW14A58UI-D}=3J+F@sUqD1r0}|-ySv-T z)%D(Q#!VIVlm>h;;apE-8>@Ker!XewZQn)vpn2-~gzPI9>sc+Lc{=;I=eZW;q`JEu9fy`2o|uTPd;|}HGW33M zFe!-)fL&2ZNsmY}zu#FEo6~G-ZQaGwx7huwcH{2u<%|Z+wZ&_1KTrK4c&L+Iw=^rN zeE08T7Ch)yOzg8fR_Y!Tyuv{^H9y~yU0eYr^>Ufdl^w?Qp@VO?0g``kkpXMQ;c!qW zfHjvE&q%wHUskpZpJo4ac^IH(Jv}`qgQHIQt;e&n;@^Z&t*z(y)d%B?3}^=@J-&^M zRM`;PR_LzEN(UVGZTF3$BFCpwmFLgT-ZP*zVStKHSTD|6 zw0mr52Gl|1*~sL3C*u;v2!OJBdLLv&`vMq1wJkAk*s)^8ibKQ0V!%5#g;Vq9&$mh0 z$nsm(pz;Iy;_*F-aau!zWoFnXU*Fupn3^h`cbRkEMvcoB>VlN;NSLU7h-o_PRva9+`V4qEd4d99xCeaX8KR%v5HIQ zNV$8OxH~how7(YpZzaEsonG7-*)SP=OABLR(VV%c=u~?!{@=Vj9U~(L$?Vf<1#w7# zWn^U7;xTGlmsQ11`${P7!;j9$jy()1vwdZt2{_*)-~DNgi<*M<0d{eTGHv5qUteGJ zio>>3FAFc2v+C^C6m<3VZG275%~w^|D<(qv-|8L;=v+k6@`$M`&FS7Y$D{O1619af zFr_(Xpe1=wm|1HM!lP75b%C0Bu_J8=mkGVm=Vph0Vch%PFE%WEx&hDE? zU-^_El}SwqzBLR38nnTi@66A5ppez1zkEhxLj$#B)46l!g669ybuJTMLsv)4UXb=f zk3JDy{B(<{>ALa91nI>kj!rmJQ(NCQ@t!#Hf%!Ai@TnssBN+-c&?V6gmp&Lcuy3Dr z_6XDu6Qw!B`dPNS@wT%Q^F52caUf`PJ6n#%C9wT2)HT=+Q<5P}3R8;zH^0JS`mxDu zqh-3NcOP2LrB619YfF#*5R)pYi**k++>V*N?W(uYz~E41t->A%W5VP`MFjA4rA=|yAJvd!^QBqQpc=A_7rp1l?4UUnKW&g6e2L_Ua9XE#C zZqN;1R_rS4Ra%6CzP9E#j&^Xs%Yc@goNP8(O~_udq@gYH^%}^u#Tj1`7p+6ln!67z zdDa@e6oC9O^EIniHM*>Fw zO*6wE5SLHKn>kKIFD>iyYnDJ4u+g&+g(@j6jq2kNpyM&>=r_I*+E-oNepGhsmDq2V zR^a8e4k-iWk%MbG8n)+~BWZt3)%m)ab>0^$sGg2a zqAOsWf9E*yczL=z&^OSZo&ylo&FSk_;YuV}zh(#=J)NjT*XU?-P1>2_;+DP-p+H?I zUX9jFoc#TBq{J+H)3S=#={0a4HHE!7|ER^=*x4mF{)`L>all4jv(wKw)hY@J@%`~q z3am-Kb#kH5q{ZN)65zZniV5AEUCx_e83eab1KP8c5R zXr8}%OYiqvhvLy{C9S~9tr3QV%>VM`%T0JtQSM zt#2;Lpo3ezdNn^Jgd!3RrkgG)p&M$fS#vDm`1varpqtv}V0!b{nkzp_fgZ_xT^>Y8 zSrYD_JBI;GN!5wxENGWyQxoGQrUy@+I%NqjG8hbK*$2W)i>{BY=$lZ7HTdx96KVJE zZGfKIzHeEYG<$*Oc7Xb|C5=#z=R>{U>{t2EYxBpeS!rs6rDz;3p}#rDc1vMV{qyIG zUv+c`@#yu%bU%VHJC?CK6RxwPT-`d_jC*#4j?(JPh zo((TsS2=pP>U$1k=j1d1#C;K-8R_3u)&ngK0NFEHEk`yXG-vI(O@(K6U2bx5aJ+6t zQ)^pQWbo_PuWsOw(%+jD6cnuVu_WDJN~)42sHNWUGWd{n{#5p%xY|#gfPkC9x24}d zw3zrUT$_+~{fwVE=^%CGnl%(3pVgY$4_$@FB=XCTA3Hp6ZMFVIPCM^a;9(v;rUrPQ z9~?Yklx3q%GS~e7l}Y`7fSmn*FZusJqWk~Lxx5J>4gKYr+qYK$vUno6juUo$%=C-d z+rqB}jBP4!(@l_R{h`@WsOhl=3*gDlv5~AM&Mo)sJPH8uFba;HG+RhQbc6dI9&ovnCr=(W+Fv^~gqDPb9Z5^u^YG!r z!$yx!9&h3zTp+h}2I@jy-jSS~1*r9#Hq}9Uq?geiyuZf(UGru`!-V!=MU?8idD@#d z4`zw(xzsl_hBttJ4R+dt0A*!m6k8Qog_S;rW$y0o zz`N(fPAx~FA+Z2JLOtQ*;6_$R`v#dnE91dfNiyd1n>));p~Kx)!5~kt@d0z-k%4ns z>7z=zulPq9+4&qQI4r_2)b+zgD$5owoIzAWZ`(Ex1??P_#afGc2Sg!|VEY*b^}sx! z4Bbj5CnnSuFILz`+I#4b3aa{qG4G6~MP*Umv!sw=?PbEDg zL$3prhJAo}fE+@~#cZTu&l7sCUo0E{rdQ}!l>+2B5J0Hv$EUaId2TeXfkBS(b||;s z{GKnT&Ds@F3+n?J&wbS5l&$i~-`S3*;cDOKwOR zSgbM|VT1wY15LPcWm0zb8W0n-^z|t&ZvjXBd*Q;_D_2yZ20IXui=1~M0)}o#S&dpc z^WJ&e%(Q?0Nw|G`%h>PVJ$;RtaLo?`1HT^*mhVEI1@&i#bD~&j2UAk^r=~8Lz5`$$ z=oz@X6(_W9qdB1tF*>G)SmnI zX;3X^pMef##VzrP`ifvHE>Z5X|B+||$0gT%Ys(wermWk5cpYqhkuCytLt0hk|& zSqyRW@_Nv+e{A$FT)2>ynW?Q z73UOhjJ3tjtAnwsp zHVb`+Lfuq_Mt)qmD^!kqwCMK@P~RIEP~Ww zop%5J{db;LmMO%UsFYQNwR6;g*HqaT=Li4L^G-|TZPfP3@?ck@ClbD4INPlwOddIoylyeIq;l3X4kj~_83Km9j!veQ6<-0zWpK5&U^RMa>n z*1MMcH!DBPh^qy(3Opv63qH zd$k6c&DMcoi+_C&giZ7D*}8xK{Z~s*Z#HmM6ktbTha;prXjibAUcF*$KVf%)(-r|R z*V57wpq&yV&V3M1pxv6rYI={qjjM&a@izr~nCtj?cSY>*#eLJC6R+~tw6Zyl@K7=%`kk8*FVffKX$bagS|Id0t|YUtd6wW~wgA}DLj_&-05fdX`j6JJ|EQ-PjY zW97=8pZjG$J}(D4phkln9Be2bTDsBqe_jB&9KtFAp~!XZv*3Bgy(R&%#M~ob_5^S^xsa(KIy=J) zi;EXRg#a;22?Y~v%zJzW@@z5&G{sI%)SB(l%6@)+56WR9!Saqgtb(%sr?Nw?fc@`4 zJbUZba%X4fi9@v$6XrRvsY(W5qsFxgKuTADiZvga_`Q4g7LClf`Eggioom_|h|m3h zBHuaNO&gLSBsoNWgl-A8?vKIF=Jby>BF*j5V|ze%?-}lC0B~Ry zGhqe+01HGW&w@EmOG`ufYt#NYdURN;<;9ErAa$ME8aY2_`|j?68y#OD|6eRB>R&Pi zH2})l78B}B6jbvgM;2PA9S71mG!(6>c<*u+o=~`P`}Q?eQ@=K%{GpkHMS$cz@GU0x zQmfmR-_$!8W7E;YHrLA6&5Fd}x83gTS*-iQl=kS6ebLD=1yM`NKR0QU;fze7KRc*zYZ{D>MSd!`TaU*4HBXE<%G zEiE%NH8mkwkNx_^g8k~(Q{b1;XK;G*+#9o>Kh00w+10%=QxFKh-oIj9J=&m{xi>DZ z_LoB9iCY_9k5lJKL{l^(YnIQh;Of}c!G?^s*y&G2R{teZ74@?~YUYVQ3Gpj0T=33m zztJ8HF_iIWn@9!V}*4L=#H@#Cf@_rszDe#|1eZ=k`7tgZf+RwxWsXwuJ zRY5!-_T}}PMdiUsvu4ju+&ovEwdek=TPshP%vpcMaq^^<-bRhAHGRM3%-yd%=Pmw| z8vx@a2RaT}2fY^`N=xNwX{|_U3Sx4}t`x zn?GJ^wYODyoPqtPteyYd{fF^a&0iLsP2mIgwZUUyZ{&S*Pl>J7QCr-pf0Yn*ki7P?<(PcY)t%dH(#`<^KNO z-ZQzmH3e;ke}Q^&>~stwXMQCYgGf}^+u8K^QsA?*JDlk`H){04_3^@op~`CZ(>M3R zju5;`xc>8q3>%uMq%vG_{(}8V$KY50h;_R16_ghu5W`DPMLpYgRmI2eAD^MBslulp z!}H~pq}_X}w#8YJC5 z$><=r?A6Mr72_Y4M|S&ud@qCiHU0R-Mt2sE$Abn_ZT0HI(59S&j9z`WNvlpj9+dk7 z9{E|;d)TpnYIBjK$4eVevGc~yln5_(Ea?9>yv>| z*>Lx4*tGC=Rp4@E?M_o+1!0i{HPJ@oUZ-#27BKyC&v8X7&cYoC{@x<~%e>DY~miA@QR>68+W zLTT1nyLRU2=qOT+0E7TJRAmj2^#3!=x;wbaZ|BZN@KId*d>dRV`vI+)%Z5;(P7MtY zkH3#-+q@9GC_wWH!!ecSH1MuCyD&w;!3n-aR6v87h4NO!>;w>oRMa+-rMUao_e2Qj z03YYHA#WD{Zy(%Nt%u&f=c=e@Da=k3t<(kcg0}KZfN*2wB_-I}x;ny} zFFvZjihL_b;QT>^>Cd2I0KDlZvWoj&96fOYk@Hwm>C$SjcsJ+|1EWs`y*4^J8oDVZ zFg8J}ny_yl;tT*K6SnZsw{M7`d;89^!kNAA-;*x6t<};(LA68f0n9D5Q%GZT=goXj zi~#C`?9vTw?eP;Q5a|v8k&BB9dh6DKZ&7dFyis1T0PF<&Gc((MF4sk!Id^UrxWb?x zNk82i^x@;jy)d1DftPw8pF`@yz~Q)r{`AJa_3PK~pimSw*g&wpCw7Bm2R#iW2~hMQ zC(cCm_Vo={E{@ZB2QEJ7J>wY#V18L~cz@29faI=;@sael=X@N%!4s}tSS%Kx9XJLn z&S&9;`ipfdg=rVBT(JxfzZUQom`kn9#;86EfDaG^unvD{{5-AU4=p-Sd#6^rUvi7a zYN$3A4nTlGpzhaFR-QAb26nD)Um%~q$K_Hj82dn5hZvk!+OA4ed-CSZ;gpp53bSW3 zgOtR^lHNpBk-2(E9L zod}!M(#mRhqdKJB=#q{vjfMI7`Ewyb1G2ptGYS38iGLGWfc(_e)lqwXUczQIL6)C| zTD)}W*^3vIt4VJhAeUafss_%s^e^eaC8?@fIytG8S5$~cKe@lY9WfWADo}a%!Nh@@ zdPCTuc*{FQGiGcC^nnh|`OvLtDNksC>uh+p(j5P;Brk7o)e7ILuNH35H$oNKKs;oQ zp_#`{t%4R8oEm15US;Tr+}&F#%aYVeY+ws|4OPsZgSvR>(%!#~Uqr;{vWhk^z(Jz{ zd8YiX=JMr;r!wz7t@`fJBd{i>Yhg@G_Pu-Ouc5l%-Bc*?_Jk0LnH=>*Rl^`58lc<9 z7#qg}^aEhH<6MhBgktKNZ3ujQ#5Jxu>kR%KFfq+UA=>ZDmvrPo6B83rdw_}Yu8dv| zN@96@S&cu^xq)6Y0|im)w0<$-vp^X8ty`18{R50+P2ULI&9_26JF7`?A8B|J<{)<) z1Ju&*^2yu5SKov*0FupZ8;DvE>k0FKWY5i;Q>Q0kufL@{3C8QJU#~oS_H6KeuB6+L z?z;dxQu1z>s(Nq7<$?kzWQYiBaFk|e2V4G1OdxBXJz?oq87hPos%A#!llCq9?E|3LatgO2Q_x={VJ zGaf+5>5a@LRznW}trx=T&(-A{ZKiDs_foGD>H=2ZQnOcQN>7w;Xo%^YkqHe7nBB9$_A9>+2CT;P<+pwSiF}wQt@v z^lk=Nt4iOuo!25(M#6G0U%nh@-PE7<@d{UmYQSPL(YHBPyk%?EKj7YlUr%j`5VZ2% zZ`(uz8GQz1c*y5|D`14YX7<{f(1Q`6j>3izxyNk~hB z>ND7R?&3v$`vsfsrZ2UasyAJUKz~Ay6AI!0B;ce41R0<;#IMN#$wW<4v-!YE_@^fo zjB(HgRR`n$IQTVJt?D&(WhNu^vzM1wd4q*^ZzIgJfX@dRBpD;ywQD3-ZPu(=<5v^G z3=Q<;dm;U%Jj*65D*8t2XNDHSK#>8Zg2#W81@{%i9>&R|GR?gYfVg+fPe#RW8Vna3!gx4 z*}Qo(5Fx|EKl1YO;7+g$>wmvPW}>g}mN)M&(VYwh4kp|MXq4#Ud2`jPzfG=roV{%> zz!ZR4g@JFfSLk|g*tjv{+NSN&&l)M37Aat*Cmb$a1zhPLPNf@o-9IY4z6X>86D;?u63Fp(0M@J}791_D9cQ$G)Jk|$$8@3x7vl*w-D(%@?&N8D<+ zE5M9{zZ`HQl>0a>;AEid*?Rb9$;k`$5IPeo1@MgXET$)nVFJf}1RWh6#k}SXS>hbaM6OY?LP~ih+1c?f4GK~?6vu6)S9Q)@Vgad>ro2R}f z+^h|b2%zX@J3fxekK0`^DT*|nomhuxgCH*fD+v-32sQ7(8jA4!(Ytpu%l@3=u+22; zy1Q>K(rN+*KxvtnxR<;E2@OH4Q1_w1fiRj42?`LS`%h=>1jCnhM1Z32nq#Mof%T6I zYAYlhR-73fl?H&Fx?=|l%3;gPm&)}Th=UF4aEvGambcLyb!dSY92}C{VOmdrZFmMd88NqyKqZHi1|i|lbflKK zY(`)jB9Zwu2(s{r0{|N*K&*rI6XY1M^8jT4JMt_{b58FDL&M(Hk9RT{PCIud0YLaR zwIDDs5Ihrc0K%bzgd7Fd4u$;Ihw*Sse$&L{B%E)#*1KiFDzj?G#}G6BZOsMhXQ4bm zt=SHzQFgcd!iS})fqmfG_3PTutO6x!CL7X%iZ;e~EZncND4s;4DT8|21(r={xMI9s zb4$DK9asPz-VQ6PI>q26+F5PTo>h~Mt7Wah)DJaJPc(z+yr=8z_3NWaaUi1Z|8qPD zDiHJ-ph|#R0_h9#10W#eJXTQ9M$P`ynEe0(LCG4{VqRy!&aKJ31gCV6&iE+^4RuM& zqV#;O9OpqAe0c2~IOrkmdgOx`7?OOnbf4y?%XAVO9*l@A=@neuU zKnR0(08l4D0fdK{Vfg(t5^6R!Hld%MZ$Qnp_eJDVH8m6zRS=0l1Rlg+e{d0S%5}dlIzMGnCr{Ckb7Sw8g+zL~f zt#Z3JekY`9&DaTNb^WQ2Ia7(#)16aaICc6K*STspfWUIpXJfnjQG_B-T|NA~e~v#jWVTFI>_w=}wAXIL64M$@(e z(kp;+;2iFP7j6Dtz?K702F@NCMVZz@;R+8AzcSV{u?^02X(Iv$ zAP&^rCAo;j_K2f-%|Khxk<82ka0mx>WTt7wLg+G>Q0&@Ag-6AsRTr9;>aMYfQT5NA zSIIsOI%I0>gbuPf(N9;_1Ym(ZK6~SalxrRi@M2S=xFw^)HZhgMNv6A&z4{6@5< z$D<*(r)YI1+PMEMx^VaI-QdxA7u#7Lm0`ME?Y9$Z2egYIed{(a87yB7CxZ6uL2W;Ew}CJT>@)Y-6M4nP8E73zVms=R5QU5lZCcU}u9cI=5{CBzGebQB<> z{L~|WrVcFv{0j#Mt9r}9DeyLB9IOPWrCUgsfqeh{n|!Y23>03l@FQv;QfF7JFqjjZ zV-X9S10*0gDnx+@?#x-1S#bICp;YO!r%y`)8(KS6VJ47+kI9&ckB+f-Q!Sq|Cp_x^%%5M(DF=z)+V5!Th}h!h97#81`lu86T7yDg0SIf zVPU5A7uu#(s-fZTo?dFVEo3$#*mhvPMJYThEs;n9l$w8;GYrD1N;}XEnAwhPMu1C zrGyzu^|>P*b{GaR3urgsags3zWCysCxm3u?M?(ma!8(nvn5ezCK7Q z5dY0Z3C#nC#l}I_hJ#_7+o$)bm0Wj9d#Pn^H_?gGc9@<#qp+I10KLt0ytQ%YQ|m^6 zB{NV+=l$vxK+i+xsHsSkU>*PmuaJPbb7y5Tgegu7G)zc8kTuKWw%~BR&z-Lp6)gZK zpWp7Cuk{oY!C7vHg!%uff3_fw+Y-Yg!~)1BY;`?jk2oMU}Hr zz+wX%_(@Vi?}-1l$^ z%zx^gbwoeE9IDbB)L2l1-T}3Rsh~{M7Eg~)kCo?Sl{$AuEZ;qN$NzfM?$#1$nzi7Z z5Cn(@oRb5by>Q8reSq8oxZG|qy#gSr`_l?oOnzU4kbL>ADu5cmhzP_*HGo`rtUpBZ zsx9FMO~;3IS$~>SU~IpA>mC7r5G>s2K-=*5kFJ1{&*bGPZIyoB0h~{j*aMCsJSzvk z5-6+K6KO||z*#y?bu0#R#FYGWVAL#n3=K;L4J#O}jFwvkak+{B+W;cgJ$a%AkpWJa z&!58$vg%lSp`0!-%@TS~HudgQNQ8D6SfjMaf284KL(I3{4^EF3+*sZYVGc)(R0}+k zKze{?@EjZryRc~sR6-=?fZo{$QwxdDCs8Yb7A;N#{VN_=?i&B<-dW_aSd<2bCZl$SwTFU(`D-Lmlq_l<|+akK`coTg!BNqKfN`uiLBknQAJbq9>0W zeSC^A3v@oP=EAubEeJbkIh3wGsST9LG(aMT1`W>m&I0xcIC^L*$I;gS<^n1*YMX|T zm^b$fiw_znU{_!g5J`Se2O@slYnpsusX}mtd|b$F@u+D=(Q@4z zqRy1q$+p98wwxHtXQ{VlF z4vYCMr0iTs-2)!`ILHpDGaz0tnM~-Vt)Qdv^MI}!(#OOjDTpSpFM!t>{`zLCoui`; z4!0QY+WGJ2Ak4OHhe5}DP!5uC-lSmIeE34_S=h{QG&uM2Wfjn2{FAL;oh=G~Ftlv> z|GWSma9p`Ikml>j$vS?VjyW#8As*e-QFzs+J zf$b9_7Hyxe_49oyXlRk`efr!6X3S)yw{Y`;ucF_wt7fe-jj}CMD=S1ftdOYzY22Ds zt7?HHMxme%5(5J_#QgfW4hT1J+~r=lpa|z{qrZaz5wTDqFG1f9gm5jS#~O_uUb_Ww zA3h5#q5zoz;0Lz>91FiyzL78-4Tklk1Dwg9vC}_`oVLLKC!n|EtX3ECEu8MCsFasw zk9>Wm_I0%N+_x#~-sfLQpw*Tf-WvLKFpB@Npzg^_+m~Pb20ssi9S~R!IBbXp~fQ4CLlo0&y zbvTpu^=a&{XE`sPH}8f2O~EiQPnA0O-{!u%KUh7p2&sN@GRcs=7O7ynMyPJv%=R6a zwD;ri+W{v*<1~w|yp+W=Kkt$VXDyJWg9$=~LSqL&4~~^D;ypgK7cjJzGISgbK=_Pp zQ$pxfkm=vmA6^M5vrlKu!jdD$RMY|YjWzmB-95nZF;^G>ues0sMjKedU@i=Zozu^2 z`Tb-0)C;eDPcNvxRkexzjY@v65CwzvIm&}NO0(Uzom3^-u=J8IcvWIbwHAUy>bv| z0j(x5nE5eH;06nqN((P_0`a^A`Z1)=0`8dF4*oUf^wcC`T0&Z~U%&Qk1gh63ItHvU zA)S>?ZSFw4X)ekeA0O9in{D`dC31Y^k3jtD)hD+~U_Ayuy;UfTtA(`QW^T9<{x1_H zC!Et>k}hnIy<1xI`{q>d=Ji)>eQqxIZHhfxZk1I3bwD=-He$!F8MZH4Em= zLri(V*~P=k1HfJmO(d8ws6f$*emVaiti5?u&+Y#I8#fY}5;9gq85oq)|&)4%>yx0qb!R>@bOi^WJ7f^wI+Adv*(yW|WqqZZSVz~x4fV*xE`EI^^d6yKq zALw0JCc{uR8c`Soi# zXjgOR^FuJq5QCplH(q#+Tf6;@XUonx?hwF2eUQ4|_uJEZE$*e$`seTL^J{&$ZFFwK zm8rzhl0jOBNtO`QAZ-$mvL9|0B`tw!eVA1dj^a z$L^s!YbZuQBfJ*=AG9@;$au%kUFnRVc|^%F*=Pfl3f!KMlc53x6Ja+%*Mg}A{r&f} zxfbjXVD75;yLazy&AdU0&g5!!Dtxh!Em51#P_x;%aX%2{uDuG1`D+qi%q?e_(w$r> z?{p0oUezMbM|%rq2m;<^ndbc(qn+yZE7i?#-|E@T1uZrg59+g5T-tPyX(K^{rZy(u zJ3HTR{Kl7spIg2N*@vTB?tUBZjH1%_lZj1vUTfCr}71|(+WR6E$ui)-XUv={rt1ffPMB2`wM>$iK!>9 znw%T|fZE+ucCXNAGJgZZQ0f?zwcg9yy99Kbr^o&}4EBDv^8O$cd5pFq7iO_4ju>(1 z(j}$H?~%jLA(-youODOJJv;CJ3)(Z<^hhsNi$hu;XN6?VUA{acXxc%FUS;SA{zph( zG)RNo_hgppUwnb;&k7kL z^2>|(Y4NqFyjL$0Zss2S@neh0=Z3&Z+YJ^kesjg9>*a%)r3BW!6ko{qZY~t zzBfPa@m^xP{ag2}s6nIkUi-!D)W3P}4Z3yYyiYIB_G98B8IVW?aJv&Us^-T0G-}X1 z_dP)>h~WZ>SQlTcwIAPay$|ee!YYjySy|KIWS#H&vmB65utP1guFXeqF}0Hn^a0sc zh-+{Gc~M?NU@vminy};N7%E4*ie-`OdvN2|XLOU7H$f4pwpnW~{{_wK+);zOFY z#hLk;=V;NQ_8n(8Z+bsXyLV(&>CXE-zJB#;JKaR8Lx;J40oo_8$(dm2BaD3}`zR<5 z`MY9A$GS2*htho0&cAF29qp~yuRhmcZe{iP|t_+e2bKeTM#N9jHR!~&jJ!yL_65m%sMJ=)3hbRkHa6zR*&x3n`ut44y z-LY3Z5X7%ANByEi<%^7hWal#r;XY}=*TFTMsoiVBrcB20UB;nk~ZK2k{UuUg;G(CJxT)9CFQ zIo-~5)BAE={;})n#D*d{r}8(C4*6s(s0`>tGMFB!gF8%0spFCHTk4g+AvEVH7zb!y z&97W?=vP(E{_y^Ca&q>Kvhi21u6u`n=7tY+q1>z z-#O{gMvC25^p}*3<&z1L{a!`R+gzSAMy6wElAgZW(HwWLwDxE0+g8TF==X_uGs3Bj z^&FBLNVL5A$g3bRPoh_!P~a2vTorWnj6!bhOYEXsRH>NgWWK}aL+K~42~erF$x)$X z%*^Jv|K9VuOtx-*+Qz)ZsjCdyPp^*hZC3EQJ9a|%{PY>^PZhK@*KK%xV>cC%YQU{A z7XR~iFZjFrp`G;CY%J8<)NhN;X6q(B8^y`f-Gxh3)oupEN1oN{v+A~P+Qk3-71mSZ zXZ&Z~?de*qq$@0^>Zgc5I^_GGN1)eU-T43eAEmXTg{H!J`S$G-^2}DyT{b4ddI&Wf z2qq#5Ar5Shy;^VZH)A!`b6e2LY<^x^n>kXP9yL+E#qb%YK(O$bXl`oIMegL8nIl^3Xb0cbYNctc|zw9xyg$sqL1@*<{(9lElq`0RF zgC%GZ_&uSAf-jul)~u~;60nLAZcsoYSB77N2ZbHd@QTuCn+3lASr?w`fF+&)_i}%Z z<+qv2Jr4d}^;9)}&iwh)Xs`}NUxzJ2BTCPoKpQl}*x0$F9P|O-R)xN$_r}}}%Kv?5 zW|x4)XduMkN~Br9@#FeTjxfU)937pLIlCXcNq4%>k|!sIi+T-!9ECABblleJl+1L$ z9!iG1{I2puBFhfg6?sE=gf1BF*6?(>imGbx-MhJ&=|2YwSBQ;njeb94G2i)_6&^yw z`e&JYsy^q7E1rd?{)a2{3K_kN+7C;CD*d0s#Z-`4)!W?HG;CXkZS3zR6^Ukfy z$;=cpCFtO2G;%W2tJ+J+uo{ykd4s1fF$;fWz@(YXlLq@~Toi*eS?imI1y9&9Qg9Ce zwDyc9Baihh9zL0Bk5G={FfUW*q@L$uLzc6zkmF>n|2s4^EpzsnO;BsX`GaTW5<7IX z-F-=~+FECnGQoB|nH_vcB>Y%^z`IbVBZN~xbGIIPX&PoSN;7B8y7^{9L>WYBFw>O7 zECRX?dUtnJzIIw>o9)xN%lEsQ{jS zDZQL_r=gFMU|yG8XG~|NvZ?D!uz#-u2Z}%~1MCiTwLDp0q;Ql1HQhR?&>ls`k#y@p zsO+I*uC6V(Lp{Ig!=r9M@nV6{YWbyWUbOAH94(Qh)1wwXWO@3pLM#iM&I*?n-cavx z@*etVPZ+!qli{9S18YFj!gtnb@{;h;KmhVi>kBX=+f=4|_xK+#i)*pkfw~2IFUxP@ z&SaF5Z*OWV>Q@0bWvw5quD;2_SV|BacF`f>qbE-E!ygJt@N4Pi^g9i5O6Xj;K?c3I z#}2hT0$VzG*s!vY28n6ytJawfP?8W%3fvZ%4nRtE#{qRl&CowleP3DHN{!&H7302V z_!x?pJ0p!iT2t+!%}h)XQp(OwYASNmx1yr1u4&_ z@%<8>OLk;rP~z3AS2v(87ld1+!xZ(FlS?SHzvA7gOP3O&gC4)$d|OlTNln z%EO0OFI>2D$0dy&pLy|XzD%!PkIn0AR+e&da)Khz#**j3}AIc zebAmn=|_~;@K!aD^ywjj*>F*%`?fRjvR z>BhvIpz_3teSIG%6Z~LzKk=OCtt>VD9;c;!!14|~43xNV7h_K^$bIwf-67*(zuLN_ z$Vs7Hi^?^x>$cuE+3Nsn$vR>a&q#HL7M2)9<`nA)I{cQ$)!IFj#Dy}VxcJ_-vA`Cl zz5JDxl&%DhvMsppf9_mA3BDjGe?afvkBRA7HvK1Ub4*q?GY+oy&1nrrU({F`zb5nY zYzMo*76Kc0R#9qw=%CH|Ove9-kJMTS4O5!x8P{`K#rWx>POZ?>1ink%5x0N1oV3n9 z$-|Ei?V&GZ3O~rM3xnn(%=|vJ%CVcBIYDoZsp;+@6+uYsf`$Ob8LFZpR3nfibV7@x z?fSz!F=kQp@#8}>#fwr$*OsQbgy+WVU~I212ysrNynO?~EJ0DLL|;VNPg%%2(GSkN z2?<7NsO5hoE>0WFR8R=;18fOm=T?|6gNsM;fy@qHo&m^0{aZwZURI8%W?D$rwdE80 zb*74BYp(YJz->*Ox{QX)+JXo3=sIPb;)>Xh3@z8202<8k^{v zM=ifkZ0N!UlNhn&wkNyzn4jM)f-$R(`yjI%>^X8ED+*>wx}C>c-H1gau*y50mS$8# zO6nvRjfre=H;bd*~?z%-hb(I5&WXI|6dNG9bc-25^aK(5!DQ5DH6(S95VV#+CIYxNKmxzyV+PG597%czZciG;^#tf2&kko^GAU#wk) z6b({-N&lnuwYeLmI(2%uLXGAi4bdHJui%=R>IwigE?Ur7x-|V9+vl)#vaE&NlJH9^ z9RQ9QJtUt|F`QN(Eu&=4oEo{oJl1htVn${`LHO4+0@#l?)swm@@tkw7%v!zr@1OEN z+4Aou&t7uVvMm#1to{Ypm;9Y?f!`o6O-)v5*zn=gwKIalPQahO*{~p3`Z#7j%wpEW z!C?0su_)o#;8}u~pfAD4=8g}T2_zKrYO5Z&w*|9{G|=N+BT6GEfKruy{c4TUia2mm zb-6KxDh(gcWTmFxgiS?legm|0FdOeT>|>nJR6d6i?o+y-c&>|Xta#SO3uw|Za!KCg zo4jJ>L4&5|#g1S)bGv57zXPq#vSTGd_n}r@p-y&tGj_mH5`fE8oxt>`a>7$^& z2^$q%Q(1D~?h7X09X4;Ku?0M!Fln0reDQ+DT#O`MW&&)QDf7XFsGn61xDzxR8?m2S z8m(}1!yAog1q7u+&-Lo)uK6JgP}QeWaG1)nhrvDgeEM2e5S6Oycc()=mBcz?PiBb; zEMfTCyI;RcRJ~%ihAj(e=fx&GaNxkJE@HHfPdWI|9f~A3cj>*_ZwlxRJ%UugE3Ve* zZ2RF!wg1#}PuIkY)?*A4p1iei-ANXsm`5K0ABA|m~7wP=&4ZXsKx%Lo3OmOh$#b*8f3~g*`cl2d=mGY zYL^l>@EB#&+`X3?w%nScX5s$tlI_~jt_tz&{v^iwFI~Or1@?ibhjDOs7Cx*GB&Y~) z$oh{0boT2k3HQJl`YD`pZ_xYm1Gcl3>svmpX%z;jL&;-kyVk*Zt*rB5gft#@8sPBXQoN1uW=+VC9YbX80n#ow?Zf4w+ zvCOQTZ>X9cKh}kFwjcjW!F2g@Z#IqmDGJ2s=sM4W$bfuA5 zQ~L7nvZl$CC;zGmcY9b+P*4$xluGDxC`gOLIFo-m=5Cj>!av&DMX zY(l41ref^s`sm;dRekGr=*)QY48NmC)eDcx3^i2NwbUS3y)tQW z(&|sop71<=5EBzYD(v}PILDf_^CBJ9^npVL3_eC6;<1G{rjIK))LZG~h{f(ng$m40 zOfzNeinT^sk5&-&%;1$Y8)Y1fU`v4lmD0{*HlU8?ajZ zc|{km(vF7>ilb~BpaP(pKM9LGt}%W*&FoK41w`J#WlQU3h$i^` z1%+0^QjfXZiJ#^TUbyKl-W5jmKLbv`y}dqb7@Z!5xCwJo$HS}1P3S*#%$VaufxB(J zU}x}8yRdUo{?*xGHvW^gpM=o5cCo8S|DfXglXUSj(i+Hgnm$qy(g5JjSaYh6Vk!s{ z1Fx7^N4T5@e+-zM3;6_a@+Hgc+_-u>y;gXw7V%$T#+u& zJ;6^y`IM>pdQWUAEGz`60}mHvNU%ANCYTwPDR~$4|ATPV;G#<80xT3#nBwB8TeghD zNj5PjXJIE9lAp(eDd;VC<*EpwJ1YekO|ZiWq;hH+7%(tVLxz?*_8hU|MUcSq0q#jt z$xoji>@j@7ja83jD5*#?yJPJJ4j*pwDmBJ=t-4>j+~Jf75vwb67Wj-BHf&~xmx>?) zy+rf;&E7|%uEKoQ%$d6`w4A5%S+Zx1hW6s=K{s!1N)EOj{W#LI!HT!e>tQS@<=lbH z7cbN;n`SOtnB!?uX>s4R)H#6OajC2H)Q4v#-_K6N?EObwg0wQo6&u4v%A; zUC)keTWYRknARm?q#=2o7rr>cx)UvA!t?!dpezAV>kO?*D=O&v4xn(MYz)sH)u&%S z$B;@o;jI3QE8pGLhy-ctuX^81RviX-+g!!_kk81i?KXma(4tBk! z;6HrBogr4fam3P>@#|>lz zPhb8$iw#aW1w~KO5W485JFJ)K@Wr^Bci;=$+~T(_h0+&Y)x?%jVYYY6%gcFo2V;l# zm;x$}0s*#^F)~~ssYXAYc z=i2QBdP~*~1RvM8-jfz$^1VyR9)S=$I=_zVASfgdF5RLd@>UTo_3VokDZY`7X z9nAZZIKS5;KIZuGF`pvyz%hleWLRtVxYx*fg(Te#M*l ze=jkW#Q}Z>@_kw{f8$`|-rs9#2E$KbSeryk`KnKk_+Q_jUa5#7{8~r6sfePnL7AKN z%nd}~As2o#CxhGt+DX+hXS$NP)9!P|%JhO(7Ue>R+^;j2k_zQaeTt~cC}Z3T z_fJdLEa6I})PZ5OSKOAkIcU)7M~Au|$vg7lM}}lc*3HInn|&4~kt=GPp4ChtZn-C$ z!>JnJ`=DGIIJ>_)l4>MB*fQhqh3-YFOP7Tn`rIz#2%B@m&}c?%t+vK6kDx-FLS?uFgn{$8W`le)-ekrXC7+c=-Hs5<$h`xLB zD@m`*N8WVX7Oy#}DKD^TYv4?7_BCIHLoz8Ejdi~m7qNOot&z^}3ytn)c{`vY; zWOL=pbt*=8ENp&yT?=ekr!i2493NYsy>j&H9)a=h`phf#TI(o~i zXYs$=$o<3|ifv?NWS7XfkzfknVy93B@e=WDLy9TJZ2i(h5%_84|D4e~B=b#XL3%}S z?|ygtPe{7{^Y*QMdd4lT-+ayX4>mMf=5xj7s(6);w`*;`T(ra1xYvO9Wzsb_?~gYW zjWC#E(1KmTS+8EDy;UgG(4O<aKG293!(u07jy=Adrk*oVZ8=0f@A2P5aCr6?$V+E?iR%=@q8=RdMW4zMdUln;N? zcYunLq0wY!6S{^?H5jD5(WS*k>#v8sB&Foa=51NA{Y;AM zeZqclQG=`knbKvl;T^5iFs7aLp7kz1*YYc{CE{mY%lGBY9S;P33vc-uPGXtnzUTA} zc#dPPUypC!C8zn_^}_g*altv(c{f#UCQS_M6qz-)`}=u?J9ln%UFx3P#v@tPdt;Z5 zT_KlrJl?EcKDE|&p0`_geC?p72ai{l+Xt)8F#_Q<3kkhoZy_ypa;)sLyAp*1brY*~ z6Xz#9bvmSZOl^J%)e)x%JZP~|?;iKwdwtf&7tc2Jk0?FY@^g7hx#yIsg63SeV)Vfp z0~LpSq`gR9_BIv5$}RAIpQ5PzjHb$nlGp0LGFCn)d7bY6Wx{PU_vG;&$pKR*PH=Vz zZrt8jSLcvAqAB0aHg$jH{_>KYpUdwpE@$024Kff6K^rNVfG#otYQ2IrW6IR-9BFAT zXt^63^4#_8+uMl6HErMObKZ#r$Aqi^T0XY)4yOCV4^bhfOYiZN;%6|vs;FDKFhK!56 z8us_G%)OEZZ6X`yb+HM1vVB_xrr8(Ve6!OwKHk2~_KD7wlM0H-N`{1X^@YExby`OA z*2J8ET}`JIJKIX-e4P#Q@YkFzwzS%Pa*AuVHSL+$09#8n3}5& z6>IC^hA&2!bqwd`e(E6h()|rLYf^VxYPNeEKEuRq-1gBqCN{h5kqrFvrM(E{Rd)-T z?DTippV_FZ7m{^r^T(OAg>>}fXv}*M8Ue1ME=?t&4;?nF^}eVrW6u~Nv!}ymVyT$& zFLZhQ074N1WN5Yt{ET=qscpaA_~1i5Omn}j+@8babF`i_mF%3{albNRzMA-qWA%M4+Gu zQ-5QVAq%bx%6FQn5dJbqh^xY8<^ahb1v~2c{}F@$+~fshTHd^2`LF}aOK5QfQh{S% zY^2R?os!rLC#_|>Eu&-y54Tb2=defQke-)9Y_9fm8gQRl0N{=UCH&a9iis1KUJou> z5<&~6POSkQNC>~N`e{$*aUN4T0-E5Pb9LGPB-~kImMzWZ@8IC>UAslBJ zsBSArxfmCBZr83|rj_NNKg&6cakxXWb^yW72-p%mWKBx;+nF$bGD?PqKBv2UTIwV2 z@IVl!M4b^F5CL(%_iY|P39k6VOUzh~m1FgIUd47;qN zRuD(8xxqUj54*%KpkrceKL}?dxjrYpl$CAX{_FmZM%<$WV4I!2fY}HmbYl#L`uNcTA@HMb^;K4U-@YJ~ zrUyRWR2DOKMSZ~1(i88pTHS{RPfQ5e=i*^ zXV!XHzQR;_H=+AMCIjXUS{t&++Ki}5~-FO?_I!pj#gbeWjsqt-Nh|A?=HfM*_BGRV>j3;|GKI*{jqAdYi+ z0yd8TXt3yESEyl9`;;9dB`rPABboM>#*=9ZdnWS1g1%@Dyy;xA1f(4BUSJ}LiF!_B z00etnUoc!(&$9tZfhbscQn1RbWhe+7z-D%m311#>{qR5!77wy3&>ab=={d`0ap6M$ zICkt<9=Yr^1{kQkb zuF{9w)4yEvcgqQ`3oG^Pyx$5~EF)tL-&N*`8f%_6yFxv@24^ooK8mekA_nr8=oiU= zhz=;oYsz=;ZUw`j)wGV@1eYe98%!vOzpZR@>-O#AaMhauZ~)~cUJX4Srf?K9M5bAC za!&D-tTg@59nmN;-Xy?)ii+8k-xT?Lq?l3xH&cgbCk@D_1uLizKy6f}y6TjOnpb-! zWi-jio18kmk@7hfIdn+YO!hLn!fF6k<#?#L*2cv+AF6XYas0SZS<7Jn8lWFE2Q00j3H;ZS)<-^3 zKVlV^JIrkaYF`9#;L9u2n6jCP&+-+2H!jxLxbxDe4LGJ+s>vZy#wN5={SC9TU~OHN zFE8M2Vqt*I0AEKR3T)j``tZO3>^dmH1)+;j4>OstpJ)MZDI}y|glv<)4)sxD%;i-# zsfF7?_h>eti*L9#Y-Zbwl%D$GQ#k98`C16$b3$?Xd|C5OI|5Tn7&rO2J_TvOFE98?W7{78Aj8lj%~lgh*{ zI~|HHU{C#AZiDQYl$m~7{h#tkM(LBs<2F)v^Nd*mB&6OE#t+}WR{-!H)N@d;o;|-q z-I!eQXUheSeElP`XYAQw#UmX*4g^hVB|bt{N(Qj{*Xx%0-SJK3f!0|^f42Mzmul<9 z#VTdMww&e=Do=~Q*Zr}_rRmwViknXm+KgQ;guJp;y83@fq; zMp(qj$-Nz(Pgu~k1S`Y9@_ysjiGYkTI5_fN0UR8zC(g8mL31o?)iz1qX(!$MN z>FGIKMddw-o&50P`Z9+L&$HG}y9PA5IM#6}(46+x?|TU8#QH_Cj#p!Ss*8&j)zl2? zxPU{I1fv8d60@UGlbu75vD7&8`qt`>0BIK|f1TA~nUGZN5-)`tY~v-RQ>(KR=l$6S z&whS&8R$H+D1y8Waf;@HLBNpg>F7xo);TYfZOak|2{{mCk21r_qz<@;WPsd93`kBh zrP+`!3XB88`Fn-*~$Smw03cBEdj*q?tS`b zt=|;-qQe3VU??wzw9yWT>97qdk@XRwHMEfc-{C3TD7 zYXr0%7?@$A(?p`s^LY>D_w+2-3MIzqn7Lh^h9=-?H=Au*%Xar4djJY~nJYcSC zi=e(9ie9!@_2anw6Gd{<4EXxacX%`I-m@%%v)x$WrSWJ2{*L(K= z{fY)MopF7^EM)3vS%}~5{sw)l=Mjz z!%YdWfW1tIZPyrm>g37I5W>;U>3j7STJ^Zv8il}uy4!dlD{;ihtD;30dxQcC^^y=2 zU3iWFYym`g^Pv{^&eC!jZ@n;FAg@3-$(Me|_rla?9tvV)1!R~an|<=9ofk}(eW9L{ zmQT5_-NKt!3QJ<$+h7znf4b7npX)x~1CB0h|M2Ws8wtU*@|Sn>p5guQs`Z?t^x8ky zMx$y)HA86*<&IG4V$ak**H>imTXgUw|C`LOf{KeGY5Dzr2-Atp!Ew(|ur%8Uk=NYc zrQSEIH7k8^VB$15^lsW0QnB3+5PVg1Jb(YHdVjCYKiRl>?g^NQZEb13m)pSoqCh-! zCn~8>E@D|k%=0Xl1M^yO)`xSlt$eRUE@D1Ht)&mH!N0DzvC(O~^SEa$I3{eURibxy zy)@PTge2n0VmT!V&`j&lsqA9b1oYb$gMf{iXH2GD8`5{m-mcZ&|A}x?I?gBCdGVlS z^+cu=VGUtHFK)tz$S;3$2#HvO$CP)c+Ju=ua`s6WS!6kfOS} zslkwBWpr2}I)ea|1MH3Z%$gyJQ^mt4krZi}n9uJ)c1JQ-Q*&$7ADyP+kOzhK3=qd~&IDSwVx+BryVUqw zg;H&@q|_-mehd|6{TABI?>s>WmL^VY4C%kSho0;2WI(aX#muy@2;c*(&?Y=2j84G1 z1+h|q-LpfDecY4niCqI)=9zcS=nwLR#!Xf)ZL`p4tu0%7KDfYP+6C*GEN$3&Xri?8 z<*~fzE5ww~fWBKorzSF)~b?TU?`wt;;bOad}I zH|}CjZa`=$`hO*k!^a7ySL5JJ#EU*?59Tad^hof`6J8Qe7`JwbT{qyr>d4azX7}&( zrJ=kf^~j+W&gLD5Xbd{tKL;6k!^n;K_cl-50_j9V?h@&^9$9L2d3gM>fPiOk3p{;z z!$1U0zCigP1#oq|-1*E;zsk;@%a>9Lijn5fef!OVjW;2)LJs${R-y2@GM@|&-J86(*ob(B}<{C4V$=7z3U>%6N^9dp@fJM7_v z_j1)*Bt&Gp>2pHXSU^t^X^@{nVQPFGso4cRZn||bnn&tVVIcscCD^nH%n<{n(2Pa6 zwKQi~4A;^+D;^H*0{x?SZy+mX$iW=?D$K^@nur&M!zPrp}nYv_oo&i^?V zDTG|u?Y>9*AY>s7q_Q6d?=I~4U*YqRO9gLIeHYfnxVDVnK01D|hSl18Q_Gr)we3b+ za67-zJV(?Nbf3s@_WzL&@ajL5loVG~OiB;e^L$K}{O6gvE(nY`HU02I^AQsg1Jaz} zCG6jC>>IPh`M|YL-!qICZbn>knKwpROcy`|;BNfb5)m#{mNnWJUp!)YT`vFFUVWWN z!GC1%rZrmmX3a%wM#-k$0xw5-uBf%W#p9gv)wm5It=d7yLNB8|SrESyIXASAfP+MjjT%t`@q%a>;j_EY zQL|5>6UFR5x$*(k@yCzToSkb!-wvFgHW{E>h*i8>>b-v}SC~qf-B=C&AZ8#=A5$>tYH-9?%ZV)N6s_&m7!|oe@&H8xMT=t>yb`{{*&r7;vMhAMYykAG>SC)d zPf_GtnJ;E|2vhWqg2tgF5yC5mpHNZZJN5uU5mEEpvr{UcV%T7s8(w2IeLc5=%i@D( zmpc5jI8e}{hWV7eJLA!#^AQo5gE$v6)l*H;pzO=K;!|h+&JEL^vMMFM`w@hZ-($zv zx@W^q4ZXjadJh^$rzulM)G)^WS~U;bcKPQGF`vvQGb#$jEt(D_hXkqj?>}d*xI7Cy z0bvaXG9wLlfI5cT9Q}6Ro6x4FrAXvFe7Dp#-y8of-r<_P4)XzP-CCRzbMT)^!228J z6Fm`DCJK<0RP0u2a$+uxa&Qg3uLLJFT~s;HMpE4h+?@-+FbtX#X8N!y_TDi zR!LqD&@mrBE*sr>JF~i)ezpd|AxO*j5vq~=(630)uspkqSg2sqh3chu3+#in4t8(2 z%Yt5cYDcFvl>MqjbFCL-?J%lKH-0H>qJj3&g7HUsFVFJ6n$J4Ipv*; zD38d%I~yy;A=hBW5%2VaRFk)MHf?iqyg-yj;YmNv4YQWSE5v72r=q}3@jmmkKDq7{ zo%(7X+IpXwh5-|gby6~F^RDJtM_@BL&h}t{8y-(iKde?H?3nnnK~{#Ik*jkP1>=zui^07A960w{plY#s%sAL3@{K<>OPwaV8a#z`!Er&Og4ba zL8!U|Kx}c;hTh;{;$+DiTPiJ#11G1G;Ig`PRQz-dd9Bz7zyzL+`&3^8REwCVW~hZC`67S-+(&i^YDI;sNLcH(ja ztxoTyYT3h5lR26By*9MqoKX}!c&B;pot<-Yj)`ltyniMeO>*1EC@m`+W%JEzX+u^wQfrcunK zEM9yjw*j#qQDengTWp61qDLnULrQ}_bUH|y%%QR$42EpwY)ru92Or;W=*g$_va zs@go)&Dg-i#JkT-aK%*ZEn$*gg@d||@ZP z32Zf1JOL`N;-+=PV*hLS6ijEF3@PC!Vue}rw!No29$OgRM?G@r$dR_@9ix87gVP(W zl??BYjFG(GaV&kgConiZetF)Rt5-{_BRhZE>{G8M(=F}Dp$k@5p8Uw(RPk;epdJwi z^QkGOvO=RwNsmZ;1#bhy`5W3nq0Fr-a5G^Dm5i6TLGv$0RAw?b!jz4|sIS!>{aY;AsbYBAaU^G%i{l=Al&nji<;a*si(& z^l{H30|-X2062t=w&^X+{nKtbBL5E5#uK%kA_}Hs&+xchW$}8+eN!S&e4H7F7 zH2b@%ILbf`JhWU1s>eJaa_M!pwgTc~1|1^{;YUI69^D|b_q4;1yEd8ezn%(_h}$on z<}JkWvFm4d6bm=33hOx`P(mQm-{IP^kUM$cvO`>(Vg@Ls(1WFcw~HaVtcqdLDE!%> zDC2|~omdKVoRF0Vg6`NpjontWp2kT403r%P&d~k3e9mgGzQst>kd#{G*0Q`sxh1zC zc*>42i@n5X^|L4`49Pi@}M(gICYZ3Gf!+yZ}-eTg9H4rHC9}SmKC5>fc}+ z{$WE@RZaRo0RDCCXvIX75UI=8z0y!j_PKgx5c0%{mZr{J83c*&e!O+!7Uj!?q>H+( z-$gr&96GesdjJE`if{hDDs8_|j80|Vs$gg1NJ3~(e0(4Zr#pMy1;YJetxIF=NQzF` zo;$CMRUbe85gJVN>%i4u))u?eOyBr%%^%pAI{aD9V^K&6G-(+0`!pkseLAvMTPi zY$H3c@_r+-s|0!r3IGBAAv3hXbc!=KCypbD0aIIASpn>KWZ!>z7t@_zMnJ)kC|Jj~ z&nMceyBTd+UCxpd+$R3Ijok)yzj%1nq5X&K#f0jph#`g&;yhrO?!itSKFt*~mh_27 z=@zPg*4Ri8+)^M;vK-hI+iSmE0CYvYS3BT32@0tp%^j_M#2X_$6djU^mtQ+ddfjBg zuNAYfnSsp${!$)eiGSIct0iAbOUW)4N5X$_I7Gm!jKin3DC)5@dC06vk+1c9pVKoP z1{?m-YKLTR#=%(}zsZ^GCr2%t2`@(gqOd>=&77olamr!d1`sC?R!G>&`=^*$O>M6} zAwXIttYS5HT5wUNLwrk9WopxFk?V^)cL#!I)`-@dqq0v-?Mz8M9VZAHN#9?kJ-XvF zl1k~oB*YL!McqlTV5#@o-tRgZ6ikwq8RFud^SlF|o(Lg#{Xw?Cj5xJ269F|`qt}rm zmoHsv^IK*P*y4$_z03U$iuVU_K}lKJ*+}vy7S-}X?Os)^rnEx4+|r=aQgYhNpRb}x zWA+4)R(I2;o2fcIP9-^{@($2ki*?7a6|7fhZj_ z9DVoJUrXn3?xFIkubuZEhBT_FeUZI9;knk>kgQ>W+6PVlpa!>=)&Bsu88l(S84e&h zr1kwdPYoR!8Wo>XW6NwGg+g5@k(3o`hwdy3$@)%Phsp|a$FAK z2{s&8^+Y<#$kW7^@1v>yXYAoxG!Ock%>b6go7?NU)8-7=@ORh5)uY;Yy||M-|IQJ& z`u%RyiwE&(WSS}KDAhVvko$p3SqvpirA?%|Dt=pgpN9%(y;xPzdarPf4>PAMO^+?I z23?MrY$!op*bZqokrkI(W(W>x)h%vcxYYqS;T4P%W?Ab?-bpHhnGy^Lj9C=;g7gHzIz%|6as`q21aB>nZB4&ciexoja@3#|fgG2X!Z}4hq^)EARM@f|Sh|zDIZKS@lu)?y+Mh z-yQWbQe~;<8-Zbp;cTzRF`PMvegxBqZgmVN&|9MG$G+bW%Tx$0;B7AFCUri#0S*A? zLvDdMYm6RENE2uhm1lPunbuR_%~4(mdmm&CoJ6Z7BW4A-Dd-%t%6Kf>`A5>)F_b1K z-nkj12QwBg?k(o7%w{9Fz#wBK#tF=f?nhE1bQC}jzyO$%MXxiFXsO-UVn7ea>PfPT z=6(wpP_Eh}>BVe`IMCWTo#|ef3}LQ@fKSLKuP@ol(A~`%BrTT%Y_?^x$#k|pqrX|b zW6$aNV(LIluAe@KSp2-of!NW<(bt2{Fwh2Jf<7+c`I1)5sx6~X z;mslTY>V6WI{EHd_ujX(tZE5NZ+2>KFmm>7u+luE^T~V05XCb_Uz0^wdn&wFBBHR5 z9yv?iMxL5~X}HSzgYh*BZZDZLxqZmUnrPYZgFW0Va+h2w@}AY!?fk&GJ-YR8CF!!W zWkrcfP0sm*UH1n(p0Qfn;hCRD%$9(-3~yaU^-V>|x{3k{yI)__k-|WN23E|lit1Oc z-Q{(b5pUF_Jt(3|u=8Ng}^ znwlp;&4()ivuLP@aosnuPQ$50lhmXMD+YjXJAXr`6kU>g#prWi%?&O?lsXU4URx1Q zl3;}{9xYR7CHgO@Xq3^}bzM6P`w&=pqVEc|wR`n}FX>tZ`Yc@7p{gW8eI#DTZ5^%Y zH!{d)Ew&$p_IOB3NSlAFupfVi6M;tuYtr>r#AFQVBh|s{r~@)-og}xDVTqDoN9?QH(OJC zKoO(P*R9QD&Rz`vdnxZBhXAwTuua$Op86Jh)KJ{U} zZVp#6$OmO>z*R{6-=J7OU&Ar~UiD~TdGou@hhf{-G>>{Y@%bJ-#qx2@Ch^%>Sz|v` zA%@uaT(Y(h(MMHcWi>(HX%m!@dUe{6y~&?B|KfJ*d8yNleSOwHiKr`1PEHp^F|c>AZV)`S^gHChP%*Ph3l*7{Pi28&_Id>iV-ZnDLGy zKh<5jac9rR%4lQ{{qtt!CJ6j(-^y<+MjW4sBX=upddwbbS zK>40|cEPc+XW^}+lf+OgNg2}?&y{jOR~l=c_4|}|CTUB-j_DkGAN1f0I#PNFxi2bb0&G~SvsmHtqT43t!@_XVC&9i zS1YjXg>)QZKyKdy;=WK#>|3eE*Ml(_0^yonHoAa;caqxc7zbsxuo5UE{(vmw3M8e9;D9n&I2-a21ULJX(2@n}I<)Uc1XZ3v34@ z0;I%gY;hK8cAg{mY|9l_S>`CVLCScix z?+3TACUA!)Ts`H)lK8C3va&=qY3ZaJtydv55M>0jgn4$+#=ZYAntw^~>87SfYtG+g z;H)sFxS=#S@LPkcheZf}LN71P$ll+=EsGn1{sWWUqjzDAo>=G<@5DYfaQ4fBf|i%I z%Q!7~j!H5sQF!)WUbmT$bF`tA^wIzG!HbSywlA{C&EycHOeJOI)DFVH;?L|tbbfJl z!|6%eHNjj~)7pyD9Zp|6b6J zqw+WtAkVG@1}a;!nea{|uT3j8lt7loYa+6(P%~tbgZJGn@a>NuQPj%ypZqpTxG;r- zz=w3Xazg&bvSl5-d`p(ZxHMVJtG({ybJO^reDX*r^--y1kC+OFiG&n#Y311=1HAWx z67uO}I-5@`d2)1CqgOwXl}<`1t|Ap8{n@(Ckdq+}5Lro-*ywqrAPw1z!$qkh?o4OY zt=h2yGMc_s)@;MLUjgyQgE6x~;76T02)-#%{%QzucO<2Qa@A7}5+@Uz+l;At+9GG)|vipPg4mp+O<;D5; z9>{-XD~#Sy%%svsG$pFdZCZsbFI*PNeY_-IkEPuFf< zK6-kq67KetDO~K;Wx;@=q^yg*{`cEFr7>M8Njf2Uz2Ieh$ET7eO?4h8Oe|U)9%j6Y z(a{)f@IP1f`*+toPRVm^%o$#p_C0X#^oA856(=uRoU}iri*L-eee;*+9Qho&M^7(B zYwsRUL$-j`ED!;XDnM@6br<>S=!p~4aE{WiETAApJT&#~8pdN`Ed(&gipJSYzG^$I zsWDz+KDFZ(+_MSR^w-YOTI+0i>92uC{}bSD4rV|#Ne!OhF9oa?_?MBsa8j7RWIw$ZOd1GkzmI`wVAqMES1`gSjtPamt0DEmKwd?^b} zZFcv}F|EXl8#8;jaAc`+(IyI%Ppu^U8REiFX#d{%uGhhXA_)EwwtREZ&W`9KCRWjZ zPFNertPC>a{GVFbesHh{$lU`N+|W%_42SVYZM|E92O&U9VhWK7kxZM`tq4Ywic;5J zikqdl>AU%Gn_&hm+2KGRt~FT;#V{8KeVFD&8(#W%v5dr#0ZNnuU&n9x+Hc=VpR;FY zT3QCxob!YtivYPJ4MAlP0}5-ly}4<%zb5WLWqiwkBsJmaPwkidadRs*QVyH?&o$RK z?VSAU+f)J@Hzc5PoW~eZkYF{lfePyyCm`UV%UVU)fS4w>Llldn<%~dbUwOZ=-+D^V z^3teFJ5K;$&^8KdBk(udc_|9thjozZcC`IarIeFc5kMW!Ap89MA;N2Ur0V_p)=6sA zDdO}_q9aagL@|Jn&bwfHNL7n&ukE&P<%+XIH2pdysae$beUEq*c3udIK=mUm+_aR2 z7@eL1iYye<70$#bN{>rXQ76e=67+8~X3p$WIB3k~{J(uX_fZE4<0WQ*&g&dJH1S&tT7{5L-B(zA4#R2l2p>qU)>#>szb z&aS=B`^ZgJjU81qzr1{#RPT=eF8LkcV~`OX8usgD4U>yc80RrqH*cf--sEpv{d>0) zm%R6%OTM|0o|06(Qm&21gI;0LOd*^zE>Su1r6R~jZotOP=^kOHd~QzpCu6+sg&01k zoC1NljLr#UNJekyC4EnyHgs@^=pnEl-acr?bQM|4vR79#v?nH3*7=@2dmy(W%C6H= z?6efRCX`g;=hxgTYzsvci+odzITA@LS?}aWa}DESE|~YK8W?M#7xZu6^E&(GOUp?n zi6zyGO^)W%*FbtP0*NjiRhTuAo~46akJm&#RFNG>2gj<^%^fT40YNCv%b=P>v3~05 zG)C4yFI?{BmxM_Yo=`0m{;p(d*Vs*)XfaeNKW$FC5TnbJLp6RoyNYEoCIS+ zrl*afT$aw<(wB@pZuGg)QsORW{QGP_Rk?t7i^>x!TsldN$BaOy2V@99a7NR%5Aibl zkgy?OH&MwT1fNCd;C?)!`UWi9wR;M2hFzVRlQUgSF1gfER5NMYh>a|MD)ds|PF2;) zGm_bd7k2Q<87GsY`FWd^SI^&xC_lLY`>C(QEEw(bV zdn`YE@I1CT;Z9+d2k8yI?{%Y&n}lHVw3gGb<*rq2X+V=SqsS2UWXqMg0guXE`cB0b z;2_s~b6w`;wOyvA>_(Hj7fY+1Gzt9w`Y957liPx?`)pn5L9Cj7)+9wuI+|0Zxv!0O%=S$56s2Wl z2WstC9TJGcqR90pM&0V!?Vz6Jo=NL%idEg}??oQUIxjqM>59e5qL*y_JU`zU2V+7Y z)c&B<~lvokQ5)yXgKt6gV6|SIXykkC! zX=g#ZaZ!*FMVn(zvqX7Bk$MB=q>-n3vphKV#;jc9T7I^VMCiocf@@*vwB5T%=&HGR z;lvUbXWpV|!NDFvIPY;}S^e$A3+=-Tva=0PlHWLD`oZbYU7zt0)`NAP1&#|lIkhyi z{;B*iNlVM3*V0G}9EFAyrYKZ$q%DSwprKql@*B(@y6sPTK=z>Fb@}9$EQq$J9}~yS zzHIFO1Pp%n?lT7n7gMJuomWl(96ko9Wp#GQgB3not3TajniaJdf2ttcLER>M1g|~R zha7VwZTWN_OqCJ2dS=Hd0#xl9o#oP0vZ5OW_93#o< zozCl8Gf8#I)4MSY_8($dPm74(c!fhQ0y9#N4>jAhd(8~KbGVo?Xy2wa{{(>VBAxaf zprOe5?)8$3MyJ(wqbTJ;Q04xOl!5gvM@V2PGDLETnJADMV)Di6^d}QnzuTMaV&(sK zOuN4ZhGOy9d$e^|>2^}>Jkn09d!XA;+nBFi9xac%%ikPsM2G}Qq!mM#XVVu>iq%PY z>5#zd;JML)^JaxvWNhr)4qo3p#Bdt^Ev?$=6`^}n*3k)M-AH7IE}m*^e58jAxLhH& z0&Ut2ym;-}ddJSk+E*GIdOrF#OjT7`Sy_^6R+C#*Q8 z1}M?>mJI3r(_IWKA;g8(5wB^%f}TM9R(Y{bq)769_sGhH`TdDfJ@3EXHlm&W>pfE4 z$M5`gr=~_A8)O=Rvp4N*y?a;9(&OL8ij$r||2n^Y#;d?;4lbs!$^}{cQ0O5+e4jgK zM03ryrhMHw$B<3g`Y)4RoiYqhU-n8Z#$Z`YmIz%zujQ_~$f9;nNjRIwF>T-1_8Ztu zWccu4UIMFMI3++MF+Ng=#+u|xPIH>6E`9F%t9S3TKi>}i7Uc z+*%xRu=-57G5Nv*B^e64t^EsqINz``rhd}DS^!ykQj{-%AiQKY+&jB*+;kfcU0O!GUA=bt*u^O0c; z)Fa-x`lmO92s8P&*z4A9tKR?bA8qiQV~>I+wdvNIHulf;47x%)D0pTZ z!1TAPbhBBzR!~2?AjqoUtO-QsW=zA%)Az`3LXM8kY1>z40!)m^(nqedSK;Q$%USQn z#?Q`w#O)^|R9VAZ+^D!2gV&Wi>n0MvD(k!to~IYEe0zHxQ%}T*H(*le<`gGS4xnn# zEw_7n+w04hFC}&*$f)j^hpp_t6!HS@J3Hq$yresS{t4l5pdAtL$X?XX;u*@VgxQqa zw{oP_B$Y%(P=ej-&Vom_#(W|XIjrRMzMX|96tuGug=l4+aFT@-+qd!tvN8BTLViN2 zqlR0vEBDm!`Jy-KK4nm2v^as}bgCURF|7pHd&j<2V8;HO^1+B5U3MD2-Vn9s{{C;< z5KbuLSRS35PFG2O6tLGWky02!l_0#X+qiKJkP00g7$(j%b)Nl}_b&MJc*t}l>Xk5a z*)&iZpK=;X+<=q6RyKD|QVVT4l6AiQ{x%6T&tB`!5_2$moV|2u?s=yqMRXqrlM3JS zpaf#fg?2i76f9*3M;IW$?%KZ1cr4SX0kS1)k-Q2}6mFDF1yMumE+i)05<4APWydv2 z=A%wU3iJHzZH!K`4#a7Qv`WZK!FOIV!9h4FsTNcLWCm}TJtEU#uOI>#XzO#&V^r}z z*0}x$uT8yV3eEN=KINU?RgZrrbOLLhYZZPBi7M0KQ#(>VE9+nUy2Q+c#ebiXIyzwoK@$zH(RtW)ZhJTFbNy5Mq z`JV?59H2G1923(!Fk||fHOhi@PARUE-d2#QsJ`|4s`m|wj439Alhn-WSgP58q(liWXQaPn^C{4Dh9OfNj zCUS^G$zh@+GdhioY%_>a4l~+jBP0u{s1!m>c2O#+geDa_Nv-$uteC@lUGMwXzTWlE z%vD-z{eHji_xV20ec#Xh%qwdMv^9d^*$<2YLQCt&k>kg+ES=;TvT#8s1))VtOSc1} zW>eNGT~=k*na_7h%<0$@oPY|Cfmw4WTCcqJ{ZQ5(<*|VAYT&HYxl)5A7t2la(YM?IbLS`?H$_}@v`jo^5kXe(*tNi{xo{D_F2VNIjWokj z6BZP1yI}D9yH1}v_3GK~-L)g5R15R}Ex@WMExlkmcMl#ou!#cRD1{ z`1yBlLHs~+NSamN=kdX>SWp6dWH$#+Wc-$q>cMlK(==mN)7Q%HPL(x}3W_0^>NGsw1U+#{}xcX(5K z?sdg~9?|t}!>oD3NQXd6rT_*1=Bkec5%42!rh`0ck$ zh2{uuZ>d_QS$=E`glR|3#JAA?{aY}ofmC?`Jjv_249&m(xh-@PEB?wd|6!94R{NS2 z+NovT9BaRL@emWYI2Celzu0nXb@W;P_)Wj29e?n2U)VCuzv*?4-1OXsgf{nfrYz5-Q7NR-pDZPzP3vE!sZAU!fDyoK#|ju6 zNqWoJy@JB*RcwA}OM%#aK5$^r!*Y|sE>zha9E&W2Z@A&4BwGYRwFp_%md10R>;FAcfTr+ZgQ#QeO^W>fqZ z)tI84VsQAQ+_>CLUAisv^d!an_1?XyOP1hyb3Dmb7Re6On0ZeYQ!sK*^KRXeJgVM5 zuGA$n73IlR%aPE#w9!KPD(st>djE+{j zi1uj?R;FfJL6`PBCmO&{FAX)cHF;TAHmqQlV^nXFu&Ben+B&&iGfOo3J*Gg(XxGLN*`UFTt8#4e&8u7mP3ADjn~8NtJ}O@w>QPoAx)-Rs<5vdMQuvC`}BKLrD`Q){2h|XOP=TJp7wV0;~g@qk%?s3=DE8>A>hwOpHHR|Ew@6XYL2qgKrlX zul%U2R{C;xANIVl0PBZSOw5sNjviVvE{bQt8)ACs2ltGuO7?v`{w-SM#vJg-+r-rE zw=Eyp8UOY&sn5|PH`_}!ABUKPl&;frqCvZMC zE-?|-8yH>9#rYnD!9%dVix;O9Q!8awH~f14{-^9*s!)$Ztu9j>ha`J%3_da$@$r!ipCy zXPF5hKZxBwfAR<+Z()r^Xc8y;s2jFw#iNo$TvTVya*W|!!MCOBT>uP#^qZ5FqBdR z#1=`*M$bKnO`2m=?Wg0$>9ZJ1Ypb{^YJrXSmgfH!3X=v{rq{eN5WU#{%$X$ngBS(h zwc3a9N5*!MQX%zL6fvYCkg(S)ysO3+5jK#lYQs!o6)6xmqYtIB$YH-DQ3JA zIyv`#-w*XP2=bmmUlQKfT+8TZd$LwYkBHLN!;7G;`=3rhfg(DjW%BHpGd&Ic=cNU! z9Lu-fA(floT4n6>%*oF9^`zEgUzpl9lt#L^O=$Mnadz%Gzq@t|F$RIWfCjzHVGBU7 ze`e)`TojuWnEll7fLLEM{_0lCj^J#d;~t-|PtT%Y!0D2-#?QNE&Pl>A3Sxj84KX&N zFk{2d^`qT3B#r42sra(3aDryfEl;m{S_Y@wuuAbfYnKiOMgtd}T*6LQomknrf!hwr z)a9a?21TgD+K_~4sO=cu72hstNAu1YROLyhHj z2DT-nN+WT=hgBoQdZvtF)*P{jKrFptAa@CF?_8wMOW4!0RoDhQP*SUyJ6xwwY(dho z;J|(aCj}(A_K8u@X3ssSNqrf2igE;6waWs)B>ssSp@ORg;y*(p%Zlu-q*LbS-{e%* z^SS0Vb)}{@V+N)>7@Lr#lY?f$;wRqguKo?YPqFt>6zI%q;U^O4C=W1Lq6Q zDYko*SHEdHhch6vi&WawA1GqE$9&~$Edh+)NOWth zIf9$z%FZCCw#wIQfYnBMX~&*iZ)HOAb8PR>&4-}1sI4C$`~3i?RmE2pGk1*HfEm-k;<*E#TFAk*gf9 z$4xLwD9B-Lh@hLYoMugJXiKcgM~=LGepm@@9J%;vba1%0?%c6lk^R>%vcaK&LEQojzOT+9K{Zv6(m3Ow2UCSW*T+S_ zQr2@a#O`_WmW>@>N4z0?rya!b+zc@(#?MNpFgHz>wwtpm&zuzmoDv$ z#$u$E?6N(w%bq(tiyhyZzOnn;l+l#X6cIJ{&)0G#;{W{RJa~JSj1*r!bbyI}=pUtx z`9m3_bL2>*)J;Z#b)CqC-~Kk^@eIEqg?3un`a7_4x>OYxVp!vT;onqm`1ov5)nBl< z4%^P#>8@zmUYo0M2pD2Vcd!+YZ%I%J14^jjOP1}W10pcb-9}6-w2vw12`nhd_9>kQ zG={>Dw%>d&U|vZ73yrp6(_l1(soBZ7=@9s%y+aPi_HIkhnNk`O?~Q-~sPQQM_-K#P z_J*D@|60C#_R9j_zPf0V$i(N(4{0t7r$4%3>zOBOtxDI<$t9r}oQmQ_0iVDYQuxF* z%mn~H?3v0Z=CCbDJrEeV<`4|utM}E>cOqq5fWAWR*%9i0vaW4NFNI>GXpMAl{O5T4 z{KCN^^_e}X{-6q0BeapeK-^BIJqiR7)CyiDW~{HVPy-Zn&|!#OA&)NZ(5wI%fqi{> zu4uWBPZjOj9N_=JgN_Z1Yf2?~B^6jWionj|U?x#MXre6`8=yIv3s0jHxL7y7HmS-y zqvwbxDSHUu+oPk+$b=Y?AmT9dB|X(>f&q1Mt+w$Akm(k{&ES-45R?J+Apd|Y2a)P3 zE^>z3xke=UN2*_-D?=g(^uz6bsnyqMZSaZ96>2;RYQ{>%Pr1 zOAc%?XRx)>d|Ccy)WcCN(f{;0e?t+HVt#z*sQJoIAN}$kawWS*zu1)<4{_5KE<_Yy zAhhPtU;yjQfWG3zs|IZ((@|-X`d?T#5UbjAMTIBayKqXt6w8rLL4=XE%rl^xK^a3Z z>Ex~0i@G>y;#bMVxS2>(2z{;MW`ze0ElJ!B&Z6lnWyuuxDXj2+!Z)*ZM z%fBlF&D`~Pc7UIKjX1oid=MSMs{tRgPRK+qmDh`hG3|OEW7Nsq2~yYZZlw+1zEtMP z$h77qust%yKsfZdVP{407uZznCmzgz8&ft%*k9hQ8L4cKQ?FLEmV3CmE(*=;_WATX zbtuKgZSY=^C4xYjHczEGyST(7_Z;Pwe*zFr4Ycnf9yBzUO7w<5Vxf{%nhqU`;IeR1 zk)=kLJWE%q*^6rwxRJ@^Vl=wF54*(Oq=W~i%c^<-t>Xj?wqNrK#FAdvBtDALlf(f& z325~nNj`a&(1e5WLfHrh{ApfPbBfEbk3U|DH8L4M;+iUFGAmJa_fq%))QSalvA6V& z@O)RbJYOP&qt!m~xY@WQXD704`YJD1uAX9DLJ~b^(Rf-*uU@?h3d=YHCs5yYKLP^K z=yfK4Yh>gBz|w8m84L-M)B>CV^mkU%zFjl?sFp>-PFas3`Rn>^A$QC-zdvl4(CohS zFDz3InNl%e^dq~4)^pp|74P4F@i;5!Oa~x@3qQ}rjH}_3Lhr|MqLVAy>3VL9v8B&o`=G-K+K!Ys$UJ79B*tqR9 zmzdRcI>-D=>MTBrw*zF8RE-5BR6$xWS>4f?DCCN?pCXgo!kz6Eo{32c^(aIhCE-@j zJ-NjtG{y*w4)B3(1K^2_s-}bz6aUrcAjNbKdp@T(1zZ4XS9{wSUH|8`kC()cUblAb zoT6Up-|#fa-L4|av2-Fvg_qVUQvGN_+J0cVqekMEf?CeD@<9XBaW9div43xk4Ei;? zJ&4g%#)Y(xlP>z2ull_EuJ}E$&}8X1wIct>PwHCcB`QCHU(mBNA}1z3Q-}}&#dlLp zx;N?Y=}78`T$?_qYRAirjl+_l!tHEEp~LMk+?KuPznXc%dy{1mKw-T2FvEE zfuaDQnJT_5sl-Q7P+ICTQ0v6;NUzjTDKZ^iR}t z>17np)}-EIe`DCwN#^j8aq?Wm5|O*!yE2Xj$cXL;sJ`<0@M&|dudY_wI+_zulMX9L z4pFAAbN0qMso_;#h~c+vwB9W?T-u3rq@~uuul1yFlB^Lq8M^TI(n?WehA$e9Tc4q2 zfL6~?iXwy4&dt@wkB?w5n&nUR4o_ibzR4N{2WH-1S104~s|3xgl@l)<6R%42RF$i@ zZdntVU9^4*tV!|DFoE*S1_?OK>q@kWb+ETzd_^)2d6UJY$nw%Rmu!QVMhrQOyuwg` zs{W1L;s5a4Z1HzQ)0F6$XG%tL`ocn)J%!TjcKh~7`3CkAWN7_O)H;X??oQhUEFzi9 z(W8Nt=gU3yuQerM!N9^j)Hb+}eD%m+M{@RAZf@<@8X(?8KymKe23?0y{$bC9ruZLK zU5&$@$;QZXRM%`c)a2lsi!0y$*B=ir9HgzG&ayv5U++ey$ezS%xVrYmix=}Z#Q&2D zWEI-LciY?8UI^C2Lf^%y^Dx;iuZ)kKTu zEgPXT1ve~7{#`I93e(f;0??vcnEsQ$9xF(9*meb|;FWP}oB$G_mDEnjOT03e4q z_ewO__q`Q;b%@^}v_j{572(kE4q#ya(`g)(vv@OPp}dD0ERaR1%J$Z(XYnUbZbt@5 zeT*tuHJ^(eXzCJsorb0P8?MXa${HVP|NYib5z44~Oz6O2Y?lZpsima?1WA{Cfz1hD zc;(u)h19Up$qT^_mLLTq1^&g#)i!Is%|r7+WoY=a5S>bddWC_ST9@{z_LDxNYeEcD z^Gi{MM$xb6_R1KmA zaB^Nkc4lTB?gcWxVA=TGA^}MV&zGO9J%u&lf`O|@qd<@#s>^=J{*=eF_`)pJvg@I? zD<{-*4>%~X3LSTRUXO`*l&xES8)1_W&`4AeW*fqdh{rLONY!JM?t<^p-gWox6&O=F z**Q%qxM^0&5e*%-Y-9pF502LiDC!;;x#1uR->y%Iw+ULFm&dnJBU})}G0Xyd;HZ*R zHm4FZ;a%_uV0dxKal8SW0K?7f%GHqCIU>-&Z&NkLG3qZ*;C&OY1NF8rTbvRwYS7Y) z5K~D+;J1U_zJmslHulGfjE zV+WOel}zu?V9<23C}4(l#hFC^zig2oq9bI&+5FZNsw-lPj)uC}%&jYQS?};bf%n*? zA-bnhzgwGf;5%#(z^ez3_{~v} zHJG1woE(d-Xa4cyLnF0y074F)JJ<4a4u>J`S2DLZ;R`?OkL+4fMIl7ac<_%fErKs_ zhL5yI_!LD&Nvw}@nW5Q3@9YmAvuE$xIeU0o%KJy;$Bw6?BHtW3^#WeV>pyi2H7dJ! Y4A}T_;Xq}88n3BSob8WKUhwsQ17^`8IRF3v literal 0 HcmV?d00001 diff --git a/jetstream/clustering/administration.md b/jetstream/clustering/administration.md new file mode 100644 index 0000000..941ee69 --- /dev/null +++ b/jetstream/clustering/administration.md @@ -0,0 +1,85 @@ +# Cluster Administration + +Once a JetStream cluster is operating interactions with the CLI and with `nats` CLI is the same as before. For these examples, lets assume we have a 5 server cluster, n1-n5 in a cluster named C1. + +## Creating clustered streams + +When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. + +```nohighlight +$ nats str add ORDERS --replicas 3 +.... +Information for Stream ORDERS_4 created 2021-02-05T12:07:34+01:00 +.... +Configuration: +.... + Replicas: 3 + +Cluster Information: + + Name: C1 + Leader: n1-c1 + Replica: n4-c1, current, seen 0.07s ago + Replica: n3-c1, current, seen 0.07s ago + +``` + +Above you can see that the cluster information will be reported in all cases where Stream info is shown such as after add or using `nats stream info`. + +Here we have a stream in the NATS cluster `C1`, its current leader is a node `n1-c1` and it has 2 followers - `n4-c1` and `n3-c1`. + +The `current` indicates that followers are up to date and have all the messages, here both cluster peers were seen very recently. + +The replica count cannot be edited once configured. + +## Forcing leader election + +Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time. + +Moving leadership away from a node does not remove it from the cluster and does not prevent it from becoming a leader again, this is merely a triggered leader election. + +```nohighlight +$ nats stream cluster step-down ORDERS +14:32:17 Requesting leader step down of "n1-c1" in a 3 peer RAFT group +14:32:18 New leader elected "n4-c1" + +Information for Stream ORDERS created 2021-02-05T12:07:34+01:00 +... +Cluster Information: + + Name: c1 + Leader: n4-c1 + Replica: n1-c1, current, seen 0.12s ago + Replica: n3-c1, current, seen 0.12s ago +``` + +## Evicting a peer + +Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down. + +There might be a case though where you know a machine will never return, and you want to signal to JetStream that the machine will not return. This will remove it from the Stream in question and all it's Consumers. + +After the node is removed the cluster will notice that the replica count is not honored anymore and will immediately pick a new node and start replicating data to it. The new node will be selected using the same placement rules as the existing stream. + +```nohighlight +$ nats s cluster peer-remove ORDERS +? Select a Peer n4-c1 +14:38:50 Removing peer "n4-c1" +14:38:50 Requested removal of peer "n4-c1" +``` + +At this point the stream and all consumers will have removed `n4-c1` from the group, they will all start new peer selection and data replication. + +```nohighlight +$ nats stream info ORDERS +.... +Cluster Information: + + Name: c1 + Leader: n3-c1 + Replica: n1-c1, current, seen 0.02s ago + Replica: n2-c1, outdated, seen 0.42s ago +``` + +We can see a new replica was picked, the stream is back to replication level of 3 and `n4-c1` is not active any more in this Stream or any of its Consumers. + From d3ebabbeb0e7e4738f878e5c4d9dd6dded5ee16c Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Fri, 5 Feb 2021 13:14:22 -0700 Subject: [PATCH 09/31] updates Signed-off-by: Colin Sullivan --- SUMMARY.md | 5 +- jetstream/about_jetstream/jetstream.md | 2 +- jetstream/clustering/clustering.md | 94 ++------------------------ jetstream/concepts/concepts.md | 2 +- 4 files changed, 10 insertions(+), 93 deletions(-) diff --git a/SUMMARY.md b/SUMMARY.md index c0c55b0..4252c67 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -206,6 +206,7 @@ ## JetStream +* [About Jetstream](jetstream/about_jetstream/jetstream.md) * [Concepts](jetstream/concepts/concepts.md) * [Streams](jetstream/concepts/streams.md) * [Consumes](jetstream/concepts/consumers.md) @@ -219,8 +220,10 @@ * [Streams](jetstream/administration/streams.md) * [Consumers](jetstream/administration/consumers.md) * [Monitoring](jetstream/monitoring/monitoring.md) +* [Clustering](jetstream/clustering/clustering.md) + * [Administration](jetstream/clustering/administration.md) * [Configuration Management](jetstream/configuration_mgmt/configuration_mgmt.md) - * [nats Admin CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) + * [NATS Admin CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) * [Terraform](jetstream/configuration_mgmt/configuration_mgmt.md#terraform) * [GitHub Actions](jetstream/configuration_mgmt/github_actions.md) * [Kubernetes Controller](jetstream/configuration_mgmt/kubernetes_controller.md) diff --git a/jetstream/about_jetstream/jetstream.md b/jetstream/about_jetstream/jetstream.md index 29261b8..dc87dcf 100644 --- a/jetstream/about_jetstream/jetstream.md +++ b/jetstream/about_jetstream/jetstream.md @@ -1,4 +1,4 @@ -# Jetstream +# JetStream JetStream was created to solve the problems identified with streaming in technology today - complexity, fragility, and a lack of scalability. Some technologies address these better than others, but no current streaming technology is truly multi-tenant, horizontally scalable, and supports multiple deployment models. No technology we are aware of can scale from edge to cloud under the same security context while having complete deployment observability for operations. diff --git a/jetstream/clustering/clustering.md b/jetstream/clustering/clustering.md index 6e6ed08..51c8d73 100644 --- a/jetstream/clustering/clustering.md +++ b/jetstream/clustering/clustering.md @@ -20,15 +20,15 @@ In order to ensure data consistency across complete restarts, a quorum of server **Meta Group** - all servers join the Meta Group and the JetStream API is managed by this group. A leader is elected and this owns the API and takes care of server placement. -![](images/meta-group.png) FIXME +![Meta Group](../../assets/images/meta-group.png) **Stream Group** - each Stream creates a RAFT group, this group synchronizes state and data between its members. The elected leader handles ACKs and so forth, if there is no leader the stream will not accept messages. -![](images/stream-groups.png) FIXME +![Stream Groups](../../assets/images/stream-groups.png) **Consumer Group** - each Consumer creates a RAFT group, this group synchronizes consumer state between its members. The group will live on the machines where the Stream Group is and handle consumption ACKs etc. Each Consumer will have its own group. -![](images/consumer-groups.png) FIXME +![Consumer Groups](../../assets/images/consumer-groups.png) ### Cluster Size Generally we recommend 3 or 5 Jetstream enabled servers in a NATS cluster. This balances scalability with a tolerance for failure. For example, if 5 servers are Jetstream enabled You would want two servers is one “zone”, two servers in another, and the remaining server in a third. This means you can lose any one “zone” at any time and continue operating. @@ -103,90 +103,4 @@ cluster { } ``` -Add nodes as necessary. Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use Jetstream. - -## Administration - -Once a JetStream cluster is operating interactions with the CLI and with `nats` CLI is the same as before. For these examples, lets assume we have a 5 server cluster, n1-n5 in a cluster named C1. - -### Creating clustered streams - -When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. - -```nohighlight -$ nats str add ORDERS --replicas 3 -.... -Information for Stream ORDERS_4 created 2021-02-05T12:07:34+01:00 -.... -Configuration: -.... - Replicas: 3 - -Cluster Information: - - Name: C1 - Leader: n1-c1 - Replica: n4-c1, current, seen 0.07s ago - Replica: n3-c1, current, seen 0.07s ago - -``` - -Above you can see that the cluster information will be reported in all cases where Stream info is shown such as after add or using `nats stream info`. - -Here we have a stream in the NATS cluster `C1`, its current leader is a node `n1-c1` and it has 2 followers - `n4-c1` and `n3-c1`. - -The `current` indicates that followers are up to date and have all the messages, here both cluster peers were seen very recently. - -The replica count cannot be edited once configured. - -### Forcing leader election - -Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time. - -Moving leadership away from a node does not remove it from the cluster and does not prevent it from becoming a leader again, this is merely a triggered leader election. - -```nohighlight -$ nats stream cluster step-down ORDERS -14:32:17 Requesting leader step down of "n1-c1" in a 3 peer RAFT group -14:32:18 New leader elected "n4-c1" - -Information for Stream ORDERS created 2021-02-05T12:07:34+01:00 -... -Cluster Information: - - Name: c1 - Leader: n4-c1 - Replica: n1-c1, current, seen 0.12s ago - Replica: n3-c1, current, seen 0.12s ago -``` - -### Evicting a peer - -Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down. - -There might be a case though where you know a machine will never return, and you want to signal to JetStream that the machine will not return. This will remove it from the Stream in question and all it's Consumers. - -After the node is removed the cluster will notice that the replica count is not honored anymore and will immediately pick a new node and start replicating data to it. The new node will be selected using the same placement rules as the existing stream. - -```nohighlight -$ nats s cluster peer-remove ORDERS -? Select a Peer n4-c1 -14:38:50 Removing peer "n4-c1" -14:38:50 Requested removal of peer "n4-c1" -``` - -At this point the stream and all consumers will have removed `n4-c1` from the group, they will all start new peer selection and data replication. - -```nohighlight -$ nats stream info ORDERS -.... -Cluster Information: - - Name: c1 - Leader: n3-c1 - Replica: n1-c1, current, seen 0.02s ago - Replica: n2-c1, outdated, seen 0.42s ago -``` - -We can see a new replica was picked, the stream is back to replication level of 3 and `n4-c1` is not active any more in this Stream or any of its Consumers. - +Add nodes as necessary. Choose a data directory that makes sense for your environment, ideally a fast SDD, and launch each server. After two servers are running you'll be ready to use Jetstream. \ No newline at end of file diff --git a/jetstream/concepts/concepts.md b/jetstream/concepts/concepts.md index 00785d4..4a533c6 100644 --- a/jetstream/concepts/concepts.md +++ b/jetstream/concepts/concepts.md @@ -4,7 +4,7 @@ In JetStream the configuration for storing messages is defined separately from h We'll discuss these 2 subjects in the context of this architecture. -![Orders](../.gitbook/assets/images/streams-and-consumers-75p.png) +![Orders](../../assets/images/streams-and-consumers-75p.png) While this is an incomplete architecture it does show a number of key points: From 6875a3fab9dccea3b9a833d6991d3fdbf13945bf Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 14:30:37 -0600 Subject: [PATCH 10/31] updates based on RI changes to README --- jetstream/monitoring/monitoring.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/jetstream/monitoring/monitoring.md b/jetstream/monitoring/monitoring.md index 13b5185..39add46 100644 --- a/jetstream/monitoring/monitoring.md +++ b/jetstream/monitoring/monitoring.md @@ -24,7 +24,11 @@ All these events have JSON Schemas that describe them, schemas can be viewed on |Consumer maximum delivery reached|`$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES..`|`io.nats.jetstream.advisory.v1.max_deliver`| |Message delivery terminated using AckTerm|`$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED..`|`io.nats.jetstream.advisory.v1.terminated`| |Message acknowledged in a sampled Consumer|`$JS.EVENT.METRIC.CONSUMER.ACK..`|`io.nats.jetstream.metric.v1.consumer_ack`| +|Clustered Stream elected a new leader|`$JS.EVENT.ADVISORY.STREAM.LEADER_ELECTED.`|`io.nats.jetstream.advisory.v1.stream_leader_elected`| +|Clustered Stream lost quorum|`$JS.EVENT.ADVISORY.STREAM.QUORUM_LOST.`|`io.nats.jetstream.advisory.v1.stream_quorum_lost` +|Clustered Consumer elected a new leader|`$JS.EVENT.ADVISORY.CONSUMER.LEADER_ELECTED..`|`io.nats.jetstream.advisory.v1.consumer_leader_elected`| +|Clustered Consumer lost quorum|`$JS.EVENT.ADVISORY.CONSUMER.QUORUM_LOST..`|`io.nats.jetstream.advisory.v1.consumer_quorum_lost`| ### Dashboards -The [NATS Surveyor](https://github.com/nats-io/nats-surveyor) system has initial support for passing JetStream metrics to Prometheus, dashboards and more will be added towards final release. \ No newline at end of file +The [NATS Surveyor](https://github.com/nats-io/nats-surveyor) system has initial support for passing JetStream metrics to Prometheus, dashboards and more will be added towards final release. From 501ecd83dc3118455199a72133eb709f785dbb22 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 14:35:01 -0600 Subject: [PATCH 11/31] updates based on RI changes to README --- jetstream/administration/consumers.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/jetstream/administration/consumers.md b/jetstream/administration/consumers.md index e3e21ac..3506676 100644 --- a/jetstream/administration/consumers.md +++ b/jetstream/administration/consumers.md @@ -184,11 +184,11 @@ Acknowledged message You can prevent ACKs by supplying `--no-ack`. -To do this from code you'd send a `Request()` to `$JS.NEXT.ORDERS.DISPATCH`: +To do this from code you'd send a `Request()` to `$JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH`: ``` -$ nats req '$JS.NEXT.ORDERS.DISPATCH' '' -Published [$JS.NEXT.ORDERS.DISPATCH] : '' +$ nats req '$JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH' '' +Published [$JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH] : '' Received [ORDERS.processed] : 'order 3' ``` @@ -216,4 +216,4 @@ Listening on [monitor.ORDERS] Note the subject here of the received message is reported as `ORDERS.processed` this helps you distinguish what you're seeing in a Stream covering a wildcard, or multiple subject, subject space. -This Consumer needs no ack, so any new message into the ORDERS system will show up here in real time. \ No newline at end of file +This Consumer needs no ack, so any new message into the ORDERS system will show up here in real time. From deb7f26a01a3f6d4497501beb4aa33f2de3e0f47 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 14:39:47 -0600 Subject: [PATCH 12/31] updates based on RI changes to README --- .../nats_api_reference/nats_api_reference.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/jetstream/nats_api_reference/nats_api_reference.md b/jetstream/nats_api_reference/nats_api_reference.md index 1e994ed..afb8466 100644 --- a/jetstream/nats_api_reference/nats_api_reference.md +++ b/jetstream/nats_api_reference/nats_api_reference.md @@ -143,9 +143,24 @@ Events and Advisories: ``` $JS.EVENT.METRIC.CONSUMER_ACK.. $JS.EVENT.ADVISORY.MAX_DELIVERIES.. +$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.. +$JS.EVENT.ADVISORY.STREAM.CREATED. +$JS.EVENT.ADVISORY.STREAM.DELETED. +$JS.EVENT.ADVISORY.STREAM.UPDATED. +$JS.EVENT.ADVISORY.CONSUMER.CREATED.. +$JS.EVENT.ADVISORY.CONSUMER.DELETED.. +$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_CREATE. +$JS.EVENT.ADVISORY.STREAM.SNAPSHOT_COMPLETE. +$JS.EVENT.ADVISORY.STREAM.RESTORE_CREATE. +$JS.EVENT.ADVISORY.STREAM.RESTORE_COMPLETE. +$JS.EVENT.ADVISORY.STREAM.LEADER_ELECTED. +$JS.EVENT.ADVISORY.STREAM.QUORUM_LOST. +$JS.EVENT.ADVISORY.CONSUMER.LEADER_ELECTED.. +$JS.EVENT.ADVISORY.CONSUMER.QUORUM_LOST.. +$JS.EVENT.ADVISORY.API ``` -This allow you to easily create ACL rules that limit users to a specific Stream or Consumer and to specific verbs for administration purposes. For ensuring only the receiver of a message can Ack it we have response permissions ensuring you can only Publish to Response subject for messages you received. +This design allows you to easily create ACL rules that limit users to a specific Stream or Consumer and to specific verbs for administration purposes. For ensuring only the receiver of a message can Ack it we have response permissions ensuring you can only Publish to Response subject for messages you received. ### Acknowledging Messages From 5f727f432a267577394a587b99ce446eee79083b Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 14:47:50 -0600 Subject: [PATCH 13/31] Updates based on RI changes to README --- jetstream/administration/account.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/jetstream/administration/account.md b/jetstream/administration/account.md index 33761ac..2053ef0 100644 --- a/jetstream/administration/account.md +++ b/jetstream/administration/account.md @@ -26,6 +26,7 @@ $ nats str add ORDERS ? Message size limit -1 ? Maximum message age limit 1y ? Maximum individual message size [? for help] (-1) -1 +? Number of replicas to store 3 Stream ORDERS was created Information for Stream ORDERS @@ -35,7 +36,7 @@ Configuration: Subjects: ORDERS.* Acknowledgements: true Retention: File - Limits - Replicas: 1 + Replicas: 3 Maximum Messages: -1 Maximum Bytes: -1 Maximum Age: 8760h0m0s @@ -54,10 +55,9 @@ Statistics: You can get prompted interactively for missing information as above, or do it all on one command. Pressing `?` in the CLI will help you map prompts to CLI options: ``` -$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old -``` +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --replicas 3``` -Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: +Additionally, one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: ``` $ nats str add ORDERS --config orders.json @@ -132,6 +132,20 @@ $ nats str info ORDERS -j This is the general pattern for the entire `nats` utility as it relates to JetStream - prompting for needed information but every action can be run non-interactively making it usable as a cli api. All information output like seen above can be turned into JSON using `-j`. +In clustered mode additional information will be included: + +```nohighlight +$ nats str info ORDERS +... +Cluster Information: + Name: JSC + Leader: S1 + Replica: S3, current, seen 0.04s ago + Replica: S2, current, seen 0.04s ago +``` + +Here the cluster name is configured as `JSC`, there is a server `S1` that's the current leader with `S3` and `S2` are replicas. Both replicas are current and have been seen recently. + #### Copying A stream can be copied into another, which also allows the configuration of the new one to be adjusted via CLI flags: From 7487129f1ea9c0be38fa8b52dba8d0ac05a68ade Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 14:52:16 -0600 Subject: [PATCH 14/31] Updates based on RI changes to README --- jetstream/concepts/streams.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jetstream/concepts/streams.md b/jetstream/concepts/streams.md index 861ec64..8b33f64 100644 --- a/jetstream/concepts/streams.md +++ b/jetstream/concepts/streams.md @@ -23,7 +23,7 @@ When defining Streams the items below make up the entire configuration of the se |MaxConsumers|How many Consumers can be defined for a given Stream, `-1` for unlimited| |Name|A name for the Stream that may not have spaces, tabs or `.`| |NoAck|Disables acknowledging messages that are received by the Stream| -|Replicas|How many replicas to keep for each message (not implemented as of January 2020)| +|Replicas|How many replicas to keep for each message in a clustered JetStream, maximum 5| |Retention|How message retention is considered, `LimitsPolicy` (default), `InterestPolicy` or `WorkQueuePolicy`| |Discard|When a Stream reached it's limits either, `DiscardNew` refuses new messages while `DiscardOld` (default) deletes old messages| |Storage|The type of storage backend, `file` and `memory` as of January 2020| From 56a90151bd6fd04ddb58d7a20f35e9dce6cb9087 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 14:54:13 -0600 Subject: [PATCH 15/31] update based on RI changes to README --- jetstream/monitoring/monitoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jetstream/monitoring/monitoring.md b/jetstream/monitoring/monitoring.md index 39add46..f95e510 100644 --- a/jetstream/monitoring/monitoring.md +++ b/jetstream/monitoring/monitoring.md @@ -8,7 +8,7 @@ Typically, NATS is monitored via HTTP endpoints like `/varz`, we do not at this JetStream publish a number of advisories that can inform operations about health and state of the Streams. These advisories are published to normal NATS subjects below `$JS.EVENT.ADVISORY.>` and one can store these advisories in JetStream Streams if desired. -The command `nats event --js-advsiroy` can view all these events on your console. The Golang package [jsm.go](https://github.com/nats-io/jsm.go) can consume and render these events and have data types for each of these events. +The command `nats event --js-advisory` can view all these events on your console. The Golang package [jsm.go](https://github.com/nats-io/jsm.go) can consume and render these events and have data types for each of these events. All these events have JSON Schemas that describe them, schemas can be viewed on the CLI using the `nats schema show ` command. From df06f3387162c48f50fe91aa39fef1c0774f46ea Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 14:59:41 -0600 Subject: [PATCH 16/31] updates based on RI changes to README --- jetstream/administration/administration.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/jetstream/administration/administration.md b/jetstream/administration/administration.md index 8dcd564..1abfecb 100644 --- a/jetstream/administration/administration.md +++ b/jetstream/administration/administration.md @@ -1,11 +1,13 @@ ## Administration and Usage from the CLI -Once the server is running it's time to use the management tool. This can be downloaded from the [GitHub Release Page](https://github.com/nats-io/jetstream/releases/) or you can use the `synadia/jsm:latest` docker image. +Once the server is running it's time to use the management tool. This can be downloaded from the [GitHub Release Page](https://github.com/nats-io/natscli/releases/) or you can use the `synadia/jsm:latest` docker image. On OS X homebrew can be used to install the latest version: -``` +```nohighlight +$ brew tap nats-io/nats-tools +$ brew install nats-io/nats-tools/nats $ nats --help usage: nats [] [ ...] -NATS Management Utility +NATS Utility Flags: --help Show context-sensitive help (also try --help-long and --help-man). @@ -27,4 +29,4 @@ We'll walk through the above scenario and introduce features of the CLI and of J Throughout this example, we'll show other commands like `nats pub` and `nats sub` to interact with the system. These are normal existing core NATS commands and JetStream is fully usable by only using core NATS. -We'll touch on some additional features but please review the section on the design model to understand all possible permutations. \ No newline at end of file +We'll touch on some additional features but please review the section on the design model to understand all possible permutations. From 8892475f8de93dcb1f70a22fca69dc9e1e37effc Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 15:01:52 -0600 Subject: [PATCH 17/31] updates based on RI changes to README --- jetstream/administration/account.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/jetstream/administration/account.md b/jetstream/administration/account.md index 2053ef0..7c3eecd 100644 --- a/jetstream/administration/account.md +++ b/jetstream/administration/account.md @@ -4,10 +4,20 @@ JetStream is multi-tenant so you will need to check that your account is enabled ```nohighlight $ nats account info - - Memory: 0 B of 6.4 GB - Storage: 0 B of 1.1 TB - Streams: 1 of Unlimited +Connection Information: + Client ID: 8 + Client IP: 127.0.0.1 + RTT: 178.545µs + Headers Supported: true + Maximum Payload: 1.0 MiB + Connected URL: nats://localhost:4222 + Connected Address: 127.0.0.1:4222 + Connected Server ID: NCCOHA6ONXJOGAEZP4WPU4UJ3IQP2VVXEPRKTQCGBCW4IL4YYW4V4KKL +JetStream Account Information: + Memory: 0 B of 5.7 GiB + Storage: 0 B of 11 GiB + Streams: 0 of Unlimited + Max Consumers: unlimited ``` ### Streams From 2b6d217e372b32fa40eafbfc402420a76607c790 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 15:07:16 -0600 Subject: [PATCH 18/31] updates from RI changes to README --- jetstream/concepts/streams.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jetstream/concepts/streams.md b/jetstream/concepts/streams.md index 8b33f64..87c8205 100644 --- a/jetstream/concepts/streams.md +++ b/jetstream/concepts/streams.md @@ -10,7 +10,7 @@ Streams can consume many subjects. Here we have `ORDERS.*` but we could also con Streams support various retention policies - they can be kept based on limits like max count, size or age but also more novel methods like keeping them as long as any Consumers have them unacknowledged, or work queue like behavior where a message is removed after first ack. -Streams support deduplication using a `Msg-Id` header and a sliding window within which to track duplicate messages. See the [Message Deduplication](#message-deduplication) section. +Streams support deduplication using a `Nats-Msg-Id` header and a sliding window within which to track duplicate messages. See the [Message Deduplication](#message-deduplication) section. When defining Streams the items below make up the entire configuration of the set. From 9d1e79110cd6b941152c79d3744020342c83716c Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 15:12:28 -0600 Subject: [PATCH 19/31] updates based on RI changes to README --- jetstream/model_deep_dive/model_deep_dive.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/jetstream/model_deep_dive/model_deep_dive.md b/jetstream/model_deep_dive/model_deep_dive.md index aa87c20..63f263f 100644 --- a/jetstream/model_deep_dive/model_deep_dive.md +++ b/jetstream/model_deep_dive/model_deep_dive.md @@ -30,16 +30,16 @@ The `WorkQueuePolicy` mode is a specialized mode where a message, once consumed ### Message Deduplication -JetStream support idempotent message writes by ignoring duplicate messages as indicated by the `Msg-Id` header. +JetStream support idempotent message writes by ignoring duplicate messages as indicated by the `Nats-Msg-Id` header. ```nohighlight -% nats req -H Msg-Id:1 ORDERS.new hello1 -% nats req -H Msg-Id:1 ORDERS.new hello2 -% nats req -H Msg-Id:1 ORDERS.new hello3 -% nats req -H Msg-Id:1 ORDERS.new hello4 +% nats req -H Nats-Msg-Id:1 ORDERS.new hello1 +% nats req -H Nats-Msg-Id:1 ORDERS.new hello2 +% nats req -H Nats-Msg-Id:1 ORDERS.new hello3 +% nats req -H Nats-Msg-Id:1 ORDERS.new hello4 ``` -Here we set a `Msg-Id:1` header which tells JetStream to ensure we do not have duplicates of this message - we only consult the message ID not the body. +Here we set a `Nats-Msg-Id:1` header which tells JetStream to ensure we do not have duplicates of this message - we only consult the message ID not the body. ```nohighlight $ nats str info ORDERS From b9d6095d36940ea5fd94b3e1041591452d54389d Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 15:13:36 -0600 Subject: [PATCH 20/31] updates based on RI changes to README --- jetstream/nats_api_reference/nats_api_reference.md | 1 - 1 file changed, 1 deletion(-) diff --git a/jetstream/nats_api_reference/nats_api_reference.md b/jetstream/nats_api_reference/nats_api_reference.md index afb8466..4d35174 100644 --- a/jetstream/nats_api_reference/nats_api_reference.md +++ b/jetstream/nats_api_reference/nats_api_reference.md @@ -61,7 +61,6 @@ The API uses JSON for inputs and outputs, all the responses are typed using a `t |-------|--------|-----------|---------------|----------------| |`$JS.API.STREAM.LIST`|`api.JSApiStreamList`|Paged list known Streams including all their current information|`api.JSApiStreamListRequest`|`api.JSApiStreamListResponse`| |`$JS.API.STREAM.NAMES`|`api.JSApiStreamNames`|Paged list of Streams|`api.JSApiStreamNamesRequest`|`api.JSApiStreamNamesResponse`| -|`$JS.API.STREAM.LOOKUP`|`api.JSApiStreamLookup`|Finds a stream with interest on a certain subject|`api.JSApiStreamLookupRequest`|`api.JSApiStreamLookupResponse`| |`$JS.API.STREAM.CREATE.*`|`api.JSApiStreamCreateT`|Creates a new Stream|`api.StreamConfig`|`api.JSApiStreamCreateResponse`| |`$JS.API.STREAM.UPDATE.*`|`api.JSApiStreamUpdateT`|Updates an existing Stream with new config|`api.StreamConfig`|`api.JSApiStreamUpdateResponse`| |`$JS.API.STREAM.INFO.*`|`api.JSApiStreamInfoT`|Information about config and state of a Stream|empty payload, Stream name in subject|`api.JSApiStreamInfoResponse`| From 70b13dbd5b969f086269c6db8ef5082f109eeda9 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 15:23:38 -0600 Subject: [PATCH 21/31] Move JetStream outline under NATS Server --- SUMMARY.md | 76 +++++++++++++++++++++++++++--------------------------- 1 file changed, 38 insertions(+), 38 deletions(-) diff --git a/SUMMARY.md b/SUMMARY.md index 4252c67..45e3ab7 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -105,6 +105,44 @@ * [Tutorial](nats-server/nats_docker/nats-docker-tutorial.md) * [Docker Swarm](nats-server/nats_docker/docker_swarm.md) * [Python and NGS Running in Docker](nats-server/nats_docker/ngs-docker-python.md) + +## JetStream + +* [About Jetstream](jetstream/about_jetstream/jetstream.md) +* [Concepts](jetstream/concepts/concepts.md) + * [Streams](jetstream/concepts/streams.md) + * [Consumes](jetstream/concepts/consumers.md) + * [Configuration](jetstream/concepts/configuration.md) +* [Getting Started](jetstream/getting_started/getting_started.md) + * [Using Docker](jetstream/getting_started/using_docker.md) + * [Using Docker with NGS](jetstream/getting_started/using_docker.md#using-docker-with-ngs) + * [Using Source](jetstream/getting_started/using_source.md) +* [Administration & Usage from CLI](jetstream/administration/administration.md) + * [Account Information](jetstream/administration/account.md) + * [Streams](jetstream/administration/streams.md) + * [Consumers](jetstream/administration/consumers.md) +* [Monitoring](jetstream/monitoring/monitoring.md) +* [Clustering](jetstream/clustering/clustering.md) + * [Administration](jetstream/clustering/administration.md) +* [Configuration Management](jetstream/configuration_mgmt/configuration_mgmt.md) + * [NATS Admin CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) + * [Terraform](jetstream/configuration_mgmt/configuration_mgmt.md#terraform) + * [GitHub Actions](jetstream/configuration_mgmt/github_actions.md) + * [Kubernetes Controller](jetstream/configuration_mgmt/kubernetes_controller.md) +* [Disaser Recovery](jetstream/disaster_recovery/disaster_recovery.md) +* [Model Deep Dive](jetstream/model_deep_dive/model_deep_dive.md) + * [Stream Limits, Retention Modes and Discard Policy](jetstream/model_deep_dive/model_deep_dive.md#stream-limits-retention-modes-and-discard-policy) + * [Message Deduplication](jetstream/model_deep_dive/model_deep_dive.md#message-deduplication) + * [Acknowledgment Models](jetstream/model_deep_dive/model_deep_dive.md#acknowledgement-models) + * [Exactly Once Delivery](jetstream/model_deep_dive/model_deep_dive.md#exactly-once-delivery) + * [Consumer Starting Position](jetstream/model_deep_dive/model_deep_dive.md#consumer-starting-position) + * [Ephemeral Consumers](jetstream/model_deep_dive/model_deep_dive.md#ephemeral-consumers) + * [Consumer Message Rates](jetstream/model_deep_dive/model_deep_dive.md#consumer-message-rates) + * [Stream Templates](jetstream/model_deep_dive/model_deep_dive.md#stream-templates) + * [Ack Sampling](jetstream/model_deep_dive/model_deep_dive.md#ack-sampling) + * [Storage Overhead](jetstream/model_deep_dive/model_deep_dive.md#storage-overhead) +* [NATS API Reference](jetstream/nats_api_reference/nats_api_reference.md) +* [Multi-tenancy & Resource Mgmt](jetstream/multi-tenancy/resource_management.md) ## NATS Tools @@ -203,41 +241,3 @@ * [Using a Load Balancer for External Access to NATS](nats-on-kubernetes/nats-external-nlb.md) * [Creating a NATS Super Cluster in Digital Ocean with Helm](nats-on-kubernetes/super-cluster-on-digital-ocean.md) * [From Zero to K8S to Leafnodes using Helm](nats-on-kubernetes/from-zero-to-leafnodes.md) - -## JetStream - -* [About Jetstream](jetstream/about_jetstream/jetstream.md) -* [Concepts](jetstream/concepts/concepts.md) - * [Streams](jetstream/concepts/streams.md) - * [Consumes](jetstream/concepts/consumers.md) - * [Configuration](jetstream/concepts/configuration.md) -* [Getting Started](jetstream/getting_started/getting_started.md) - * [Using Docker](jetstream/getting_started/using_docker.md) - * [Using Docker with NGS](jetstream/getting_started/using_docker.md#using-docker-with-ngs) - * [Using Source](jetstream/getting_started/using_source.md) -* [Administration & Usage from CLI](jetstream/administration/administration.md) - * [Account Information](jetstream/administration/account.md) - * [Streams](jetstream/administration/streams.md) - * [Consumers](jetstream/administration/consumers.md) -* [Monitoring](jetstream/monitoring/monitoring.md) -* [Clustering](jetstream/clustering/clustering.md) - * [Administration](jetstream/clustering/administration.md) -* [Configuration Management](jetstream/configuration_mgmt/configuration_mgmt.md) - * [NATS Admin CLI](jetstream/configuration_mgmt/configuration_mgmt.md#nats-admin-cli) - * [Terraform](jetstream/configuration_mgmt/configuration_mgmt.md#terraform) - * [GitHub Actions](jetstream/configuration_mgmt/github_actions.md) - * [Kubernetes Controller](jetstream/configuration_mgmt/kubernetes_controller.md) -* [Disaser Recovery](jetstream/disaster_recovery/disaster_recovery.md) -* [Model Deep Dive](jetstream/model_deep_dive/model_deep_dive.md) - * [Stream Limits, Retention Modes and Discard Policy](jetstream/model_deep_dive/model_deep_dive.md#stream-limits-retention-modes-and-discard-policy) - * [Message Deduplication](jetstream/model_deep_dive/model_deep_dive.md#message-deduplication) - * [Acknowledgment Models](jetstream/model_deep_dive/model_deep_dive.md#acknowledgement-models) - * [Exactly Once Delivery](jetstream/model_deep_dive/model_deep_dive.md#exactly-once-delivery) - * [Consumer Starting Position](jetstream/model_deep_dive/model_deep_dive.md#consumer-starting-position) - * [Ephemeral Consumers](jetstream/model_deep_dive/model_deep_dive.md#ephemeral-consumers) - * [Consumer Message Rates](jetstream/model_deep_dive/model_deep_dive.md#consumer-message-rates) - * [Stream Templates](jetstream/model_deep_dive/model_deep_dive.md#stream-templates) - * [Ack Sampling](jetstream/model_deep_dive/model_deep_dive.md#ack-sampling) - * [Storage Overhead](jetstream/model_deep_dive/model_deep_dive.md#storage-overhead) -* [NATS API Reference](jetstream/nats_api_reference/nats_api_reference.md) -* [Multi-tenancy & Resource Mgmt](jetstream/multi-tenancy/resource_management.md) From 648be33d81a5f91684ba2a1ed9827ec36012bf2c Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 5 Feb 2021 15:34:57 -0600 Subject: [PATCH 22/31] Jetstream ->JetStream --- jetstream/getting_started/getting_started.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/jetstream/getting_started/getting_started.md b/jetstream/getting_started/getting_started.md index 91b2364..6e5ff6c 100644 --- a/jetstream/getting_started/getting_started.md +++ b/jetstream/getting_started/getting_started.md @@ -1,16 +1,16 @@ # Getting Started -Getting started with Jetstream is straightforward. While we speak of Jetstream as if it is a seperate component, it's actually a subsystem built into the NATS server that needs to be enabled. +Getting started with JetStream is straightforward. While we speak of Jetstream as if it is a seperate component, it's actually a subsystem built into the NATS server that needs to be enabled. ## Command line -Enable Jetstream by specifying the `-js` flag when starting the NATS server. +Enable JetStream by specifying the `-js` flag when starting the NATS server. `$ nats-server -js` ## Configuration File -Enable Jetstream through a configuration file. By default, the Jetstream subsytem will store data in the /tmp directory. Here's a minimal file that will store data in a local "nats" directory, suitable for development and local testing. +Enable JetStream through a configuration file. By default, the JetStream subsytem will store data in the /tmp directory. Here's a minimal file that will store data in a local "nats" directory, suitable for development and local testing. `$ nats-server -c js.conf` @@ -21,6 +21,6 @@ jetstream { } ``` -Normally Jetstream will be run in clustered mode and will replicate data, so the best place to store Jetstream data would be locally on a fast SSD. One should specifically avoid NAS or NFS storage for Jetstream. +Normally JetStream will be run in clustered mode and will replicate data, so the best place to store JetStream data would be locally on a fast SSD. One should specifically avoid NAS or NFS storage for JetStream. -See [Using Docker](./using_docker.md) and [Using Source](./using_source.md) for more information. \ No newline at end of file +See [Using Docker](./using_docker.md) and [Using Source](./using_source.md) for more information. From a3224f0245cd7ce0972f57a83ca1218d77f4e2ec Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Tue, 9 Feb 2021 16:50:24 -0600 Subject: [PATCH 23/31] updates based on RI updates to README 020921 --- jetstream/clustering/administration.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/jetstream/clustering/administration.md b/jetstream/clustering/administration.md index 941ee69..92efe7f 100644 --- a/jetstream/clustering/administration.md +++ b/jetstream/clustering/administration.md @@ -32,9 +32,9 @@ The `current` indicates that followers are up to date and have all the messages, The replica count cannot be edited once configured. -## Forcing leader election +## Forcing Stream and Consumer leader election -Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time. +Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process, but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time. Moving leadership away from a node does not remove it from the cluster and does not prevent it from becoming a leader again, this is merely a triggered leader election. @@ -53,6 +53,18 @@ Cluster Information: Replica: n3-c1, current, seen 0.12s ago ``` +The same is true for consumers, `nats consumer cluster step-down ORDERS NEW`. + +#### Forcing Meta Group leader election + +Similar to Streams and Consumers above the Meta Group allows leader stand down. The Meta Group is cluster wide and spans all accounts, therefore to manage the meta group you have to use a `SYSTEM` user. + +```nohighlight +$ nats server raft step-down --user system +17:44:24 Current leader: n2-c2 +17:44:24 New leader: n1-c2 +``` + ## Evicting a peer Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down. From 2b9d45ad7eeb067a20206e1505e9e379c23db4e4 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 11 Feb 2021 15:56:18 -0600 Subject: [PATCH 24/31] updates based on RI changes to README --- jetstream/clustering/administration.md | 92 +++++++++++++++++++++++++- 1 file changed, 90 insertions(+), 2 deletions(-) diff --git a/jetstream/clustering/administration.md b/jetstream/clustering/administration.md index 92efe7f..7c05e83 100644 --- a/jetstream/clustering/administration.md +++ b/jetstream/clustering/administration.md @@ -2,6 +2,10 @@ Once a JetStream cluster is operating interactions with the CLI and with `nats` CLI is the same as before. For these examples, lets assume we have a 5 server cluster, n1-n5 in a cluster named C1. +## Account Level + +Within an account there are operations and reports that show where users data is placed and which allow them some basic interactions with the RAFT system. + ## Creating clustered streams When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. @@ -32,7 +36,26 @@ The `current` indicates that followers are up to date and have all the messages, The replica count cannot be edited once configured. -## Forcing Stream and Consumer leader election +### Viewing Stream Placement and Stats + +Users can get overall statistics about their streams and also where these streams are placed: + +``` +$ nats stream report +Obtaining Stream stats ++----------+-----------+----------+--------+---------+------+---------+----------------------+----------+ +| Stream | Consumers | Messages | Bytes | Storage | Lost | Deleted | Cluster | Template | ++----------+-----------+----------+--------+---------+------+---------+----------------------+----------+ +| ORDERS | 4 | 0 | 0 B | File | 0 | 0 | n1-c1*, n2-c1, n3-c1 | | +| ORDERS_3 | 4 | 0 | 0 B | File | 0 | 0 | n1-c1*, n2-c1, n3-c1 | | +| ORDERS_4 | 4 | 0 | 0 B | File | 0 | 0 | n1-c1*, n2-c1, n3-c1 | | +| ORDERS_5 | 4 | 0 | 0 B | File | 0 | 0 | n1-c1, n2-c1, n3-c1* | | +| ORDERS_2 | 4 | 1,385 | 13 MiB | File | 0 | 1 | n1-c1, n2-c1, n3-c1* | | +| ORDERS_0 | 4 | 1,561 | 14 MiB | File | 0 | 0 | n1-c1, n2-c1*, n3-c1 | | ++----------+-----------+----------+--------+---------+------+---------+----------------------+----------+ +``` + +#### Forcing Stream and Consumer leader election Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process, but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time. @@ -55,6 +78,71 @@ Cluster Information: The same is true for consumers, `nats consumer cluster step-down ORDERS NEW`. +## System Level + +Systems users can view state of the Meta Group - but not individual Stream or Consumers. + +### Viewing the cluster state + +We have a high level report of cluster state: + +```nohighlight +$ nats server report jetstream --user system ++--------------------------------------------------------------------------------------------------+ +| JetStream Summary | ++--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+ +| Server | Cluster | Streams | Consumers | Messages | Bytes | Memory | File | API Req | API Err | ++--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+ +| n3-c2 | c2 | 0 | 0 | 0 | 0 B | 0 B | 0 B | 1 | 0 | +| n3-c1 | c1 | 6 | 24 | 2,946 | 27 MiB | 0 B | 27 MiB | 3 | 0 | +| n2-c2 | c2 | 0 | 0 | 0 | 0 B | 0 B | 0 B | 3 | 0 | +| n1-c2 | c2 | 0 | 0 | 0 | 0 B | 0 B | 0 B | 14 | 2 | +| n2-c1 | c1 | 6 | 24 | 2,946 | 27 MiB | 0 B | 27 MiB | 15 | 0 | +| n1-c1* | c1 | 6 | 24 | 2,946 | 27 MiB | 0 B | 27 MiB | 31 | 0 | ++--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+ +| | | 18 | 72 | 8,838 | 80 MiB | 0 B | 80 MiB | 67 | 2 | ++--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+ ++---------------------------------------------------+ +| RAFT Meta Group Information | ++-------+--------+---------+---------+--------+-----+ +| Name | Leader | Current | Offline | Active | Lag | ++-------+--------+---------+---------+--------+-----+ +| n1-c1 | yes | true | false | 0.00s | 0 | +| n1-c2 | | true | false | 0.05s | 0 | +| n2-c1 | | true | false | 0.05s | 0 | +| n2-c2 | | true | false | 0.05s | 0 | +| n3-c1 | | true | false | 0.05s | 0 | +| n3-c2 | | true | false | 0.05s | 0 | ++-------+--------+---------+---------+--------+-----+ +``` + +This is a full cluster wide report, the report can be limited to a specific account using `--account`. + +Here we see the distribution of streams, messages, api calls etc by across 2 super clusters and an overview of the RAFT meta group. + +In the Meta Group report the server `n2-c1` is not current and has not been seen for 9 seconds, it's also behind by 2 raft operations. + +This report is built using raw data that can be obtained from the monitor port on the `/jsz` url, or over nats using: + +```nohightlight +$ nats server req jetstream --help +... + --name=NAME Limit to servers matching a server name + --host=HOST Limit to servers matching a server host name + --cluster=CLUSTER Limit to servers matching a cluster name + --tags=TAGS ... Limit to servers with these configured tags + --account=ACCOUNT Show statistics scoped to a specific account + --accounts Include details about accounts + --streams Include details about Streams + --consumer Include details about Consumers + --config Include details about configuration + --leader Request a response from the Meta-group leader only + --all Include accounts, streams, consumers and configuration +$ nats server req jetstream --leader +``` + +This will produce a wealth of raw information about the current state of your cluster - here requesting it from the leader only. + #### Forcing Meta Group leader election Similar to Streams and Consumers above the Meta Group allows leader stand down. The Meta Group is cluster wide and spans all accounts, therefore to manage the meta group you have to use a `SYSTEM` user. @@ -65,7 +153,7 @@ $ nats server raft step-down --user system 17:44:24 New leader: n1-c2 ``` -## Evicting a peer +### Evicting a peer Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down. From 4dc2d7f0b7d61192f53fee3173429e374144059f Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Sat, 27 Feb 2021 17:30:51 -0700 Subject: [PATCH 25/31] update docs Signed-off-by: Colin Sullivan --- jetstream/administration/administration.md | 2 +- jetstream/administration/consumers.md | 2 +- jetstream/administration/streams.md | 68 +++++++++++++++------- jetstream/clustering/administration.md | 4 +- jetstream/getting_started/using_docker.md | 23 ++------ jetstream/getting_started/using_source.md | 4 +- 6 files changed, 58 insertions(+), 45 deletions(-) diff --git a/jetstream/administration/administration.md b/jetstream/administration/administration.md index 1abfecb..4585bbe 100644 --- a/jetstream/administration/administration.md +++ b/jetstream/administration/administration.md @@ -1,6 +1,6 @@ ## Administration and Usage from the CLI -Once the server is running it's time to use the management tool. This can be downloaded from the [GitHub Release Page](https://github.com/nats-io/natscli/releases/) or you can use the `synadia/jsm:latest` docker image. On OS X homebrew can be used to install the latest version: +Once the server is running it's time to use the management tool. This can be downloaded from the [GitHub Release Page](https://github.com/nats-io/natscli/releases/) or you can use the `natsio/nats-box:latest` docker image. On OS X homebrew can be used to install the latest version: ```nohighlight $ brew tap nats-io/nats-tools diff --git a/jetstream/administration/consumers.md b/jetstream/administration/consumers.md index 3506676..9526c4d 100644 --- a/jetstream/administration/consumers.md +++ b/jetstream/administration/consumers.md @@ -156,7 +156,7 @@ More details about the `State` section will be shown later when discussing the a #### Consuming Pull-Based Consumers -Pull-based Consumers require you to specifically ask for messages and ack them, typically you would do this with the client library `Request()` feature, but the `jsm` utility has a helper: +Pull-based Consumers require you to specifically ask for messages and ack them, typically you would do this with the client library `Request()` feature, but the `nats` utility has a helper: First we ensure we have a message: diff --git a/jetstream/administration/streams.md b/jetstream/administration/streams.md index 72e8689..1fcaf4c 100644 --- a/jetstream/administration/streams.md +++ b/jetstream/administration/streams.md @@ -42,7 +42,7 @@ Statistics: You can get prompted interactively for missing information as above, or do it all on one command. Pressing `?` in the CLI will help you map prompts to CLI options: ``` -$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old +nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --dupe-window="0s" --replicas 1 ``` Additionally one can store the configuration in a JSON file, the format of this is the same as `$ nats str info ORDERS -j | jq .config`: @@ -68,26 +68,29 @@ Information about the configuration of the Stream can be seen, and if you did no ```nohighlight $ nats str info ORDERS -Information for Stream ORDERS +Information for Stream ORDERS created 2021-02-27T16:49:36-07:00 Configuration: Subjects: ORDERS.* - No Acknowledgements: false + Acknowledgements: true Retention: File - Limits Replicas: 1 - Maximum Messages: -1 - Maximum Bytes: -1 - Maximum Age: 8760h0m0s - Maximum Consumers: -1 + Discard Policy: Old + Duplicate Window: 2m0s + Maximum Messages: unlimited + Maximum Bytes: unlimited + Maximum Age: 1y0d0h0m0s + Maximum Message Size: unlimited + Maximum Consumers: unlimited -Statistics: +State: - Messages: 0 - Bytes: 0 B - FirstSeq: 0 - LastSeq: 0 - Active Consumers: 0 + Messages: 0 + Bytes: 0 B + FirstSeq: 0 + LastSeq: 0 + Active Consumers: 0 ``` Most commands that show data as above support `-j` to show the results as JSON: @@ -105,14 +108,20 @@ $ nats str info ORDERS -j "max_msgs": -1, "max_bytes": -1, "max_age": 31536000000000000, + "max_msg_size": -1, "storage": "file", - "num_replicas": 1 + "discard": "old", + "num_replicas": 1, + "duplicate_window": 120000000000 }, - "stats": { + "created": "2021-02-27T23:49:36.700424Z", + "state": { "messages": 0, "bytes": 0, "first_seq": 0, + "first_ts": "0001-01-01T00:00:00Z", "last_seq": 0, + "last_ts": "0001-01-01T00:00:00Z", "consumer_count": 0 } } @@ -128,14 +137,29 @@ A stream can be copied into another, which also allows the configuration of the $ nats str cp ORDERS ARCHIVE --subjects "ORDERS_ARCVHIVE.*" --max-age 2y Stream ORDERS was created -Information for Stream ARCHIVE +Information for Stream ORDERS created 2021-02-27T16:52:46-07:00 Configuration: - Subjects: ORDERS_ARCVHIVE.* -... - Maximum Age: 17520h0m0s -... + Subjects: ORDERS_ARCHIVE.* + Acknowledgements: true + Retention: File - Limits + Replicas: 1 + Discard Policy: Old + Duplicate Window: 2m0s + Maximum Messages: unlimited + Maximum Bytes: unlimited + Maximum Age: 2y0d0h0m0s + Maximum Message Size: unlimited + Maximum Consumers: unlimited + +State: + + Messages: 0 + Bytes: 0 B + FirstSeq: 0 + LastSeq: 0 + Active Consumers: 0 ``` #### Editing @@ -208,7 +232,7 @@ To delete all data in a stream use `purge`: ```nohighlight $ nats str purge ORDERS -f ... -Statistics: +State: Messages: 0 Bytes: 0 B @@ -231,5 +255,5 @@ Finally for demonstration purposes, you can also delete the whole Stream and rec ``` $ nats str rm ORDERS -f -$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 +$ nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --dupe-window="0s" --replicas 1 ``` diff --git a/jetstream/clustering/administration.md b/jetstream/clustering/administration.md index 7c05e83..ef213fd 100644 --- a/jetstream/clustering/administration.md +++ b/jetstream/clustering/administration.md @@ -8,12 +8,12 @@ Within an account there are operations and reports that show where users data is ## Creating clustered streams -When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. +When adding a stream using the `nats` CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored o multiple nodes in your cluster using the RAFT protocol as above. The replica count must be less than the maximum number of servers. ```nohighlight $ nats str add ORDERS --replicas 3 .... -Information for Stream ORDERS_4 created 2021-02-05T12:07:34+01:00 +Information for Stream ORDERS created 2021-02-05T12:07:34+01:00 .... Configuration: .... diff --git a/jetstream/getting_started/using_docker.md b/jetstream/getting_started/using_docker.md index a16549e..f46a7fa 100644 --- a/jetstream/getting_started/using_docker.md +++ b/jetstream/getting_started/using_docker.md @@ -1,29 +1,17 @@ # Using Docker -The `synadia/jsm:latest` docker image contains both the JetStream enabled NATS Server and the `nats` utility this guide covers. +The `synadia/nats-box:latest` docker image contains the `nats` utility this guide covers. -In one window start JetStream: +In one window start a JetStream enabled nats server: ``` -$ docker run -ti -p 4222:4222 --name jetstream synadia/jsm:latest server -[1] 2020/01/20 12:44:11.752465 [INF] Starting nats-server version 2.2.0-beta -[1] 2020/01/20 12:44:11.752694 [INF] Git commit [19dc3eb] -[1] 2020/01/20 12:44:11.752875 [INF] Starting JetStream -[1] 2020/01/20 12:44:11.753692 [INF] ----------- JETSTREAM (Beta) ----------- -[1] 2020/01/20 12:44:11.753794 [INF] Max Memory: 1.46 GB -[1] 2020/01/20 12:44:11.753822 [INF] Max Storage: 1.00 TB -[1] 2020/01/20 12:44:11.753860 [INF] Store Directory: "/tmp/jetstream" -[1] 2020/01/20 12:44:11.753893 [INF] ---------------------------------------- -[1] 2020/01/20 12:44:11.753988 [INF] JetStream state for account "$G" recovered -[1] 2020/01/20 12:44:11.754148 [INF] Listening for client connections on 0.0.0.0:4222 -[1] 2020/01/20 12:44:11.754279 [INF] Server id is NDYX5IMGF2YLX6RC4WLZA7T3JGHPZR2RNCCIFUQBT6C4TP27Z6ZIC73V -[1] 2020/01/20 12:44:11.754308 [INF] Server is ready +$ docker run --network host -p 4222:4222 nats -js ``` And in another log into the utilities: ``` -$ docker run -ti --link jetstream synadia/jsm:latest +$ docker run -ti --network host synadia/nats-box ``` This shell has the `nats` utility and all other NATS cli tools used in the rest of this guide. @@ -34,6 +22,7 @@ Now skip to the `Administer JetStream` section. You can join a JetStream instance to your [NGS](https://synadia.com/ngs/pricing) account, first we need a credential for testing JetStream: +You'll want to do this outside of docker to keep the credentials that are generated. ``` $ nsc add user -a YourAccount --name leafnode --expiry 1M ``` @@ -42,7 +31,7 @@ You'll get a credential file somewhere like `~/.nkeys/creds/synadia/YourAccount/ ``` $ docker run -ti -v ~/.nkeys/creds/synadia/YourAccount/leafnode.creds:/leafnode.creds --name jetstream synadia/jsm:latest server -[1] 2020/01/20 12:44:11.752465 [INF] Starting nats-server version 2.2.0-beta +[1] 2020/01/20 12:44:11.752465 [INF] Starting nats-server version 2.2.0 ... [1] 2020/01/20 12:55:01.849033 [INF] Connected leafnode to "connect.ngs.global" ``` diff --git a/jetstream/getting_started/using_source.md b/jetstream/getting_started/using_source.md index b21eb8a..7b7e821 100644 --- a/jetstream/getting_started/using_source.md +++ b/jetstream/getting_started/using_source.md @@ -17,7 +17,7 @@ Starting the server you can use the `-js` flag. This will setup the server to re ``` $ ./nats-server -js -[16928] 2019/12/04 19:16:29.596968 [INF] Starting nats-server version 2.2.0-beta +[16928] 2019/12/04 19:16:29.596968 [INF] Starting nats-server version 2.2.0 [16928] 2019/12/04 19:16:29.597056 [INF] Git commit [not set] [16928] 2019/12/04 19:16:29.597072 [INF] Starting JetStream [16928] 2019/12/04 19:16:29.597444 [INF] ----------- JETSTREAM (Beta) ----------- @@ -35,7 +35,7 @@ You can override the storage directory if you want. ``` $ ./nats-server -js -sd /tmp/test -[16943] 2019/12/04 19:20:00.874148 [INF] Starting nats-server version 2.2.0-beta +[16943] 2019/12/04 19:20:00.874148 [INF] Starting nats-server version 2.2.0 [16943] 2019/12/04 19:20:00.874247 [INF] Git commit [not set] [16943] 2019/12/04 19:20:00.874273 [INF] Starting JetStream [16943] 2019/12/04 19:20:00.874605 [INF] ----------- JETSTREAM (Beta) ----------- From ce3996faabba9196572c7ac1ffb868962b7b1214 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Sun, 28 Feb 2021 14:43:16 -0700 Subject: [PATCH 26/31] add replication Signed-off-by: Colin Sullivan --- assets/images/replication-setup.png | Bin 0 -> 18016 bytes assets/images/replication.png | Bin 0 -> 15453 bytes jetstream/data_replication/replication.md | 169 ++++++++++++++++++++++ 3 files changed, 169 insertions(+) create mode 100644 assets/images/replication-setup.png create mode 100644 assets/images/replication.png create mode 100644 jetstream/data_replication/replication.md diff --git a/assets/images/replication-setup.png b/assets/images/replication-setup.png new file mode 100644 index 0000000000000000000000000000000000000000..7d14c7d0e31828d5f2e59d0b6f98b18e32d9851b GIT binary patch literal 18016 zcmd74cR1F6_y_vP-kZmW;xR&a5Ry^$ij0uGS7wPPd+(8zog$@#Y@uw5BrC}#D_J3% zb3c85zjMxYoxjdM=Q`i(`d(M@JfF{d-uLT%y?_E9g44*6o4XQc#@)V}O_BI&5R>*hPY<$Xf<8jfw zgu7;I)*#c7_D8TQkWdhPo7YAE}B{KrT z&*@nNgK?#&)dix^kGiTJcsFb>qa;HwR5H-8-k`dp@r^`~j^rX8N$b*f>-<)9=d4GI zb4`~S(gq!;^-^P>DWjLGx3H)P|Jf~~Sr4PN_UpxpC>rVq!VPX1X|{UPvhdwY<}$YN zt&U={yH)Mm+Xwyy0x76h0gk%r$|P=>&R507!6VKQ1CIJ{=osRZJUtsHJu|LfK$MTN z_`Dxl(VuY8(n_>uitK$`Q9%taNmf-{YgdauriCm;aPFFIu`Mqo1SOfsVlD-*Op1e&Rp$CorEOWBCfJhWcJyQsSP^U%qdFcY1L1< z?VkkVh20n_7k>5ZuIwZxS~G;>0M+l$mq zUBP+}Nix_P4zW@L(#A=O=4owuJ{(iufwq$3;_EgxoagIaUQnYkHa4E|kW}<~G2(1h zHKwSqEX~ryAE}Wvdb4UwfJ6F6&M^XTO5m`f$e!uEkk zWj=~&^G5M<8Aa+Ryt?t5yMi&QDdDhtFNZF1Natxq!B}tqau~#8BqNyhkhD$jUELj| z6eh+yhx2{STjyE4GN-?+h4B@7-H^eEVYfO?Dx&yCn`A*n4r)*IH|Lsyzy{N@v})-t zeUB6roWL0S;7_)p(;F{)(&gJ`g>$uD7Hd*Oah7m0Of68GEj5`7T^S#uGnYH)%&e16 zdGMi25&OOQt?Ms`Vq~wx4i9&4?8OCA7Y;;sbt%`xiP?*Zi^nd)38R>BlTf?dPDBv) zc6?3kJyI@Xh9f!JG4hYP@__Lnb&R5A-MFZ^*Rgz0;g!b8CnEzZm^OG6j#$Zvl>LL9 zEc|ZHT3Dq0lT-91vSiZ&B?s{wQDusD6C%1&J0a=U^NjP?FKn6~vDD0jwqA%=E-ZFb zT!hCe4voxuJby4e`TaX@iTjGFf%Wg`4@i^JZw?=jN$MVg8Km*(?=IpEq4qc-*AZ8* zUj4e_)K7{x!##tiHw z;cjthsgjx+@e8fIST2pnGsg$pm1~LYa*2_Vk(qX_q4B&*1 zT#zicR&Mm4KlcR8n~;8gf47>rzrP|*=#A&`si=WJwD}Sttrga~nwrkv8-Esh5y7XYfyaOAnj`w` z!$zDtC*Iw!aOh{#%n^4b>;L=_=_HfDFIhKEUTTo@r#1W*Z}`N7>GH0xX9@+YWbmg{ zo-lMIJ-@-z=4N)p*YWWbRx z{AS`F)Bnc&J@e{lc{jW-IU|Ed*|p5{9f$1D*jUU+iD6v7fLQ}wUtgc2n_Ji8%O-C< zX5}5lm<238u)?xP1{_v@H?*?4DDL_LE@47J0S8+3{?T98vAy+`l^f>fY&*U(vIjqD z;nrzpi75T%OoyPi?A4H#=Qky-WykojD);5Ua2SvFlVEgYQ(Hy-C zItxtcHxg)i!u?tzHZ)F=!!o{gb(g!;Lp@h6h%Fus3vJpQXjWU++-$LfT{7cZK{+}0 zc=YedM#JZm^_~j3Kegpf^lMMH=DS;wVakQcMSHuuG5lfX<=wCJUUoEm`J&}15|u%Y zkUibE5iqU4b=u){#jF#Y3-z; zMF4&m^2d)KbiFw)0t}>|?e9yjnlTD_Z3ny8ZY?%g{PYTOpM;C76^PmcCrl*h;ZWMpI= zKMTLvBx8gb%3rugX(ah$`s0_nligasWsZ@8{c zDIj*`ip=iMb5tjUdM~x;oz!_S?UCeY9i#(_{#98SMLfD`tBVEkll}-6W%<`Hc|$|S zALHXX1_qJ0OAJ2AmhZGXxw_K#5fqu++TXK`w`vO~-d}Irpka%%ZMS%nDuD3`&ieJdy*@MbKzQtLT2)LoZhYXcWfGKpA~F5VQg~*&UoOeyi#`nz-kcQw3-4FAR~RlE zKgV>kQZ!dCE-l501%{Ry@VkiT6z7WPkn=$J9P3u)8&qKrK=tx8Sg-E-R%}+!o`fDp zrDkMA!BvS3Oo2G!|98wGmh%-QyR4$CYw4RH+xOeCp<8m35OD1Wk#$km?#5I34cUJ9~SD4GmIm zc-=4SH~XJl^I3n@J@@sQg0)u?ISP+$GHMMjoF&ICsy_w1&+ElW%myzi4P+jwheY z=!&5jXl!WcxQ+QW{3@C!;w_C*qshyJ#k)Q60^%oJugY7@ z(Q9Ed=WJXY%LYB3aM!a3?%zmTfEgg3$odi>49m?sJL4E}I!-;xwr)Icc)&|GN$sZc{U^Z#qhXHltdI~K&ku!5Rut3Sn<4T4 zhKbVxVaBd+&9OB-BV)cRfd!pE8`6??Ne?0u2~EO&cV@hW(v!Wbn*D7rZsxQg73}Pa z=6q$Tbc~JpxR1_J4QW7p;~bPd+0pYXC>RRq2=&{X2|>yxePFLFFW-1Y-}vT@=FiTY z!GZga1x@QO@NI6s)AW7McFp^D^w?5I3~eh?LsBY30F$p(IdgwzxT*VL^^#Nd?gi$l z!)PAK(>q;A?~SSRRqM0uQD0Yg(f7Z=jlL!&HK!b-lE|t+^oSBU zjaf9k!j+u9$HTGblVHL+#J^To?F{<^j{e>#9B6@y^lPM~vY&~JK)7qOm5}}8VBv;Q z^ty|ylN0rdnVA`qm9-=9n$OqaVbqH#fa_U`{ZC~0Wru$J;2S(TJoH%n%(V5JZ;BCZ z(?v#&uejYfeWLZ7r=cL@@nZ-tAG0MP>J!)~~eyG?i2MPF|<5$HKXQn2cI2Rf%y)g6r)}bokvWx+4vh>)$IBCd%xp#%|rZg@9-W z!M8_BD~U~pkB_{ZZs?^pWrzUTk_I3LK7RhKFV{9c@(Bui zN$PoOBX0f->%V zb4nPa&vby{w}dT$-TY~`E}`5l<$$HnS)8<{dU~{f{`|S6uaEcMThTb}PpkV<yn3uUVb6G_$h&M{~Fb>f4!3=t1=WKpY26uWU_CesED9w zNx)b-E7DFK2g+3&Uw&i2Z)R3SW$rXM`>7Ag8tV%9L{Z;04x-&;6fr$Ly=8pDFB~e) zCj%@LQC1>ZPoa9w;#7&3NkPRdJ>%Mbb+P|kb77Zrjjou)7bh1Nx(!RVwfWyF4|+pU zYL!0pgC)44jOI?&#;!li-?q)Z{e4oZKiviOG?jhT9nF%5kH2+_PNve65x}!skYU~P zI`8i&#OGQH6P1kc71U}<`oKq_L_sxp%4=xE6&Lm&Q7kSstPDe;#7L;uso`GGXzAmd&SBF4T#36__d$=k^&L0iUAHF1 z`cK>qZps!OYu?w+g#ETVh6!w)5}w-B7?*myGj()vVcpY#dVP-1NJrGk?Q(%eJdL6cB`4-8EO9pZ0tysEbrYEoNN8l+rICeSkeyA4yyjV9?MU;>;o~M`^v((@!Js77q@wBCszG$c#p{u3D^h8XtEbol&2;51gf% zMeE%mjFRBI%n;|uw8A^;hI*ok=5klEV1Jvk3m$G5uE|lg ztRxf|VKFPTl~I3gVH830>qe+v$i|k8*Vk4O{=;WecSoa_me{wFFvLQ$M2W>_bn-Yf zx$3N4R!!jc5xTIMqWi$RS&*O^6{3Ix6~}~JG45*3JwL;8KaOmsCuzk zlCl|?&f7M1TBV^h=HVH;rg&kICHw6y3Sz+o$xu!VlTdrijxVt&&YM27Re0=GtMDz+W3n^GGedYSi?}r> ztmOBENoU3gjFzegQ;Xa0E>cX0vVmTU6>FFFCheORlA1XAcvu87D>TE9&BGM3p!gBF zId<$$C_kDJiH1QBh0&)fGB#|I!Wy=fe~mMTp~O?eSLV=c^X{7xa9q?Re3LHXB3sJ0 z?jo%^WG3`r@saYOy~1v@3}nJz3w^sDUCmV3_=IJOnLaY*w(?^l*lHyST-wEb*(?|o zLVIuhwWLHMWm|ci8M$n(Mu%Bi8YPUtxK(N3MfO|#sp(qF15b{Th>0SomG@UkrZ#fQ-JIgvzF4j&}xTsEQ90K|7+-g z@EA8CGm%r(j5Kc{i8-`1fRV>%lZD}c%t@&uC3%Ael z@q3#Va*6Nxr>ECW8+(fUynnx)FOoZbl4Zw z?6SQmamsi@acxe9wtR+u>-;JCDS7qRIXGM}z!L9)Xt5AH39UyLW~b+;;R42Jy2*8F z&xY%*%_T5TbtInpF$vAnH-!du$(5K%bKJbpntyW=vMag_TQW4{QoGq_d+sVZ45QtspUzh=aJ9^z44iGjAQ@^oy-_ zX#cb=ryzKtPJ(ky+%@Z_P1)PukNe5%espr_!Lp0;o(*FI}A4E9BZmbt?w8_op zx3pL*>shprGNF0Om`C8McSk3L%x-M@FkM@SP^EwsQ!TK++dw+kzqN$66K}cRrz1~I zGa+e@sk$<`)S=WLfL#coO~zqXE*kkzmL)|A6DI4W4#=zF@)ySjzker6>ZbMad}j(A zP$dWAM0P-1eaC z?fe!b%Yb!-(UZGOPO=AQ)3;I&r_E36Jj5tIRYIcD3CC-NCA$hT2eAK&x7GguRVy!f zwq11=bG9{d7CgEN)QqAqG92}a_R`kz2|x`TmYzTh?w+jF)WUr;O!Y@j`I;WB=Y*(w z0YD7Vj&BMI&M6n7l`XYi-dCLKseWNmO$4+n&`QCDBULir;N;cH(J$=1NyTrkjdzcc>I{5CQcni3>TOxMsa5Q z01KP)v>?J|g%(g^7Y|B_ry|W|7xJ z^}=RhSn9KsBx28ylRyg(J)XOC+`CY47)5fxU@AQp+2VGXOmN)d%bc{EP$;C{M} zX+EEy~-1E!OT< z7lu|&(LlpDIH-t%(aQHY=jL{O)$Uz_TLlY}CZ25K<0HYExGyAaUSW;T%*<@t+w^c7 z_7a4!fuW&Pk26VB0?&EV6X&aH=F)PBlG5uV4#PyM~9`FI@8am-LRGsQ0Gx8+I+WBvAp4-i3T% z@tlZe0%=)e3Pk>o6ItvgBd4Ge&n1(rHj z`k~0;{gacE{mpjDs3&DP`ANW5%x})Nf3J0AfjYWVCTDVTvi`v;!y;6UgMiC{(8EFC z;^Inr&fx(SjkMwATjtNMM7NEz`AV`FFU&=BPV94xStS%NBqaa?t zI(u`-tDb|Bf`a28Ukx-o*SdFpz+Wa910P>Xk&tPyC z3wGzbcUzI|@CpP1ODEr3SzV1Ao8DO&YDFq2D1-^TfA$O~9<7-#Yxel`U@1TD1jZ?3 z9|T;5W&CWfP&1vA-`;w{Q2^p8uv^b=sd}1;K(T0C%3xkDvG6cSLe--`2JT&_wN4`v z5y=+n1i^)9?xxseb5L`^jG%cFuk85Ll;9vxfC7U+tR?OS04@w%8Ik!Lg zN#6+dqd=*hvk`FR-qOkv!DkV7q3UY~a)Qr|QQM=mv{c?!!hMQv#m>$SxKY?d;_YeJ zTDOUM9LwMMgk*jE^L?XJQ|4n0gM+#*KdRLpKUVeVeSEontjd8lFy9smkU{Q0{q``& z8^TCUixx!wWE$=@G_urIR&*v@J5G* zIC}P>q8&YPhRrE9y_gjvp z2SX0Fmu!IlnhRokP-@mFZf_ze3B1P2(96Q-&lTtJnXdh>7r<+#-1Hp{tT5g9RKo)e z#Ro@!*H}Cs*3V!%uK3TcDsxIAj4Ex3kv5f!%Ru*R{kU7+>B2HuZO~)l+PW2$Me~VM zCNPgly@4oEMSME&=rREzAzD&08e$;zd*DucYnMWM`}&mb->=$lb)WtqV?W)r9aue} z?1cx*$5y3pXegIDQxL&I5&=p9BCItSOC#BjV{Q(^On)vez7~`Onj}<1ZEfbw-w+a+a;#`!+_>a%gBhsi3N3yJNH^Q{uO_Dq5L`Psxg{CpAToV`APP0 zHe(Av0rBkU>Z%0lNNijj?P4m~esTGa%d%0Npv*8B+9{&S&A;&?I0Ir{|#B#bsL746sykqYw`$5MC{LA;0Y}t;9lF)>*5_;0<7;BN*#wVf zbg?d%K7+ymqWpf9zQ+O~DlSb|)vYxEMP1?ss%FJ2_C-ni?9(qxWGMLSKMR6R$K$R? zGy3u=D9n5FpFe*FqX|G$Bq2q}1ocq@aa21XnzultEPB8&lDZ3fR@W=jfx67^=TW}SWFwrV2@1d1 zoonrYNr?cCEGO=buXsrTu7deIEh>sQ+)#$W+~B1P?Vb+-u00H|X+=%P*!u>Qfxx(d z9Rn06DmDvXE=>g){bi@?6L*07g0aosdj}am;`2Ttq|-WNbhYTi7XiRpO5nqRwE>Or zPSxYRg)PUVI3YR`JOApoW)(24NV8G$@$q%=@>l@g`t5xUUJ%-9*%jnq{qxaGWv(hJ z6s#wJDuCxo{q%f)Mg_1CmYZyz<vax&MdqY_q?(cwb=Mw|Sb~rx_(^EpVZgTCK2_Vc+6RV@;Q8o{5GvSxnU<{0DsR z`HfJ{;Yd6c6gds+GjWt8hts1OqMD2Fg)lsoVshd{Rc4f=ysN5*;vhW5Ss|1&g@8Em z7R&#CACd6Gc!sfgv*6jY8v=62n+k3HKwXag1;U*Gfz@8{@$u(^LXpJSxek9jXK+_y zD(Eagxa%K@9MgCLR6kEGB&bf%9kB@r0a73z(nd1q)ckATjl*baid5voA+NPsfEL4!h`Kr0d`~u#CS^J_RT^*x^{}%r1*RQ`041|EXPMMyb&T-iI zaGM(>SUfn2QSBjEU<&3Cj&cZcZy?v-9W7JhrIUv@fi)#Fv``6!iycuU>H5@rzXIY0 zn{JAVNq~V*M_0ENGA#H=+*SsQz+~_rrt1IzeM?uDBfAM!*7^j%pb{I1njic6@NLMN z&rUWx=Q@ZX>kW^N&O~iVpUuqQ;8@L5U;~Oe#yNWw!ZF`d(DReA?!J)DBSfHbX))! z^FKeOf^~qHgoNZ#etr^|6e^~{{{wP5fa`MzV-pjl`-hOGmzS5XtEi~_p`c0D$DZ$7 zG!x)%iO4+NwK$_kfTj};Rv^R{ex28fZq^Lqdif})kdP4fG?S=(Fq)fo2+jxW;mw-~ zF5^&MM5|IEU?cFlkiUNY>Y5><`gd0%Q)U3AL6v|lA@1Z)IN~{gZh&uywVPzR4A($7 z1X=#{aJAAkU-}_OxZl#3^XR#2fZEYl3Bnt&7nEAt15i}yBXtnK3`&i;zHw=IfWe7j zdo{BCXcCqJcn`Res?4Sy=g*%9wCLe+>kYq<5aZ2u@RuY{ef^5)LN0#E;~yX0CK2=v zo)-HA{r!LgD%!?~az1rY@<;su_U=9;r0 zh8n&1tueCD)eYr0EJGjw(QvzU&uQGb!vuef1f&}s0Km?=FZ&8FKR@kw`s2qS=FuXN zNUG4EebdV;yig_?y}QSSw889-7@~dFO+nK8JzhtNcMNX|6%I=jg@nYb_c=C%mSTaTIF< zfC>x|TM{8#Qq*%*XQ{j{qv!sbU{al9L6cYT4#nHOwdprxQpD5l^UkcG?( z{LqA3-A9X(5!u_JC^>oHfgaOdu5wHUppn%C)i&@ZZBEAmh5zy@QS&q|N$V&H&I{+LT7NI3mV zy*(RR9YD9v0w#oMU98@^P?Zd29$gXmEdf&uN*Qx?5zv?t&;a8E5JQX^%485#0V|wS zKW`om37!b7so6Eadx4P=2s&rR1l4oEOk-gZ`~(cas(mY7x^#b7;S~TyGBvq{yFf_= zo9sV`I1r5v#mneYO~Fg*=H$5YPe8m@mXr4hMi2U2n|&4W*gI1}sc%fcUy~CPgn3A^ z7HGpG2PrQ{fT#z+ZXmuCJh;M0{YIk>(d)oy!9u+;D9H?4eCd~~&jWgzbp@m(AgKy^ zg+YP>Vc#Q+#E1_N(iQSlp0_=j%?0BN>GMo3k&IL>?i}9*?|}U zf=bZH2@qlg92sDpe%T(46KB@ROIXO9m zE95wCDIy~mWK>`Cj-iXKTuf>y*zNL?(tErF$p!8DLku=>!11kBJHZ>HigS>iE#of= z2~mRGk#5|_+uM5g*T~$Kyher~6v(34IXM+}y$oOtq+(7W060L1sIIAbR$NTx%b&%JA)q%n!jXpE)twUB9&^ZtJOFO3JBPpyMEopR`Td(t_Z_6>&!_)m za}>6h!%{N=z;pB})A#4gQFbABd;>lQ9vvNZKJ5k5i`{B{+TZm_{y|_@E~55UUgn=e z#Ky*Yz;;z6Om6_U1H3K-Vt;z0(r>+k~8XKZLAW~+ee zj-OwnDmxOX%3FW5x6zv&B|R6%z>4tS8Bl>7XekW-SPLR|Fbu~oG?iX`Z;O zrsg)$0uXN;GRZUGEKPD9ul+PQN9+bqhm8i06}XCNkA4GcH2DUeVb~eWmUS|Nw#PqO zX|_*dT7SY&y7d*#%{c>h8dDl*K!D2U_0Q6O%n-FsUzVs!Z9w%57?gtlhxmE_{ycoy zsKz&2F#kB0Y#}Z z|7w@UADzfiSi0y*w9>?plFar=VSq8hcD_$PNw;I1(~OoogTn#Z*GAmn-SXvHoFD&EvR8601q1 z5CyH}F?+NwEG`0Jc_-Q#c&6C#?z;tl|JPra0q6}F2mp>YfDHZ_i@6)#Wn+DMor*c8 z9^i}Yf;mO-Ha0fWuJh;Nbx-5pC?+Hpj48g;8YQ7q=Q75D=l%P8MdljqZfhRA|Aml% zIlHMCFhF!inqq_r+H27t@5?6Xn1*`Fm@bPBIe*eP%?BY?ME|U zoVDHFnM>9R70u}rn>Dn9!vZDv^BsyvN8o;VxaaP%$@qo?hXh6sz(;_u;vRsFS(_fa z#6Z~L&UEt>e9pZpYIPMH)u_XTv>9A#@DW&73L((Qwr$*Ctj9PFEhUqkBzG!#8Lzuv zMn+u?dnmhXMP7-d)IL}>xEKb+WHK8#{Ph<+B*XQ=1jS$L4rb4BFeJ}@&Ju;-0k<3| zLb}9_Shyzy4m%*q3MvBC7)XWI7?gt;^U6c3J6H+?Fah*BT)OB)M687my$`@6XuZU`l0dQ+ zo!dTuEMO^;wQQaK0#akg>TaxKqEwM+6!04*BP3l&IA#wdo}TN06gzsX({lH1@KuVoFFz7{=3$fliVEI<5*k zZukWkWMH>Cg_03@P-kX6Zl&_#1B|FQ6@-Nn7T<{!j(s7@NMo7}`W7H4h~F(p+RD35 zPCAW_cNBm|>0JmNaHJamSI`T_E+~BK$^Hjc*oqL3LF>>hQSUU+v4GfxMWlWO?m?hR zVpFQfK!XDqG_!ZqNhH9Y3kur~=WS4}3ZZ8TjA3&a_#}Xt*x^c`lo+7#2gIu7syK!x zy$fKlny*h`%g6#lR2ZIr#h42;_|R=7(n67=`meQ!(fvW}X^2D44r~fE4AI$viR(Xk zuWGK5rZ5TwT2+*cpsb)A_^M$wfO5Uv3A7b8^k@CkOgT~FrnXJ*oa)8_Nvhe01F4q? z>=QKGfNu{ts5zN3f3U6LMG3)JK`|{fW~ES>IJvaoDw5R^mj-erEEep<&Djf}%W~4R zakX(F-+Nr-yXGEq;xu&_}4<_+Bnv|s>bDgsR}*=hcTYIf!Wrl23y)YR}^x^(0AZTfFzW^q^o zAXQsd&FL!0eEfKIuH)hmef0FgfC04x^E2FaDE`0!>CCwC4*DQuu?&f)LrW+=>0A3g zW{}-N(b!^ead8pdzc=pQ+?m3q0Y>@fYN?98AQJ(6fW`^}0)j5-X-C~G#+Rj~3UC_&)aJ&E6_Rs8L)B(vip2%7@^IR=~#MxaHh|u z{iIL&SFT4EswBl$0~G+JctJyhac!yY!_)-khxU9-LiJEh$FgHoRH9t+ppgVHPUjgZ zg8;%tfD@nbM5Yj*5CPyT9_=#yA>i|Nz@MLLns^xkANJ=+!Iler2)9hRB5WA&3ucYp zp9Y0Y>k&Znof;N4t=x+jiHK)pMWm&t11Q4EgmOXV&wYG+=Apbr1cN)8^ajLsTG&y1 zM6T2W68HDJ4A_Pv0F^d>etrT@9Un2#=tGI!a&S6h_Xw!?o(_0dpD2o!H#V{~dat#C=J|DMDsgW*h#m3{WD3W-cWIA+ zErHq^+l>TBvQvx&?5PPrLSdU$004AjbxCZ0i_3T|`IReIu!k=oKn8r^M3%*rR}P=0 z0RmiJUr)>#1!C!~+oTj0kM^Wpg3b#xHm+Bk!bX0txyuMtmdDBA1DB~LCP)m>nd9r6 zzW_Xv(ohhFvHO`q6wTwh!a`i&6*_*@+=W{XYeJmg9|geSb0!)WFh64RX!?u|yBzUM zqSk_%0JS@Euin1}qzou3v4_QJOBoh{F)(=S7qE;K4gw4hzmN8}<`Gzs1si`JhFkBo zLgVc0e9O=<3b;5#%ZGp?>}tY^vi_991w^1a2DbYRmFZ55L%h{OPx|uOTDt4~-`~+d zPyqec9z{71k2I>XPuSPIaRUchlmL`L>qHz3{sPnhQ1O6KoSJ5`!iF&TnSClqc`WWt zIEeo!GYAHVU})5c`S=>p_|S`+*uE5Eus7VluLH;6lHQA_Aa2D&-wc7ySQ=~(9Qn<| z)$%y$0{pLFg$8P7ey#4G>ljO*T=YR;dwsrJK&=1_JyxNnJc1QOSmXNux=mi%bi@F~ zhOGw5%-_edNVr`u8~_fi5KaXG=u0fM1|%hvBj8oX2CJ4^UixNcprL;&w@AXy6;N>O zL$wqvw!xkXhcCIUqZUErVXauxH_$~m2>|Gm;K zu2>t)?OK>zk&ri3tin6?DD!uiq7MLgTv(=bSoDU?BtGwnXy*IdFYH@4f4A0PBLH-Ou6SVF(HU7-OqP&HW$(PBc9fUEhmin1Cw< z$B&!O36ycoZGUKH_}JS^dxq@}>)-aF@^FfhzlczfiUIl7#xp3Vpal&(E#mEL=%mujz>{M)eLB#c zZIKqtesniq8LFX=8F5w^ymI|1U?rg*1zvZ5%Lck0pB5G>Y)L+WAjK^y_|2~Oag6J} z=fZ~V?H*AiSr729>_NvBze8C9PW%5a_H{a7&+K!>PsT~8b!2uWi>}?_l(gg}3p!e> z>x>y*-QE8??%wIk)#^IgKp&4D53*Sp4VCAA!vUmq!@T(N?o< z5ti|vUg$xMEUy@!ODVg5;YBFKg(!61kMB^bGS8v{g1I2PLM)omQl`*Mvdj{k=b}Ce2QD>NDSl;@F znSyHqD4`@p$7unvNDJfO)VT|a*6#M z^^g(gCVi$8)Z_w>>px}$sTBQ@O`5?B`R!ZMBhWXMFZC}`=T5poQOYP}9{*{(C!L=} zg}3S>ruEYA=c^Do@ZB_V+?knW@S8vL@T5b1XPaX z>9Hh#y2O(i3~fVwz0vbL89#>q)B;50;Y}u%Uvi6+7g}z`g=-5N^rZGjVyLp$w1CQDq*w-x|^ zmyb_ea?di+(EL^h(a4&2w~GYpRGp}4B7{_iWG6^toy@8a(6s0CzOFjAlgeSl5N+?KKHp%d=14a1cl zYvXb>dr`Wc2&Uswp!85sAhH_fiu|^nN zJFK)7n*zowC}rmt4}c~_Tfc|G8U$?5kSF4<%+S>nZh3_{!NqF_SSjmq;CA45*vJh+ z*iFgh^h>r)Gcdbzko&-&1N~Rv#)zf;dwR6dTc!>$i$?A>@EWtw4-^hNz;pV3Z;E#O z@9yrivNA1~ex+9aZmiA*9aCMAFK|Hm&{nMh4;949^qYFv+U3BX<1)}oVB+Y)u160x zKSLu~r++tu_{|RCiu;$@nP}XDhMnI`C z8)z_e9B(BB#pjFyLW10ajq#C@*bb~&@MSUnS=2#KpHbj#f<}Mwkxc|_DwF%Z%RKL zN>{OlTXFcS1nabeqXE{eu=jp@v8QVPS~l%;*5GBfORzc;+_7J+qWUGW!njYw?ux)Z}Es#)l&<@sun6K?Bdn@zeq})41#I%}#s>V8O=11Ht|taPHXN zeB-!2*~rtlOeSXzt!uF>CUGiYKm@wS0|4L7&W^NSNR%gJ3xvXuJ#^W6LUhk;RC!X= z)N(89;PL|~(>zyu!5%lSrJ9)G04Xtg$Jf_a`N|^^@Jd0J0uarUo0R1Q$0piAH0~w5o4>hOKM#Shj<|0*L(IrFtqJ8`fEk4*l_*-R$KnR qEJgqOI5POc6{Md7Re3tI+ zTsDrD_8yk5BrZ2wGF*cDt^^xLJ4aiaKW+GU_;@e#@LlE;(BaTHcZY&pJkY@fHA`zpH!^|baPdzITe7A7pWZaR)Uf9KcwZS$2Ywq-RUb8y zwb!3EUbY@2M}pg*X1oGid|aY`ZjgQ5ZU0=gvGsDawneV;%fp2J9yvVF@$a!3s<;UG zSSh+_lSQ0)Wt53HYajbRow!-L{@sZt4EHkcpH77Jom8#K1aAdR9!phG4IgE^pPHJ3 zqos|KArDT^QP)V%$WVcoq-(G3VC17N=j9=w?PAZz?<*vy?XF1TQ8~vaYwJX`u@%v}1<<4KBGA76Pj4FV2YlC14ry;XhmaE{vASY;wfo5$PG z>KvbftUXasL0`Z}PT!uar{#uKaB3qG~?MIJ}RhGZ8QE=;p2I?X0G#&8wuO zZ>8%hYz2(ZfgB%T?G4 zYp9?{QWfSWDhMhNlnreivCaff8+iwJTR|ZgHw_O@XCWhdeqRG`lB}GNzA&E_fuyY> zNOX2}b5kVp2^w00Ey>~pU4>Kyy)=Ccv5vNOI32u-H-YTSFYL|h=&Pt|WTogWLbekS z6tQ!4MV3NDRFEL-z=y1{t+I%nrmHp{FK8)CQugvwR1t&*4#KLc27(S+hVZ*Oj?bDP z=%}lTSMYYT7BLiX^42o2cJS0tF;wt2a#r$Hw$>vHYwF2cYdPuSZS-}$+;j*64n_na zK^Kym3W;CY#n9Km(N=&Ur{pVRXFwo|>T2-n3wT1`G8%#^Sb1ePIR`#{FntAYf|jn4 zf{uVZ509<8ySj|B5bUC>pR*^{)m>2-EJs$r$#_!#J^+432R2oTf_Gr_5JiUob-$|-Q?7TJ^1;wk-_o%`N9uXC1p=nC1|T9f-}-{Fm$p~6;RaE5#k3& zA#3IB2zR_RaCWxN8p29mh)H^BC^|Sh%L*eK;wfZk!(-(oYpbn?*OXE5BPiIqxw?tS z;hf#{)HUQSmE5reWdj=l4|}4XF!E4-6%`jjIW6c$*g@9~D^FI^bX9iKQWC|3EvmW+ z@ar4uBF!}Uy!4fH!CDmDg|Xg#B8Y$CmxDk5;ZwF94E+7K)8v=eXq}osp-!Qcurj(n z@zZ_5y1GjVTXRJ+HEGGTpDr9Dy*X>~*4uiX%6UFKK84kLgQ=s`E-&NA3quXgOp%>V zs>!)br`nn6ZL3{^+`0wZbT1Iwusdn^7)@QP*s}&6zsJh1U%hw3`MQp zQM%V=tExPjp2=jLK7E>Hso#J0l}zZK0$LWah529KYq_ylZ2Ya;x7SIks$UkxXUq9D zutDsY=y1i9mWtCD8joOZjI$cM&d2JjJL7PXOw^8*-=p{!sj0$MbMH@HxOlN9<_yk# z{)bX)M@MpQE;=hK3l(zn=AL-zVAcl@9*FoYTsXukLAiVPh)d_|9Unh_q|!3gx*l()N*n;&l_vu&i!gwjblxDg2WBR`*ZPJ7wd z+Y8?a3o}1NN4Iqa%bv!1938`@5XHRmvzR!{_2B;f{PZXy;r_#ig;6JkXIj#Tip}4F|oe#n@ET<9XBqGGYpUHkGSyGz%b-S9wehRa6RbE-C1Z|3}Ya*maOmey6+>{Fba zv?>odWndQZ>Z#mf{@;v@j5IYhQ4w|5;Dpk1S1cU4#KGZDt$YjXbEjlnai%kOXP$kd zf`h|_r$MWZHgkzW!h+kKJVcl;aC77A?AU)c#D%?n&GRKeoKf6+rhGj59I_;_ zQoCF2@=kMb98gkH+DAuMZ|dE*JFn~Z?c20JMn;|&7atF!V^ffq-#z6e&cwvjYd@%- zN>8OyU-wL>rKMHFLDn=ex!v1qVq;@d?IL8|TTDcH=`A6}Ja}*eHbu%cho=R5)MoW) z#iX8|p5;JgU}wG~{XEO+EUuZVi`bx)V zvCH0)FSe<@{U&MjYmEiZDN)fAZEbC+5S>(Y%kF$fa0(6%4$C#u4o*(8o}Pl~85uFL zvEh#&A74u3HZ6Vk_mh`q2WICMr>0!6IGjJW^nCWHxR_Wk0bK5J?5_GKmRc3=itA%z z4g$i${^jPbu9va9W-Hg+atdqD2@5mM&CN-=(ZR0Nq}MAF1}a$!3JQvS<}gT@C@vOB zdGW#*>(f){%1Wp9P2wrR!QqOv^_j2nqM`Zu0*_VM^NWhCR~N<&@*QG2?x@G|?N)ij zHQnX~8{axWJ?=<|CRA;-pLt7vUVXs_tj0%TfXZw6c2~ecGY%CJPJiBN8SKBt&-ny_$MX3R zgDb5_{5F1_VlmxaP|zTlfg27`P`lon#=U!s^s87iFobq`ZY)TK^GQ_Z5 z=$flL_xcsEaDlN>@qM!GI?TW4=Xqdk(fioMy)t`}U%eXNd&U%x-wXEP`0eHMH*bzW zIE4_JR?l*_qU#^eG1b@qIk%W)TqKKihm|guSrT8+~*)8wR*cibEQux0oX%BmZHObN%@fI<3Y=++; z7AtJ~j)ERFt+IcOgoA6%4w>7L%$WDw>C-jO#ZopG21DuE!#@nC?^VN+RUkT-OgySu zw~d6@C+Fq0D4*P1EK9u|PFb~SNl`ih?(K4V@`MWM=NCsMRn`2ork}-yDjy ze@X;n&6sgc6c7@^h>CXc2CXkMZpY$O81)u$u==Vi*rc-_OOJ7`=Vbds1Ve#Wlzfgv z^dM^iCiCd1D6gZ(jum7aA`k-RKc3HmaJ~rO$W0po1{Lp4M!wHV#zhxaZEdbKT)uqO z^JD3&SNr3H32x!HZ#^n2T#J9i4@BNzmaGBCUyCtuo6Qc*cT zLql`t?p+UG7)}h4+BM-|+-em#Hd$9d^jpTB&;`ud78AKL15nsyh_ zG&76(@qd~zDv_HkqKbm^{9E;Okz#C+kket$IV=1n05EVlLhxpN6>Dk{!j zKYx}3xOIq4>h`T$GysVN1O$p3s}t1EPfSi)7WKgFuF8cSLIOldNeM($=b+W`)GB_a zNb2+F&m-CIIp$DoZheSuo8dxhxtFP6U*1x@S!3a2zrH-zQ-Aw}4q#a=dN#H#H3x?@ z{U52L!AOLkmIc_qRon&7gxQv{^{8p?r|0d=PozFNYxPY z=2Ui`Lr19nH!OBX(xhR&c+2)x+L^quXq_f%u&4NC<~btG zo2n|-NSCK-^ypV3(?$zaI^@@jL%ulNr`Si*>RnyFYJQ`i%XGf}Qm4aUK z*Izt<#kS9=Hw5^4UqXH!Mfx2dANrXsZx z=2+hUF2A7Q7`P^wTulEbcX)xP9`}UR-pSfwhes=h1roCLEAX=%}h>w^#QW- zeJi_^$maV49WU;E==aZtP>My}tz(Y`of)vZZrr%hXU2iia$1BPRxHLOBqT&4F{-cB z6A3fOS+7-OlL|PF>iuq76(osz_WU`lfGFuF3iag4lT#SzM!%@zQm-o}X>jn?LS^9E zDU70`V)dxS0Ml4wxVX6XppeLhxpw0K<6o%m?rsQfJihZk+;a4ad$8D=nq!V- zeKVCubDcY~*={d9;J`~@DhLBoOLV(ZH4+a&G=g9=RL5w3oiYgPJ-uk;=?`Xi)+A=n z;Ho7npoG4d8NUN&fOT>(8(sE^H8y-D$^t8A9b8J zX>oIHHiucjLDTum^<4ZYXTnd)#(;MLV1<8W=Kspf|CO2lD>MIBX8wOWeSc4oadB~Z zhc^$r(cNuS6&MxwhZe};HLeU0#qLTBT#T*nL9%C>gteGIpkT=vsE?mtVu$CRAX!;y zDH94IV?t}1rb%C)2p#}{vJ;h(k}{Yoy>StBik*EAclobt_09m50XUciQuU?=s;YW~ zN?5ZSKSH+odV=9|d%ON|+CA{SzCJ3wr0m z?^v|t;_)N&^hDrTQPltr?mT`>!Lj=8LfuJB+zO2`D|Sb?t{teZt}aSc)cdv!d|%Gx z%a?CZ273rqo?Bax+k5&$*31tiOlxV~o3ylj?DzK1RxGgsbOA6Rfk}vs9iQ`4`*sQ= zLi$Pj>GS8)7!l&o^?GL{n@mYf9scxyLzrber*f4Bb4^2fqjjSja&hkk)0mlqmMP&E zK9CP|LAv=mV8z3vl)N8v?adp$Xhm-GN@hsy)3UPaa;@um{1;u`y?cj3^#`rFK308s z@+P0OG#lm`b+W@>VrXcnT!9IJaTEX_|9~jJ7JCT(Siiri0Km93eynJc2s3L(62OwE z`}dDVD@HtWu3A}HL51MWe0KwnRB>woNFzzsJ)zI*6-P%#O1OQQMch%CYp|-(iW>1E z&BHMPEFu&KIXHSRS)zHwey#BlKG^R7xI;`F~=y%)z zVnQq|EJa5hJ4S=%>KK8u>9wChXanz=kOVPL*ab-8JR?S@y7K&QFC=a*oTu#LGA-SO znsIK$MKQ~eNc_O2>;vEiSQjN6>>c(Q$iKcvc)-AA@~iNqKGy8u9Z24v*}H?Z@oUBp z7|N@Z!TFKTkNIqA-n@Au@zlGn;@LA4%KiJ>-HZ)STv=>FO zsAynJYJ9Tl`#wN#GB1re14mk3s4UNa=5Frn94l%h=55oZ$)$0nM-Wc*_JrGnGO~|+ z>1D4twk;=Z=4JMO)aWtf=uh=o)FtbXp9o=Kc9PHL!wBR7X>O@p06%%jW13;x9JG;* z&5xQ2|E2VqRIbITuF&RWoO8OpI+o1;6YO#Q?nKChCGd!zTh1-g6|Y6`{)*HlV7pxr zrXgqRM%IR!p5C$B3f9yllWDgIz0j`o1ffxfxCZ^6%oim_#S=L@(9MymN{Wi^X@_1( zOD+$FdSd2_DU>=Vjv@m=9|@2A1cr=48?9`wqHgZp=orQ`xF{3qXYvBy7fxhu_fg;_!EXUE$Rz^`uS(s z*^ROXM!94v7pp{>zVu&9uok!e*v3A&dE<0a58giI;p43Ctvr|3r&FsYE-q8tPkn!$ zz>!FK6P1ffK%RknU>UBD4zB5;{h~j;!SzVY;_JAVP=zxws_@Dq?tSPO3dJ^g(Pp#& z?|r+xf{TxDo^9_T=FUQ(JoMXgD~=f#xw)P|fU2%;E4^^M{Ag@`$>$ezzZH6NRzk6u zmWCmd>C>dBht9c9hqyb;y!*xiBNc`xo12@Xl9H0V+P+UmYDlf52wuAMNy_8L2epdW zNYQ$)_rF_iw%e5~KAwylr$_sKCyLe%AKGI)Dn03%ec|$SU08%LQ=~C2n$u6ooQ~(zznn8!FuGvULmSc7ZdK@nZz$<*bXz%a z1^c6+#pH6YOPiNxSw-d=U9Cl~Q}M^1h9Lms8Bh>&jbOO!bb*ue`nPf2BR?NLWPoDL zsdMK>=Ap=7d=T1TJZ2lde)eo6kDk%MzPp>DR`{bc+aykBB|>$7}|-t~5!< zLZ?~pwbAlXL+WVYvVcYui+DT3ks~de_Y&7rxcK?$M@B~6Mb=3f1yIHUZibJapBI>@ zQs4v&qglkgswiJ3Cd)ZR3mY%%A2baPDfO}Ty|y;588t1vwkk@Y_~kECcW&9wDa7@C zp0@RsT0MGC{w(RsW8ON1Z2_i)3jCJ+x0aSCKq?}Pd~FzAa#GS?US%u?4<4N1cTfEm zzPjoMJdUoe?qJh%xeHue3T|#(E%1p%U~Xo?=mUZLP5x-p&4G!<41M!|#5n43;EQNs zmD$#!{K@<(!O>19v8s}hMlMd;xslOxn7g*^1GvXoh1}vf(uwnw{H>XdivO7Bkp!lY z3^}U5fx%;bDXGmk3Wf3n7^wK;(n0HCFzjGsEFbm<8`dU1u}}wFS%v*EU9!;eX@+;y z!Cri{$lt4dPBYl;0J=|sq3YE^{!xcSF9k#6c>L3*ktgIL{`#J-$n1vHs%-hVEF)X{ zG}l+Nn?y=foC0=3YjslIk|E=z=gpCCbp)qc=zMckuBbmIb)kb!!6>uvOh@I%uBeo3 zZkFeMagdQWz*OO}WjS@7iz9rKvu8{iV@dfh7&7*3J|YWqQ~RV3!R^AY!I1$mo23j{ z$cH*nJ7F7L6Rt|dKJn*CpZu0B$gynR+V`Z(uwJfj07c5r8_#*rmyEQf7-V+$BosyJ z84EElpUlbm*!+gvpBZH9Wob$2N+0ny>&Y?9msvAzgEU>HIzBmTR-!gTW59>UjJmks z{gAmQ*vo9yowMKI=`%ln;D}1H1>;EhcG*`m*;JwmpZ_-J!WYM1MElwnIw?J5HV!+7 zb5?0AXwa;XEQ+hscl8}LWxE_NjrLW9i^YXbaapx4ZFhAEjnkjvGd8}sv}GH4E3v*|^nUiKfGeUnHid$lU}M`yZt z=fJxNe?~yUDPTb6+?` z$i_0!2l&xcX~`@HT+R1x8ozGF0q!qk9&M@+Iwrc;L0+7O_0^;buK1ytpmBk6_A^{7 z+j{jaucIcVT%MnfEl^gk_GiLJZ4tvl?$Q8xa@5NjrRIcnn{Q?%o zvg?hli2EQ4g&cP{a3PdaNGY<&Z8r*dYH)r#f|qOAP$(#D!rd$W{^IR9=Fw1W%QP-J zw15A8R7g?3-vKDsP{PBn)>+8}21<7K^Z@&<4*>}yCdR~p&&C=XpFE>RK4a26vp6Jn zj{yz0tlHA`5JEzSxmNP9Y#;}6w#qxTh#nT*@sY4 zxdy##{7{b?2wH=B5rhtdavxsoE(B?TyTtA~z`%LM$cPEMYqBE;X6*Oo%j(*`j$2+o znO%kP`IB2Hi=~-90|3KG$;o|0N%Gho@ub=E-%)Fsz!I)b<}6)m0Sb8!${_f_^(bAj8F+-K_hzny{lzr{lrV$V-Ww4sMy1P)zkAhboxy5AXd*d*g(xd!pPtTBCSC2^}O@Fi!5{3>Ad?P8)F7J=G7 zMn~naugT-uneZgcwGK1?gKU8dXs9W^cIy+{WJ%hrJ*Lnt~~K-%(cX=^-)v^1^mK4_oo*Y5 z1c}2iSZ$v=JJZRNbHFW@3}pMUW2I`59`0|*m+WZcKRtf}1{(O{S&g(JRh1HObE#PS z)N0Tx9XP5*q9O2?g^^3!!?>(&y*#}#r>|{6FToeyAC$~K`hiol+e9U*-7eS30Ged< zvts(@Sn)LE@sh$53>jH3fS=#f)+3j~0Z*u@{zr6Q>w9b@_2}fYo|}_=duNF+t`s4` zxGr6^ck-XhZ@%hhPnvvD1!S&OTjkigrIqcfv+Yt^KRc!>&eY(z)z@Y(^R)=Vnnx^k z37uXPo^~E_b&p8`Jq7eM#tlgay9d&t@z#z#;Dma_b_VhZE?L~4P{XXA&dZAmD^=&@ z)NjZ|<~QMrnLiMr=U>L(^oaB1q4mg#>C+QgT{mu=w##@pu;MS6W8hKn5gzd&l~dG6 ze^hYHFG6ok9F~vF0ReAos(do}oMeZP97KYZcSFq61L8aT;vs4nzy<1Vd{c5z9Yhys zdI+}UJwM$9H-mA(NP>5u$;eyPqxmf)SN`k-*G{o7`j?fEKJJR**J8WUO_96zTe1q- zkkAIv2I(4tON=zxbfWdH^_m6@p_Ky(JqgmMuGgh!>4M|kBu|^)Hj#VD3g< zUfv#XSzjg4E3p&_c1+NgJp(@f%!J{@Y73enV_GpJ^BE&KaR09VGfh_WM%gAPH{3x8 z01jwqtT|c4=hrEaQM|US-E;Q*`3~t{)jJ>EJjBus*DcoI>L+=d5{kswx2(}yURdc1 z2+Sc~_Dugx<5$zicw-ZA{6uxgj-A<3D?jdeLeZcOYP?NNO@oij=>g;d`lQY4G6{y_ zNzYz7)?S!jE)M&2Vlt{&KR*w`d`S|ER-?p2v4&jJx#9VJ8!x&2B7~3Ap4>}&yea14 zLoZ*@5SRs97qb5Q$}QnCIdJ+Fh>MQgJG{ZepY#VX`R|q-;QG=!yPA}`Um=-y3n#s* z)wh=|psBaNKikG#GPh^?!`H8FG<0-T%&e@NS1GDv4eExO-O5~|%4BB~jodN;)%e8dwmVopmH5q=;e(kHx*+^dQ^1)KU4OYBM;jY?Fis6q06B zjotXtkol}5aV;q#^{JmzA85Ja^ysP|*)=@zT=@<%vG#)K@so7#xeJ{bO!um^Hoh3` zwll~qWLTo{Gh?tTWmz475DrUH8iBTG3Cjj9~xWC zIm+t6878XcZ3?fo;Mnht4+2}FJDbSmHQjyVdu`aGHhuN z#KyO}64EG0hhD2lv0>1@b(PB@hz^Z$A;(^TiRB;H_P9NoC4`Q*ashsp_F^D`NMib3ql2z0dljZUglm78X*uR2nrqp&hBY_ z&xRltGyiFL+1fmS2a5umtQ)Ke1%YEuhtygtA$ZN0_5|N9Ae{Q1{_Z#&{vpGy{?yt~ z=;_i-gWSIW%t79_I4ZTKs-=aVDVvolUml8(tbA`ScNUZ5C&AFg$VQ7sMX^aofh_J_ zU7aP6n+}fqGtE`nTU)PRHZB|l0cjlolV8*h^MPN~O(s}#&)VUbz`2^krk5^VdTrOh zbp6H+%a3=@{Yk4A>X_Y({zw-9+CI&IZ#>N7=4Xgd0SfI;%XfN=Rh!_cEoTot_2bikvYl1jo)q2GlHndHREg4z3 z&Ib*4a{BrVckkXk%*q;7HdA`QF7E_PS@O^)3}7w4VtdAQm1(R)R8Xb7hBB#3cm94vuIJ|`#;*3{I%t?A z=HAgC7jii+D<{Y6J8uJ`wzn2M7NFIkL`2A0Sz-A2_z-M{eDK`4b0YqWnHoin+iMZv zx89!}I5)1aM{L_3v;#j(Djy%O+}=u#wy<(w%Vxy?%ihAyDteR2gCKkZoo72yn9$di zXCM5F(Qp;n#0ToQs!D{%EbHr^o=(E208GAx5W9OqYp6<=JjZ{16`w6O|1G`sTY6^H zBpM`UpcebSjeJ3j89h2YG;{|{1juz%M-crim93TvMu_$I_b)0sNxKJVOnRt4lL?fKQ@LfDT4!8X4VsrIT7JtpL00HQS#RoPy6WI|u}9Uzv9@=o)~K z2dLqe8%Nkjx7bKmf|S-sI&0@B53ewMamP zynnAi+2=LW8xCZ?uSB*<$qC6<|iF7LC?J0r+K3=E_@-<&z3LtwflEj%05J) zfqEK~otb%^QVpuffhy2bfj0K;+TGV<=yB=w?{tUJIaSY^3zA_?XSRZ2^sVjfLm*8B z4*ooL7ldR~Ij*#%6t;lpA(}VSv7Ns=FYg$J4R-ObcXtS1HdbD=Uo$L2xxCbzspe5{Es^#x@(3 zQlwbsWdn$4MzJ2Yc<5dDOxUFMjUs2m?66637E}PF)=25-Fbm6cWj{4;VEtwcyLQgs z!{H$gA==qxftC*G(#225#zBbrwJ$9@W={yv=Yz<;Uf)mO%VrH+!PwYX6L$;ntv4vo zKtgXkh?K4<`#^$RFZeWqksDdKnaTESeoI13c2*V?nBoYP450b&65JX6PzV(^A>Rr4jsn>oTqO5j>Sk#(FNpe%5ZW>tQLkp2mN8(iMN6%I$O)Y5%!PLWDEopoel!1WXYQ>fAMSg02AhiLB&n^_;++pBEs6LxeA39vJsPlbtJ8^KFwY0Nc59s{M5w2X{e&=PHvGF9Kz8zE5#r9X3sbBq}%+|}bnu7W`PudO+@ ztsUqZD9#-mTq_eVbnCkX)f`_5ur&Ymp`RlomT#{E_hP;=fj?a0<&DGP_JaKTPC`Q8 z@VS%!JQlYWs4OC0Q;%`haB_j1A(I6vmxm4r0DDxmwN@qh^{)T74A^E=T*J@G6ynOf zEwgkGGv*o;ghUe5$z*cztk6Xcj&H&(uCA^#nq{o8^lOXLq^kD?-%Pm$QPtp^5bZUf zL$5N8)fgNrF}Qw*TR`Fh;yX8oO7{%D0VasDPd!m85@gMsX62p3flw_QBhFgkw}WI` z#_+Pjk`icJr}m!d1Eu{5emkG?0dUuZbbJSnsn|=1%QRc5Or6FQapmRnpE!bMBRU@M z=ulq);z2qY;%j}!E}F8wD9}y%ZRCWM&LggxBZ02$tOP44sl&)!;2=FZK{P1T!y{1ntY8fySc z5j?pQ-VrDXfMQwO+RF&=t?+_P>-XQtg8`R-#Z9ykap{bO>xyXw^&mC$L zpU|L_TDC}_NX#_;c@Q_`EFNv#!7HC{EJ0Y14ObN_zp zZvye4>_3FCV|-UI6X>;}Gcc_xpP#Yz%pKfOEH1#v#Kgx>A*}p?(nEJBe(x%m6`{N) zeTZ2Eyj-DcR(cqb1I)LZ&!OpAJG=~V{yy!WW_Ow$0wl%6y0svK$@b%^n1gqD@0>df=#|{=zH=BBHHO8y(U~JLuP}E3;q;$? zwEOovMn^_UvOqzJm4V^s?da$&b_^!T4uZ=0_t9)y-(f5R y3^BCmmA&W_sbm7>!kyR1n_d5_uh!`G;KA=Vr_W^>JHTthC?$DyY__b$wf_duoh%Rl literal 0 HcmV?d00001 diff --git a/jetstream/data_replication/replication.md b/jetstream/data_replication/replication.md new file mode 100644 index 0000000..632f22f --- /dev/null +++ b/jetstream/data_replication/replication.md @@ -0,0 +1,169 @@ +## Data Replication + +Replication allow you to move data between streams in either a 1:1 mirror style or by multiplexing multiple source streams into a new stream. In future builds this will allow data to be replicated between accounts as well, ideal for sending data from a Leafnode into a central store. + +![](../../assets/images/replication.png) + +Here we have 2 main streams - _ORDERS_ and _RETURNS_ - these streams are clustered across 3 nodes. These Streams have short retention periods and are memory based. + +We create a _ARCHIVE_ stream that has 2 _sources_ set, the _ARCHIVE_ will pull data from the sources into itself. This stream has a very long retention period and is file based and replicated across 3 nodes. Additional messages can be added to the ARCHIVE by sending to it directly. + +Finally, we create a _REPORT_ stream mirrored from _ARCHIVE_ that is not clustered and retains data for a month. The _REPORT_ Stream does not listen for any incoming messages, it can only consume data from _ARCHIVE_. + +### Mirrors + +A *mirror* copies data from 1 other stream, as far as possible IDs and ordering will match exactly the source. A *mirror* does not listen on a subject for any data to be added. The Start Sequence and Start Time can be set, but no subject filter. A stream can only have 1 *mirror* and if it is a mirror it cannot also have any *source*. + +### Sources + +A *source* is a stream where data is copied from, one stream can have multiple sources and will read data in from them all. The stream will also listen for messages on it's own subject. We can therefore not maintain absolute ordering, but data from 1 single source will be in the correct order but mixed in with other streams. You might also find the timestamps of streams can be older and newer mixed in together as a result. + +A Stream with sources may also listen on subjects, but could have no listening subject. When using the `nats` CLI to create sourced streams use `--subjects` to supply subjects to listen on. + +A source can have start time or start sequence and can filter by a subject. + +### Configuration + +The ORDERS and RETURNS streams as normal, I will not show how to create them. + +```nohighlight +$ nats s report +Obtaining Stream stats + ++---------+---------+----------+-----------+----------+-------+------+---------+----------------------+ +| Stream | Storage | Template | Consumers | Messages | Bytes | Lost | Deleted | Cluster | ++---------+---------+----------+-----------+----------+-------+------+---------+----------------------+ +| ORDERS | Memory | | 0 | 0 | 0 B | 0 | 0 | n1-c2, n2-c2*, n3-c2 | +| RETURNS | Memory | | 0 | 0 | 0 B | 0 | 0 | n1-c2*, n2-c2, n3-c2 | ++---------+---------+----------+-----------+----------+-------+------+---------+----------------------+ +``` + +We now add the ARCHIVE: + +```nohighlight +$ nats s add ARCHIVE --source ORDERS --source RETURNS +? Storage backend file +? Retention Policy Limits +? Discard Policy Old +? Message count limit -1 +? Message size limit -1 +? Maximum message age limit -1 +? Maximum individual message size -1 +? Duplicate tracking time window 2m +? Number of replicas to store 3 +? ORDERS Source Start Sequence 0 +? ORDERS Source UTC Time Stamp (YYYY:MM:DD HH:MM:SS) +? ORDERS Source Filter source by subject +? RETURNS Source Start Sequence 0 +? RETURNS Source UTC Time Stamp (YYYY:MM:DD HH:MM:SS) +? RETURNS Source Filter source by subject +``` + +And we add the REPORT: + +```nohighlight +$ nats s add REPORT --mirror ARCHIVE +? Storage backend file +? Retention Policy Limits +? Discard Policy Old +? Message count limit -1 +? Message size limit -1 +? Maximum message age limit 1M +? Maximum individual message size -1 +? Duplicate tracking time window 2m +? Number of replicas to store 1 +? Mirror Start Sequence 0 +? Mirror Start Time (YYYY:MM:DD HH:MM:SS) +? Mirror subject filter +``` + +When configured we'll see some additional information in a `nats stream info` output: + +```nohighlight +$ nats stream info ARCHIVE +... +Source Information: + + Stream Name: ORDERS + Lag: 0 + Last Seen: 2m23s + + Stream Name: RETURNS + Lag: 0 + Last Seen: 2m15s +... + +$ nats stream info REPORT +... +Mirror Information: + + Stream Name: ARCHIVE + Lag: 0 + Last Seen: 2m35s +... +``` + +Here the `Lag` is how far behind we were reported as being last time we saw a message. + +We can confirm all our setup using a `nats stream report`: + +```nohighlight +$ nats s report ++-------------------------------------------------------------------------------------------------------------------+ +| Stream Report | ++---------+---------+----------+-------------+-----------+----------+-------+------+---------+----------------------+ +| Stream | Storage | Template | Replication | Consumers | Messages | Bytes | Lost | Deleted | Cluster | ++---------+---------+----------+-------------+-----------+----------+-------+------+---------+----------------------+ +| ARCHIVE | File | | Sourced | 1 | 0 | 0 B | 0 | 0 | n1-c2*, n2-c2, n3-c2 | +| ORDERS | Memory | | | 1 | 0 | 0 B | 0 | 0 | n1-c2, n2-c2*, n3-c2 | +| REPORT | File | | Mirror | 0 | 0 | 0 B | 0 | 0 | n1-c2* | +| RETURNS | Memory | | | 1 | 0 | 0 B | 0 | 0 | n1-c2, n2-c2, n3-c2* | ++---------+---------+----------+-------------+-----------+----------+-------+------+---------+----------------------+ + ++---------------------------------------------------------+ +| Replication Report | ++---------+--------+---------------+--------+-----+-------+ +| Stream | Kind | Source Stream | Active | Lag | Error | ++---------+--------+---------------+--------+-----+-------+ +| ARCHIVE | Source | ORDERS | never | 0 | | +| ARCHIVE | Source | RETURNS | never | 0 | | +| REPORT | Mirror | ARCHIVE | never | 0 | | ++---------+--------+---------------+--------+-----+-------+ +``` + +We then create some data in both ORDERS and RETURNS: + +```nohighlight +$ nats req ORDERS.new "ORDER {{Count}}" --count 100 +$ nats req RETURNS.new "RETURN {{Count}}" --count 100 +``` + +We can now see from a Stream Report that the data has been replicated: + +```nohighlight +$ nats s report --dot replication.dot +Obtaining Stream stats + ++---------+---------+----------+-----------+----------+---------+------+---------+----------------------+ +| Stream | Storage | Template | Consumers | Messages | Bytes | Lost | Deleted | Cluster | ++---------+---------+----------+-----------+----------+---------+------+---------+----------------------+ +| ORDERS | Memory | | 1 | 100 | 3.3 KiB | 0 | 0 | n1-c2, n2-c2*, n3-c2 | +| RETURNS | Memory | | 1 | 100 | 3.5 KiB | 0 | 0 | n1-c2*, n2-c2, n3-c2 | +| ARCHIVE | File | | 1 | 200 | 27 KiB | 0 | 0 | n1-c2, n2-c2, n3-c2* | +| REPORT | File | | 0 | 200 | 27 KiB | 0 | 0 | n1-c2* | ++---------+---------+----------+-----------+----------+---------+------+---------+----------------------+ + ++---------------------------------------------------------+ +| Replication Report | ++---------+--------+---------------+--------+-----+-------+ +| Stream | Kind | Source Stream | Active | Lag | Error | ++---------+--------+---------------+--------+-----+-------+ +| ARCHIVE | Source | ORDERS | 14.48s | 0 | | +| ARCHIVE | Source | RETURNS | 9.83s | 0 | | +| REPORT | Mirror | ARCHIVE | 9.82s | 0 | | ++---------+--------+---------------+--------+-----+-------+ +``` + +Here we also pass the `--dot replication.dot` argument that writes a GraphViz format map of the replication setup. + +![](../../assets/images/replication-setup.png) \ No newline at end of file From 615153208434ba2ae0dcf78e219b87029ee79eb7 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Sun, 28 Feb 2021 14:44:14 -0700 Subject: [PATCH 27/31] add replication Signed-off-by: Colin Sullivan --- SUMMARY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/SUMMARY.md b/SUMMARY.md index 45e3ab7..1e59b38 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -122,6 +122,7 @@ * [Streams](jetstream/administration/streams.md) * [Consumers](jetstream/administration/consumers.md) * [Monitoring](jetstream/monitoring/monitoring.md) +* [Data Replication](jetstream/data_replication/replication.md) * [Clustering](jetstream/clustering/clustering.md) * [Administration](jetstream/clustering/administration.md) * [Configuration Management](jetstream/configuration_mgmt/configuration_mgmt.md) From 19de4f5841dff5dd003aa9ef1b877d306fcfaaa8 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Sun, 28 Feb 2021 16:58:09 -0700 Subject: [PATCH 28/31] use natsio/nats-box Signed-off-by: Colin Sullivan --- jetstream/getting_started/using_docker.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/jetstream/getting_started/using_docker.md b/jetstream/getting_started/using_docker.md index f46a7fa..f36feef 100644 --- a/jetstream/getting_started/using_docker.md +++ b/jetstream/getting_started/using_docker.md @@ -1,6 +1,6 @@ # Using Docker -The `synadia/nats-box:latest` docker image contains the `nats` utility this guide covers. +The `natsio/nats-box:latest` docker image contains the `nats` utility this guide covers. In one window start a JetStream enabled nats server: @@ -11,7 +11,7 @@ $ docker run --network host -p 4222:4222 nats -js And in another log into the utilities: ``` -$ docker run -ti --network host synadia/nats-box +$ docker run -ti --network host natsio/nats-box ``` This shell has the `nats` utility and all other NATS cli tools used in the rest of this guide. From 6d4e66410a2577fe7473542e866ef0c0f73d1de8 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 1 Mar 2021 09:31:13 -0600 Subject: [PATCH 29/31] Grammar/syntax updates --- jetstream/data_replication/replication.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/jetstream/data_replication/replication.md b/jetstream/data_replication/replication.md index 632f22f..7561645 100644 --- a/jetstream/data_replication/replication.md +++ b/jetstream/data_replication/replication.md @@ -1,6 +1,6 @@ ## Data Replication -Replication allow you to move data between streams in either a 1:1 mirror style or by multiplexing multiple source streams into a new stream. In future builds this will allow data to be replicated between accounts as well, ideal for sending data from a Leafnode into a central store. +Replication allows you to move data between streams in either a 1:1 mirror style or by multiplexing multiple source streams into a new stream. In future builds this will allow data to be replicated between accounts as well, ideal for sending data from a Leafnode into a central store. ![](../../assets/images/replication.png) @@ -20,7 +20,7 @@ A *source* is a stream where data is copied from, one stream can have multiple s A Stream with sources may also listen on subjects, but could have no listening subject. When using the `nats` CLI to create sourced streams use `--subjects` to supply subjects to listen on. -A source can have start time or start sequence and can filter by a subject. +A source can have Start Time or Start Sequence and can filter by a subject. ### Configuration @@ -166,4 +166,4 @@ Obtaining Stream stats Here we also pass the `--dot replication.dot` argument that writes a GraphViz format map of the replication setup. -![](../../assets/images/replication-setup.png) \ No newline at end of file +![](../../assets/images/replication-setup.png) From bc3b2b9cd7b09577f9f21846ed910c3f03f0d820 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Tue, 2 Mar 2021 21:39:26 -0700 Subject: [PATCH 30/31] add jsz http endpoint Signed-off-by: Colin Sullivan --- nats-server/configuration/monitoring.md | 55 +++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/nats-server/configuration/monitoring.md b/nats-server/configuration/monitoring.md index 0416bad..15ec707 100644 --- a/nats-server/configuration/monitoring.md +++ b/nats-server/configuration/monitoring.md @@ -246,6 +246,61 @@ You can also report detailed subscription information on a per connection basis } ``` +### JetStream Information + +The `/jsz` endpoint reports information about the JetStream subsystem. + +**Endpoint:** `http://server:port/jsm` + +| Result | Return Code | +| :--- | :--- | +| Success | 200 \(OK\) | +| Error | 400 \(Bad Request\) | + +#### Arguments + +| Argument | Values | Description | +| :--- | :--- | :--- | +| acc | account name | Provide information for a specfic account | +| accounts | true, 1, false, 0 | Provide information for all accounts. The default is false. | +| streams | true, 1, false, 0 | Include stream information. The default is false. | +| consumers | true, 1, false, 0 | Include consumer information. The default is false. | +| config | true, false | Include configuration with streams and consumers. The default is false. | +| offset | integer > 0 | Pagination offset. Default is 0. | +| limit | integer > 0 | Number of results to return. Default is 1024. | +| leader-only | true, false | TODO | + +As noted above, the `routez` endpoint does support the `subs` argument from the `/connz` endpoint. For example: [http://demo.nats.io:8222/routez?subs=1](http://demo.nats.io:8222/jsz?accounts=1&streams=1&consumers=1&config=1) + +#### Example + +* Get Jetstream information: http://host:port/jsz?accounts=1&streams=1&consumers=1&config=1 + +#### Response + +```javascript +{ + "server_id": "NACDVKFBUW4C4XA24OOT6L4MDP56MW76J5RJDFXG7HLABSB46DCMWCOW", + "now": "2019-06-24T14:29:16.046656-07:00", + "num_routes": 1, + "routes": [ + { + "rid": 1, + "remote_id": "de475c0041418afc799bccf0fdd61b47", + "did_solicit": true, + "ip": "127.0.0.1", + "port": 61791, + "pending_size": 0, + "in_msgs": 0, + "out_msgs": 0, + "in_bytes": 0, + "out_bytes": 0, + "subscriptions": 0 + } + ] +} +``` + ### Route Information The `/routez` endpoint reports information on active routes for a cluster. Routes are expected to be low, so there is no paging mechanism with this endpoint. From 042264f33b07206dfae7a1b778a40d762ac103de Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Wed, 3 Mar 2021 09:30:05 -0600 Subject: [PATCH 31/31] Grammar/syntax review --- nats-server/configuration/monitoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats-server/configuration/monitoring.md b/nats-server/configuration/monitoring.md index 15ec707..a93734c 100644 --- a/nats-server/configuration/monitoring.md +++ b/nats-server/configuration/monitoring.md @@ -274,7 +274,7 @@ As noted above, the `routez` endpoint does support the `subs` argument from the #### Example -* Get Jetstream information: http://host:port/jsz?accounts=1&streams=1&consumers=1&config=1 +* Get JetStream information: http://host:port/jsz?accounts=1&streams=1&consumers=1&config=1 #### Response