From f30806c9b57d08c8fe319ca5be6aad3ab2bb7e01 Mon Sep 17 00:00:00 2001 From: Scott Fauerbach Date: Sat, 17 Apr 2021 07:07:16 -0400 Subject: [PATCH 1/6] Update streams.md MaxAge should be nanos Name -> Cannot have spaces, tabs, period . greater than >, asterix * --- jetstream/concepts/streams.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jetstream/concepts/streams.md b/jetstream/concepts/streams.md index fd776d6..f60b09f 100644 --- a/jetstream/concepts/streams.md +++ b/jetstream/concepts/streams.md @@ -16,7 +16,7 @@ When defining Streams the items below make up the entire configuration of the se | Item | Description | | :--- | :--- | -| MaxAge | Maximum age of any message in the stream, expressed in microseconds | +| MaxAge | Maximum age of any message in the stream, expressed in nanoseconds | | MaxBytes | How big the Stream may be, when the combined stream size exceeds this old messages are removed | | MaxMsgSize | The largest message that will be accepted by the Stream | | MaxMsgs | How many messages may be in a Stream, oldest messages will be removed if the Stream exceeds this size | From 49fcccf0ecaa8ab6d739a4c5e2fb69247473e079 Mon Sep 17 00:00:00 2001 From: scottf Date: Sat, 17 Apr 2021 07:24:23 -0400 Subject: [PATCH 2/6] name description --- jetstream/concepts/streams.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jetstream/concepts/streams.md b/jetstream/concepts/streams.md index f60b09f..7a00969 100644 --- a/jetstream/concepts/streams.md +++ b/jetstream/concepts/streams.md @@ -21,7 +21,7 @@ When defining Streams the items below make up the entire configuration of the se | MaxMsgSize | The largest message that will be accepted by the Stream | | MaxMsgs | How many messages may be in a Stream, oldest messages will be removed if the Stream exceeds this size | | MaxConsumers | How many Consumers can be defined for a given Stream, `-1` for unlimited | -| Name | A name for the Stream that may not have spaces, tabs or `.` | +| Name | A name for the Stream that may not have spaces, tabs, period (`.`), greater than (`>`) or asterix (`*`) | | NoAck | Disables acknowledging messages that are received by the Stream | | Replicas | How many replicas to keep for each message in a clustered JetStream, maximum 5 | | Retention | How message retention is considered, `LimitsPolicy` \(default\), `InterestPolicy` or `WorkQueuePolicy` | From be6dfc4cbf9c1b9aac68ae796c2cce248318c8bd Mon Sep 17 00:00:00 2001 From: scottf Date: Sun, 18 Apr 2021 08:02:27 -0400 Subject: [PATCH 3/6] consumer changes --- jetstream/concepts/consumers.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/jetstream/concepts/consumers.md b/jetstream/concepts/consumers.md index a411233..a6dcec3 100644 --- a/jetstream/concepts/consumers.md +++ b/jetstream/concepts/consumers.md @@ -1,8 +1,8 @@ # Consumers -Each Consumer, or related group of Consumers, of a Stream will need an Consumer defined. It's ok to define thousands of these pointing at the same Stream. +Each Consumer, or related group of Consumers, of a Stream will need a Consumer defined. It's ok to define thousands of these pointing at the same Stream. -Consumers can either be `push` based where JetStream will deliver the messages as fast as possible to a subject of your choice or `pull` based for typical work queue like behavior. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging. +Consumers can either be `push` based where JetStream will deliver the messages as fast as possible to a subject of your choice or `pull` to have control by asking the server for messages. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging. In the orders example above we have 3 Consumers. The first two select a subset of the messages from the Stream by specifying a specific subject like `ORDERS.processed`. The Stream consumes `ORDERS.*` and this allows you to receive just what you need. The final Consumer receives all messages in a `push` fashion. @@ -21,8 +21,8 @@ When defining Consumers the items below make up the entire configuration of the | AckPolicy | How messages should be acknowledged, `AckNone`, `AckAll` or `AckExplicit` | | AckWait | How long to allow messages to remain un-acknowledged before attempting redelivery | | DeliverPolicy | The initial starting mode of the consumer, `DeliverAll`, `DeliverLast`, `DeliverNew`, `DeliverByStartSequence` or `DeliverByStartTime` | -| DeliverySubject | The subject to deliver observed messages, when not set, a pull-based Consumer is created | -| Durable | The name of the Consumer | +| DeliverySubject | The subject to deliver observed messages. Useful to setup an alternate subject for a regular NatsSubcriber can listen on that subject. Not allowed for pull subscriptions. | +| Durable | The name of the Consumer, which the server will track, allowing resuming consumption where left off. | | FilterSubject | When consuming from a Stream with many subjects, or wildcards, select only a specific incoming subjects, supports wildcards | | MaxDeliver | Maximum amount times a specific message will be delivered. Use this to avoid poison pills crashing all your services forever | | OptStartSeq | When first consuming messages from the Stream start at this particular message in the set | From 470c950f3b89382d5f446211f0e6127b515fc7bb Mon Sep 17 00:00:00 2001 From: scottf Date: Mon, 19 Apr 2021 07:45:11 -0400 Subject: [PATCH 4/6] discard policy --- jetstream/concepts/streams.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/jetstream/concepts/streams.md b/jetstream/concepts/streams.md index 7a00969..fc871a5 100644 --- a/jetstream/concepts/streams.md +++ b/jetstream/concepts/streams.md @@ -16,10 +16,10 @@ When defining Streams the items below make up the entire configuration of the se | Item | Description | | :--- | :--- | -| MaxAge | Maximum age of any message in the stream, expressed in nanoseconds | -| MaxBytes | How big the Stream may be, when the combined stream size exceeds this old messages are removed | +| MaxAge | Maximum age of any message in the stream, expressed in nanoseconds. | +| MaxBytes | How big the Stream may be. Adheres to Discard Policy, removing oldest or refusing new messages if the Stream exceeds this number of messages. | | MaxMsgSize | The largest message that will be accepted by the Stream | -| MaxMsgs | How many messages may be in a Stream, oldest messages will be removed if the Stream exceeds this size | +| MaxMsgs | How many messages may be in a Stream. Adheres to Discard Policy, removing oldest or refusing new messages if the Stream exceeds this size | | MaxConsumers | How many Consumers can be defined for a given Stream, `-1` for unlimited | | Name | A name for the Stream that may not have spaces, tabs, period (`.`), greater than (`>`) or asterix (`*`) | | NoAck | Disables acknowledging messages that are received by the Stream | From 9962dae9d0794a36cca63cbcb2e3c5e74565819d Mon Sep 17 00:00:00 2001 From: scottf Date: Mon, 19 Apr 2021 09:50:09 -0400 Subject: [PATCH 5/6] fine tuning --- jetstream/concepts/streams.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/jetstream/concepts/streams.md b/jetstream/concepts/streams.md index fc871a5..58ab6fd 100644 --- a/jetstream/concepts/streams.md +++ b/jetstream/concepts/streams.md @@ -2,7 +2,7 @@ Streams define how messages are stored and retention duration. Streams consume normal NATS subjects, any message found on those subjects will be delivered to the defined storage system. You can do a normal publish to the subject for unacknowledged delivery, else if you send a Request to the subject the JetStream server will reply with an acknowledgement that it was stored. -As of January 2020, in the tech preview we have `file` and `memory` based storage systems, we do not yet support clustering. +As of January 2020, in the tech preview we have `File` and `Memory` based storage systems, we do not yet support clustering. In the diagram above we show the concept of storing all `ORDERS.*` in the Stream even though there are many types of order related messages. We'll show how you can selectively consume subsets of messages later. Relatively speaking the Stream is the most resource consuming component so being able to combine related data in this manner is important to consider. @@ -16,17 +16,17 @@ When defining Streams the items below make up the entire configuration of the se | Item | Description | | :--- | :--- | -| MaxAge | Maximum age of any message in the stream, expressed in nanoseconds. | -| MaxBytes | How big the Stream may be. Adheres to Discard Policy, removing oldest or refusing new messages if the Stream exceeds this number of messages. | -| MaxMsgSize | The largest message that will be accepted by the Stream | -| MaxMsgs | How many messages may be in a Stream. Adheres to Discard Policy, removing oldest or refusing new messages if the Stream exceeds this size | -| MaxConsumers | How many Consumers can be defined for a given Stream, `-1` for unlimited | | Name | A name for the Stream that may not have spaces, tabs, period (`.`), greater than (`>`) or asterix (`*`) | -| NoAck | Disables acknowledging messages that are received by the Stream | +| Storage | The type of storage backend, `File` and `Memory` | +| Subjects | A list of subjects to consume, supports wildcards | | Replicas | How many replicas to keep for each message in a clustered JetStream, maximum 5 | +| MaxAge | Maximum age of any message in the stream, expressed in nanoseconds. | +| MaxBytes | How many bytes the Stream may contain. Adheres to Discard Policy, removing oldest or refusing new messages if the Stream exceeds this number of messages. | +| MaxMsgs | How many messages may be in a Stream. Adheres to Discard Policy, removing oldest or refusing new messages if the Stream exceeds this size | +| MaxMsgSize | The largest message that will be accepted by the Stream | +| MaxConsumers | How many Consumers can be defined for a given Stream, `-1` for unlimited | +| NoAck | Disables acknowledging messages that are received by the Stream | | Retention | How message retention is considered, `LimitsPolicy` \(default\), `InterestPolicy` or `WorkQueuePolicy` | | Discard | When a Stream reached it's limits either, `DiscardNew` refuses new messages while `DiscardOld` \(default\) deletes old messages | -| Storage | The type of storage backend, `file` and `memory` as of January 2020 | -| Subjects | A list of subjects to consume, supports wildcards | -| Duplicates | The window within which to track duplicate messages | +| Duplicates | The window within which to track duplicate messages, expressed in nanoseconds. | From d491d2035f10f997b1d0ead7b8a22cfc9b5b9148 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Mon, 19 Apr 2021 09:36:05 -0500 Subject: [PATCH 6/6] Update consumers.md --- jetstream/concepts/consumers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jetstream/concepts/consumers.md b/jetstream/concepts/consumers.md index a6dcec3..cbfd431 100644 --- a/jetstream/concepts/consumers.md +++ b/jetstream/concepts/consumers.md @@ -2,7 +2,7 @@ Each Consumer, or related group of Consumers, of a Stream will need a Consumer defined. It's ok to define thousands of these pointing at the same Stream. -Consumers can either be `push` based where JetStream will deliver the messages as fast as possible to a subject of your choice or `pull` to have control by asking the server for messages. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging. +Consumers can either be `push` based where JetStream will deliver the messages as fast as possible to a subject of your choice or `pull` based to have control by asking the server for messages. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging. In the orders example above we have 3 Consumers. The first two select a subset of the messages from the Stream by specifying a specific subject like `ORDERS.processed`. The Stream consumes `ORDERS.*` and this allows you to receive just what you need. The final Consumer receives all messages in a `push` fashion.