From 4737b0f2958efd2a7aa1e55d841dba5c0f2096fc Mon Sep 17 00:00:00 2001 From: scottf Date: Thu, 20 May 2021 08:22:23 -0400 Subject: [PATCH 1/7] concepts consumers update 21-05-20 --- jetstream/concepts/consumers.md | 116 +++++++++++++++++++++++++++----- 1 file changed, 99 insertions(+), 17 deletions(-) diff --git a/jetstream/concepts/consumers.md b/jetstream/concepts/consumers.md index 1fe7d3d..19d6967 100644 --- a/jetstream/concepts/consumers.md +++ b/jetstream/concepts/consumers.md @@ -2,7 +2,7 @@ Each Consumer, or related group of Consumers, of a Stream will need a Consumer defined. It's ok to define thousands of these pointing at the same Stream. -Consumers can either be `push` based where JetStream will deliver the messages as fast as possible to a subject of your choice or `pull` based to have control by asking the server for messages. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging. +Consumers can either be `push` based where JetStream will deliver the messages as fast as possible to a subject of your choice or `pull` to have control by asking the server for messages. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging. In the orders example above we have 3 Consumers. The first two select a subset of the messages from the Stream by specifying a specific subject like `ORDERS.processed`. The Stream consumes `ORDERS.*` and this allows you to receive just what you need. The final Consumer receives all messages in a `push` fashion. @@ -10,25 +10,107 @@ Consumers track their progress, they know what messages were delivered, acknowle Acknowledgements default to `AckExplicit` - the only supported mode for pull-based Consumers - meaning every message requires a distinct acknowledgement. But for push-based Consumers, you can set `AckNone` that does not require any acknowledgement, or `AckAll` which quite interestingly allows you to acknowledge a specific message, like message `100`, which will also acknowledge messages `1` through `99`. The `AckAll` mode can be a great performance boost. -Some messages may cause your applications to crash and cause a never ending loop forever poisoning your system. The `MaxDeliver` setting allow you to set a upper bound to how many times a message may be delivered. +Some messages may cause your applications to crash and cause a never ending loop forever poisoning your system. The `MaxDeliver` setting allow you to set an upper bound to how many times a message may be delivered. To assist with creating monitoring applications, one can set a `SampleFrequency` which is a percentage of messages for which the system should sample and create events. These events will include delivery counts and ack waits. When defining Consumers the items below make up the entire configuration of the Consumer: -| Item | Description | -| :--- | :--- | -| AckPolicy | How messages should be acknowledged, `AckNone`, `AckAll` or `AckExplicit` | -| AckWait | How long to allow messages to remain un-acknowledged before attempting redelivery | -| DeliverPolicy | The initial starting mode of the consumer, `DeliverAll`, `DeliverLast`, `DeliverNew`, `DeliverByStartSequence` or `DeliverByStartTime` | -| DeliverySubject | The subject to deliver observed messages. Useful to set up an alternate subject for a regular NatsSubcriber can listen on that subject. Not allowed for pull subscriptions. | -| Durable | The name of the Consumer, which the server will track, allowing resuming consumption where left off. See [naming](../administration/naming.md). | -| FilterSubject | When consuming from a Stream with many subjects, or wildcards, select only a specific incoming subjects, supports wildcards | -| MaxDeliver | Maximum amount times a specific message will be delivered. Use this to avoid poison pills crashing all your services forever | -| OptStartSeq | When first consuming messages from the Stream start at this particular message in the set | -| ReplayPolicy | How messages are sent `ReplayInstant` or `ReplayOriginal` | -| SampleFrequency | What percentage of acknowledgements should be samples for observability, 0-100 | -| OptStartTime | When first consuming messages from the Stream start with messages on or after this time | -| RateLimit | The rate of message delivery in bits per second | -| MaxAckPending | The maximum number of messages without acknowledgement that can be outstanding, once this limit is reached message delivery will be suspended | +### AckPolicy +How messages should be acknowledged. The server will consider an ack ony if it comes within the Ack Wait window. +If the ack is not received in time, the message(s) will be redelivered. +#### AckExplicit + +This is the default policy. It means that each individual message must be acknowledged. It is the only allowed option for pull consumers. + +#### AckNone + +You do not have to ack any messages, the server will assume ack on delivery. + +#### AckAll + +If you receive a series of messages, you only have to ack the last one you received. +All the previous messages received are automatically acknowledged. + +### AckWait + +Ack Wait is the time in nanoseconds that the server will wait for an ack for any individual message. +If an ack is not received in time, the message will be redelivered. + + +### DeliverPolicy / OptStartSeq / OptStartTime + +When a consumer is first created, it can specify where in the stream it wants to start receiving messages. +This is the `DeliverPolicy` and it's options are as follows: + +#### DeliverAll + +All is the default policy. The consumer will start receiving from the earliest available message. + +#### DeliverLast + +The consumer will start receiving messages with the last message added to the stream, so the very last message in the stream when the server realizes the consumer is ready. + +#### DeliverNew + +The consumer will only start receiving messages that were created after the consumer was created. + +#### DeliverByStartSequence + +When first consuming messages from the Stream, start at this particular message in the set. The consumer is required to specify `OptStartSeq`, the sequence number to start on. It will receive the closest available sequence if that message was removed based on the stream limit policy. + +#### DeliverByStartTime + +When first consuming messages from the Stream start with messages on or after this time. The consumer is required to specify `OptStartTime`, the time in the stream to start at. It will receive the closest available message on or after that time. + +### DeliverSubject + +The subject to deliver observed messages. Useful to set up an alternate subject for a regular NatsSubcriber can listen on that subject. Not allowed for pull subscriptions. +Deliver subject essentially creates an alias core NATS subject for the stream. This means a core NATS subscriber could be set up to receive messages on the DeliverSubject, +starting wherever the DeliverPolicy was configured for the consumer. You could use a core NATS subscriber to access the stream by its original subject, but that would always start at +the next message that appears on the subject, instead of where the consumer configured start sequence or start time. This is only allowed for push subscriptions. + +### Durable (Name) + +The name of the Consumer, which the server will track, allowing resuming consumption where left off. +By default, a consumer is ephemeral. To make the consumer durable, set the name. + +### FilterSubject + +When consuming from a stream with a wildcard subject, this allows you to select a subset of the full wildcard subject to receive messages from. + +### FlowControl + +Flow control is another way for the consumer to manage back pressure. Instead of relying on the rate limit, it relies on the pending limits of max messages and/or max bytes. +If the server sends the number of messages or bytes without receiving an ack, it will send a status message letting you know it has reached this limit. +Once flow control is tripped, the server will not start sending messages again until the client tells the server, even if all messages have been acknowledged. +The message status header will have a code of 100 and the description "FlowControl Request" + +### IdleHeartbeat + +If the idle heartbeat period is set, the server will send a status message with to the client when the period has elapsed but it has not received any new messages. +This lets the client know that it's still there, but just isn't receiving messages. +The message status header will have a code of 100 and the description "Idle Heartbeat" + +### MaxAckPending + +The maximum number of messages without an acknowledgement that can be outstanding, once this limit is reached message delivery will be suspended. + +### MaxDeliver + +The maximum number of times a specific message will be delivered. Applies to any message that is re-sent due to ack policy. + +### RateLimit + +Used to throttle the delivery of messages to the consumer, in bits per second. + +### ReplayPolicy + +The replay policy applies when the DeliverPolicy is `All`, `ByStartSequence` or `ByStartTime` since those deliver policies begin reading the stream at a position other than the end. +If the policy is `ReplayOriginal`, the messages in the stream will be pushed to the client at the same rate that they were originally received, simulating the original timing of messages. +If the policy is `ReplayInstant` (the default), the messages will be pushed to the client as fast as possible while adhering to the Ack Policy, Max Ack Pending and the client's ability to consume those messages. + +### SampleFrequency + +Sets the percentage of acknowledgements that should be sampled for observability, 0-100 From 06e4b70a6bb9bdbf2bc09f3e2a72092f0f6a95c4 Mon Sep 17 00:00:00 2001 From: scottf Date: Thu, 20 May 2021 11:20:25 -0400 Subject: [PATCH 2/7] clarify deliver subject --- jetstream/concepts/consumers.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/jetstream/concepts/consumers.md b/jetstream/concepts/consumers.md index 19d6967..7281033 100644 --- a/jetstream/concepts/consumers.md +++ b/jetstream/concepts/consumers.md @@ -38,7 +38,6 @@ All the previous messages received are automatically acknowledged. Ack Wait is the time in nanoseconds that the server will wait for an ack for any individual message. If an ack is not received in time, the message will be redelivered. - ### DeliverPolicy / OptStartSeq / OptStartTime When a consumer is first created, it can specify where in the stream it wants to start receiving messages. @@ -66,10 +65,8 @@ When first consuming messages from the Stream start with messages on or after th ### DeliverSubject -The subject to deliver observed messages. Useful to set up an alternate subject for a regular NatsSubcriber can listen on that subject. Not allowed for pull subscriptions. -Deliver subject essentially creates an alias core NATS subject for the stream. This means a core NATS subscriber could be set up to receive messages on the DeliverSubject, -starting wherever the DeliverPolicy was configured for the consumer. You could use a core NATS subscriber to access the stream by its original subject, but that would always start at -the next message that appears on the subject, instead of where the consumer configured start sequence or start time. This is only allowed for push subscriptions. +The subject to deliver observed messages. Not allowed for pull subscriptions. +Deliver subject is required for queue subscribing as it configures a subject that all the queue consumers should listen on. ### Durable (Name) From 1b96fa5c05198ed4f88725f1180ddf629bbaf1cd Mon Sep 17 00:00:00 2001 From: scottf Date: Thu, 20 May 2021 11:39:27 -0400 Subject: [PATCH 3/7] clarify based on comments --- jetstream/concepts/consumers.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/jetstream/concepts/consumers.md b/jetstream/concepts/consumers.md index 7281033..28f796e 100644 --- a/jetstream/concepts/consumers.md +++ b/jetstream/concepts/consumers.md @@ -2,7 +2,7 @@ Each Consumer, or related group of Consumers, of a Stream will need a Consumer defined. It's ok to define thousands of these pointing at the same Stream. -Consumers can either be `push` based where JetStream will deliver the messages as fast as possible to a subject of your choice or `pull` to have control by asking the server for messages. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging. +Consumers can either be `push` based where JetStream will deliver the messages as fast as possible (while adhering to the rate limit policy) to a subject of your choice or `pull` to have control by asking the server for messages. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging. In the orders example above we have 3 Consumers. The first two select a subset of the messages from the Stream by specifying a specific subject like `ORDERS.processed`. The Stream consumes `ORDERS.*` and this allows you to receive just what you need. The final Consumer receives all messages in a `push` fashion. @@ -17,8 +17,12 @@ To assist with creating monitoring applications, one can set a `SampleFrequency` When defining Consumers the items below make up the entire configuration of the Consumer: ### AckPolicy -How messages should be acknowledged. The server will consider an ack ony if it comes within the Ack Wait window. -If the ack is not received in time, the message(s) will be redelivered. +How messages should be acknowledged. If an ack is required but is not received within the AckWait window, the message will be redelivered. +> IMPORTANT +> +> The server may consider an ack arriving out of the window. If a first process fails to ack within the window +> it's entirely possible, for instance in queue situation, that the message has been redelivered to another consumer. +> Since this will technically restart the window, the ack from the first consumer will be considered. #### AckExplicit @@ -49,19 +53,19 @@ All is the default policy. The consumer will start receiving from the earliest a #### DeliverLast -The consumer will start receiving messages with the last message added to the stream, so the very last message in the stream when the server realizes the consumer is ready. +When first consuming messages, the consumer will start receiving messages with the last message added to the stream, so the very last message in the stream when the server realizes the consumer is ready. #### DeliverNew -The consumer will only start receiving messages that were created after the consumer was created. +When first consuming messages, the consumer will only start receiving messages that were created after the consumer was created. #### DeliverByStartSequence -When first consuming messages from the Stream, start at this particular message in the set. The consumer is required to specify `OptStartSeq`, the sequence number to start on. It will receive the closest available sequence if that message was removed based on the stream limit policy. +When first consuming messages, start at this particular message in the set. The consumer is required to specify `OptStartSeq`, the sequence number to start on. It will receive the closest available sequence if that message was removed based on the stream limit policy. #### DeliverByStartTime -When first consuming messages from the Stream start with messages on or after this time. The consumer is required to specify `OptStartTime`, the time in the stream to start at. It will receive the closest available message on or after that time. +When first consuming messages, start with messages on or after this time. The consumer is required to specify `OptStartTime`, the time in the stream to start at. It will receive the closest available message on or after that time. ### DeliverSubject From 1bd39ddf814f836a62bf263d8e461b14c8297393 Mon Sep 17 00:00:00 2001 From: scottf Date: Thu, 20 May 2021 12:09:51 -0400 Subject: [PATCH 4/7] clarify sample freq --- jetstream/concepts/consumers.md | 1 + 1 file changed, 1 insertion(+) diff --git a/jetstream/concepts/consumers.md b/jetstream/concepts/consumers.md index 28f796e..7644c6d 100644 --- a/jetstream/concepts/consumers.md +++ b/jetstream/concepts/consumers.md @@ -115,3 +115,4 @@ If the policy is `ReplayInstant` (the default), the messages will be pushed to t ### SampleFrequency Sets the percentage of acknowledgements that should be sampled for observability, 0-100 +This value is a string and for example allows both `30` and `30%` as valid values. From fd2513dcb3d241632c5218f90dafa1cb1abc8041 Mon Sep 17 00:00:00 2001 From: scottf Date: Thu, 20 May 2021 16:26:34 -0400 Subject: [PATCH 5/7] clarify fc and hb --- jetstream/concepts/consumers.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/jetstream/concepts/consumers.md b/jetstream/concepts/consumers.md index 7644c6d..0d45b1a 100644 --- a/jetstream/concepts/consumers.md +++ b/jetstream/concepts/consumers.md @@ -86,13 +86,15 @@ When consuming from a stream with a wildcard subject, this allows you to select Flow control is another way for the consumer to manage back pressure. Instead of relying on the rate limit, it relies on the pending limits of max messages and/or max bytes. If the server sends the number of messages or bytes without receiving an ack, it will send a status message letting you know it has reached this limit. Once flow control is tripped, the server will not start sending messages again until the client tells the server, even if all messages have been acknowledged. -The message status header will have a code of 100 and the description "FlowControl Request" +The message status header will have a code of 100 and have an address in the message reply to field. The reply to address is where (the subject) +to publish an empty message to. The status message may have a description like "FlowControl Request" ### IdleHeartbeat If the idle heartbeat period is set, the server will send a status message with to the client when the period has elapsed but it has not received any new messages. This lets the client know that it's still there, but just isn't receiving messages. -The message status header will have a code of 100 and the description "Idle Heartbeat" +The message status header will have a code of 100. Unlike FlowControl, it will have no reply to address. +It may have a description like "Idle Heartbeat" ### MaxAckPending From 232d1908f33038967e221793a870a34046507bcd Mon Sep 17 00:00:00 2001 From: Scott Fauerbach Date: Fri, 21 May 2021 09:54:19 -0400 Subject: [PATCH 6/7] Update jetstream/concepts/consumers.md Co-authored-by: Waldemar Quevedo --- jetstream/concepts/consumers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jetstream/concepts/consumers.md b/jetstream/concepts/consumers.md index 0d45b1a..2742f54 100644 --- a/jetstream/concepts/consumers.md +++ b/jetstream/concepts/consumers.md @@ -98,7 +98,7 @@ It may have a description like "Idle Heartbeat" ### MaxAckPending -The maximum number of messages without an acknowledgement that can be outstanding, once this limit is reached message delivery will be suspended. +The maximum number of messages without an acknowledgement that can be outstanding, once this limit is reached message delivery will be suspended. It cannot be used with AckNone ack policy. ### MaxDeliver From 1f5756a6f107aa1a09cc8b1003055ae0b28c84d3 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Fri, 21 May 2021 10:22:06 -0500 Subject: [PATCH 7/7] syntax and grammar review --- jetstream/concepts/consumers.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/jetstream/concepts/consumers.md b/jetstream/concepts/consumers.md index 2742f54..eeb7747 100644 --- a/jetstream/concepts/consumers.md +++ b/jetstream/concepts/consumers.md @@ -61,7 +61,7 @@ When first consuming messages, the consumer will only start receiving messages t #### DeliverByStartSequence -When first consuming messages, start at this particular message in the set. The consumer is required to specify `OptStartSeq`, the sequence number to start on. It will receive the closest available sequence if that message was removed based on the stream limit policy. +When first consuming messages, start at this particular message in the set. The consumer is required to specify `OptStartSeq`, the sequence number to start on. It will receive the closest available message moving forward in the sequence should the message specified have been removed based on the stream limit policy. #### DeliverByStartTime @@ -110,7 +110,7 @@ Used to throttle the delivery of messages to the consumer, in bits per second. ### ReplayPolicy -The replay policy applies when the DeliverPolicy is `All`, `ByStartSequence` or `ByStartTime` since those deliver policies begin reading the stream at a position other than the end. +The replay policy applies when the DeliverPolicy is `DeliverAll`, `DeliverByStartSequence` or `DeliverByStartTime` since those deliver policies begin reading the stream at a position other than the end. If the policy is `ReplayOriginal`, the messages in the stream will be pushed to the client at the same rate that they were originally received, simulating the original timing of messages. If the policy is `ReplayInstant` (the default), the messages will be pushed to the client as fast as possible while adhering to the Ack Policy, Max Ack Pending and the client's ability to consume those messages.