taigrr/nats.docs
1
0
Fork 0
mirror of https://github.com/taigrr/nats.docs synced 2025-01-18 04:03:23 -08:00
Code Issues Projects Releases Wiki Activity

GitBook: [master] 2 pages modified

Browse Source
This commit is contained in:
Ginger Collison 2021-05-25 14:27:08 +00:00 committed by gitbook-bot
parent 0820ff2091
commit f4a24c2858
No known key found for this signature in database
GPG Key ID: 07D2180C7B12D0FF
2 changed files with 41 additions and 57 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

90
jetstream/concepts/consumers.md
View File

@ -2,7 +2,7 @@
Each Consumer, or related group of Consumers, of a Stream will need a Consumer defined. It's ok to define thousands of these pointing at the same Stream.
Consumers can either be `push` based where JetStream will deliver the messages as fast as possible (while adhering to the rate limit policy) to a subject of your choice or `pull` to have control by asking the server for messages. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging.
Consumers can either be `push` based where JetStream will deliver the messages as fast as possible \(while adhering to the rate limit policy\) to a subject of your choice or `pull` to have control by asking the server for messages. The rate of message delivery in both cases is subject to `ReplayPolicy`. A `ReplayInstant` Consumer will receive all messages as fast as possible while a `ReplayOriginal` Consumer will receive messages at the rate they were received, which is great for replaying production traffic in staging.
In the orders example above we have 3 Consumers. The first two select a subset of the messages from the Stream by specifying a specific subject like `ORDERS.processed`. The Stream consumes `ORDERS.*` and this allows you to receive just what you need. The final Consumer receives all messages in a `push` fashion.
@ -16,105 +16,91 @@ To assist with creating monitoring applications, one can set a `SampleFrequency`
When defining Consumers the items below make up the entire configuration of the Consumer:
### AckPolicy
How messages should be acknowledged. If an ack is required but is not received within the AckWait window, the message will be redelivered.
> IMPORTANT
>
> The server may consider an ack arriving out of the window. If a first process fails to ack within the window
> it's entirely possible, for instance in queue situation, that the message has been redelivered to another consumer.
> Since this will technically restart the window, the ack from the first consumer will be considered.
## AckPolicy
#### AckExplicit
How messages should be acknowledged. If an ack is required but is not received within the AckWait window, the message will be redelivered.
> IMPORTANT
>
> The server may consider an ack arriving out of the window. If a first process fails to ack within the window it's entirely possible, for instance in queue situation, that the message has been redelivered to another consumer. Since this will technically restart the window, the ack from the first consumer will be considered.
### AckExplicit
This is the default policy. It means that each individual message must be acknowledged. It is the only allowed option for pull consumers.
#### AckNone
### AckNone
You do not have to ack any messages, the server will assume ack on delivery.
#### AckAll
### AckAll
If you receive a series of messages, you only have to ack the last one you received.
All the previous messages received are automatically acknowledged.
If you receive a series of messages, you only have to ack the last one you received. All the previous messages received are automatically acknowledged.
### AckWait
## AckWait
Ack Wait is the time in nanoseconds that the server will wait for an ack for any individual message.
If an ack is not received in time, the message will be redelivered.
Ack Wait is the time in nanoseconds that the server will wait for an ack for any individual message. If an ack is not received in time, the message will be redelivered.
### DeliverPolicy / OptStartSeq / OptStartTime
## DeliverPolicy / OptStartSeq / OptStartTime
When a consumer is first created, it can specify where in the stream it wants to start receiving messages.
This is the `DeliverPolicy` and it's options are as follows:
When a consumer is first created, it can specify where in the stream it wants to start receiving messages. This is the `DeliverPolicy` and it's options are as follows:
#### DeliverAll
### DeliverAll
All is the default policy. The consumer will start receiving from the earliest available message.
#### DeliverLast
### DeliverLast
When first consuming messages, the consumer will start receiving messages with the last message added to the stream, so the very last message in the stream when the server realizes the consumer is ready.
#### DeliverNew
### DeliverNew
When first consuming messages, the consumer will only start receiving messages that were created after the consumer was created.
#### DeliverByStartSequence
### DeliverByStartSequence
When first consuming messages, start at this particular message in the set. The consumer is required to specify `OptStartSeq`, the sequence number to start on. It will receive the closest available message moving forward in the sequence should the message specified have been removed based on the stream limit policy.
#### DeliverByStartTime
### DeliverByStartTime
When first consuming messages, start with messages on or after this time. The consumer is required to specify `OptStartTime`, the time in the stream to start at. It will receive the closest available message on or after that time.
### DeliverSubject
## DeliverSubject
The subject to deliver observed messages. Not allowed for pull subscriptions.
Deliver subject is required for queue subscribing as it configures a subject that all the queue consumers should listen on.
The subject to deliver observed messages. Not allowed for pull subscriptions. Deliver subject is required for queue subscribing as it configures a subject that all the queue consumers should listen on.
### Durable (Name)
## Durable \(Name\)
The name of the Consumer, which the server will track, allowing resuming consumption where left off.
By default, a consumer is ephemeral. To make the consumer durable, set the name.
The name of the Consumer, which the server will track, allowing resuming consumption where left off. By default, a consumer is ephemeral. To make the consumer durable, set the name.
### FilterSubject
## FilterSubject
When consuming from a stream with a wildcard subject, this allows you to select a subset of the full wildcard subject to receive messages from.
### FlowControl
## FlowControl
Flow control is another way for the consumer to manage back pressure. Instead of relying on the rate limit, it relies on the pending limits of max messages and/or max bytes.
If the server sends the number of messages or bytes without receiving an ack, it will send a status message letting you know it has reached this limit.
Once flow control is tripped, the server will not start sending messages again until the client tells the server, even if all messages have been acknowledged.
The message status header will have a code of 100 and have an address in the message reply to field. The reply to address is where (the subject)
to publish an empty message to. The status message may have a description like "FlowControl Request"
Flow control is another way for the consumer to manage back pressure. Instead of relying on the rate limit, it relies on the pending limits of max messages and/or max bytes. If the server sends the number of messages or bytes without receiving an ack, it will send a status message letting you know it has reached this limit. Once flow control is tripped, the server will not start sending messages again until the client tells the server, even if all messages have been acknowledged. The message status header will have a code of 100 and have an address in the message reply to field. The reply to address is where \(the subject\) to publish an empty message to. The status message may have a description like "FlowControl Request"
### IdleHeartbeat
## IdleHeartbeat
If the idle heartbeat period is set, the server will send a status message with to the client when the period has elapsed but it has not received any new messages.
This lets the client know that it's still there, but just isn't receiving messages.
The message status header will have a code of 100. Unlike FlowControl, it will have no reply to address.
It may have a description like "Idle Heartbeat"
If the idle heartbeat period is set, the server will send a status message with to the client when the period has elapsed but it has not received any new messages. This lets the client know that it's still there, but just isn't receiving messages. The message status header will have a code of 100. Unlike FlowControl, it will have no reply to address. It may have a description like "Idle Heartbeat"
### MaxAckPending
## MaxAckPending
The maximum number of messages without an acknowledgement that can be outstanding, once this limit is reached message delivery will be suspended. It cannot be used with AckNone ack policy.
### MaxDeliver
## MaxDeliver
The maximum number of times a specific message will be delivered. Applies to any message that is re-sent due to ack policy.
### RateLimit
## RateLimit
Used to throttle the delivery of messages to the consumer, in bits per second.
### ReplayPolicy
## ReplayPolicy
The replay policy applies when the DeliverPolicy is `DeliverAll`, `DeliverByStartSequence` or `DeliverByStartTime` since those deliver policies begin reading the stream at a position other than the end.
If the policy is `ReplayOriginal`, the messages in the stream will be pushed to the client at the same rate that they were originally received, simulating the original timing of messages.
If the policy is `ReplayInstant` (the default), the messages will be pushed to the client as fast as possible while adhering to the Ack Policy, Max Ack Pending and the client's ability to consume those messages.
The replay policy applies when the DeliverPolicy is `DeliverAll`, `DeliverByStartSequence` or `DeliverByStartTime` since those deliver policies begin reading the stream at a position other than the end. If the policy is `ReplayOriginal`, the messages in the stream will be pushed to the client at the same rate that they were originally received, simulating the original timing of messages. If the policy is `ReplayInstant` \(the default\), the messages will be pushed to the client as fast as possible while adhering to the Ack Policy, Max Ack Pending and the client's ability to consume those messages.
### SampleFrequency
## SampleFrequency
Sets the percentage of acknowledgements that should be sampled for observability, 0-100 This value is a string and for example allows both `30` and `30%` as valid values.
Sets the percentage of acknowledgements that should be sampled for observability, 0-100
This value is a string and for example allows both `30` and `30%` as valid values.

8
jetstream/concepts/streams.md
View File

@ -2,8 +2,6 @@
Streams define how messages are stored and retention duration. Streams consume normal NATS subjects, any message found on those subjects will be delivered to the defined storage system. You can do a normal publish to the subject for unacknowledged delivery, else if you send a Request to the subject the JetStream server will reply with an acknowledgement that it was stored.
As of January 2020, in the tech preview we have `File` and `Memory` based storage systems, we do not yet support clustering.
In the diagram above we show the concept of storing all `ORDERS.*` in the Stream even though there are many types of order related messages. We'll show how you can selectively consume subsets of messages later. Relatively speaking the Stream is the most resource consuming component so being able to combine related data in this manner is important to consider.
Streams can consume many subjects. Here we have `ORDERS.*` but we could also consume `SHIPPING.state` into the same Stream should that make sense \(not shown here\).
@ -16,17 +14,17 @@ When defining Streams the items below make up the entire configuration of the se
| Item | Description |
| :--- | :--- |
| Name | A name for the Stream that may not have spaces, tabs, period \(`.`\), greater than \(`>`\) or asterisk \(`*`\). See \[naming\]\(../administration/naming.md\] |
| Name | A name for the Stream that may not have spaces, tabs, period \(`.`\), greater than \(`>`\) or asterisk \(`*`\). See [naming](../administration/naming.md). |
| Storage | The type of storage backend, `File` and `Memory` |
| Subjects | A list of subjects to consume, supports wildcards |
| Replicas | How many replicas to keep for each message in a clustered JetStream, maximum 5 |
| MaxAge | Maximum age of any message in the stream, expressed in nanoseconds. |
| MaxAge | Maximum age of any message in the Stream, expressed in nanoseconds. |
| MaxBytes | How many bytes the Stream may contain. Adheres to Discard Policy, removing oldest or refusing new messages if the Stream exceeds this number of messages. |
| MaxMsgs | How many messages may be in a Stream. Adheres to Discard Policy, removing oldest or refusing new messages if the Stream exceeds this size |
| MaxMsgSize | The largest message that will be accepted by the Stream |
| MaxConsumers | How many Consumers can be defined for a given Stream, `-1` for unlimited |
| NoAck | Disables acknowledging messages that are received by the Stream |
| Retention | How message retention is considered, `LimitsPolicy` \(default\), `InterestPolicy` or `WorkQueuePolicy` |
| Discard | When a Stream reached it's limits either, `DiscardNew` refuses new messages while `DiscardOld` \(default\) deletes old messages |
| Discard | When a Stream reaches it's limits either, `DiscardNew` refuses new messages while `DiscardOld` \(default\) deletes old messages |
| Duplicates | The window within which to track duplicate messages, expressed in nanoseconds. |