Commit Graph

410 Commits

Author SHA1 Message Date
Tomasz Pietrek
692f384f2d Fix consumer reply subject escaping
If the Consumer had a name containing `%`, it could result in
reply subject failing to format with `fmt.Sprintf`, as the `%`
was not properly escaped with `%%`.

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-04-12 09:22:08 +02:00
Derek Collison
c6b2a97ef4 Use entry pool
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-08 19:58:43 -07:00
Derek Collison
ebe4f8957f Spelling based on review feedback
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 21:08:59 -07:00
Derek Collison
dcbefd5cc4 We can receive these on push consumers, so error if we do
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 21:07:08 -07:00
Derek Collison
07b34f707f Make sure to never process next message requests inline
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 20:50:01 -07:00
Derek Collison
e6447c982a Protect against concurrent creation of streams and consumers.
Also make sure we have exited monotoring routines when doing resets for both streams and consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-02 14:29:52 -07:00
Derek Collison
b752b8b30d Snapshot on clean shutdown if needed or interest based retention
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-02 03:53:03 -07:00
Derek Collison
ad5bb366a0 Updates to preacks when multiple consumers are present but mutually exlusive (filtered).
Signed-off-by: Derek Collison <derek@nats.io>
2023-03-31 10:43:28 -07:00
Derek Collison
9a714e7d7d Update based on review feedback
Signed-off-by: Derek Collison <derek@nats.io>
2023-03-29 15:47:54 -07:00
Derek Collison
c4da37ecc7 Make sure consumer is valid and state was returned
Signed-off-by: Derek Collison <derek@nats.io>
2023-03-29 12:44:01 -07:00
Derek Collison
e516c47a4b Improvements to consumers attached to an interest retention stream.
1. Do not process an ack if we are closed.
2. When checking for needing an ack for a given consumer, hold lock entire time.
3. During recovery and restarts we check if we need to replay acks to the parent stream.

Signed-off-by: Derek Collison <derek@nats.io>
2023-03-29 12:43:49 -07:00
Derek Collison
5bb6f167b9 Make sure to cleanup messages on a follower consumer for an interest based stream when the consumer leader sends a state snapshot.
Signed-off-by: Derek Collison <derek@nats.io>
2023-03-15 20:11:16 -07:00
Derek Collison
5a1878b015 Fix for workqueue stream scaling up and not removing acked messages.
Make sure when scaling up streams that are workqueue or interest policy that consumers scale as well.

Signed-off-by: Derek Collison <derek@nats.io>
2023-03-13 17:13:49 -07:00
Tomasz Pietrek
df282a221c Fix Pull Consumer not sending request timeout
Server did check for timeouts in `processWaiting`,
but that needs to be also checked in `nextWaiting` in case of
tight timings, as `nextWaiting` can remove Pull Request based on
timeouts too.

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-03-03 14:49:04 +01:00
Derek Collison
724160ebac Fix flapping tests
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-28 14:30:23 -08:00
Derek Collison
24cb570646 Do not lock on stream name for consumer write state error
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-28 13:24:18 -08:00
Derek Collison
d85bec2007 Do not block in place on warning, and only warn if consumer not closed
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-28 11:45:31 -08:00
Derek Collison
b19fe508c4 Do not block routes/gws on internal stream and consumer info requests
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-28 11:17:29 -08:00
Derek Collison
2642a8c03d Optimize locking for when under heavy loads.
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-27 18:56:55 -08:00
Derek Collison
13167f46b9 Optimize some locking for when under heavy loads.
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-27 18:56:55 -08:00
Derek Collison
daacbf5580 Added optimized store NumPending() call.
Optimized and fixed a bug in filestore filteredPending().
Optimized memstore FilteredState().

Added comprehensive tests for NumPending() and FilteredState().

Signed-off-by: Derek Collison <derek@nats.io>
2023-02-25 17:26:26 -08:00
Neil Twigg
68961ffedd Refactor ipQueue to use generics, reduce allocations 2023-02-21 14:50:09 +00:00
Derek Collison
3c64d07691 Warn of consumer state update failures.
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-20 17:28:11 -08:00
Derek Collison
d2179e0939 Make sure to also cleanup pending if below our stream ack floor
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-20 11:56:37 -08:00
Derek Collison
b6149c51f0 Make sure to clean up redelivered state on purge.
Make sure to update ack floors on messages being expired out from underneath of us.

Signed-off-by: Derek Collison <derek@nats.io>
2023-02-20 11:16:12 -08:00
Derek Collison
6c9a9fb45e Fixed bug that would lose ack pending state during partial stream purge.
General code cleanup to be more correct.

Signed-off-by: Derek Collison <derek@nats.io>
2023-02-19 14:21:53 -08:00
Derek Collison
06fd81d096 Fixed a bug where a named consumer under interest policy was spinning up inactive threshold timers in all replicas not just the leader.
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-19 06:08:43 -08:00
Derek Collison
efa3bcc49d Parallel consumer creation could drop responses (create and info) and could also run monitorConsumer twice.
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-18 05:16:05 -08:00
Derek Collison
11b0f214d0 Do not re-calculate NumPending on consumer info calls.
We noticed this was being called alot in user environments.
When the consumer was filtered with a wilcard and the stream had a high cardinality of subjects and was falling behind this could take a substantial amount of time.

Signed-off-by: Derek Collison <derek@nats.io>
2023-02-16 16:30:14 -08:00
Derek Collison
b611e37e95 For updating a consumer filter subject make sure locking ordere correct and that our sublist is present.
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-06 21:34:48 +04:00
Derek Collison
1252653c16 Merge pull request #3829 from nats-io/jarema/fix-message-after-update
Fix Consumer not getting messages after filter update
2023-01-30 19:59:32 -08:00
Derek Collison
6058056e3b Minor fixes and optimizations for snapshots.
We were snappshotting more then needed, so double check that we should be doing this at the stream and consumer level.
At the raft level, we should have always been compacting the WAL to last+1, so made that consistent. Also fixed bug that would not skip last if more items behind the snapshot.

Signed-off-by: Derek Collison <derek@nats.io>
2023-01-30 17:54:18 -08:00
Tomasz Pietrek
836848ca64 Fix Consumer not getting messages after filter update
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-01-30 20:47:17 +01:00
Derek Collison
d0a7a8169a Merge branch 'main' into force-consumer-replicas 2023-01-26 08:35:49 -08:00
Derek Collison
4a3c27a251 Fix MQTT test for consumer replica override.
This was ill-advised by me, not understanding that the messages stream for MQTT was interested policy based.
Interest policy based streams require consumers to match the replica count.

Signed-off-by: Derek Collison <derek@nats.io>
2023-01-25 17:58:57 -08:00
Tomasz Pietrek
c89883b6aa Fix assigning signal subscription to consumer
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-01-24 12:26:22 +01:00
Derek Collison
f62d929018 Consumer must match replica of parent stream if interest based policy.
Signed-off-by: Derek Collison <derek@nats.io>
2023-01-23 20:16:42 -08:00
Derek Collison
786298469c Simplify locking for consumer info requests.
Signed-off-by: Derek Collison <derek@nats.io>
2022-12-14 18:52:15 -08:00
Derek Collison
c90fe9a2fa Improve performance and latency with large number of sparse consumers.
When a stream had a large number of consumers on a server that were sparse, the signaling mechanism would do a linear scan to signal matching consumers. As usage patterns have continued to have more consumers that are filteres and sparse, meaning a message is destined for a single or small number of consumers.

This change moves selection to a sublist that tracks only active consumer leaders for selection, which optimizes selection of consumers to signal when the number of consumers is large.

Signed-off-by: Derek Collison <derek@nats.io>
2022-12-13 15:25:55 -08:00
Derek Collison
549b77ca2d Ensure that ephemeral consumers that are deleted on startup properly are removed from the system.
Signed-off-by: Derek Collison <derek@nats.io>
2022-12-06 15:07:46 -08:00
Derek Collison
5e8c1993cb Server might crash if a pull consumer with inactivity threshold acks a msg then immediately deletes the consumer.
Signed-off-by: Derek Collison <derek@nats.io>
2022-11-21 15:22:23 -08:00
Ivan Kozlovic
49faba9e33 [FIXED] JetStream: WorkQueue not preventing overlapping consumers
A stream with a WorkQueue retention policy is supposed to allow
more than one consumer if they user filtered subjects, but those
subjects should not overlap.

There was an issue that if a new consumer had a filter subject
"wider" than an existing one, the error was not detected and
the new consumer was incorrectly accepted.

Resolves #3639

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-11-16 17:09:30 -07:00
Derek Collison
6cb7f68ef7 Updates based on feedback w/ Ivan
Signed-off-by: Derek Collison <derek@nats.io>
2022-11-16 11:53:46 -08:00
Derek Collison
aa57adbcd0 Use default here
Signed-off-by: Derek Collison <derek@nats.io>
2022-11-16 11:31:49 -08:00
Derek Collison
08c94096db Allow any type of activity to prolong auto cleanup of a consumer.
Signed-off-by: Derek Collison <derek@nats.io>
2022-11-15 17:25:18 -08:00
Derek Collison
b92ea86b80 When reading state, even on consumer init, should lock.
Signed-off-by: Derek Collison <derek@nats.io>
2022-11-15 07:40:44 -08:00
Derek Collison
36ef788112 When determing whether we need an ack, no need to copy since under consumer lock.
Signed-off-by: Derek Collison <derek@nats.io>
2022-11-14 11:47:31 -08:00
Derek Collison
47dd97e389 Fix logic bug that would prevent some messages from being deleted on an interest based stream.
Signed-off-by: Derek Collison <derek@nats.io>
2022-11-13 17:32:38 -08:00
Derek Collison
c6031382a1 Fix for #3499
When we deleted a consumer from an interest policy stream we would make sure to clean up any unacked messages.
However we only based start from the ack floor for the consumer and did not take into account the first sequence of the stream.

Signed-off-by: Derek Collison <derek@nats.io>
2022-11-05 13:56:45 -07:00
Derek Collison
72ff2edb5f Fix for #3603.
Signed-off-by: Derek Collison <derek@nats.io>
2022-11-03 12:46:41 -07:00