Commit Graph

126 Commits

Author SHA1 Message Date
Derek Collison
4efce40bbd Small improvements to send performance to a full stream.
Cleaned up some locking and if fifo make index updates lazy like writeMsgRecord.

Signed-off-by: Derek Collison <derek@nats.io>
2022-02-17 05:39:27 -08:00
Derek Collison
5a93b0e9d8 Allow pull requests to specify a heartbeat when idle to detect when a request is invalidated.
Signed-off-by: Derek Collison <derek@nats.io>
2022-02-11 09:51:51 -08:00
Ivan Kozlovic
55ffde7251 Fixed consumer dlv count and num pending wrong due to redeliveries
Introduced by #2848, so should not have impacted existing releases.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-02-10 17:33:53 -07:00
Derek Collison
0cc7302be9 A stream name is tied to its identity and can not be changed on a restore.
Signed-off-by: Derek Collison <derek@nats.io>
2022-02-09 12:38:45 -08:00
Derek Collison
c13a84cf44 Fixed a bug that would calculate the first sequence of a filteredPending incorrectly.
Also added in more optimized version to select the first matching message in a message block for LoadNextMsg.

Signed-off-by: Derek Collison <derek@nats.io>
2022-02-08 13:29:38 -08:00
Derek Collison
d50febeeff Improved sparse consumers replay time.
When a stream has multiple subjects and a consumer filters the stream to a small and spread out list of messages the logic would do a linear scan looking for the next message for the filtered consumer.
This CL allows the store layer to utilize the per subject info to improve the times.

Signed-off-by: Derek Collison <derek@nats.io>
2022-02-07 17:26:32 -08:00
Derek Collison
55b7f11c9a Fixed flow control stall under specific conditions of message size.
Signed-off-by: Derek Collison <derek@nats.io>
2022-02-05 20:15:48 -08:00
Derek Collison
5da0453964 Add in NumSubjects to StreamInfo
Signed-off-by: Derek Collison <derek@nats.io>
2022-02-02 08:51:13 -08:00
Derek Collison
6a3cf0f71e Added in ability to get number of subjects from StreamInfo, and optionally details per subject on how many messages each subject has.
This can also be filtered, meaning you can filter out the subjects when asking for details.

Signed-off-by: Derek Collison <derek@nats.io>
2022-02-02 08:51:13 -08:00
Derek Collison
12f5ea3655 When a consumer had not filtered subject and was attached to a interest policy retention stream we could incorrectly drop messages.
Signed-off-by: Derek Collison <derek@nats.io>
2022-02-01 14:21:05 -08:00
Derek Collison
fa814f7cee Fixed behavior for when MaxMsgsPerSubject is set and DiscardNew is also set.
Signed-off-by: Derek Collison <derek@nats.io>
2022-01-31 08:36:37 -08:00
Derek Collison
b38ced51b2 A true no wait pull request was not considering redeliveries.
Signed-off-by: Derek Collison <derek@nats.io>
2022-01-30 11:52:35 -08:00
Derek Collison
275d42628b Fix for #2828. The original design of the consumer and the subsequent store did not allow updates.
Now that we do, we need to store the new config into our storage layer.

Signed-off-by: Derek Collison <derek@nats.io>
2022-01-30 09:45:05 -08:00
Derek Collison
7f572983ac The 2.7 update broke the one-shot pull consumer fetch behavior due to change and a fix to a bug that allowed it to work before.
This change tries to lock down all expected behaviors, and now does out of order timeouts for requests.

Signed-off-by: Derek Collison <derek@nats.io>
2022-01-20 18:06:29 -08:00
Ivan Kozlovic
3ce22adb76 Fixed some tests
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-01-13 13:14:05 -07:00
Derek Collison
43eff407b8 Add in explicit subscription for import responses when bound to a leafnode.
When we want to track service import response interest across a leafnode we need to send sub and unsub for all response _R_ subjects versus using a wildcard.

Signed-off-by: Derek Collison <derek@nats.io>
2022-01-12 18:24:03 -08:00
Derek Collison
32c3c9ecfb Track interest properly across accounts for pull consumers
Signed-off-by: Derek Collison <derek@nats.io>
2022-01-12 12:16:53 -08:00
Derek Collison
279f31ecb5 Add in ability to have ephemeral pull based consumers
Signed-off-by: Derek Collison <derek@nats.io>
2022-01-10 20:42:39 -08:00
Derek Collison
e12c8cda92 Add in ability to limit aspects of a pull request, specifically batch size and expiration.
Signed-off-by: Derek Collison <derek@nats.io>
2022-01-10 17:29:04 -08:00
Derek Collison
5592d923c4 Updated pull consumers.
Cleaned up code, made more consistent, utilize loopAndGather.
Allow pull consumers to have AckAll as well as AckExplicit.

Signed-off-by: Derek Collison <derek@nats.io>
2022-01-10 16:59:01 -08:00
Derek Collison
c4198d603c Added test to show cross account interest for push consumers works
Signed-off-by: Derek Collison <derek@nats.io>
2021-12-21 19:30:35 -08:00
Matthias Hanel
3e8b66286d Js leaf deny (#2693)
Along a leaf node connection, unless the system account is shared AND the JetStream domain name is identical, the default JetStream traffic (without a domain set) will be denied.

As a consequence, all clients that wants to access a domain that is not the one in the server they are connected to, a domain name must be specified.
Affected from this change are setups where: a leaf node had no local JetStream OR the server the leaf node connected to had no local JetStream. 
One of the two accounts that are connected via a leaf node remote, must have no JetStream enabled.
The side that does not have JetStream enabled, will loose JetStream access and it's clients must set `nats.Domain` manually.

For workarounds on how to restore the old behavior, look at:
https://github.com/nats-io/nats-server/pull/2693#issuecomment-996212582

New config values added:
`default_js_domain` is a mapping from account to domain, settable when JetStream is not enabled in an account.
`extension_hint` are hints for non clustered server to start in clustered mode (and be usable to extend)
`js_domain` is a way to set the JetStream domain to use for mqtt.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-12-16 16:53:20 -05:00
Derek Collison
ca12a11be3 There were situations where invalid subjects could be assigned to streams.
This will patch them on the fly during recovery. Specifically subjects with leading or trailing spaces and mirror streams with any subjects at all.

Signed-off-by: Derek Collison <derek@nats.io>
2021-12-01 14:00:23 -07:00
Ivan Kozlovic
1cf8b40304 Merge pull request #2719 from nats-io/js_mem_corruption
[FIXED] Corrupted headers receiving from consumer with meta-only
2021-12-01 13:42:47 -07:00
Ivan Kozlovic
9f30bf00e0 [FIXED] Corrupted headers receiving from consumer with meta-only
When a consumer is configured with "meta-only" option, and the
stream was backed by a memory store, a memory corruption could
happen causing the application to receive corrupted headers.

Also replaced most of usage of `append(a[:0:0], a...)` to make
copies. This was based on this wiki:
https://github.com/go101/go101/wiki/How-to-efficiently-clone-a-slice%3F

But since Go 1.15, it is actually faster to call make+copy instead.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-12-01 10:50:15 -07:00
R.I.Pienaar
c025d25899 prevent stream update to add subjects to mirrors
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-12-01 18:12:49 +01:00
R.I.Pienaar
4f1bfa969f ensure streams have only valid interest subjects
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-12-01 17:03:28 +01:00
Derek Collison
72ad68fada [FIXED] Bug in memstore that when setting max msgs per subject to 1 would not work properly.
Signed-off-by: Derek Collison <derek@nats.io>
2021-11-19 09:13:43 -08:00
Derek Collison
14469ccfc8 Fix for #2662.
Upon server restart a server would set the check expiration to the configured amount vs delta of next to expire.

Signed-off-by: Derek Collison <derek@nats.io>
2021-11-01 18:04:37 -07:00
Derek Collison
c9f615933a Honor JetStream server settings of 0.
Signed-off-by: Derek Collison <derek@nats.io>
2021-10-12 14:16:47 -07:00
Derek Collison
cbbab295ec Merge pull request #2616 from nats-io/fix_2611
[FIXED] #2611
2021-10-12 08:46:42 -07:00
Derek Collison
8951bc54f4 Fix for stream purge where purge would remove all pending messages for a filtered consumer.
Signed-off-by: Derek Collison <derek@nats.io>
2021-10-12 07:26:13 -07:00
Derek Collison
c789ad9211 Simplified a bit
Signed-off-by: Derek Collison <derek@nats.io>
2021-10-12 06:12:56 -07:00
Ivan Kozlovic
3b9c358a81 Test that reproduces the issue reported in #2611
Issue seem to be from the fact that the filter subject is == to
the stream's subjects list and so setInitialPendingAndStart()
considers that the consumer is not filtered.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-10-11 13:32:04 -06:00
Derek Collison
58e5c7c681 Allow consumers to request only headers to be delivered
Signed-off-by: Derek Collison <derek@nats.io>
2021-10-05 18:54:52 -07:00
Ivan Kozlovic
d34b8fdd7d [FIXED] JetStream: data race on shutdown
Replaced use of eventsEnabled() with EventsEnabled() that will
check under server lock. Also found another reference when
creating templates.

Resolves #2588

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-09-30 11:17:12 -06:00
Ivan Kozlovic
a3be36fcd4 Moving 2 new tests to jetstream test file
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-09-22 17:02:29 -06:00
Derek Collison
9534372113 Fix for #2551
When a mirror would be processed before the origin stream we would not recover the consumers due to failure on looking up source's subjects.
This change processes all streams first then does all consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2021-09-21 08:53:12 -07:00
R.I.Pienaar
3be0b23a3e Merge pull request #2546 from ripienaar/negative_dupe_window_protection
protect against negative dupe window via negative max age
2021-09-20 18:09:19 +02:00
R.I.Pienaar
34b5a11e20 protect against negative dupe window via negative max age
Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-09-20 17:34:48 +02:00
Derek Collison
cecb6246f2 Test tweaks
Signed-off-by: Derek Collison <derek@nats.io>
2021-09-19 13:28:37 -07:00
Derek Collison
40a4d40337 Make large batch requests expire more efficiently.
Signed-off-by: Derek Collison <derek@nats.io>
2021-09-16 15:00:18 -07:00
Ivan Kozlovic
108d4060ff [CHANGED] JetStream: flow control requires heartbeats
The server will now reject the creation of a push consumer with
flow control if no heartbeat is set.

Resolves #2520

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-09-15 09:50:15 -06:00
Derek Collison
e18a278196 Added ability to check if no message exists as a test to store the message.
Signed-off-by: Derek Collison <derek@nats.io>
2021-09-09 11:41:23 -07:00
Derek Collison
dadc3b9fae Fixed a bug when an interest retention stream with noack consumers is in clustered mode.
We were not properly propagating the ack state and proper cleanup of the stream messages.

Signed-off-by: Derek Collison <derek@nats.io>
2021-09-08 15:02:09 -07:00
Ivan Kozlovic
ba36aa452b Fix some timing and bump server version
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-09-01 18:03:14 -06:00
Ivan Kozlovic
44c57a5702 [FIXED] Pull requests: don't send 408 when request expires
When expiring requests, the server would send 408 if interest was
still present, which can happen for pull subscribe implementations
that maintain interest for the duration of the pull subscription.

Let's keep the 408 for when a request is "force expired", that
is, a request was removed from the queue because it queue was
full but interest is still found.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-09-01 16:57:13 -06:00
Derek Collison
752fd295a5 Consumer num pending fixes for multiple matches and merging.
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-24 07:52:29 -07:00
Derek Collison
70d28bd221 Use append in case pubAck is larger than stack []byte
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-20 15:53:37 -07:00
Derek Collison
6871d1240b When we expired all messages on a restart we did not properly setup lmb.
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-17 13:45:50 -07:00