Commit Graph

283 Commits

Author SHA1 Message Date
Derek Collison
47c6bfded4 Update server/jetstream_test.go
Fix spelling

Co-authored-by: Tomasz Pietrek <tomasz@nats.io>
2023-04-24 22:29:05 -07:00
Derek Collison
cae91b8cad In single server mode healthz could mistake a snapshot staging directory during a restore as an account.
If the restore took a long time, stalled, or was aborted, would cause healthz to fail.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-24 22:14:04 -07:00
Derek Collison
034975e767 Fix for a regression in behavior, needed to make sure when we went back to 1 entry for a subject we cleared firstNeedsUpdate.
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-15 10:00:44 -07:00
Tomasz Pietrek
a66c67baa5 Fix stream sourcing & mirroring overlap errors
When adding or updating sources/mirrors, server was checking if the stream with
a given name exists to check for subject overlaps, among other things.
However, if sourced/mirrored stream was `External`, checks should
not be executed, as not only stream would never be found,
but also, if `External` stream had the same name as the sourcing stream,
the check would be wrongly performed against itself.

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-04-14 21:00:11 +02:00
Tomasz Pietrek
692f384f2d Fix consumer reply subject escaping
If the Consumer had a name containing `%`, it could result in
reply subject failing to format with `fmt.Sprintf`, as the `%`
was not properly escaped with `%%`.

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-04-12 09:22:08 +02:00
Ivan Kozlovic
a4df4f8727 Fixed some tests
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2023-03-30 15:02:59 -06:00
Tomasz Pietrek
df282a221c Fix Pull Consumer not sending request timeout
Server did check for timeouts in `processWaiting`,
but that needs to be also checked in `nextWaiting` in case of
tight timings, as `nextWaiting` can remove Pull Request based on
timeouts too.

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-03-03 14:49:04 +01:00
Derek Collison
6bda358fa3 Fix tests that made assumptions about single server processing.
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-28 13:24:18 -08:00
Derek Collison
b19fe508c4 Do not block routes/gws on internal stream and consumer info requests
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-28 11:17:29 -08:00
Derek Collison
3c64d07691 Warn of consumer state update failures.
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-20 17:28:11 -08:00
Derek Collison
b6149c51f0 Make sure to clean up redelivered state on purge.
Make sure to update ack floors on messages being expired out from underneath of us.

Signed-off-by: Derek Collison <derek@nats.io>
2023-02-20 11:16:12 -08:00
Derek Collison
6c9a9fb45e Fixed bug that would lose ack pending state during partial stream purge.
General code cleanup to be more correct.

Signed-off-by: Derek Collison <derek@nats.io>
2023-02-19 14:21:53 -08:00
Derek Collison
11b0f214d0 Do not re-calculate NumPending on consumer info calls.
We noticed this was being called alot in user environments.
When the consumer was filtered with a wilcard and the stream had a high cardinality of subjects and was falling behind this could take a substantial amount of time.

Signed-off-by: Derek Collison <derek@nats.io>
2023-02-16 16:30:14 -08:00
Derek Collison
0cb01f9e7a Make sure we update storage accounting on extended version purge for filestore.
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-16 13:18:40 +04:00
Derek Collison
b611e37e95 For updating a consumer filter subject make sure locking ordere correct and that our sublist is present.
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-06 21:34:48 +04:00
Tomasz Pietrek
46af979871 Fix current consumers not getting messages after purge
Until now, purge updated all consumers sequences
even if purge subject was only a subset of given consumer filter.
Because of that, even messages from not purged subjects were not fetched
or properly accounted for existing consumers.

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-02-02 16:27:32 +01:00
Tomasz Pietrek
836848ca64 Fix Consumer not getting messages after filter update
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-01-30 20:47:17 +01:00
Derek Collison
d9cb1e6286 Fix for #3734
When a msg blk was not writen correctly, but the idx file was, max bytes for a stream would no longer be honored since the deletion of any messages in that empty block were not being handled properly.

Signed-off-by: Derek Collison <derek@nats.io>
2022-12-31 18:50:46 -08:00
Neil
1d53b47b33 Merge pull request #3754 from nats-io/neil/golangcilint
Switch to `golangci-lint`
2022-12-30 20:12:03 +00:00
Neil Twigg
14d0ba1c65 Fix some lint errors after move to golangci-lint 2022-12-30 20:00:08 +00:00
Derek Collison
713f632fa7 If a stream's meta was not properly written but the file existed, we could re-add the stream but a subsequent restart would lose the stream again.
Also added in healthz for single server systems to make sure all stream directories resulted in recovered streams.

Signed-off-by: Derek Collison <derek@nats.io>
2022-12-29 20:08:56 -08:00
Waldemar Quevedo
861f98128a Remove individual import subs from leafnodes
When a leafnode connection is bound to an account where there was already
a wildcard response import subscription to handle the requests (e.g. `_R_.foo.>`),
this would have created message duplicates due to an extra subscription
being created that also matched the wildcard (e.g. `_R_.foo.bar`).

To avoid this condition, we now skip creating the latter extra subscription
for leafnode connections.

Signed-off-by: Waldemar Quevedo <wally@nats.io>
2022-12-29 15:30:17 -08:00
Todd Beets
c463b398db Validate no overlapping stream subscriptions on update config (non-clustered jetstream) 2022-12-16 12:58:59 -08:00
Marco Primi
f8a030bc4a Use testing.TempDir() where possible
Refactor tests to use go built-in temporary directory utility for tests.

Also avoid binding to default port (which may be in use)
2022-12-12 13:18:44 -08:00
Byron Ruth
566d1adfa7 Fix /healthz?js-enabled=true behavior
When js-enabled is set to true, the condition was only checked if
the `getJetStream()` call returned `nil`. However, if it non-nil,
all remaining checks were executed, including assessing the health
of the assets (streams and consumers).

This change addresses two issues:

- Switch to use `js.isEnabled()` which will check whether the value
  is nil OR `js.disabled = true` which can occur if the subsystem
  is temporarily disabled (insufficient resources).
- Correctly exit the check after the assertion and before meta and
  asset checks are performed.

In addition, the option has been renamed to `js-enabled-only` to align
with the `js-server-only` naming. The previous `js-enabled` name still
works, but is mapped to this new option. A warning is emitted noting
the previous option is deprecated.

Fix #3703

Signed-off-by: Byron Ruth <b@devel.io>
2022-12-10 07:34:32 -05:00
Derek Collison
5e8c1993cb Server might crash if a pull consumer with inactivity threshold acks a msg then immediately deletes the consumer.
Signed-off-by: Derek Collison <derek@nats.io>
2022-11-21 15:22:23 -08:00
Derek Collison
fe57c687f6 Make sure header keys do not have additional prefixes.
Signed-off-by: Derek Collison <derek@nats.io>
2022-11-17 06:52:26 -08:00
Ivan Kozlovic
49faba9e33 [FIXED] JetStream: WorkQueue not preventing overlapping consumers
A stream with a WorkQueue retention policy is supposed to allow
more than one consumer if they user filtered subjects, but those
subjects should not overlap.

There was an issue that if a new consumer had a filter subject
"wider" than an existing one, the error was not detected and
the new consumer was incorrectly accepted.

Resolves #3639

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-11-16 17:09:30 -07:00
Derek Collison
f696c6559e Adjust test for jitter on ephemerals.
Signed-off-by: Derek Collison <derek@nats.io>
2022-11-15 17:51:01 -08:00
Derek Collison
b6ef2c8910 Auto cleanup dangling messages from interest policy streams on server start.
Signed-off-by: Derek Collison <derek@nats.io>
2022-11-14 15:02:22 -08:00
Ivan Kozlovic
abcfe2e7ac Add the pending msgs/bytes on 409 Shutdown
This is related to PR #3572 and PR #3576

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-10-27 13:59:21 -06:00
Tomasz Pietrek
f0219e1d95 Merge pull request #3572 from nats-io/jarema/add-pending-info-to-request-timeout
Added pending messages/bytes info to request statuses and errors
2022-10-26 21:20:20 +02:00
Tomasz Pietrek
ef764598ee Add pending messages/bytes info to request errors and statuses
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2022-10-26 20:02:11 +02:00
Ivan Kozlovic
39f31b0dbe [FIXED] JetStream: InactivityThreshold updates not always working
This is based of @neilalexander PR #3558.

It ensures that the timer is reset/canceled on configuration
update (by the leader only).

Fixed also the issue with a super-cluster where the delete timer
would always be reset at every gateway interval check.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-10-25 09:54:01 -06:00
Ivan Kozlovic
7ca85e0e80 [FIXED] JetStream: Update of an R1 consumer would not get a response
The update was accepted but the server would not respond to the
client/CLI.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-10-25 09:04:35 -06:00
Ivan Kozlovic
1039df088c [UPDATED] Dependencies
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-09-29 15:00:44 -06:00
Ivan Kozlovic
46aec649e4 [FIXED] JetStream: redeliveries for LastPerSubject delivery policy
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-09-29 14:00:18 -06:00
Ivan Kozlovic
ff0bda415b [FIXE] JetStream: Pull requests closed due to max_bytes were silent
If the client pull requests has a max_bytes value and the server
cannot deliver a single message (because size is too big), it
is sending a 409 to signal that to the client library. However,
if it sends at least a message then it would close the request
without notifying the client with a 409, which would cause the
client library to have to wait for its expiration/timeout.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-09-15 16:55:41 -06:00
Tomasz Pietrek
dbf7636e15 Add error if Consumer Durable and Name are not equal
This error will happen only if both Name and Durable are specified.

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2022-09-14 20:31:18 +02:00
Derek Collison
d979937bbd Merge pull request #3456 from nats-io/max-bytes-pull
[IMPROVED] Pull request logic
2022-09-08 12:08:10 -07:00
Derek Collison
dedf21d45d Fix for issue #3455
When hitting max ack pending from getNextMsg would remove one shots incorrectly.

Signed-off-by: Derek Collison <derek@nats.io>
2022-09-08 11:56:57 -07:00
Derek Collison
b32814d5fd Better accounting for max-bytes for pull consumers
Signed-off-by: Derek Collison <derek@nats.io>
2022-09-08 11:56:57 -07:00
Ivan Kozlovic
b69ffe244e Fixed some tests
Code change:
- Do not start the processMirrorMsgs and processSourceMsgs go routine
if the server has been detected to be shutdown. This would otherwise
leave some go routine running at the end of some tests.
- Pass the fch and qch to the consumerFileStore's flushLoop otherwise
in some tests this routine could be left running.

Tests changes:
- Added missing defer NATS connection close
- Added missing defer server shutdown

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-09-08 11:28:23 -06:00
Ivan Kozlovic
8c1c6951dc JetStream: R1 durables were incorrectly migrated on shutdown
This could happen for stream with R>1 but with a durable that
has an override of R=1.

Fixed a test to make sure assets have an elected leader.

Also fixed a gateway test that would cause a data race.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-09-07 19:49:16 -06:00
Derek Collison
84b95be56e NoAck allowed now on pull consumers
Signed-off-by: Derek Collison <derek@nats.io>
2022-09-06 15:19:17 -07:00
Derek Collison
9c3bd17059 Updates to tests
Signed-off-by: Derek Collison <derek@nats.io>
2022-09-06 13:33:39 -07:00
Derek Collison
b850a95d4c Remove auto-promotion of direct get. Force stream config to set AllowDirect to true.
Signed-off-by: Derek Collison <derek@nats.io>
2022-09-06 13:33:39 -07:00
Derek Collison
452e3c3955 Make sure update covered
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-31 16:05:32 -07:00
Derek Collison
b649ae8bc3 We detected overlap between different streams but not same stream with multiple subjects.
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-31 15:51:39 -07:00
Derek Collison
aa94a0bc0f New consumer create that allows elevation of stream and consumer names, and optional filter subject to the request subject.
Similar to changes in direct get allows proper security if needed for filter subject selection.

Signed-off-by: Derek Collison <derek@nats.io>
2022-08-30 09:29:38 -07:00