Commit Graph

230 Commits

Author SHA1 Message Date
Ivan Kozlovic
f0b098af92 [FIXED] JetStream: issue with max deliver and server/cluster restart
This is a regression introduced in v2.8.3. If a message reaches
the max redeliver count, it stops being delivered to the consumer.
However, after a server or cluster restart, those messages would
be redelivered again.

Resolves #3361

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-16 17:05:47 -06:00
Derek Collison
ef91d67708 Support auto-conversion
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-16 08:41:39 -07:00
Derek Collison
827b34a77a Add support for AES cipher encryption for filestore.
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-15 14:21:37 -07:00
Ivan Kozlovic
00345cac64 [FIXED] JetStream: subject overlap error should be returned
In standalone mode, when attempting to create a stream which has
subjects that overlap with an existing stream, the generic
stream create error "10049" was returned instead of the more
accurate "10065" error code corresponding to subject overlap,
as it was the case in clustered mode.

Resolves #3362

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-11 13:32:29 -06:00
Derek Collison
9a61537b1e Merge pull request #3351 from nats-io/fs-kv
[IMPROVED] DirectGet performance and memory usage for large streams.
2022-08-09 15:27:39 -07:00
Derek Collison
8c04adc009 Improvements to filestore for large KVs.
Use better indexing for lookups, we used to do simple linear scan backwards, now track first and last block.
Will expire the fss cache at will to reduce memory usage.

Signed-off-by: Derek Collison <derek@nats.io>
2022-08-09 15:51:13 -05:00
Ivan Kozlovic
a4bf4e87f6 Merge pull request #3326 from mfaizanse/health_endpoint_params
Added param options to /healthz endpoint
2022-08-09 08:49:22 -06:00
Muhammad Faizan
1634f33de7 Added param options to /healthz endpoint 2022-08-09 08:32:54 +02:00
Derek Collison
758b733d43 Attempt to improve long RTT catchup time during stream moves.
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-08 11:06:10 -06:00
Ivan Kozlovic
3c9a7cc6e5 Move to Go 1.19, remote io/util, fix data race and a flapper
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-05 09:55:37 -06:00
Derek Collison
748890adb1 Auto-set and upgrade AllowDirect when MaxMsgsPerSubject is set.
Also allow mirrors to inherit properly.

Signed-off-by: Derek Collison <derek@nats.io>
2022-08-03 12:36:52 -07:00
Derek Collison
c82c49451c Allow direct get by subject to be all subject based.
This avoids marshalling or unmarshalling but also allows subject based permissioning.

Signed-off-by: Derek Collison <derek@nats.io>
2022-08-02 18:19:33 -07:00
Byron Ruth
095cfef9eb Add check and test to prevent updating consumer MaxWaiting
Signed-off-by: Byron Ruth <b@devel.io>
2022-07-29 15:05:00 -04:00
Ivan Kozlovic
a3e62f000c Merge pull request #3265 from abegaj/fix-max-deliver-update
[FIXED] Maximum Deliveries update has no effect
2022-07-21 16:06:41 -06:00
Ardit Begaj
2871dcb235 increase maxDeliver 2022-07-21 23:24:56 +02:00
Ardit Begaj
c04ab08f2a improve tests based on feedback 2022-07-21 23:21:58 +02:00
Ivan Kozlovic
1da5ecfb96 [IMPROVED] JetStream: stream already exists error description
The `JSStreamNameExistErr` will now include in the description that
the stream exists with a different configuration, because that is
the error clients would get when trying to add a stream with a
different configuration (otherwise this is a no-op and client
don't get an error).

Since that error was used in case of restore, a new error is added
but uses the same description prefix "stream name already in use"
but adds ", cannot restore" to indicate that this is a restore
failure because the stream already exists.

Resolves #3273

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-07-21 10:20:07 -06:00
Ardit Begaj
94d22642d1 move test to appropriate file 2022-07-20 19:49:13 +02:00
Ardit Begaj
3f26dfee25 added cluster test 2022-07-19 20:01:36 +02:00
Todd Beets
302f85dcc2 Merge pull request #3271 from nats-io/repub-fix-srcignored
Fix and test for RePub src ignored at runtime.
2022-07-18 15:02:09 -07:00
Todd Beets
7a47631051 optimize for > or empty src (any subject) 2022-07-17 13:45:05 -07:00
Todd Beets
ae9f872ef5 Fix static code check violation on two unit tests and normalize unit test values to ADR-28. 2022-07-15 21:25:29 -07:00
Todd Beets
b145952e78 Fix and test for RePub src ignored at runtime. 2022-07-15 13:50:35 -07:00
Ivan Kozlovic
5f0ee2344a [FIXED] JetStream: stream mirror updates not rejected in standalone
Updates to stream mirror config are rejected in cluster mode, but
were not in standalone. This PR adds the check in standalone mode.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-07-15 10:51:57 -06:00
Ardit Begaj
4d81b8e4a2 Make MaxDeliver update changes take effect 2022-07-14 15:38:59 +02:00
Ivan Kozlovic
be8734e32b [CHANGED] JetStream: accept "Nats-Expected-Last-Sequence" with "0"
We use to ignore if the seq was 0, but now would treat it as
a requirement that the stream be empty if header is present but
set to 0.

This relates to client PR: https://github.com/nats-io/nats.go/pull/958

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-07-07 09:44:27 -06:00
Derek Collison
1ea608eabf Allows direct get to also do get next for subject with starting sequence
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-06 14:22:28 -07:00
Derek Collison
52baa95e2a When stalled for MaxAckPending, expire all pull requests that are one shot, meaning had at least 1 delivered message
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-04 19:14:49 -07:00
Derek Collison
1b580c67f3 Make pull consumers FIFO per message, not per request.
This effectively means that requests with batch > 1 will process a message and go to the end of the line.

Signed-off-by: Derek Collison <derek@nats.io>
2022-07-04 13:05:57 -07:00
Ivan Kozlovic
c519df7e0d [FIXED] Pull consumer may be incorrectly removed after InactiveThreshold
When creating a pull consumer with InactiveThreshold set, if the
application is doing pull requests with "no_wait" at regular interval
(lower than InactiveThreshold), the JS consumer should be considered
active and not deleted.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-06-29 10:15:09 -06:00
Derek Collison
9154fca7f1 Allow consumer filter subjects to be updated
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-24 12:38:01 -07:00
Derek Collison
b4238259b3 Change test to not have path separators
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-20 12:16:46 -07:00
Derek Collison
1ade8fc881 When stream or consumer names contained path separators it prevented backup and restore.
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-20 11:59:18 -07:00
Derek Collison
4c8110c3ff Add in support for inactivity thresholds for durable consumers.
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-14 06:51:00 -07:00
Derek Collison
fb51162e37 Make error on make bytes exceeded on a pull request a 409
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-07 06:59:08 -07:00
Derek Collison
0979bce543 Updates based on group feedback.
1. Do not use original subject since this could use Request() and we want to use muxing.
2  Place original subject and timestamp into headers.

Signed-off-by: Derek Collison <derek@nats.io>
2022-05-31 19:15:52 -07:00
Derek Collison
c8a730ce55 Stream get for KV was going through API layer, but with popularity needed a more peformant and lighter weight and direct approach.
Signed-off-by: Derek Collison <derek@nats.io>
2022-05-30 16:34:54 -07:00
Derek Collison
e08f6d863d Allow for republish to be headers only
Signed-off-by: Derek Collison <derek@nats.io>
2022-05-30 12:05:17 -07:00
Derek Collison
daa4b97eeb Don't do advisories or API stats for a direct get msg from a stream.
Signed-off-by: Derek Collison <derek@nats.io>
2022-05-30 09:32:07 -07:00
Ivan Kozlovic
a52f12613e Bump version to v2.8.4-beta.2 and fix flapper
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-05-25 13:25:45 -06:00
Derek Collison
72ed48d096 Merge pull request #3149 from nats-io/pull_perf_stable
[FIXED] Spurious pull consumer 408s under load
2022-05-25 09:44:27 -07:00
Derek Collison
d69394efad Fix spurious 408s under load and move processing of acks to their own Go routine.
Signed-off-by: Derek Collison <derek@nats.io>
2022-05-25 09:27:34 -07:00
Derek Collison
46f7f7bfc9 Consumer pending was not correct when stream had max msgs per subject set > 1 and a consumer that filtered out part of the stream was created.
Also make sure to update stream's config on a stream restore in case of changes.

Signed-off-by: Derek Collison <derek@nats.io>
2022-05-24 14:44:15 -07:00
Ivan Kozlovic
53e3c53d96 [FIXED] JetStream: consumer with deliver new may miss messages
This could happen when a consumer had not sent anything to the
attached NATS subscription and there was a consumer leader
step down or server restart.

Signed-off-by: Derek Collison <derek@nats.io>
2022-05-23 12:01:48 -06:00
Derek Collison
c166c9b199 Enable republishing of messages once stored in a stream.
This enables lightweight distribution of messages to very large number of NATS subscribers.
We add in metadata as headers that allows for gap detection which enables initial value (via JetStream, maybe KV) and realtime NATS core updates but all globally ordered.

Signed-off-by: Derek Collison <derek@nats.io>
2022-05-17 15:18:54 -07:00
Derek Collison
6bbc5f627c Support for MaxBytes for pull requests.
Signed-off-by: Derek Collison <derek@nats.io>
2022-05-16 08:43:33 -07:00
Derek Collison
bcecae42ac Fix for #3119
Signed-off-by: Derek Collison <derek@nats.io>
2022-05-12 15:45:29 -07:00
Derek Collison
b35988adf9 Remember the last timestamp by not removing last msgBlk when empty and during purge pull last timestamp forward until new messages arrive.
When a downstream stream uses retention modes that delete messages, fallback to timebased start time for the new source consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2022-05-09 09:04:19 -07:00
Ivan Kozlovic
cadf921ed1 [FIXED] JetStream: PullConsumer MaxWaiting==1 and Canceled requests
There was an issue with MaxWaiting==1 that was causing a request
with expiration to actually not expire. This was because processWaiting
would not pick it up because wq.rp was actually equal to wq.wp
(that is, the read pointer was equal to write pointer for a slice
of capacity of 1).

The other issue was that when reaching the maximum of waiting pull
requests, a new request would evict an old one with a "408 Request Canceled".

There is no reason for that, instead the server will first try to
find some existing expired requests (since some of the expiration
is lazily done), but if none is expired, and the queue is full,
the server will return a "409 Exceeded MaxWaiting" to the new
request, and not a "408 Request Canceled" to an old one...

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-05-03 15:17:20 -06:00
Derek Collison
e0f5fcffb8 Fix for subject transforms and JetStream delivery subjects.
Signed-off-by: Derek Collison <derek@nats.io>
2022-04-28 15:50:28 -07:00