Commit Graph

233 Commits

Author SHA1 Message Date
Derek Collison
ef71087d56 Fixed a bug that would not track per subject info for streams that were mirrors or sources.
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-23 15:46:57 -07:00
Derek Collison
c48bd6ea54 Inline turning on direct subs for non-clustered streams
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-23 11:20:18 -07:00
Ivan Kozlovic
11b00ed281 Add test that demonstrate cons pending < stream first seq is OK
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-22 18:20:25 -06:00
Ivan Kozlovic
f0b098af92 [FIXED] JetStream: issue with max deliver and server/cluster restart
This is a regression introduced in v2.8.3. If a message reaches
the max redeliver count, it stops being delivered to the consumer.
However, after a server or cluster restart, those messages would
be redelivered again.

Resolves #3361

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-16 17:05:47 -06:00
Derek Collison
ef91d67708 Support auto-conversion
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-16 08:41:39 -07:00
Derek Collison
827b34a77a Add support for AES cipher encryption for filestore.
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-15 14:21:37 -07:00
Ivan Kozlovic
00345cac64 [FIXED] JetStream: subject overlap error should be returned
In standalone mode, when attempting to create a stream which has
subjects that overlap with an existing stream, the generic
stream create error "10049" was returned instead of the more
accurate "10065" error code corresponding to subject overlap,
as it was the case in clustered mode.

Resolves #3362

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-11 13:32:29 -06:00
Derek Collison
9a61537b1e Merge pull request #3351 from nats-io/fs-kv
[IMPROVED] DirectGet performance and memory usage for large streams.
2022-08-09 15:27:39 -07:00
Derek Collison
8c04adc009 Improvements to filestore for large KVs.
Use better indexing for lookups, we used to do simple linear scan backwards, now track first and last block.
Will expire the fss cache at will to reduce memory usage.

Signed-off-by: Derek Collison <derek@nats.io>
2022-08-09 15:51:13 -05:00
Ivan Kozlovic
a4bf4e87f6 Merge pull request #3326 from mfaizanse/health_endpoint_params
Added param options to /healthz endpoint
2022-08-09 08:49:22 -06:00
Muhammad Faizan
1634f33de7 Added param options to /healthz endpoint 2022-08-09 08:32:54 +02:00
Derek Collison
758b733d43 Attempt to improve long RTT catchup time during stream moves.
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-08 11:06:10 -06:00
Ivan Kozlovic
3c9a7cc6e5 Move to Go 1.19, remote io/util, fix data race and a flapper
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-05 09:55:37 -06:00
Derek Collison
748890adb1 Auto-set and upgrade AllowDirect when MaxMsgsPerSubject is set.
Also allow mirrors to inherit properly.

Signed-off-by: Derek Collison <derek@nats.io>
2022-08-03 12:36:52 -07:00
Derek Collison
c82c49451c Allow direct get by subject to be all subject based.
This avoids marshalling or unmarshalling but also allows subject based permissioning.

Signed-off-by: Derek Collison <derek@nats.io>
2022-08-02 18:19:33 -07:00
Byron Ruth
095cfef9eb Add check and test to prevent updating consumer MaxWaiting
Signed-off-by: Byron Ruth <b@devel.io>
2022-07-29 15:05:00 -04:00
Ivan Kozlovic
a3e62f000c Merge pull request #3265 from abegaj/fix-max-deliver-update
[FIXED] Maximum Deliveries update has no effect
2022-07-21 16:06:41 -06:00
Ardit Begaj
2871dcb235 increase maxDeliver 2022-07-21 23:24:56 +02:00
Ardit Begaj
c04ab08f2a improve tests based on feedback 2022-07-21 23:21:58 +02:00
Ivan Kozlovic
1da5ecfb96 [IMPROVED] JetStream: stream already exists error description
The `JSStreamNameExistErr` will now include in the description that
the stream exists with a different configuration, because that is
the error clients would get when trying to add a stream with a
different configuration (otherwise this is a no-op and client
don't get an error).

Since that error was used in case of restore, a new error is added
but uses the same description prefix "stream name already in use"
but adds ", cannot restore" to indicate that this is a restore
failure because the stream already exists.

Resolves #3273

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-07-21 10:20:07 -06:00
Ardit Begaj
94d22642d1 move test to appropriate file 2022-07-20 19:49:13 +02:00
Ardit Begaj
3f26dfee25 added cluster test 2022-07-19 20:01:36 +02:00
Todd Beets
302f85dcc2 Merge pull request #3271 from nats-io/repub-fix-srcignored
Fix and test for RePub src ignored at runtime.
2022-07-18 15:02:09 -07:00
Todd Beets
7a47631051 optimize for > or empty src (any subject) 2022-07-17 13:45:05 -07:00
Todd Beets
ae9f872ef5 Fix static code check violation on two unit tests and normalize unit test values to ADR-28. 2022-07-15 21:25:29 -07:00
Todd Beets
b145952e78 Fix and test for RePub src ignored at runtime. 2022-07-15 13:50:35 -07:00
Ivan Kozlovic
5f0ee2344a [FIXED] JetStream: stream mirror updates not rejected in standalone
Updates to stream mirror config are rejected in cluster mode, but
were not in standalone. This PR adds the check in standalone mode.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-07-15 10:51:57 -06:00
Ardit Begaj
4d81b8e4a2 Make MaxDeliver update changes take effect 2022-07-14 15:38:59 +02:00
Ivan Kozlovic
be8734e32b [CHANGED] JetStream: accept "Nats-Expected-Last-Sequence" with "0"
We use to ignore if the seq was 0, but now would treat it as
a requirement that the stream be empty if header is present but
set to 0.

This relates to client PR: https://github.com/nats-io/nats.go/pull/958

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-07-07 09:44:27 -06:00
Derek Collison
1ea608eabf Allows direct get to also do get next for subject with starting sequence
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-06 14:22:28 -07:00
Derek Collison
52baa95e2a When stalled for MaxAckPending, expire all pull requests that are one shot, meaning had at least 1 delivered message
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-04 19:14:49 -07:00
Derek Collison
1b580c67f3 Make pull consumers FIFO per message, not per request.
This effectively means that requests with batch > 1 will process a message and go to the end of the line.

Signed-off-by: Derek Collison <derek@nats.io>
2022-07-04 13:05:57 -07:00
Ivan Kozlovic
c519df7e0d [FIXED] Pull consumer may be incorrectly removed after InactiveThreshold
When creating a pull consumer with InactiveThreshold set, if the
application is doing pull requests with "no_wait" at regular interval
(lower than InactiveThreshold), the JS consumer should be considered
active and not deleted.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-06-29 10:15:09 -06:00
Derek Collison
9154fca7f1 Allow consumer filter subjects to be updated
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-24 12:38:01 -07:00
Derek Collison
b4238259b3 Change test to not have path separators
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-20 12:16:46 -07:00
Derek Collison
1ade8fc881 When stream or consumer names contained path separators it prevented backup and restore.
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-20 11:59:18 -07:00
Derek Collison
4c8110c3ff Add in support for inactivity thresholds for durable consumers.
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-14 06:51:00 -07:00
Derek Collison
fb51162e37 Make error on make bytes exceeded on a pull request a 409
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-07 06:59:08 -07:00
Derek Collison
0979bce543 Updates based on group feedback.
1. Do not use original subject since this could use Request() and we want to use muxing.
2  Place original subject and timestamp into headers.

Signed-off-by: Derek Collison <derek@nats.io>
2022-05-31 19:15:52 -07:00
Derek Collison
c8a730ce55 Stream get for KV was going through API layer, but with popularity needed a more peformant and lighter weight and direct approach.
Signed-off-by: Derek Collison <derek@nats.io>
2022-05-30 16:34:54 -07:00
Derek Collison
e08f6d863d Allow for republish to be headers only
Signed-off-by: Derek Collison <derek@nats.io>
2022-05-30 12:05:17 -07:00
Derek Collison
daa4b97eeb Don't do advisories or API stats for a direct get msg from a stream.
Signed-off-by: Derek Collison <derek@nats.io>
2022-05-30 09:32:07 -07:00
Ivan Kozlovic
a52f12613e Bump version to v2.8.4-beta.2 and fix flapper
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-05-25 13:25:45 -06:00
Derek Collison
72ed48d096 Merge pull request #3149 from nats-io/pull_perf_stable
[FIXED] Spurious pull consumer 408s under load
2022-05-25 09:44:27 -07:00
Derek Collison
d69394efad Fix spurious 408s under load and move processing of acks to their own Go routine.
Signed-off-by: Derek Collison <derek@nats.io>
2022-05-25 09:27:34 -07:00
Derek Collison
46f7f7bfc9 Consumer pending was not correct when stream had max msgs per subject set > 1 and a consumer that filtered out part of the stream was created.
Also make sure to update stream's config on a stream restore in case of changes.

Signed-off-by: Derek Collison <derek@nats.io>
2022-05-24 14:44:15 -07:00
Ivan Kozlovic
53e3c53d96 [FIXED] JetStream: consumer with deliver new may miss messages
This could happen when a consumer had not sent anything to the
attached NATS subscription and there was a consumer leader
step down or server restart.

Signed-off-by: Derek Collison <derek@nats.io>
2022-05-23 12:01:48 -06:00
Derek Collison
c166c9b199 Enable republishing of messages once stored in a stream.
This enables lightweight distribution of messages to very large number of NATS subscribers.
We add in metadata as headers that allows for gap detection which enables initial value (via JetStream, maybe KV) and realtime NATS core updates but all globally ordered.

Signed-off-by: Derek Collison <derek@nats.io>
2022-05-17 15:18:54 -07:00
Derek Collison
6bbc5f627c Support for MaxBytes for pull requests.
Signed-off-by: Derek Collison <derek@nats.io>
2022-05-16 08:43:33 -07:00
Derek Collison
bcecae42ac Fix for #3119
Signed-off-by: Derek Collison <derek@nats.io>
2022-05-12 15:45:29 -07:00