Commit Graph

430 Commits

Author SHA1 Message Date
Derek Collison
b9e7b58f5c Merge branch 'main' into dev 2023-04-02 18:58:54 -07:00
Derek Collison
64b22011dc Better use of LoadAndStore based on review feedback
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-02 14:50:22 -07:00
Derek Collison
1fb1efd748 Make sure to remove any inflight entries when done
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-02 14:41:49 -07:00
Derek Collison
e6447c982a Protect against concurrent creation of streams and consumers.
Also make sure we have exited monotoring routines when doing resets for both streams and consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-02 14:29:52 -07:00
Derek Collison
48a5d270b2 Merge branch 'main' into dev 2023-04-02 04:23:52 -07:00
Derek Collison
299d9b28a9 Add original message timestamp to republished message headers (#3933)
Adds original message timestamp to republished message as
"Nats-Time-Stamp" header

 - [ ] Link to issue, e.g. `Resolves #NNN`
 - [ ] Documentation added (if applicable)
 - [x] Tests added
- [x] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [x] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [ ] Build is green in Travis CI
- [x] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

Resolves #

### Changes proposed in this pull request:

- Add original message timestamp to republished message as header
"Nats-Time-Stamp"
 -
 -
2023-04-02 04:05:53 -07:00
Derek Collison
872a9e7927 Add in monitor status similar to consumer
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-02 03:52:59 -07:00
Derek Collison
d057889b13 Merge branch 'main' into dev 2023-03-31 11:36:35 -07:00
Derek Collison
ad5bb366a0 Updates to preacks when multiple consumers are present but mutually exlusive (filtered).
Signed-off-by: Derek Collison <derek@nats.io>
2023-03-31 10:43:28 -07:00
Derek Collison
6ced966776 Merge branch 'main' into dev 2023-03-30 21:09:45 -07:00
Derek Collison
8c0a45edf9 Make sure to lock on clearing if not removing.
Signed-off-by: Derek Collison <derek@nats.io>
2023-03-30 20:42:28 -07:00
Derek Collison
937ef0d2a6 Improvements to preAcks.
Better handling of multiple consumers so as to not delete too early.

Signed-off-by: Derek Collison <derek@nats.io>
2023-03-30 20:29:15 -07:00
Derek Collison
625ffa8685 Merge branch 'main' into dev 2023-03-29 17:19:45 -07:00
Derek Collison
152b25c314 Update server/stream.go
Pre-allocate

Co-authored-by: Neil <neil@nats.io>
2023-03-29 15:29:51 -07:00
Derek Collison
5cabc365df General improvements around handling interest retention.
1. During ackMsg processing hold write lock to block concurrent access.
2. Check for presence of preAcks before and force removal if present.
3. Rework check for orphan msgs on startup to use checkStateForInterestStream().

Signed-off-by: Derek Collison <derek@nats.io>
2023-03-29 12:43:51 -07:00
Derek Collison
2eaede8b1d Merge branch 'main' into dev 2023-03-20 21:35:15 -07:00
Derek Collison
ed9de4b0a1 Improved publisher performance under some instances of asymmetric network latency clusters on interest based streams.
Under asymmetric network latency based clusters, if a node in an R3 was replicating a consumer and the parent stream, but was the leader of neither, but the path from the stream leader was faster then the consumer leader a replicated ack could arrive before the message itself.

In this case we used to forward a delete message request to the stream leader which would then replicate that to all stream replicas, causing more work which could lead to increased publisher times on clients connected to the slow node.

Signed-off-by: Derek Collison <derek@nats.io>
2023-03-20 20:53:45 -07:00
Andres Morey
2d4c12c5dd added original message timestamp to republished message headers 2023-03-01 22:08:52 +03:00
Derek Collison
6507a913b3 Merge branch 'main' into dev 2023-03-01 05:05:41 -08:00
Derek Collison
9f1580686a Revert behavior for JetStream published directly from client to be handled inline.
Signed-off-by: Derek Collison <derek@nats.io>
2023-03-01 04:35:52 -08:00
Derek Collison
7bd7cda021 Merge branch 'main' into dev 2023-02-28 15:17:24 -08:00
Derek Collison
3807441fd7 Always process inbound messages in separate execution context.
Do not duplicate work on leader, sealed and clustered state.

Signed-off-by: Derek Collison <derek@nats.io>
2023-02-28 11:45:31 -08:00
Derek Collison
b19fe508c4 Do not block routes/gws on internal stream and consumer info requests
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-28 11:17:29 -08:00
Derek Collison
2642a8c03d Optimize locking for when under heavy loads.
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-27 18:56:55 -08:00
Tomasz Pietrek
d400664b67 Fix source header name retrieval
Stream subject mapping added index prefix to Nats-Stream-Source which
was stripped when retrieving that header. That caused
startingSequenceForSources to iterate over whole stream because of
name mismatch.
Stripping was removed in this commit.

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-02-27 13:18:23 +01:00
Derek Collison
7ce23c46ce Merge branch 'main' into dev 2023-02-21 08:34:08 -08:00
Neil Twigg
68961ffedd Refactor ipQueue to use generics, reduce allocations 2023-02-21 14:50:09 +00:00
Derek Collison
cac712b1d1 Merge branch 'main' into dev 2023-02-20 18:30:26 -08:00
Derek Collison
e270e9538f Do not warn if consumer replicas condigured to 0
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-18 11:50:26 -08:00
Derek Collison
efa3bcc49d Parallel consumer creation could drop responses (create and info) and could also run monitorConsumer twice.
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-18 05:16:05 -08:00
Derek Collison
6920f24217 Merge branch 'main' into dev 2023-02-17 10:45:07 -08:00
Derek Collison
11b0f214d0 Do not re-calculate NumPending on consumer info calls.
We noticed this was being called alot in user environments.
When the consumer was filtered with a wilcard and the stream had a high cardinality of subjects and was falling behind this could take a substantial amount of time.

Signed-off-by: Derek Collison <derek@nats.io>
2023-02-16 16:30:14 -08:00
Tomasz Pietrek
af338d0d59 Add multiple subject filters 2023-02-13 09:38:40 +01:00
Derek Collison
3fe85e6cc4 Merge branch 'main' into dev 2023-02-06 21:57:49 +04:00
Derek Collison
b611e37e95 For updating a consumer filter subject make sure locking ordere correct and that our sublist is present.
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-06 21:34:48 +04:00
Derek Collison
f7615acb7d Merge branch 'main' into dev 2023-02-04 22:23:50 -08:00
Derek Collison
b22ed47a26 Use fast state in case many interior deletes and small fix for staticcheck
Signed-off-by: Derek Collison <derek@nats.io>
2023-02-04 13:55:41 -08:00
Derek Collison
2c4ec09cc7 Merge branch 'main' into dev 2023-02-02 11:01:50 -07:00
Tomasz Pietrek
46af979871 Fix current consumers not getting messages after purge
Until now, purge updated all consumers sequences
even if purge subject was only a subset of given consumer filter.
Because of that, even messages from not purged subjects were not fetched
or properly accounted for existing consumers.

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-02-02 16:27:32 +01:00
Derek Collison
2d6b316efd Merge branch 'main' into dev 2023-01-30 20:07:26 -08:00
Derek Collison
1252653c16 Merge pull request #3829 from nats-io/jarema/fix-message-after-update
Fix Consumer not getting messages after filter update
2023-01-30 19:59:32 -08:00
Derek Collison
aaacca8549 Merge branch 'main' into dev 2023-01-30 19:54:37 -08:00
Derek Collison
6058056e3b Minor fixes and optimizations for snapshots.
We were snappshotting more then needed, so double check that we should be doing this at the stream and consumer level.
At the raft level, we should have always been compacting the WAL to last+1, so made that consistent. Also fixed bug that would not skip last if more items behind the snapshot.

Signed-off-by: Derek Collison <derek@nats.io>
2023-01-30 17:54:18 -08:00
Tomasz Pietrek
836848ca64 Fix Consumer not getting messages after filter update
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-01-30 20:47:17 +01:00
Jean-Noël Moyne
3518c9460a Small formatting improvements 2023-01-28 12:59:36 -08:00
Jean-Noël Moyne
ccc492bdce Improve checking that subject filters and destination transforms are valid subjects.
Improve error messages when bad filters/transforms are encountered.
Cover all cases of updating an existing stream's subject transform or sources
2023-01-27 17:54:37 -08:00
Derek Collison
11ffd69575 Merge branch 'main' into dev 2023-01-27 13:41:48 -08:00
Derek Collison
f4e6481ce7 Allow report cycles between source streams if subjects truly form a cycle.
Signed-off-by: Derek Collison <derek@nats.io>
2023-01-27 13:03:24 -08:00
Derek Collison
b9f220ec80 Merge pull request #3823 from ripienaar/mappings_breaking_changes
Mappings breaking changes
2023-01-27 07:48:04 -08:00
R.I.Pienaar
348c84a469 restore public APIs and fix breaking interface renames
Signed-off-by: R.I.Pienaar <rip@devco.net>
2023-01-27 12:36:15 +01:00