Commit Graph

471 Commits

Author SHA1 Message Date
Jean-Noël Moyne
69e137c3d2 [FIXED] Stream config idempotency (#4292)
- [X] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [X] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [x] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

Fixes a behavior where idempotency of re-defining the same stream more
than once (with the same attributes) was broken due to the DeepEqual
failing due to the StreamSource struct received from the client app not
having a value for the `iname` structure field (as it's internal) but
the StreamSource struct return from `mset.config()` would have it set.

Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-07-07 11:32:09 -07:00
Neil Twigg
d2615b76f2 Annotate CPU and goroutine profiles with account/stream/consumer info
Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-20 19:02:40 +01:00
Derek Collison
3501ca3c1f Merge branch 'main' into dev 2023-06-15 17:49:19 -07:00
Derek Collison
087a28a13e When creating replicated mirrors where the source stream had a very large starting sequence number, the server would use excessive CPU and Memory.
This is due to the mirroring functionality trying to skip messages when it detects a gap. In a replicated stream this puts excessive stress on the raft system.
This step is not needed at all if the mirror stream has no messages, we can simply jump ahead.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-15 17:20:15 -07:00
Jean-Noël Moyne
7ff114162c Adds the same check for valid stream name for Mirror
Fix test using invalid stream names

Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-06-08 07:49:47 -07:00
Jean-Noël Moyne
bd6c15d24e Adds a check that the stream name of a stream source is valid and associated new error if it isn't.
Adresses https://github.com/nats-io/nats-server/issues/4141

Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-06-08 07:49:46 -07:00
Derek Collison
df5df3ce99 Panic fixes (#4214)
- [ ] Link to issue, e.g. `Resolves #NNN`
 - [ ] Documentation added (if applicable)
 - [ ] Tests added
- [ ] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [ ] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [x] Build is green in Travis CI
- [x] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

Resolves panics in the code.

### Changes proposed in this pull request:

 - This PR fixes some of the panics in the code
2023-06-05 13:02:05 -07:00
Nikita Mochalov
5141b87dff Refactor code 2023-06-05 22:42:28 +03:00
Derek Collison
30d9dfd305 Merge branch 'main' into dev 2023-06-03 18:17:28 -07:00
Derek Collison
238282d974 Fix some data races detected in internal testing
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-03 13:58:15 -07:00
Artem Seleznev
27a8b96ee3 different panic fixes
Signed-off-by: Artem Seleznev <seleznyov.artyom@gmail.com>
2023-06-02 13:19:22 +03:00
R.I.Pienaar
fb1d86d506 Record the stream and consumer info timestamps
This records the server time when info for streams and
consumers are created so that tools such as the nats cli
can calculate time deltas for last ack, last delivered and
so forth in the context of the server clock.

This will help aleviate problems with client devices experiencing
clock jitter that can show up in user interfaces as negative
seconds since last ack etc

Signed-off-by: R.I.Pienaar <rip@devco.net>
2023-05-05 18:53:09 +02:00
Derek Collison
68f6b59fc7 Merge branch 'main' into dev 2023-05-03 19:51:24 -07:00
Derek Collison
21239022bd Protect against usage drift for any unforseen reason and if detected correct.
Signed-off-by: Derek Collison <derek@nats.io>
2023-05-03 17:09:06 -07:00
Derek Collison
eb1eb3c49e Merge branch 'main' into dev 2023-05-01 16:29:35 -07:00
Derek Collison
f098c253aa Make sure we adjust accounting reservations when deleting a stream with any issues.
Signed-off-by: Derek Collison <derek@nats.io>
2023-05-01 15:54:37 -07:00
Derek Collison
f5ac5a4da0 Fix for a bug that could leave a raft node running when stopping a stream.
This can happen when we reset a stream internally and the stream had a prior snapshot.

Also make sure to always release resources back to the account regardless if the store is no longer present.

Signed-off-by: Derek Collison <derek@nats.io>
2023-05-01 13:22:06 -07:00
Derek Collison
0321eb6484 Merge branch 'main' into dev 2023-04-29 19:52:57 -07:00
Derek Collison
b27ce6de80 Add in a few more places to check on jetstream shutting down.
Add in a helper method.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-29 11:27:18 -07:00
Derek Collison
db972048ce Detect when we are shutting down or if a consumer is already closed when removing a stream.
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-29 11:18:10 -07:00
Derek Collison
ac27fd046a Fix data race
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-28 17:57:03 -07:00
Neil Twigg
85923c4342 Apply stream compression from file store config
Signed-off-by: Neil Twigg <neil@nats.io>
2023-04-18 16:31:59 +01:00
Tomasz Pietrek
82220d9103 Merge branch 'main' into dev
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-04-14 22:38:45 +02:00
Tomasz Pietrek
a66c67baa5 Fix stream sourcing & mirroring overlap errors
When adding or updating sources/mirrors, server was checking if the stream with
a given name exists to check for subject overlaps, among other things.
However, if sourced/mirrored stream was `External`, checks should
not be executed, as not only stream would never be found,
but also, if `External` stream had the same name as the sourcing stream,
the check would be wrongly performed against itself.

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-04-14 21:00:11 +02:00
Derek Collison
a319d24345 Merge branch 'main' into dev 2023-04-13 21:03:05 -07:00
Derek Collison
bafd585ce4 Fix data race
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-13 18:40:06 -07:00
Derek Collison
dfeac4a214 Merge branch 'main' into dev 2023-04-09 19:31:01 -07:00
Derek Collison
313dd424a3 Optimize to not allocate converting strings to []byte
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-08 20:46:05 -07:00
Tomasz Pietrek
5715918ad1 Fix panic in startingSequenceForSources
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-04-07 17:31:51 +02:00
Derek Collison
83f08999a7 Fix data race
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-07 07:30:03 -07:00
Derek Collison
ff8701b724 Merge branch 'main' into dev 2023-04-06 08:37:11 -07:00
Derek Collison
2da50512e2 Optimize non-inline direct gets to not use simple go routines
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-06 07:50:57 -07:00
Derek Collison
4525bdea0d [IMPROVE] Improvements and fixes to stream source consumer creation on leadership change or on stream config source updates (#4009)
- [X] Tests added
- [X] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [X] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [x] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)
2023-04-04 20:09:15 -07:00
Derek Collison
ce115ab1e6 Allow republish for mirroring/sourcing streams (#4010)
- [X] Link to issue, e.g. `Resolves #NNN`
 - [ ] Documentation added (if applicable)
 - [ ] Tests added
- [X] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [X] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [ ] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

Resolves #4000

### Changes proposed in this pull request:

Removes the check on the republish filter needing to have an overlap
with the listened subjects, as with the new stream subject
transformation changes we do not assume anymore that the subjects in a
stream must match the subjects being listened to.

This allows the use of republish for mirroring and/or sourcing streams
2023-04-04 18:38:27 -07:00
Derek Collison
c5e19e19e7 Merge branch 'main' into dev 2023-04-03 21:22:53 -07:00
Derek Collison
ebe4f8957f Spelling based on review feedback
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 21:08:59 -07:00
Derek Collison
01a2c0472d Merge branch 'main' into dev 2023-04-03 15:33:12 -07:00
Derek Collison
6b01a21965 No inline jetstream msg processing, always queue inbound
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-03 08:27:52 -07:00
Derek Collison
b9e7b58f5c Merge branch 'main' into dev 2023-04-02 18:58:54 -07:00
Derek Collison
64b22011dc Better use of LoadAndStore based on review feedback
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-02 14:50:22 -07:00
Derek Collison
1fb1efd748 Make sure to remove any inflight entries when done
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-02 14:41:49 -07:00
Derek Collison
e6447c982a Protect against concurrent creation of streams and consumers.
Also make sure we have exited monotoring routines when doing resets for both streams and consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2023-04-02 14:29:52 -07:00
Derek Collison
48a5d270b2 Merge branch 'main' into dev 2023-04-02 04:23:52 -07:00
Derek Collison
299d9b28a9 Add original message timestamp to republished message headers (#3933)
Adds original message timestamp to republished message as
"Nats-Time-Stamp" header

 - [ ] Link to issue, e.g. `Resolves #NNN`
 - [ ] Documentation added (if applicable)
 - [x] Tests added
- [x] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [x] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [ ] Build is green in Travis CI
- [x] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

Resolves #

### Changes proposed in this pull request:

- Add original message timestamp to republished message as header
"Nats-Time-Stamp"
 -
 -
2023-04-02 04:05:53 -07:00
Derek Collison
872a9e7927 Add in monitor status similar to consumer
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-02 03:52:59 -07:00
Jean-Noël Moyne
11d88ef157 Implement comments
Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-04-01 12:24:54 -07:00
Jean-Noël Moyne
b454a577e4 Removes the check on the republish filter needing to have an overlap with the listened subjects, as with the new stream subject transformation changes we do not assume anymore that the subjects in a stream must match the subjects being listened to.
This allows the use of republish for mirroring and/or sourcing streams

Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-03-31 16:10:23 -07:00
Jean-Noël Moyne
0420c6e317 - Improve and fix stream source consumer sequence number finding on leadership change or stream config update to work with the new stream sourcing
- Updates source index name format and adds two fields (source's filter and transform) to the `"Nats-Stream-Source"` message header
- Backwards compatibility for streams containing previous (2.9) message headers for seamless upgrade
- Update TestJetStreamSourceBasics as it could flap sometimes

Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-03-31 15:49:10 -07:00
Derek Collison
d057889b13 Merge branch 'main' into dev 2023-03-31 11:36:35 -07:00
Derek Collison
ad5bb366a0 Updates to preacks when multiple consumers are present but mutually exlusive (filtered).
Signed-off-by: Derek Collison <derek@nats.io>
2023-03-31 10:43:28 -07:00