Commit Graph

5167 Commits

Author SHA1 Message Date
Derek Collison
f342f6a758 Merge branch 'main' into dev 2023-06-05 14:13:18 -07:00
Derek Collison
2e2ac33920 [IMPROVED] When R1 consumers were recreated with the same name when they became inactive. (#4216)
When consumers were R1 and the same name was reused, server restarts
could try to cleanup old ones and effect the new ones. These changes
allow consumer name reuse more effectively during server restarts.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-05 14:04:53 -07:00
Derek Collison
df5df3ce99 Panic fixes (#4214)
- [ ] Link to issue, e.g. `Resolves #NNN`
 - [ ] Documentation added (if applicable)
 - [ ] Tests added
- [ ] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [ ] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [x] Build is green in Travis CI
- [x] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

Resolves panics in the code.

### Changes proposed in this pull request:

 - This PR fixes some of the panics in the code
2023-06-05 13:02:05 -07:00
Derek Collison
4ac45ff6f3 When consumers were R1 and the same name was reused, server restarts could try to cleanup old ones and effect the new ones.
These changes allow consumer name reuse more effectively during server restarts.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-05 12:48:18 -07:00
Nikita Mochalov
5141b87dff Refactor code 2023-06-05 22:42:28 +03:00
Nikita Mochalov
4c181bc99a Use sentinel error 2023-06-05 22:41:09 +03:00
Nikita Mochalov
f71c49511b Fix client panic on absent server field 2023-06-05 15:27:45 +03:00
Derek Collison
af318be5db Merge branch 'main' into dev 2023-06-04 13:30:15 -07:00
Maurice van Veen
132567de39 Fix PurgeEx replay with sequence & keep succeeds 2023-06-04 11:56:28 +02:00
Derek Collison
30d9dfd305 Merge branch 'main' into dev 2023-06-03 18:17:28 -07:00
Derek Collison
e1f8064e9e [FIXED] Make sure to process extended purge operations correctly when being replayed. (#4212)
This is an extension to the excellent work by @MauriceVanVeen and his
original PR #4197 to fully resolve for all use cases.

Signed-off-by: Derek Collison <derek@nats.io>

Resolves #4196
2023-06-03 18:12:22 -07:00
Derek Collison
dee532495d Make sure to process extended purge operations correctly when being replayed on a restart.
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-03 17:49:45 -07:00
Derek Collison
eb09ddd73a [FIXED] Killed server on restart could render encrypted stream unrecoverable (#4210)
When a server was killed on restart before an encrypted stream was
recovered the keyfile was removed and could cause the stream to not be
recoverable.

We only needed to delete the key file when converting ciphers and right
before we add the stream itself.

Signed-off-by: Derek Collison <derek@nats.io>

Resolves #4195
2023-06-03 17:36:10 -07:00
Derek Collison
238282d974 Fix some data races detected in internal testing
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-03 13:58:15 -07:00
Derek Collison
4c1b93d023 Make sure to put the keyfile back if we did not recover the stream.
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-03 11:21:58 -07:00
Derek Collison
d5ae96f54d When a server was killed on restart before an encrypted stream was recovered the keyfile was removed and could cause the stream to not be recoverable.
We only needed to delete the key file when converting ciphers and right before we add the stream itself.

Signed-off-by: Derek Collison <derek@nats.io>
2023-06-03 11:21:47 -07:00
Derek Collison
df901dc1aa Merge branch 'main' into dev 2023-06-02 16:45:07 -07:00
Derek Collison
22c97d67ff [FIXED] Daisy chained leafnodes sometimes would not propagate interest (#4207)
When we were optimizing for single cluster and large numbers of
leafnodes we inadvertently broke a daisy chained scenario where a server
was a spoke and a hub within a single hub server.

So interest on D would not propagate properly to server A as a
publisher.

```
     B
   /    \
A       C -- D (SUB)
 |
PUB
```
2023-06-02 16:43:21 -07:00
Derek Collison
b2ac621212 Bump to 2.9.18-beta (#4182) 2023-06-02 16:40:23 -07:00
Derek Collison
1bce79750e When we were optimizing for single cluster but large number of leafnodes we inadvertently broke a daisy chained scenarion where a server was a spoke and a hub with a single hub cluster.
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-02 15:16:36 -07:00
Derek Collison
8278a7a0b2 Merge branch 'main' into dev 2023-06-02 12:44:02 -07:00
Derek Collison
27bbfb7a85 Only check ack floor if we are interest policy based.
Signed-off-by: Derek Collison <derek@nats.io>
2023-06-02 11:04:00 -07:00
Artem Seleznev
27a8b96ee3 different panic fixes
Signed-off-by: Artem Seleznev <seleznyov.artyom@gmail.com>
2023-06-02 13:19:22 +03:00
R.I.Pienaar
c24547eb4e Record the stream and consumer info timestamps (#4133)
This records the server time when info for streams and consumers are
created so that tools such as the nats cli can calculate time deltas for
last ack, last delivered and so forth in the context of the server
clock.

This will help aleviate problems with client devices experiencing clock
jitter that can show up in user interfaces as negative seconds since
last ack etc
2023-06-02 08:53:28 +03:00
Derek Collison
c3234dc059 [IMPROVED] Subject transform validation and error reporting (#4202)
- [X] Tests added
- [X] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [X] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [ ] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

### Changes proposed in this pull request:
1: Improves error reporting for weighted mappings:
As it was, any error in a weighted mapping would return a very
unhelpfull error message.

e.g. `nats-server: mappingtest.cfg:38:39: interface conversion:
interface {} is []interface {}, not string`

This was because the line `err := &configErr{tk, fmt.Sprintf("Error
adding mapping for %q to %q : %v", subj, v.(string), err)}` would panic
on the `v.(string)` since in weighted mapping that interface{} is
actually a map[string]interface{} (since there's can be more than one
mapping in weighted mappings).

Now returns the actual error:

e.g. `nats-server: mappingtest.cfg:40:3: Error adding mapping for "bla"
: invalid mapping destination: wildcard index out of range in
{{wildcard(1)}}`

2: improves subject transform checking and catches if the destination is
using a mapping function and there are no partial wildcards in the
source.
2023-06-01 16:57:48 -07:00
Jean-Noël Moyne
0bd3fa4a13 Harmonize subject mapping error variable names
Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-06-01 14:15:27 -07:00
Ivan Kozlovic
a744cb8cd2 Fixed delivery of retained messages after transfer.
I was running a manual test moving from dev to this branch and
noticed that the consumer would receive only 1 message of the 10
messages sent as retained. So I modified the test to verify that
we receive them all and we did not.

The reason was that after the transfer we need to refresh the state
of the stream (stream info) since we attempt to load all messages
based on the state's sequences.

I have also modified a bit the code to update the MaxMsgsPer once
all messages have been transferred.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2023-06-01 10:00:18 +01:00
Neil Twigg
4f797a54e0 Add test for MQTT retained message migration
Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-01 10:00:18 +01:00
Neil Twigg
007565ffd0 Migrate old retained messages to new subjects
Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-01 10:00:18 +01:00
Neil Twigg
74690388f5 Per-subject limits for MQTT retained messages
Signed-off-by: Neil Twigg <neil@nats.io>
2023-06-01 10:00:18 +01:00
Tomasz Pietrek
261f39bb7d Make sure that o.subjf is nil or checked if empty
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-06-01 08:29:09 +02:00
Tomasz Pietrek
a463dfe5c4 Improve consumer error handling
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-06-01 08:29:03 +02:00
Tomasz Pietrek
9a792482e9 Improve consumer with multiple filters
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-06-01 08:28:19 +02:00
Jean-Noël Moyne
08987bd173 1: Improves error reporting for weighted mappings:
As it was, any error in a weighted mapping would return a very unhelpfull error message.

e.g. `nats-server: mappingtest.cfg:38:39: interface conversion: interface {} is []interface {}, not string`

This was because the line `err := &configErr{tk, fmt.Sprintf("Error adding mapping for %q to %q : %v", subj, v.(string), err)}` would panic on the `v.(string)` since in weighted mapping that interface{} is actually a map[string]interface{} (since there's can be more than one mapping in weighted mappings).

Now returns the actual error:

e.g. `nats-server: mappingtest.cfg:40:3: Error adding mapping for "bla" : invalid mapping destination: wildcard index out of range in {{wildcard(1)}}`

2: improves subject transform checking and catches if the destination is using a mapping function and there are no partial wildcards in the source.
Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>
2023-05-31 16:28:31 -07:00
Ivan Kozlovic
86a319a50e [FIXED] LeafNode: data race during validation and create leafnode
The issue really was that the test was sharing remote options. The
way options are used is not ideal since we reference the user provided
options (but it is relied upon now in many tests and possibly users
setups). The other side of the issue was that when no local account
is specified in a "remote" specification, we set it to the global
account, but that was done when creating the leafnode object (when
soliciting), which in the case of the test could race with the
second server doing the validation.

In this PR we move the setting to global account during the validation,
but also fixed the tests to not share the remote options configuration
slice between the two servers.

Resolves #4191

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2023-05-24 17:52:54 -06:00
Derek Collison
0db9d20383 When in async mode make sure to not flush or create new files after close. (#4193)
Some async tests were showing artifacts being left after a message block
was being closed.

Signed-off-by: Derek Collison <derek@nats.io>
2023-05-24 14:27:04 -07:00
Derek Collison
aac9135e0f When in async mode make sure to not flush or create new files after being closed.
Signed-off-by: Derek Collison <derek@nats.io>
2023-05-24 12:03:41 -07:00
Ivan Kozlovic
7cf00c8ef7 Updates based on PR code review
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2023-05-19 12:16:24 -06:00
Ivan Kozlovic
607b0ca7f3 Fixed cluster permissions configuration reload
This is a rework of incorrect changes made in PR #4001.
This affects only the `dev` branch.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2023-05-18 19:02:03 -06:00
Byron Ruth
b24f0f393a Bump to 2.9.18-beta
Signed-off-by: Byron Ruth <byron@nats.io>
2023-05-18 14:22:22 -04:00
Byron Ruth
f3dac91d2a Prepare v2.9.17 release
Include fix with GoReleaser for nightly.

Signed-off-by: Byron Ruth <byron@nats.io>
2023-05-18 13:57:40 -04:00
Derek Collison
646ae9eddf Merge branch 'main' into dev 2023-05-18 09:18:34 -07:00
Derek Collison
7e3f3f4908 Make health checks more consistent with stream health checks.
Check for closed state on leader change for consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2023-05-18 08:18:53 -07:00
Derek Collison
ee87df250c Merge branch 'main' into dev 2023-05-17 19:27:58 -07:00
Derek Collison
8e825001d2 When we receive a catchup request for an item beyond our current state, we should stepdown.
Signed-off-by: Derek Collison <derek@nats.io>
2023-05-17 17:30:35 -07:00
Derek Collison
75e8df13d7 Allows cluster filtering in account subject mapping (#4175)
- [X] Branch rebased on top of current main (`git pull --rebase origin
main`)
- [X] Changes squashed to a single commit (described
[here](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html))
 - [ ] Build is green in Travis CI
- [X] You have certified that the contribution is your original work and
that you license the work to the project under the [Apache 2
license](https://github.com/nats-io/nats-server/blob/main/LICENSE)

### Changes proposed in this pull request:

Allows (exposes) the existing cluster filtering option of account level
subject mapping, thereby allowing you to define different mapping
destination per cluster (e.g. to insert the cluster name as a subject
token).

Example:
```
mappings {
	"foo":[
		{destination:"foo-east", cluster: "east", weight:100},
		{destination:"foo-west", cluster: "west", weight:100},
	]
}
```
2023-05-17 17:07:24 -07:00
Derek Collison
fbfa593b3d Merge branch 'main' into dev 2023-05-17 16:48:47 -07:00
Derek Collison
7dfe5e528e Bump to 2.9.17-RC.3
Signed-off-by: Derek Collison <derek@nats.io>
2023-05-17 16:46:10 -07:00
Derek Collison
94457e2d55 [IMPROVED] Reset logic for streams (#4177)
When we detect conditions to reset streams, make sure we properly clean
up old NRG nodes etc.

Signed-off-by: Derek Collison <derek@nats.io>
2023-05-17 16:45:00 -07:00
Derek Collison
a8d7d3886e Make sure to delete the stream assignment node here
Signed-off-by: Derek Collison <derek@nats.io>
2023-05-17 16:19:39 -07:00