Commit Graph

6216 Commits

Author SHA1 Message Date
Ivan Kozlovic
e851fdb66b Increase test timeout for nightly code coverage
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-24 14:38:34 -06:00
Ivan Kozlovic
284e35132b Merge pull request #3387 from nats-io/fix_3317
[ADDED] Monitoring: TLS Peer Certificates in Connz when auth is on
2022-08-24 14:28:01 -06:00
Ivan Kozlovic
03ac1f256f Update based on code review
- Change finger_prints to cert_sha256 and use hex.EncodeToString
- Add spki_sha256 for RawSubjectPublicKeyInfo with hex.EncodeToString

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-24 14:16:37 -06:00
Matthias Hanel
f5ba11736b bump version to 2.9.0-RC.8
Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-08-23 17:44:56 -07:00
Matthias Hanel
970491debc scale down happened too soon
when currentCount != replicas

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-08-23 17:44:56 -07:00
Derek Collison
3e41f77247 Bump to 2.9.0-RC.7
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-23 16:38:01 -07:00
Derek Collison
547bf0540c Merge pull request #3394 from nats-io/fs-bug
Fixed a bug that would not track per subject info for streams that were mirrors/sources.
2022-08-23 16:27:27 -07:00
Matthias Hanel
eae3ffa859 [FIXED] Service import response invoking svc import (#3393)
past processing the import response, c.pa was not reset to the
appropriate state, which lead to an unintended recursion

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-08-23 16:22:46 -07:00
Derek Collison
ef71087d56 Fixed a bug that would not track per subject info for streams that were mirrors or sources.
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-23 15:46:57 -07:00
Derek Collison
36f443dcf8 Merge pull request #3392 from nats-io/chaos
General improvements to clustered streams during server restart and KV/CAS.
2022-08-23 11:32:36 -07:00
Derek Collison
c48bd6ea54 Inline turning on direct subs for non-clustered streams
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-23 11:20:18 -07:00
Derek Collison
1896af3201 Updates based on review feedback
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-23 09:19:33 -07:00
Ivan Kozlovic
380fa4499f Merge pull request #3383 from nats-io/gw_switch_to_interest_only_right_away
[CHANGED] Gateway: Switch all accounts to interest-only mode
2022-08-23 08:44:15 -06:00
Matthias Hanel
a43c1e38eb fix signing key template processing dropping allow (#3390)
Scoped signing keys allow for optional values in allow rules
If an allow rule therefore gets removed because a tag is not present,
the removal needs to be compensated by adding in a deny >

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-08-23 01:16:06 -07:00
Derek Collison
212adf5775 General improvements to clustered streams during server restart and KV/CAS scenarios.
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-22 18:36:15 -07:00
Ivan Kozlovic
4dd4b42ec1 Merge pull request #3391 from nats-io/js_cons_pending
Add test that demonstrate cons pending < stream first seq is OK
2022-08-22 19:16:31 -06:00
Ivan Kozlovic
11b00ed281 Add test that demonstrate cons pending < stream first seq is OK
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-22 18:20:25 -06:00
Matthias Hanel
c02d1ad69e fix consumer subject validation on recovery (#3389)
This fixes an issue introduced in #3080
The consumer filter subject check was skipped on recovery.

The intent was to bypass the upstream stream subjects.
But it also filtered the downstream stream subject.
This became a problem when the downstream was itself an upstream.

Then during recover, the stream subject was not checked, which
lead to delivery of filtered messages that should never have been
delivered.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-08-22 14:30:00 -07:00
Ivan Kozlovic
4a2287a50c Merge pull request #3388 from nats-io/js_shorter_tests
Reduce length of some clustering tests
2022-08-22 14:02:14 -06:00
Ivan Kozlovic
d2784589a0 Change json tag name to finger_prints
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-22 12:40:20 -06:00
Ivan Kozlovic
5663bc2fa3 Reduce length of some clustering tests
Since PR #3381, the 2 tests modified here would take twice as
long (around 245 seconds) to complete.
Talking with Matthias, he suggested using a variable instead of
a const and set it to 0 for those 2 tests since they don't really
need that to be set.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-22 12:35:37 -06:00
Ivan Kozlovic
951b7c38f6 [ADDED] Monitoring: TLS Peer Certificates in Connz when auth is on
Add basic peer certificates information in /connz endpoint when
the "auth" option is provided.

Resolves #3317

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-22 11:48:49 -06:00
Ivan Kozlovic
700fe48149 Merge pull request #3386 from nats-io/js_meta_nil_panic
Some more checks for cc.meta == nil
2022-08-22 11:48:09 -06:00
Ivan Kozlovic
b1822e1b4c Some more checks for cc.meta == nil
Missed those when re-running the previous test for longer period
of time.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-22 11:06:04 -06:00
Ivan Kozlovic
8871453c75 Merge pull request #3385 from nats-io/js_monitor_stream_panic
Fixed possible panic in monitorStream
2022-08-22 10:36:05 -06:00
Ivan Kozlovic
c30445657f Fixed possible panic in monitorStream
Saw this panic in code coverage run:
```
=== RUN   TestJetStreamClusterPeerExclusionTag
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x88 pc=0x8acd55]

goroutine 97850 [running]:
github.com/nats-io/nats-server/v2/server.(*jetStream).monitorStream(0xc002b94780, 0xc001ecb500, 0xc003229b00, 0x0)
	/home/runner/work/nats-server/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:1653 +0x495
github.com/nats-io/nats-server/v2/server.(*jetStream).processClusterCreateStream.func1()
	/home/runner/work/nats-server/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:2953 +0x3b
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine
	/home/runner/work/nats-server/src/github.com/nats-io/nats-server/server/server.go:3063 +0xa7
```

Was able to reproduce and reason was `meta` was nil.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-22 09:52:05 -06:00
Matthias Hanel
e6ae36cb7e Fix bad sys request for different account (#3382)
When a request for a system service like $SYS.REQ.ACCOUNT.*.CONNZ
is imported/exported we ensured that the requesting account is identical
to the account referenced in the subject.

In #3250 this check was extended from CONNZ to all $SYS.REQ.ACCOUNT.*.*
requests.

In general this check interferes with monitoring accounts that need
to query all other accounts, not just itself.
There the use case is that account A sends a request with account B
in the subject. The check for equal accounts prevents this.

This change removes the check to support these use cases.

Instead of the check, the default export now uses exportAuth
tokenPos to ensure that the 4th token is the importer account id.
This guarantees that an explicit export (done by user) can only import
for the own account.

This change also ensures that an explicit export is not overwritten
by the system.
This is not a problem when the export is public.
Automatic imports set the account id correctly and do not use wildcards.

To cover cases where the export is private, automatically added imports
are not subject a token check.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-08-19 17:16:53 -07:00
Ivan Kozlovic
f6c4e5fcee [CHANGED] Gateway: Switch all accounts to interest-only mode
We are phasing out the optimistic-only mode. Servers accepting
inbound gateway connections will switch the accounts to interest-only
mode.

The servers with outbound gateway connection will check interest
and ignore the "optimistic" mode if it is known that the corresponding
inbound is going to switch the account to interest-only. This is
done using a boolean in the gateway INFO protocol.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-19 16:41:44 -06:00
Matthias Hanel
6bf50dbb77 induce delay prior to scale down (#3381)
This is to avoid a narrow race between adding server and them catching
up where they also register as current.

Also wait for all peers to be caught up.

This also avoids clearing catchup marker once catchup stalled.
A stalled catchup would remove the marker causing the peer to
register as current.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-08-18 13:47:40 -07:00
Jean-Noël Moyne
2a709aaf61 - Changes to make adding new mapping functions easier (#3305)
* - Changes to make adding new mapping functions easier
- Adds new subject mapping functions:
{{SplitFromLeft(wildcard index, position)}}
{{SplitFromRight(wildcard index, position)}}
{{SliceFromLeft(wildcard index, slice size)}}
{{SliceFromRight(wildcard index, slice size)}}
{{Split(wildcard index, deliminator)}}

Examples:
	shouldMatch("*", "{{splitfromleft(1,3)}}", "12345", "123.45")
	shouldMatch("*", "{{SplitFromRight(1,3)}}", "12345", "12.345")
	shouldMatch("*", "{{SliceFromLeft(1,3)}}", "1234567890", "123.456.789.0")
	shouldMatch("*", "{{SliceFromRight(1,3)}}", "1234567890", "1.234.567.890")
	shouldMatch("*", "{{split(1,-)}}", "-abc-def--ghi-", "abc.def.ghi")
	shouldMatch("*.*", "{{split(2,-)}}.{{splitfromleft(1,2)}}", "foo.-abc-def--ghij-", "abc.def.ghij.fo.o")

- Subject mapping functions can now be all lower case or Pascal case (or a combination): e.g. splitfromleft, SplitFromLeft, splitFromleft, etc...
2022-08-18 09:52:28 -07:00
Ivan Kozlovic
aa02c12711 Merge pull request #3379 from mprimi/chaos-test-kv
Chaos tests for KeyValue
2022-08-18 10:22:09 -06:00
Ivan Kozlovic
76fec26af2 Bump version to RC.6
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-18 09:02:51 -06:00
Derek Collison
1228a32bc5 Merge pull request #3380 from nats-io/direct-get-delay
Don't immediately listen on the direct get subjects.
2022-08-17 17:26:52 -07:00
Matthias Hanel
904b7aeefc fixed consumer source update receiving upstream msgs (#3364)
if an origin stream contains:
1M msgs with subject foo and 1M msgs with subject bar

IF the source consumer changes their filter from foo to bar
Then it would have received messages for subject bar.
This happens because this tail was filtered and their
respective seqno was not communicated to the consumer.

This is somewhat unexpected. It is also coincidental.
Had the last message in the stream had subject foo then
this wouldn't happen.

Therefore, when completely changing the subject say,
from foo to bar, we only receive messages received
after the time the change was made.

However, if the old and new subject overlap in any way,
we go by sequence number. Meaning in these cases the
outlined behavior remains in order to not induce artificial
message loss for the part of the subject space that is
covered by old and new filter.

Signed-off-by: Matthias Hanel <mh@synadia.com>

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-08-17 17:16:03 -07:00
Derek Collison
ce2d5fa173 Don't immediately listen on the direct get subjects.
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-17 16:39:34 -07:00
Marco Primi
d5277bf8ad Chaos tests for KeyValue 2022-08-17 16:25:14 -07:00
Derek Collison
c61465b344 Bump to 2.9.0-RC.5
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-17 15:17:58 -07:00
Matthias Hanel
9892a132e7 Improve StreamMoveInProgressError (#3376)
by adding progress indicators

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-08-17 15:12:32 -07:00
Derek Collison
8612ca66c8 Merge pull request #3377 from nats-io/no-fss
When filestore is used for raft layer do not attempt to track subjects
2022-08-17 15:10:56 -07:00
Derek Collison
3739e50070 Merge pull request #3378 from nats-io/meta-cleanup-fix
We can't purge directories here since not 100% sure all state is in snapshot.
2022-08-17 15:10:42 -07:00
Derek Collison
9c9de656c6 We can't purge directories here since not 100% sure all state is in snapshot.
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-17 14:57:19 -07:00
Derek Collison
35135948a0 Make sure llts update under lock, fss can be force expired so remove.
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-17 14:54:35 -07:00
Derek Collison
d48ccf4c5a When filestore is used for raft layer do not attempt to track subject metadata.
Signed-off-by: Derek Collison <derek@nats.io>
2022-08-17 13:46:13 -07:00
Ivan Kozlovic
81a8016938 Merge pull request #3374 from nats-io/gw_possible_panic
[FIXED] Gateway: possible panic if monitor endpoint inspected too soon
2022-08-17 13:55:06 -06:00
Ivan Kozlovic
5d3ee8ebf4 [FIXED] Gateway: possible panic if monitor endpoint inspected too soon
The monitoring http server is started early and the gateway setup
(when configured) may not be fully ready when the `/gatewayz`
endpoint is inspected and could cause a panic.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-17 13:30:58 -06:00
Matthias Hanel
c67d6aad79 fix jwt template ordering issue and error message (#3373)
ordering of templates got messed up by a map (now removed)
Also improved error message when template generation fails

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-08-16 19:00:41 -07:00
Ivan Kozlovic
4b4de20c25 Merge pull request #3365 from nats-io/fix_3361
[FIXED] JetStream: issue with max deliver and server/cluster restart
2022-08-16 19:13:49 -06:00
Ivan Kozlovic
02ecda535c Stop the raft node to not cause test to flap.
Test TestNoRaceJetStreamClusterCorruptWAL() would start to flap
because of the snapshot on cluster shutdown. Disable the snapshot
on exit for this test by stopping the raft node before shutdown.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-16 18:44:32 -06:00
Ivan Kozlovic
7de4497815 Install consumer snapshot on clean exit and few other fixes
- didRemove in applyMetaEntries() could be reset when processing
multiple entries
- change "no race" test names to include JetStream
- separate raft nodes leader stepdown and stop in server
shutdown process
- in InstallSnapshot, call wal.Compact() with lastIndex+1

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-16 17:05:49 -06:00
Ivan Kozlovic
f0b098af92 [FIXED] JetStream: issue with max deliver and server/cluster restart
This is a regression introduced in v2.8.3. If a message reaches
the max redeliver count, it stops being delivered to the consumer.
However, after a server or cluster restart, those messages would
be redelivered again.

Resolves #3361

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-16 17:05:47 -06:00