Commit Graph

274 Commits

Author SHA1 Message Date
cui fliter
f1f5a59e9b fix some comments
Signed-off-by: cui fliter <imcusg@gmail.com>
2023-04-25 11:28:59 +08:00
Derek Collison
9a3e0b783c Fix for a data race when setting up service import subscriptions.
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-17 06:40:09 -07:00
Julien Viard de Galbert
1b1610f2da Adding hard_delete option on resolver configuration
Signed-off-by: Julien Viard de Galbert <jviarddegalbert@scaleway.com>
2023-04-05 10:05:48 +02:00
Tomasz Pietrek
10c2c387b7 Fix panic if serviceExport is nil
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-02-28 15:36:37 +01:00
Tomasz Pietrek
8732022d26 Fix JWT claims update if headers are passed in request
Claims update message requires only payload to be passed,
but passing headers should not fail the request.
This change ensures we extract payload from raw message
before decoding it.
Before this change, passing claims update with headers
would return cryptic `expected x chunks` error.

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-02-28 11:09:18 +01:00
Maurice van Veen
05695d304c Fixed a bug where partition was used with multiple wildcard token position 2023-02-20 10:27:29 +01:00
Neil Twigg
ba0b1f8bd3 Fix stack overflow panic in reverse entry check when inbox ends with wildcard 2023-02-13 09:45:50 +00:00
Derek Collison
9c02be2409 Various fixes for snapshots.
Due to bug, in rare circumstances could write an empty snapshot for aplied == 0. This would cause a spinning at the raft layer.

1. Allow Truncate() to also properly do a reset of the store when terms were only mismatch.
2. During testing fixed memstore truncate and also made sure per subject info was also cleaned up.
3. Then added fix to detect a bad snapshot on initialization and remove.
4. Do not allow snapshots for applied == 0.

Signed-off-by: Derek Collison <derek@nats.io>
2023-02-04 13:46:06 -08:00
Neil Twigg
14d0ba1c65 Fix some lint errors after move to golangci-lint 2022-12-30 20:00:08 +00:00
Waldemar Quevedo
861f98128a Remove individual import subs from leafnodes
When a leafnode connection is bound to an account where there was already
a wildcard response import subscription to handle the requests (e.g. `_R_.foo.>`),
this would have created message duplicates due to an extra subscription
being created that also matched the wildcard (e.g. `_R_.foo.bar`).

To avoid this condition, we now skip creating the latter extra subscription
for leafnode connections.

Signed-off-by: Waldemar Quevedo <wally@nats.io>
2022-12-29 15:30:17 -08:00
Waldemar Quevedo
6df92f40dc Fix to reenable JetStream for account when reenabled
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2022-12-06 15:53:03 -08:00
Ivan Kozlovic
9e36918f41 [FIXED] JetStream: Account removal leaks subscriptions
Resolve #3623

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-11-14 16:40:21 -07:00
Ivan Kozlovic
da287b0aea [FIXED] Weighted subject mappings updates not applied
Suppose an account is updated to have the following weighted mapping:
```
foo -> bar 40%
```
The server automatically adds foo -> foo at 60%. Sending messages
to "foo" will result in the expected distribution of 60% messages
going to "foo" and 40% going to bar.

However, if a successive update is pushed to the server(s):
```
foo -> bar 40%
foo -> baz 60%
```
The subject mapping should now be as described, that is, no more
mapping from "foo" to "foo" and 40% to bar and 60% to baz, however,
what was happening is that the server would always use the original
mapping.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-11-11 15:45:32 -07:00
Ivan Kozlovic
3ec42d5b85 Updates to PR #3611
- Save the TLS name only if not already set
- Use the passed URLs slice instead of using s.getOpts().Routes
- Enhanced the test
- Fixed an unrelated DATA RACE report

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-11-08 09:36:08 -07:00
jnmoyne
a1f90b8776 Fixes mishandling of an edge condition in the {{Split()}} subject mapping function 2022-09-09 12:42:03 -07:00
Ivan Kozlovic
a8318d1f62 [FIXED] Service import/export cycles causing stack overflow
There was a way to detect a cycle but I believe it needs to be
a stack of "si" not just the one before invoking processServiceImport.

Changes in #3393 would solve issue reported with test TestAccountImportCycle,
but would not address the new reported issue represented by new test
TestLeafNodeSvcImportExportCycle. This current approach seems to solve
all known cases.

Resolves #3397
Replaces #3393
2022-08-26 14:40:02 -06:00
Matthias Hanel
eae3ffa859 [FIXED] Service import response invoking svc import (#3393)
past processing the import response, c.pa was not reset to the
appropriate state, which lead to an unintended recursion

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-08-23 16:22:46 -07:00
Matthias Hanel
e6ae36cb7e Fix bad sys request for different account (#3382)
When a request for a system service like $SYS.REQ.ACCOUNT.*.CONNZ
is imported/exported we ensured that the requesting account is identical
to the account referenced in the subject.

In #3250 this check was extended from CONNZ to all $SYS.REQ.ACCOUNT.*.*
requests.

In general this check interferes with monitoring accounts that need
to query all other accounts, not just itself.
There the use case is that account A sends a request with account B
in the subject. The check for equal accounts prevents this.

This change removes the check to support these use cases.

Instead of the check, the default export now uses exportAuth
tokenPos to ensure that the 4th token is the importer account id.
This guarantees that an explicit export (done by user) can only import
for the own account.

This change also ensures that an explicit export is not overwritten
by the system.
This is not a problem when the export is public.
Automatic imports set the account id correctly and do not use wildcards.

To cover cases where the export is private, automatically added imports
are not subject a token check.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-08-19 17:16:53 -07:00
Jean-Noël Moyne
2a709aaf61 - Changes to make adding new mapping functions easier (#3305)
* - Changes to make adding new mapping functions easier
- Adds new subject mapping functions:
{{SplitFromLeft(wildcard index, position)}}
{{SplitFromRight(wildcard index, position)}}
{{SliceFromLeft(wildcard index, slice size)}}
{{SliceFromRight(wildcard index, slice size)}}
{{Split(wildcard index, deliminator)}}

Examples:
	shouldMatch("*", "{{splitfromleft(1,3)}}", "12345", "123.45")
	shouldMatch("*", "{{SplitFromRight(1,3)}}", "12345", "12.345")
	shouldMatch("*", "{{SliceFromLeft(1,3)}}", "1234567890", "123.456.789.0")
	shouldMatch("*", "{{SliceFromRight(1,3)}}", "1234567890", "1.234.567.890")
	shouldMatch("*", "{{split(1,-)}}", "-abc-def--ghi-", "abc.def.ghi")
	shouldMatch("*.*", "{{split(2,-)}}.{{splitfromleft(1,2)}}", "foo.-abc-def--ghij-", "abc.def.ghij.fo.o")

- Subject mapping functions can now be all lower case or Pascal case (or a combination): e.g. splitfromleft, SplitFromLeft, splitFromleft, etc...
2022-08-18 09:52:28 -07:00
Ivan Kozlovic
3c9a7cc6e5 Move to Go 1.19, remote io/util, fix data race and a flapper
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-05 09:55:37 -06:00
Ivan Kozlovic
f19908979f Revert direct changes to main
This reverts commit cf784c19f0.
This reverts commit c269a1ca09.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-07-28 17:25:37 -06:00
jnmoyne
cf784c19f0 - Adds new subject mapping functions:
{{SplitFromLeft(wildcard index, position)}}
{{SplitFromRight(wildcard index, position)}}
{{SliceFromLeft(wildcard index, slice size)}}
{{SliceFromRight(wildcard index, slice size)}}
{{Split(wildcard index, deliminator)}}

Examples:
	shouldMatch("*", "{{splitfromleft(1,3)}}", "12345", "123.45")
	shouldMatch("*", "{{SplitFromRight(1,3)}}", "12345", "12.345")
	shouldMatch("*", "{{SliceFromLeft(1,3)}}", "1234567890", "123.456.789.0")
	shouldMatch("*", "{{SliceFromRight(1,3)}}", "1234567890", "1.234.567.890")
	shouldMatch("*", "{{split(1,-)}}", "-abc-def--ghi-", "abc.def.ghi")
	shouldMatch("*.*", "{{split(2,-)}}.{{splitfromleft(1,2)}}", "foo.-abc-def--ghij-", "abc.def.ghij.fo.o")

- Subject mapping functions can now be all lower case or Pascal case (or a combination): e.g. splitfromleft, SplitFromLeft, splitFromleft, etc...
2022-07-28 12:53:14 -07:00
jnmoyne
c269a1ca09 New expendable implementation of subject mapping destinations to transform processing 2022-07-28 12:30:22 -07:00
Todd Beets
6de64667bc follow convention (_EMPTY_, fwcs). Add special case short-circuit to skip unnecessary tokenization and transform. 2022-07-18 14:20:10 -07:00
Todd Beets
7a47631051 optimize for > or empty src (any subject) 2022-07-17 13:45:05 -07:00
Jean-Noël Moyne
23a9036b9b Improves error handling for subject mapping destination mustache functions (#3255) 2022-07-11 14:56:34 -07:00
Jean-Noël Moyne
e46b00639a Resolves #3151 plus redo and improve subject mapping destination validation and error handling (#3231)
* 1: Allows spaces to be used inside {{}} subject mapping functions:

2: Rework and improve mapping destinations validation and error handling with much more helpful error messages, e.g.:

* Error adding mapping for "foo.*.*" to "bar.{{wildcard(1)}}" : invalid mapping destination: not using all of the token wildcard(s) in bar.{{wildcard(1)}}

* Error adding mapping for "myservice.request.*" to "myservice.request.{{ partition(10) }}.{{wildcard(1)}}" : invalid mapping destination: not enough arguments passed to the function in {{ partition(10) }}

* Error adding mapping for "myservice.request.*" to "myservice.request.{{ partition(10,2) }}.{{wildcard(1)}}" : invalid mapping destination: wildcard index out of range in {{ partition(10,2) }}: [2]

* Error adding mapping for "myservice.request.*" to "myservice.request.{{ partition(10,1) }}.{{wildcard()}}" : invalid mapping destination: not enough arguments passed to the function in {{wildcard()}}

* Error adding mapping for "myservice.request.*" to "myservice.request.{{ xxxpartition(10,1) }}.{{wildcard(1)}}" : invalid mapping destination: unknown function in {{ xxxpartition(10,1) }}

* Error adding mapping for "myservice.request.*" to "myservice. request.{{ xxxpartition(10,1) }}.{{wildcard(1)}}" : invalid mapping destination: invalid subject

* implement PR comments
2022-06-30 14:21:53 -07:00
Matthias Hanel
6e52608936 [added] support for jwt account option DisallowBearer (#3127)
* [added] support for jwt account option DisallowBearer

change 3 out of 3. Fixes #3084
corresponds to:
https://github.com/nats-io/jwt/pull/177
https://github.com/nats-io/nsc/pull/495

update jwt library to 2.3.0

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-06-29 18:19:14 +02:00
Matthias Hanel
aba1da090b [ADD] account specific in/out msgs/bytes stats to CONNS (#3187)
* [ADD] account specific in/out msgs/bytes stats to CONNS

This subject $SYS.ACCOUNT.%s.SERVER.CONNS will now respond with account
specific datastats for Received and sent messages as well as number of slow
consumers for the account.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-06-28 18:59:29 +02:00
Derek Collison
830d659aec Do not default to sharing all client info for JetStream, only server and cluster info.
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-26 07:11:02 -07:00
R.I.Pienaar
52a1c542f5 export the correct subject transformer subject
While the TransformSubject function was doing the right
thing it did not match first and so would panic for subjects
that do not match the mapping.

The map function does the right thing so this is a more
appropriate function to export.

This undoes the exporting of unsafe TransformSubject and
exports the safer Match instead.

Signed-off-by: R.I.Pienaar <rip@devco.net>
2022-06-02 18:26:12 +02:00
R.I.Pienaar
dc9d6776f8 Export the subject transformer
This exports the one key function of the subject transformer
allowing external tools to be written to test mappings are
valid and see how they would interact without the hassle of
configuring a serrver

The APIs are specifically marked as being unsupported and
having kept the transform struct itself unexported one can
not cast from the interface to the real implementation

Signed-off-by: R.I.Pienaar <rip@devco.net>
2022-05-27 10:33:59 +02:00
Ivan Kozlovic
3cdbba16cb Revert "[added] support for jwt operator option DisallowBearerToken" 2022-05-04 11:11:25 -06:00
Matthias Hanel
bd2883122e [added] support for jwt operator option DisallowBearerToken
I modified an existing data structure that held a similar attribute already.
Instead this data structure references the claim.

change 3 out of 3. Fixes #3084
corresponds to:
https://github.com/nats-io/jwt/pull/177
https://github.com/nats-io/nsc/pull/495

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-29 14:18:11 -04:00
Matthias Hanel
7752a5becc Fixed mixed mode server without JS dropping js export on jwt update (#3044)
* Fixed mixed mode server without JS dropping JS export on jwt update

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-16 15:09:36 -04:00
Matthias Hanel
d9da66d67e returns -1 for new unlimited/unset limits and tests/fixes info counts (#3002)
iterates on tiered limits

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-05 12:25:55 -04:00
Matthias Hanel
92f4dc986a added max_ack_pending setting to js account limits (#2982)
* added max_ack_penind setting to js account limits

because of the addition, defaults now have to be set later (depend on
these new limits now)

also re-organized the code to closer track how stream create looks

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-03-31 14:17:16 -04:00
Ivan Kozlovic
c0ab2d4959 [FIXED] Possible panic due to data races
A panic was reported that looked like this:
```
fatal error: concurrent map read and map write
goroutine 200 [running]:
runtime.throw({0xa366ce, 0xe620e0})
	/home/travis/.gimme/versions/go1.17.8.linux.amd64/src/runtime/panic.go:1198 +0x71 fp=0xc00105f098 sp=0xc00105f068 pc=0x434ff1
runtime.mapaccess1_faststr(0x0, 0x0, {0xc0054b6f18, 0x11})
	/home/travis/.gimme/versions/go1.17.8.linux.amd64/src/runtime/map_faststr.go:21 +0x3a5 fp=0xc00105f100 sp=0xc00105f098 pc=0x412285"
github.com/nats-io/nats-server/v2/server.(*consumer).processNextMsgReq(0xc000681000, 0xc00105f2a8, 0x4503e9, 0x11, {0x0, 0xc000246900}, {0xc0054b6f18, 0x11}, {0xc0002469c4, 0x90, ...})
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/consumer.go:2454 +0x8ce fp=0xc00105f250 sp=0xc00105f100 pc=0x77dc2e
github.com/nats-io/nats-server/v2/server.(*consumer).processNextMsgReq-fm(0x9c, 0x7f302e954fff, 0xc00105f2f8, {0xc000774280, 0x400}, {0xc0054b6f18, 0x40}, {0xc0002469c4, 0x90, 0x63c})
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/consumer.go:2380 +0x77 fp=0xc00105f2b8 sp=0xc00105f250 pc=x91e337
github.com/nats-io/nats-server/v2/server.(*client).deliverMsg(0xc0015f8000, 0xc003034f00, 0x41642f, {0xc000246969, 0x4b6166, 0x697}, {0xc0002469a9, 0x4b60be, 0x657}, {0xc0015f9480, ...}, ...)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:3180 +0xbb0 fp=0xc00105f530 sp=0xc00105f2b8 pc=0x764470
github.com/nats-io/nats-server/v2/server.(*client).processMsgResults(0xc0015f8000, 0x8cd7a5, 0xc0089fb440, {0xc0002469c4, 0x92, 0x63c}, {0x0, 0x0, 0x4}, {0xc000246969, ...}, ...)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:4163 +0x9af fp=0xc00105fa48 sp=0xc00105f530 pc=0x769e4f
github.com/nats-io/nats-server/v2/server.(*client).processInboundRoutedMsg(0xc0015f8000, {0xc0002469c4, 0xc0015f8220, 0x63c})
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/route.go:443 +0x159 fp=0xc00105fae8 sp=0xc00105fa48 pc=0x8ce299
github.com/nats-io/nats-server/v2/server.(*client).processInboundMsg(0xc0015f8000, {0xc0002469c4, 0x92, 0x79e})
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:3493 +0x36 fp=0xc00105fb18 sp=0xc00105fae8 pc=0x765c76
github.com/nats-io/nats-server/v2/server.(*client).parse(0xc0015f8000, {0xc000246800, 0x800, 0xc087258a5d30c937})
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/parser.go:497 +0x246a fp=0xc00105fd98 sp=0xc00105fb18 pc=0x8a4f6a
github.com/nats-io/nats-server/v2/server.(*client).readLoop(0xc0015f8000, {0x0, 0x0, 0x0})"
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:1227 +0xe1f fp=0xc00105ffb0 sp=0xc00105fd98 pc=0x75841f
github.com/nats-io/nats-server/v2/server.(*Server).createRoute.func1()
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/route.go:1372 +0x25 fp=0xc00105ffe0 sp=0xc00105ffb0 pc=0x8d46a5
runtime.goexit
```

Writting a test showed the data race:
```
==================
WARNING: DATA RACE
Read at 0x00c0008ea240 by goroutine 62:
  runtime.mapaccess1_faststr()
      /usr/local/go/src/runtime/map_faststr.go:12 +0x0
  github.com/nats-io/nats-server/v2/server.(*consumer).processNextMsgRequest()
      /Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/consumer.go:2567 +0xa64
(...)
Previous write at 0x00c0008ea240 by goroutine 15:
  runtime.mapdelete_faststr()
      /usr/local/go/src/runtime/map_faststr.go:300 +0x0
  github.com/nats-io/nats-server/v2/server.(*Account).checkForReverseEntry()
      /Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/accounts.go:1759 +0x61c
  github.com/nats-io/nats-server/v2/server.(*client).unsubscribe()
      /Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/client.go:2838 +0xa27
(...)
```

After fixing this data race, another showed up:
```
==================
WARNING: DATA RACE
Read at 0x00c000352200 by goroutine 99:
  github.com/nats-io/nats-server/v2/server.(*Account).checkForReverseEntry()
      /Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/accounts.go:1752 +0x4b3
  github.com/nats-io/nats-server/v2/server.(*client).unsubscribe()
      /Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/client.go:2838 +0xa27
(...)
Previous write at 0x00c000352200 by goroutine 92:
  runtime.slicecopy()
      /usr/local/go/src/runtime/slice.go:284 +0x0
  github.com/nats-io/nats-server/v2/server.(*Account).checkForReverseEntry()
      /Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/accounts.go:1737 +0x871
  github.com/nats-io/nats-server/v2/server.(*Account).removeRespServiceImport()
      /Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/accounts.go:1622 +0x24c
(...)
```

This PR addresses both.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-03-30 13:51:52 -06:00
Matthias Hanel
1445153130 Adding max stream bytes check (#2970)
* Adding max stream bytes check

Also start checking on  stream update

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-03-30 15:50:28 -04:00
Matthias Hanel
0c5f3688a7 [ADDED] Tiered limits and fix limit issues on updates (#2945)
* Adding tiered limits and fix limit issues on updates

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-03-28 20:47:54 -04:00
Derek Collison
7e4a4c8fdd Merge pull request #2890 from nats-io/jnm/partition_mapping
[ADDED] deterministic subject tokens to partition mapping
2022-03-25 11:30:24 -07:00
jnmoyne
6818a232f7 improve mapping template regex to allow for some extra spaces and associated test upgrade 2022-03-10 11:30:51 -08:00
Ivan Kozlovic
0fae8067ae [FIXED] Some lock inversions
The established ordering is client -> Account, so fixed few places
where we had Account -> client.

Added a new file, locksordering.txt with the list of known ordering
for some of the objects.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-03-09 09:47:37 -07:00
jnmoyne
1d37629d95 source cleanup 2022-03-04 15:46:15 -08:00
jnmoyne
2138c96cdd Optimizations, cleanups and new mapping destination error 2022-03-04 14:10:18 -08:00
jnmoyne
ceb47f0298 Change mapping function name being passed as a string to being passed as a 'constant' regex and small simplification because the hash's Write never returns an error 2022-03-02 09:32:29 -08:00
jnmoyne
7fa088c804 Adds deterministic subject tokens to partition mapping
introduces 'Moustache' style subject mapping format (e.g. foo.*.* -> foo.{{wildcard(1)}}.{{wildcard(2)}}.{{partition(10,1,2)}})
2022-03-01 17:04:49 -08:00
jnmoyne
130b254699 Revert "Initial commit"
This reverts commit b10388c979.
2022-02-21 15:06:40 -08:00
jnmoyne
b10388c979 Initial commit 2022-02-19 23:57:30 -08:00
Matthias Hanel
fa12d096cd Fix jwt based user/activation token revocation and revocation granularity
user and activation token did not honor the jwt value for all * on
connect.

activation token where not re evaluated when the export revoked a key.
In part this is a consistency measure so servers that already have an
account and servers that don't behave the same way.

in jwt activation token revocations are stored per export.
The server stored them per account, thus effectively merging
revocations. Now they are stored per export inside the server too.

fixes nats-io/nsc/issues/442

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-01-25 13:48:12 -05:00