Commit Graph

256 Commits

Author SHA1 Message Date
Jean-Noël Moyne
2a709aaf61 - Changes to make adding new mapping functions easier (#3305)
* - Changes to make adding new mapping functions easier
- Adds new subject mapping functions:
{{SplitFromLeft(wildcard index, position)}}
{{SplitFromRight(wildcard index, position)}}
{{SliceFromLeft(wildcard index, slice size)}}
{{SliceFromRight(wildcard index, slice size)}}
{{Split(wildcard index, deliminator)}}

Examples:
	shouldMatch("*", "{{splitfromleft(1,3)}}", "12345", "123.45")
	shouldMatch("*", "{{SplitFromRight(1,3)}}", "12345", "12.345")
	shouldMatch("*", "{{SliceFromLeft(1,3)}}", "1234567890", "123.456.789.0")
	shouldMatch("*", "{{SliceFromRight(1,3)}}", "1234567890", "1.234.567.890")
	shouldMatch("*", "{{split(1,-)}}", "-abc-def--ghi-", "abc.def.ghi")
	shouldMatch("*.*", "{{split(2,-)}}.{{splitfromleft(1,2)}}", "foo.-abc-def--ghij-", "abc.def.ghij.fo.o")

- Subject mapping functions can now be all lower case or Pascal case (or a combination): e.g. splitfromleft, SplitFromLeft, splitFromleft, etc...
2022-08-18 09:52:28 -07:00
Ivan Kozlovic
3c9a7cc6e5 Move to Go 1.19, remote io/util, fix data race and a flapper
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-08-05 09:55:37 -06:00
Ivan Kozlovic
f19908979f Revert direct changes to main
This reverts commit cf784c19f0.
This reverts commit c269a1ca09.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-07-28 17:25:37 -06:00
jnmoyne
cf784c19f0 - Adds new subject mapping functions:
{{SplitFromLeft(wildcard index, position)}}
{{SplitFromRight(wildcard index, position)}}
{{SliceFromLeft(wildcard index, slice size)}}
{{SliceFromRight(wildcard index, slice size)}}
{{Split(wildcard index, deliminator)}}

Examples:
	shouldMatch("*", "{{splitfromleft(1,3)}}", "12345", "123.45")
	shouldMatch("*", "{{SplitFromRight(1,3)}}", "12345", "12.345")
	shouldMatch("*", "{{SliceFromLeft(1,3)}}", "1234567890", "123.456.789.0")
	shouldMatch("*", "{{SliceFromRight(1,3)}}", "1234567890", "1.234.567.890")
	shouldMatch("*", "{{split(1,-)}}", "-abc-def--ghi-", "abc.def.ghi")
	shouldMatch("*.*", "{{split(2,-)}}.{{splitfromleft(1,2)}}", "foo.-abc-def--ghij-", "abc.def.ghij.fo.o")

- Subject mapping functions can now be all lower case or Pascal case (or a combination): e.g. splitfromleft, SplitFromLeft, splitFromleft, etc...
2022-07-28 12:53:14 -07:00
jnmoyne
c269a1ca09 New expendable implementation of subject mapping destinations to transform processing 2022-07-28 12:30:22 -07:00
Todd Beets
6de64667bc follow convention (_EMPTY_, fwcs). Add special case short-circuit to skip unnecessary tokenization and transform. 2022-07-18 14:20:10 -07:00
Todd Beets
7a47631051 optimize for > or empty src (any subject) 2022-07-17 13:45:05 -07:00
Jean-Noël Moyne
23a9036b9b Improves error handling for subject mapping destination mustache functions (#3255) 2022-07-11 14:56:34 -07:00
Jean-Noël Moyne
e46b00639a Resolves #3151 plus redo and improve subject mapping destination validation and error handling (#3231)
* 1: Allows spaces to be used inside {{}} subject mapping functions:

2: Rework and improve mapping destinations validation and error handling with much more helpful error messages, e.g.:

* Error adding mapping for "foo.*.*" to "bar.{{wildcard(1)}}" : invalid mapping destination: not using all of the token wildcard(s) in bar.{{wildcard(1)}}

* Error adding mapping for "myservice.request.*" to "myservice.request.{{ partition(10) }}.{{wildcard(1)}}" : invalid mapping destination: not enough arguments passed to the function in {{ partition(10) }}

* Error adding mapping for "myservice.request.*" to "myservice.request.{{ partition(10,2) }}.{{wildcard(1)}}" : invalid mapping destination: wildcard index out of range in {{ partition(10,2) }}: [2]

* Error adding mapping for "myservice.request.*" to "myservice.request.{{ partition(10,1) }}.{{wildcard()}}" : invalid mapping destination: not enough arguments passed to the function in {{wildcard()}}

* Error adding mapping for "myservice.request.*" to "myservice.request.{{ xxxpartition(10,1) }}.{{wildcard(1)}}" : invalid mapping destination: unknown function in {{ xxxpartition(10,1) }}

* Error adding mapping for "myservice.request.*" to "myservice. request.{{ xxxpartition(10,1) }}.{{wildcard(1)}}" : invalid mapping destination: invalid subject

* implement PR comments
2022-06-30 14:21:53 -07:00
Matthias Hanel
6e52608936 [added] support for jwt account option DisallowBearer (#3127)
* [added] support for jwt account option DisallowBearer

change 3 out of 3. Fixes #3084
corresponds to:
https://github.com/nats-io/jwt/pull/177
https://github.com/nats-io/nsc/pull/495

update jwt library to 2.3.0

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-06-29 18:19:14 +02:00
Matthias Hanel
aba1da090b [ADD] account specific in/out msgs/bytes stats to CONNS (#3187)
* [ADD] account specific in/out msgs/bytes stats to CONNS

This subject $SYS.ACCOUNT.%s.SERVER.CONNS will now respond with account
specific datastats for Received and sent messages as well as number of slow
consumers for the account.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-06-28 18:59:29 +02:00
Derek Collison
830d659aec Do not default to sharing all client info for JetStream, only server and cluster info.
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-26 07:11:02 -07:00
R.I.Pienaar
52a1c542f5 export the correct subject transformer subject
While the TransformSubject function was doing the right
thing it did not match first and so would panic for subjects
that do not match the mapping.

The map function does the right thing so this is a more
appropriate function to export.

This undoes the exporting of unsafe TransformSubject and
exports the safer Match instead.

Signed-off-by: R.I.Pienaar <rip@devco.net>
2022-06-02 18:26:12 +02:00
R.I.Pienaar
dc9d6776f8 Export the subject transformer
This exports the one key function of the subject transformer
allowing external tools to be written to test mappings are
valid and see how they would interact without the hassle of
configuring a serrver

The APIs are specifically marked as being unsupported and
having kept the transform struct itself unexported one can
not cast from the interface to the real implementation

Signed-off-by: R.I.Pienaar <rip@devco.net>
2022-05-27 10:33:59 +02:00
Ivan Kozlovic
3cdbba16cb Revert "[added] support for jwt operator option DisallowBearerToken" 2022-05-04 11:11:25 -06:00
Matthias Hanel
bd2883122e [added] support for jwt operator option DisallowBearerToken
I modified an existing data structure that held a similar attribute already.
Instead this data structure references the claim.

change 3 out of 3. Fixes #3084
corresponds to:
https://github.com/nats-io/jwt/pull/177
https://github.com/nats-io/nsc/pull/495

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-29 14:18:11 -04:00
Matthias Hanel
7752a5becc Fixed mixed mode server without JS dropping js export on jwt update (#3044)
* Fixed mixed mode server without JS dropping JS export on jwt update

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-16 15:09:36 -04:00
Matthias Hanel
d9da66d67e returns -1 for new unlimited/unset limits and tests/fixes info counts (#3002)
iterates on tiered limits

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-05 12:25:55 -04:00
Matthias Hanel
92f4dc986a added max_ack_pending setting to js account limits (#2982)
* added max_ack_penind setting to js account limits

because of the addition, defaults now have to be set later (depend on
these new limits now)

also re-organized the code to closer track how stream create looks

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-03-31 14:17:16 -04:00
Ivan Kozlovic
c0ab2d4959 [FIXED] Possible panic due to data races
A panic was reported that looked like this:
```
fatal error: concurrent map read and map write
goroutine 200 [running]:
runtime.throw({0xa366ce, 0xe620e0})
	/home/travis/.gimme/versions/go1.17.8.linux.amd64/src/runtime/panic.go:1198 +0x71 fp=0xc00105f098 sp=0xc00105f068 pc=0x434ff1
runtime.mapaccess1_faststr(0x0, 0x0, {0xc0054b6f18, 0x11})
	/home/travis/.gimme/versions/go1.17.8.linux.amd64/src/runtime/map_faststr.go:21 +0x3a5 fp=0xc00105f100 sp=0xc00105f098 pc=0x412285"
github.com/nats-io/nats-server/v2/server.(*consumer).processNextMsgReq(0xc000681000, 0xc00105f2a8, 0x4503e9, 0x11, {0x0, 0xc000246900}, {0xc0054b6f18, 0x11}, {0xc0002469c4, 0x90, ...})
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/consumer.go:2454 +0x8ce fp=0xc00105f250 sp=0xc00105f100 pc=0x77dc2e
github.com/nats-io/nats-server/v2/server.(*consumer).processNextMsgReq-fm(0x9c, 0x7f302e954fff, 0xc00105f2f8, {0xc000774280, 0x400}, {0xc0054b6f18, 0x40}, {0xc0002469c4, 0x90, 0x63c})
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/consumer.go:2380 +0x77 fp=0xc00105f2b8 sp=0xc00105f250 pc=x91e337
github.com/nats-io/nats-server/v2/server.(*client).deliverMsg(0xc0015f8000, 0xc003034f00, 0x41642f, {0xc000246969, 0x4b6166, 0x697}, {0xc0002469a9, 0x4b60be, 0x657}, {0xc0015f9480, ...}, ...)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:3180 +0xbb0 fp=0xc00105f530 sp=0xc00105f2b8 pc=0x764470
github.com/nats-io/nats-server/v2/server.(*client).processMsgResults(0xc0015f8000, 0x8cd7a5, 0xc0089fb440, {0xc0002469c4, 0x92, 0x63c}, {0x0, 0x0, 0x4}, {0xc000246969, ...}, ...)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:4163 +0x9af fp=0xc00105fa48 sp=0xc00105f530 pc=0x769e4f
github.com/nats-io/nats-server/v2/server.(*client).processInboundRoutedMsg(0xc0015f8000, {0xc0002469c4, 0xc0015f8220, 0x63c})
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/route.go:443 +0x159 fp=0xc00105fae8 sp=0xc00105fa48 pc=0x8ce299
github.com/nats-io/nats-server/v2/server.(*client).processInboundMsg(0xc0015f8000, {0xc0002469c4, 0x92, 0x79e})
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:3493 +0x36 fp=0xc00105fb18 sp=0xc00105fae8 pc=0x765c76
github.com/nats-io/nats-server/v2/server.(*client).parse(0xc0015f8000, {0xc000246800, 0x800, 0xc087258a5d30c937})
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/parser.go:497 +0x246a fp=0xc00105fd98 sp=0xc00105fb18 pc=0x8a4f6a
github.com/nats-io/nats-server/v2/server.(*client).readLoop(0xc0015f8000, {0x0, 0x0, 0x0})"
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:1227 +0xe1f fp=0xc00105ffb0 sp=0xc00105fd98 pc=0x75841f
github.com/nats-io/nats-server/v2/server.(*Server).createRoute.func1()
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/route.go:1372 +0x25 fp=0xc00105ffe0 sp=0xc00105ffb0 pc=0x8d46a5
runtime.goexit
```

Writting a test showed the data race:
```
==================
WARNING: DATA RACE
Read at 0x00c0008ea240 by goroutine 62:
  runtime.mapaccess1_faststr()
      /usr/local/go/src/runtime/map_faststr.go:12 +0x0
  github.com/nats-io/nats-server/v2/server.(*consumer).processNextMsgRequest()
      /Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/consumer.go:2567 +0xa64
(...)
Previous write at 0x00c0008ea240 by goroutine 15:
  runtime.mapdelete_faststr()
      /usr/local/go/src/runtime/map_faststr.go:300 +0x0
  github.com/nats-io/nats-server/v2/server.(*Account).checkForReverseEntry()
      /Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/accounts.go:1759 +0x61c
  github.com/nats-io/nats-server/v2/server.(*client).unsubscribe()
      /Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/client.go:2838 +0xa27
(...)
```

After fixing this data race, another showed up:
```
==================
WARNING: DATA RACE
Read at 0x00c000352200 by goroutine 99:
  github.com/nats-io/nats-server/v2/server.(*Account).checkForReverseEntry()
      /Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/accounts.go:1752 +0x4b3
  github.com/nats-io/nats-server/v2/server.(*client).unsubscribe()
      /Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/client.go:2838 +0xa27
(...)
Previous write at 0x00c000352200 by goroutine 92:
  runtime.slicecopy()
      /usr/local/go/src/runtime/slice.go:284 +0x0
  github.com/nats-io/nats-server/v2/server.(*Account).checkForReverseEntry()
      /Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/accounts.go:1737 +0x871
  github.com/nats-io/nats-server/v2/server.(*Account).removeRespServiceImport()
      /Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/accounts.go:1622 +0x24c
(...)
```

This PR addresses both.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-03-30 13:51:52 -06:00
Matthias Hanel
1445153130 Adding max stream bytes check (#2970)
* Adding max stream bytes check

Also start checking on  stream update

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-03-30 15:50:28 -04:00
Matthias Hanel
0c5f3688a7 [ADDED] Tiered limits and fix limit issues on updates (#2945)
* Adding tiered limits and fix limit issues on updates

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-03-28 20:47:54 -04:00
Derek Collison
7e4a4c8fdd Merge pull request #2890 from nats-io/jnm/partition_mapping
[ADDED] deterministic subject tokens to partition mapping
2022-03-25 11:30:24 -07:00
jnmoyne
6818a232f7 improve mapping template regex to allow for some extra spaces and associated test upgrade 2022-03-10 11:30:51 -08:00
Ivan Kozlovic
0fae8067ae [FIXED] Some lock inversions
The established ordering is client -> Account, so fixed few places
where we had Account -> client.

Added a new file, locksordering.txt with the list of known ordering
for some of the objects.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-03-09 09:47:37 -07:00
jnmoyne
1d37629d95 source cleanup 2022-03-04 15:46:15 -08:00
jnmoyne
2138c96cdd Optimizations, cleanups and new mapping destination error 2022-03-04 14:10:18 -08:00
jnmoyne
ceb47f0298 Change mapping function name being passed as a string to being passed as a 'constant' regex and small simplification because the hash's Write never returns an error 2022-03-02 09:32:29 -08:00
jnmoyne
7fa088c804 Adds deterministic subject tokens to partition mapping
introduces 'Moustache' style subject mapping format (e.g. foo.*.* -> foo.{{wildcard(1)}}.{{wildcard(2)}}.{{partition(10,1,2)}})
2022-03-01 17:04:49 -08:00
jnmoyne
130b254699 Revert "Initial commit"
This reverts commit b10388c979.
2022-02-21 15:06:40 -08:00
jnmoyne
b10388c979 Initial commit 2022-02-19 23:57:30 -08:00
Matthias Hanel
fa12d096cd Fix jwt based user/activation token revocation and revocation granularity
user and activation token did not honor the jwt value for all * on
connect.

activation token where not re evaluated when the export revoked a key.
In part this is a consistency measure so servers that already have an
account and servers that don't behave the same way.

in jwt activation token revocations are stored per export.
The server stored them per account, thus effectively merging
revocations. Now they are stored per export inside the server too.

fixes nats-io/nsc/issues/442

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-01-25 13:48:12 -05:00
Ivan Kozlovic
f6acc9d28b [FIXED] Possible deadlock
This is due to a re-entrant RLock(). It works sometimes, but if there
is a go routine requesting the write lock, then the second RLock()
will not be granted which will lead to a deadlock.
In summary: one should never make re-entrant RLock calls.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-01-20 13:53:52 -07:00
Derek Collison
7dc99c3840 Merge from main
Signed-off-by: Derek Collison <derek@nats.io>
2022-01-13 10:01:33 -08:00
Matthias Hanel
78bbcd791f [Adding] support for JS MaxBytesRequired
Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-01-12 22:57:34 -05:00
Derek Collison
43eff407b8 Add in explicit subscription for import responses when bound to a leafnode.
When we want to track service import response interest across a leafnode we need to send sub and unsub for all response _R_ subjects versus using a wildcard.

Signed-off-by: Derek Collison <derek@nats.io>
2022-01-12 18:24:03 -08:00
Derek Collison
b43cb5b352 Added in ability to have account limits configured in server config.
Signed-off-by: Derek Collison <derek@nats.io>
2021-12-21 18:31:07 -08:00
Derek Collison
65d098f526 Fix for latency tracking bug.
The bug occures when latency tracking is on, a requestor and responder are not connected to the same server, and the responder sends two responses for a single request.

Signed-off-by: Derek Collison <derek@nats.io>
2021-11-12 17:37:15 -08:00
Phil Pennock
fc6df0fbbc Redact URLs before logging or returning in error (#2643)
* Redact URLs before logging or returning in error

This does not affect strings which failed to parse, and in such a scenario
there's a mix of "which evil" to accept; we can't sanely find what should be
redacted in those cases, so we leave them alone for debugging.

The JWT library returns some errors for Operator URLs, but it rejects URLs
which contain userinfo, so there can't be passwords in those and they're safe.

Fixes #2597

* Test the URL redaction auxiliary functions

* End-to-end tests for secrets in debug/trace

Create internal/testhelper and move DummyLogger there, so it can be used from
the test/ sub-dir too.

Let DummyLogger optionally accumulate all log messages, not just retain the
last-seen message.

Confirm no passwords logged by TestLeafNodeBasicAuthFailover.

Change TestNoPasswordsFromConnectTrace to check all trace messages, not just the
most recent.

Validate existing trace redaction in TestRouteToSelf.

* Test for password in solicited route reconnect debug
2021-10-27 12:44:59 -04:00
Derek Collison
15fb253c7c Revert back to old subject, don't suppress updates on non client/leaf
Signed-off-by: Derek Collison <derek@nats.io>
2021-09-23 16:07:35 -07:00
Derek Collison
3740a707f6 Do not count non client or leafnode connections for account limit reporting.
Signed-off-by: Derek Collison <derek@nats.io>
2021-09-23 15:43:07 -07:00
Ivan Kozlovic
4076abfd57 Use a better seed for a rand generator
This was causing some test to flap.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-08-26 15:16:22 -06:00
Derek Collison
02c702f7af Added test for system account update as well
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-17 11:09:13 -07:00
Derek Collison
84ff537e66 Make sure jwt claim update does not wipe system imports
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-17 10:03:30 -07:00
Derek Collison
10167b1bcf Added in ability for normal accounts to access scoped connz info.
Added in client kind and sub type for clients.
Added in ability to filter connections based on matching subject interest.

Signed-off-by: Derek Collison <derek@nats.io>
2021-08-13 10:19:12 -07:00
Derek Collison
f13fa767c2 Remove the swapping of accounts during processing of service imports.
When processing service imports we would swap out the accounts during processing.
With the addition of internal subscriptions and internal clients publishing in JetStream we had an issue with the wrong account being used.
This was specific to delyaed pull subscribers trying to unsubscribe due to max of 1 while other JetStream API calls were running concurrently.
2021-07-26 07:57:10 -07:00
Derek Collison
26e63a14a6 Merge pull request #2324 from nats-io/large-r1
Improvements to large numbers of JetStream R1 consumers per stream.
2021-06-29 08:35:17 -07:00
Derek Collison
3c16f776dc Protect account prand, use faster Uint64 variant
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-29 06:13:41 -07:00
Derek Collison
99fed910f0 Improvements to large numbers of JetStream R1 consumers per stream.
1. We were holding open FDs longer than we should for consumers causing issues with open FD limits. We now do not hold them open and cap updates a bit better.

2. When doing a stream delete, consumer delete was repeating alot of work that was not necessary, causing longer delays. This has been optimized a bit, still more improvements to be made.

3. We cover all JS under a single export, but that was also trapping GetNext for pull based consumers, and since this was a no-op (is handled at user account level) we were creating alot of garbage service import responses and reverse map entries that had to be garbage collected. We have a fix in to avoind this but still looking for a better one.

4. Still had some lingering references to all exports vs single JS export.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-29 05:45:55 -07:00
Ivan Kozlovic
1d3cddfa7c [CHANGED] Reduce print for an account subs limit to every 2 sec
We could make it for all limits by having a map of error types
instead of applying just to max subs.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-06-22 11:00:41 -06:00