Commit Graph

704 Commits

Author SHA1 Message Date
Derek Collison
ff8701b724 Merge branch 'main' into dev 2023-04-06 08:37:11 -07:00
Derek Collison
2da50512e2 Optimize non-inline direct gets to not use simple go routines
Signed-off-by: Derek Collison <derek@nats.io>
2023-04-06 07:50:57 -07:00
Sandy Kellagher
5ae83b7983 Alternative fix for issue 4014, where we always send pings on ROUTER connections, updating c.lastIn timestamp on receiving client data or ping 2023-04-05 13:03:46 +01:00
Ivan Kozlovic
bd1b7b8d55 Cleanup use of s.opts and fixed some lock (deadlock/inversion) issues
One should not access s.opts directly but instead use s.getOpts().
Also, server lock needs to be released when performing an account
lookup (since this may result in server lock being acquired).
A function was calling s.LookupAccount under the client lock, which
technically creates a lock inversion situation.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2023-04-03 09:32:28 -06:00
Ivan Kozlovic
105237cba8 [ADDED] Multiple routes and ability to have per-account routes
New configuration fields:
```
cluster {
   ...
   pool_size: 5
   accounts: ["A", "B"]
}
```

The configuration `pool_size` in the example above means that this
server will create 5 routes to a remote server, assuming that that
server has the same `pool_size` setting.

Accounts (which are not part of the `accounts[]` configuration)
are assigned a specific route in this pool, and this will be the
same route on all servers in the cluster.

Accounts that are defined in the `accounts` field will each have
a dedicated route connection. This will allow suppression of the
account name in some of the route protocols, reducing bytes transmitted
which may increase performance.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2023-04-03 09:32:25 -06:00
Neil Twigg
f2bffec366 Refactor outbound queues, remove dynamic sizing, add buffer reuse
Also try to reduce flakiness of `TestClusterQueueSubs` and `TestCrossAccountServiceResponseTypes`
2023-03-15 09:37:40 +00:00
Waldemar Quevedo
beb179ec15 Check if connection name was already set when storing it
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-02-19 07:58:56 -08:00
Waldemar Quevedo
c6e263f910 Check if connection name was already set when storing it
Signed-off-by: Waldemar Quevedo <wally@nats.io>
2023-01-27 11:04:38 -08:00
Jean-Noël Moyne
a953e84437 Add stream input subject transform functionality
Extract subject transformation code out of accounts.go
Stream sources can now have a subject mapping transform
You can source the same stream more than once
Remove limitation that the subject filter for a source, mirror or consumer must have an overlap with the sourced/mirrored's stream or the stream's subjects
2023-01-25 12:25:51 -08:00
Derek Collison
235ce2caed Merge branch 'main' into dev 2023-01-11 18:34:01 -08:00
peaaceChoi
038037381b Fix some typos in code comment 2023-01-12 10:31:32 +09:00
Neil Twigg
e4b6ba2f23 Refactor outbound queues, remove dynamic sizing, add buffer reuse
Also try to reduce flakiness of `TestClusterQueueSubs` and `TestCrossAccountServiceResponseTypes`
2023-01-09 09:35:22 +00:00
Derek Collison
2daf90493b Authentication and Authorization callouts for server configuration mode.
This adds the ability to augment or override the NATS auth system.

A server will send a signed request to $SYS.REQ.USER.AUTH on the specified account. The request will contain client information, all client options sent to the server, and optionally TLS information and client certificates.
The external auth service will respond with an empty message if not authorized, or a signed User JWT that the user will bind to.

The response can change the account the client will be bound to.

Signed-off-by: Derek Collison <derek@nats.io>
2022-12-28 10:32:45 -08:00
Derek Collison
89b5f8319f Merge branch 'main' into dev 2022-12-07 13:01:06 -08:00
Sandy Kellagher
7907950a54 Fix for #3682: do not delay PINGs for GATEWAY or solicited LEAF connections, to ensure failover of leaf node connections 2022-12-07 16:19:33 +00:00
R.I.Pienaar
e6c78f1a59 allow expiration timer logic to be called unlocked
This ensures the logic that triggers user disconnections
share the same code paths and sets the same variables

Signed-off-by: R.I.Pienaar <rip@devco.net>
2022-12-02 15:38:54 +01:00
Derek Collison
8365fb3ef4 Add in expiration to user info.
This is only added if set by a user or account expiration claim.
It is represented as a duration til expiration vs absolute time which would involve time zone and clock sync issues.

Signed-off-by: Derek Collison <derek@nats.io>
2022-11-28 09:20:14 -08:00
Derek Collison
d2f1b04d34 Add in user info requests to have connected users get info for bound account and permissions.
Signed-off-by: Derek Collison <derek@nats.io>
2022-11-27 18:16:02 -08:00
Derek Collison
be73018f66 Merge branch 'main' into dev 2022-11-17 07:27:53 -08:00
Derek Collison
fe57c687f6 Make sure header keys do not have additional prefixes.
Signed-off-by: Derek Collison <derek@nats.io>
2022-11-17 06:52:26 -08:00
R.I.Pienaar
3bb473c67d adds the notion of a connection deadline to User
This will be used mainly by CustomClientAuthentication implementations
to indicate that the user connection should be disconnected at some
point in future - like when a certificate or token expires

Signed-off-by: R.I.Pienaar <rip@devco.net>
2022-10-27 12:57:30 +02:00
Ivan Kozlovic
6113c52ae1 [FIXED] Solicited route may not retry to reconnect
Originally, only solicited routes were retried in case of a disconnect,
but that was before gossip protocol was introduced. Since then, two
servers that connect to each other due to gossip should retry to
reconnect if the connection breaks, even if the route is not explicit.
However, server will retry only once or more accurately, ConnectRetries+1.

This PR solves the issue that the reconnect attempt was not initiated
for a "solicited route" that was not explicit.

Maybe related to #3571

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-10-24 10:39:23 -06:00
Ivan Kozlovic
dde94987ce [FIXED] MQTT: Subjects mapping were not handled
A simple configuration like this:
```
...
mappings = {
  foo: bar
}

mqtt {
   port: 1883
}
```
would cause an MQTT subscription on "bar" to not receive messages
published on "foo".

In otherwords, the subject transformation was not done when parsing
a PUBLISH packet.

This PR also handles the case of service imports where transformation
occurs after the initial publish parsing.

Resolves #3547

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-10-13 16:00:05 -06:00
Ivan Kozlovic
3358247e6b Added warning if internal sub callback takes too long
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-10-10 14:39:37 -06:00
Ivan Kozlovic
c3adf78702 [FIXED] Stack overflow when account imports its own export
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-10-10 13:20:26 -06:00
Ivan Kozlovic
8d9c57ad44 [IMPROVED] Fan-out performance
There was an observed degradation (around 5%) for large fan out in
v2.9.0 compared to earlier release. This is because we added
accounting of the in/out messages for the account, which result
in 4 atomic operations, 2 for in and 2 for out, however, it means
that for a fan-out of say 100 matching subscriptions, it is now
2 + 2 * 100 = 202.

This PR rework how the stats accounting is done which removes
the regression and even boost a bit the numbers since we are
doing the server stats update as an aggregate too.

There are still degradation for queues and no-sub at all that
need to be looked at.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-09-27 19:43:32 -06:00
Caleb Lloyd
3babdda3bb [FIXED] Format protocol error []byte with %q
Protocol errors print arguments that contain arbitrary []byte
and are possibly not formattable strings; use %q to escape

Signed-off-by: Caleb Lloyd <caleb@synadia.com>
2022-09-26 13:52:56 -04:00
Ivan Kozlovic
a8318d1f62 [FIXED] Service import/export cycles causing stack overflow
There was a way to detect a cycle but I believe it needs to be
a stack of "si" not just the one before invoking processServiceImport.

Changes in #3393 would solve issue reported with test TestAccountImportCycle,
but would not address the new reported issue represented by new test
TestLeafNodeSvcImportExportCycle. This current approach seems to solve
all known cases.

Resolves #3397
Replaces #3393
2022-08-26 14:40:02 -06:00
Derek Collison
5690059dac Reserve a system queue group
Signed-off-by: Derek Collison <derek@nats.io>
2022-07-06 13:16:13 -07:00
Derek Collison
e6479dafd2 Close leafnode connection when same cluster name detected
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-30 15:34:22 -07:00
Derek Collison
abc5905aa9 Merge pull request #3221 from nats-io/direct
Made direct get from a stream part of the $JS.API hierarchy vs separate.
2022-06-28 09:59:44 -07:00
Matthias Hanel
aba1da090b [ADD] account specific in/out msgs/bytes stats to CONNS (#3187)
* [ADD] account specific in/out msgs/bytes stats to CONNS

This subject $SYS.ACCOUNT.%s.SERVER.CONNS will now respond with account
specific datastats for Received and sent messages as well as number of slow
consumers for the account.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-06-28 18:59:29 +02:00
Derek Collison
b8ef9b19a0 Made direct get from a stream part of the $JS.API hierarchy vs separate.
Also for direct get and for pull requests, if we are not on a client connection check how long we have been away from the readloop.
If need be execute in a separate go routine.

Signed-off-by: Derek Collison <derek@nats.io>
2022-06-28 08:53:48 -07:00
Derek Collison
830d659aec Do not default to sharing all client info for JetStream, only server and cluster info.
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-26 07:11:02 -07:00
Derek Collison
af43dd3c74 When internal system messages were destined for a queue subscriber across a route the reply subject would be empty but not nil which caused mangling of the RMSG proto.
Signed-off-by: Derek Collison <derek@nats.io>
2022-06-11 14:12:05 -07:00
R.I.Pienaar
52a1c542f5 export the correct subject transformer subject
While the TransformSubject function was doing the right
thing it did not match first and so would panic for subjects
that do not match the mapping.

The map function does the right thing so this is a more
appropriate function to export.

This undoes the exporting of unsafe TransformSubject and
exports the safer Match instead.

Signed-off-by: R.I.Pienaar <rip@devco.net>
2022-06-02 18:26:12 +02:00
R.I.Pienaar
dc9d6776f8 Export the subject transformer
This exports the one key function of the subject transformer
allowing external tools to be written to test mappings are
valid and see how they would interact without the hassle of
configuring a serrver

The APIs are specifically marked as being unsupported and
having kept the transform struct itself unexported one can
not cast from the interface to the real implementation

Signed-off-by: R.I.Pienaar <rip@devco.net>
2022-05-27 10:33:59 +02:00
Ivan Kozlovic
c4adf0ffed [FIXED] Accounts Export/Import isolation with overlap subjects
I tracked down this issue to have been introduced with PR #2369,
but the code also touched PR #1891 and PR #3088.

I added a test as described in issue #3108 but did not need
JetStream to demonstrate the issue. With the proposed fix, all
tests that were added in aforementioned PRs still pass, including
the new test.

Resolves #3108

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-05-09 12:59:12 -06:00
Derek Collison
806877ebaa Merge pull request #3090 from nats-io/qsub-deny
Combined canSubscribe and canQueueSubscribe
2022-04-29 09:20:57 -07:00
Derek Collison
c20b52251b Combined canSubscribe and canQueueSubscribe for consistency in specialized deny clause handling.
Signed-off-by: Derek Collison <derek@nats.io>
2022-04-29 09:18:45 -07:00
Derek Collison
e0f5fcffb8 Fix for subject transforms and JetStream delivery subjects.
Signed-off-by: Derek Collison <derek@nats.io>
2022-04-28 15:50:28 -07:00
Matthias Hanel
254c970876 Fix subject renaming for leaf connections and queue subs (#3062)
* [fix] on queue sub, a consumers  delivery subject, was not changed

to the original publish subject the stream received
the code added is a copy of what regular subs do

* [fixed] subject renaming for leaf node connections as well

also updated multi server test to test for queue and non queue scenarios

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-20 19:23:21 -04:00
Matthias Hanel
0f113aa3d5 [FIXED] subject renaming with hand crafted reply subject (#3026)
do so by rejecting jsackprefix in reply subjects

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-11 22:32:02 -04:00
Matthias Hanel
02d25cc640 [FIXED] Consumer deliver subject incorrect when imported and crossing gateway (#3025)
followup to #3017

Signed-off-by: Matthias Hanel <mh@synadia.com>
2022-04-11 20:27:25 -04:00
Derek Collison
cd7f16f28a Tweak timing for test to prevent flapping
Signed-off-by: Derek Collison <derek@nats.io>
2022-04-09 11:13:49 -07:00
Derek Collison
331c2faaa6 When using a stream import for a push consumer's messages, if the message crossed a route we dropped the delivered subject.
Signed-off-by: Derek Collison <derek@nats.io>
2022-04-09 06:42:22 -07:00
Ivan Kozlovic
7fa2676353 Fixed comment typos and some rewording
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-07 09:22:51 -06:00
Ivan Kozlovic
9e6f965913 [ADDED] LeafNode min_version new option
If set, a server configured to accept leafnode connections will
reject a remote server whose version is below that value. Note
that servers prior to v2.8.0 are not sending their version
in the CONNECT protocol, which means that anything below 2.8.0
would be rejected.

Configuration example:
```
leafnodes {
    port: 7422
    min_version: 2.8.0
}
```
The option is a string and can have the "v" prefix:
```
min_version: "v2.9.1"
```
Note that although suffix such as `-beta` would be accepted,
only the major, minor and update are used for the version comparison.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-06 18:40:33 -06:00
Ivan Kozlovic
366d217f44 Some changes based on review
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-01 17:55:33 -06:00
Ivan Kozlovic
19783a9f11 [CHANGED] Rate limit similar warnings
Some warnings, especially when dealing with JS limits that were
printed on a per-message basis, are now limited to ~1 per second
if the content of the warning is already found in a map.

This is also for "client" warnings, but the client porting of the
warning is not taken into account so that helps with reducing logging
for similar content, but coming from different clients.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-04-01 15:24:03 -06:00