Commit Graph

167 Commits

Author SHA1 Message Date
Derek Collison
d962500827 Track reply subjects for pending pull requests across clustered consumers.
We will only send if all peers in our group are >= 2.7.1 and we will check for updates.
When a consumer follower takes over it will notify all pending requests that those requests are invalid now.

Signed-off-by: Derek Collison <derek@nats.io>
2022-01-21 16:31:59 -08:00
Ivan Kozlovic
84f6cbb760 Pooling pubMsg and jsPubMsg objects
This should help with GC pressure, however, it may have an effect
on performance (based on some benchmark). Calling sync.Pool.Get/Put
too often has a performance impact...

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-01-13 13:14:25 -07:00
Ivan Kozlovic
92e8997506 Replaced system event queue
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2022-01-13 13:03:33 -07:00
Derek Collison
52da55c8c6 Implement overflow placement for JetStream streams.
This allows stream placement to overflow to adjacent clusters.
We also do more balanced placement based on resources (store or mem). We can continue to expand this as well.
We also introduce an account requirement that stream configs contain a MaxBytes value.

We now track account limits and server limits more distinctly, and do not reserver server resources based on account limits themselves.

Signed-off-by: Derek Collison <derek@nats.io>
2022-01-06 19:33:08 -08:00
Matthias Hanel
377a46eedd Merge pull request #2580 from nats-io/hdr-ping
[fixed] missing support for header in some request handler
2021-09-28 17:41:15 -04:00
Matthias Hanel
3379b3b612 [fixed] missing support for header in some request handler
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-09-28 14:28:03 -04:00
Derek Collison
8223275c44 On cold start in mixed mode if the js servers were not > non-js we could stall.
Signed-off-by: Derek Collison <derek@nats.io>
2021-09-27 16:59:42 -07:00
Derek Collison
15fb253c7c Revert back to old subject, don't suppress updates on non client/leaf
Signed-off-by: Derek Collison <derek@nats.io>
2021-09-23 16:07:35 -07:00
Derek Collison
3740a707f6 Do not count non client or leafnode connections for account limit reporting.
Signed-off-by: Derek Collison <derek@nats.io>
2021-09-23 15:43:07 -07:00
Derek Collison
d91a0061ed Fix for data race around c.echo
Signed-off-by: Derek Collison <derek@nats.io>
2021-09-16 11:59:52 -07:00
Derek Collison
cfbc69b12c Allow clustered JetStream to allow duplicate stream creation like single server mode.
Resolves #2528

Signed-off-by: Derek Collison <derek@nats.io>
2021-09-15 20:18:44 -07:00
Ivan Kozlovic
0411ba0c03 Changed ClientID to MQTTClient and client_id to mqtt_client
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-09-09 14:34:54 -06:00
Ivan Kozlovic
49024a0353 [ADDED] Monitoring: ClientID (for MQTT clients) on connection events
ClientID has been added to various monitoring objects. Also, added
the ability to filter connections on `client_id`.

On auth violation, the proper code was not invoked, which meant
that no disconnect event (with auth reason) would be published.

Resolves #2270

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-09-09 13:34:46 -06:00
Matthias Hanel
2e40bdd5bc [fixed] error print on re adding existing system import
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-08-24 14:29:44 -04:00
Derek Collison
3a20582ad5 Add in optional compression schemes for Accept-Encoding on server api requests.
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-23 13:06:18 -07:00
Derek Collison
84ff537e66 Make sure jwt claim update does not wipe system imports
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-17 10:03:30 -07:00
Derek Collison
944dd248c4 Fix for tests
Signed-off-by: Derek Collison <derek@nats.io>
2021-08-14 17:39:51 -07:00
Derek Collison
10167b1bcf Added in ability for normal accounts to access scoped connz info.
Added in client kind and sub type for clients.
Added in ability to filter connections based on matching subject interest.

Signed-off-by: Derek Collison <derek@nats.io>
2021-08-13 10:19:12 -07:00
Derek Collison
925a6fe6b2 Fix for #2388. Leafnodes with no JS can seamlessly access a HUB with JS.
This is the reverse of the early work to have LNs extend a non-JS cluster.
Also have mixed mode tests as well.

Signed-off-by: Derek Collison <derek@nats.io>
2021-08-01 14:57:47 -07:00
Matthias Hanel
a40ea298e5 [fixed] jetstream unique server name requirement across domains (#2378)
* [fixed] jetstream unique server name requirement across domains

including domain in server info
adding check for cluster name in duplicate leaf node connection check

This does not address non unique domains in the same domain, say within
super cluster.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-07-27 18:42:19 -04:00
Derek Collison
f13fa767c2 Remove the swapping of accounts during processing of service imports.
When processing service imports we would swap out the accounts during processing.
With the addition of internal subscriptions and internal clients publishing in JetStream we had an issue with the wrong account being used.
This was specific to delyaed pull subscribers trying to unsubscribe due to max of 1 while other JetStream API calls were running concurrently.
2021-07-26 07:57:10 -07:00
Matthias Hanel
c68ffe5ad5 [adding] kind and client_type to account connect/disconnect events (#2351)
* [adding] kind and client_type to client info. specifically account connect/disconnect events

Kind is Client/Leafnode but can take the value of Router/Gateway/JetStream/Account/System in the future.
When kind is Client, then client_type is set to mqtt/websocket/nats
This fixes #2291

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-07-07 17:43:50 -04:00
Matthias Hanel
fcb3382f4b [fixed] CONNS system endpoint which had issues with header and empty msg
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-07-06 16:46:36 -04:00
Matthias Hanel
a72ca8a9bf [fixed] header handling in system services
On export/import a header was inserted which broke parsing of the
message.
Fixed unit test broken by .beta in version

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-07-01 19:00:52 -04:00
Derek Collison
637973a1c7 fomatting
Signed-off-by: Derek Collison <derek@nats.io>
2021-06-10 10:04:41 -07:00
Derek Collison
4965a5c494 Merge branch 'master' into active_servers 2021-06-10 09:57:55 -07:00
Matthias Hanel
2caf2303f2 [adding] jetstream info to statsz (#2269)
* [adding] jetstream info to statsz

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-06-10 11:54:56 -04:00
Derek Collison
cc63915f54 Added active servers to statsz.
This is generally useful but will also help with cli actions since we know from first response how many total responses are expected.

Signed-off-by: Derek Collison <derek@nats.io>
2021-06-09 19:14:37 -07:00
Matthias Hanel
230128ed01 [added] filter system requests by JS domain
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-05-07 18:58:40 -04:00
Derek Collison
e438d2f5fa Mixed mode improvements.
1. When in mixed mode and only running the global account we now will check the account for JS.
2. Added code to decrease the cluster set size if we guessed wrong in mixed mode setup.

Signed-off-by: Derek Collison <derek@nats.io>
2021-04-09 14:58:35 -07:00
Derek Collison
c8d812d805 Release server lock when getting usage
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-27 17:38:57 -07:00
Derek Collison
282def10fd Fix data race
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-25 17:07:34 -07:00
Waldemar Quevedo
775afd01e7 Fix for JS reload and exports
Signed-off-by: Waldemar Quevedo <wally@synadia.com>
2021-03-14 15:16:56 -07:00
Derek Collison
bfb8e3432e Move RAFT comms off internal sendq.
Move route and gateway msgs our of fast path for inbound stream msgs.

Signed-off-by: Derek Collison <derek@nats.io>
2021-03-04 14:45:34 -08:00
Ivan Kozlovic
0f53bf6580 Fixed data race with nodeInfo
Took the approach of storing struct instead of pointer. Of course,
when changing the offline bool from false to true, it means that
we need to call Store again (with same key).

This is based on the assumption that those Load/Store are not too
frequent. Otherwise, we may need to use locking (and keep *nodeInfo)

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-03-03 13:28:45 -07:00
Derek Collison
2ecf6be3ef Mark raft node as offline when server is removed
Signed-off-by: Derek Collison <derek@nats.io>
2021-03-03 08:06:45 -08:00
Matthias Hanel
c50ee2a1c6 [Changed] all times exposed will be computed in UTC (#1943)
This also applies to times that end up in that json.
Where applicable moved time.Now() to where it is used.
Moved calls to .UTC() to where time is created it that time is converted
later anyway.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-03-02 21:37:42 -05:00
Derek Collison
b9e1a921ff Use internal wildcard inbox for stream and consumer info requests.
More gateway friendly but suffers from no echo, so added new client based internal send.

Signed-off-by: Derek Collison <derek@nats.io>
2021-02-28 10:01:01 -08:00
Derek Collison
78bdc34637 General stability improvements. Fixes to subscription state not cleaning up.
Signed-off-by: Derek Collison <derek@nats.io>
2021-02-24 08:44:34 -08:00
Derek Collison
6444038660 Make sure this is GW and leafnode friendly
Signed-off-by: Derek Collison <derek@nats.io>
2021-02-23 10:57:10 -08:00
Ivan Kozlovic
61bd1b8d86 MQTT clustering
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-02-19 08:50:00 -07:00
Derek Collison
d803189eca Changes needed to properly support multi-layer service imports for system services like JS.
Signed-off-by: Derek Collison <derek@nats.io>
2021-02-12 17:01:32 -08:00
R.I.Pienaar
b36abc0f31 skip responses from non leaders in jsz
Adds an error type and a check to avoid sending them
to clients

Signed-off-by: R.I.Pienaar <rip@devco.net>
2021-02-09 17:34:29 +01:00
Derek Collison
7ae8d66c6d Do ramp down on server stats on startup
Signed-off-by: Derek Collison <derek@nats.io>
2021-02-08 06:46:57 -08:00
Derek Collison
15355d783b For larger superclusters we send out our server information after a short delay on startup.
For determing leaderless make sure the raft node has been running for long enough.

Signed-off-by: Derek Collison <derek@nats.io>
2021-02-08 06:26:33 -08:00
Derek Collison
f209c60123 Send our own serverinfo when we receive an update from a new server
Signed-off-by: Derek Collison <derek@nats.io>
2021-02-07 06:53:07 -08:00
Derek Collison
6d32c307ef Remove pretty indent for json.
Signed-off-by: Derek Collison <derek@nats.io>
2021-02-06 20:09:44 -08:00
Derek Collison
1622c2b60e Send serverInfo on remote shutdown and remove mapping entries
Signed-off-by: Derek Collison <derek@nats.io>
2021-02-05 17:21:55 -08:00
Matthias Hanel
7b7543d298 [added] jsz nats and http monitoring endpoint for jetstream (#1881)
The new endpoints are /jsz on http and "$SYS.REQ.SERVER.PING.JSZ" and "$SYS.REQ.SERVER.%s.JSZ".
$SYS.REQ.ACCOUNT.%s.JSZ will only return info for the particular account

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-02-05 18:46:04 -05:00
Derek Collison
a1e0f7dc1a First pass at supercluster enablement.
This allows metacontrollers to span superclusters. Also includes placement directives for streams. By default they select the request origin cluster.

Signed-off-by: Derek Collison <derek@nats.io>
2021-02-03 17:28:13 -08:00