Commit Graph

86 Commits

Author SHA1 Message Date
Derek Collison
c16f6e193d Move JetStream direct APIs to private.
Signed-off-by: Derek Collison <derek@nats.io>
2021-02-07 15:19:22 -08:00
Derek Collison
c9e1cbd571 Modified jetstream varz information
Signed-off-by: Derek Collison <derek@nats.io>
2021-02-02 08:35:35 -08:00
Derek Collison
e5c1d65fff Added in JS disable per server on reload. Also removing peerw from a stream and leader stepdown for streams and consumers.
Various bug fixes, stability improvments.

Signed-off-by: Derek Collison <derek@nats.io>
2021-02-01 19:39:08 -08:00
Derek Collison
095a83bc2a Make sure to suppress duplicate create/delete audit events
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-31 09:16:16 -08:00
Derek Collison
8bd290c77a Fix for #1864.
When trying to make sure we properly created all subs for service imports we would check the internal client to see if we should process.
With JS enabled on the server we would place system imports that would break that check and orphan other service imports.

Signed-off-by: Derek Collison <derek@nats.io>
2021-01-29 17:51:14 -08:00
Derek Collison
c889321a83 Change to API.Total and API.Errors
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-29 09:42:20 -08:00
Derek Collison
0a3124e27d Track API calls per account. Track success and errors.
These tracking data are ephemeral per server. so on restart they reset.
That should be ok since these will most likely be used more for rates.

Signed-off-by: Derek Collison <derek@nats.io>
2021-01-28 17:16:50 -08:00
Derek Collison
6f2b50a374 Added support for clustered account info and limit enforcement
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-21 18:47:21 -08:00
Derek Collison
55d750733b Fix based on feedback from Ivan
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-20 11:58:31 -08:00
Derek Collison
78747b2414 Stability improvements around startup and restore.
We were incorrectly starting clustering before enabling accounts and restoring state.

Signed-off-by: Derek Collison <derek@nats.io>
2021-01-17 13:44:49 -08:00
Derek Collison
b606dceb59 Stabilize restart/catchup for raft.
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-16 05:47:48 -08:00
Derek Collison
4b84decc7f Fix for race
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 12:59:12 -08:00
Derek Collison
b68d7066c4 Remove alpha banner
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 06:05:55 -08:00
Derek Collison
4bfe9d4c24 Fixes to PR.
Add nats to default storage directory
Fix race in raft, change leader notice
Fix test crash on failure

Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 05:56:05 -08:00
Derek Collison
f0cdf89c61 JetStream Clustering WIP
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 01:14:52 -08:00
Derek Collison
bfb726e8e9 Make sure to clear JS resources on reload
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-30 17:18:33 -08:00
Derek Collison
7e27042e6e Fix for #1736
When a system account was configured and not the default when we did a reload we would lose the JetStream service exports.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-30 16:11:50 -08:00
Derek Collison
44a1373f89 JetStream changes.
Made several changes based on feedback.

1. Made PubAckResponse only optionally include an ApiError and not force an API type.
2. Allow FilterSubject to be set on a consumer config and cleared if it matches the only stream subject.
3. Remove LookupStream by subject, and add in filters for stream names API.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-25 06:50:25 -08:00
Derek Collison
5ff28b6087 Allow stream lookup by subject.
Allow an API endpoint and public API to lookup a stream by subject. The subject needs to be an exact match or a subset. If the subject is considered a filtered subject for the stream that will also be returned.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-14 13:35:05 -08:00
Derek Collison
164f9fdf2b Updates to consumer store to support delta updates.
In preparation for clustering we need to have the consumer filestore update state with deltas vs original design.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-10 19:16:55 -08:00
Derek Collison
1f8ec6c964 Separate out disk available for Windows
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-31 12:56:27 -07:00
Derek Collison
fe2b354414 Stability and performance updates.
The original design had a shared filestore write buffer and individual message blocks had a read cache.
This presented some performance and stability issues when lots of reads and deletes were happening to a
message block that was also being written to actively.

This change eliminates the shared write buffer and uses the message block's cache as a write through as
well as read cache and handles partials correctly.

Signed-off-by: Derek Collison <derek@nats.io>
2020-10-28 07:58:47 -07:00
Matthias Hanel
a3c5fd4575 Updating jetstream account settings from jwt
Signed-off-by: Matthias Hanel <mh@synadia.com>
2020-09-04 16:07:38 -04:00
Ivan Kozlovic
22833c8d1a Fix sysSubscribe races
Made changes to processSub() to accept subscription properties,
including the icb callback so that it is set prior to add the
subscription to the account's sublist, which prevent races.
Fixed some other racy conditions, notably in addServiceImportSub()

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-08-03 14:59:00 -06:00
R.I.Pienaar
b80e40340e allow account limits to be applied to unlimited streams
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-07-27 19:02:37 +02:00
R.I.Pienaar
ca0dc72841 allow max consumers to be set
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-07-24 17:30:41 +02:00
R.I.Pienaar
5037f00b30 ensure unlimited account limits are calculated correctly
Previously unlimited accounts - ones who inherit server values - would
be unable to publish any messags at all

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-07-20 10:46:34 +02:00
Derek Collison
8727315eb9 Updated snapshots, added restore, generic hashes
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-26 19:53:16 -07:00
Derek Collison
710ef00383 Don't allow JetStream on system account. Warn when accounts configured but no JS
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-25 12:17:18 -07:00
Derek Collison
3caf6265d4 Properly recover ephemeral consumers after restart
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-25 11:06:55 -07:00
R.I.Pienaar
5504c1212d adds a warning banner about the preview status
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-22 18:32:24 +02:00
Derek Collison
476e339cf9 Race detection fix
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 16:36:08 -07:00
Derek Collison
d3a447e6e9 Be more efficient when cleaning up all consumers for a stream
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
5256781ba0 Addressed issues raised in https://github.com/nats-io/nats-server/pull/1369
API still only turned on for account info in disabled accounts. Issues with advisories. Plan is still to have all endpoints on in all accounts.
Stream list and Consumer list return names only, page limit increased to 1024.
Stream, Consumer and Template names limited to 256 for now.
Subject API for stream messages, delete and get, not have STREAM.MSG.
Subject API for Durable is now CONSUMER.DURABLE.
Subject API for Templates now STREAM.TEMPLATE.
All subject APIs for list reverted back, so STREAM.LIST, CONSUMER.LIST etc.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
cadd39a01c Major rewrite for NATS JetStream API
API made more consistent. Noun followed by verb.
Name arguments in request subejcts are always at the end now.
Remove enabled call, just use account info.
Getting a message directly from a stream is treated like an admin API and requires JSON request.
Deleting a message directly as well.
StreamList and ConsumerList now include details and support paging.
Streams and Consumers now contain a created field in their info.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
513bff051b Redesign message-based API
Removed usage of +OK and -ERR. All responses are valid json objects now and optionally can include an ApiError which will have Code and Description.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
4139777ea8 First pass at account config for jetstream server reload support
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:21:27 -07:00
Derek Collison
ec0bc1dbec First pass account configuration for jetstream
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:21:27 -07:00
Derek Collison
4774d41f9d Fixes for filestore consistency on server restarts under heavy load with purge operations
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
3ab76a6dcd Write performance tweaks
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
c9d3bdd5ab Added Stream Update abilities for certain properties
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
d5aa55a103 Audit advisories, work based off of #1253 from ripienaar
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
313dcd21ba Removed condition not needed
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
5c99355fa5 Separate out consumer API for Durable vs Ephemeral
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
e703d62bd5 Tweak ACK reply subject
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
f0b5fd2882 Add in API support for templates, proto error helper
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
4046babb77 Make other API calls for next, ack ,and byseq more aligned with others
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
R.I.Pienaar
88760724e1 also validate that subject matches request 2020-05-19 14:16:03 -07:00
R.I.Pienaar
25e535eab8 add stream name to create subject
I anticipate it would be desirable to allow a microservice running
in an account to create only the stream it needs when it starts up
if there is no stream but no others, so a ACL would be written to
allow that.

Thus adding the T pattern to stream create too.
2020-05-19 14:16:03 -07:00
Derek Collison
72cda966f4 Changed API subjects to be authorization friendly
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00