Commit Graph

79 Commits

Author SHA1 Message Date
Derek Collison
6f2b50a374 Added support for clustered account info and limit enforcement
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-21 18:47:21 -08:00
Derek Collison
55d750733b Fix based on feedback from Ivan
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-20 11:58:31 -08:00
Derek Collison
78747b2414 Stability improvements around startup and restore.
We were incorrectly starting clustering before enabling accounts and restoring state.

Signed-off-by: Derek Collison <derek@nats.io>
2021-01-17 13:44:49 -08:00
Derek Collison
b606dceb59 Stabilize restart/catchup for raft.
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-16 05:47:48 -08:00
Derek Collison
4b84decc7f Fix for race
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 12:59:12 -08:00
Derek Collison
b68d7066c4 Remove alpha banner
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 06:05:55 -08:00
Derek Collison
4bfe9d4c24 Fixes to PR.
Add nats to default storage directory
Fix race in raft, change leader notice
Fix test crash on failure

Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 05:56:05 -08:00
Derek Collison
f0cdf89c61 JetStream Clustering WIP
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 01:14:52 -08:00
Derek Collison
bfb726e8e9 Make sure to clear JS resources on reload
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-30 17:18:33 -08:00
Derek Collison
7e27042e6e Fix for #1736
When a system account was configured and not the default when we did a reload we would lose the JetStream service exports.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-30 16:11:50 -08:00
Derek Collison
44a1373f89 JetStream changes.
Made several changes based on feedback.

1. Made PubAckResponse only optionally include an ApiError and not force an API type.
2. Allow FilterSubject to be set on a consumer config and cleared if it matches the only stream subject.
3. Remove LookupStream by subject, and add in filters for stream names API.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-25 06:50:25 -08:00
Derek Collison
5ff28b6087 Allow stream lookup by subject.
Allow an API endpoint and public API to lookup a stream by subject. The subject needs to be an exact match or a subset. If the subject is considered a filtered subject for the stream that will also be returned.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-14 13:35:05 -08:00
Derek Collison
164f9fdf2b Updates to consumer store to support delta updates.
In preparation for clustering we need to have the consumer filestore update state with deltas vs original design.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-10 19:16:55 -08:00
Derek Collison
1f8ec6c964 Separate out disk available for Windows
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-31 12:56:27 -07:00
Derek Collison
fe2b354414 Stability and performance updates.
The original design had a shared filestore write buffer and individual message blocks had a read cache.
This presented some performance and stability issues when lots of reads and deletes were happening to a
message block that was also being written to actively.

This change eliminates the shared write buffer and uses the message block's cache as a write through as
well as read cache and handles partials correctly.

Signed-off-by: Derek Collison <derek@nats.io>
2020-10-28 07:58:47 -07:00
Matthias Hanel
a3c5fd4575 Updating jetstream account settings from jwt
Signed-off-by: Matthias Hanel <mh@synadia.com>
2020-09-04 16:07:38 -04:00
Ivan Kozlovic
22833c8d1a Fix sysSubscribe races
Made changes to processSub() to accept subscription properties,
including the icb callback so that it is set prior to add the
subscription to the account's sublist, which prevent races.
Fixed some other racy conditions, notably in addServiceImportSub()

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-08-03 14:59:00 -06:00
R.I.Pienaar
b80e40340e allow account limits to be applied to unlimited streams
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-07-27 19:02:37 +02:00
R.I.Pienaar
ca0dc72841 allow max consumers to be set
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-07-24 17:30:41 +02:00
R.I.Pienaar
5037f00b30 ensure unlimited account limits are calculated correctly
Previously unlimited accounts - ones who inherit server values - would
be unable to publish any messags at all

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-07-20 10:46:34 +02:00
Derek Collison
8727315eb9 Updated snapshots, added restore, generic hashes
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-26 19:53:16 -07:00
Derek Collison
710ef00383 Don't allow JetStream on system account. Warn when accounts configured but no JS
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-25 12:17:18 -07:00
Derek Collison
3caf6265d4 Properly recover ephemeral consumers after restart
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-25 11:06:55 -07:00
R.I.Pienaar
5504c1212d adds a warning banner about the preview status
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-22 18:32:24 +02:00
Derek Collison
476e339cf9 Race detection fix
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 16:36:08 -07:00
Derek Collison
d3a447e6e9 Be more efficient when cleaning up all consumers for a stream
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
5256781ba0 Addressed issues raised in https://github.com/nats-io/nats-server/pull/1369
API still only turned on for account info in disabled accounts. Issues with advisories. Plan is still to have all endpoints on in all accounts.
Stream list and Consumer list return names only, page limit increased to 1024.
Stream, Consumer and Template names limited to 256 for now.
Subject API for stream messages, delete and get, not have STREAM.MSG.
Subject API for Durable is now CONSUMER.DURABLE.
Subject API for Templates now STREAM.TEMPLATE.
All subject APIs for list reverted back, so STREAM.LIST, CONSUMER.LIST etc.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
cadd39a01c Major rewrite for NATS JetStream API
API made more consistent. Noun followed by verb.
Name arguments in request subejcts are always at the end now.
Remove enabled call, just use account info.
Getting a message directly from a stream is treated like an admin API and requires JSON request.
Deleting a message directly as well.
StreamList and ConsumerList now include details and support paging.
Streams and Consumers now contain a created field in their info.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
513bff051b Redesign message-based API
Removed usage of +OK and -ERR. All responses are valid json objects now and optionally can include an ApiError which will have Code and Description.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
4139777ea8 First pass at account config for jetstream server reload support
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:21:27 -07:00
Derek Collison
ec0bc1dbec First pass account configuration for jetstream
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:21:27 -07:00
Derek Collison
4774d41f9d Fixes for filestore consistency on server restarts under heavy load with purge operations
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
3ab76a6dcd Write performance tweaks
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
c9d3bdd5ab Added Stream Update abilities for certain properties
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
d5aa55a103 Audit advisories, work based off of #1253 from ripienaar
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
313dcd21ba Removed condition not needed
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
5c99355fa5 Separate out consumer API for Durable vs Ephemeral
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
e703d62bd5 Tweak ACK reply subject
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
f0b5fd2882 Add in API support for templates, proto error helper
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
4046babb77 Make other API calls for next, ack ,and byseq more aligned with others
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
R.I.Pienaar
88760724e1 also validate that subject matches request 2020-05-19 14:16:03 -07:00
R.I.Pienaar
25e535eab8 add stream name to create subject
I anticipate it would be desirable to allow a microservice running
in an account to create only the stream it needs when it starts up
if there is no stream but no others, so a ACL would be written to
allow that.

Thus adding the T pattern to stream create too.
2020-05-19 14:16:03 -07:00
Derek Collison
72cda966f4 Changed API subjects to be authorization friendly
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
R.I.Pienaar
24939d51db use "advisory" instead of "notification" 2020-05-19 14:16:03 -07:00
R.I.Pienaar
200ebbd47e proposed schemas for events
This is a proposal to add schema, timestamps and unique ids to
events.  For now just JS ones but I propose to do these for all
events the server sends that might be end user consumed.

These exist to help the user identify a message even if it was
sent to a 3rd party system, the schema will be translatable to
a well known url like https://nats.io/schames/jetstream/event/v1/obs_ack.json
or to another wll known identifier like a NATS subject name

We'd publish JSON schema documents at these well known locations
describing each key and so forth
2020-05-19 14:16:03 -07:00
R.I.Pienaar
c9da1ca64a fix purge export subject 2020-05-19 14:16:03 -07:00
Derek Collison
47c28b2fb0 JetStream major refactor for name changes.
MsgSet -> Stream
Observable -> Consumer

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
23da0b08ac Pulled out K/V support for now
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
d836769cab Added in basic K/V scenario
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
2237915c97 Added in template support
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00