Commit Graph

73 Commits

Author SHA1 Message Date
Derek Collison
e40c3e6f55 Templates not supported currently in clustered mode
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-25 17:13:31 -08:00
Derek Collison
9c858d197a Added ability to properly restore consumers from a snapshot.
This made us add forwarding proposals functionality in the raft layer.
More general cleanup and bug fixes as well.

Signed-off-by: Derek Collison <derek@nats.io>
2021-01-24 19:30:34 -08:00
Derek Collison
d1d2d5b24e Fix for consumer names list in clustered mode
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-23 10:09:44 -08:00
Derek Collison
4c965323f2 Additional stabilizations for split votes, general fixes
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-20 11:58:31 -08:00
Derek Collison
ff54c9dc9c Reworked snapshot and restore.
Underestimated the effort to get stream restore working properly in cluster mode.
Some good bug fixes and stability improvments.

Signed-off-by: Derek Collison <derek@nats.io>
2021-01-20 11:58:31 -08:00
Derek Collison
2e9545d587 Make snapshot API cluster aware
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-20 11:58:31 -08:00
Derek Collison
a1730f1b31 Report on RAFT group information.
This adds in optional reporting to stream and consumer info when running in clsutered mode.

Signed-off-by: Derek Collison <derek@nats.io>
2021-01-20 11:58:31 -08:00
Ivan Kozlovic
f5df209022 Fixed SIGSEGV when sending update for unknown stream
Will now return an error that the stream is unknown.

Resolves #1827

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-01-20 12:42:14 -07:00
Derek Collison
8d568d41e4 Copy off before starting Go routine
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-17 13:05:25 -08:00
Derek Collison
a18a6803c1 Added support for stream and consumer lists.
This utilizes a scatter and gather approach.

Signed-off-by: Derek Collison <derek@nats.io>
2021-01-16 12:42:45 -08:00
Derek Collison
cb69df7118 Add proper support for stream update
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-16 06:29:37 -08:00
Derek Collison
f0cdf89c61 JetStream Clustering WIP
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 01:14:52 -08:00
Ivan Kozlovic
b048b6b3de Merge pull request #1754 from nats-io/mqtt
[ADDED] MQTT Support
2020-12-07 09:06:12 -07:00
Derek Collison
7564768027 Added Compact to store interface for WAL functionality
Signed-off-by: Derek Collison <derek@nats.io>
2020-12-03 16:18:58 -08:00
Ivan Kozlovic
3e91ef75ab Some updates based on code review
- Added non-public stream and consumer configuration options to
achieve the "no subject" and "no interest" capabilities. Had
to implement custom FileStreamInfo and FileConsumerInfo marshal/
unmarshal methods so that those non public fields can be
persisted/recovered properly.
- Restored some of JS original code (since now can use config
instead of passing booleans to the functions).
- Use RLock for deliveryFormsCycle() check (unrelated to MQTT).
- Removed restriction on creating streams with MQTT prefix.
- Preventing API deletion of internal streams and their consumers.
- Added comment on Sublist's ReverseMatch method.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-01 14:05:54 -07:00
Ivan Kozlovic
718c995914 Allow "nats" utility to display internal MQTT streams
MQTT streams are special in that we do not set subjects in the config
since they capture all subjects. Otherwise, we would have been forced
to create a stream on say "MQTT.>" but then all publishes would have
to be prefixed with "MQTT." in order for them to be captured.

However, if one uses the "nats" tool to inspect those streams, the tool
would fail with:

```
server response is not a valid "io.nats.jetstream.api.v1.stream_info_response" message:
(root): Must validate one and only one schema (oneOf)
config: subjects is required
config: Must validate all the schemas (allOf)
```

To solve that, if we detect that user asks for the MQTT streams, we
artificially set the returned config's subject to ">".

Alternatively, we may want to not return those streams at all, although
there may be value to see the info for mqtt streams/consumers.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-11-30 20:08:44 -07:00
Derek Collison
7e27042e6e Fix for #1736
When a system account was configured and not the default when we did a reload we would lose the JetStream service exports.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-30 16:11:50 -08:00
Derek Collison
954f5a9093 Flattened filters for stream names API
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-25 07:46:56 -08:00
Derek Collison
44a1373f89 JetStream changes.
Made several changes based on feedback.

1. Made PubAckResponse only optionally include an ApiError and not force an API type.
2. Allow FilterSubject to be set on a consumer config and cleared if it matches the only stream subject.
3. Remove LookupStream by subject, and add in filters for stream names API.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-25 06:50:25 -08:00
Derek Collison
a1c1ead39d Enable JetStream streams and consumer access to be exported to another account
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-20 10:17:16 -08:00
R.I.Pienaar
ca8cbcdc63 support subjects starting with {
still doesnt support a subject {}

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-11-17 16:33:22 +01:00
R.I.Pienaar
02eaf6d831 trim json text
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-11-17 16:06:12 +01:00
R.I.Pienaar
a4116cc0fa support json requests for stream lookup 2020-11-17 15:33:44 +01:00
Derek Collison
5ff28b6087 Allow stream lookup by subject.
Allow an API endpoint and public API to lookup a stream by subject. The subject needs to be an exact match or a subset. If the subject is considered a filtered subject for the stream that will also be returned.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-14 13:35:05 -08:00
R.I.Pienaar
b337edb447 always include purged number
When purging an empty stream success is true but the
purged is then omitted since its, correctly, 0. It's
better to include it even when zero so it's not weirdly
inconsistent between succesful purges.

Also helps with validation of payloads

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-10-23 12:58:42 +02:00
Derek Collison
3b18f188ed Switched behavior to never refuse new request, and to alert when expiring ones with interest
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-21 10:35:29 -07:00
Derek Collison
c2b8de377c Formalize requests for next msg, support NoWait and Expires
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-20 16:25:19 -07:00
Derek Collison
610d2d21b7 More robust waiting queue for pull mode consumers
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-19 19:51:46 -07:00
R.I.Pienaar
f2106ab9e4 fix npe for offsets larger than items
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-10-15 11:27:31 +02:00
Andrew Harris
17930baa18 add client provided info into server side client logs when available 2020-07-24 15:40:39 -04:00
Derek Collison
c969e7e424 Do proper ubsubscribe when shutting off restore endpoint
Signed-off-by: Derek Collison <derek@nats.io>
2020-06-04 08:58:14 -07:00
Derek Collison
164f44ed18 Require reply subjects for restore chunks
Signed-off-by: Derek Collison <derek@nats.io>
2020-06-04 06:56:07 -07:00
Derek Collison
012d517ba1 Better error handling and reporting for failures
Signed-off-by: Derek Collison <derek@nats.io>
2020-06-04 06:17:50 -07:00
Derek Collison
660ea3c807 Snapshot restore now works across leafnodes.
This also introduces the ability to have flow control inbound for restoring a stream.
If the system detects a reply subject it will respond with a nil payload.
For the last EOF message if a reply is present it will respond with a stream info response or error.

Signed-off-by: Derek Collison <derek@nats.io>
2020-06-03 20:00:59 -07:00
R.I.Pienaar
920dd4269a fix argument order in snapshots
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-06-02 13:51:50 +02:00
R.I.Pienaar
c57b86128d publish audit advisories to the correct subject
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-06-02 12:40:03 +02:00
R.I.Pienaar
3fc5c9284a send stream advisories using a helper 2020-06-02 08:48:11 +02:00
Derek Collison
5271be49ee Fix race for snapshots
Signed-off-by: Derek Collison <derek@nats.io>
2020-06-01 09:21:22 -07:00
Derek Collison
e584d4efee Merge pull request #1435 from nats-io/js-hdrs
First pass header support for JetStream
2020-05-31 06:01:01 -07:00
Derek Collison
8e407f8db4 Do snapshot setup in go routine as well for checkMsgs
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-30 11:28:43 -07:00
Derek Collison
eca04c6fce First pass header support for JetStream
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-30 10:04:23 -07:00
Derek Collison
e12907ffa6 Allow snapshots to optionally check all message checksums
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-29 09:57:33 -07:00
Derek Collison
0a206b4c64 Snapshot performance tweaks
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-29 08:07:31 -07:00
Derek Collison
d3ac95a5e6 Add in a terminate delivery for https://github.com/nats-io/jetstream/issues/189
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-28 08:32:34 -07:00
Derek Collison
bc0fedbaba Udpates based on PR feedback
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-28 06:08:35 -07:00
Derek Collison
fa59cff105 Add in snapshot and restore JSApi
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-27 20:01:30 -07:00
R.I.Pienaar
c407a3b9d7 create advisories for stream and consumer add/delete/modify
We now publish advisories when streams and consumers are added,
deleted and modified

Also rework how TypedEvents are created to be easier to use

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-22 15:20:16 +02:00
R.I.Pienaar
859a36e7ac ensures that the streams value is never null
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:33:06 -07:00
R.I.Pienaar
85da492096 initialize api response structs properly
when go json marshal a unset []string it puts
null in the body and not [] which then makes
it an invalid list of strings.

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:33:06 -07:00
R.I.Pienaar
ce26c65174 fix the type hint for msg_get
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:33:06 -07:00