Commit Graph

530 Commits

Author SHA1 Message Date
Derek Collison
7f41b5a6ae Added support for DiscardPolicy on streams
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
50443ead89 Change to consumer info, removed state, made pending and redelivery counts
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
7d7a9e8f23 Make sure messages are not redelivered if ack'd late (pull-based this can happen)
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
0bf6b2afbc Allow addition of streams and consumers (durables) with same config to be acceptable
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
a7f1bca534 Additional service latency upgrades.
We now share more information about the responder and the requestor. The requestor information by default is not shared, but can be when declaring the import.

Also fixed bug for error handling on old request style requests that would always result on a 408 response.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
ea5e5bd364 Services rewrite #2
This contains a rewrite to the services layer for exporting and importing. The code this merges to already had a first significant rewrite that moved from special interest processing to plain subscriptions.

This code changes the prior version's dealing with reverse mapping which was based mostly on thresholds and manual pruning, with some sporadic timer usage. This version uses the jetstream branch's code that understands interest and failed deliveries. So this code is much more tuned to reacting to interest changes. It also removes thresholds and goes only by interest changes or expirations based around a new service export property, response thresholds. This allows a service provider to provide semantics on how long a response should take at a maximum.

This commit also introduces formal support for service export streamed and chunked response types send an empty message to signify EOF.

This commit also includes additions to the service latency tracking such that errors are now sent, not only successful interactions. We have added a Status field and an optional Error fields to ServiceLatency.

We support the following Status codes, these are directly from HTTP.

400 Bad Request (request did not have a reply subject)
408 Request Timeout (when system detects request interest went away, old request style to make dependable)..
503 Service Unavailable (no service responders running)
504 Service Timeout (The new response threshold expired)

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
R.I.Pienaar
fb37c121d1 expose the arrival timestamp in the message metadata
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:21:27 -07:00
Derek Collison
0a36706958 PubAck details that provide stream name and sequence assigned
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:21:27 -07:00
Derek Collison
4139777ea8 First pass at account config for jetstream server reload support
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:21:27 -07:00
Derek Collison
b5daac9638 Properly parse and use jetstream server config
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:21:27 -07:00
Derek Collison
ec0bc1dbec First pass account configuration for jetstream
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:21:27 -07:00
Derek Collison
1394a7118d Breaking change to upgrade ConsumerConfig for consumer creation.
This is a breaking change and will not be able to restore consumer's from a filestore when upgraded.
We are getting close to settling on the API an once that happens we will not be introducnig any
breaking changes.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:20:02 -07:00
Derek Collison
ffae57e20e Fold replay original logic into main loop, fixed pull bug on replay original
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:20:02 -07:00
Derek Collison
7f458282b3 Double check we receive on the correct subject
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:20:02 -07:00
Derek Collison
d2ff4311d4 Rebase with master, updates to go.mod and vendor, bumped version
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:20:02 -07:00
Derek Collison
2d94caee87 Fix for https://github.com/nats-io/jetstream/issues/144
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
4a30f593bd Fix for https://github.com/nats-io/jetstream/issues/143
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
0b970130b0 Allow filter subjects to have wildcards. Fix for https://github.com/nats-io/jetstream/issues/136
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
3c1dcd2401 Remove pull based consumers that are no longer valid or have registered interest
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
4774d41f9d Fixes for filestore consistency on server restarts under heavy load with purge operations
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
ade58bea87 Do not deadlock when triggering lots of didNotDeliver notices. Don't hold lock on chan send
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
323b3d0b11 Beginning basics for stream and consumer snapshotting
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
df774e44b0 Rework how service imports are handled to avoid performance hits
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
3ab76a6dcd Write performance tweaks
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
3d80ff71a4 Fix for queue subscribers as consumers
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
5e9f487b61 Fixes for test and locking, bumped version
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
4a46a92cea Redesigned file storage engine.
The old engine was a simple design to get us started but had a single lock and held that lock around I/O.
This design allows message blocks to have the own RW locks and operate independently and to also not hold
locks while doing most I/O.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
623fa5ccec Impl for GetSeqFromTime for FileStore was missing
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
c9d3bdd5ab Added Stream Update abilities for certain properties
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
8d1f3cc7c2 Allow JetStream consumers to work across multi-server hops
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
d5aa55a103 Audit advisories, work based off of #1253 from ripienaar
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
5c99355fa5 Separate out consumer API for Durable vs Ephemeral
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
cd052a42d6 Fix for non-js msg failed delivery stopping normal subscribers
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
f0b5fd2882 Add in API support for templates, proto error helper
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
4046babb77 Make other API calls for next, ack ,and byseq more aligned with others
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
R.I.Pienaar
88760724e1 also validate that subject matches request 2020-05-19 14:16:03 -07:00
R.I.Pienaar
25e535eab8 add stream name to create subject
I anticipate it would be desirable to allow a microservice running
in an account to create only the stream it needs when it starts up
if there is no stream but no others, so a ACL would be written to
allow that.

Thus adding the T pattern to stream create too.
2020-05-19 14:16:03 -07:00
Derek Collison
72cda966f4 Changed API subjects to be authorization friendly
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
47c28b2fb0 JetStream major refactor for name changes.
MsgSet -> Stream
Observable -> Consumer

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
23da0b08ac Pulled out K/V support for now
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
119c48ee53 Do not allow overlapping message sets for now
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
7d1431d903 Add in K/V Delete
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
d836769cab Added in basic K/V scenario
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
2237915c97 Added in template support
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
d4c3c481bd Allow ephemeral observables on work queue message sets
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
a7d7b6ff56 Fix for writing messages after restart and delivery count suppression on max
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
e3be6f45c4 Add in delete for observables for filestore impl
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
1028798d3a Move Subject to FilterSubject
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
89ff13a5be Add MaxDeliver for observables
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
b3739f9bbf MaxMsgSize for MsgSets, -ERR api cleanup
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00