Commit Graph

520 Commits

Author SHA1 Message Date
Derek Collison
ec0bc1dbec First pass account configuration for jetstream
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:21:27 -07:00
Derek Collison
1394a7118d Breaking change to upgrade ConsumerConfig for consumer creation.
This is a breaking change and will not be able to restore consumer's from a filestore when upgraded.
We are getting close to settling on the API an once that happens we will not be introducnig any
breaking changes.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:20:02 -07:00
Derek Collison
ffae57e20e Fold replay original logic into main loop, fixed pull bug on replay original
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:20:02 -07:00
Derek Collison
7f458282b3 Double check we receive on the correct subject
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:20:02 -07:00
Derek Collison
d2ff4311d4 Rebase with master, updates to go.mod and vendor, bumped version
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:20:02 -07:00
Derek Collison
2d94caee87 Fix for https://github.com/nats-io/jetstream/issues/144
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
4a30f593bd Fix for https://github.com/nats-io/jetstream/issues/143
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
0b970130b0 Allow filter subjects to have wildcards. Fix for https://github.com/nats-io/jetstream/issues/136
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
3c1dcd2401 Remove pull based consumers that are no longer valid or have registered interest
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
4774d41f9d Fixes for filestore consistency on server restarts under heavy load with purge operations
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
ade58bea87 Do not deadlock when triggering lots of didNotDeliver notices. Don't hold lock on chan send
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
323b3d0b11 Beginning basics for stream and consumer snapshotting
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
df774e44b0 Rework how service imports are handled to avoid performance hits
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
3ab76a6dcd Write performance tweaks
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
3d80ff71a4 Fix for queue subscribers as consumers
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
5e9f487b61 Fixes for test and locking, bumped version
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
4a46a92cea Redesigned file storage engine.
The old engine was a simple design to get us started but had a single lock and held that lock around I/O.
This design allows message blocks to have the own RW locks and operate independently and to also not hold
locks while doing most I/O.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
623fa5ccec Impl for GetSeqFromTime for FileStore was missing
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
c9d3bdd5ab Added Stream Update abilities for certain properties
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
8d1f3cc7c2 Allow JetStream consumers to work across multi-server hops
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
d5aa55a103 Audit advisories, work based off of #1253 from ripienaar
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
5c99355fa5 Separate out consumer API for Durable vs Ephemeral
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
cd052a42d6 Fix for non-js msg failed delivery stopping normal subscribers
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
f0b5fd2882 Add in API support for templates, proto error helper
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
4046babb77 Make other API calls for next, ack ,and byseq more aligned with others
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
R.I.Pienaar
88760724e1 also validate that subject matches request 2020-05-19 14:16:03 -07:00
R.I.Pienaar
25e535eab8 add stream name to create subject
I anticipate it would be desirable to allow a microservice running
in an account to create only the stream it needs when it starts up
if there is no stream but no others, so a ACL would be written to
allow that.

Thus adding the T pattern to stream create too.
2020-05-19 14:16:03 -07:00
Derek Collison
72cda966f4 Changed API subjects to be authorization friendly
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
47c28b2fb0 JetStream major refactor for name changes.
MsgSet -> Stream
Observable -> Consumer

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
23da0b08ac Pulled out K/V support for now
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
119c48ee53 Do not allow overlapping message sets for now
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
7d1431d903 Add in K/V Delete
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
d836769cab Added in basic K/V scenario
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
2237915c97 Added in template support
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
d4c3c481bd Allow ephemeral observables on work queue message sets
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
a7d7b6ff56 Fix for writing messages after restart and delivery count suppression on max
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
e3be6f45c4 Add in delete for observables for filestore impl
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
1028798d3a Move Subject to FilterSubject
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
89ff13a5be Add MaxDeliver for observables
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
b3739f9bbf MaxMsgSize for MsgSets, -ERR api cleanup
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
7e8c74fdbd Pull-based observables must be durable
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
1c939d5cc8 Adjust delivered for obs info via -1
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00
Derek Collison
37be43ee5a Make sure purge works with redeliveries pending
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:15:11 -07:00
Derek Collison
f78efe2d91 Add Name to Observable info
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:15:11 -07:00
Derek Collison
5efdf63625 Fix case where request for next msg times out
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:15:11 -07:00
Derek Collison
54e2de912f Update observables on purge
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:15:11 -07:00
Derek Collison
58d98d42cb Allow notifications for literal subjects.
This allows us to register for notifications on subjects to determine when interest comes and goes.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:15:11 -07:00
Derek Collison
a1e802c04b Flush state on shutdown and re-initialize pending checks on restore to do redelivery
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:15:11 -07:00
Derek Collison
b3b2feebc5 Include config in info for msgset
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:15:11 -07:00
Derek Collison
ca5b58e27f Bug fix for original replay with no initial interest, README updates
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:15:11 -07:00