Commit Graph

56 Commits

Author SHA1 Message Date
Derek Collison
9ad408e0b3 Use closed variable on consumer, don't check consumer sequences
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-02 16:21:23 -07:00
Derek Collison
8a9f6eaf42 Additional fixes to interest retention based streams and offline durables with redelivery.
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-02 12:58:59 -07:00
Derek Collison
a75be04b0a Various fixes for this PR.
- Fix for updating delivery subject and adjusting next delivery sequences.
- When acking explicitly but out of order, need to make sure we set floor correctly.
- Only update ack floors on an ack if the message is present.
- Fix for needAck for explicitAck out of order consumers detecting if message has been acked.
- Fix for race not locking stream when checking interest during stop.
- Fix for filestore determing if a message block still has a message. Added check to first sequence as well as cache.
- Some additions to the original test.

Signed-off-by: Derek Collison <derek@nats.io>
2020-10-01 21:24:40 -07:00
Ivan Kozlovic
3977ea04f5 Fixed possible panic on consumer.Delete() during server Shutdown()
The panic was caused by the closing of an already closed Go channel.
The Delete() relied on the consumer's mset being nil to consider
the consumer already closed. However, the consumer's mset is set
to nil after invoking sendDeleteAdvisoryLocked() which internally
invokes sendAdvisory() which releases/reacquires the consumer lock.
This left an open door for a race to occur and Delete() to be
invoked twice on the same consumer.

Moving setting the consumer's mset to nil too early would prevent
the sendAdvisory() to actually do its job. We could pass the mset
to sendAvisory(), but a simpler approach is to simply use a "closed"
boolean on the Consumer object that is set to true at the beginning
of the Delete() function.

Resolves #1621

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-09-29 10:01:47 -06:00
Derek Collison
7f44d075f7 Merge pull request #1601 from ripienaar/skip_ack_on_acknxt
Do not ack to AckNxt
2020-09-28 17:19:35 -07:00
Derek Collison
9129d7b62f Set to 1 if no acks received, fixes #1619
Signed-off-by: Derek Collison <derek@nats.io>
2020-09-28 15:56:50 -07:00
R.I.Pienaar
cd376aa753 Do not ack to AckNxt
This causes nil messages mid content stream and is confusing,
the test for this is amended to test the content received for
validity - where previously the test assumed any content is
good content but in fact it only received half the content.

Removing this behaviour until we can design this properly

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-09-21 17:28:53 +02:00
Derek Collison
bcbf28fb3a Make sure to clear messages from stream when consumer deleted
Signed-off-by: Derek Collison <derek@nats.io>
2020-09-03 11:41:14 -07:00
Derek Collison
3faa2e7637 Fix for #1580
Signed-off-by: Derek Collison <derek@nats.io>
2020-09-02 17:18:28 -07:00
Derek Collison
6e1a892740 Merge pull request #1531 from ripienaar/max_consumers
allow max consumers to be set
2020-07-29 09:09:11 -07:00
R.I.Pienaar
b80e40340e allow account limits to be applied to unlimited streams
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-07-27 19:02:37 +02:00
Derek Collison
154d4303a9 Add in consumer rate limits
Signed-off-by: Derek Collison <derek@nats.io>
2020-07-24 10:11:32 -07:00
Derek Collison
48b2335cbf Fix for stream delete with many consumers - https://github.com/nats-io/jetstream/issues/260
Signed-off-by: Derek Collison <derek@nats.io>
2020-07-22 07:35:38 -07:00
Derek Collison
ad21074177 Merge pull request #1508 from nats-io/eo
Provide exactly once semantics
2020-07-09 16:27:56 -07:00
Derek Collison
b74c2eb2c4 Provide exactly once semantics
Signed-off-by: Derek Collison <derek@nats.io>
2020-07-08 16:56:52 -07:00
Derek Collison
aecdca874f Pending timers could go negative, this is a fix for #1502
Signed-off-by: Derek Collison <derek@nats.io>
2020-07-06 14:59:05 -07:00
Derek Collison
afc7fc367b Remove hdrs for now, find better way to deliver in client
Signed-off-by: Derek Collison <derek@nats.io>
2020-06-02 07:10:23 -07:00
Derek Collison
4d62a7237d Allow redelivery for AckAll policy, general upgrades to pending beahviors. Fixes #1436
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-31 07:50:50 -07:00
Derek Collison
dbde2479c2 Add in headers to consumer delivered messages
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-30 15:03:54 -07:00
Derek Collison
eca04c6fce First pass header support for JetStream
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-30 10:04:23 -07:00
Derek Collison
10e49ca1c4 Fix more flappers
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-28 14:19:11 -07:00
Derek Collison
8c9a30b8a1 Avoid deadlock by releasing consumer lock
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-28 12:30:12 -07:00
Derek Collison
d3ac95a5e6 Add in a terminate delivery for https://github.com/nats-io/jetstream/issues/189
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-28 08:32:34 -07:00
Derek Collison
4c91b69c4f Merge pull request #1413 from nats-io/fix_flappers
Fix flappers
2020-05-26 09:01:04 -07:00
Derek Collison
3caf6265d4 Properly recover ephemeral consumers after restart
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-25 11:06:55 -07:00
Derek Collison
f02ec860b6 Fix race when active state has not been updated yet
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-25 07:30:58 -07:00
Derek Collison
e42dc4915f Write consumer state during stop if not deleting
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-23 13:07:56 -07:00
R.I.Pienaar
c407a3b9d7 create advisories for stream and consumer add/delete/modify
We now publish advisories when streams and consumers are added,
deleted and modified

Also rework how TypedEvents are created to be easier to use

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-22 15:20:16 +02:00
Derek Collison
476e339cf9 Race detection fix
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 16:36:08 -07:00
Derek Collison
d3a447e6e9 Be more efficient when cleaning up all consumers for a stream
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
5256781ba0 Addressed issues raised in https://github.com/nats-io/nats-server/pull/1369
API still only turned on for account info in disabled accounts. Issues with advisories. Plan is still to have all endpoints on in all accounts.
Stream list and Consumer list return names only, page limit increased to 1024.
Stream, Consumer and Template names limited to 256 for now.
Subject API for stream messages, delete and get, not have STREAM.MSG.
Subject API for Durable is now CONSUMER.DURABLE.
Subject API for Templates now STREAM.TEMPLATE.
All subject APIs for list reverted back, so STREAM.LIST, CONSUMER.LIST etc.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
cadd39a01c Major rewrite for NATS JetStream API
API made more consistent. Noun followed by verb.
Name arguments in request subejcts are always at the end now.
Remove enabled call, just use account info.
Getting a message directly from a stream is treated like an admin API and requires JSON request.
Deleting a message directly as well.
StreamList and ConsumerList now include details and support paging.
Streams and Consumers now contain a created field in their info.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
R.I.Pienaar
63845b8577 add type hints to service latency, use time.Time for timestamp
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:26:46 -07:00
Derek Collison
513bff051b Redesign message-based API
Removed usage of +OK and -ERR. All responses are valid json objects now and optionally can include an ApiError which will have Code and Description.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
36d5421318 Don't do sample ack unless msg was on pending
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
50443ead89 Change to consumer info, removed state, made pending and redelivery counts
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
ae8bc2d22b Move ack to +ACK, +OK still works
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
7d7a9e8f23 Make sure messages are not redelivered if ack'd late (pull-based this can happen)
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
0bf6b2afbc Allow addition of streams and consumers (durables) with same config to be acceptable
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
R.I.Pienaar
3182db4c3a move to events having Type not Schema 2020-05-19 14:22:53 -07:00
R.I.Pienaar
fb37c121d1 expose the arrival timestamp in the message metadata
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:21:27 -07:00
Derek Collison
1394a7118d Breaking change to upgrade ConsumerConfig for consumer creation.
This is a breaking change and will not be able to restore consumer's from a filestore when upgraded.
We are getting close to settling on the API an once that happens we will not be introducnig any
breaking changes.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:20:02 -07:00
Derek Collison
ffae57e20e Fold replay original logic into main loop, fixed pull bug on replay original
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:20:02 -07:00
Derek Collison
4a30f593bd Fix for https://github.com/nats-io/jetstream/issues/143
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
0b970130b0 Allow filter subjects to have wildcards. Fix for https://github.com/nats-io/jetstream/issues/136
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
3c1dcd2401 Remove pull based consumers that are no longer valid or have registered interest
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
R.I.Pienaar
871edd996a add the stream name in consumer info 2020-05-19 14:18:34 -07:00
Derek Collison
4774d41f9d Fixes for filestore consistency on server restarts under heavy load with purge operations
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
ade58bea87 Do not deadlock when triggering lots of didNotDeliver notices. Don't hold lock on chan send
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:18:34 -07:00
Derek Collison
3d80ff71a4 Fix for queue subscribers as consumers
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:16:03 -07:00