Commit Graph

69 Commits

Author SHA1 Message Date
R.I.Pienaar
96c53b77f5 Merge pull request #1683 from ripienaar/bare_acknxt
ensure bare AckNxt requesting >1 is supported
2020-10-30 16:15:20 +01:00
R.I.Pienaar
bf68068fa6 address review
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-10-30 15:27:42 +01:00
R.I.Pienaar
0e9ca7614f ensure bare AckNxt requesting >1 is supported
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-10-30 14:46:39 +01:00
Derek Collison
cce7195a2c Fixed bug that would not properly handle wildcard streams with interest retention and filtered consumers.
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-29 13:14:17 -07:00
Derek Collison
0f22e99601 Merge pull request #1675 from nats-io/filestore
Stability and performance updates for filestore.
2020-10-28 19:16:44 -07:00
Derek Collison
283c632d8b Updates based on PR feedback and fix for merge regression with pending PR
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-28 18:58:42 -07:00
R.I.Pienaar
a88c2ee349 rename consumer pending fields for clarity
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-10-28 16:12:01 +01:00
Derek Collison
df4ee081a5 Track number of stream pending for each consumer.
This will track the stream pending state for each consumer.
This code does account for filtered consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2020-10-27 19:30:42 -07:00
Derek Collison
df29f1fb50 Add detailed info on error for NextMsg() request
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-21 11:26:22 -07:00
Derek Collison
3b18f188ed Switched behavior to never refuse new request, and to alert when expiring ones with interest
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-21 10:35:29 -07:00
Derek Collison
7fb22206f5 Allow AckNxt to also process a next request struct
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-20 16:45:54 -07:00
Derek Collison
c2b8de377c Formalize requests for next msg, support NoWait and Expires
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-20 16:25:19 -07:00
Derek Collison
610d2d21b7 More robust waiting queue for pull mode consumers
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-19 19:51:46 -07:00
Derek Collison
9ad408e0b3 Use closed variable on consumer, don't check consumer sequences
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-02 16:21:23 -07:00
Derek Collison
8a9f6eaf42 Additional fixes to interest retention based streams and offline durables with redelivery.
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-02 12:58:59 -07:00
Derek Collison
a75be04b0a Various fixes for this PR.
- Fix for updating delivery subject and adjusting next delivery sequences.
- When acking explicitly but out of order, need to make sure we set floor correctly.
- Only update ack floors on an ack if the message is present.
- Fix for needAck for explicitAck out of order consumers detecting if message has been acked.
- Fix for race not locking stream when checking interest during stop.
- Fix for filestore determing if a message block still has a message. Added check to first sequence as well as cache.
- Some additions to the original test.

Signed-off-by: Derek Collison <derek@nats.io>
2020-10-01 21:24:40 -07:00
Ivan Kozlovic
3977ea04f5 Fixed possible panic on consumer.Delete() during server Shutdown()
The panic was caused by the closing of an already closed Go channel.
The Delete() relied on the consumer's mset being nil to consider
the consumer already closed. However, the consumer's mset is set
to nil after invoking sendDeleteAdvisoryLocked() which internally
invokes sendAdvisory() which releases/reacquires the consumer lock.
This left an open door for a race to occur and Delete() to be
invoked twice on the same consumer.

Moving setting the consumer's mset to nil too early would prevent
the sendAdvisory() to actually do its job. We could pass the mset
to sendAvisory(), but a simpler approach is to simply use a "closed"
boolean on the Consumer object that is set to true at the beginning
of the Delete() function.

Resolves #1621

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-09-29 10:01:47 -06:00
Derek Collison
7f44d075f7 Merge pull request #1601 from ripienaar/skip_ack_on_acknxt
Do not ack to AckNxt
2020-09-28 17:19:35 -07:00
Derek Collison
9129d7b62f Set to 1 if no acks received, fixes #1619
Signed-off-by: Derek Collison <derek@nats.io>
2020-09-28 15:56:50 -07:00
R.I.Pienaar
cd376aa753 Do not ack to AckNxt
This causes nil messages mid content stream and is confusing,
the test for this is amended to test the content received for
validity - where previously the test assumed any content is
good content but in fact it only received half the content.

Removing this behaviour until we can design this properly

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-09-21 17:28:53 +02:00
Derek Collison
bcbf28fb3a Make sure to clear messages from stream when consumer deleted
Signed-off-by: Derek Collison <derek@nats.io>
2020-09-03 11:41:14 -07:00
Derek Collison
3faa2e7637 Fix for #1580
Signed-off-by: Derek Collison <derek@nats.io>
2020-09-02 17:18:28 -07:00
Derek Collison
6e1a892740 Merge pull request #1531 from ripienaar/max_consumers
allow max consumers to be set
2020-07-29 09:09:11 -07:00
R.I.Pienaar
b80e40340e allow account limits to be applied to unlimited streams
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-07-27 19:02:37 +02:00
Derek Collison
154d4303a9 Add in consumer rate limits
Signed-off-by: Derek Collison <derek@nats.io>
2020-07-24 10:11:32 -07:00
Derek Collison
48b2335cbf Fix for stream delete with many consumers - https://github.com/nats-io/jetstream/issues/260
Signed-off-by: Derek Collison <derek@nats.io>
2020-07-22 07:35:38 -07:00
Derek Collison
ad21074177 Merge pull request #1508 from nats-io/eo
Provide exactly once semantics
2020-07-09 16:27:56 -07:00
Derek Collison
b74c2eb2c4 Provide exactly once semantics
Signed-off-by: Derek Collison <derek@nats.io>
2020-07-08 16:56:52 -07:00
Derek Collison
aecdca874f Pending timers could go negative, this is a fix for #1502
Signed-off-by: Derek Collison <derek@nats.io>
2020-07-06 14:59:05 -07:00
Derek Collison
afc7fc367b Remove hdrs for now, find better way to deliver in client
Signed-off-by: Derek Collison <derek@nats.io>
2020-06-02 07:10:23 -07:00
Derek Collison
4d62a7237d Allow redelivery for AckAll policy, general upgrades to pending beahviors. Fixes #1436
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-31 07:50:50 -07:00
Derek Collison
dbde2479c2 Add in headers to consumer delivered messages
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-30 15:03:54 -07:00
Derek Collison
eca04c6fce First pass header support for JetStream
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-30 10:04:23 -07:00
Derek Collison
10e49ca1c4 Fix more flappers
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-28 14:19:11 -07:00
Derek Collison
8c9a30b8a1 Avoid deadlock by releasing consumer lock
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-28 12:30:12 -07:00
Derek Collison
d3ac95a5e6 Add in a terminate delivery for https://github.com/nats-io/jetstream/issues/189
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-28 08:32:34 -07:00
Derek Collison
4c91b69c4f Merge pull request #1413 from nats-io/fix_flappers
Fix flappers
2020-05-26 09:01:04 -07:00
Derek Collison
3caf6265d4 Properly recover ephemeral consumers after restart
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-25 11:06:55 -07:00
Derek Collison
f02ec860b6 Fix race when active state has not been updated yet
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-25 07:30:58 -07:00
Derek Collison
e42dc4915f Write consumer state during stop if not deleting
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-23 13:07:56 -07:00
R.I.Pienaar
c407a3b9d7 create advisories for stream and consumer add/delete/modify
We now publish advisories when streams and consumers are added,
deleted and modified

Also rework how TypedEvents are created to be easier to use

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-22 15:20:16 +02:00
Derek Collison
476e339cf9 Race detection fix
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 16:36:08 -07:00
Derek Collison
d3a447e6e9 Be more efficient when cleaning up all consumers for a stream
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
5256781ba0 Addressed issues raised in https://github.com/nats-io/nats-server/pull/1369
API still only turned on for account info in disabled accounts. Issues with advisories. Plan is still to have all endpoints on in all accounts.
Stream list and Consumer list return names only, page limit increased to 1024.
Stream, Consumer and Template names limited to 256 for now.
Subject API for stream messages, delete and get, not have STREAM.MSG.
Subject API for Durable is now CONSUMER.DURABLE.
Subject API for Templates now STREAM.TEMPLATE.
All subject APIs for list reverted back, so STREAM.LIST, CONSUMER.LIST etc.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
cadd39a01c Major rewrite for NATS JetStream API
API made more consistent. Noun followed by verb.
Name arguments in request subejcts are always at the end now.
Remove enabled call, just use account info.
Getting a message directly from a stream is treated like an admin API and requires JSON request.
Deleting a message directly as well.
StreamList and ConsumerList now include details and support paging.
Streams and Consumers now contain a created field in their info.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
R.I.Pienaar
63845b8577 add type hints to service latency, use time.Time for timestamp
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:26:46 -07:00
Derek Collison
513bff051b Redesign message-based API
Removed usage of +OK and -ERR. All responses are valid json objects now and optionally can include an ApiError which will have Code and Description.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
36d5421318 Don't do sample ack unless msg was on pending
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
50443ead89 Change to consumer info, removed state, made pending and redelivery counts
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
ae8bc2d22b Move ack to +ACK, +OK still works
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00