Commit Graph

155 Commits

Author SHA1 Message Date
Derek Collison
cdfb7ba03b Remove delivery count check, could flap
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-02 16:30:40 -07:00
Derek Collison
9ad408e0b3 Use closed variable on consumer, don't check consumer sequences
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-02 16:21:23 -07:00
Derek Collison
8a9f6eaf42 Additional fixes to interest retention based streams and offline durables with redelivery.
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-02 12:58:59 -07:00
Derek Collison
a75be04b0a Various fixes for this PR.
- Fix for updating delivery subject and adjusting next delivery sequences.
- When acking explicitly but out of order, need to make sure we set floor correctly.
- Only update ack floors on an ack if the message is present.
- Fix for needAck for explicitAck out of order consumers detecting if message has been acked.
- Fix for race not locking stream when checking interest during stop.
- Fix for filestore determing if a message block still has a message. Added check to first sequence as well as cache.
- Some additions to the original test.

Signed-off-by: Derek Collison <derek@nats.io>
2020-10-01 21:24:40 -07:00
Ivan Kozlovic
eff27e26be AckExplicit removes message for "offline" durable
The test shows the issue.
It seems that Consumer.needAck() for AckExplicit should consider
that an Ack is needed if sseq > o.asflr and there is no pending
ack at all. However making this change would break the test
TestJetStreamInterestRetentionStream.

Also, running the new test with `-count 100` and by adding an
artificial delay in stream.ackMsg() (just before calling
mset.store.RemoveMsg(seq)) causes sometimes delete requests
for the same sequence to be processed twice, which causes
the new test to fail (even with an attempted fix as discussed
above). I think that the attempt to remove the same sequence twice
is messing up the state.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-10-01 12:22:02 -06:00
Ivan Kozlovic
3977ea04f5 Fixed possible panic on consumer.Delete() during server Shutdown()
The panic was caused by the closing of an already closed Go channel.
The Delete() relied on the consumer's mset being nil to consider
the consumer already closed. However, the consumer's mset is set
to nil after invoking sendDeleteAdvisoryLocked() which internally
invokes sendAdvisory() which releases/reacquires the consumer lock.
This left an open door for a race to occur and Delete() to be
invoked twice on the same consumer.

Moving setting the consumer's mset to nil too early would prevent
the sendAdvisory() to actually do its job. We could pass the mset
to sendAvisory(), but a simpler approach is to simply use a "closed"
boolean on the Consumer object that is set to true at the beginning
of the Delete() function.

Resolves #1621

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-09-29 10:01:47 -06:00
Derek Collison
7f44d075f7 Merge pull request #1601 from ripienaar/skip_ack_on_acknxt
Do not ack to AckNxt
2020-09-28 17:19:35 -07:00
Ivan Kozlovic
5506b96baa AddConsumer causes redelivery of first message
It seems that when updating the delivery subject, we use as the
first sequence the ack floor, but if no message was ever ack'ed
then it causes the first message to be redelivered twice.

Resolves #1619

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-09-28 12:28:57 -06:00
R.I.Pienaar
cd376aa753 Do not ack to AckNxt
This causes nil messages mid content stream and is confusing,
the test for this is amended to test the content received for
validity - where previously the test assumed any content is
good content but in fact it only received half the content.

Removing this behaviour until we can design this properly

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-09-21 17:28:53 +02:00
Ivan Kozlovic
7ccbaca782 Added an allowed connection type filter for users
Users and NKey users will now have the option to specify a list
of allowed connection types.

This will allow for instance a certain user to be allowed to
connect as a standard NATS client, but not as Websocket, or
vice-versa.

This also fixes the websocket auth override. Indeed, with
the original behavior, the websocket users would have been bound
to $G, which would not work when there are accounts defined, since
when that is the case, no app can connect/bind to $G account.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-09-16 18:22:44 -06:00
Derek Collison
bcbf28fb3a Make sure to clear messages from stream when consumer deleted
Signed-off-by: Derek Collison <derek@nats.io>
2020-09-03 11:41:14 -07:00
Derek Collison
3faa2e7637 Fix for #1580
Signed-off-by: Derek Collison <derek@nats.io>
2020-09-02 17:18:28 -07:00
Derek Collison
cdbea93a87 Interest based retention should not store if no consumers present.
Fix for https://github.com/nats-io/jetstream/issues/314

Signed-off-by: Derek Collison <derek@nats.io>
2020-09-02 16:04:38 -07:00
Derek Collison
3a61a0cef8 Fix for https://github.com/nats-io/jetstream/issues/297
Signed-off-by: Derek Collison <derek@nats.io>
2020-08-17 10:35:07 -07:00
Ivan Kozlovic
22833c8d1a Fix sysSubscribe races
Made changes to processSub() to accept subscription properties,
including the icb callback so that it is set prior to add the
subscription to the account's sublist, which prevent races.
Fixed some other racy conditions, notably in addServiceImportSub()

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-08-03 14:59:00 -06:00
Derek Collison
6e1a892740 Merge pull request #1531 from ripienaar/max_consumers
allow max consumers to be set
2020-07-29 09:09:11 -07:00
R.I.Pienaar
b80e40340e allow account limits to be applied to unlimited streams
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-07-27 19:02:37 +02:00
Derek Collison
154d4303a9 Add in consumer rate limits
Signed-off-by: Derek Collison <derek@nats.io>
2020-07-24 10:11:32 -07:00
R.I.Pienaar
ca0dc72841 allow max consumers to be set
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-07-24 17:30:41 +02:00
Derek Collison
48b2335cbf Fix for stream delete with many consumers - https://github.com/nats-io/jetstream/issues/260
Signed-off-by: Derek Collison <derek@nats.io>
2020-07-22 07:35:38 -07:00
R.I.Pienaar
5037f00b30 ensure unlimited account limits are calculated correctly
Previously unlimited accounts - ones who inherit server values - would
be unable to publish any messags at all

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-07-20 10:46:34 +02:00
Derek Collison
177ca51356 More checks on duplicate window config, signal in PubAck if duplicate
Signed-off-by: Derek Collison <derek@nats.io>
2020-07-10 07:02:34 -07:00
Derek Collison
ad21074177 Merge pull request #1508 from nats-io/eo
Provide exactly once semantics
2020-07-09 16:27:56 -07:00
Derek Collison
2b577b5197 Recover dedupe in memory structures on stream recovery
Signed-off-by: Derek Collison <derek@nats.io>
2020-07-09 16:13:20 -07:00
Derek Collison
b74c2eb2c4 Provide exactly once semantics
Signed-off-by: Derek Collison <derek@nats.io>
2020-07-08 16:56:52 -07:00
Derek Collison
aecdca874f Pending timers could go negative, this is a fix for #1502
Signed-off-by: Derek Collison <derek@nats.io>
2020-07-06 14:59:05 -07:00
Ivan Kozlovic
b9bd5c2d35 Fixed flappers
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-06-09 15:34:52 -06:00
Derek Collison
c969e7e424 Do proper ubsubscribe when shutting off restore endpoint
Signed-off-by: Derek Collison <derek@nats.io>
2020-06-04 08:58:14 -07:00
Derek Collison
164f44ed18 Require reply subjects for restore chunks
Signed-off-by: Derek Collison <derek@nats.io>
2020-06-04 06:56:07 -07:00
Derek Collison
660ea3c807 Snapshot restore now works across leafnodes.
This also introduces the ability to have flow control inbound for restoring a stream.
If the system detects a reply subject it will respond with a nil payload.
For the last EOF message if a reply is present it will respond with a stream info response or error.

Signed-off-by: Derek Collison <derek@nats.io>
2020-06-03 20:00:59 -07:00
Derek Collison
afc7fc367b Remove hdrs for now, find better way to deliver in client
Signed-off-by: Derek Collison <derek@nats.io>
2020-06-02 07:10:23 -07:00
R.I.Pienaar
920dd4269a fix argument order in snapshots
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-06-02 13:51:50 +02:00
Derek Collison
f6ce833751 Fix flapper
Signed-off-by: Derek Collison <derek@nats.io>
2020-06-01 11:37:06 -07:00
Derek Collison
4d62a7237d Allow redelivery for AckAll policy, general upgrades to pending beahviors. Fixes #1436
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-31 07:50:50 -07:00
Derek Collison
e584d4efee Merge pull request #1435 from nats-io/js-hdrs
First pass header support for JetStream
2020-05-31 06:01:01 -07:00
Derek Collison
dbde2479c2 Add in headers to consumer delivered messages
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-30 15:03:54 -07:00
Derek Collison
eca04c6fce First pass header support for JetStream
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-30 10:04:23 -07:00
Derek Collison
e12907ffa6 Allow snapshots to optionally check all message checksums
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-29 09:57:33 -07:00
Derek Collison
0a206b4c64 Snapshot performance tweaks
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-29 08:07:31 -07:00
Derek Collison
10e49ca1c4 Fix more flappers
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-28 14:19:11 -07:00
Derek Collison
4ca05d9719 Fix gap test from flapping
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-28 13:43:15 -07:00
Derek Collison
625129f20a Fix flapper test where no messages to receive at end
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-28 13:23:00 -07:00
Derek Collison
d3ac95a5e6 Add in a terminate delivery for https://github.com/nats-io/jetstream/issues/189
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-28 08:32:34 -07:00
Derek Collison
bc0fedbaba Udpates based on PR feedback
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-28 06:08:35 -07:00
Derek Collison
fa59cff105 Add in snapshot and restore JSApi
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-27 20:01:30 -07:00
Derek Collison
8727315eb9 Updated snapshots, added restore, generic hashes
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-26 19:53:16 -07:00
Derek Collison
4c91b69c4f Merge pull request #1413 from nats-io/fix_flappers
Fix flappers
2020-05-26 09:01:04 -07:00
Derek Collison
710ef00383 Don't allow JetStream on system account. Warn when accounts configured but no JS
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-25 12:17:18 -07:00
Derek Collison
3caf6265d4 Properly recover ephemeral consumers after restart
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-25 11:06:55 -07:00
Derek Collison
54aa40b352 Wait a bit longer to get subs
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-25 09:30:36 -07:00