Commit Graph

693 Commits

Author SHA1 Message Date
Matthias Hanel
06e4e93185 Speeding up unit test fail case
Signed-off-by: Matthias Hanel <mh@synadia.com>
2020-11-20 13:57:34 -05:00
Matthias Hanel
f24aec4af7 Incorporating comments and fixed an issue where code was not invoked
had to change failing tests to use insecure as to not fail due to the
outgoing connection being not trusted.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2020-11-20 13:57:34 -05:00
Matthias Hanel
74642e024e [Added] verify_and_implicit_allow to tie subject alt name to url in cfg
Only works for gateways and routes. When true the subject alt DNS name
must match one url in the corresponding configuration

Signed-off-by: Matthias Hanel <mh@synadia.com>
2020-11-20 13:57:34 -05:00
Derek Collison
28d8a4be9c Merge pull request #1721 from nats-io/fsfix2
Fix for not properly recovering first sequence number on recovery.
2020-11-17 15:22:02 -08:00
Derek Collison
d358aaddf6 Fixes for filestore not remember first sequence when all messages deleted.
Thsi fixed a few minor bugs as well as the one where we did not remember our sequence numbers.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-17 15:04:57 -08:00
Ivan Kozlovic
5e9bad5b26 Test showing issue with filestore after a restart
I have pin-pointed that the issue started at this commit:
d737ccef55

This was PR: https://github.com/nats-io/nats-server/pull/1685

Prior to this PR, the test would pass.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-11-17 09:01:36 -07:00
R.I.Pienaar
ca8cbcdc63 support subjects starting with {
still doesnt support a subject {}

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-11-17 16:33:22 +01:00
R.I.Pienaar
a4116cc0fa support json requests for stream lookup 2020-11-17 15:33:44 +01:00
Derek Collison
67f4bf1ac1 Make sure to honor MaxAckPending when streaming directly to consumers
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-16 16:51:34 -08:00
Derek Collison
ce42addfe4 Merge pull request #1711 from nats-io/stream_lookup
Allow stream lookup by subject.
2020-11-16 05:59:13 -08:00
Derek Collison
5ff28b6087 Allow stream lookup by subject.
Allow an API endpoint and public API to lookup a stream by subject. The subject needs to be an exact match or a subset. If the subject is considered a filtered subject for the stream that will also be returned.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-14 13:35:05 -08:00
Derek Collison
9893aac8cb Add support for MaxAckPending
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-14 06:28:07 -08:00
Derek Collison
28cb4e8c34 Fix bug when removing the same message from a stream.
We would release locks and call into upper layers when removing a message. The upper layers may call back into the lower layers to get more information, such as the subject.
This fix has the storage updates optionally supply the subject for filtered consumers and fixes the bug of double deletes.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-13 17:05:24 -08:00
Derek Collison
e57f6c9dee Larger AckWait to prevent flapping
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-13 08:06:43 -08:00
Derek Collison
e6797efde7 Make interest notifications explicit match only
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-12 07:10:47 -08:00
Derek Collison
a733c12086 Fix flapper, be pedantic about signaling on delivery subject update
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-11 18:07:51 -08:00
Derek Collison
e31001a782 Remove conditional and broadcast for signalling consumers
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-11 13:53:45 -08:00
Derek Collison
164f9fdf2b Updates to consumer store to support delta updates.
In preparation for clustering we need to have the consumer filestore update state with deltas vs original design.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-10 19:16:55 -08:00
Derek Collison
398ef78aac Fix flappers
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-05 18:10:52 -08:00
Derek Collison
a4abbea0d2 Make sure to set timer for additional redeliveries
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-05 14:07:17 -08:00
Derek Collison
4a26b1f514 Redeliver pending right away on change of delivery subject for a consumer.
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-05 13:20:27 -08:00
Derek Collison
67b1a33401 Fix for partial cache overwrite bug.
When we moved to a write through cache architecture we also moved the cache write to offset based instead of APPEND.
We were inadvertently clearing our offset from our cache when we would clear which meant if the next operation was another write we would have the wrong offset and overwrite previous messages.

Signed-off-by: Derek Collison <derek@nats.io>
2020-11-04 20:01:53 -08:00
Ivan Kozlovic
5a6bb37564 Test that shows message disappear from filestore
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-11-04 15:37:38 -07:00
R.I.Pienaar
c0f031cc39 support next structure in acknxt
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-11-02 17:47:19 +01:00
Derek Collison
d729beb24c Flush after purge too
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-01 13:21:53 -08:00
Derek Collison
65caff18c3 Fix for flapper and additional consumer perf test
Signed-off-by: Derek Collison <derek@nats.io>
2020-11-01 12:06:45 -08:00
R.I.Pienaar
96c53b77f5 Merge pull request #1683 from ripienaar/bare_acknxt
ensure bare AckNxt requesting >1 is supported
2020-10-30 16:15:20 +01:00
R.I.Pienaar
bf68068fa6 address review
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-10-30 15:27:42 +01:00
R.I.Pienaar
0e9ca7614f ensure bare AckNxt requesting >1 is supported
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-10-30 14:46:39 +01:00
Derek Collison
cce7195a2c Fixed bug that would not properly handle wildcard streams with interest retention and filtered consumers.
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-29 13:14:17 -07:00
Derek Collison
0f22e99601 Merge pull request #1675 from nats-io/filestore
Stability and performance updates for filestore.
2020-10-28 19:16:44 -07:00
R.I.Pienaar
a88c2ee349 rename consumer pending fields for clarity
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-10-28 16:12:01 +01:00
Derek Collison
fe2b354414 Stability and performance updates.
The original design had a shared filestore write buffer and individual message blocks had a read cache.
This presented some performance and stability issues when lots of reads and deletes were happening to a
message block that was also being written to actively.

This change eliminates the shared write buffer and uses the message block's cache as a write through as
well as read cache and handles partials correctly.

Signed-off-by: Derek Collison <derek@nats.io>
2020-10-28 07:58:47 -07:00
Derek Collison
df4ee081a5 Track number of stream pending for each consumer.
This will track the stream pending state for each consumer.
This code does account for filtered consumers.

Signed-off-by: Derek Collison <derek@nats.io>
2020-10-27 19:30:42 -07:00
Derek Collison
987892b260 Purge clears dedupe window as well
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-23 05:52:05 -07:00
Derek Collison
63477acb61 Merge pull request #1658 from nats-io/waitq
[JetStream] Upgrade for wait queues for pull based consumers
2020-10-22 09:59:57 -07:00
Derek Collison
511cdaa5ac Do not report bad latency on auto-unsubscribe triggers
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-22 05:41:33 -07:00
Derek Collison
3b18f188ed Switched behavior to never refuse new request, and to alert when expiring ones with interest
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-21 10:35:29 -07:00
Derek Collison
705f8b6ed7 Do not forward service import messages to a stream export.
Addresses stack overflow issue wally was seeing with configs
that mix and match streams and services between each other.

Signed-off-by: Derek Collison <derek@nats.io>
2020-10-20 19:20:44 -07:00
Derek Collison
c2b8de377c Formalize requests for next msg, support NoWait and Expires
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-20 16:25:19 -07:00
Derek Collison
610d2d21b7 More robust waiting queue for pull mode consumers
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-19 19:51:46 -07:00
Ivan Kozlovic
3b8d00e046 [FIXED] Possible panic when server accepts TLS leafnode connection
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-10-19 10:29:32 -06:00
R.I.Pienaar
f2106ab9e4 fix npe for offsets larger than items
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-10-15 11:27:31 +02:00
Ivan Kozlovic
26cd1f99ab [FIXED] Stream's subscription propagation issue with gateways
When creating shadow subscriptions for import streams, we were
not invoking code for gateway subscription accounting, which means
that when the account (for leafnodes) was switched to interest
only, those shadow subscriptions were not sent.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-10-13 18:10:27 -06:00
Ivan Kozlovic
df9d5f5fd9 Accepting route warns if remote server has same name
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-10-08 17:59:33 -06:00
Derek Collison
cdfb7ba03b Remove delivery count check, could flap
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-02 16:30:40 -07:00
Derek Collison
9ad408e0b3 Use closed variable on consumer, don't check consumer sequences
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-02 16:21:23 -07:00
Derek Collison
8a9f6eaf42 Additional fixes to interest retention based streams and offline durables with redelivery.
Signed-off-by: Derek Collison <derek@nats.io>
2020-10-02 12:58:59 -07:00
Derek Collison
a75be04b0a Various fixes for this PR.
- Fix for updating delivery subject and adjusting next delivery sequences.
- When acking explicitly but out of order, need to make sure we set floor correctly.
- Only update ack floors on an ack if the message is present.
- Fix for needAck for explicitAck out of order consumers detecting if message has been acked.
- Fix for race not locking stream when checking interest during stop.
- Fix for filestore determing if a message block still has a message. Added check to first sequence as well as cache.
- Some additions to the original test.

Signed-off-by: Derek Collison <derek@nats.io>
2020-10-01 21:24:40 -07:00
Ivan Kozlovic
eff27e26be AckExplicit removes message for "offline" durable
The test shows the issue.
It seems that Consumer.needAck() for AckExplicit should consider
that an Ack is needed if sseq > o.asflr and there is no pending
ack at all. However making this change would break the test
TestJetStreamInterestRetentionStream.

Also, running the new test with `-count 100` and by adding an
artificial delay in stream.ackMsg() (just before calling
mset.store.RemoveMsg(seq)) causes sometimes delete requests
for the same sequence to be processed twice, which causes
the new test to fail (even with an attempted fix as discussed
above). I think that the attempt to remove the same sequence twice
is messing up the state.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-10-01 12:22:02 -06:00