Underestimated the effort to get stream restore working properly in cluster mode.
Some good bug fixes and stability improvments.
Signed-off-by: Derek Collison <derek@nats.io>
I noticed that some consumer go routines were left running at the end
of the test suite.
It turns out that there was a race the way the consumer's qch was closed.
Since it was closed and then set to nil, it is possible that the go
routines that are started and then try to capture o.qch would actually
get qch==nil, wich then when doing a select on that nil channel would
block forever.
So we know pass the qch to the 2 go routines loopAndGatherMsgs() and
loopAndDeliverMsgs() so that when we close the channel there is
no risk of that race happening.
I do believe that there is still something that should be looked at:
it seems that a consumer's delivery loop can now be started/stopped
many times based on leadership acquired/lost. If that is the case,
I think that the consumer should wait for previous go routine to
complete before trying to start new ones.
Also moved 3 JetStream tests to the test/norace_test.go file because
they would consumer several GB of memory when running with the -race flag.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Just increased the AckWait from 20ms to 100ms and reduced max
deliveries from 4 to 3.
I believe that there is still the risk that the message is redelivered
while the server is being shutdown and that message is not making it
to the sub.
But using those new values (100ms/3), I have ran 200 rounds on a Linux
VM and did not get the failure (but did before the change).
Again, this is not proper test fix, but may help. This test has been
failing 11 times already (keeping track in spreadsheet) and causes
several minutes of tests to have to be recycled.
Note that the test ran in about 0.4s and now 0.7s, so not that much
and would be worth the added delay if it helps not breaking the whole
test suite!
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Had a deadlock with new preconditions. We need to hold lock across Store() call but that call could call into storeUpdate() such that we may need to acquire the lock. We can enter this callback from the storage layer itself and the lock would not be held so added an atomic.
Signed-off-by: Derek Collison <derek@nats.io>
When a system account was configured and not the default when we did a reload we would lose the JetStream service exports.
Signed-off-by: Derek Collison <derek@nats.io>
Made several changes based on feedback.
1. Made PubAckResponse only optionally include an ApiError and not force an API type.
2. Allow FilterSubject to be set on a consumer config and cleared if it matches the only stream subject.
3. Remove LookupStream by subject, and add in filters for stream names API.
Signed-off-by: Derek Collison <derek@nats.io>
Allow an API endpoint and public API to lookup a stream by subject. The subject needs to be an exact match or a subset. If the subject is considered a filtered subject for the stream that will also be returned.
Signed-off-by: Derek Collison <derek@nats.io>
We would release locks and call into upper layers when removing a message. The upper layers may call back into the lower layers to get more information, such as the subject.
This fix has the storage updates optionally supply the subject for filtered consumers and fixes the bug of double deletes.
Signed-off-by: Derek Collison <derek@nats.io>
In preparation for clustering we need to have the consumer filestore update state with deltas vs original design.
Signed-off-by: Derek Collison <derek@nats.io>
When we moved to a write through cache architecture we also moved the cache write to offset based instead of APPEND.
We were inadvertently clearing our offset from our cache when we would clear which meant if the next operation was another write we would have the wrong offset and overwrite previous messages.
Signed-off-by: Derek Collison <derek@nats.io>
