Use monitor check for streams like consumers.
Make sure to stop raft layer if exiting monitorConsumer early.
Allow consumers to force a snapshot on leadership change.
Signed-off-by: Derek Collison <derek@nats.io>
This PR has general improvements and fixes to filestore, raft, and the
clustering layer.
Summary
1. Additional support for preAck handling for interest based streams
when replicated acks arrive before the message itself.
2. Better handling when checking state to determine whether to remove an
interest based message.
3. Improved StepDown() and leadership transfer handling after restarts.
4. Improved voting logic for high load systems.
5. Various improvements and fixes for filestore Compact(), which is used
heavily in the raft layer when updating snapshots and the raft wal.
Signed-off-by: Derek Collison <derek@nats.io>
1. If we are retention based, make sure our consumers are running before entering into monitorStream logic.
2. If we skip messages and are interest based, make sure we check for a preAck state.
3. On finalization of recovery for consumers have them check against the interest based stream.
4. Do not process ack state updates if consumer is closed and shutting down.
5. When processing final state for a stream after upper layer catchup, check all attached consumers for ack skew.
6. During catchup of stream messages consult preAck state and skip messages as needed.
Signed-off-by: Derek Collison <derek@nats.io>
1. During ackMsg processing hold write lock to block concurrent access.
2. Check for presence of preAcks before and force removal if present.
3. Rework check for orphan msgs on startup to use checkStateForInterestStream().
Signed-off-by: Derek Collison <derek@nats.io>
1. Do not process an ack if we are closed.
2. When checking for needing an ack for a given consumer, hold lock entire time.
3. During recovery and restarts we check if we need to replay acks to the parent stream.
Signed-off-by: Derek Collison <derek@nats.io>
1. If reset ignore Applied() that are greater then our commit.
2. Improved StepDown() by placing at back of queue if preferred.
3. Improved handling of leadership transfer during StepDown().
4. Do not store EntryLeaderTransfer records on disk.
5. Remove un-needed processing of older terms.
6. If append entry has higher term, also inherit pterm.
7. Only inherit a candidate's term if we decide to vote for them.
Signed-off-by: Derek Collison <derek@nats.io>
1. Fixed a bug that would process a removal of a message after the message block was closed.
2. Improved removal of non-existant message when we know the store is empty.
3. Improved last write index size tracking when opening the file descriptor after being closed.
4. Improved Compact() by not loading messages for last block twice.
5. Improved Compact() determination of calling purge by determing last sequence under write lock.
6. Improved Compact() by only compacting underlying message block if over certain size threshold.
7. Improved Compact() by writing the index file if needed while still holding lock avoiding an unecessary re-lock.
8. Improved Compact() by not calling out to upper layers on no messages being purged.
9. Fixed a bug in Compact() that would not delete members from a block's delete map.
10. Fixed a bug in reset() when a callback was not registered (raft logs) which avoiding msg block cleanup.
11. Improved consumer store Update() call for when to avoid an outdated update.
Signed-off-by: Derek Collison <derek@nats.io>
Under asymmetric network latency based clusters, if a node in an R3 was replicating a consumer and the parent stream, but was the leader of neither, but the path from the stream leader was faster then the consumer leader a replicated ack could arrive before the message itself.
In this case we used to forward a delete message request to the stream leader which would then replicate that to all stream replicas, causing more work which could lead to increased publisher times on clients connected to the slow node.
Signed-off-by: Derek Collison <derek@nats.io>
