gogrlx/nats-server
1
0
Fork 0
mirror of https://github.com/gogrlx/nats-server.git synced 2026-04-02 03:38:42 -07:00
Code Issues Packages Projects Releases Wiki Activity
8,379 Commits 30 Branches 48 Tags
842d600e3faae31ead3ab97e70b8747f27c626b2
Commit Graph

407 Commits

Author SHA1 Message Date
Derek Collison
842d600e3f Grab blk fn while mb lock held 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-10-11 07:54:36 -07:00
Derek Collison
f4387ec74e Fix for compaction with compression and added an out of band compaction in syncBlocks to reclaim more space. 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-10-10 17:17:55 -07:00
Derek Collison
dd646f6b71 Set initial min on dmap caused subtle bugs with dmap. Some minor cleanup. 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-10-06 09:42:09 -07:00
Derek Collison
dba03dbc2f Optimizations to reduce contention for high connections in a JetStream enabled account with high API usage. 
Several strategies which are listed below.

1. Checking a RaftNode to see if it is the leader now uses atomics.
2. Checking if we are the JetStream meta leader from the server now uses an atomic.
3. Accessing the JetStream context no longer requires a server lock, uses atomic.Pointer.
4. Filestore syncBlocks would hold msgBlock locks during sync, now does not.

Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-30 14:52:15 -07:00
Derek Collison
cb74f3f26e Add in additional warning when subject skew detected 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-28 14:16:27 -07:00
Derek Collison
b0743ec059 Additional markers for dirty state 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-27 20:32:17 -07:00
Derek Collison
aeef0eff53 Add in warnings for filestore recover state if happy path fails. 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-27 16:22:15 -07:00
Derek Collison
2716248b5e Protect against going upside down on mb.msgs. 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-22 14:17:45 -07:00
Derek Collison
4824085171 Hold lock here as well 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-22 10:25:58 -07:00
Derek Collison
21e81a2961 We will panic since we unlock the mb in removeMsgBlock 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-21 18:34:38 -07:00
Derek Collison
acffa0668a Various fixes and improvements to tombstone and buffer gaps. 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-17 19:49:01 -07:00
Derek Collison
787f6acf31 Fix for a call into fs.recalculateFirstForSubj() from fs.recalculateFirstForSubj() that did not lock the mb properly. 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-13 15:35:34 -07:00
Derek Collison
244ff4489c Fix for panic from a bug in selecting a block and an index when num blocks > 32 and we used new binary search in NumPending(). 
The reason would be that we were not accounting for gaps as mb.first.seq can move. The behavior should always return a valid index and mb if seq is inclusive of range from first to last.
The panic could orphan held locks for filestore, consumer and possibly stream.

Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-12 15:20:09 -07:00
Neil Twigg
25564087f9 Fix race condition in hashing stream snapshots 
Signed-off-by: Neil Twigg <neil@nats.io>
 2023-09-06 11:32:43 +01:00
Derek Collison
e7e8a330d4 Allow sync intervals to be set and the ability to have all data writes synchronous. 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-04 11:05:13 -07:00
Derek Collison
1768f9c770 Always check for orphaned fss and idx files 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-03 14:31:55 -07:00
Derek Collison
2d2bb77f6e Optimize for restore time. 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-03 14:14:00 -07:00
Derek Collison
e11ddb8bfe Merge branch 'main' into dev 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-02 14:22:57 -07:00
Derek Collison
34ae2bf4cb Fix for a bug that would make normal streams use the wrong block size. 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-02 13:56:34 -07:00
Derek Collison
ad380d48f2 Merge branch 'main' into dev 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-01 11:19:33 -07:00
Derek Collison
4422a95a8e We should update accounting before clearing ebit 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-09-01 09:31:12 -07:00
Derek Collison
4df5f515ca Fix for filestore data race on hash during snapshots 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-31 19:38:09 -07:00
Derek Collison
83fab5c9a7 [FIXED] Unlock panic on start when filestore needs to remove msgs for enforcement. (#4469) 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-31 19:26:03 -07:00
Derek Collison
0ec42f85f0 Fix for merge issue that duplicated the index increment, causing snapshot tests to fail 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-31 18:51:34 -07:00
Derek Collison
60fa2d8781 Only have removeMsg release lock if it really has a callback. 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-31 16:56:40 -07:00
Derek Collison
9ff3261af2 On startup make sure to hold lock for enforcing limits due to removeMsg() needing to remove msgs possibly. 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-31 16:56:35 -07:00
Derek Collison
2bfa14d9bd Fix from main merge 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-31 15:52:36 -07:00
Derek Collison
49c30b6d2f Merge branch 'main' into dev 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-31 15:52:00 -07:00
Derek Collison
c110ceea94 Check for checksum violations for all records and before sequence processing. 
Also fix for bitrot test and a small bug fix for a leaking fd.

Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-31 13:53:28 -07:00
Derek Collison
2834142bdd Revert lock guard 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-31 08:59:15 -07:00
Derek Collison
0bd4763584 Revert lock guard 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-31 08:58:22 -07:00
Derek Collison
fbaed8f220 Merge branch 'main' into dev 2023-08-31 08:29:30 -07:00
Derek Collison
2b677c231a Unlock needed to be guarded 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-31 08:16:47 -07:00
Derek Collison
9e26574707 Make sure we unlock only if we did not acquire 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-31 08:09:16 -07:00
Neil Twigg
af2ff3d17d Fix lock issue in filestore 
Signed-off-by: Neil Twigg <neil@nats.io>
 2023-08-31 15:16:15 +01:00
Derek Collison
abae24086c Make sure order correct 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-30 16:13:56 -07:00
Derek Collison
adef8281a2 Updates to the way meta indexing is handled for filestore. 
Historically we kept indexing information, either by sequence or by subject, as a per msg block operation. These were the "*.idx" and "*.fss" indexing files. When streams became very large this could have an impact on recovery time. Also, for encryption the fast path for determining if the indexing was current would require loading and decrypting the complete block.

This design moves to a more traditional WAL and snapshot approach. The snapshots for the complete stream, including sumary information, global per subject information maps (PSIM) and per msg block details including summary and dmap, are processed asynchronously. The snapshot includes the msh block and has for the last record considered in the snapshot. On recovery the snapshot is read and processed and any additional records past the point of the snapshot itself are processed. To this end, any removal of a message has to be expressed as a delete tombstone that is always added the the fs.lmb file. These are processed on recovery and our indexing layer knows to skip them.

Changing to this method drastically improves startup and recovery times, and has simplified the code. Some normal performance benefits have been seen as well.

Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-30 16:12:45 -07:00
Derek Collison
8841432d03 Allow 2.10 tombstones to be skipped and allow us to recover on downgrade from 2.10 to 2.9. 
Also fixed small bug that could set bad first seq.

Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-30 14:38:41 -07:00
Derek Collison
7f884062d1 Merge branch 'main' into dev 2023-08-29 20:01:26 -07:00
Derek Collison
e4a1b81d30 Fix on rebuild first when rebuild results in empty block (from dev branch) 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-29 19:50:33 -07:00
Derek Collison
acfb593ed5 Merge branch 'main' into dev 2023-08-29 16:48:04 -07:00
Derek Collison
8865c2a703 Fix for update to max msgs per where recalculating first was not checking for seq < mb.first.seq 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-29 16:02:52 -07:00
Derek Collison
f50b772a14 Merge branch 'main' into dev 2023-08-27 14:20:45 -07:00
Derek Collison
b66a7f6e9f When expiring complete blocks make sure to update global subject index psim. 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-27 12:03:44 -07:00
Derek Collison
f1bf4127c5 Merge branch 'main' into dev 2023-08-25 11:03:54 -07:00
Derek Collison
22ed97c6c9 Fix for purge with keep bug and improved search for large number of blocks. 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-25 08:59:47 -07:00
Derek Collison
346c22788e Merge branch 'main' into dev 2023-08-24 16:20:46 -07:00
Derek Collison
48bf7ba151 When a consumer reached a max delivered condition, we did not properly synchronize the state such that on a restore or leader switch the ack pending could jump and be higher than max ack pending and block the consumer. 
This propagates a delivered update and we updated the store state engine to do the right thing when the condition is reached.

Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-24 16:00:27 -07:00
Derek Collison
3b235059fa We were trying to be too smart to save space at the expense of encoding time for filestore. 
Revert back to very simple but way faster method. Sometimes 100x faster and only ~8% size increase.

Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-05 12:33:30 -07:00
Derek Collison
42752ec551 Merge branch 'main' into dev 
Signed-off-by: Derek Collison <derek@nats.io>
 2023-08-01 21:46:54 -07:00