b9b284dffa
Historically we kept indexing information, either by sequence or by subject, as a per msg block operation. These were the "*.idx" and "*.fss" indexing files. When streams became very large this could have an impact on recovery time. Also, for encryption the fast path for determining if the indexing was current would require loading and decrypting the complete block. This design moves to a more traditional WAL and snapshot approach. The snapshots for the complete stream, including summary information, global per subject information maps (PSIM) and per msg block details including summary and dmap, are processed asynchronously. The snapshot includes the msg block and has for the last record hash that was considered in the snapshot. On recovery the snapshot is read and processed and any additional records past the point of the snapshot itself are processed. To this end, any non-system removal of a message has to be expressed as a delete tombstone that is always added the the fs.lmb file. These are processed on recovery and our indexing layer knows to skip them. Changing to this method drastically improves startup and recovery times, and has simplified the code. Some normal performance benefits have been seen as well. Signed-off-by: Derek Collison <derek@nats.io>
NATS is a simple, secure and performant communications system for digital systems, services and devices. NATS is part of the Cloud Native Computing Foundation (CNCF). NATS has over 40 client language implementations, and its server can run on-premise, in the cloud, at the edge, and even on a Raspberry Pi. NATS can secure and simplify design and operation of modern distributed systems.
Documentation
- Official Website
- Official Documentation
- FAQ
- Watch a video overview of NATS.
- Watch this video from SCALE 13x to learn more about its origin story and design philosophy.
Contact
- Twitter: Follow us on Twitter!
- Google Groups: Where you can ask questions
- Slack: Click here to join. You can ask question to our maintainers and to the rich and active community.
Contributing
If you are interested in contributing to NATS, read about our...
Roadmap
The NATS product roadmap can be found here.
Security
Security Audit
A third party security audit was performed by Cure53, you can see the full report here.
Reporting Security Vulnerabilities
If you've found a vulnerability or a potential vulnerability in the NATS server, please let us know at nats-security.
License
Unless otherwise noted, the NATS source files are distributed under the Apache Version 2.0 license found in the LICENSE file.