From fcf5c61997eb859723c7261f7ebb2125d056b73d Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Wed, 22 May 2019 17:04:30 -0600 Subject: [PATCH 01/31] Update the introduction and 2.0 section with newer content Signed-off-by: Colin Sullivan --- README.md | 41 ++++++-- whats_new/whats_new_20.md | 216 ++++++++++++++++++++++++++++++++------ 2 files changed, 214 insertions(+), 43 deletions(-) diff --git a/README.md b/README.md index b9cbb91..aaaa734 100644 --- a/README.md +++ b/README.md @@ -1,24 +1,45 @@ # Introduction -NATS is simple and secure messaging made for developers and operators who want to spend more time developing applications than worrying about a messaging system. +## Why Messaging is Important + +Developing and deploying applications that communicate in distributed systems is complex and difficult. A communication infrastructure should provide features to make this easier, including multiple messaging patterns bundled into one technology, location transparency, the decoupling of data producers and consumers, and asynchronous communications to build event driven applications. + +### Distributed Computing Needs of Today + +A modern messaging system needs to support multiple communication patterns, be secure by default, support multiple qualities of service, and provide secure multi-tenancy for shared infrastructure. A modern system needs to include: + +* Secure by default communications for microservices, edge platforms and devices +* Secure multi-tenancy in a single distributed communication technology +* Decoupled from IP for addressing and security +* Resiliency with an emphasis on the health of the system as a whole +* Ease of use for agile development, CI/CD, and operations, at scale +* Highly scalable with built-in load balancing and no config auto-scaling +* Consistent identity and security mechanisms from edge devices to backend services + +## NATS + +NATS was built to meet the distributed computing needs of today. NATS is +simple and secure messaging made for developers and operators who want to +spend more time developing modern applications than worrying about a +distributed communication system. * Easy to use for developers and operators * High-Performance * Always on and available * Extremely lightweight * At Most Once (NATS) or At Least Once Delivery (NATS Streaming) -* Common Messaging Pattern Support +* Common Messaging Pattern Support (Services, Streams, Load Balancing) * Client support for over 30 different programming languages * Cloud Native, a CNCF project with Kubernetes and Prometheus integrations -NATS is used for: +### Use Cases + +NATS can run anywhere, from devices, to edge computers, to cloud. Use cases NATS include: * Cloud Messaging -* Microservices + * Services (microservices, service mesh) + * Event/Data Streaming (observability, analytics, ML/AI) * Command and Control -* Service Discovery -* Event Streaming -* IoT and Edge -* Telemetry / Sensor Data -* Mobile and Big Data -* Augmenting or Replacing Legacy Messaging + * IoT and Edge + * Telemetry / Sensor Data / Command and Control +* Augmenting or Replacing Legacy Messaging Systems diff --git a/whats_new/whats_new_20.md b/whats_new/whats_new_20.md index e7c8316..d26e70a 100644 --- a/whats_new/whats_new_20.md +++ b/whats_new/whats_new_20.md @@ -1,63 +1,213 @@ # What's New in NATS 2.0 -NATS 2.0 is the largest feature release since the original code base for the server was released. It includes features for security, multi-tenancy, larger networks, and secure sharing, allowing a new way of thinking about NATS as a globally shared utility and as a new and innovative way to solve on-premise communications for modern systems design. +NATS 2.0 is the largest feature release since the original code base for the +server was released. NATS 2.0 was created to allow a new way of thinking about +NATS as a shared utility, solving problems at scale through distributed +security, multi-tenancy, larger networks, and secure sharing of data. -## Multi-Tenancy and Accounts +## Rationale -Ever since NATS was originally designed, it has always been considered to be single tenant. Even when we added support for multiple users, the subject space was still a shared space, and care had to be taken when designing subject spaces so as to not interfere with other applications and services on the NATS systems. -As a matter of fact, designing the subject space, and most importantly the prefixes based on organization, or business unit or business function/service, was encouraged as an early exercise. However, even with the best intentions, large NATS systems became overly complex and managing and understanding the subject space was challenging at large scale. Even as we introduced authorization primitives to control which subjects users could publish or subscribe to, the challenge still remained. -NATS 2.0 introduces accounts. Accounts can be viewed as containers for a subject space. Instead of there being a global subject space that was bound to a server, NATS 2.0 now recognizes accounts, including the default global account $G that defines which data structures the server will utilize to match publishers to subscribers. These subject spaces are isolated and by default, no messages cross from one account to another. This allows multiple groups or organizations to use a simplified subject space that in the past, could have overlapped with others in a larger installation. It also allows for a single NATS infrastructure to be utilized by different users without the need to maintain many silos of NATS servers and clusters. -An account binds the subject space data structures, a sublist, and its associated caches, to a list of users. A client application is not aware of this account binding and this was by design to preserve existing clients and the simplicity of the NATS philosophy. When connected to a NATS server, the user is authenticated and validated, and the configuration tells the server which account to bind to the user. The messages then flow as normal, as published messages are matched to subscribers. +NATS 2.0 was created to address problems in large scale distributed computing. -## Streams and Services - Account Sharing +It is difficult at best to combine identity management end-to-end +(or end-to-edge), with data sharing, while adhering to policy and compliance. +Current distributed systems increase significantly in operational complexity +as they scale upward. Problems arise around service discovery, connectivity, +scaling for volume, and application onboarding and updates. +Disaster recovery is difficult, especially as systems have evolved to +operate in silos defined by technology rather than business needs. +As complexity increases, systems become expensive to operate. The become +fragile making it difficult to deploy services and applications hindering +innovation. -With the introduction of secure multi-tenancy in NATS 2.0, the desire for accounts expanded beyond just isolation. The NATS core team felt that this could possibly be used to simplify and ease the use of NATS when being utilized for many applications and services that want to communicate over certain subjects. Think of this as the microservices equivalent for NATS applications. In NATS 2.0 we define two abstract concepts that form the basis of sharing messages across accounts; Streams and Services. Streams are published messages defined by a subject, which could include a wildcard. Services are listening endpoints bound to subscribers that usually receive requests and return responses. +We decided to: -Both Streams and Services can be exported by an account owner for subsequent import by other accounts. There are two very important aspects of export and import that make account sharing even more powerful. First, exports can be public, meaning anyone can import, or then can require authorization. An account that wants to import the stream or service must have permission from the exporting account owner for the messages to flow between accounts. Second, the account owner always controls their own subject space. To this end, when importing, the account owner can pick and choose where the import will show up in their own subject space. This is a very powerful concept that allows some tremendous opportunities. +* __Reduce total cost of ownership__: Users want reduced TCO for their +distributed systems. This is addressed by an easy to use technology that +can operate at global scale with simple configuration and a resilient +and cloud-native architecture. +* __Decrease Time to Value__: As systems scale, _time to value_ increases. +Operations resist change due to risk in touching a complex and fragile +system. Providing isolation contexts can mitigate this. +* __Support manageable large scale deployments__: No data silos defined by +software, instead easily managed though software to provide exactly what the +business needs. We wanted to provide easy to configure disaster recovery. +* __Decentralize security__: Provide security supporting one +technology end-to-end where organizations may self-manage will make it +easier to support a massive number of endpoints. -For instance, in NGS, a commercial offering from Synadia, the power of accounts and sharing is utilized by providing a service imported to all new users of the system allowing them to get their current usage. As users connect and disconnect from the system, the usage servers in NGS gather the events from the system account and calculate the real-time usage information for each account. There is also a usage demo account that can export a single service with a wildcard, ____ Anyone on the demo account can send a request to **ngs.usage.ACCOUNT_PUB_NKEY** with the body of the message being duration, e.g. 1h, 2d, etc. and receive the usage for that time period for the account. Although the usage is available in NGS, no one actually has access to the demo account since that would mean anyone on the system could get the usage for any known account just by the account’s public key. Which would not be desirable. The export on ____ requires authorization, so no one can just blindly import the usage data. When a new account registers on the NGS system, the system will get a signed authorization token for the ability to import ONLY their account usage, e.g. ****. Moreover, the new account owner is allowed to specify where the import is placed, and in this case, we place it on ****. Thus for every account, the account can send a simple request to **** and receive their own, and only their own, account usage. All accounts look exactly the same. And in this example we utilized three (3) accounts to essentially have secure domains, or DMZs, to provide the security and control what we want to achieve. +To achieve this, we added a number of new features that are transparent +to existing clients with 100% backward client compatablity. -Under the covers, the NGS system does several things to pull this off. For streams, the import that is present, will be checked when the importing account does a subscription. If any of the import is covered by the subscription, the system will create a secure shadow subscription on the exporting account subject space for the exported subject. This will map message sent to the original export subject (or a subset scope if it contains wildcards) into the import account. Services are a bit trickier, in essence, you are importing and mapping a subject that the import account can publish to. In our example above with ****, this is exactly what happens. When a new account publishes a message to **** the system can map that message into the exporting account’s subject space. But only doing this would be insufficient, since most of the time the published message is a request, meaning it has a reply subject and expects a response. The system needs to ensure that the response can also cross the account boundary in a secure and meaningful way. NATS 2.0 goes further and anonymizes the reply subject such that the responder has no way of knowing the requestor (unless the body of the message has a clue). NATS knows the requestor is guaranteed to be authorized, but there is no other way to understand or track the requestor. The system sets up ephemeral state to map the random and anonymous reply subject back into the importing account and with the correct reply subject. -We believe as time goes on, accounts and more specifically account sharing of streams and services, will be the most influential feature we introduced in the NATS 2.0 release. It will change the way people architect system on technologies like NATS. It will allow systems to scale but maintain simplicity and security in a way that was not possible before. +## Accounts -## Security and Authentication +Accounts are securely isolated communication contexts that allow multi-tenancy +spanning a NATS deployment. Accounts allow users to bifurcate technology from +business driven use cases, where data silos are created by design, not software +limitations. When a client connects, it specifies an account or will default +to authentication with a global account. -Prior to NATS 2.0, NATS allowed multi-user deployments and supported username and password. As NATS evolved we supported bcrypt-based password and encouraged NATS users to utilize the tool provided to avoid having any plaintext passwords in their configuration files. However, this still felt insufficient. For NATS 2.0 we wanted to have a system that never had any access whatsoever to private keys, etc. The most common way to do this is via public key technologies. There are ways to prove one’s identity without ever sharing the private key from a public/private key pair. The team chose [Ed25519](https://ed25519.cr.yp.to/index.html) as its key algorithm for NATS 2.0. We may add others in the future, but our selection was specific in that Ed25519 can only be used to sign, and has very few branches in its code, providing a higher level of resistance to side-channel attacks like Spectre and Meltdown. NATS has an encoding of these keys making them easily distinguishable and selectable from a copy/paste perspective. +At least some applications need to share data outside of their account. +Data can be securely shared between accounts with secure services and +streams, where only mutual agreement between account owners permit data flow. -The NATS 2.0 server upon client connect will pass a nonce field in the INFO. Clients use this nonce and their private key to sign the nonce. Servers are configured with authorized users’ public keys only. They can use the public key to validate that the client has the private key that corresponds to the public key. This combined with TLS provides a powerful way to secure and manage NATS clients and users without ever having access to a client’s private key. +This means within an account, limitations may be set and subjects can be used +without worry of collisions with other groups or organizations. Development +groups choose any subjects without affecting the rest of the system, and open +up accounts to export or import only the services and streams they need. -In addition for NATS 2.0, we also allow the system to extract a user identity from an x.509 client certificate. So for those who require client certificates and have the management and system infrastructure in place for client-side certificates, you will no longer have to duplicate identity to map to authorization primitives in NATS 2.0. +Accounts are easy, secure, and cost effective. There is one NATS deployment +to manage, but organizations and development teams can self manage with more +autonomy reducing time to value with faster, more agile development practices. -## Network Topologies (Rewrite for Routes and Introduction of Gateways) +### Service and Streams -With the introduction of accounts, the existing clustering protocol for NATS needed to be rewritten. The protocol had some inefficiencies we wanted to correct for quite some time, and the need to introduce account scoping into the clustering protocol gave us an opportunity to correct some of these wrongs. Each message traveling between servers now had to designate which account it was bound to, or the account under which it was published. This allows remote servers to then match the account and subsequently match subscribers from that account for any given message. +Services and streams are mechanisms to share data between accounts. -Two inefficiencies that existed in the original protocol were handling of duplicates in the interest graph, and message delivery across a route in the presence of both normal subscribers and queue-based subscribers. Subscriptions (the interest graph) were always sent across a route to the other side even if the subjects were the same. The other side was smart enough not to send the message more than once for a given subject, however, the interest graph traffic could get very chatty even under normal behaviors. We also would send the message again between servers for every queue group that existed for a given message. This has served us well to date, but the team wanted to be able to update and improve this. +Think of a service as a RPC endpoint into an account. Behind that account +there might be many microservices working in concert to handle requests, but +from outside the account there simply one subject exposed. -In NATS 2.0 we have greatly reduced the interest graph traffic for duplicative subjects and there are more optimizations we can make without needing to change the protocol. We also only send a message once regardless of the interest being represented by normal and queue-based subscribers. The way these two features work is fairly straightforward. For interest propagation, we keep a suppression map that uses reference counts to only send interest for a given subject once, when it goes from 0->1, and when it goes from 1->0. All other changes on the local cluster simply increment or decrement the count in the map. For sending a message only once in the face of any interest, we force the protocol for clusters to require an affirmative list of queue groups to which the receiving server can possibly deliver the message. This allows the distributed queues to work correctly without requiring the “directed” message sends between servers. +__Services__ definition to share an endpoint: -Also in NATS 2.0 we introduced a new connection type called gateways. Gateways allow multiple NATS clusters to be connected together into a supercluster. The team was faced with many challenges in the design of gateways. We wanted the abstract to look fully connected, but we did not want an explosion of connections at large scale. We were also concerned with interest graph traffic that would need to propagate globally when all changes were actually local to a cluster. +* Export a service to allow other accounts to import +* Import a service to allow requests to be sent and securely and seamlessly to +another account -Gateways can either be inbound or outbound, but not both. Every server in an origin cluster has exactly one outbound connection where it can send messages to the remote cluster. In addition, they may have zero or more inbound connections for which they send interest and receive messages themselves from the remote cluster. This forms a type of full mesh but with much fewer connections per server, and of course avoids any SPOF scenarios where only one server would be selected to perform communications between the clusters. +Use cases include most applications - anything that accepts a request and returns +a response. -In addition, outbound connections can be in one of two states, optimistic mode or full knowledge. In full knowledge, all interest graph changes need to be propagated globally for every 0->1 or 1->0 change. This could result in quite a bit of wasted control messages being sent and received. The team designed a way to have a connection in optimistic send mode to start. In this mode, a message will be sent over if we have no information at all about the message’s subject. If the remote side truly did not have any interest, it can asynchronously send a message indicating so, either for the subject itself, or the whole account. Obviously, this has some edge cases where we are sending more messages (both data and control) then we were saving. So a connection can change its state and move to full knowledge, meaning it wants to know all interest and it will not send a message unless it has knowledge that it should do so. We are continually tweaking this algorithm, but extremely excited about the power it has to create large superclusters. Synadia uses this technology in its NGS offering. +__Stream__ definitions allow continuous data flow between accounts: -Gateways and SuperClusters support distributed queue subscriptions just like today. However, we have added selection criteria that will favor queue subscribers in a local cluster to the publisher, and only cross a gateway connection if no local cluster queue subscribers for a given group are available. We feel this is a powerful feature on top of what distributed queues give us already with transparent horizontal scalability. Combined with the recently introduced drain capabilities, allowing queue subscribers to exit a group without dropping any messages, this is an extremely powerful way to design scalable service endpoints for modern systems. This also provides zero configuration and high availability failover across multiple locations - entirely transparent to clients.. +* Export a stream to allow egress +* Import a stream to allow ingress -## System Account +Use cases include Observability, Metrics, and Data analytics. Any application +or endpoint reading a stream of data. -With NATS 2.0, NATS servers can now send and receive messages themselves. Prior to 2.0, NATS could only send and receive messages on behalf of connected clients. NATS servers now can generate events on things like connect and disconnect events and can receive messages and respond to requests. This is exciting for the team as we can now enable many things that were only possible before with protocol changes. In addition, internal messaging is bound to an account just like all users are in NATS 2.0. So all of the security, sharing of stream and services and mapping of subject spaces applies equally to system accounts as it does to any other user and account. Synadia’s NGS system uses system accounts to enforce limits globally and generate events that allow us to track usage. It also allows push-based updates in our decentrally managed system, described below. +Note that services and streams operate with __zero__ client configuration or +API changes. Services may even move between accounts, entirely transparent +to end clients. -## Decentralized Management +### System Accounts -As systems get larger, management through config files only becomes challenging and brittle. With the introduction of accounts and superclusters, we wanted to encourage large, shared NATS clusters that could power large organizations, even entire companies and possibly even a global utility. Config file-based management would not suffice. +The system account publishes system messages under established subject patterns. +These are internal NATS system messages that may be useful to operators. -In NATS 2.0 the system can be configured to be completely de-centrally managed. In addition to Nkeys (Ed25519) keys, we introduce JWTs (only signed by Nkeys) into our system and a hierarchy that would utilize a root of trust. In NATS 2.0, the system understands operators, accounts and users. Each one is represented by a JWT. The operator JWT is self-signed and is the only thing required to be configured for each server. Account owners have JWTs signed by an operator and then generate user JWTs which are self-contained in terms of identity and authorization. +Server initiated events and data include: -When a server receives a client connect request, the client presents its user JWT (signed by an account) and a signed nonce, generated by its private key. The server validates the user JWT, checks the signature and if all are valid, will then use a resolver to pull the account JWT based on the public key in the user JWT. It will then validate the account JWT and finally check that the account JWT was signed by a signing key owned by the operator for which the server was configured. +* Client connection events +* Account connection status +* Authentication errors +* Leaf node connection events +* Server stats summary -This allows a very scalable, yet very secure mechanism to allow rapid change of permissions, authentication, limits, etc. to a global shared NATS system. +Tools and clients with proper privileges can request: -## LeafNodes - The best of both worlds +* Service statistics +* Server discovery and metrics -We are also introducing leaf nodes. A leaf node is a NATS server or small NATS cluster, running in a designated mode, that will connect to another cluster/supercluster. It will securely and transparently bind to an account on the remote cluster, and bridge communications between local clients and those on the remote cluster. This allows existing clients and systems to integrate with NATS 2.0 systems with increased security, accounts, and sharing all with no changes to an existing NATS application. For instance, we could run a leaf node server that allows existing applications with their own authorization to communicate with new clients that are secured by accounts and nkeys and utilize accounts for communication isolation and the sharing as the systems interfaces. With NATS 2.0 and a local leaf node server (a normal NATS 2.0 server) you could import streams and services and have them accessed by current applications with their own authentication (or none if physical security is in place). This means you can transparently use NGS, our global utility, and combine that with on-premise servers and clusters. You could securely and transparently use NGS to connect different regions or access meaningful streams and services as they become available on NGS. Leaf nodes utilizing accounts effectively create a DMZ between a local or private NATS deployment and NGS or external NATS clusters or superclusters. +Account servers will also publish messages when an account changes. + +With this information and system metadata you can build useful +monitoring and anomaly detection tools. + +## Global Deployments + +NATS 2.0 supports global deployments, allowing for global topologies that +optimize for WANs while extend to the edge or devices. + +### Self healing + +While self healing features have been part of NATS 1.X releases, we +ensured they continue to work in global deployments. These include: + +* Client and server connections automatically reconnect +* Auto-Discovery where servers exchange server topology changes with each +other and with clients, in real time with zero configuration changes and +zero downtime while being entirely transparent to clients. Clients can +failover to servers they were not originally configured with. +* NATS server clusters dynamically adjust to new or removed servers allowing +for seamless rolling upgrades and scaling up or down. + +### Superclusters + +Conceptually, superclusters are clusters of NATS server clusters. Create +superclusters to deploy a truly global NATS network. Superclusters use +a novel spline based technology with a unique apporoach to toplology, keeping +one hop semantics and optimizing WAN traffic through optimistic sends with +interest graph pruning. Superclusters provide transparent, intelligent support +for geo-distributed queue subscribers. + +### Disaster Recovery + +Superclusters inherently support disaster recovery. With geo-distributed queue +subscribers, local clients are preferred, then an RTT is used to find the lowest +latency NATS cluster containing a matching queue subscriber in the supercluster. + +What does this mean? + +Let's say you have a set of load balanced services in US East Coast +(US-EAST), another set in the EU (EU-WEST), and a supercluster consisting +of a NATS cluster in US-EAST connected to a NATS cluster in EU-WEST. Clients +in the US would connect to a US-EAST, and services connected to that cluster +would service those clients. Clients in Europe would automatically use +services connected to EU-WEST. If the services in US-EAST disconnect, +clients in US-EAST will begin using services in EU-WEST. + +Once the Eastern US services have reconnected to US-EAST, those services +will immediately begin servicing the Eastern US clients since they're local to +the NATS cluster. This is automatic and entirely transparent to the client. +There is no extra configuration in NATS servers. + +This is __zero configuration disaster recovery__. + +### Leaf Nodes + +Leaf nodes are a NATS servers running in a special configuration, allowing +hub and spoke topologies to extend superclusters. + +Leaf can bridge separate security domains. e.g. IoT, mobile, web. They are +ideal for edge computing, IoT hubs, or data centers that need to be connected +to a global NATS deployment. + +Leaf nodes: + +* Transparently and securely bind to a remote NATS account +* Securely bridge specific local data to a wider NATS deployment +* Are 100% transparent to clients which remain simple, lightweight, and easy to develop +* Allow for a local security scheme while using new NATS security features globally +* Can create a DMZ between a local NATS deployment and external NATS cluster or supercluster. + +## Decentralized Security + +### Operators, Accounts, and Users + +NATS 2.0 Security consists of defining Operators, Accounts, and Users +within a NATS deployment. + +* An __Operator__ provides the root of trust for the system, may represent +a company or enterprise + * Creates __Accounts__ for account administrators. An account represents +an organization with a secure context within the NATS deployment, for example +an IT system monitoring group, a set of microservices, a regional IoT +deployment. Account creation would likely be managed by a central group. +* __Accounts__ define limits and may securely expose services and streams. + * Account managers create __Users__ with permissions +* __Users__ have specific credentials and permissions. + +### Trust Chain + +PKI (NKeys encoded [Ed25519](https://ed25519.cr.yp.to/)) and signed JWTs +create a hierarchy of Operators, Accounts, and Users creating a scalable +and flexible distributed security mechanism. + +* __Operators__ are represented by a self signed JWT and is the only thing that +is required to be configured in the server. + * Operators will sign __Account__ JWTs with various signing keys. + * __Accounts__ sign __User__ JWTs, again with various signing keys. +* Clients or leaf nodes present __User__ credentials and a signed nonce when connecting. + * The server uses resolvers to obtain JWTs and verify the client trust chain. + +This allows for rapid change of permissions, authentication and limits, to a +the secure multi-tenant NATS system. \ No newline at end of file From 5a458036a23fd2c3342994bf8aaa7d6525b0a18a Mon Sep 17 00:00:00 2001 From: Ivan Kozlovic Date: Wed, 22 May 2019 18:11:42 -0600 Subject: [PATCH 02/31] Major rewrite of NATS Streaming Server concepts section and updates to the developing section. Signed-off-by: Ivan Kozlovic --- SUMMARY.md | 41 +++ developer/resources/ring_buffer.png | Bin 25282 -> 0 bytes developer/resources/start_positions.png | Bin 23794 -> 0 bytes developer/streaming/README.md | 10 +- developer/streaming/acks.md | 2 +- developer/streaming/connecting.md | 15 +- developer/streaming/durables.md | 8 +- developer/streaming/embedding.md | 111 ++++++ developer/streaming/protocol.md | 265 +++++++++++++++ developer/streaming/publishing.md | 10 +- developer/streaming/queues.md | 34 +- developer/streaming/receiving.md | 28 +- nats_streaming/channels/channels.md | 7 + nats_streaming/channels/message-log.md | 5 + .../channels/subscriptions/durable.md | 9 + .../channels/subscriptions/queue-group.md | 11 + .../channels/subscriptions/redelivery.md | 11 + .../channels/subscriptions/regular.md | 3 + .../channels/subscriptions/subscriptions.md | 13 + nats_streaming/client-connections.md | 9 + .../clustering/auto-configuration.md | 17 + nats_streaming/clustering/clustering.md | 9 + nats_streaming/clustering/configuration.md | 95 ++++++ nats_streaming/clustering/containers.md | 9 + nats_streaming/clustering/supported-stores.md | 9 + .../fault-tolerance/active-server.md | 7 + nats_streaming/fault-tolerance/failover.md | 9 + nats_streaming/fault-tolerance/ft.md | 15 + .../fault-tolerance/shared-state.md | 3 + .../fault-tolerance/standby-server.md | 3 + nats_streaming/gettingstarted/configuring.md | 3 + nats_streaming/gettingstarted/install.md | 81 +++++ nats_streaming/gettingstarted/intro.md | 6 + .../gettingstarted/process-signaling.md | 27 ++ nats_streaming/gettingstarted/run.md | 96 ++++++ .../tls.md} | 0 .../gettingstarted/windows-service.md | 28 ++ .../{nats-streaming-intro.md => intro.md} | 16 +- nats_streaming/monitoring/enabling.md | 24 ++ nats_streaming/monitoring/endpoints.md | 319 ++++++++++++++++++ nats_streaming/monitoring/monitoring.md | 3 + nats_streaming/nats-streaming-install.md | 102 ------ nats_streaming/nats-streaming-protocol.md | 190 ----------- nats_streaming/nats-streaming-quickstart.md | 82 ----- nats_streaming/partitioning.md | 119 +++++++ nats_streaming/relation-to-nats.md | 12 + nats_streaming/store-encryption.md | 33 ++ nats_streaming/store-interface.md | 15 + .../{nats-streaming-swarm.md => swarm.md} | 0 49 files changed, 1521 insertions(+), 403 deletions(-) delete mode 100644 developer/resources/ring_buffer.png delete mode 100644 developer/resources/start_positions.png create mode 100644 developer/streaming/embedding.md create mode 100644 developer/streaming/protocol.md create mode 100644 nats_streaming/channels/channels.md create mode 100644 nats_streaming/channels/message-log.md create mode 100644 nats_streaming/channels/subscriptions/durable.md create mode 100644 nats_streaming/channels/subscriptions/queue-group.md create mode 100644 nats_streaming/channels/subscriptions/redelivery.md create mode 100644 nats_streaming/channels/subscriptions/regular.md create mode 100644 nats_streaming/channels/subscriptions/subscriptions.md create mode 100644 nats_streaming/client-connections.md create mode 100644 nats_streaming/clustering/auto-configuration.md create mode 100644 nats_streaming/clustering/clustering.md create mode 100644 nats_streaming/clustering/configuration.md create mode 100644 nats_streaming/clustering/containers.md create mode 100644 nats_streaming/clustering/supported-stores.md create mode 100644 nats_streaming/fault-tolerance/active-server.md create mode 100644 nats_streaming/fault-tolerance/failover.md create mode 100644 nats_streaming/fault-tolerance/ft.md create mode 100644 nats_streaming/fault-tolerance/shared-state.md create mode 100644 nats_streaming/fault-tolerance/standby-server.md create mode 100644 nats_streaming/gettingstarted/configuring.md create mode 100644 nats_streaming/gettingstarted/install.md create mode 100644 nats_streaming/gettingstarted/intro.md create mode 100644 nats_streaming/gettingstarted/process-signaling.md create mode 100644 nats_streaming/gettingstarted/run.md rename nats_streaming/{nats-streaming-tls.md => gettingstarted/tls.md} (100%) create mode 100644 nats_streaming/gettingstarted/windows-service.md rename nats_streaming/{nats-streaming-intro.md => intro.md} (81%) create mode 100644 nats_streaming/monitoring/enabling.md create mode 100644 nats_streaming/monitoring/endpoints.md create mode 100644 nats_streaming/monitoring/monitoring.md delete mode 100644 nats_streaming/nats-streaming-install.md delete mode 100644 nats_streaming/nats-streaming-protocol.md delete mode 100644 nats_streaming/nats-streaming-quickstart.md create mode 100644 nats_streaming/partitioning.md create mode 100644 nats_streaming/relation-to-nats.md create mode 100644 nats_streaming/store-encryption.md create mode 100644 nats_streaming/store-interface.md rename nats_streaming/{nats-streaming-swarm.md => swarm.md} (100%) diff --git a/SUMMARY.md b/SUMMARY.md index eb642bc..3949c59 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -129,6 +129,47 @@ * [Durable Subscriptions](developer/streaming/durables.md) * [Queue Subscriptions](developer/streaming/queues.md) * [Acknowledgements](developer/streaming/acks.md) +* [Embedding NATS Streaming Server](developer/streaming/embedding.md) +* [Writing your own client library](developer/streaming/protocol.md) + +## NATS Streaming Server + +* [Concepts](nats_streaming/intro.md) + * [Relation to NATS](nats_streaming/relation-to-nats.md) + * [Client Connections](nats_streaming/client-connections.md) + * [Channels](nats_streaming/channels/channels.md) + * [Message Log](nats_streaming/channels/message-log.md) + * [Subscriptions](nats_streaming/channels/subscriptions/subscriptions.md) + * [Regular](nats_streaming/channels/subscriptions/regular.md) + * [Durable](nats_streaming/channels/subscriptions/durable.md) + * [Queue Group](nats_streaming/channels/subscriptions/queue-group.md) + * [Redelivery](nats_streaming/channels/subscriptions/redelivery.md) + * [Store Interface](nats_streaming/store-interface.md) + * [Store Encryption](nats_streaming/store-encryption.md) + * [Clustering](nats_streaming/clustering/clustering.md) + * [Supported Stores](nats_streaming/clustering/supported-stores.md) + * [Configuration](nats_streaming/clustering/configuration.md) + * [Auto Configuration](nats_streaming/clustering/auto-configuration.md) + * [Containers](nats_streaming/clustering/containers.md) + * [Fault Tolerance](nats_streaming/fault-tolerance/ft.md) + * [Active Server](nats_streaming/fault-tolerance/active-server.md) + * [Standby Servers](nats_streaming/fault-tolerance/standby-server.md) + * [Shared State](nats_streaming/fault-tolerance/shared-state.md) + * [Failover](nats_streaming/fault-tolerance/failover.md) + * [Partitioning](nats_streaming/partitioning.md) + * [Monitoring](nats_streaming/monitoring/monitoring.md) + * [Enabling](nats_streaming/monitoring/enabling.md) + * [Endpoints](nats_streaming/monitoring/endpoints.md) + +* [Getting started](nats_streaming/gettingstarted/intro.md) + * [Installing](nats_streaming/gettingstarted/install.md) + * [Running](nats_streaming/gettingstarted/run.md) + * [Configuring](nats_streaming/gettingstarted/configuring.md) + * [Securing](nats_streaming/gettingstarted/tls.md) + * [Process Signaling](nats_streaming/gettingstarted/process-signaling.md) + * [Windows Service](nats_streaming/gettingstarted/windows-service.md) + +* [Docker Swarm](nats_streaming/swarm.md) ## NATS Streaming Server diff --git a/developer/resources/ring_buffer.png b/developer/resources/ring_buffer.png deleted file mode 100644 index 44fd040b25ddf13f687e7c25894389d95f03c4c6..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 25282 zcmc$`WmH_j(l&|=8f8|doE_te|ccRo(WKod`k)fcVQ03*M)S;kYD50RBLlEJCZ*Yly{sMnN zyQ<4dKvj(s9|52BY&CV=bd{9^%$*!qOf8(uELl7qAiyXnC?QV);G=`3n<=@cgT14x zfTu9!zYzk!=hv@UDarp0akCSq)Kyj^mvnNmBS_t`R{b# zU&53&Zf+0(R@RRnKeBw}WN~t_W@YE+=VxW(VCCRo21YQudO5n8dNMn@QvHX>|Im@L zbTxOeg}B)|Ig-E9H8pc`cN3;KXQ zrV71&E1>FPYYD9UN?(Ls=-e~ir5;*6 zC@3)~c`0#CPw2CbH>t#PUKj75vU8^nO);$AqOx`2BfS;FU&5{B`%u-@gTNPTBAo!9 zGTC~AwvQOx5eP@z-Hilc14}K6K`DabNOAFrPfaMB30W2?@hrtDsUyh?qmsTkSAv4z zJ{=uAy~t`6ew|yJ5uQo#{(G;#p*dG&=o7HvbU*w2;li*DMN%9jCHDXQDVHMGEcQ+N zc8%DVKg~3hLH?AOKrSu^uda|8B~?LT%!~D6PLDT7F=mkZ+mn>Un*3Zo66@A_tzKdP zC+Uf=gdB#F9440v<1Sm8gN_VZfc!H#b7RV5%?sf!$9sd?wRe0=Xtp74lG%hd=ET#e zwGp*^kn(4@UyX}tW8cY!uuAB_7|^c|L-xo{1a3S<` zk)qx0+p`y=zTfY?X^-z9Mrk?HCXFV&X+o9NlcWEW0_$FTv(4Sox)I8!SRGRnz-%G7 z>oBA)S4I&ZIy=&8#e1hCiLWhIhk>>xguZ0v=r!8Kd|SW0QZv!zIsfLwnm5#T@w)Jz z+yK#WQ%}ZDiv+f#>%rF&YVc@kn>P|a{s^h*j*-M)gq(#f?kta*50HemLwWVan(x^0 zQ)y=@Nm>UT`J->SD^h4@M&!H=J;=0r6S}~*_5l%=B(8~K{&w{j;n%%8Q$$Ik&?67Y z%IifE>}5z_J3Z3@+>rKl=7Rp8nKBbnZNl_^zk#t}-5|82{>a#@3&){0jIY!Y>M$pv z3*+oqoC$OLA}8TqoN-#XC&{ix%jB`v-bNDwMc>7(wDp%|1(_W!s{RinT>}wu1VoI; z_hcb8c-_c~wY{#+j6JJOgAEk%gZ7t;JEmc9i>7}SID&E#FaCW325E+bkb)9c@alJ<)^J|?6H{2J0ygvGYe3fow zf4V}hZQ{NB^cBa{ac7Z3Re+RN+*6nYLjZ9P)J8HJ&~Ucxy#H)quZQckVROC_eJ13Z zcZD(`QL0L82FKx#C$XYOh726R>!YRtWAD3|_$|;ryheNlxwc3Bl#RtvI`i?anWyjS zK}62c6BfMUkThYQXUn|Rf`h>`i<17m2cDakNg9-c;-eb}yKl)oeCOKo*aohp=EWRz zz}SB$_tNBGn^~2(NYdP#6|W*!+~a6bFQ97W201hlLZMJS9;~mAcuY4!sQrPIOjIQU znpyI3cbPfVeYNKg3SA$tKj@btSI|R?Fb5F1knT!r)DYN=R&}SV4*TG<_n5pe*LJth zq_Uc>z;CJbkYIdmBT;7((%apAraVv(wp`%8|CCBF72?p;lGj!{y}09ZhM=`98tB_3 zjw^APcq_I{t|RVEo{lz~;GRgtM9Y%*++yp_^iLqBJpcL(86d*?=r0m=c~+Ek3s~i9e-VXy1Wbn=C3eGZ zm{=w=C~j{p5pUFdpNuy_5B{Fs&Hgl~1KL?5llAO-ysdfFGaf}eMc%U$d2Z|SEo~cI zA`yi)`QRSLv2@PPS6wA%U2PlLW4_QoGydlZ5sSSF#5F?51jJnTI_7q8=huIcBP|6T z3Y0G-Dh0Z$B#=P+l&E^^Ak&-_AAXbtswYQ{$`^5B0T#$%s`vi!p~aJzaTFdn_rPyt zd*Ve0a3Dd1Sj{+lmTw}T@{-Y@h`JgoZ(?8eyxBfvlvl-*iL`4#N!ZEJBNV_SYX>%xj zHkH$C8#9x%JD9#x7y3I%xLp4s*|u7qwr%2r79jde(L(Iip8rrBGXCpjv^=3v+&bI6 zYXdVtXC5Q3nB?VScKOSP!}MzZO1|8Kmtz{vHDr4nVwrs6>UGm{#T-l|p=h@iaxw{) z;7eO;^j(6X_ZW<)gX1k2>s&`_m#eJg8WetsI^?1eY<~qquY5U_#EnASvK|-lnOqS< zEw5<0HuTU#Z9T**yG}$9jslf9sqRfzP=9B6`LZ@ve}23}b@yJvb*8?fU5P(yn|%tT zkDZvy`z?x8z>FeDQ6gE2=U|aq$7)}|Nq#}B&4D3LEzK>g#dl6B-_aK7tDkzI5R08X-Imd}fBfn%>jtdfie&ztDywD~Xc!+_ad!a}4$R>u2w!%>O5& z0x#x6(B!(AQYMS>q0}z+&{(1mxgf4yVE90`xWQK z1FwB8rkYN@gx|mDJaWdP0udtv5g#(U6c)#6?Ra{$ZFe@Fbk5l0fU8IgW5u;$7vgae@`e1841uE4Cfk*H@9 z-n4fFelEKA+*CI7cZUh7)4;sIz`2+Sf9y^RPGSUwc5l>L5Js=@ai1Gg%#pSvngl*x z1lwqct|eQpL`k@SYk)v#x57Qm-Pr2oC$(-uDuPGp?lwdrsV~6sKMtMkZIE5k_D2jJ zI&7XhbjefI&i9aZA~W;~XOo2YK|4&ljy{y9oU8*A>^olT3cTo##EvHDVNb1=R)oh^ z=p)Wub|p(xV|+aRyi40l`TOB~l%B#(Ev9gkp68v}SoQ`4s!xKWe#^5~Sc)5M7}fyY zF1sc%KDTZ$+KF)w^141z=OT0e)%|cDsmoRGl_(ER$1vvjHfO|{XVM;ln&7>_&gjUH z#Fj*!&-UFS)#jbr>q;X}JQ|*;)qc?bdsft7L@FE~^u_!-Xj5}n#y({GK#%tb`yTEp ztH{LG6!xj7LGkcAP8?MHyLEH1=0aidlXDo1La`(R?@_AW*JQYHN;p7}M=zScPMY0gUj*Egw1Dy;M|2S*!8(8RZzoFTly8DO(@ zo!4*zbV9~*hlzSBdIemglHG6B%E$r(qhGCL0kIu9#6zqT$!jE)wr1@OKiMzqL>Jfy zYL6wBZT8+ENunLaMI(fxFr~0bXFWtOP!G~@wPa;|`~33Yd6$J&?gXCtZ^{U8#!8AA zi3WN_Eb8J$7VSl@kaKKZ6(gk8aVqS_41k*F^!ZDoRL?fo2E2{fVU(cVNbI058bI9c zr0E(67e;v`7iE{fly8FjmF+b;=8w2zCBEp%9LN;JwjTXC-Y8&n!3^!oBE^*Yk@s z^hRQte0PE`{bzj}xy3!#t|elxg&eVZRa~>we_(!>6R5Fwb3?L@zYb^6O{lU9UKFXe zM9_xxN8A}vqpNm%+8u1R`HnT**QW8MrU@z$E9#y*eK|)Xr5|`bGI|+V0oKuI>JqjEOwayb%k=|LkQp zN~}g@uUr>gDj1|h!^uL7CaU#Oy6C?#E07pWrqj8hd~tU6#L0&E#d??K$M?jie+ZWq z#zpE_+GuY+M+`{S8SqhzlE$p6FAfhg^*)CEmq6kx0qs_$=TtErorPR6atLa;?fgd% zk6)Zy|MjfgZ^>ng!fUirIgu*gH=WZ?p0+$cKM!V?71U$CCUO^;iB#Kbt!b8)7&bOG z&2T?g@R;@8+zB%1%l?V83lRw<4`s<= z_(AGrv%HgRMv)~qUi>9)bU-RV3H=wnVKc5?n3_h1gA5B`XJvb&ZX2!2V9Qlwj zRgBT)oAj7;-qja{rIVVxo?v_VEjML7;N^)(^!IMc29s`$Nty`d;h?d&|7Uj5BCU8i zf1YR(C9(J+6H)=pAaM|z@ved!>qoOltMPm+0X`*@llgCP0oS|%GY{)N7lu*c2OFD- ze{Yki6_?Rz@x2dp-!AdGaTG(ap#S3X>1hd5I+oW995KiiscdRh3L=_x%CVAwS2Y(* za=_!3YD1DxPM!EGjSfV&tF9B=olkcw8C&1JGyx=0Y5uBAr1+{~FAj7)bmhyDcm|2R ziGBEnhmwRkM-3=Zt8ZEx0zhzRgw33paAb~@t)!f$3!44;DDS!Kcb0W7LD-r74P8g@ zdQk?&RXJ|uXpUSyj>0Y9JNBbpCP16BwaBIi3P5+~Yd(T)pRPiGtQIf`D137haLBi| zK=Do;XuldDvhEdM0$J~pA-7|GYXQZVcSO_AEU&CoEPP}PpJ^lb3BGb%8?HmfJUl;F zDUc1VzFF5EhDgH2c(`kCTAG{>Et0)F4DJN{9x=}Kyxi=*FpspOwJ5vg!JH1I@T=m{ z97d-!K{G!;co_r4ixDIQ1gG7gLOSq}ER-adunpUfFeJ~76yB|gr73SMs3Xq*;s}LS3M$(qg@MTM zY};P^Rimp%);B&P9n1Mb-+GeJ^PWdm`4bmCr`>#InoT`NB6?Q?mi}Glr}Kyz>u<~) z7!giTB=Nm10oDTTBJI{^%MER#cE_DtCnH-n545OB;Vr<#+MDV&BX_kxCyAfnGFVSu zTzc8QxCWY>69p(NHYYLIW*pUP5Qd+vTh5SOPIXVDH!hau4sSl0DT zO++(al@-mOl8oaCxsS1SuQ{?~tZN+1bLq^Pcy^i-9(e3xo`U#Ng)t&p=o}y>4>uiH zb#JcR6g()`1Xs{YmIwYEJVM921v(ySWaJv)zxjg36!e{PPRn$P0~1_qoq^~dqNkad29KG8;u?+sy3@`tz3o=sYF4BN=Ar`oQ{ zeh{%oJ&ZXEJeRvivtR8m{+lrB@W5`dkaxJj`DeuM3))}5QxhIrVq-g5n}=ncI|U^r zQe6;+aHQK1wp2bH^m7UW;K@?up@b=%3ZOesRd-L=kA4w1sSHboOohYy@DQH!H9GsU zrwOCMqVmRnB=?7HypaAfm4p9*&rE_IKUJ!#p<#{>&=mB-!Oa}Gx4%%i)K={A8Vj=n znn*Gl?2a2?@!rA1jnqm!(@Gv@`=YLR98RIFeB5mQ3^TZkN9C?mlRo_Gu+J^v;c&Dx zd`BHMs-xgBe>5!w``1tCY~j1e_B||D%Y8(oAus`jnBloGjOVf+HtJOv7LAfgh$}3S zaLL4*O=9?)?NJ#?ux$0P`axH9PuSyVECa2ud^m# zG?Sg|OOi1fLd3P`FZe@nh-B)pPn|-N6^T2Q&u~tzW+$mms^u6Swcu*&-7z2dY?k>!51&f?^-OJpe>7~MTbYJ!wElRp-f<9$rGg*# zOUKNyKjf6UZakw-n$+h?|5DH9ui+JFj1+@ufCul6CDbniUSjF8H0!B$=%G7)DDW&R zu=eF34k1o+6TN(mNDKKyzo@%CQ=8BJ{gt3l74BXc$~~4^LgeF#8t!Zslw0EI9+Th!9~iPUqeLv=yQc!br1qsNTMA+flS4sIuGZU_MM{ zs^)q1VypMxBaW=;F0F0c@Iar`19BFS3S_z~r?|M6AQ0JIF!7j0>H=)I1=qW|lQaX< z$K6Ox{L;E>T4+8h1HKc*OjZ{`!UNHuUWgBM1P$8-Rs7!!v+=>=#ZEM9YvyoKmI@1k zp{7$P1e?kB$U08QZ@4A3Kj&X()Thj>q~aCfMSo{pmR0t8^T+nzRP?l%Ux{u;uYY4$ ztYV4~2QUI#{t+gaP73Peoe*w(IBrRupKD^c>R2z@{^UqQbJ<7A6hDZ_Jo(G})mb04 z!YlOD%<*?Wf$}6#EK*Xs=DUuX>GCO0Z?l{30DDhFq!JwE)1Z`xoW>p$eoE5P5&bM6 z%4-9dgcWg9YwmLmotKY^!9D}N5R+`ElU}~hqrSlg60HSlxN&Ez^UgqdyVXgkg1FN( zs>84IAwzwsGACyBpdg|Z!Nl3-M&WAmm&8d2wzz5&n|3R+OZV9D0t#&@1~aRlf=DAi1B4zftivAq^IuC z-E+nl62!>Uy+Ci$rTI_QKXf@!;eQzS&Hh98AZ#!bz5cyv%d@KgPMq}{XK(?*ZDi`7 zFKVsOFqVIbqB0R9#VBBWKU3o)8;Bmq&R!|aOy+WNkim~%v|E|JFR;7R6ngWEDLWAl za(x~S>q1GxxlT`8gu!mz_x)ga2qdyGsqO6)S?sk#5M>^l?+knV2lvji;J`%Gr)v?* z>aI>#eyzQjs!&U7%^-w(-cXp58@_70T|Q9JHzHnvtPgWz$A%)~!Le)j)6WiZf4w_{ zwz=$G#6fd7Vhom$Ct)synB%9r7(>xBbn$%lD;qa;njRh35!6HGH(`UtGg0W6I_A^t z838{?R8F7ptMp_du;Y(TBHPF$U8>VQH-$_$nW#6=R0xzxWLHJwa=Ch=%8L$SZNlha z(JG&kJZ+$nIv5*ren!D3bBr%G-@2S7dxo3|Lc~~t*_*n5=YMxmr60)?T=yhnXtud3 zRE@2i;**%!2>$MzE{uLx-c6y09I?Cd5O*r>^pp9)gp>qp0rbqeNtGzS__eX0yFoiZ z`qsQGfJ6>91FK8pbo#P+)bES+JxPk4!bVli)CSgi(r>=zHwRzygIWj}?VsQB=EO^P zQJV1V8&titCFV&wiuaDYe;_8?v~Swt?S%W2d+2FdoNtb36CY__E)m_FF2?4gW6v<~ zd>y(dZ`&&eR|3~N@OTK#Z>S@AoEoQu z7uiJt(`~caIq>r*d+}wYiULuPjmI{Q4%2h5`ZQx4;x1L!7U3vot1vdQbMw9g%7T4_ zcTv_U%B%SZ-w6yMNBO^TfDH)t7(BVeEG3 zw<7^J&1K$vV}y2%i->?pI(-p7@97k4sj&O{%xOdHDcs0P_<>KQqHMl_L0E^+Tgs}3 zKo7k)x3^+@qipcK{$bYdz&wUYYh+*8(V{jRRk>xoxj6o%m`LFkE3p;h;De9Jj z6c$0$opG1%j%1jSUX)pDlbCO5yR>w-K#&IQXO$*2U#ybw56+gl*mx;e>XL_~^bDt9 zL5pxi%`~XC_Nl$Ue63wS?B2o#tlz8j<)d)Ma>UxxRvGe-p15Zr?oyiwKxY(KZ-2Np z_uklE;qk)K;lnh}nK1O_>`X03#2*ys}V7*VN@JsytfX$!*Y00VsS_i z3GVO@Jl$UB19m@nFXKLcC}+DA5ER*ul&?R~`cAxeCJaOP>B=Z)xDXl47Gc&%*~a7H zHG))zp=+su1(P}S2iIGAJQz04Qnu$k9+NKHw@+~9s7dle#v^SySaq`3=~UZO(P?^A zR9yx5Lh_x^dNH)^BV6ag9J#HnBA~@LFbIDXQlEwSkl*uJXfP{pUe=R4@l(#hoNoE< zMS3>7SnB@<@nMI2zExP&Bn$M#QB6Zn&x@(TfVKt)az4Gq6&FO%Hw@Xh_htfb7}%n2 z)z5&Z2r1LxPeT6gMn2>xTbkK|9-h!T9RgP^|hl|Itv&wKo(|C7(Z)aoGOz!7vI5gNe;+BUc)^xMi>{SIDKEg(P zT{z)9F)TeNzotBmFD-H1lw^ zvvF~B^fUEy!t`TCATHOn)7s;*ix4!+;Mo02AOt!)F{d+}V#iN45UYW`@)wMl8~TAD z_o3ct2-bAq$x&c?g3$8G)iE|53=X-NHApg@m%O56D;A>nK!GRH zJ2}d_73y2Mo`|d&i2fZY{PZf^r~Gu27?t;2fw&qL)!K>9b=s34LI3^nmw-6-f+Ar| zlAj79@rnK5STwQG>8<6le!y$NXu|tGY%Uus_!2wzj6Ltt*R&}XJ4gY%xhw#YIS>4D zsf9e?^hr+~hmL11;xOu_nbwggu=fNqwOQafZlBOCTPHY5{!-Bh&N4~!oTNW1{? zsQm;HQp-W-N59LVtmJNZD{mTs$ri!u!+!NhU$GFfVcS!ldB9a)3}5dk#Oa(0W^;5d zMN%Ir0AdSYy;;%xC1O79t_#ZG>ZrQ}WM`GP-M}pkKu;nt`xPBKG%DH6(FY+@UA=J{ zSnO1pBy!_MNhVw~u@OODzpHezfzU%P`<4u+za<$ecYI&$DuVpLHU~pTyCBtr54zJm z9HK{)k`@2bGzSq=iK{0)My=I!65yN*eR{DZv15dh1Se8)U{mCLgbRv~kI%})s>2-~ zmsO4IB-x49Pf2!s+qI=K+R@5{!GX)0lZ1~mWGhXHrJl#$leM!|+P7wt=T;>gl%1z` zhNBim$)*mI-HVFp5EF=qO^XpF)rlQz`S=BE#tVx_MOo(31md%v z5{(^14AjG10uLt*L8A?*4|9_%K-uw<^P?@RgLIc4i0>UY(NN`wPr3Mpa*v13Wwk*y z?69uVU-1;ac!FyHzf&t`b0Pra8<$v(C2{XQ$?QM0yq&7OF1l)n3_R z&Vq^EJR*)E;`Zqkh#5!baM2C?7&)835OrXK4YpL5A?P7AOz-hMJHa_j<-ZcJvVA|?FBC?W zxKU^=uu^|0t3)zZs429iiWwVk{F;2&9V`~0;NryiNjSBmA@dfqb*|V-@r)27kxCtQ zh@sF0WPdF|`puwfSRN#mj}P6MgNp)JP2?_f{9tzUD<>u=nb;Bpt5YXSz2Vg#1^9y} zcmqkXYY77lE%jM*(51(+a#B>;Q}?^DjZ}IIj(H;X9_BI=%Bhd$1zgehyg@ZT#xibI zsA}Aff>W@fHa~@jhrgMx)SIzBvL1iq6?rXDV>ATKX!IVqy6_3(!P`vLM#&6a$B9ha z21P2Cx>-FY8}&_I6KgWn_Sqlz9QPw#4`p)8(alp|x)*ZPO&Enwc-yVzsXe5WfjYZ` z>2_$|Y%7XnOx{16;$AdRKTEB^=zugLF!YZ#^z@4>6Vin z2b+KSS8dw(tNc|*nX~Cg6j(@z#tU*m+T7a8GGTkb+|Lx!K5tOCAOke{uf{@r<4rUp zuK!Ot_z8O(!5jIj0ZuG-I$uF=8PbT+LEogOcpkA~pjW?3jS(l~)D=blz(*bN9!F{u z5ttt@Rv77xF(ehs4U|VD;I+*yOsWy*$uk45_VA|q+rE(slqmtiOTyYLx>lp(BIEb_ z*c}WvBa{2Z<&qH4YNLvGc*`?WwGgyf5&miKQriqyLTqrY2aD=J&dkISnniId8yw4uWgFK<^Tv4o_L2GTMLDWw5(yTl!1p7bmrWm(m7nw!;*m*YNSxCjP4;H)W4Hdou~ z*VUR(UQ1O9b0c+7es39lRI_zA0!ei(fgTt{yrms`C9FR+AD@<&XL-0teF)cwTS78H zx_6R^ALa@bNeOU_6lOL)ip$MBRV`Nk#K?gdizd~T1DgN#elV&6-dj4oWd_F9P#!p0 z26+vyG$|94LVdQjLgx==n(}z$?{?_veZz$=BmNS5l`YWa06nBHNRaL0kKE34Q|_8r zd%Kadpzp}FSY1*LL_LFFVF1s7%lbW!u|A=HEEaHw{{HdpYJ~l-#}fgC#}G>Oi_2T~ z!XsgRx}Fe-E}9|bznJ6#g$Xk+YTJ+MXETGNkh3F|GTMZXXlAtHyf(^$+0y1FL6k{D%!f-u@Ul4m8ei2|9x!r)a#J;5UCn>bh7@MM zZ|-x!6~l?$%K4PhC+%ple02hPGB90OS7CpQ2uIz(q53qg5*Ej^`_uAm%A4E;B%o}j z1MWh&G79UlyJ}dMH7ssU#R2Y7@%O@&&(4&toS>7b;dJ{ms( z%zNDXI&>bNs1sIEWMc=(+#(b)va1GZ(|x@OBD%<9+5_^7uv)}k^C=((-_Pe%a9cYe zqIRJk(pK?E4FeQ4qr09UGL9P6a>bBVS~7mh^D;(N+i0TkTeictTn>eSSLE;+%IM&1 zAiBawUtgbv%!8(NG7S)AbKUalH?IA#zOBgm?eCk%sLmg?6nwy?bkKa1_hJvYalr{d zG`f0cDcG;MYNIXIIWiF}C{24r0rcepVDQWRlJs>Yn{F562xo%XO%Q^~h&5k^J%Y>dlTDq5V1X4S9OMwd5K?GYm`6{v zEdfcU3v&oUfl7J>}Es{1ng2!Rmt?OO1JOMCP7C=7`>raxc z@w4v~w88;)has9Q(ui&z9s`@9zRcLjWxPd*v^hZA^&u510>Ym4R=h7A|_<0S#5XOWVr3b{POEE_|3Zz1y5w-`NTs?Rl%kd~jaV znj}EE2@m5N+1)#e$$Q8^5&=NUq08o@a!jO6nw|;o7tIt+MbK9ZfNBV$$uvWTU*7!G z)Re#>Q0I^YXj+#~XN6zN!`BUDEz2_kte0T1(S8$&dqgDrfGs2kj6h(Vqo*;|wL*5Y zJvuw{>!%lxM+e5;g9EX8;KsRxb~?)$<<0pisZG@jw=gzqP%J_^899l)TV((zL>VYw zQQZxWvQ-AhL^d@7!qLi$uY>1}A9oXYD*BR|%mHl7k-7%=KDF3U%g7s#2C!TTep*pZ27@r&iEu!V?P!G_|#l>_2P~Y~qKW?$y`!%D4gLH4D0TCN0 z{|gS2RMVW!+kO16Mnm>mYz508k1jE5zr%1GwOGM9Y8Qq?C^J_7y;EB%g&D)4iE z@sMfnkjvEE>zT1-2Da=zD(e2qK^H_Or68B>qry32+JnOmkntH3U!f-xcwL?B^t zyyIqFq$ENE7$`KBkVeXk$-H$~tSKldBw*xxWfTf}RM>~$bF+zO1OQ+zW{WkLvf@h8 z982h^gn>=BOLt}a%vbb}wQE!Z0@cV+seKz?HS@ne%XYO7j^5>3KG(~|M~a?we9|_~ zR49IEQ2KXayA&C?yf71&yWpwCq4SlS7LfP;P{;fHC9UO(1&@yRgLO)vY%!?3_)Uw4 zW-PZPG%8?-#V$%(ZwbiCPIq4ee;F{RQZ(pmQ)^YAWi)!6nr~nyFBUJyZ0pm`lthyH@@^a}tGN0~pgwq5O)*}MpeD7R4f=abeH#Qh z2Mic?&@lMnVJE>n2*Ts`_iAvEF{8qz903TgxB2JY981#WQRSqSx>;{w(5M41##YTN z;&fXnoD(FqQuV1{kpm?iHKS73DPfMKM;fk5Dd+21szJl#YMq()oJ0nQvwoG+3e6ftsNv4guA<`)cbDrBjYrg8JIZ!c5-7QK(d}X2bnB~y>PHN zxM2U(m|sJ~Rtp)}QL&o(r$SJ0_ol_$MJ=u*m47mTL(J!Wi;GCW%axN(*n3L2oFell z|4Nk2w6g(bi7`gt75O3@8ZlzJN-mqus8lae`v6LiElZvx=gaBqCV)?zMxe%m-z$}L zr-}Gon$$J7ak>;6@Iwj(MO@NICWXgbZvhgi$*;@BrgnAt)|+@p@7Vr)ch_!4TM}!} z!DbDl2flRb5CUM2EER0DT z579@-(Io3T^xK}00;{cR=)})qOLA(|)VJ+N@->DRWg>cph|$>m^*Wuk&=9dRqb)|I z!g?tDeKqDQkA0v9%pulicYHVE>%h|B|MbVtaZ1k{*dhSa{#7L}OHZ4DzEKB71T&IV z+}E-wI&axoMi=~OK@T99eaX$P+`~p019tP1`|X-k7c6MHdvtg)Ra25(gYsJ@Bj%`* zQ8UD}T~{E-6&K)Zyd0hP11F9or{HR7vl}iDtr389G0Zji-52WVWQKAG6tn(Jy@fd) zp=XheM2;IaFekno?Vi3@=Oe{2P_CSlHwQz|gWz`xfY!*AI(EEDk3mxS=-W}8EFA^! z;xbWS!fiFIj$Ie8l}Vt{n!7u*f)^OV5>%{+mxp?$vR|WER&aD<@f|$;x43ikJwxrE zuPIdO#?MA{ehLCP{u6J)hu{ILIRMK^&jf_QjZhMM&~r%#M@D93V2 z3)YHJMQtTB4m7SPkn&2(ju31(m4I-8zzka+tcoZ2X^t+G> zubA_!bn&2O@rN`M9JU?<*`~VRKrS=1(@1OtXJOc=(EDBvCD5^=$y%#>NGH995)o5X zGu-_bL0p3 zMF<`Bw5iTf$pWFkEMj%AqbBUjgJ~pihokma(f5BBnnGVMH2wRfgJR4RhJRCLt@!nq zCd5sndNPjj_u9XE3Hd3XOApF*vwm8zQLRl{%#}}SeFe;U4@2zY)PFSvsG!4+QP}Q9 z@)c1Qg};&oo`)pBIg8GFhuO>U+A!uyaOpKj2&KgVc_m$Bz5Xu6YBJkI7_QX{w69HPq-lbzgu+hoP+$E&1S$yj5NzuaNUC0- z1enP%=<99>V(j298$V@8;vw+LQ9dbXlIT|+T?W=6j=?K57-3L4e)T4=PsS^ZQ+JqO zPv`MnC@8t>FPD0yimyaYjCLVj9Z6GxHn1H{#ZOsPN<&ZopLZ}~#gL(0r2rQ2-$vwW z6!X`cWmnGkguPh_7B_d`Ik8|a{tQsgLkyKDz}~McQ+72i1`_$OG;OCe*tmg^C9%yn zxAT&F078unlqSznnL(Ors%9c*Ssn&Ezq8~QpOcQkRYtOcF}BTJ%E%YF@-$E-S1H9m z0(>nw3=422sjnDEhqVFr-3~(S&!4)xnM$q@O&)I3p3uG9Y3aRxti_2D-TBxLF1sVW zx~Vcc+R~^N3Qm}CXBEkng7pa??^7dB?3hPtl}C8x7X$j^Yh`ZSyCP;DXnbr9ltpEN zg}4;Cmy^-(Q3nqDWAG*_Vk8(eUlmOWMt9GP$)lmG@J#^NMan?e_EwFcgTk4QyhKJH zVFJ$XQ@Z^);S$d6q`c+7%}7JpMaG5z)tkU(8%Eha=`<}>t&puQDrM`pGH@>b>{Ie( znc$zfY@4OC8scfp|D?RB0K-2P&FGLgtwqySkRir2gPD{0BH5-~LTwh5SYVXIun3^#6BtAL*D;@zNpQG+#y*RIM1P^!o zH_ej$b$WUlO8~vI{?3{v{BOO-X`dz)aq^O4I+N{vYKK3#pwf<0H)TTXs0 zt)L-e%Du(ZiAYeemdN`l%4;zh=2_gd1+;ckl-Ky_j1Yj8DAizL{%(?3T-y0@)9FaZ zW&;M1#Cck02at0UAJ&2sj{&AO;K@Cwu!f`pLm0M=-mJ) z50FqaL8hsAc6E~c&U$PS&wl~7pa@juGTV_t_E?dwjcg!bVhvnH5uC*dn zTOi=14#<}wV;mkIFNbK1>4Ltsh+#^<`U$Z)L=zkvWzBDdYs(f)vPC@E9Y(_}H?gHU zeJ2e}k5XMhAT~XJj@fA`ErfaHGKQaE`n1$(9I7sPGw-k_e2Sb`7>W!%`tEah>_<&% zS4(}YR~-pnS_y+NgN4#At@8hQmj;^rf-HTkYKRw5!a$i#SlXId5xmsw`5g4`p7HFb zE^Hx`4xJMTEu+2HHW0C*=t1A-!2QUF-+1IJ()s?N z1Lb^@>ZyJyz$}3G#iUKHw)00r;VYv*Qsm$t>B z4e0bb$jV&zYuma)>EEQ+x;2M?>+;(`e#*6GifmT$fA+!%lbPosQq-osSC)PJYS1KA z{-d`zd{oY≥$dv-+S)PMQ#Lb%~YGZ^k;`>6WJZpYqJipx2hwrGO*yOnpyzLI8-v z5cEOLWV%S6Q*D)H6n)9K*Gx+j=aokoO;Iab72FlvC@C)0z64RIB_s3aae4iR*8!;IB4b#RoU9|@E8D#h zdvGZQv};*lmaf2N8X;~)VYw8L6{L0jVPEO+xknqtViz4blLgi30b+m$)o6Ap3EiBs zWuTRcM==zB0yIT4rE22WIdB=Ydv^gd3N+}5^Un?FgX;a|8iwX6i~;)$Jl~^a7y9Bw zX;fvhL<2KU4x#Xtx>>H`JT~jo2ku(2OIVuFLdk}jH#+-d-vPps%2x+9OF?BIZze;h z4Qu?K%>cCj&#!xfqaV+=)lJ=W{vWNf4`Nfq0ol9Lny`Rtls7WUe6%!`0Q(vvIzYf^Cr~43;}x79%ZUk7x`$rO6*{LZMvx{gu%!!5RPYS5081=`Bc&vo;9&+1W`bsg4^j0r z+LE=@nO_eT&N3jq7I$2(L{vEguuqsY%C4x3rNG)sFDqjV!w2il&R-J6zqCchfvW_4 z1h<>~%$Fb5R83}Mfv$=61OaWxt0v2kCmpu2b5J;hCXC|)K}Lox`AK^WSx1~3$Y_gb zGM{uJe}7n;$<%G^3YZ@;19qfieK}Vz%P)&nen)vBex3ET^ znJjw4QP*mG{{mO>OCD1KD*=~?&k_Wr|2!~3D@3Y_1E*eG+1Y4BdOHD3dUZu>q}6`+ zuQ7kU)irECdn-N{j5^U=*jQAA$QRKSI&Us0xieimxhUB|k!}@)=q>&AEtM@*!!Ur1 z*8ufvZhI!K?g+OgHkMXb0};!UW1y^(e9n&D*QnYma>GBE&~ma95jc-Yq1x^ga7`P~ z)v%z&J~wQJY!U0wvsePUb}VD!e6anyfX3Y3;_v2qVlOh>cbS=43ofEj{})KMhk+sh z)$A(9g`>?qt(fcQinQ)Ev$$}AS}_svkJ8vteqd`*x)^X^B3i2SN~-)zT|{1^L^aid z-Ov^RA5{igd-Q>~oyDvBJ{sE zMz2=qH|+XQCLKb4aM!(B!zU2sR@c4xSs8@hQtJIol9;3i<$m;`D`qj2s|^5z(Z1oM zm%EF4%T(Rx8*;ohStLNXtndH!+t2f{PwThy_8F~~a;BKaiS7HqRzEJQ+H7#Y3h!(A z>bKICuW|~iEaz3SwyZF7Z!u7c$z+}6B70ei!-r~*Y#^!N&upf_-UfWCLtd5k7lgJ@ad0<|_3Xr4V(g4xceg}7oO+khg{ zs+O@Xx&s~*6vdm@{|lgM-UCP{{$k`7=$%(p$GFGFY2E0H^x;tF-cCxN^ZQ9)wQ*pV5}Zi>-9r3&Z1V}_;3(S;s^OG+TNxqti2CA zO)Hjsy&+`N@2U3<=sD4vZYcTD_KY+ztw`nj!T?375$8GA&pZyt%r74dx3qysx2o^T zGVe3?iQDfGrv<@ja`pJXl;MzSctQ7JPGJ%hx8a(J6-0GFH!58JSAP7I&CR<^9OF6* z`8I^H6nS9BmY60^h@>k1tsZB~hL%Ro%i>d5zu@4Nwr zg$Q{j6%POUR9>rlD!%H=k}DeEAG2VeA`DP2#$sD|@igq9 zDCWFY2auIfxpKx*uu;;J+V3m5w>|FC*nqq7l?!y0oD}rtBvm-SDfrDn0BF*owVDo$ zXUM+MYADOu>nk!k(wfbHzzh*@?(eU~cT04=et&DF%a=G&0m92#Mb`#$jSfgvVq@Hb zqvGaPDc}XyqT>Z|(-oc5 zF);yommIr*fDR|j>nk)tJ@XRhReA*sK;oLu0($5QYB4pKy(DsfCl)3EwAsP3S`k0e zy7}0x$BZkZDxRN7hP;h2Hem<#uc7qRq@s}n{R1f_v&)b zs{oBmM9tv9-}%l3zc=4SF{MR}yef+qoVT){>-O=pFP08f(&RT8w#$C0LRI7QTmAkK zQh@kp=S=iF{BVoTZ%E~xrN5G6fy>;OG}PtnK__bK`B4?HF{E&@mSdcc!7_B0qTFQ- z1Kfq%Qu#0-pk?pLTz9lqW7<1_Y*E4_Bmp=sS%zsvt4iY^{Iu zg4^I0Bm@Q$G{GgfTX1)m5F~?z1PQ?j?j8sZ!CfcVpuxF^=Y5|a-}nFixK*c4)zqo# zBi+4v_u6aiNta$hW_vtu2P`oRU#J<{y~U{ThWkj6j*nm8UiZSMh1sF(X69F*-iS|T zm}7!2$`7nLW-+H0Fhp>Oq`9}ezf)J?c9qvrDt)%fAps=9Y8*=l3&#gnP+t;n_Kj0O z!Tl9-3_+Ad6mI;?wi^cfIx%~iBeqC@UmQ&@P1D6*Q=jGf#;hC$SN!-eyPA!o)4)A8VL|xH2tFu~BJ^Y&QncCU>CqfFrXK zOs1zXa;B27cJkj};X~olJa9iws1RMH7(`R#ffd_${v_+geOQ-jqBqlu9%xICk2cv$ z8W!mofFVaymX`q`W|J|%Mn~hIQIr3xm`x-mA|Xn|XJJ;ngYbfJC|5Hqwv z-4uHxx{-tV5v?lFYxyO8%L9*MPXTJo{0miB|IyD%1*j{{R&saD*9nn-A_^X(DT5Fa zN)sguqkU!V7Y;!746vbWp=D7%GeD+FPeO$Ng3QJqrER5mg4CpW%O_wgR1_c%)JZ{^ zl)yrr&Hg+-vhdr4Snu=Bpu@g7jn&uBY@l2cTtuo2F=Z=WU%%+k05|$WJldT2r}E5Z z7+|g6R&&+b9#!=i{j!K7WiIXUk=uJBl$B~TKr`!eDPWN>-3mwiFAH3tnAEror;8vJs{MTg{_I2$;VS%bTdi0O}W@D3fOwb>B zrKd|oFGmWhNfd!iF3gva+oH#s19%C)hnMi{HRO_ckaA6j)qd?nFeuC?%lyNFb*=-7 znM{6$$0SiDgDu(!V1Wu(1M~#pBQPDB@ZO0t4ktWR{~fMsDS;Npfc_ny!X3S)9yduk z-1hfOLBA>36kcpUETKHy<|3`Q?rc?Jbq~;)^8lUIIPX6<=%Sfpd<<-*Zya*4vVwX5 zcz!)NMrw4XA9c$AV6QR+9D|o;qMKv|H2|tcV6Oxmy7j5z7x_4J%pC$A2(FOy#nE~c@ zdGw?E>t|)6m@yyspB>tKr3g~PwuZ{>aY^GLvq@Rg;&H!-@25sR;<|A zK#_7={=4@KMs*I0q|Ae=Ix+2q=mKxfG$`guP*KZT(|?*T1y&>N7k)K<(_-1q%cX^V z`BZ+eZu3gJ)gRI>54yg-7VcWIzX)jT;Px0Co+DbhXb#4j3`J>jUkz-=^8&PVW>e4H zuFv;TG^oBDj7cT-wUgu80{(X?FwspjwxWnc_DMgC(_G;xGqT4Yv4O7~z} zSrx1)rJ8V|PGGOX6C2%jGdZLTL%f)cdPE;fG+ZHob#Fxpw1>UAi*(v5bR~hZhA?C( z^D^bA?Z3#bz(2Zcld<(#+rT|&jJ z&oAmrtroXe4NNQMpuLWbE;P4|`~Bv&+xfPA*Q(a>p<8qT#qL}89bH`H^E0w=Ia69a zkIt&yog^;|HP0-RTI?=CS2|o+bd+=!mJ=7bWDAl#dU@-^{k&+NdZi#sE`1VB=efo! zU5?O~rD`t~rA+FPf%bkm_)c?g9 zcL6J*J3gxJWrS_y+=|_ff`;#WF2ix6bwRA5)qD3P13ma+K4ly(TN#!Pp0JHAw%8~?F-;jLOP3!u)~vycgj)N(I8Pe^RcQn8?Xrr@ z%3!pleV7;7@;9|hOG{ulPugQ0MY z7!x5bqCi#EE~R*rvpEQ+c@n&Y*U-%g!zzhv>~wBK@m5>kjrN`Bv(#dBHGnaSin_gL zG=_hOr@QeNDM%+IAY9SF+8)4Wi-3=7X99dWyXs`br3uu6J{(FPjG$Zzubqg)vMZ-a zh`76vjQbe+-c0^VWjkmou@~7EcjxFA*LELZ_S5)@+uYm_+xO&M?aQJR>`mmue`QZG z4EZ##OuZ2BhX4m?S!S%y%gL7-*3T?m(O(KYVS*f=uK{sWx!h9s#2Q{N;Kfb>;@d9( z$tYJOV%9yElBB|NX2t7=sL$O#d<2D5SE$c=xyVM3jW@g{2B`eEN$18QTP$^|JVT^L z^_#oV>D^a+Ps;Cb?XGxhv0GlPcQBMGs(jWU#tWmj%}$EMaAGA4cw&3nLYWQSpPXpA zw9qZfa4Or*#oTZ_T0zoo$r;2=|B)I{%{L=1!bHU%nha^g90&#M)D})>J-cJ1$FZ0y zH0B_OIeSl=FDX9?^YT^)Tv$;F22?Pa)E|N9jEw3ZKU5$Bg zlmg8@Ted6@{ z7r(Y(&DY_0+E0?7b-=?)a`)w30X%1u#rMH^>Dz*28rD|><>Tn_7sNV^D4W}KAU9bg zT{M7N^lt!A4OyzRvwCSKCn(`r)l5=;`zK{b&P*ekooMCW1l@5Uwq?cGj4^pguk$c~ z`WkxHWK09=XRvwtgqulTG_rlQQXUh*qD8BOBHU&Hg?w5Gy_s|bh6bAO34ODcE5Y7! zMBy*tp~sbUx*D&4?xKGeeupChsR@|OhtJ>Cd&c}q*n@0y%EzuZ%j1Pd5czA`*8 z%O}F9qUE!i#xi>wnlJv^6D$hAlP^Y$d5X#rSJ#E>p1UuvWW<(m*mTK3x)_oNZw+?PDRUDD_iytI2bw4K4>2?NrN`e!*wS?^2I1;uvK6oNOai zdC2PhC#j3qGwkJj#`}G#plv6aVfK$H^35e#c;HJ!xTNsK30|)=GhZ`{4D!NKENE09 z;EXC3qRqtxtBFe(bP<|%U?bcEmCH0c^!+@@KQKgOC-08hb|xUGH^8Fo?h`w2Zb_eY z5gf!){cP!&{w;Zx?^+546u?Y8NIjf)_p?gX0cOuvIZqd|1V7WdRbpud0X!tTw7c!c z^yLZ^wG+a2Vv&eli~1=7Rs?=uKAQL;>U55&j}If&jL20Row%rgkgM)cP#svPZ@kjU z^ABM_{drawYt6bXXiwrL3rXGDdslLQslPW{!fltW;V=AdQ;^?}n9nLCaC;rYSBrOD z7WP^(n??YaO2b^AlPYDnElk*Tft)tWp+XE2yFHIzsgay_r|x2~sjLlic-+1!ej~-T ze|Q)GhAD}8vyQAW1-r;Dyv-l?c?w_>b#x-(@OptTu7xC_;)eJFa_Kner#r9ntG><_ zKr&JHyfyP6fta~+LB~0JQHXx}E{OYCR$(!#KUjc#x^v|`5ix0N_2ai?$652)x5@64x` zg~S!KM5loZc#DR8@Xwc5Ms9C8U*!B(4@xishV_#b^=BuB)gQHNIk_*E8uIu!u1`+Z zn<_kitARnA^kBUjM*NvWx>VMP0`TA@-Fu|v0Adu;3~ z-TH{xKUxhPu5-W&(zk6Uqx^4`g0-`!T=D2&;dSgi0w`p}b8*Rgo-!8Wz!nat(uk%X zdtse_n6-BlBSs;Wqr%dbr&IJYZ!2B^ZJDxj{B8#o(S5%(;UqnrgtivnnDTMzaqiD= zQ=7Z>3qzk)DHgC8f=G{Z*U3)PWz)fY5=28croVIugi;%Eo*zre_V&$xW?28R#PY`) z?FqMJ(2JOP@$dq3R}BAZ{MBu6C;oQXo9cq{4P3h9#5UB)RldnhlA;<6$BqRK%n3*; zvmmVuKu^aPh0i?=0(KATh+}1@uCnP8QjBG}N=mNmOr%=o&cDhZcNTKmOThhvPdgfcTXFD?2iX{WXqt z>;drLo?3vQnI72B3)1!(qMYm(eC=OB ztX~w|v*W)UBxEJB&tvC#-!K|p6uFtLy^nHSbNV`)-V)w>AlhurOX!@HZi^(M{ZRO- z+`!PlktykR&$t#$z!B44rK(3u-Z-@_vJ} zW!}b6Md2#dC~*3E+UC{jzd8ADbDnSY;v9g2B#^XzY_`v3mU78-d1fE!wf65Xz7 z=WsAgGyOSDxkOL<>CM>9*Skb5W`PBGk$sOML@oTgnTT<~m1T)cK>e|PgYPR-$_N%{5jLx_*a zjwjqvnc4A+_lX@eck*`J*8_taISa-!E2r7!B&GQ+YI7_Yl`X8jaT!_|YHNgCda zEepP@%4-+d&DQ;UkN@=p!bwHGkjF($&f#+M5nPgFMfMZ!F%N+J3oCl)ytn7V^X#%d(Fr1*dBj3x@NtP-%@<4ATwF;r;7`f6Z> zut}~Zs$NOb_D2a;iOAheL8UrOIWU{RIh;{qG$q*8ry{4f;T9^)bCV)=Dx+vMzA<7* z-jCtWvvvQ6!{2cLJ@z($a**a{`|b`TBd}SIaWVup&Lv9cCIraLVTE|YGTaX}EydBJ z@2j^Pq+boQ1ct9USe^4jA>|?(a@L#jpjyKZq$_r#)W%jl{!}nUemFtv53_bQtDYx# zt4>&9dG;%Zg#oWSw!V*DG?+tyeZs^H$OrSiLTvZa=x}>q(#z&54`oyha!yb-AeHT& znBrJ}-Y$Xu@ea``QheLH#k+1)v=df=@Gm0!BYq4d6hz!bFpy|+l@KrsRlJbH%&p**V=sDadgUn zA3#86)b{R9BqJm!g=Q|CKDmA z33g=-0PW#F*ZTgcs1-e-Mag8h?? z9!3{czw=NcIp#8dY%cg^GS&3b`uICrQ>D9&Rfy>%zmB5tjv8z9*|FpXBtkNn0U(Yg4K zjOz_u8{MeT$zoDdzt354qd;+#yhRD3lI;HEGdVG9N_UQ4shqoiE9e2aJvC#F=owt= zI`STcgg7IrPK-FUH8RnscqNPt&aum&Rb;&T@SMv`;Um+qlFAhBMGKCghIfQzUaIMs+dlV-lpq_Ia!~v7&qVlVE}%+I>ETm?f1M@1 zOQhV2bQFsLHrtj0x`&B8Ubiq2Ygp-Qfrs@R4J?N#6ve6Aqx0 zU^3OZNQ(L(OkMcTPicQ=to4lTLr6^&!7{sR%ur%bEzUS;+|HVfLD77! z1Q{yD-*|(J;I~hU&6L&t1Y6v^w%a!mCa}WHDqck&#BOeE@A>s99t&-7Sau89 zj;F1^AifmLIkE9aZ7J3Wx`s`=UWm~alQKRH8zWDQot`cL7@Qzx~K6-J~PH|ATzkT2VVG{Op$jpmB zLEL}zdIhe9Uw2d>N4Fx|>=nCdtIRC3FJ6D{j}q#EB*^aN(P5S*;-4$+6}ranN5~;Y zZXVK(z@e|qxFg%b5Ip>T_Vl3ZZg;D4%IZzwe*AAY4nE7l@Yvxna!-7){nnsDY770I z#LArxgjH5SE;~dYt#V-Fk~)!%$j7P|)knU*R3Z1yW8UR#IsAmJzaddNUZwpgD^WQd zt>2C_6c0i5Hi$MSSUt@JF8|w}^GwnC-!iG{=_pO>Y=^Sia)yCUp5bz2`$Qbgadf@B z_z|G*yHtH5G<$PWR=uuG*qt@a@zy%hR6ylf3+fTM^sP5o zAkC5lc_~$NrajJ0^JM3zx~ndG&A6)#gpAEg%8GhBlXhVn6FHasw>S@l$v1(Dm^bOe z>}fF2dV2h&OHS1z)7i+yWKb4=4D;E$rH@0}UNvLUW+K&$6O87WVaO|OCGZ!r+a#6$ zmcq10AFwxu8!u0&o3d`m_LE@wa2WRb#x)m5iX9lg(K!W85rP>~3bprD{or6fg_prD|4fd4TF@W35zun`{c1KLSR z;v-b~DA7LfS<_NQ!&&2#ET6I6XJ$haJ0nwOx6k&#C@3g?H$LFwXH#cGu-j)FTPHp@ z0rG!F@ByD+Z?lkt{~6+JEkLgENdYWk=V%J%V&-IKB^N{jgTefcCT4s}qT>H~9QaLu z+``$}o{xp)%aIh>-3*(0UKm_{f32&nU&>#<^~?+ zf4$2m?`UZXto%B^ARGTbBmb{^|5=Bh<#qA@gP4DJ`p;cptAa@UEdRqcK_uBu8cirD z2$Ynlu!aXNAu_DuS2=}D=bkXtA%YR$Z~1O&pG5`%sB{JPO=qK=jaua zFrQJy)g37+E$2(J*a&bv+QD$<%8g{(n=I!^3R;MUyrXzaEVfTcndASbb?oVBiN4vZ z!Rw**U`1KA?egxisYEWl@5NJo^?uHMnm0vU0z?V<|Mkjx3thoiqJV{IPw7V;Hwbgr z`6J&rThbR+S`NbE{!MmV4%T0x4x4>S;a-)fEBZZ|KrkuV9hQbKt-0{WXz_l_Xl4Xr z#AvA33q=wVaX}bu0^eZRuF=ef=eLf49)vK5_T3mkf)5i6joBaO>s&d}5DP*6CP3(j z+>KTo{09+(M*PF_r$7mWNd)}rxX0G5_#-7uvG(q-9ECc>mW`m)>U?61I+d;bP!=N!Ra;cVEH@ert zR#BrBz9l`YcgX+A1Z$HOm&r(pL|n*)?kB8^PFiyc*xLl<-6!8O`ufm{$-*!!4{v-# z52I;JW)8$UkLBzc2oKf~BFCjG$JkXe<4D`PBQs&9j3Yl?dxR40{f_Jq)4D}UPszRs zh(moWeaL?GMC`Z@0ulYcN^-LNn`|+W3hs)K6p%RU4DkMU2p= zuGdTm^E$5j&uQ1UUz8??wtBmqLd!s3d~J0|;eOPAk6sUM{$@=59wWof zv{6{-b+@!!&;&b5SheyVEFv>r(Hs%EcprXTl6^;aSRyMFISD-S&1OPPvXu>8)LuJ?{0bT9E_OSxQUSwg&hpQkkjN9w9PB-A@*Os}6QTDJT{ zEX~uagta4q^cXNrQ|iEA@x8-n{AhxHMD+6qoUPsHMDS$GSGp~pemihmK!mU4d^mWC zY%+-Hc{NA}kr(;mck~O=Ii!0{02X5o_Aa%xnnimtX4S_}l7}s>LPRj&QKJ6``aty( z53T^!*fqMt+{f;A>qlwX_4n0G!kWfO1kTHle0wq17Ai*D>2{r;G zGkp9a`~ty+BD0Xc!j3NNuFOOqqr=c0a!Q*KLd%{1V}I-spz*7{-Z(;$Z)O$z>}kMb zn@SIhT}Hobb^n7X)L`C7$Ky_LK_VJ)-4d-YOqsS%x%>&81C>|WjrSV~1=&WqZ-2d= z6%TLX6=kN=tR@z49v@E*pI$3(~^%f5bQtgKI!Vk_yL zzLYg}9?pfeBpPKb$2CE4YUam0msT3(cA}?ld(zK=PcV^7V7ARg{{C=3JnxC)F?7Ug zwEt1YI%=@E3%96N{UkeAlaqt=f0dL{!4@<|CbHl73EMJk^2fh+>!YvkLIBjpu|Gm6 zh9DhPf$Wh7PRxo+8fIYO|I2c-qC>lyqKO7SzYdB{_w_U!H-OzAC5ihY^{+Ydh5bos z&w=?uWF%+riMD$|^-=KTBkYQ0|D-25JL7+>W^}tc>@b||ifuK6xh6&Vdvw!Exf&v+NYLiXIf9BKll0&iS2h?Of)V?@avs`(qXPZo}aqQ9^nupR4 zg(Rks5kGYRZhZmahjFFcg=HX0L0g<6&LjS?RFbFK4k4q!3eBs1z_f>gW_!y5>2BxV z*31y6$Y6|m2yT=jgO>w3e73GVuxF@^YcHS9BdoCx3f*nR`Co=gp(A5Fe{H?UNZTrl*@ zg^o2Flr!>=yTY@=lmg1K0St3hq7^)BJ4&tC(WdB9 z`sH6fUbx7huM_uAp-ahJ$rDHr-KXtQdB;5KMsK4}=spUvl{<`6GI9KA;qig0A|!82 zhsmb+mo;?IY{5~GS>bMl(Di}e+O-p*4C3R?lSm0y3HT(+5!P%GlaAc!czSH&Mn|&0 zv1krQkT;VCQ|gyQePSa7m(Pr|MDD%R})GU!~Ekffg|FX|As}85M?cDz{WqMM4tap zOJKND5XdI~eq&4E64aL-eN8v#m@**$7#ppU%j{ijOU+FA8PWV3o*-MoU1NSIm<8I5 zZ^yw)`r`%WFZ%68{!d(M9BXtA73_8eM5kOQ1IV#u>3y?{%I>ccp%Bon989p2WJg!K zXihnslu%TImyg}_encp9NF$+r%Kj9kU#0&p`jBkW$Ok@Vv8vZpji&cXxF4rqGujwj z4Hk%2`HhS_`5}6d0lY24Rbxhkn0V67g2LvWzq-rykI4IqApVJPlsc5viu;6aIZ*B^ zDN_y?Q;JNURYOU&;sl&jgi>Q>-~;+agM)l$5O`86pwOJ3h4gZ8(M`E*q_phLZ~!tn4RaULZ&) zxEJuZb};f?3CLRs-?)D(S<-}ryKhLDc3Z{;B9TNk4t^Q0XL)0Zk$OJ-u%n1#BPgfG zxZ<01K9ZZQeD{^#$M{0S%`Yg=kq$fsLH= z9zFy9jcz(K=k~M0Bgy1;sjofo+F!V#LoQ9wT{aWKm48Kua#VnHE^O z#ni77V?qh(O2a@F_$&=EGabak{wLm?Pk=(L{#+XQtq@C+I<_q+)ALm7A0uB!1?jR` zD#_j0hxPBs@;u~Wdxzw3BFaak7Yl_rAN=}M$+vHB#j{#kbaBl0NVu%t^(DUv=y?FF z_#kGMp<10)k%k{^-Fj>>B3!R2=c`&2@c_}qH+5)iS}N!W4w)~1Ba{jTYF;gKRvN5D z(vVcDyQUmenQ3zJNV(0B6bB6OP$|MVtF~CezFHuZ*6=C0u{XOa#{F>If9yFsdi!Kr z=2)iCD#lWqDJ@A0Fd31YFOyk_`Y)OA-va0eXJ5?{)da&ofm~^T4c!prJtrgBNKs-| znQ{NifJk5~$0ZR+XyQ*PmUC0tx z?Wd;HTTBwTlhi?*f8Dr?X-ksdLu;m7UNsx@fx8rk_5IJk2|lDr{=Kc^A~{ z0wt!7V@Tat#o}r@fX%?EJ^W1-VTKMBn%}@XJWhv~V`=x2_+dSSRPUw-SKqVQpu>Ng z$eLy3B`#LE<7MF8quF1)OO7-h745k1!3#JuqEweu=qT`}iN#^r3jOgd9|_;v{(Ro` zdEPy^S@n6rtjTyGMqBaGYIbu_CLxj_`zf1~>jVuw+;@L15-`;GYr~4b^Dh zC0YJ;x=68{*t~bx>htB!NbQ)=i|6%+(`F}Vvt?U9zh`q#crGv2X*bC?6aI_uHF#FO zp!nez*+sH2D5Y}_e^Y^-`4!mCZ3DyG5fk@0juxfrZ|dq}!O4PoW?7Ge93#$Gul3HL z3gM63^zf>+(NGap!~M_sL7J@l{D9Q$LWc_; ze@f*}hYe`K>AhPW_s>Wqn}e@O`O1DfHDm}Ikv>oT_v)%YP>K%@q2BTF6ZDBnHVW*x z)(U9zk6Q3gn|Qb+3%su+@Dsh?33mnd^J2Y2%h*qa$xaqqzJ4)RS64?d+*$y4K|8;t z0dEe`ZkQE+3BjG=?#VT*`=qKk;fdH@-yF^p5LvjBS{~c!bJp=DBy4(a?qxepS*`)D zk2DL;iv=e&+8l;-TCU^>deA=u!@G?iDplJ3Fn%!UE(XGjkVSRN+p8jx`!LIem8`Hs zDfpER;dH*z7^~WRthZWA9wI1A-dVOJX~UxDI&7Ev@Ho!kFphTP&9Z*hFk2oO8A&Xj z_KkGQiA}%#rkdVoCRxmLPWRkVBOfZ5Lk}V#*y^}Fn5h~6W@;n~N_+V3+{oeqW1a*7 zcI&H|#3)a^eAd|p8#LN1b&X@Hfh=-LP~?ehS#hycD!ZvP(W=mk_2fnahI{b3( zNT^%qZ;F?E^u~9fqRiw8i>r%wEE#G^fYt^v_%3M686cy~a&8%EcQH0SuXXTZY)9ko z7`demQ$!e`k^Ee8s2ICle-y)OpK=hQ2|3d;HqPbty1&xvUJ6W18e;qU^#jV`l%&Df z!AxnB!@GeY;oUVqm^Uc~!(zMCc}%6!y1|br(RKV^?yq_jQU1OS{EBP(NAVy7)o_k* zzopsMY`*$5v191R4e_=&TR*Z+GR#mo@pH&uzpwJi)n=}mfb5z3b!q>lffD(JDn6vf z;G1n-iQzHk*MYQ-cW`_hk5g>v_PfJ}>v1&%W7*cTV!0y6!1G=g&3R&cJj-xn4!VYR z_4?r{u?xXY4sprX(>v2KeXck=arhhV{_Zd?YMu}YJDh0)LLc?sL}1mB-l8r!>$Dw? zbFXs`MbYv;KhR zN2PpoF+O(qZ{=IO7F@wur~Bi--`-u}(-55sBg;1i%Irk)Ur3^2wfr@7sUg7=Ve8kW zlHm^tjBwD~@7S9QCo8kTQ?k??>u4d&UP5U$=^wsVcX;Bv@?w|@?}ungd;G9CU7eAQ zM2%-v)eraRY`fw9T~qUCabcmsI?K)WqVHu%_Hp&Iy>jxgf*OI@hiOa~#!%mhPd7;9 zU(bEzJJ-C<&-+NxO0Ud;V5|Fmw-XA5C*M_Rh;vvw!V`M%MYJ0A3HGsg1YUN!=f%L+ zkhjE*q~dmTD6J8s=Fa032zr$f@0l(*BW!VFa2u96r&)=dIu4kxh-8HYY=dnbeH27c zd)03vLumwR5n=oA+*ym^QjJ4`S@JnPrL<2^zO=F946tbefkXklX0$Bq zxR@s!_3uahKj9tol7|g8a;`^yWH!RV;biPmhxq+Bg7 zUFmqS&klmR9i}kx#^Swh)zgOh0x7`+W&2JYE?9baXu6A}k zxRh4*yRm73kGcDN37wSD| zKjr+HpL*|9Cq|Av##x)5gxcY@C58+TTd$vLYgEc;ZC5FCCAU7}8Cc?vi6Wo46IC$X z*KWnp@c?w3_ zWy4MSoopb}FLQbCS=>fo0Wi^9Sc!<0bs z4=!M&0d7~?;Yr=RVr zln>eaE`*YddGh_kvaeMvbDyp&e|i{Fv6gvQne_YVs|!TZqxk{m-Vd1g{CsueLUaW z-tGz}my~VEG3_!YG;gNMc}I_DRa6IoM^{J8-jvLb@_mz_-u0>2nyFew#?Et7G3$s2 zHU=#=l~PXFtyIHLa|tMSrF!}s%BSw1eLl1aX2$h^b6qMySlQg2;M1P^V@ftHIqt!` zGex?evOE5QeRqUazY~;Q%9C*>I0f9HgDFol%AxjVvy%06Xzq9$rmX!JP+x0GN)D`ed!b65a;~w~ ziD!9qQ27gHzj&5iopa5XoADQP66R7Ns?Ns03)UT?S!#r<&lamYC?TQ}rBWmsbHXN3 zi%rMR$ERd)&P5?tHk4lHzeBtpeD%A29(w8=jb+^Kz7QOB)J1=KxH%<_99uSd+Yuj) z%f@>^R_LWf)rNe9?&+U6(nT+z$d>8hsH{G(-`R_W}PvFhD)xiNLTbPc&_##YWhum!G%mg~!^#WDLZGlCFG`0A2W%kU3W-$84} zI1JCT35c}1tes8IWX34!J~->Ll3d+U3*DLYHL>>Zf^)SEB6{3b(}x`6#prtb6VNrL zoA;2JidAuwZ*lNev6QD+wGzu%$LBVEOmMZv-y2d_m(hGOOfsXRX5n7vVDqMr*IGbI z2Q07(zDm9`?D(MNHLCod4<>TAp;szhA3U|}e%omqwVA&;`P2<_BkrvJ%qT!qA=OR^ zX7R|Ch)UR2Jq>&$a>R;oQ_NgRwoH5oUG3F+921Iqakte|w7uosN$lecBT_bD#Z>U~ zrE!6xy{2j5S``^zA z*>wlWHL*cr-pTv$Pt)LboN9t=2$QrV6xy(R#-iK9c*A;Y<+iMpgy5Owo4SYJUj!Sp zNNSuZqj8~Y_-M#G9Ya0!t_7MRgp3;|;hl|NknYVqEk}?+FxuikLNqkazWxp4jPkTN zgNSgPHe9dwcloBa7S3o{QpWIn9jw88oychWe6{oE3jb{H#kZ@FF500^^2nravn8Jl zqIZRcqtKWE^C6Jcqk|F?TN!q_$Xl*9>QcXn9}r(1=mI{2r=qj5u~mrzKlrstl@<#?u*LMov}UIdc%g4v(Y9MKb zqsMcTW&M=_rN$;wQHh47PSRc+jK20wpNh&_*C_4Llvuk!hZ5Rda^3G5v~oo%!2pC0 zvBu7r3Mo)K@{G}j1hnqZT2oYnXbFhsx5NV9ISc7geFupt+i@<1-|;JkL6InNWR3CR zZ8urQsg~d$7-2)-1tP&alTGSd4;Ru(sU^WZf1>QRJ&GA;lqcsL-d-Qpxd21u5`7$6 zhZAjp7rzVR&ta!UoKJy@xU^;+-4!MYcFzeu0nKtN!G zMSKT=YD*+-Ut0+&I)vvKwO|{<%xtSDFJDoFs6{9VgE)t=15bN@n#$szSjM$1SwPog zzzoTP;IN|Y9t_j$0@l6Aw13y>l?mkSFWyR0l0QJE7D54_>^uDsV#(~>oKWzaZ)nSN zi9Ff-j4+5Q{=RtTnk6_a4fr&HQr3&H<88X{tH1b=C*MAR%$(J8G=`_Zkgj2$pel^^ zt_|kq{r86CNKF+(;_3Lj@xGNR$EECyadnf41#fCd;b=qyo+&mdJh>L3Wvgr)=-<#J zgawrj3nWS2)hy5_xa?FqPX85xu$oev!^jLiu$t`TZq1bRib2U_ODM*>KXflA1ik zC(px?vdBHgODs{XnkKraYv`Q6u-abw&LSyYDfU|ejNPeIc8ZPZ7lJlrwVIfsgfhf_ zbyR!Ywd$h%+maXJcjTbpAguTTkq zRO3tDn$sG4YST4s{PBM>)pryx)3$C} z+^hJHP_k+Dn-gB#!6RL<0t9oevT+Uh7s)~dLoVJn04^xL!k;IB4UwD)+wQxZd0MFmdTcZ7JSrq)E=Izw4QL> z_oz4tHw+TK!3~Z_#EBf#35@Oa7yC(M9u1pFeV`w{Ie-d&S_R;ouNpg*+ul~2SS2OK zIT;${z4(hy?L{trr{nvQS#F>U_(;ucLbir2l%JB6OCu9NMN&f+s%(4xhlNRBG_O9< z3Gw~*m!SaDoZ1wLHc_oV$tP14RltET46XicaMD@$tJ)8mkArIUuI~_{ zuA5W-a3*`?_bvD|olD)A*a@XMO;ajtgqf|Dn5*{X@5E(M-5p>&7}9BQ`wpUEo+F5P!{9y2uWm z8&v5=S#m_6&XLCW5BS3qsk_QzSA)5MYz;GHyOi6bp5UCkB`lWnl1hE$>}Zgg#cpYP zcCpdIv1;x^>CV=Qu7=*F2V1`q0;7dJ%i9lU_G|T3$5!!3P4Q43b&ZOD8B7~=A{6Hu z;sXx03k(5ALH|>qG5VH9_T4%B!R9Bt#6<=M6#S*W^r-UB-W59Q_lAEdqgkMvWrYsU z-Sm3!u8r?{O>ZxNB$XM>Tzkjyya%~0TTEa|iCYIP2mCu{`;o^)us8K)vC2Rt1y)B6 zjE&*!bI&Y{u1rr=iXDAg=fzWZx`zwF85uVz%Su7nnSheo)Zh1jxbnIDm1hYO3@(TVk z?Ub^aZ`qcE3Py!sP^qtK(ba!tZs2nD60Ms_sNaQ&}mZeLY}V z0Rw4K8_lw0HI0f2`}98_8YNgq23<~3~V1@WkWGWS>XXnE2Jt5!JV zD0}iGkSb8wfKV0XfD-;ZX84Ha(9faH!SoqO%xOdzkCdo@ufcxg2nIuH$p{76GtGIS zCU~{5fbh#eO(05LeR@2jOjMPRoZivOIM_6rca?jIv=K_QCJF{>V+tw0van6DSRM4* z5}d>F+ZD1WNw*GVb}F)CU%RezjasUn(%9j$$$=^p;H-iI43aX1PZxAw*(-(x$PGdE z6IL`1OK*7MEDDT_fOBMAc>wBIJaHDoY4{k%GQSPOTAR`}`4YYXeOT(CTTd(G zj5n#KXn%D!S#>M{XgD!AmqO{sY}Gqhq5e$X90=g;qal^BCm{~4L1jBeY&*D_!Ms9x zk#E0~u5r<>rAIPTA_En@Xbxyk!I>cJ4%eBvIsGIjN_fT;Aa(5n4#Yw66b7;g#B?Hh@l%}wynweZEaym=wqP3T_Wmg}iszg0+7|)^VxNXv zCpj{`wF6EW#w5NPVtvQ=IDgX1?r9i*oo5RSil+jL=&}pPSzBo9qX9OLLybG+)yD`o zROd`stP1`dq?hgiXrh;c1ULS=9q)4cu}gA)oKHU)r=Xe$DWz5ZYne(yp9Rd2JF?2|=T7D>b^$#?JI zgvz;_@9K83LJVKKIG~mwC(uF6N1Ls_oIl9`yM(HquiT?%OJNviZaj&bxT?}vP(*}S zBn8nsICpH*_;h^qD#TBRtf!c35>!j|E&WpZ?zzUWfE=Q?Q?hCnI7!RM$iVo;WdH}h zpG=>b)4cz2F>!Zh2Aftca-%jkSzFWyf7#+UBB?jFbXQeYCF>q_LV8aX3%WpW-!9T_g*$=ix{5XbrMU{Cbqd z;z9|OF^~;7Y34!-6`1DPD(5B0%R>=uqNzzV#_!W4hiWZ zzN!RN&ZiiJ?540)=B2!Fetr8)5;&=M|stQjgzdUIv=PeM>JF@n?0@%PB5tk zOaQ@?TmwT7>$;=jW(Sb)(|rR@L-E!I%l++&S(8mJ6E+n<3pTXzi8LB6A>F51fQiAm z2gQNoGt>EvF|SJ3-U=IGK1}qG03jZ+OqwZ1X5ivwzzH9)W&Nk|zyQ%tlzbv6qam6q ztcg*P04RdjPt;C#jpY&PO)!!msw6=r+{X{(XCsxahaOob5i5SzL}m+Fp4YrP13Cl% z@B)wJrRkT%CG~uk$$la1rj(p@a~MqT%bx zax28s19CK>4MY-4& zR^_`h$%n)ecB(8xAmto#BIdH17B!h%URi0KyBJP_2884g@#^niJ;I`>KF<)Ens!CD zsNc+YSaTwD=)e*At*J!qoV!s&`iM&orCz%ikK}>KGV5ffyo^_nTSZ!V=H1N3o0_%` z*lGBO#PUaAA!eb=s;$tYEI@?Ru|h@3S1*Z4ZrrS}yomUPQ8?fCG~<2t z9-c8kV&wmKo!~HiH}=CC&~65*!!Jox0s=lsHoWj94a2c0;=5mKY&G(q3~ly z1M&*@_9|rsyd7$1^EPXm^kN2JD^Ca!uP*=eSN-aTqoLUW~MCzjjQdDt}_9t*9kH76R2Xp6G*X zG@Qc$9971lVS9sS)(eqb66sRy;;nNd-2)5Qcw$<#My4pNf0FE+*y|NZWRvg%mL38=Z#X?%AB)kt)%D44F#gcrO%?j6kd4SAK!RG`WrbH=DbdN}WQY+m z!w@b4DnZw1a5Qb>s;u4*>WQ4)ba#GuEo@f>ZMHna&c`ojzNvV%U9Vj_Ij+p8^DOzKfpD~bjINTNsoezk44fYZ139ZlE7s^1$bJ9eKqiBXF{$uruf)4`t}S>z!F{2fUy*}lc!e` z)dMiH3`ruQpH7AX1#Pzf?P>+01T00eG?O%QE}+K)aTvbi@GPNm4&Fma3?OM2Yd!?W zDg82Ba}jfrH8G(OsrY-h2{M5hwv|6$wC@mNR_&uXI2y6o`MM!n1m#rqF8(LroH5o}Hu>KSJUjPeF0 z@cMu*$}`Ag>OMJ<>2v9CD*2$JSs*?z-=0;k<&#mjg51C zR#V+fY!RDLCfC!I*HcB*O``c{^N@33^Wo+i6Yuz;pMD$+{K(q5bDUMr5-hh?U5wt@ z*>T5KjYtL>$Lcen)X0e75p>RI$6jCwcKH2@-w~B52Fc9Ha(VWGMPS5|Vks5TVRA^YInO;|03&-^@`Y~QM?cdj3Vwknd zsMlOSVAhk@2s|(fRtTxBy5Y^G-DZG8b-L3?q%4%AbPTc+SMV4~Yg?yeL z?}6lj=#@cnp%57E%G)AQ%LZzE0y9EY7ak(L17iH;Q{t2d9HmCmvYp%wJ)hc}wuPc1 zmH1bA(UI{DUZ7TGVbY75b8L zW(j7o(1Yds(Z$>EM~2BuNj9U+;FmCLy|$Y}m-R^;rYDgGu|wPlJ&gdW=jk0Lgcx{0Nj)Oadnq zOpG^&U(uNHPe|1O`>Q(cXC{!DL$IemhfByB)WfIy_roaJbl zgzQ`3@ouw^RnbMDm=ussC&V>g$}wE0X*osDPahsT1*anD}pH7y;ZTw zS7Vca1K^iR&GAxy4zF9o(-w@%QM`AgLFF_CKa&-HyB;r+%yrQMZ@W1`kE=vY0(~vW ze6xm%F?=&F`LD`2O5v-D#Cu*E`WiTaGBy;yWNi@>c**D8ny@~z_8i7WB z0O|4PL3P47Q;GamHe0v;e}0EG!Q}o%HaGPdd@IuGr0oj$Wlu7V1)4 zF2uP3Pk2JC@HNk$L~MS$Nu;DD@|s($;emi|!;oGGEgnkiM$$*sEHL0teD%s1&o|wW zg<|M*pnvLgC!-#53tu28AI$$o zIzm#D^xlBcBExY|kqi*BC5*F&miLE^f3NFFnh49bU6-&ShC!A|*yTjXDu=!fw$)%= zYAHYj$U*EsIE$@i4$OiD_FM&^$Kn%pRXm5nrMql+ja=B<3sHmuxH+`6vzwh_C$at-OU_?nX*q=40#fU{+Nt=P>=8T$l@ops5=?78gz&k0lV?dA3&~%VYk-8-QiRGTM28%!P8ajq^BzN>Tvr+X$jY>!%NI zqrM8%h6yh7cjT!pzq_UuIDd#!o-II{U|p)&KvMmBeK4aN`k|_!!QCUu%SV|q+;jrlai8~m(>;zI!+}@GdN)vc0tHU)%V;W#$){krmW0x1|%J=|O6PqgQ>s6CK z(1Il^twKzAv;JIs^nFN_Mo5TY=h2OqoU_$(CDQ+jQK#O%oC2x)Bfv8axvYjV_txLy zggc}QZMB}P#_~#~$wa8YODd!!SDRA8fR9A(G<%WU8$E&0@gNRAs}5jBLo^Zgv>yGy zjiuG5J3{mab1h+;=aZy^IFUD10BScn+E@7{{{vy~-D9gfDUr4`P6QBazJtZ7VZN;+ z85h4hG}errGE}rG5Vv-z_-ioh+ngK^Y(|ZLw8fHn0zXVtD@|KE;opZI1)oAX3X)h7 zYe1+DZF=!U0hMOVFR94CPz)`SI3Nx`=AJBg^*3`as)_*haL2UJoAi`sFDBl!>tE55 z`&8$u<(yPNb8`65On(hO=0pxeOIXzA@a(OYk>?|`$=5?lKOl4nZq4U?6Iy?zffk=p zA$pRtX{(NjT#0H4W_1g?^$SQkW_V#FnXkjn(zsRY2pgP{Fkr)^#j~z2#A0JH#= z19pJ=i-c99>zm!mBvD79N!H!b!5?JmuCxD?{%pu)M}>a>guxLnJ(b(@TC^Zxd85wk=mDSW4+(Vff1*vZ{ufD zv#@V)@(Yp|EKsM&=p2wOcN+3Rzy$lLfY$BZ=*lzon_NH3mk5oOQ2?%+SX(>)&&mpk zceTQ#b#j#UI!?J2*9z5Jdixnj@bdFzTynm z=};gzw*s}av|JcHldHl1F>=w9efc-rb7+aGNjWd@^dkSEF}iH1^KuG-Vy5ujb~n%J zM?Uv|zF}r)Wbes0R4bsLR)Vz~OdqF>ST;;3%EzxI17M=0cE{$VRU;H@%{1)TF;U-O zUM%eUYYa&E-B+FltfBMol$Zc3cd^<*B3>*TZfriCcloNn138BeWxKo_7D z!{+Fwoc0o&m|)GB6h+aB4r>2_o6!*_8NAleMshPT!vY&P;-}~{wjJQ0?=C=>QY0d{f3MTd^2Ot)SW6Yr zT*o)8Xcz3rHCD-C8|(6bzUf@|>z83EBiADF14+;mVgx?N4*l)bPrVk8PcI*uOWpzC zhV+l$nyvF-uMaBTM2erM*IQXRQkvqeew3l;>|6-I0Dz0nQr z9VkE!1Jp?Fp@2$=-tOi|j8o&&w|PF?7n80{Ub!M5B9A=hVTGkd{?j_M9GL_%byq(+ zI;zV}`xc0>=CB|vbM&6o%`jc6zZ*${KSBYmBt`5h>Trblr?C#S5fTEEse3KGU)=sM zh-;<+K`_vWd+x^mj8>8u@ajDcEGvSL$Dl)!=TG9BBLJRVH>@1ISbG`KHtYh z;pHMad;2iS8on|>t2F}DE+tCc`M%^;ihNR%_>CFOY}0ID_wjDmILRtNo3}{I_Mfi0 zpzDj<7()K8jc1iy#)AXzniLvQ%yZF4ANrqysuTbk#cF0KgZnptbI|IUVVt#8lf}w< zdjcrSe7xidK#q(;n`l`iVmy8?(Dm5^AQFHM1-~}So*orLK@om<{aXMq^haIw7XsL7 zK?`UQ-(Ytr)~iw3|Lq&@RB>I~qW|e|O$l5Td3tmQQX>Tikei_(mNa9qj*f$AQM)hT zEZ&HA5hnlR9oD*wgr)?h#)O#pWG|^A`!t;UuQmY)Agc;CLOypfyNBG_@U-|R_prb# zh-KsKRzd0qSBBhL_NGSh`CdclE>uCVY`{FD)t!(`+<>Cz8s0l z;$J`-1(=#yETG1E9Z@#u7ra1U^;Y`*PPh=hMT7^ZCsKUH<@boFvso&^a8YvgEyBcg zfFvZEc8BfG1yw%kzGr$}m~Xo*gzA)kJqkl^JZjIHG}~eZ)nbY@epRDF-+p%DmTr~+ zQIY#PZ0BptJ$q7qu?mYZVK)WxDjo58*FmIJ@K}3{*BoS*S5;Lt_hK1B+%(JYThHNo zy5gk;lq#RS)?EF?bfq97V>#1Q=sLGpT-6 z{bNy-@V+zAX;0L{-;dJY-gIk)z1eII?QKB!j}QUfVML30ojrKHtu({`TC|%8CBh)= zYdkEiBC;Ldel(Ii&*+8Ac}L3H+qbFlW*-$k0Pc%WxUe_@#o^&${{p}RoZL8fR#vl@8mHVbV_0wC4qTI;zJkw){b z{@;cl2QjBIC0?IJZ%0=V@YZK(XI+KQeYCMuGNq4y<)qX-(l(RbAC0AZ)1n$!p_5n# z{(_`gN2>{gw6AC5JZNZ9OZB4W^!ORE5C_-@ce2B?B`=~S9zjJNtB5irXt`&ld}{il zke_ekBq_N&-1MSIzZ9yL^z(3E{P z^rSu*qjdP7;l~L3K;HN)}Su` zi6=GlL)SYmXMT=Q`go`3rOztX9SMw<^F0{;{K@R6@PNh-1;Q2I_`^!0McSpFjXFG*@N%T-TC;dw@{ApQ_5GMV?&el9yG)2UIFcmL}tDq8MszWh1wqAe zPxZzUDGScQ>le444hwDkGDLBFW)H*Bz^KQ(Ryb~oUJ)k6>U4^ODxOu#0|wf7CWci#n^b@wYfLD$QA%9P+xmsUU|b9s$m#%_Z@E9B_vpb)eRE0u`k3YZ*@0KmGKp=(p!+~oI0 z=>))zuw5RUf9wE9!<(p zX>;G&X(U*d+M08{H{_9UB%`3 z6=4A;5`X-+Eu8YhUP;kYH82VkkdK;q%~>ygD(VhA;sU_vi>*IMF!uOX0@VpQ=G(gc@(eR9iaL8NDRtn<%w@LxbZ zPbx+S6T5`ngiYoZZP!zd@6otCDpRzS@&Q(6i5>@jo?4VOvh+h5v4W*j zil&gff$C&I3)}Z)|2g{sHNe{bK*IuDrg)QlpD9LoQfX!i{w6p$a!q?~Y^i71#LUI4 zbAC`FKkFR5noJl8w0Z=W)Tb;YEUy!EF7BH4J-Dr1cs<_Q%dIt>|F=6~N;mw+XIVDB zIc~17n~s>OruRZVDIxmjNDQ1|?fu;Apb#sJ>O@V_Im4-z25zJpX=l!{ItFXIq7<`krYsNj85CL~Zmq+ynfSJ>~Wj>;h;JeXkunm^` zVmo(cxLbp_l!N$Qmeai>Sht3X_wVssqbp7p^#K#0v#$;77@eO5s>bcpsg?<4zj;un z180?{g;3Y}v>c;k6B$7>?bDI|zslZ*U$Hh}%r%7%FJQcv#z31gG)uT*&(AmxA?)o% z1f;7O`A&Ey%LmG^?9i|7gyeiE8s5m|aUp9mF$5XcYP}peoDq~2@iF0}XePlgxBIg9 zU`ir87t5<-m41y_opgj!PkOn1MYe7pa>^ZICXed&w^|$fMZj1Mwv3Of2XERdyIYFS zQhIGPyD~TMeosJ(0M@K>MSN^{a-AW21NZx`uX&%PXHHJem%?mTqSG_$BZ(j7&Q4BF z4BYe-$GY>hAGQ8cwlJju=6yd56F2XLDvV_1r|HAe4ZqFr8u2gVrlzRXsNZZ1-z9wm zbem7WajZ7RcHxsI#(Qc$aZ8R0lHVZ?Wvw6wlr`Gdx)<}LTHO1zQ@inttmQ@4^iA7k zj;Z&>@6wO;<(q%^MI}l)bXo(_a=6P>S7b(O=^Kh#Ru?+~p$k8b>!@p7A(9U~dgxP@ew3Ow&O7%WJ7a3SE2YModbyX5P0Qy3n4rCij8O#+U~AN2nA zZ?@Cag|5vmoU$j?ZL$QPAwBL$%t-TkYG1WXecB4&%NrIy`N}0dC+io%RdJ84`2k=x zcQ~InkG{(NB7b?j%!)AnI&lR~zt!@iPF6kDHs1=|W_A*Fyf+_HHvTMC9d8x3^IOx) zLrY9|W|0wGsV3I9B=E^GqF1@Od2YRLY;_=YTMZU1|0mG(_wV1q#3~J>%|={(`EsIt zKIJ-}4;|u7Mo-(7tRLzJUXMNzmAO{S{l573gz2~H=f1Na<^ry;m8Si~cX%odhgU*vo%y*z8PtN(K zKY-mn@Kb)WO(QPg4ZN?e?hj=lWWJ?0Y_jQ>g4fqdq8v8l4pGvGuwj`8)PU}eT=;>( zo<35%H8Q5Z2gp0rFh(G;neOQkRn5}7udz56|1nTZ3ff<1o`@*OR!G5tDX(7YPZUK#et?J}B&SGq$HmYQ)58x^d3)SpC zg5FhAd92s9bIX^-nkE@^-W=I2$Ivilsb42}x@>wpyHfuv{TC!Zd42WVjt}B|L^56{ zCzWoiOx{4*+U#hJbR1>-S+Iltz{wTDqy43)vpp^Fz7a^iC%pu)%VaCH2It+Itkfz<$~dsqMonrd4i!J32b5B9mGzh&ZRtqsqwR zhupQ4b}*Umgj58P2!BWeY!kI9?o>|8*SP}kw!ELE4fYqpn2$G1N@` z!<4IPo$C#kfAvE!G#T2gLlcMti3wfOymsyz@=147>jz#1vSVa;z~t(al|f@AO(;$i zn*<1KwEMXEE4#>@(Pv&@%4(d!n$!(|K(S8CR$;u4#q>=YYST^H@tG;zYu+pG9Fn0) zHJLoA|B)*@-BK4G2%39eP?+{?0OO&|w5DX2ul<;f#_xL)A2vt=>*wmhJEP~TR_J_H z{Jxnrr2dS=pmj~m!N`h4iI;CV6Y0pNwTQ2U<_e4OlBS{n>&l^lJ^%!KCw0K5c=K z@2HZy)iV$_g?dZbE9zL0i_%~AkBkSD0TyMx#@>Rr6X!8$TIK0qGTa_S4PHGj~59Wg5a?UzGz$N zP4h+_tGa#PkMRN)er1lvwkEUU0nH9~LXKJ^Mjv(ktga3&;*HZY5rAH4{|oaMg?u~Idg)P@p7rtwd{>{LT*lE(P!(|1*vh}I3Z!w;G6 z@qYDl>b7{X6|Uv1l|(UIQ|q@vP?Ht1b~fABJE_5~qQh9w%g*{j`{g(daWbZ{{FdyI6#z$rY;9{kb^A~{e($}R|>pS|Ih3GP~?WnH|zrCw5g@+a6a|$Ip#hHgQ^U&3{le6>0P_mpY zAd|)HSRgcgIMZ*zi!EO7r0pY3jXJ7FeICwW=j3+uS;kW|5p%o9ewMP8r9I;Lml3+# z5ql`om|#A)nV(Vn5<9=Un_G5p2NkydLJZQ2*#3jU%YwI8Q1pd=*hLxslKsuM{8O`M z3a?eB4{1h~ESY>_Y3Y}@@Mm>pW3trE>u{JPXe#7cU#W{g)SrbWa;8?of~I!MEIl*n zn{X$4R?6iEJv!zDL&1U_-hv&%56mNyUEA7;lXsrddL{;kw4UDF2dwd%mCC_ko*yfN zh$TAdGfnO4e805JhhX$tW?Ayy9oz#@aNs$k(@9aG7o;_pvmFf|o2MJ!mavYznEFGI z9rP@>#Way~Ea0`pN!QhB^95-YA&2%$5F-~N)r#^L9MI~qhjoRmypDLXetmdP_3atT zbmFh9tp}ePxB6C-!X%>K6aeb z6EJIhrvDGvg#elmG&5H;`}IlM*zIvvYSEs9Am2rt^<_qg{=oRwtIuDkn8-CioNR12 z<=6i*_6sziSFL`pDR=jr7kw{cX8OC8@Gd9>X|EuF8T@arB#s4dyHd2(yE5&rVQA5h z*G9EPw~75({}W`VZ38M29;l8;lvyY#`I_r_!B|KmPMus4hNe0p!99jIqV9NBM3|E~ zMPi9?w6QtpZM_HEM@k5x8ffGh&!gJ$GtMqAU1HlRfe5%PmIxFGokDk=$Rnk&(>0+T zM;h2$SZOExK)(u_H>yUV@V4!Yy_v#4*<)d3H%XvZc0~RkV9h&l^{p4?!*F$ViWr83 zZ$XpG)BI(bvLFlJl6BS&fBqeNLaA#)b^9m@5oiZ@T9kQw$U{d2RS}^S@me;UgXsg^ zog$1{aN!!ys*@UJ7XVoS;C)yh4GGlDAE~~jI`o8gIkOT)9C(YJ2)?8q|62}k zdcQn+ZBQbP>X)~(G2QTdm@YGPi!I-mMEFg1(lzi=SWRea#~~MTD!P@y&gR|*^-5tH zujg2&IdN+alCCFM(gC3>U)p0u(fIA>dyTLS0BQr?pSC3kYR5&SlthVaN&Mk?8!NUH zErS*L?EZw{0nwCKajsWJmS)1^_d3Zna*-dz309t`NNr{xj3HN4_X_%7}&{Mc}jm|3igGKZ>&DBMn{ zC#8g8rXAe`hg-zXWulvq+tE!K8xC>vn&?V+gQjdS1tkA<)4N#rN#M0p!~R|Oz<<$z zHEOQD<&9pT!rrCvf94TxhXAXX`fu>blRk)sjj4G<&10=x;!4Z~)Em;r>5w^2FC#3n z8r1{xeIuZ@)?K@nK+b9=gEXnf918KK=fVLESu&+n#gS5irv_eBL9UGse-jHfpBv~H z&`X+{(VQ_Cf`tYgm$VLG$ulVAM*zqkAZxi{FqUtnniK|LVWcOVGCP1j8n#r_NW_OJKSwmA*3>$X921>)F~p<1Ch(zgZ>po2Ro#qOTtNZl`}hyjkP(?pB(Niq^u9#O$*zX+B1<>q7Z> zI~_Zo9a-vR+;F&Ss2yT|fd{=45h-#5JGv_ziIN!#ouQJXtiFNZjQqc$fG-?ZrM26E zc3Br4>7d(K9ArPAkub5n-7Q$FaHP1>?jc3$%m1hm(>*BIl`y}S*_%5~dqfqx&biz- zhLdK+u@|OjdNmMLGmhO*_3c;_!bTw7kT(Mqs|G1~pvWe$-RSS-!Q2VbF%W|+ zf#$*r$tOj>NeoV@o~ESV)>Fn+#K7yyXEKB)?4#ilJh|)8HQ^R1NJ^PmOsF&zthGcg z-(X3u>aFex#S`1o3Gb8?LpR_79rq@po=Y~JNr+&9|DU9>>HAUAkXJK6rsW+w$chfO9I3(Y>Q`J_uiMGj(ft(pi4FYgA@%~u zAEbmp6m}wbqg}&-sM-A?OHwo&jBt{rh^8C<^z3{JsxJtDgoN^UeDkBF}E6 zlx-=%zH-f~S{9?uCY5|(mcp{*N%=}fS@WxYu*4(m%o~RWx^)?$Me}BtkCp?H-d2d*J_ia)Q`pKjl<`1wX{;&6^e=nFcVOP}TL&k~( Sometimes the maintainers will refer to NATS as "nats core" and streaming as "stan" or "streaming." +> Sometimes the maintainers will refer to NATS as "nats core" and streaming as "stan" or "streaming". Messages to the streaming service are opaque byte arrays, just as they are with NATS. However, the streaming server protocol uses protocol buffers to wrap these byte arrays. So if you listen to the NATS traffic the messages will appear as protocol buffers, while the actual data sent and received will simply be byte arrays. NATS streaming uses the concept of a channel to represent an ordered collection of messages. Clients send to and receive from channels instead of subjects. The subjects used by the streaming libraries and server are managed internally. Channels do not currently support wildcard. Channels aren’t raw subjects. Streaming isn’t raw NATS. The streaming libraries hide some of the differences. -Think of channels as a ring buffer. Messages are added until the configured limit is reached. Old messages are removed to make room for new ones. Old messages can expire, based on configuration. Subscriptions don’t affect channel content. Subscriptions are like a cursor on the ring buffer. - -![ring buffer](../resources/ring_buffer.png) +Think of channels as a First In First Out (FIFO) queue. Messages are added until the configured limit is reached. Old messages are removed to make room for new ones. Old messages can expire, based on configuration. Subscriptions don’t affect channel content, that is, when a message is acknowledged, it is not removed from the channel. Positions in the channel are specified in multiple ways: @@ -20,8 +18,6 @@ Positions in the channel are specified in multiple ways: New subscriptions can also specify last received to indicate they only want new messages. Sequence numbers are persistent, when message #1 goes away the oldest message is message #2. Trying to go to a position before the oldest message will be moved to the oldest message. -![starting positions](../resources/start_positions.png) - ## Subscription Types NATS Streaming supports several types of subscriptions: diff --git a/developer/streaming/acks.md b/developer/streaming/acks.md index 72eeb94..c753258 100644 --- a/developer/streaming/acks.md +++ b/developer/streaming/acks.md @@ -14,7 +14,7 @@ sub, err := sc.Subscribe("foo", }, stan.SetManualAckMode(), stan.AckWait(aw)) ``` -## Max In Flight +# Max In Flight Subscribers can set max in flight to rate limit incoming messages. The server will send at most “max in flight” messages before receiving an acknowledgement. Setting max in flight to 1 insures every message is processed in order. diff --git a/developer/streaming/connecting.md b/developer/streaming/connecting.md index 20e83c2..eb12cb6 100644 --- a/developer/streaming/connecting.md +++ b/developer/streaming/connecting.md @@ -1,11 +1,22 @@ # Connecting to NATS Streaming -NATS Streaming is a service on top of NATS. To connect to the service you first connect to NATS and then use the client library to communicate with the server over your NATS connection. Most of the libraries provide a convenience mechanism for connecting in a single step. These convenience methods will take some NATS options, like the server, and perform the NATS connection first, then then run the protocol to connect to the streaming server. +First, it is recommended to understand the relation between Streaming and core NATS. You should familiarize yourself with the [concept](/nats_streaming/relation-to-nats.md). + +NATS Streaming is a service on top of NATS. To connect to the service you first connect to NATS and then use the client library to communicate with the server over your NATS connection. Most of the libraries provide a convenience mechanism for connecting in a single step. These convenience methods will take some NATS options, like the cluster ID, and perform the NATS connection first, then run the protocol to connect to the streaming server. Connecting to a streaming server requires a cluster id, defined by the server configuration, and a client ID defined by the client. +_Client ID should contain only alphanumeric characters, `-` or `_`_ + +Connecting to a server running locally on the default port is as simple as this: + ```go -sc, err := stan.Connect(clusterID, clientID, stan.NatsURL(“nats://localhost:4222”)) +sc, err := stan.Connect(clusterID, clientID) +``` + +If the server runs on port `1234`: +```go +sc, err := stan.Connect(clusterID, clientID, stan.NatsURL(“nats://localhost:1234)) ``` Sometimes you may want to provide NATS settings that aren't available in the streaming libraries connect method. Or, you may want to reuse a NATS connection instead of creating a new one. In this case the libraries generally provide a way to connect to streaming with an existing NATS connection: diff --git a/developer/streaming/durables.md b/developer/streaming/durables.md index e10023b..c5d2550 100644 --- a/developer/streaming/durables.md +++ b/developer/streaming/durables.md @@ -1,7 +1,13 @@ # Durable Subscriptions -Regular subscriptions remember their position while the client is connected. If the client disconnects the position is lost. Durable subscriptions remember their position even if the client is disconnected. Durable subscriptions identify themselves with a name. Connect and disconnect won’t affect the durable subscriptions position in the channel. Unsubscribe will clear the durable subscription. +Regular subscriptions remember their position while the client is connected. If the client disconnects the position is lost. Durable subscriptions remember their position even if the client is disconnected. + +Durable subscriptions identify themselves with a name. Connect and disconnect won’t affect the durable subscriptions position in the channel. ```go sc.Subscribe("foo", func(m *stan.Msg) {...}, stan.DurableName("my-durable")) ``` + +Unsubscribe will cause the server to completely remove the durable subscription. + +Check the [concepts](/nats_streaming/channels/subscriptions/durable.md) section for more information. \ No newline at end of file diff --git a/developer/streaming/embedding.md b/developer/streaming/embedding.md new file mode 100644 index 0000000..5fc5f4c --- /dev/null +++ b/developer/streaming/embedding.md @@ -0,0 +1,111 @@ +# Embedding NATS Streaming + +Embedding a NATS Streaming Server in your own code is easy. Simply import: + +``` + stand "github.com/nats-io/nats-streaming-server/server" +``` + +(Note: we chose `stand` here, but you don't have to use that name) + +Then if you want to use default options, it is as simple as doing: + +``` + s, err := stand.RunServer("mystreamingserver") +``` + +If you want a more advance configuration, then you need to pass options. For instance, let's start the server with a file store instead of memory. + +First import the stores package so we have access to the store type. + +``` + stores "github.com/nats-io/nats-streaming-server/stores" +``` + +Then get the default options and override some of them: + +``` + opts := stand.GetDefaultOptions() + opts.StoreType = stores.TypeFile + opts.FilestoreDir = "datastore" + s, err := stand.RunServerWithOpts(opts, nil) +``` + +However, since the NATS Streaming Server project vendors NATS Server (that it uses as the communication layer with its clients and other servers in the cluster, there are some limitations. + +If you were to import `github.com/nats-io/nats-server/server`, instantiate a NATS `Options` structure, configure it and pass it to the second argument of `RunServerWithOpts`, you would get a compiler error. For instance doing this does not work: + +``` +import ( + natsd "github.com/nats-io/nats-server/server" + stand "github.com/nats-io/nats-streaming-server/server" + stores "github.com/nats-io/nats-streaming-server/stores" +) + +(...) + + nopts := &natsd.Options{} + nopts.Port = 4223 + + s, err := stand.RunServerWithOpts(nil, nopts) +``` + +You would get: + +``` +./myapp.go:36:35: cannot use nopts (type *"myapp/vendor/github.com/nats-io/nats-server/server".Options) as type *"myapp/vendor/github.com/nats-io/nats-streaming-server/vendor/github.com/nats-io/gnatsd/server".Options in argument to "myapp/vendor/github.com/nats-io/nats-streaming-server/server".RunServerWithOpts +``` + +To workaround this issue, the NATS Streaming Server package provides a function `NewNATSOptions()` that is suitable for this approach: + +``` + nopts := stand.NewNATSOptions() + nopts.Port = 4223 + + s, err := stand.RunServerWithOpts(nil, nopts) +``` + +That will work. + +But, if you want to do advanced NATS configuration that requires types or interfaces that belong to the NATS Server package, then this approach won't work. In this case you need to run the NATS Server indepently and have the NATS Streaming Server connects to it. Here is how: + +``` + // This configure the NATS Server using natsd package + nopts := &natsd.Options{} + nopts.HTTPPort = 8222 + nopts.Port = 4223 + + // Setting a customer client authentication requires the NATS Server Authentication interface. + nopts.CustomClientAuthentication = &myCustomClientAuth{} + + // Create the NATS Server + ns := natsd.New(nopts) + + // Start it as a go routine + go ns.Start() + + // Wait for it to be able to accept connections + if !ns.ReadyForConnections(10 * time.Second) { + panic("not able to start") + } + + // Get NATS Streaming Server default options + opts := stand.GetDefaultOptions() + + // Point to the NATS Server with host/port used above + opts.NATSServerURL = "nats://localhost:4223" + + // Now we want to setup the monitoring port for NATS Streaming. + // We still need NATS Options to do so, so create NATS Options + // using the NewNATSOptions() from the streaming server package. + snopts := stand.NewNATSOptions() + snopts.HTTPPort = 8223 + + // Now run the server with the streaming and streaming/nats options. + s, err := stand.RunServerWithOpts(opts, snopts) + if err != nil { + panic(err) + } +``` + +The above seem involved, but it really only if you use very advanced NATS Server options. \ No newline at end of file diff --git a/developer/streaming/protocol.md b/developer/streaming/protocol.md new file mode 100644 index 0000000..a964caa --- /dev/null +++ b/developer/streaming/protocol.md @@ -0,0 +1,265 @@ +# Writing your own client library + +You can find a list of all supported client libraries [here](https://nats.io/download/). There is also links to community contributed clients. + +In the event you would want to write your own NATS Streaming library, you could have a look at existing libraries to understand the flow. But you need to use [Google Protocol Buffers](https://developers.google.com/protocol-buffers/) to exchange protocols between the client and the server. + +## NATS Streaming Protocol + +The NATS streaming protocol sits atop the core NATS protocol and uses [Google's Protocol Buffers](https://developers.google.com/protocol-buffers/). Protocol buffer messages are marshaled into bytes and published as NATS messages on specific subjects described below. In communicating with the NATS Streaming Server, the NATS request/reply pattern is used for all protocol messages that have a corresponding reply. + +### NATS streaming protocol conventions + +**Subject names**: Subject names, including reply subject (INBOX) names, are case-sensitive and must be non-empty alphanumeric strings with no embedded whitespace, and optionally token-delimited using the dot character (`.`), e.g.: + +`FOO`, `BAR`, `foo.bar`, `foo.BAR`, `FOO.BAR` and `FOO.BAR.BAZ` are all valid subject names + +`FOO. BAR`, `foo. .bar` and`foo..bar` are *not- valid subject names + +**Wildcards**: NATS streaming does **not** support wildcards in subject subscriptions + +**Protocol definition**: The fields of NATS streaming protocol messages are defined in the NATS streaming client [protocol file](https://github.com/nats-io/stan.go/blob/master/pb/protocol.proto). + +### NATS streaming protocol messages + +The following table briefly describes the NATS streaming protocol messages. + +Click the name to see more detailed information, including usage: + +#### Protocols + +| Message Name | Sent By | Description +| ------------------------------------------------- |:--------|:-------------------------------------------- +| [`ConnectRequest`](#connectrequest) | Client | Request to connect to the NATS Streaming Server +| [`ConnectResponse`](#connectresponse) | Server | Result of a connection request +| [`SubscriptionRequest`](#subscriptionrequest) | Client | Request sent to subscribe and retrieve data +| [`SubscriptionResponse`](#subscriptionresponse) | Server | Result of a subscription request +| [`UnsubscribeRequest`](#unsubscriberequest) | Client | Unsubscribe from a subject +| [`PubMsg`](#pubmsg) | Client | Publish a message to a subject +| [`PubAck`](#puback) | Server | An acknowledgement that a published message has been processed on the server +| [`MsgProto`](#msgproto) | Server | A message from the NATS Streaming Server to a subscribing client +| [`Ack`](#ack) | Client | Acknowledges that a message has been received +| [`Ping`](#ping) | Client | Ping sent to server to detect connection loss +| [`PingResponse`](#pingresponse) | Server | Result of a Ping +| [`CloseRequest`](#closerequest) | Client | Request sent to close the connection to the NATS Streaming Server +| [`CloseResp`](#closeresponse) | Server | Result of the close request + +The following sections explain each protocol message. + +#### ConnectRequest + +##### Description + +A connection request is sent when a streaming client connects to the NATS Streaming Server. The connection request contains a unique identifier representing the client, and an inbox subject the client will listen on for incoming heartbeats. The identifier **must** be unique; a connection attempt with an identifier currently in use will fail. The inbox subject is the subject where the client receives incoming heartbeats, and responds by publishing an empty NATS message to the reply subject, indicating it is alive. The NATS Streaming Server will return a [ConnectResponse](#connectresponse) message to the reply subject specified in the NATS request message. + +More advanced libraries can set the protocol to 1 and send a connection ID which in combination with ping interval and ping max out allows the library to detect that the connection to the server is lost. + +This request is published to a subject comprised of the `.cluster-id`, for example, if a NATS Streaming Server was started with a cluster-id of `mycluster`, and the default prefix was used, the client publishes to `_STAN.discover.mycluster` + +##### Message Structure + +- `clientID`: A unique identifier for a client +- `heartbeatInbox`: An inbox to which the NATS Streaming Server will send heartbeats for the client to process +- `protocol`: Protocol the client is at +- `connID`: Connection ID, a way to uniquely identify a connection (no connection should ever have the same) +- `pingInterval`: Interval at which client wishes to send PINGs (expressed in seconds) +- `pingMaxOut`: Maximum number of PINGs without a response after which the connection can be considered lost + +[Back to table](#protocols) + + +#### ConnectResponse + +##### Description + +After a `ConnectRequest` is published, the NATS Streaming Server responds with this message on the reply subject of the underlying NATS request. The NATS Streaming Server requires the client to make requests and publish messages on certain subjects (described above), and when a connection is successful, the client saves the information returned to be used in sending other NATS streaming protocol messages. In the event the connection was not successful, an error is returned in the `error` field. + +##### Message Structure + +- `pubPrefix`: Prefix to use when publishing +- `subRequests`: Subject used for subscription requests +- `unsubRequests`: Subject used for unsubscribe requests +- `closeRequests`: Subject for closing a connection +- `error`: An error string, which will be empty/omitted upon success +- `subCloseRequests`: Subject to use for subscription close requests +- `pingRequests`: Subject to use for PING requests +- `pingInterval`: Interval at which client should send PINGs (expressed in seconds). +- `pingMaxOut`: Maximum number of PINGs without a response after which the connection can be considered lost +- `protocol`: Protocol version the server is at +- `publicKey`: Reserved for future use + +[Back to table](#protocols) + + +#### SubscriptionRequest + +##### Description + +A `SubscriptionRequest` is published on the subject returned in the `subRequests` field of a [ConnectResponse](#connectresponse), and creates a subscription to a subject on the NATS Streaming Server. This will return a [SubscriptionResponse](#subscriptionresponse) message to the reply subject specified in the NATS protocol request message. + +##### Message Structure + +- `clientID`: Client ID originally provided in the [ConnectRequest](#connectrequest) +- `subject`: Formal subject to subscribe to, e.g. foo.bar +- `qGroup`: Optional queue group +- `inbox`: Inbox subject to deliver messages on +- `maxInFlight`: Maximum inflight messages without an acknowledgement allowed +- `ackWaitInSecs`: Timeout for receiving an acknowledgement from the client +- `durableName`: Optional durable name which survives client restarts +- `startPosition`: An enumerated type specifying the point in history to start replaying data +- `startSequence`: Optional start sequence number +- `startTimeDelta`: Optional start time + +##### StartPosition enumeration + +- `NewOnly`: Send only new messages +- `LastReceived`: Send only the last received message +- `TimeDeltaStart`: Send messages from duration specified in the `startTimeDelta` field. +- `SequenceStart`: Send messages starting from the sequence in the `startSequence` field. +- `First`: Send all available messages + +[Back to table](#protocols) + + +#### SubscriptionResponse + +##### Description + +The `SubscriptionResponse` message is the response from the `SubscriptionRequest`. After a client has processed an incoming [MsgProto](#msgproto) message, it must send an acknowledgement to the `ackInbox` subject provided here. + +##### Message Structure + +- `ackInbox`: subject the client sends message acknowledgements to the NATS Streaming Server +- `error`: error string, empty/omitted if no error + +[Back to table](#protocols) + + +#### UnsubscribeRequest + +##### Description + +The `UnsubscribeRequest` closes or unsubcribes the subscription from the specified subject. The inbox specified is the `inbox` returned from the NATS Streaming Server in the `SubscriptionResponse`. Depending on which subject this request is sent, the action will result in close (if sent to subject `subCloseRequests`) or unsubscribe (if sent to subject `unsubRequests`) + +##### Message Structure + +- `clientID`: Client ID originally provided in the [ConnectRequest](#connectrequest) +- `subject`: Subject for the subscription +- `inbox`: Inbox subject to identify subscription +- `durableName`: Optional durable name which survives client restarts + +[Back to table](#protocols) + + +#### PubMsg + +##### Description + +The `PubMsg` protocol message is published from a client to the NATS Streaming Server. The GUID must be unique, and is returned in the [PubAck](#puback) message to correlate the success or failure of storing this particular message. + +##### Message Structure + +- `clientID`: Client ID originally provided in the [ConnectRequest](#connectrequest) +- `guid`: a guid generated for this particular message +- `subject`: subject +- `data`: payload +- `connID`: Connection ID. For servers that know about this field, clientID can be omitted + +[Back to table](#protocols) + + +#### PubAck + +##### Description + +The `PubAck` message is an acknowledgement from the NATS Streaming Server that a message has been processed. The message arrives on the subject specified on the reply subject of the NATS message the `PubMsg` was published on. The GUID is the same GUID used in the `PubMsg` being acknowledged. If an error string is present, the message was not persisted by the NATS Streaming Server and no guarantees regarding persistence are honored. `PubAck` messages may be handled asynchronously from their corresponding `PubMsg` in the client. + +##### Message Structure + +- `guid`: GUID of the message being acknowledged by the NATS Streaming Server +- `error`: An error string, empty/omitted if no error + +[Back to table](#protocols) + + +#### MsgProto + +##### Description + +The `MsgProto` message is received by client from the NATS Streaming Server, containing the payload of messages sent by a publisher. A `MsgProto` message that is not acknowledged with an [Ack](#ack) message within the duration specified by the `ackWaitInSecs` field of the subscription request will be redelivered. + +##### Message Structure + +- `sequence`: Globally ordered sequence number for the subject's channel +- `subject`: Subject +- `data`: Payload +- `timestamp`: Time the message was stored in the server. +- `redelivered`: Flag specifying if the message is being redelivered + +[Back to table](#protocols) + + +#### Ack + +##### Description + +An `Ack` message is an acknowledgement from the client that a [MsgProto](#msgproto) message has been considered received. It is published to the `ackInbox` field of the [SubscriptionResponse](#subscriptionresponse). + +##### Message Structure + +- `subject`: Subject of the message being acknowledged +- `sequence`: Sequence of the message being acknowledged + +[Back to table](#protocols) + + +#### Ping + +##### Description + +A `Ping` message is sent to the server at configured interval to check that the connection ID is still valid. This should be used only if client is at protocol 1, and has sent a `connID` in the [ConnectRequest](#connectrequest) protocol. + +##### Message Structure + +- `connID`: The connection ID + +[Back to table](#protocols) + + +#### PingResponse + +##### Description + +This is a response from the server to a `Ping` from the client. If the content is not empty, it will be the error indicating to the client why the connection is no longer valid. + +##### Message Structure + +- `error`: Error string, empty/omitted if no error + +[Back to table](#protocols) + + +#### CloseRequest + +##### Description + +A `CloseRequest` message is published on the `closeRequests` subject from the [ConnectResponse](#connectresponse), and notifies the NATS Streaming Server that the client connection is closing, allowing the server to free up resources. This message should **always** be sent when a client is finished using a connection. + +##### Message Structure + +- `clientID`: Client ID originally provided in the [ConnectRequest](#connectrequest) + +[Back to table](#protocols) + + +#### CloseResponse + +##### Description + +The `CloseResponse` is sent by the NATS Streaming Server on the reply subject of the `CloseRequest` NATS message. This response contains any error that may have occurred with the corresponding close call. + +##### Message Structure + +- `error`: error string, empty/omitted if no error + +[Back to table](#protocols) \ No newline at end of file diff --git a/developer/streaming/publishing.md b/developer/streaming/publishing.md index b9eaa28..bc140fa 100644 --- a/developer/streaming/publishing.md +++ b/developer/streaming/publishing.md @@ -1,6 +1,6 @@ # Publishing to a Channel -The streaming client library can provide a method for publishing synchronously. .These publish methods block until the ACK is returned by the server. An error or exception is used to indicate a timeout or other error. +The streaming client library can provide a method for publishing synchronously. These publish methods block until the ACK is returned by the server. An error or exception is used to indicate a timeout or other error. ```go err := sc.Publish("foo", []byte("Hello World")) @@ -12,4 +12,10 @@ Streaming libraries can also provide a way to publish asynchronously. An ACK cal ackHandler := func(ackedNuid string, err error){ ... } nuid, err := sc.PublishAsync("foo", []byte("Hello World"), ackHandler) -``` \ No newline at end of file +``` + +Even in this mode, the call will still block if the library has a number of published messages without having received an ACK from the server. The default can be changed when creating the connection. + +```go +sc, err := sc.Connect(clusterID, clientName, stan.MaxPubAcksInflight(1000)) +``` diff --git a/developer/streaming/queues.md b/developer/streaming/queues.md index 0cad323..bb86eb4 100644 --- a/developer/streaming/queues.md +++ b/developer/streaming/queues.md @@ -1,11 +1,37 @@ # Queue Subscriptions -Queue subscriptions are created like other subscriptions with the addition of a queue name. All subscriptions, across clients, share the queue based on this unique name. Other subscriptions can receive messages independently of the queue groups. Unsubscribe removes a client from a group, the last unsubscribe kills the group. Max in flight is per subscription. +Queue subscriptions are created like other subscriptions with the addition of a queue name. ```go -qsub1, _ := sc.QueueSubscribe(channelID, +qsub1, _ := sc.QueueSubscribe(channelName, queueName, func(m *stan.Msg) {...}) -qsub2, _ := sc.QueueSubscribe(channelID, +qsub2, _ := sc.QueueSubscribe(channelName, queueName, func(m *stan.Msg) {...}) -``` \ No newline at end of file +``` + +Multiple subscriptions using the same channel and queue name are members of the same queue group. That means that if a message is published on that channel, only one member of the group receives the message. Other subscriptions receive messages independently of the queue groups, that is, a message is delivered to all subscriptions and one member of each queue group. + +To create a durable queue subscription, simply add a durable name: +```go +qsub, err := sc.QueueSubscribe(channelName, + queueName, func(m *stan.Msg) {...}, + stan.DurableName("durable-name")) +``` + +Subscriptions options apply to each member independently, notably, the `AckWait` and `MaxInflight`. Those two members of the same queue group use different options for redelivery and max inflight. +```go +qsub1, _ := sc.QueueSubscribe(channelName, + queueName, func(m *stan.Msg) {...}, + stan.AckWait(5*time.Second), + stan.MaxInflight(5)) + +qsub2, _ := sc.QueueSubscribe(channelName, + queueName, func(m *stan.Msg) {...}, + stan.AckWait(20*time.Second), + stan.MaxInflight(10)) +``` + +If the queue subscription is durable, only the last member calling `Unsubscribe()` will cause the durable queue group to be removed from the server. + +Check the [concepts](/nats_streaming/channels/subscriptions/queue-group.md) section for more information. \ No newline at end of file diff --git a/developer/streaming/receiving.md b/developer/streaming/receiving.md index 0811e06..7841e6c 100644 --- a/developer/streaming/receiving.md +++ b/developer/streaming/receiving.md @@ -9,6 +9,10 @@ Subscriptions come in several forms: * Queue * Queue/Durable +For more details on the various types, check the [concept](/nats_streaming/channels/subscriptions/subscriptions.md) section. + +***Note: message callbacks are invoked serially, one message at a time. If your application does not care about processing ordering and would prefer the messages to be dispatched concurrently, it is the application responsibility to move them to some internal queue to be picked up by threads/go routines.*** + Subscriptions set their starting position on creation using position or time. For example, in Go you can start at: * The last message received @@ -43,4 +47,26 @@ var startTime time.Time sub, err := sc.Subscribe("foo", func(m *stan.Msg) {...}, stan.StartAtTime(startTime)) -``` \ No newline at end of file +``` + +To set the delay after which the server should attempt to redeliver a message for which it has not receive an acknowledgment: + +```go +sub, err := sc.Subscribe("foo", + func(m *stan.Msg) {...}, + stan.AckWait(20*time.Second)) +``` + +When an application wishes to stop receiving, but want to maintain the connection opened, the subscription should be closed. There are two ways to stop a subscription, either "close" it, or "unsubscribe" it. For non durable subscriptions, this is equivalent since the subscription will be completely removed. For durable subscriptions, close means that the server will stop delivering, but remember the durable subscription. Unsubscribe, however, means that the server will remove the state of this subscription. + +To simply close: +```go +err := sub.Close() +``` + +To unsubscribe: +```go +err := sub.Unsubscribe() +``` + +_Note: If a connection is closed without explicitly closing the subscriptions, the subscriptions are implicitly closed, not unsubscribed._ diff --git a/nats_streaming/channels/channels.md b/nats_streaming/channels/channels.md new file mode 100644 index 0000000..f38ef3f --- /dev/null +++ b/nats_streaming/channels/channels.md @@ -0,0 +1,7 @@ +# Channels + +Channels are at the heart of the NATS Streaming Server. Channels are subjects clients send data to and consume from. + +***Note: NATS Streaming server does not support wildcard for channels, that is, one cannot subscribe on `foo.*`, or `>`, etc...*** + +The number of channels can be limited (and is by default) through configuration. Messages produced to a channel are stored in a message log inside this channel. \ No newline at end of file diff --git a/nats_streaming/channels/message-log.md b/nats_streaming/channels/message-log.md new file mode 100644 index 0000000..058f632 --- /dev/null +++ b/nats_streaming/channels/message-log.md @@ -0,0 +1,5 @@ +# Message Log + +You can view a message log as a First In First Out (FIFO) queue. Messages are appended to the end of the log. If a limit is set globally for all channels, or specifically for this channel, when the limit is reached, older messages are removed to make room for the new ones. + +But except for the administrative size/age limit set for a message log, messages are not removed due to consumers consuming them. In fact, messages are stored regardless of the presence of subscriptions on that channel. diff --git a/nats_streaming/channels/subscriptions/durable.md b/nats_streaming/channels/subscriptions/durable.md new file mode 100644 index 0000000..16c0864 --- /dev/null +++ b/nats_streaming/channels/subscriptions/durable.md @@ -0,0 +1,9 @@ +# Durable + +If an application wishes to resume message consumption from where it previously stopped, it needs to create a durable subscription. It does so by providing a durable name, which is combined with the client ID provided when the client created its connection. The server then maintain the state for this subscription even after the client connection is closed. + +***Note: The starting position given by the client when restarting a durable subscription is ignored.*** + +When the application wants to stop receiving messages on a durable subscription, it should close - but *not unsubscribe*- this subscription. If a given client library does not have the option to close a subscription, the application should close the connection instead. + +When the application wants to delete the subscription, it must unsubscribe it. Once unsubscribed, the state is removed and it is then possible to re-use the durable name, but it will be considered a brand new durable subscription, with the start position being the one given by the client when creating the durable subscription. \ No newline at end of file diff --git a/nats_streaming/channels/subscriptions/queue-group.md b/nats_streaming/channels/subscriptions/queue-group.md new file mode 100644 index 0000000..527b8ae --- /dev/null +++ b/nats_streaming/channels/subscriptions/queue-group.md @@ -0,0 +1,11 @@ +# Queue Group + +When consumers want to consume from the same channel but each receive a different message, as opposed to all receiving the same messages, they need to create a queue subscription. When a queue group name is specified, the server will send each messages from the log to a single consumer in the group. The distribution of these messages is not specified, therefore applications should not rely on an expected delivery scheme. + +After the first queue member is created, any other member joining the group will receive messages based on where the server is in the message log for that particular group. That means that starting position given by joining members is ignored by the server. + +When the last member of the group leaves (subscription unsubscribed/closed/or connection closed), the group is removed from the server. The next application creating a subscription with the same name will create a new group, starting at the start position given in the subscription request. + +A queue subscription can also be durable. For that, the client needs to provide a queue and durable name. The behavior is, as you would expect, a combination of queue and durable subscription. Unlike a durable subscription, though, the client ID is not part of the queue group name. It makes sense, because since client ID must be unique, it would prevent more than one connection to participate in the queue group. The main difference between a queue subscription and a durable one, is that when the last member leaves the group, the state of the group will be maintained by the server. Later, when a member rejoins the group, the delivery will resume. + +***Note: For a durable queue subscription, the last member to * unsubscribe * (not simply close) causes the group to be removed from the server.*** diff --git a/nats_streaming/channels/subscriptions/redelivery.md b/nats_streaming/channels/subscriptions/redelivery.md new file mode 100644 index 0000000..ac76c8b --- /dev/null +++ b/nats_streaming/channels/subscriptions/redelivery.md @@ -0,0 +1,11 @@ +# Redelivery + +When the server sends a message to a consumer, it expects to receive an ACK from this consumer. The consumer is the one specifying how long the server should wait before resending all unacknowledged messages to the consumer. + +When the server restarts and recovers unacknowledged messages for a subscription, it will first attempt to redelivery those messages before sending new messages. However, if during the initial redelivery some messages don't make it to the client, the server cannot know that and will enable delivery of new messages. + +***So it is possible for an application to receive redelivered messages mixed with new messages. This is typically what happens outside of the server restart scenario.*** + +For queue subscriptions, if a member has unacknowledged messages, when this member `AckWait` (which is the duration given to the server before the server should attempt to redeliver unacknowledged messages) time elapses, the messages are redelivered to any other member in the group (including itself). + +If a queue member leaves the group, its unacknowledged messages are redistributed to other queue members. \ No newline at end of file diff --git a/nats_streaming/channels/subscriptions/regular.md b/nats_streaming/channels/subscriptions/regular.md new file mode 100644 index 0000000..a3be9c6 --- /dev/null +++ b/nats_streaming/channels/subscriptions/regular.md @@ -0,0 +1,3 @@ +# Regular + +The state of these subscriptions is removed when they are unsubscribed or closed (which is equivalent for this type of subscription) or the client connection is closed (explicitly by the client, or closed by the server due to timeout). They do, however, survive a *server* failure (if running with a persistent store). diff --git a/nats_streaming/channels/subscriptions/subscriptions.md b/nats_streaming/channels/subscriptions/subscriptions.md new file mode 100644 index 0000000..3fea537 --- /dev/null +++ b/nats_streaming/channels/subscriptions/subscriptions.md @@ -0,0 +1,13 @@ +# Subscriptions + +A client creates a subscription on a given channel. Remember, there is no support for wildcards, so a subscription is really tied to +one and only one channel. The server will maintain the subscription state on behalf of the client until the later closes the subscription (or its connection). + +If there are messages in the log for this channel, messages will be sent to the consumer when the subscription is created. The server will +send up to the maximum number of inflight messages as given by the client when creating the subscription. + +When receiving ACKs from the consumer, the server will then deliver more messages, if more are available. + +A subscription can be created to start at any point in the message log, either by message sequence, or by time. + +Following pages describe all types of subscription. \ No newline at end of file diff --git a/nats_streaming/client-connections.md b/nats_streaming/client-connections.md new file mode 100644 index 0000000..fd04ede --- /dev/null +++ b/nats_streaming/client-connections.md @@ -0,0 +1,9 @@ +# Client Connections + +As described, clients are not directly connected to the streaming server. Instead, they send connection requests. The request includes a `client ID` which is used by the server to uniquely identify, and restrict, a given client. That is, no two connections with the same client ID will be able to run concurrently. + +This client ID links a given connection to its published messages, subscriptions, especially durable subscriptions. Indeed, durable subscriptions are stored as a combination of the client ID and durable name. More on durable subscriptions later. + +It is also used to resolve the issue of not having direct client connections to the server. For instance, say that a client crashes without closing the connection. It later restarts with the same client ID. The server will detect that this client ID is already in-use. It will try to contact that known client to its original private inbox. If the server does not receive a response - which would be the case if the client crashed - it will replace the old client with this new one.
+ +Otherwise, the server would reject the connection request since the client ID is already in-use. \ No newline at end of file diff --git a/nats_streaming/clustering/auto-configuration.md b/nats_streaming/clustering/auto-configuration.md new file mode 100644 index 0000000..2a19c63 --- /dev/null +++ b/nats_streaming/clustering/auto-configuration.md @@ -0,0 +1,17 @@ +# Auto Configuration + +We can also bootstrap a NATS Streaming cluster by starting one server as the seed node using the `-cluster_bootstrap` flag. This node will elect itself leader, so it's important to avoid starting multiple servers as seed. Once a seed node is started, other servers will automatically join the cluster. If the server is recovering, it will use the recovered cluster configuration. + +Here is an example of starting three servers in a cluster by starting one as the seed and letting the others automatically join: + +``` +nats-streaming-server -store file -dir store-a -clustered -cluster_bootstrap -nats_server nats://localhost:4222 + +nats-streaming-server -store file -dir store-b -clustered -nats_server nats://localhost:4222 + +nats-streaming-server -store file -dir store-c -clustered -nats_server nats://localhost:4222 +``` + +For a given cluster ID, if more than one server is started with `cluster_bootstrap` set to true, each server with this parameter will report the misconfiguration and exit. + +The very first server that bootstrapped the cluster can be restarted, however, the operator must remove the datastores of the other servers that were incorrectly started with the bootstrap parameter before attempting to restart them. If they are restarted -even without the `-cluster_bootstrap` parameter- but with existing state, they will once again start as a leader. diff --git a/nats_streaming/clustering/clustering.md b/nats_streaming/clustering/clustering.md new file mode 100644 index 0000000..4125f77 --- /dev/null +++ b/nats_streaming/clustering/clustering.md @@ -0,0 +1,9 @@ +# Clustering + +NATS Streaming Server supports clustering and data replication, implemented with the [Raft consensus algorithm](https://raft.github.io/), for the purposes of high availability. + +There are two ways to bootstrap a cluster: with an explicit cluster configuration or with "auto" configuration using a seed node. With the first, we provide the IDs of the nodes participating in the cluster. In this case, the participating nodes will elect a leader. With the second, we start one server as a seed node, which will elect itself as leader, and subsequent servers will automatically join the seed (note that this also works with the explicit cluster configuration once the leader has been established). With the second method, we need to be careful to avoid starting multiple servers as seed as this will result in a split-brain. Both of these configuration methods are shown in the sections below. + +It is recommended to run an odd number of servers in a cluster with a minimum of three servers to avoid split-brain scenarios. Note that if less than a majority of servers are available, the cluster cannot make progress, e.g. if two nodes go down in a cluster of three, the cluster is unavailable until at least one node comes back. + +Note about Channels Partitioning and Clustering. These two features are mutually exclusive. Trying to start a server with channels Partitioning and Clustering enabled will result in a startup error. Clustering requires all channels to be replicated in the cluster. \ No newline at end of file diff --git a/nats_streaming/clustering/configuration.md b/nats_streaming/clustering/configuration.md new file mode 100644 index 0000000..6160294 --- /dev/null +++ b/nats_streaming/clustering/configuration.md @@ -0,0 +1,95 @@ +# Configuration + +We can bootstrap a NATS Streaming cluster by providing the cluster topology using the `-cluster_peers` flag. This is simply the set of node IDs participating in the cluster. Note that once a leader is established, we can start subsequent servers without providing this configuration as they will automatically join the leader. If the server is recovering, it will use the recovered cluster configuration. + +Here is an example of starting three servers in a cluster. For this example, we run a separate NATS server which the Streaming servers connect to. + +``` +nats-streaming-server -store file -dir store-a -clustered -cluster_node_id a -cluster_peers b,c -nats_server nats://localhost:4222 + +nats-streaming-server -store file -dir store-b -clustered -cluster_node_id b -cluster_peers a,c -nats_server nats://localhost:4222 + +nats-streaming-server -store file -dir store-c -clustered -cluster_node_id c -cluster_peers a,b -nats_server nats://localhost:4222 +``` + +Note that once a leader is elected, subsequent servers can be started without providing the cluster configuration. They will automatically join the cluster. Similarly, the cluster node ID does not need to be provided as one will be automatically assigned. As long as the file store is used, this ID will be recovered on restart. + +``` +nats-streaming-server -store file -dir store-d -clustered -nats_server nats://localhost:4222 +``` + +The equivalent clustering configurations can be specified in a configuration file under the `cluster` group. See the [Configuring](#configuring) section for more information. + +Here is an example of a cluster of 3 nodes using the following configuration files. The nodes are running on `host1`, `host2` and `host3` respectively. + +NOTE If you have an existing NATS cluster and want to run NATS Streaming Cluster on top of that, see details at the end of this section. + +On `host1`, this configuration indicates that the server will accept client connections on port 4222. It will accept route connections on port 6222. It creates 2 routes, to `host2` and `host3` cluster port. + +It defines the NATS Streaming cluster name as `mycluster`, uses a store file that points to the `store` directory. The `cluster` section inside `streaming` makes the NATS Streaming server run in cluster mode. This configuration explicitly define each node id (`a` for `host1`) and list its peers. +``` +# NATS specific configuration +port: 4222 +cluster { + listen: 0.0.0.0:6222 + routes: ["nats://host2:6222", "nats://host3:6222"] +} + +# NATS Streaming specific configuration +streaming { + id: mycluster + store: file + dir: store + cluster { + node_id: "a" + peers: ["b", "c"] + } +} +``` + +Below is the configuration for the server running on `host2`. Notice how the routes are now to `host1` and `host3`. The other thing that changed is the node id that is set to `b` and peers are updated accordingly to `a` and `c`. + +Note that the `dir` configuration is also `store` but these are local directories and do not (actually must not) be shared. Each node will have its own copy of the datastore. You could have each configuration have a different value for `dir` if desired. +``` +# NATS specific configuration +port: 4222 +cluster { + listen: 0.0.0.0:6222 + routes: ["nats://host1:6222", "nats://host3:6222"] +} + +# NATS Streaming specific configuration +streaming { + id: mycluster + store: file + dir: store + cluster { + node_id: "b" + peers: ["a", "c"] + } +} +``` + +As you would expect, for `host3`, the routes are now to `host1` and `host2` and the node id is `c` while its peers +are `a` and `b`. +``` +# NATS specific configuration +port: 4222 +cluster { + listen: 0.0.0.0:6222 + routes: ["nats://host1:6222", "nats://host2:6222"] +} + +# NATS Streaming specific configuration +streaming { + id: mycluster + store: file + dir: store + cluster { + node_id: "c" + peers: ["a", "b"] + } +} +``` + +In the example above, the configuration assumes no existing NATS Cluster and therefore configure the NATS routes between each node. Should you want to use an existing NATS cluster, do not include the "NATS specific configuration" section, instead, add `nats_server_url` in the `streaming` section to point to the NATS server you want. \ No newline at end of file diff --git a/nats_streaming/clustering/containers.md b/nats_streaming/clustering/containers.md new file mode 100644 index 0000000..71cf971 --- /dev/null +++ b/nats_streaming/clustering/containers.md @@ -0,0 +1,9 @@ +# Containers + +When running the docker image of NATS Streaming Server, you will want to specify a mounted volume so that the data can be recovered. Your `-dir` parameter then points to a directory inside that mounted volume. However, after a restart you may get a failure with a message similar to this: +``` +[FTL] STREAM: Failed to start: streaming state was recovered but cluster log path "mycluster/a" is empty +``` +This is because the server recovered the streaming state (as pointed by `-dir` and located in the mounted volume), but did not recover the RAFT specific state that is by default stored in a directory named after your cluster id, relative to the current directory starting the executable. In the context of a container, this data will be lost after the container is stopped. + +In order to avoid this issue, you need to specify the `-cluster_log_path` and ensure that it points to the mounted volume so that the RAFT state can be recovered along with the Streaming state. diff --git a/nats_streaming/clustering/supported-stores.md b/nats_streaming/clustering/supported-stores.md new file mode 100644 index 0000000..0fb3a3e --- /dev/null +++ b/nats_streaming/clustering/supported-stores.md @@ -0,0 +1,9 @@ +# Supported Stores + +In order to run NATS Streaming Server in clustered mode, you need to specify a persistent store. At this time you have the choice between `FILE` and `SQL` + +The NATS Streaming stores server meta information, messages and subscriptions to the storage you configure using the `--store` option. + +However, in clustered mode, we use RAFT for leader election. The raft layer uses its own stores which are currently necessarily file based. The location of the RAFT stores defaults to the current directory under a sub-directory named after the cluster ID, or you can configure it using `--cluster_log_path`. + +This means that even if you select an SQL Store, there will still be a need for storing data on the file system. diff --git a/nats_streaming/fault-tolerance/active-server.md b/nats_streaming/fault-tolerance/active-server.md new file mode 100644 index 0000000..6ae7b8d --- /dev/null +++ b/nats_streaming/fault-tolerance/active-server.md @@ -0,0 +1,7 @@ +# Active Server + +There is a single Active server in the group. This server was the first to obtain the exclusive lock for storage. For the `FileStore` implementation, it means trying to get an advisory lock for a file located in the shared datastore. For the `SQLStore` implementation, a special table is used in which the owner of the lock updates a column. Other instances will steal the lock if the column is not updated for a certain amount of time. + +If the elected server fails to grab this lock because it is already locked, it will go back to standby. + +***Only the active server accesses the store and service all clients.*** diff --git a/nats_streaming/fault-tolerance/failover.md b/nats_streaming/fault-tolerance/failover.md new file mode 100644 index 0000000..c1dce65 --- /dev/null +++ b/nats_streaming/fault-tolerance/failover.md @@ -0,0 +1,9 @@ +# Failover + +When the active server fails, all standby servers will try to activate. The process consists of trying to get an exclusive lock on the storage. + +The first server that succeeds will become active and go through the process of recovering the store and service clients. It is as if a server in standalone mode was automatically restarted. + +All other servers that failed to get the store lock will go back to standby mode and stay in this mode until they stop receiving heartbeats from the current active server. + +It is possible that a standby trying to activate is not able to immediately acquire the store lock. When that happens, it goes back into standby mode, but if it fails to receive heartbeats from an active server, it will try again to acquire the store lock. The interval is random but as of now set to a bit more than a second. diff --git a/nats_streaming/fault-tolerance/ft.md b/nats_streaming/fault-tolerance/ft.md new file mode 100644 index 0000000..b3833ce --- /dev/null +++ b/nats_streaming/fault-tolerance/ft.md @@ -0,0 +1,15 @@ +# Fault Tolerance + +To minimize the single point of failure, NATS Streaming server can be run in Fault Tolerance mode. It works by having a group of servers with one acting as the active server (accessing the store) and handling all communication with clients, and all others acting as standby servers. + +It is important to note that is not possible to run Nats Stream as Fault Tolerance mode and Clustering mode at the same time. + +To start a server in Fault Tolerance (FT) mode, you specify an FT group name. + +Here is an example on how starting 2 servers in FT mode running on the same host and embedding the NATS servers: + +``` +nats-streaming-server -store file -dir datastore -ft_group "ft" -cluster nats://localhost:6222 -routes nats://localhost:6223 -p 4222 + +nats-streaming-server -store file -dir datastore -ft_group "ft" -cluster nats://localhost:6223 -routes nats://localhost:6222 -p 4223 +``` \ No newline at end of file diff --git a/nats_streaming/fault-tolerance/shared-state.md b/nats_streaming/fault-tolerance/shared-state.md new file mode 100644 index 0000000..3226b49 --- /dev/null +++ b/nats_streaming/fault-tolerance/shared-state.md @@ -0,0 +1,3 @@ +# Shared State + +Actual file replication to multiple disks is not handled by the Streaming server. This - if required - needs to be handled by the user. For the FileStore implementation that we currently provide, the data store needs to be mounted by all servers in the FT group (e.g. an NFS Mount (Gluster in Google Cloud or EFS in Amazon). diff --git a/nats_streaming/fault-tolerance/standby-server.md b/nats_streaming/fault-tolerance/standby-server.md new file mode 100644 index 0000000..da8fff6 --- /dev/null +++ b/nats_streaming/fault-tolerance/standby-server.md @@ -0,0 +1,3 @@ +# Standby servers + +There can be as many as you want standby servers on the same group. These servers do not access the store and do not receive any data from the streaming clients. They are just running waiting for the detection of the active server failure. diff --git a/nats_streaming/gettingstarted/configuring.md b/nats_streaming/gettingstarted/configuring.md new file mode 100644 index 0000000..9e3f666 --- /dev/null +++ b/nats_streaming/gettingstarted/configuring.md @@ -0,0 +1,3 @@ +## Configuration and Administration + +NATS Streaming provides a rich set of commands and parameters to configure all aspects of the server. Please refer to the [README](https://github.com/nats-io/nats-streaming-server#configuring) for further information on usage, configuration, and administration. \ No newline at end of file diff --git a/nats_streaming/gettingstarted/install.md b/nats_streaming/gettingstarted/install.md new file mode 100644 index 0000000..390b20b --- /dev/null +++ b/nats_streaming/gettingstarted/install.md @@ -0,0 +1,81 @@ +# NATS Streaming Server Installation + +NATS philosophy is simplicity. Installation is just decompressing a zip file and copying the binary to an appropriate directory; you can also use your favorite package manager. + +## Installing via a Package Manager + +On Mac OS: +``` +> brew install nats-streaming-server +``` + +Via Docker: +``` +> docker pull nats-streaming +``` + +## Installing a release build + +You can find the latest release of `nats-streaming-server` [here](https://github.com/nats-io/nats-streaming-server/releases/latest). + +Download the zip file matching your systems architecture, and unzip. For this example, assuming version 0.14.2 of the server, and a Linux AMD64: +``` +> curl -L https://github.com/nats-io/nats-streaming-server/releases/download/v0.14.2/nats-streaming-server-v0.14.2-linux-amd64.zip -o nats-streaming-server.zip + +> unzip nats-streaming-server.zip -d tmp +Archive: nats-streaming-server.zip + creating: tmp/nats-streaming-server-v0.14.2-linux-amd64/ + inflating: tmp/nats-streaming-server-v0.14.2-linux-amd64/README.md + inflating: tmp/nats-streaming-server-v0.14.2-linux-amd64/LICENSE + inflating: tmp/nats-streaming-server-v0.14.2-linux-amd64/nats-streaming-server + +> cp tmp/nats-streaming-server-v0.14.2-linux-amd64/nats-streaming-server /usr/local/bin +``` + +## Installing from the source + +If you have go installed, installing the binary is easy: + +``` +> go get github.com/nats-io/nats-streaming-server +``` + +This mechanism will install a build of master, which almost certainly will not be a released version. If you are a developer and want to play with the latest, this is the easiest way of obtaining it. + +You can run the test suite from the `nats-streaming-server` root directory: +``` +go test -v -p=1 ./... +``` + +Some of the store tests require a SQL server to be running. To skip those, use this command instead: +``` +go test -v -p=1 ./... -sql=false +``` + +## Testing Your Installation + +To test your installation (provided the executable is visible to your shell): + +``` +> nats-streaming-server +[58061] 2019/05/22 13:56:45.463562 [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.14.2 +[58061] 2019/05/22 13:56:45.463639 [INF] STREAM: ServerID: Avb51sMf9imRPVVwv6Ts0v +[58061] 2019/05/22 13:56:45.463657 [INF] STREAM: Go version: go1.11.10 +[58061] 2019/05/22 13:56:45.463659 [INF] STREAM: Git commit: [not set] +[58061] 2019/05/22 13:56:45.464086 [INF] Starting nats-server version 1.4.1 +[58061] 2019/05/22 13:56:45.464092 [INF] Git commit [not set] +[58061] 2019/05/22 13:56:45.464310 [INF] Listening for client connections on 0.0.0.0:4222 +[58061] 2019/05/22 13:56:45.464328 [INF] Server is ready +[58061] 2019/05/22 13:56:45.495045 [INF] STREAM: Recovering the state... +[58061] 2019/05/22 13:56:45.495055 [INF] STREAM: No recovered state +[58061] 2019/05/22 13:56:45.749604 [INF] STREAM: Message store is MEMORY +[58061] 2019/05/22 13:56:45.749658 [INF] STREAM: ---------- Store Limits ---------- +[58061] 2019/05/22 13:56:45.749664 [INF] STREAM: Channels: 100 * +[58061] 2019/05/22 13:56:45.749668 [INF] STREAM: --------- Channels Limits -------- +[58061] 2019/05/22 13:56:45.749671 [INF] STREAM: Subscriptions: 1000 * +[58061] 2019/05/22 13:56:45.749675 [INF] STREAM: Messages : 1000000 * +[58061] 2019/05/22 13:56:45.749678 [INF] STREAM: Bytes : 976.56 MB * +[58061] 2019/05/22 13:56:45.749682 [INF] STREAM: Age : unlimited * +[58061] 2019/05/22 13:56:45.749686 [INF] STREAM: Inactivity : unlimited * +[58061] 2019/05/22 13:56:45.749690 [INF] STREAM: ---------------------------------- +``` diff --git a/nats_streaming/gettingstarted/intro.md b/nats_streaming/gettingstarted/intro.md new file mode 100644 index 0000000..6be1634 --- /dev/null +++ b/nats_streaming/gettingstarted/intro.md @@ -0,0 +1,6 @@ +# Getting Started + +The best way to get the NATS Streaming Server is to use one of the pre-built release binaries which are available for OSX, Linux (x86-64/ARM), Windows. + +Of course you can build the latest version of the server from the master branch. The master branch will always build and pass tests, but may not work correctly in your environment. You will first need Go installed on your machine (version 1.11+ is required) to build the NATS Streaming server. + diff --git a/nats_streaming/gettingstarted/process-signaling.md b/nats_streaming/gettingstarted/process-signaling.md new file mode 100644 index 0000000..a3e88b8 --- /dev/null +++ b/nats_streaming/gettingstarted/process-signaling.md @@ -0,0 +1,27 @@ +# Process Signaling + +On Unix systems, the NATS Streaming Server responds to the following signals: + +| Signal | Result | +| --------------- | ------------------------------------- | +| SIGKILL | Kills the process immediately | +| SIGINT, SIGTERM | Stops the server gracefully | +| SIGUSR1 | Reopens the log file for log rotation | + +The `nats-streaming-server` binary can be used to send these signals to running NATS Streaming Servers using the `-sl` flag: + +```sh +# Reopen log file for log rotation +nats-streaming-server -sl reopen + +# Stop the server +nats-streaming-server -sl quit +``` + +If there are multiple `nats-streaming-server` processes running, specify a PID: + +```sh +nats-streaming-server -sl quit= +``` + +See the [Windows Service](#windows-service) section for information on signaling the NATS Streaming Server on Windows. diff --git a/nats_streaming/gettingstarted/run.md b/nats_streaming/gettingstarted/run.md new file mode 100644 index 0000000..c24918b --- /dev/null +++ b/nats_streaming/gettingstarted/run.md @@ -0,0 +1,96 @@ +# Getting Started with NATS Streaming + +This tutorial demonstrates NATS Streaming using example [Go NATS Streaming clients](https://github.com/nats-io/stan.go.git). + +## Prerequisites + +- [Set up your Git environment](https://help.github.com/articles/set-up-git/). +- [Set up your Go environment](https://golang.org/doc/install). + +## Setup + +Download and install the [NATS Streaming Server](https://github.com/nats-io/nats-streaming-server/releases). + +Clone the following repositories: + +- NATS Streaming Server: `git clone https://github.com/nats-io/nats-streaming-server.git` +- NATS Streaming Client: `git clone https://github.com/nats-io/stan.go.git` + +## Start the NATS Streaming Server + +Two options: + +Run the binary that you downloaded, for example: `$ ./nats-streaming-server` + +Or, run from source: + +```sh +> cd $GOPATH/src/github.com/nats-io/nats-streaming-server +> go run nats-streaming-server.go +``` + +You should see the following, indicating that the NATS Streaming Server is running: + +```sh +> go run nats-streaming-server.go +[59232] 2019/05/22 14:24:54.426344 [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.14.2 +[59232] 2019/05/22 14:24:54.426423 [INF] STREAM: ServerID: 3fpvAuXHo3C66Rkd4rmfFX +[59232] 2019/05/22 14:24:54.426440 [INF] STREAM: Go version: go1.11.10 +[59232] 2019/05/22 14:24:54.426442 [INF] STREAM: Git commit: [not set] +[59232] 2019/05/22 14:24:54.426932 [INF] Starting nats-server version 1.4.1 +[59232] 2019/05/22 14:24:54.426937 [INF] Git commit [not set] +[59232] 2019/05/22 14:24:54.427104 [INF] Listening for client connections on 0.0.0.0:4222 +[59232] 2019/05/22 14:24:54.427108 [INF] Server is ready +[59232] 2019/05/22 14:24:54.457604 [INF] STREAM: Recovering the state... +[59232] 2019/05/22 14:24:54.457614 [INF] STREAM: No recovered state +[59232] 2019/05/22 14:24:54.711407 [INF] STREAM: Message store is MEMORY +[59232] 2019/05/22 14:24:54.711465 [INF] STREAM: ---------- Store Limits ---------- +[59232] 2019/05/22 14:24:54.711471 [INF] STREAM: Channels: 100 * +[59232] 2019/05/22 14:24:54.711474 [INF] STREAM: --------- Channels Limits -------- +[59232] 2019/05/22 14:24:54.711478 [INF] STREAM: Subscriptions: 1000 * +[59232] 2019/05/22 14:24:54.711481 [INF] STREAM: Messages : 1000000 * +[59232] 2019/05/22 14:24:54.711485 [INF] STREAM: Bytes : 976.56 MB * +[59232] 2019/05/22 14:24:54.711488 [INF] STREAM: Age : unlimited * +[59232] 2019/05/22 14:24:54.711492 [INF] STREAM: Inactivity : unlimited * +[59232] 2019/05/22 14:24:54.711495 [INF] STREAM: ---------------------------------- +``` + +## Run the publisher client + +Publish several messages. For each publication you should get a result. + +```sh +> cd $GOPATH/src/github.com/nats-io/stan.go/examples/stan-pub +> go run main.go foo "msg one" +Published [foo] : 'msg one' +> go run main.go foo "msg two" +Published [foo] : 'msg two' +> go run main.go foo "msg three" +Published [foo] : 'msg three' +``` + +## Run the subscriber client + +Use the `--all` flag to receive all published messages. + +```sh +> cd $GOPATH/src/github.com/nats-io/stan.go/examples/stan-sub +> go run main.go --all -c test-cluster -id myID foo +Connected to nats://localhost:4222 clusterID: [test-cluster] clientID: [myID] +subscribing with DeliverAllAvailable +Listening on [foo], clientID=[myID], qgroup=[] durable=[] +[#1] Received on [foo]: 'sequence:1 subject:"foo" data:"msg one" timestamp:1465962202884478817 ' +[#2] Received on [foo]: 'sequence:2 subject:"foo" data:"msg two" timestamp:1465962208545003897 ' +[#3] Received on [foo]: 'sequence:3 subject:"foo" data:"msg three" timestamp:1465962215567601196 +``` + +## Explore other subscription options + +```sh + --seq Start at seqno + --all Deliver all available messages + --last Deliver starting with last published message + --since Deliver messages in last interval (e.g. 1s, 1hr, https://golang.org/pkg/time/#ParseDuration) + --durable Durable subscriber name + --unsubscribe Unsubscribe the durable on exit +``` diff --git a/nats_streaming/nats-streaming-tls.md b/nats_streaming/gettingstarted/tls.md similarity index 100% rename from nats_streaming/nats-streaming-tls.md rename to nats_streaming/gettingstarted/tls.md diff --git a/nats_streaming/gettingstarted/windows-service.md b/nats_streaming/gettingstarted/windows-service.md new file mode 100644 index 0000000..6a52b78 --- /dev/null +++ b/nats_streaming/gettingstarted/windows-service.md @@ -0,0 +1,28 @@ +# Windows Service + +The NATS Streaming Server supports running as a Windows service. There is currently no installer and instead users should use `sc.exe` to install the service: + +Here is how to create and start a NATS Streaming Server named `nats-streaming-server`. Note that the server flags should be passed in when creating the service. +```sh +sc.exe create nats-streaming-server binPath="\"\nats-streaming-server.exe\" [NATS Streaming flags]" +sc.exe start nats-streaming-server +``` + +You can create several instances, giving it a unique name. For instance, this is how you would create two services, named `nss1` and `nss2`, each one with its own set of parameters. +``` +sc.exe create nss1 binPath="\"c:\nats-io\nats-streaming\nats-streaming-server.exe\" --syslog --syslog_name=nss1 -p 4222" + +sc.exe create nss2 binPath="\"c:\nats-io\nats-streaming\nats-streaming-server.exe\" --syslog --syslog_name=nss2 -p 4223" +``` +By default, when no logfile is specified, the server will use the system log. The default event source name is `NATS-Streaming-Server`. However, you can specify the name you want, which is especially useful when installing more than one service as described above. + +Once the service is running, it can be controlled using `sc.exe` or `nats-streaming-server.exe -sl`: + +```batch +REM Stop the server +nats-streaming-server.exe -sl quit +``` +The above commands will default to controlling the service named `nats-streaming-server`. If the service has another name, it can be specified like this: +```batch +nats-streaming-server.exe -sl quit= +``` diff --git a/nats_streaming/nats-streaming-intro.md b/nats_streaming/intro.md similarity index 81% rename from nats_streaming/nats-streaming-intro.md rename to nats_streaming/intro.md index 1ce104c..9d1ea42 100644 --- a/nats_streaming/nats-streaming-intro.md +++ b/nats_streaming/intro.md @@ -32,15 +32,8 @@ digraph nats_streaming { In addition to the features of the core NATS platform, NATS Streaming provides the following: -- **Enhanced message protocol** - NATS Streaming implements its own enhanced message format using [Google Protocol Buffers] (https://developers.google.com/protocol-buffers/). These messages are transmitted as binary message payloads via core NATS platform, and thus require no changes to the basic NATS protocol. NATS Streaming messages contain the following fields: - - Sequence - a globally ordered sequence number for the subject's channel - - Subject - The NATS Streaming delivery subject - - Reply - The optional "reply-to" subject - - Data - The message payload - - Timestamp - the received timestamp, in nanoseconds. - - Redelivered - A flag signifying whether this message has been redelivered by the server - - CRC32 - An optional IEEE CRC32 -- **Message/event persistence** - NATS Streaming offers configurable message persistence either in-memory or via flat files. The storage subsystem uses a public interface that allows contributors to develop their own custom implementations. +- **Enhanced message protocol** - NATS Streaming implements its own enhanced message format using [Google Protocol Buffers](https://developers.google.com/protocol-buffers/). These messages are transmitted as binary message payloads via core NATS platform, and thus require no changes to the basic NATS protocol. +- **Message/event persistence** - NATS Streaming offers configurable message persistence: in-memory, flat files or database. The storage subsystem uses a public interface that allows contributors to develop their own custom implementations. - **At-least-once-delivery** - NATS Streaming offers message acknowledgements between publisher and server (for publish operations) and between subscriber and server (to confirm message delivery). Messages are persisted by the server in memory or secondary storage (or other external storage) and will be redelivered to eligible subscribing clients as needed. - **Publisher rate limiting** - NATS Streaming provides a connection option called `MaxPubAcksInFlight` that effectively limits the number of unacknowledged messages that a publisher may have in-flight at any given time. When this maximum is reached, further async publish calls will block until the number of unacknowledged messages falls below the specified limit. - **Rate matching/limiting per subscriber** - Subscriptions may specify a `MaxInFlight` option that designates the maximum number of outstanding acknowledgements (messages that have been delivered but not acknowledged) that NATS Streaming will allow for a given subscription. When this limit is reached, NATS Streaming will suspend delivery of messages to this subscription until the number of unacknowledged messages falls below the specified limit. @@ -52,10 +45,11 @@ In addition to the features of the core NATS platform, NATS Streaming provides t - A specific message sequence number - **Durable subscriptions** - Subscriptions may also specify a "durable name" which will survive client restarts. Durable subscriptions cause the server to track the last acknowledged message sequence number for a client and durable name. When the client restarts/resubscribes, and uses the same client ID and durable name, the server will resume delivery beginning with the earliest unacknowledged message for this durable subscription. + ## Installation -NATS provides a [server binary](nats-streaming-install.md) for Linux, Mac, and Windows. You can install the server from source on any platform you choose. +NATS provides a [server binary](gettingstarted/install.md) for Linux, Mac, and Windows. You can install the server from source on any platform you choose. ## Usage, Configuration and Administration -NATS Streaming provides a rich set of commands and parameters to configure all aspects of the server. Please refer to the [README](https://github.com/nats-io/nats-streaming-server/) for further info on usage, configuration, and administration. +NATS Streaming provides a rich set of commands and parameters to configure all aspects of the server. Please refer to the [README](https://github.com/nats-io/nats-streaming-server#configuring) for further information on usage, configuration, and administration. \ No newline at end of file diff --git a/nats_streaming/monitoring/enabling.md b/nats_streaming/monitoring/enabling.md new file mode 100644 index 0000000..675cc58 --- /dev/null +++ b/nats_streaming/monitoring/enabling.md @@ -0,0 +1,24 @@ +### Enabling + +To enable the monitoring server, start the NATS Streaming Server with the monitoring flag -m (or -ms) and specify the monitoring port. + +Monitoring options +``` +-m, --http_port PORT HTTP PORT for monitoring +-ms,--https_port PORT Use HTTPS PORT for monitoring (requires TLS cert and key) +``` +To enable monitoring via the configuration file, use `http: "host:port"` or `https: "host:port"` (there is no explicit configuration flag for the monitoring interface). + +For example, after running this: +``` +nats-streaming-server -m 8222 +``` +you should see that the NATS Streaming server starts with the HTTP monitoring port enabled: + +``` +(...) +[53359] 2017/12/18 17:44:31.594407 [INF] Starting http monitor on 0.0.0.0:8222 +[53359] 2017/12/18 17:44:31.594462 [INF] Listening for client connections on 0.0.0.0:4222 +(...) +``` +You can then point your browser (or curl) to [http://localhost:8222/streaming](http://localhost:8222/streaming) diff --git a/nats_streaming/monitoring/endpoints.md b/nats_streaming/monitoring/endpoints.md new file mode 100644 index 0000000..fdd042a --- /dev/null +++ b/nats_streaming/monitoring/endpoints.md @@ -0,0 +1,319 @@ +### Endpoints + +The following sections describe each supported monitoring endpoint: serverz, storez, clientsz, and channelsz. + +#### /serverz + +The endpoint [http://localhost:8222/streaming/serverz](http://localhost:8222/streaming/serverz) reports various general statistics. +``` +{ + "cluster_id": "test-cluster", + "server_id": "JEzjfVQS4JIEzM7lZmWHm9", + "version": "0.14.2", + "go": "go1.11.10", + "state": "STANDALONE", + "now": "2019-05-21T11:09:35.364637-06:00", + "start_time": "2019-05-21T11:09:24.204869-06:00", + "uptime": "11s", + "clients": 0, + "subscriptions": 0, + "channels": 0, + "total_msgs": 0, + "total_bytes": 0 +} +``` + +In clustering mode, there is an additional field that indicates the RAFT role of the given node. Here is an example: +``` +{ + "cluster_id": "test-cluster", + "server_id": "t9W9zbOIIi5Y9Guppxl0lF", + "version": "0.14.2", + "go": "go1.11.10", + "state": "CLUSTERED", + "role": "Follower", + "now": "2019-05-21T11:10:15.765261-06:00", + "start_time": "2019-05-21T11:10:12.21284-06:00", + "uptime": "3s", + "clients": 0, + "subscriptions": 0, + "channels": 0, + "total_msgs": 0, + "total_bytes": 0 +} +``` +The possible values are: `Leader`, `Follower` or `Candidate`. + +#### /storez + +The endpoint [http://localhost:8222/streaming/storez](http://localhost:8222/streaming/storez) reports information about the store. +``` +{ + "cluster_id": "test-cluster", + "server_id": "8AjZq57k4JY7cfKEvuZ8iF", + "now": "2019-04-16T09:57:32.857406-06:00", + "type": "MEMORY", + "limits": { + "max_channels": 100, + "max_msgs": 1000000, + "max_bytes": 1024000000, + "max_age": 0, + "max_subscriptions": 1000, + "max_inactivity": 0 + }, + "total_msgs": 130691, + "total_bytes": 19587140 +} +``` + +#### /clientsz + +The endpoint [http://localhost:8222/streaming/clientsz](http://localhost:8222/streaming/clientsz) reports more detailed information about the connected clients. + +It uses a paging mechanism which defaults to 1024 clients. + +You can control these via URL arguments (limit and offset). For example: [http://localhost:8222/streaming/clientsz?limit=1&offset=1](http://localhost:8222/streaming/clientsz?limit=1&offset=1). +``` +{ + "cluster_id": "test-cluster", + "server_id": "J3Odi0wXYKWKFWz5D5uhH9", + "now": "2017-06-07T14:47:44.495254605+02:00", + "offset": 1, + "limit": 1, + "count": 1, + "total": 11, + "clients": [ + { + "id": "benchmark-sub-0", + "hb_inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayQK" + } + ] +} +``` +You can also report detailed subscription information on a per client basis using `subs=1`. For example: [http://localhost:8222/streaming/clientsz?limit=1&offset=1&subs=1](http://localhost:8222/streaming/clientsz?limit=1&offset=1&subs=1). +``` +{ + "cluster_id": "test-cluster", + "server_id": "J3Odi0wXYKWKFWz5D5uhH9", + "now": "2017-06-07T14:48:06.157468748+02:00", + "offset": 1, + "limit": 1, + "count": 1, + "total": 11, + "clients": [ + { + "id": "benchmark-sub-0", + "hb_inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayQK", + "subscriptions": { + "foo": [ + { + "client_id": "benchmark-sub-0", + "inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayvC", + "ack_inbox": "_INBOX.J3Odi0wXYKWKFWz5D5uhem", + "is_durable": false, + "is_offline": false, + "max_inflight": 1024, + "ack_wait": 30, + "last_sent": 505597, + "pending_count": 0, + "is_stalled": false + } + ] + } + } + ] +} +``` +You can select a specific client based on its client ID with `client=`, and get also get detailed statistics with `subs=1`. For example: [http://localhost:8222/streaming/clientsz?client=me&subs=1](http://localhost:8222/streaming/clientsz?client=me&subs=1). +``` +{ + "id": "me", + "hb_inbox": "_INBOX.HG0uDuNtAPxJQ1lVjIC2sr", + "subscriptions": { + "foo": [ + { + "client_id": "me", + "inbox": "_INBOX.HG0uDuNtAPxJQ1lVjIC389", + "ack_inbox": "_INBOX.Q9iH2gsDPN57ZEvqswiYSL", + "is_durable": false, + "is_offline": false, + "max_inflight": 1024, + "ack_wait": 30, + "last_sent": 0, + "pending_count": 0, + "is_stalled": false + } + ] + } +} +``` + +#### /channelsz + +The endpoint [http://localhost:8222/streaming/channelsz](http://localhost:8222/streaming/channelsz) reports the list of channels. +``` +{ + "cluster_id": "test-cluster", + "server_id": "J3Odi0wXYKWKFWz5D5uhH9", + "now": "2017-06-07T14:48:41.680592041+02:00", + "offset": 0, + "limit": 1024, + "count": 2, + "total": 2, + "names": [ + "bar" + "foo" + ] +} +``` +It uses a paging mechanism which defaults to 1024 channels. + +You can control these via URL arguments (limit and offset). For example: [http://localhost:8222/streaming/channelsz?limit=1&offset=1](http://localhost:8222/streaming/channelsz?limit=1&offset=1). +``` +{ + "cluster_id": "test-cluster", + "server_id": "J3Odi0wXYKWKFWz5D5uhH9", + "now": "2017-06-07T14:48:41.680592041+02:00", + "offset": 1, + "limit": 1, + "count": 1, + "total": 2, + "names": [ + "foo" + ] +} +``` +You can also get the list of subscriptions with `subs=1`. For example: [http://localhost:8222/streaming/channelsz?limit=1&offset=0&subs=1](http://localhost:8222/streaming/channelsz?limit=1&offset=0&subs=1). +``` +{ + "cluster_id": "test-cluster", + "server_id": "J3Odi0wXYKWKFWz5D5uhH9", + "now": "2017-06-07T15:01:02.166116959+02:00", + "offset": 0, + "limit": 1, + "count": 1, + "total": 2, + "channels": [ + { + "name": "bar", + "msgs": 0, + "bytes": 0, + "first_seq": 0, + "last_seq": 0, + "subscriptions": [ + { + "client_id": "me", + "inbox": "_INBOX.S7kTJjOcToXiJAzGWgINit", + "ack_inbox": "_INBOX.Y04G5pZxlint3yPXrSTjTV", + "is_durable": false, + "is_offline": false, + "max_inflight": 1024, + "ack_wait": 30, + "last_sent": 0, + "pending_count": 0, + "is_stalled": false + } + ] + } + ] +} +``` +You can select a specific channel based on its name with `channel=name`. For example: [http://localhost:8222/streaming/channelsz?channel=foo](http://localhost:8222/streaming/channelsz?channel=foo). +``` +{ + "name": "foo", + "msgs": 649234, + "bytes": 97368590, + "first_seq": 1, + "last_seq": 649234 +} +``` +And again, you can get detailed subscriptions with `subs=1`. For example: [http://localhost:8222/streaming/channelsz?channel=foo&subs=1](http://localhost:8222/streaming/channelsz?channel=foo&subs=1). +``` +{ + "name": "foo", + "msgs": 704770, + "bytes": 105698990, + "first_seq": 1, + "last_seq": 704770, + "subscriptions": [ + { + "client_id": "me", + "inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayvC", + "ack_inbox": "_INBOX.J3Odi0wXYKWKFWz5D5uhem", + "is_durable": false, + "is_offline": false, + "max_inflight": 1024, + "ack_wait": 30, + "last_sent": 704770, + "pending_count": 0, + "is_stalled": false + }, + { + "client_id": "me2", + "inbox": "_INBOX.jAHSY3hcL5EGFQGYmfaywG", + "ack_inbox": "_INBOX.J3Odi0wXYKWKFWz5D5uhjV", + "is_durable": false, + "is_offline": false, + "max_inflight": 1024, + "ack_wait": 30, + "last_sent": 704770, + "pending_count": 0, + "is_stalled": false + }, + (...) + ] +} +``` + +For durables that are currently running, the `is_offline` field is set to `false`. Here is an example: +``` +{ + "name": "foo", + "msgs": 0, + "bytes": 0, + "first_seq": 0, + "last_seq": 0, + "subscriptions": [ + { + "client_id": "me", + "inbox": "_INBOX.P23kNGFnwC7KRg3jIMB3IL", + "ack_inbox": "_STAN.ack.pLyMpEyg7dgGZBS7jGXC02.foo.pLyMpEyg7dgGZBS7jGXCaw", + "durable_name": "dur", + "is_durable": true, + "is_offline": false, + "max_inflight": 1024, + "ack_wait": 30, + "last_sent": 0, + "pending_count": 0, + "is_stalled": false + } + ] +} +``` + +When that same durable goes offline, `is_offline` is be set to `true`. Although the client is possibly no longer connected (and would not appear in the `clientsz` endpoint), the `client_id` field is still displayed here. +``` +{ + "name": "foo", + "msgs": 0, + "bytes": 0, + "first_seq": 0, + "last_seq": 0, + "subscriptions": [ + { + "client_id": "me", + "inbox": "_INBOX.P23kNGFnwC7KRg3jIMB3IL", + "ack_inbox": "_STAN.ack.pLyMpEyg7dgGZBS7jGXC02.foo.pLyMpEyg7dgGZBS7jGXCaw", + "durable_name": "dur", + "is_durable": true, + "is_offline": true, + "max_inflight": 1024, + "ack_wait": 30, + "last_sent": 0, + "pending_count": 0, + "is_stalled": false + } + ] +} +``` diff --git a/nats_streaming/monitoring/monitoring.md b/nats_streaming/monitoring/monitoring.md new file mode 100644 index 0000000..4ef4fff --- /dev/null +++ b/nats_streaming/monitoring/monitoring.md @@ -0,0 +1,3 @@ +## Monitoring + +To monitor the NATS streaming system, a lightweight HTTP server is used on a dedicated monitoring port. The monitoring server provides several endpoints, all returning a JSON object. diff --git a/nats_streaming/nats-streaming-install.md b/nats_streaming/nats-streaming-install.md deleted file mode 100644 index 8717bef..0000000 --- a/nats_streaming/nats-streaming-install.md +++ /dev/null @@ -1,102 +0,0 @@ -# Install and Run NATS Streaming Server - -In this tutorial you install and run the NATS Streaming server (`nats-streaming-server`). -You can follow this same procedure anytime you want to run the NATS Streaming server. - -### Install the NATS Streaming server - -There are numerous ways to install the NATS Streaming server. - -#### GitHub releases - -The latest official release binaries are always available on the [GitHub releases page](https://github.com/nats-io/nats-streaming-server/releases). -The following platforms are available: - -- Linux (x86, x86_64, ARM) -- Windows (x86, x86_64) -- macOS - -The following methods may also be used. _Please note that these methods may not install the latest released version_: - -#### Go - -Make sure [your Go environment is set up](https://golang.org/doc/install) - -```sh -% go get github.com/nats-io/nats-streaming-server -``` - -Note that this method may not install the latest released version. - -#### Docker Hub - -The latest [official Docker image](https://hub.docker.com/_/nats-streaming/) is always available on Docker Hub. - -#### Windows - -On Windows, the NATS Streaming server can also be installed via [Chocolatey](https://chocolatey.org/packages/nats-streaming-server): - -```sh -% choco install nats-streaming-server -``` - -#### macOS - -On macOS, the NATS Streaming server can be installed via [Homebrew](http://brewformulas.org/NatsStreamingServer): - -```sh -% brew install nats-streaming-server -``` - -### Start the NATS Streaming server - -You can invoke the NATS Streaming server binary, with no options and no configuration file, to start a server with acceptable standalone defaults (no authentication, no clustering). - -```sh -% nats-streaming-server -``` - -When the server starts successfully, you will see that the NATS Streaming server listens for client connections on TCP Port 4222: - -```sh -[18085] 2016/10/31 13:11:44.059012 [INF] Starting nats-streaming-server[test-cluster] version 0.3.1 -[18085] 2016/10/31 13:11:44.059830 [INF] Starting nats-server version 0.9.4 -[18085] 2016/10/31 13:11:44.061544 [INF] Listening for client connections on 0.0.0.0:4222 -[18085] 2016/10/31 13:11:44.061966 [INF] Server is ready -[18085] 2016/10/31 13:11:44.396819 [INF] STAN: Message store is MEMORY -[18085] 2016/10/31 13:11:44.396832 [INF] STAN: --------- Store Limits --------- -[18085] 2016/10/31 13:11:44.396837 [INF] STAN: Channels: 100 * -[18085] 2016/10/31 13:11:44.396839 [INF] STAN: -------- channels limits ------- -[18085] 2016/10/31 13:11:44.396842 [INF] STAN: Subscriptions: 1000 * -[18085] 2016/10/31 13:11:44.396844 [INF] STAN: Messages : 1000000 * -[18085] 2016/10/31 13:11:44.396855 [INF] STAN: Bytes : 976.56 MB * -[18085] 2016/10/31 13:11:44.396858 [INF] STAN: Age : unlimited * -[18085] 2016/10/31 13:11:44.396859 [INF] STAN: -------------------------------- -``` - -### Start the NATS Streaming Server with NATS monitoring enabled (optional) - -The NATS Streaming server exposes the monitoring interface of its embedded NATS Server (`nats-server`) on port 8222. - -```sh -% nats-streaming-server -m 8222 -``` - -If you run the NATS Streaming server with monitoring enabled, you see the following messages: - -```sh -[18122] 2016/10/31 13:13:10.048663 [INF] Starting nats-streaming-server[test-cluster] version 0.3.1 -[18122] 2016/10/31 13:13:10.048843 [INF] Starting nats-server version 0.9.4 -[18122] 2016/10/31 13:13:10.048890 [INF] Starting http monitor on 0.0.0.0:8222 -[18122] 2016/10/31 13:13:10.048968 [INF] Listening for client connections on 0.0.0.0:4222 -[18122] 2016/10/31 13:13:10.048992 [INF] Server is ready -[18122] 2016/10/31 13:13:10.388282 [INF] STAN: Message store is MEMORY -[18122] 2016/10/31 13:13:10.388301 [INF] STAN: --------- Store Limits --------- -[18122] 2016/10/31 13:13:10.388309 [INF] STAN: Channels: 100 * -[18122] 2016/10/31 13:13:10.388312 [INF] STAN: -------- channels limits ------- -[18122] 2016/10/31 13:13:10.388316 [INF] STAN: Subscriptions: 1000 * -[18122] 2016/10/31 13:13:10.388319 [INF] STAN: Messages : 1000000 * -[18122] 2016/10/31 13:13:10.388333 [INF] STAN: Bytes : 976.56 MB * -[18122] 2016/10/31 13:13:10.388338 [INF] STAN: Age : unlimited * -[18122] 2016/10/31 13:13:10.388341 [INF] STAN: -------------------------------- -``` diff --git a/nats_streaming/nats-streaming-protocol.md b/nats_streaming/nats-streaming-protocol.md deleted file mode 100644 index 3533a23..0000000 --- a/nats_streaming/nats-streaming-protocol.md +++ /dev/null @@ -1,190 +0,0 @@ -# NATS Streaming Protocol - - -The NATS streaming protocol sits atop the core [NATS protocol](/documentation/internals/nats-protocol) and uses [Google's Protocol Buffers](https://developers.google.com/protocol-buffers/). Protocol buffer messages are marshaled into bytes and published as NATS messages on specific subjects described below. In communicating with the NATS Streaming Server, the NATS [request/reply](/documentation/writing_applications/concepts) pattern is used for all protocol messages that have a corresponding reply. - -## NATS streaming protocol conventions - -**Subject names**: Subject names, including reply subject (INBOX) names, are case-sensitive and must be non-empty alphanumeric strings with no embedded whitespace, and optionally token-delimited using the dot character (`.`), e.g.: - -`FOO`, `BAR`, `foo.bar`, `foo.BAR`, `FOO.BAR` and `FOO.BAR.BAZ` are all valid subject names - -`FOO. BAR`, `foo. .bar` and`foo..bar` are *not- valid subject names - -**Wildcards**: NATS streaming does **not*- support wildcards in subject subscriptions - -**Protocol definition**: The fields of NATS streaming protocol messages are defined in the go-nats-streaming [protocol file](https://github.com/nats-io/go-nats-streaming/blob/master/pb/protocol.proto). - -## NATS streaming protocol messages - -The following table briefly describes the NATS streaming protocol messages. - -Click the name to see more detailed information, including usage: - -| Message Name | Sent By | Description -| --------------------------------- |:--------|:-------------------------------------------- -| [`ConnectRequest`](#CONNREQ) | Client | Request to connect to the NATS Streaming Server -| [`ConnectResponse`](#CONNRESP) | Server | Result of a connection request -| [`SubscriptionRequest`](#SUBREQ) | Client | Request sent to subscribe and retrieve data -| [`SubscriptionResponse`](#SUBRESP)| Server | Result of a subscription request -| [`UnsubscribeRequest`](#UNSUBREQ) | Client | Unsubscribe from a subject -| [`PubMsg`](#PUBMSG) | Client | Publish a message to a subject, with optional reply subject -| [`PubAck`](#PUBACK) | Server | An acknowledgement that a published message has been processed on the server -| [`MsgProto`](#MSGPROTO) | Server | A message from the NATS Streaming Server to a subscribing client -| [`Ack`](#ACK) | Client | Acknowledges that a message has been received -| [`CloseRequest`](#CLOSEREQ) | Client | Request sent to close the connection to the NATS Streaming Server -| [`CloseResp`](#CLOSERESP) | Server | Result of the close request - -The following sections explain each protocol message. - -## ConnectRequest - -#### Description - -A connection request is sent when a streaming client connects to the NATS Streaming Server. The connection request contains a unique identifier representing the client, and an inbox subject the client will listen on for incoming heartbeats. The identifier **must*- be unique; a connection attempt with an identifier currently in use will fail. The inbox subject is the subject where the client receives incoming heartbeats, and responds by publishing an empty NATS message to the reply subject, indicating it is alive. The NATS Streaming Server will return a [ConnectResponse](#CONNRESP) message to the reply subject specified in the NATS request message. - -This request is published to a subject comprised of the `.cluster-id`, for example, if a NATS Streaming Server was started with a cluster-id of `mycluster`, and the default prefix was used, the client publishes to `_STAN.discover.mycluster` - -#### Message Structure - -- `clientID`: A unique identifier for a client -- `heartbeatInbox`: An inbox to which the NATS Streaming Server will send heartbeats for the client to process - -## ConnectResponse - -#### Description - -After a `ConnectRequest` is published, the NATS Streaming Server responds with this message on the reply subject of the underlying NATS request. The NATS Streaming Server requires the client to make requests and publish messages on certain subjects (described above), and when a connection is successful, the client saves the information returned to be used in sending other NATS streaming protocol messages. In the event the connection was not successful, an error is returned in the `error` field. - -#### Message Structure - -- `pubPrefix`: Prefix to use when publishing -- `subRequests`: Subject used for subscription requests -- `unsubRequests`: Subject used for unsubscribe requests -- `closeRequests`: Subject for closing a connection -- `error`: An error string, which will be empty/omitted upon success -- `publicKey`: Reserved for future use - -## SubscriptionRequest - -#### Description - -A `SubscriptionRequest` is published on the subject returned in the `subRequests` field of a [`ConnectResponse`](#CONNRESP), and creates a subscription to a subject on the NATS Streaming Server. This will return a [SubscriptionResponse](#SUBRESP) message to the reply subject specified in the NATS protocol request message. - -#### Message Structure - -- `clientID`: Client ID originally provided in the [ConnectRequest](#CONNREQ) -- `subject`: Formal subject to subscribe to, e.g. foo.bar -- `qGroup`: Optional queue group -- `inbox`: Inbox subject to deliver messages on -- `maxInFlight`: Maximum inflight messages without an acknowledgement allowed -- `ackWaitInSecs`: Timeout for receiving an acknowledgement from the client -- `durableName`: Optional durable name which survives client restarts -- `startPosition`: An enumerated type specifying the point in history to start replaying data -- `startSequence`: Optional start sequence number -- `startTimeDelta`: Optional start time - -#### StartPosition enumeration - -- `NewOnly`: Send only new messages -- `LastReceived`: Send only the last received message -- `TimeDeltaStart`: Send messages from duration specified in the `startTimeDelta` field. -- `SequenceStart`: Send messages starting from the sequence in the `startSequence` field. -- `First`: Send all available messages - -## SubscriptionResponse - -#### Description - -The `SubscriptionResponse` message is the response from the `SubscriptionRequest`. After a client has processed an incoming [MsgProto](#MSGPROTO) message, it must send an acknowledgement to the `ackInbox` subject provided here. - -#### Message Structure - -- `ackInbox`: subject the client sends message acknowledgements to the NATS Streaming Server -- `error`: error string, empty/omitted if no error - -## UnsubscribeRequest - -#### Description - -The `UnsubscribeRequest` unsubcribes the connection from the specified subject. The inbox specified is the `inbox` returned from the NATS Streaming Server in the `SubscriptionResponse`. - -#### Message Structure - -- `clientID`: Client ID originally provided in the [ConnectRequest](#CONNREQ) -- `subject`: Subject for the subscription -- `inbox`: Inbox subject to identify subscription -- `durableName`: Optional durable name which survives client restarts - -## PubMsg - -#### Description - -The `PubMsg` protocol message is published from a client to the NATS Streaming Server. The GUID must be unique, and is returned in the [PubAck](#PUBACK) message to correlate the success or failure of storing this particular message. - -#### Message Structure - -- `clientID`: Client ID originally provided in the [ConnectRequest](#CONNREQ) -- `guid`: a guid generated for this particular message -- `subject`: subject -- `reply`: optional reply subject -- `data`: payload -- `sha256`: optional sha256 of payload data - -## PubAck - -#### Description - -The `PubAck` message is an acknowledgement from the NATS Streaming Server that a message has been processed. The message arrives on the subject specified on the reply subject of the NATS message the `PubMsg` was published on. The GUID is the same GUID used in the `PubMsg` being acknowledged. If an error string is present, the message was not persisted by the NATS Streaming Server and no guarantees regarding persistence are honored. `PubAck` messages may be handled asynchronously from their corresponding `PubMsg` in the client. - -#### Message Structure - -- `guid`: GUID of the message being acknowledged by the NATS Streaming Server -- `error`: An error string, empty/omitted if no error - -## MsgProto - -#### Description - -The `MsgProto` message is received by client from the NATS Streaming Server, containing the payload of messages sent by a publisher. A `MsgProto` message that is not acknowledged with an [Ack](#ACK) message within the duration specified by the `ackWaitInSecs` field of the subscription request will be redelivered. - -#### Message Structure - -- `sequence`: Globally ordered sequence number for the subject's channel -- `subject`: Subject -- `reply`: Optional reply -- `data`: Payload -- `timestamp`: Time the message was stored in the server. -- `redelivered`: Flag specifying if the message is being redelivered -- `CRC32`: Optional IEEE CRC32 - -## Ack - -#### Description - -An `Ack` message is an acknowledgement from the client that a [MsgProto](#MSGPROTO) message has been considered received. It is published to the `ackInbox` field of the [`SubscriptionResponse`](#SUBRESP). - -#### Message Structure - -- `subject`: Subject of the message being acknowledged -- `sequence`: Sequence of the message being acknowledged - -## CloseRequest - -#### Description - -A `CloseRequest` message is published on the `closeRequests` subject from the [`ConnectResponse`](#CONNRESP), and notifies the NATS Streaming Server that the client connection is closing, allowing the server to free up resources. This message should **always*- be sent when a client is finished using a connection. - -#### Message Structure - -- `clientID`: Client ID originally provided in the [ConnectRequest](#CONNREQ) - -## CloseResponse - -#### Description - -The `CloseResponse` is sent by the NATS Streaming Server on the reply subject of the `CloseRequest` NATS message. This response contains any error that may have occurred with the corresponding close call. - -#### Message Structure - -- `error`: error string, empty/omitted if no error diff --git a/nats_streaming/nats-streaming-quickstart.md b/nats_streaming/nats-streaming-quickstart.md deleted file mode 100644 index 5812482..0000000 --- a/nats_streaming/nats-streaming-quickstart.md +++ /dev/null @@ -1,82 +0,0 @@ -# Getting Started with NATS Streaming - -This tutorial demonstrates NATS Streaming using example [Go NATS Streaming clients](https://github.com/nats-io/go-nats-streaming.git). - -## Prerequisites - -- [Set up your Git environment](https://help.github.com/articles/set-up-git/). -- [Set up your Go environment](https://golang.org/doc/install). - -## Setup - -Download and install the [NATS Streaming Server](https://github.com/nats-io/nats-streaming-server/releases). - -Clone the following repositories: - -- NATS Streaming Server: `git clone https://github.com/nats-io/nats-streaming-server.git` -- NATS Streaming Client: `git clone https://github.com/nats-io/go-nats-streaming.git` - -## Start the NATS Streaming Server - -Two options: - -Run the binary that you downloaded, for example: `$ ./nats-streaming-server` - -Or, run from source: - -```sh -% cd $GOPATH/src/github.com/nats-io/nats-streaming-server -% go run nats-streaming-server.go -``` - -You should see the following, indicating that the NATS Streaming Server is running: - -```sh -% go run nats-streaming-server.go -[89999] 2016/06/25 08:54:35.399071 [INF] Starting nats-streaming-server[test-cluster] version 0.1.0 -[89999] 2016/06/25 08:54:35.399315 [INF] Starting nats-server version 0.9.0.beta -[89999] 2016/06/25 08:54:35.399326 [INF] Listening for client connections on localhost:4222 -[89999] 2016/06/25 08:54:35.400721 [INF] Server is ready -[89999] 2016/06/25 08:54:35.737589 [INF] STAN: Message store is MEMORY -[89999] 2016/06/25 08:54:35.737610 [INF] STAN: Maximum of 1000000 will be stored -``` - -## Run the publisher client - -Publish several messages. For each publication you should get a result. - -```sh -% cd $GOPATH/src/github.com/nats-io/go-nats-streaming/examples/stan-pub -% go run main.go foo "msg one" -Published [foo] : 'msg one' -% go run main.go foo "msg two" -Published [foo] : 'msg two' -% go run main.go foo "msg three" -Published [foo] : 'msg three' -``` - -## Run the subscriber client - -Use the `--all` flag to receive all published messages. - -```sh -% cd $GOPATH/src/github.com/nats-io/go-nats-streaming/examples/stan-sub -% go run main.go --all -c test-cluster -id myID foo -Connected to nats://localhost:4222 clusterID: [test-cluster] clientID: [myID] -subscribing with DeliverAllAvailable -Listening on [foo], clientID=[myID], qgroup=[] durable=[] -[#1] Received on [foo]: 'sequence:1 subject:"foo" data:"msg one" timestamp:1465962202884478817 ' -[#2] Received on [foo]: 'sequence:2 subject:"foo" data:"msg two" timestamp:1465962208545003897 ' -[#3] Received on [foo]: 'sequence:3 subject:"foo" data:"msg three" timestamp:1465962215567601196 -``` - -## Explore other subscription options - -```sh - --seq Start at seqno - --all Deliver all available messages - --last Deliver starting with last published message - --since Deliver messages in last interval (e.g. 1s, 1hr, https://golang.org/pkg/time/#ParseDuration) - --durable Durable subscriber name - --unsubscribe Unsubscribe the durable on exit -``` diff --git a/nats_streaming/partitioning.md b/nats_streaming/partitioning.md new file mode 100644 index 0000000..1e6ab4c --- /dev/null +++ b/nats_streaming/partitioning.md @@ -0,0 +1,119 @@ +# Partitioning + +***Note, this feature is incompatible with Clustering mode. Trying to start a server with Partitioning and Clustering enabled will result in a startup error.*** + +It is possible to limit the list of channels a server can handle. This can be used to: + +* Prevent creation of unwanted channels +* Share the load between several servers running with the same cluster ID + +In order to do so, you need to enable the `partitioning` parameter in the configuration file, and also specify the list of allowed channels in the `channels` section of the `store_limits` configuration. + +Channels don't need to override any limit, but they need to be specified for the server to service only these channels. + +Here is an example: + +``` +partitioning: true +store_limits: { + channels: { + "foo": {} + "bar": {} + # Use of wildcards in configuration is allowed. However, applications cannot + # publish to, or subscribe to, wildcard channels. + "baz.*": {} + } +} +``` + +When partitioning is enabled, multiple servers with the same cluster ID can coexist on the same NATS network, each server handling its own set of channels. ***Note however that in this mode, state is not replicated as it is in Clustering mode. The only communication between servers is to report if a given channel is handled in more than one serve.r*** + +## Wildcards + +NATS Streaming does not support sending or subscribing to wildcard channels (such as `foo.*`). + +However, it is possible to use wildcards to define the partition that a server can handle. For instance, with the following configuration: +``` +partitioning: true +store_limits: { + channels: { + "foo.*": {} + "bar.>": {} + } +} +``` +The streaming server would accept subscriptions or published messages to channels such as: +1. `foo.bar` +2. `bar.baz` +3. `bar.baz.bat` +4. ... + +But would ignore messages or subscriptions on: + +1. `foo` +2. `foo.bar.baz` +3. `bar` +4. `some.other.channel` +5. ... + +## A given channel must be defined in a single server + +When a server starts, it sends its list of channels to all other servers on the same cluster in an attempt to detect duplicate channels. When a server receives this list and finds that it has a channel in common, it will return an error to the emitting server, which will then fail to start. + +However, on startup, it is possible that the underlying NATS cluster is not fully formed. The server would not get any response from the rest of the cluster and therefore start successfully and service clients. Anytime a Streaming server detects that a NATS server was added to the NATS cluster, it will resend its list of channels. It means that currently running servers may suddenly fail with a message regarding duplicate channels. Having the same channel on different servers means that a subscription would be created on all servers handling the channel, but only one server will receive and process message acknowledgements. Other servers would then redeliver messages (since they would not get the acknowledgements), which would cause duplicates. + +***In order to avoid issues with channels existing on several servers, it is ultimately the responsibility of the administrator to ensure that channels are unique.*** + +## Fault Tolerance and Partitioning + +You can easily combine the Fault Tolerance and Partitioning feature. + +To illustrate, suppose that we want two partitions, one for `foo.>` and one for `bar.>`. + +The configuration for the first server `foo.conf` would look like: +``` +partitioning: true +store_limits: { + channels: { + foo.>: {} + } +} +``` + +The second configuration `bar.conf` would be: +``` +partitioning: true +store_limits: { + channels: { + bar.>: {} + } +} +``` + +If you remember, Fault Tolerance is configured by specifying a name (`ft_group_name`). Suppose there is an NFS mount called `/nss/datastore` on both `host1` and `host2`. + +Starting an FT pair for the partition `foo` could look like this: +``` +host1$ nats-streaming-server -store file -dir /nss/datastore/foodata -sc foo.conf -ft_group_name foo -cluster nats://host1:6222 -routes nats://host2:6222,nats://host2:6223 + +host2$ nats-streaming-server -store file -dir /nss/datastore/foodata -sc foo.conf -ft_group_name foo -cluster nats://host2:6222 -routes nats://host1:6222,nats://host1:6223 +``` +Notice that each server on each note points to each other (the `-routes` parameter). The reason why we also point to `6223` will be explained later. They both listen for routes connections on their host's `6222` port. + +We now start the FT pair for `bar`. Since we are running from the same machines (we don't have to), we need to use a different port: +``` +host1$ nats-streaming-server -store file -dir /nss/datastore/bardata -sc bar.conf -ft_group_name bar -p 4223 -cluster nats://host1:6223 -routes nats://host2:6222,nats://host2:6223 + +host2$ nats-streaming-server -store file -dir /nss/datastore/bardata -sc bar.conf -ft_group_name bar -p 4223 -cluster nats://host2:6223 -routes nats://host1:6222,nats://host1:6223 +``` +You will notice that the `-routes` parameter points to both `6222` and `6223`, this is so that both partitions belong to the same cluster and be view as "one" by a Streaming application connecting to this cluster. Effectively, we have created a full mesh of 4 NATS servers that can all communicate with each other. Two of these servers are backups for servers running on the same FT group. + +## Applications behavior + +When an application connects, it specifies a cluster ID. If several servers are running with that same cluster ID, the application will be able to publish/subscribe to any channel handled by the cluster (as long as those servers are all connected to the NATS network). + +A published message will be received by only the server that has that channel defined. If no server is handling this channel, no specific error is returned, instead the publish call will timeout. Same goes for message acknowledgements. Only the server handling the subscription on this channel should receive those. + +However, other client requests (such as connection and subscription requests) are received by all servers. For connections, all servers handle them and the client library will receive a response from all servers in the cluster, but use the first one that it received. + +For subscriptions, a server receiving the request for a channel that it does not handle will simply ignore the request. Again, if no server handle this channel, the client's subscription request will simply time out. \ No newline at end of file diff --git a/nats_streaming/relation-to-nats.md b/nats_streaming/relation-to-nats.md new file mode 100644 index 0000000..cce74fa --- /dev/null +++ b/nats_streaming/relation-to-nats.md @@ -0,0 +1,12 @@ +# Relation to NATS + +NATS Streaming Server by default embeds a [NATS](https://github.com/nats-io/nats-server) server. That is, the Streaming server is not a server per-se, but instead, a client to a NATS Server.
+It means that Streaming clients are not directly connected to the streaming server, but instead communicate with the streaming server *through* NATS Server. + +This detail is important when it comes to Streaming clients connections to the Streaming server. Indeed, since there is no direct connection, the server knows if a client is connected based on heartbeats. + +***It is therefore strongly recommended for clients to close their connection when the application exit, otherwise the server will consider these clients connected (sending data, etc...) until it detects missing heartbeats.*** + +The streaming server creates internal subscriptions on specific subjects to communicate with its clients and/or other servers. + +Note that NATS clients and NATS Streaming clients cannot exchange data between each other. That is, if a streaming client publishes on `foo`, a NATS client subscribing on that same subject will not receive the messages. Streaming messages are NATS messages made of a protobuf. The streaming server is expected to send ACKs back to producers and receive ACKs from consumers. If messages were freely exchanged with the NATS clients, this would cause problems. diff --git a/nats_streaming/store-encryption.md b/nats_streaming/store-encryption.md new file mode 100644 index 0000000..9dbf744 --- /dev/null +++ b/nats_streaming/store-encryption.md @@ -0,0 +1,33 @@ +# Store Encryption + +The server can be configured to encrypt a message's payload when storing them, providing encryption at rest. This can be done from the command line or from the configuration file. Check `encrypt` and `encryption_key` in the [Configuring](#configuring) section. + +It is recommended to provide the encryption key through the environment variable `NATS_STREAMING_ENCRYPTION_KEY` instead of `encryption_key`. If encryption is enabled and `NATS_STREAMING_ENCRYPTION_KEY` is found, this will take precedence over `encryption_key` value. + +You can pass this from the command line this way: +``` +$ env NATS_STREAMING_ENCRYPTION_KEY="mykey" nats-streaming-server -store file -dir datastore -encrypt +``` + +We currently support two ciphers for encryption: [AES](https://godoc.org/crypto/aes) and [CHACHA](https://godoc.org/golang.org/x/crypto/chacha20poly1305). +The default selected cipher depends on the platform. For ARM, we use `CHACHA`, otherwise we default to `AES`. You can always override that decision by explicitly specifying the cipher like this: +``` +$ env NATS_STREAMING_ENCRYPTION_KEY="mykey" nats-streaming-server -store file -dir datastore -encrypt -encryption_cipher "CHACHA" +``` +or, to select `AES`: +``` +$ env NATS_STREAMING_ENCRYPTION_KEY="mykey" nats-streaming-server -store file -dir datastore -encrypt -encryption_cipher "AES" +``` + +Note that only message payload is encrypted, all other data stored by NATS Streaming server is not. + +When running in clustering mode (see below), the server uses RAFT, which uses its own log files. Those will be encrypted too. + +Starting a server with `encrypt` against a datastore that was not encrypted may result in failures when it comes to decrypt a message, which may not happen immediately upon startup. Instead, +it will happen when attempting to deliver messages to consumers. However, when possible, the server will detect if the data was not encrypted and return the data without attempting to decrypt it. +The server will also detect which cipher was used to encrypt the data and use the proper cipher to decrypt, even if this is not the currently selected cipher. + +If the data is encrypted with a key and the server is restarted with a different key, the server will fail to decrypt messages when attempting to load them from the store. + +Performance considerations: As expected, encryption is likely to decrease performance, but by how much is hard to define. In some performance tests on a MacbookPro 2.8 GHz Intel Core i7 with SSD, we have +observed as little as 1% decrease to more than 30%. In addition to CPU cycles required for encryption, the encrypted payload is bigger, which result in more data being stored or read. diff --git a/nats_streaming/store-interface.md b/nats_streaming/store-interface.md new file mode 100644 index 0000000..bf0f591 --- /dev/null +++ b/nats_streaming/store-interface.md @@ -0,0 +1,15 @@ +# Store Interface + +Every store implementation follows the [Store interface](https://github.com/nats-io/nats-streaming-server/blob/master/stores/store.go). + +On startup, the server creates a unique instance of the `Store`. The constructor of a store implementation can do some initialization and configuration check, but *must not* access, or attempt to recover, the storage at this point. This is important because when the server runs on Fault Tolerance mode, the storage must be shared across many servers but only one server can be using it. + +After instantiating the store, the server will then call `Recover()` in order to recover the persisted state. For implementations that do not support persistence, such as the provided `MemoryStore`, this call will simply return `nil` (without error) to indicate that no state was recovered. + +The `Store` is used to add/delete clients, create/lookup channels, etc... + +Creating/looking up a channel will return a `ChannelStore`, which points to two other interfaces, the `SubStore` and `MsgStore`. These stores, for a given channel, handle subscriptions and messages respectively. + +If you wish to contribute to a new store type, your implementation must include all these interfaces. For stores that allow recovery (such as file store as opposed to memory store), there are additional structures that have been defined and should be returned by `Recover()`. + +The memory and the provided file store implementations both use a generic store implementation to avoid code duplication. When writing your own store implementation, you can do the same for APIs that don't need to do more than what the generic implementation provides. You can check [MemStore](https://github.com/nats-io/nats-streaming-server/blob/master/stores/memstore.go) and [FileStore](https://github.com/nats-io/nats-streaming-server/blob/master/stores/filestore.go) implementations for more details. diff --git a/nats_streaming/nats-streaming-swarm.md b/nats_streaming/swarm.md similarity index 100% rename from nats_streaming/nats-streaming-swarm.md rename to nats_streaming/swarm.md From b4ecd58b337192933fe73ff3f0a23d88b7a0cf74 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:01:24 -0500 Subject: [PATCH 03/31] Update subscriptions.md --- nats_streaming/channels/subscriptions/subscriptions.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/nats_streaming/channels/subscriptions/subscriptions.md b/nats_streaming/channels/subscriptions/subscriptions.md index 3fea537..78b173b 100644 --- a/nats_streaming/channels/subscriptions/subscriptions.md +++ b/nats_streaming/channels/subscriptions/subscriptions.md @@ -1,13 +1,11 @@ # Subscriptions -A client creates a subscription on a given channel. Remember, there is no support for wildcards, so a subscription is really tied to -one and only one channel. The server will maintain the subscription state on behalf of the client until the later closes the subscription (or its connection). +A client creates a subscription on a given channel. Remember, there is no support for wildcards, so a subscription is really tied to one and only one channel. The server will maintain the subscription state on behalf of the client until the later closes the subscription (or its connection). -If there are messages in the log for this channel, messages will be sent to the consumer when the subscription is created. The server will -send up to the maximum number of inflight messages as given by the client when creating the subscription. +If there are messages in the log for this channel, messages will be sent to the consumer when the subscription is created. The server will send up to the maximum number of inflight messages as given by the client when creating the subscription. When receiving ACKs from the consumer, the server will then deliver more messages, if more are available. A subscription can be created to start at any point in the message log, either by message sequence, or by time. -Following pages describe all types of subscription. \ No newline at end of file +Following pages describe all types of subscription. From 687d45bdecee86123056006c02df7c91a23e82b7 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:03:47 -0500 Subject: [PATCH 04/31] Update client-connections.md --- nats_streaming/client-connections.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nats_streaming/client-connections.md b/nats_streaming/client-connections.md index fd04ede..a4794ff 100644 --- a/nats_streaming/client-connections.md +++ b/nats_streaming/client-connections.md @@ -4,6 +4,6 @@ As described, clients are not directly connected to the streaming server. Instea This client ID links a given connection to its published messages, subscriptions, especially durable subscriptions. Indeed, durable subscriptions are stored as a combination of the client ID and durable name. More on durable subscriptions later. -It is also used to resolve the issue of not having direct client connections to the server. For instance, say that a client crashes without closing the connection. It later restarts with the same client ID. The server will detect that this client ID is already in-use. It will try to contact that known client to its original private inbox. If the server does not receive a response - which would be the case if the client crashed - it will replace the old client with this new one.
+It is also used to resolve the issue of not having direct client connections to the server. For instance, say that a client crashes without closing the connection. It later restarts with the same client ID. The server will detect that this client ID is already in-use. It will try to contact that known client to its original private inbox. If the server does not receive a response - which would be the case if the client crashed - it will replace the old client with this new one. -Otherwise, the server would reject the connection request since the client ID is already in-use. \ No newline at end of file +Otherwise, the server would reject the connection request since the client ID is already in-use. From a88d987d43f8e0e04162ae6757df4e5d52550e4d Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:04:30 -0500 Subject: [PATCH 05/31] Update client-connections.md --- nats_streaming/client-connections.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/nats_streaming/client-connections.md b/nats_streaming/client-connections.md index a4794ff..3d03fa6 100644 --- a/nats_streaming/client-connections.md +++ b/nats_streaming/client-connections.md @@ -4,6 +4,4 @@ As described, clients are not directly connected to the streaming server. Instea This client ID links a given connection to its published messages, subscriptions, especially durable subscriptions. Indeed, durable subscriptions are stored as a combination of the client ID and durable name. More on durable subscriptions later. -It is also used to resolve the issue of not having direct client connections to the server. For instance, say that a client crashes without closing the connection. It later restarts with the same client ID. The server will detect that this client ID is already in-use. It will try to contact that known client to its original private inbox. If the server does not receive a response - which would be the case if the client crashed - it will replace the old client with this new one. - -Otherwise, the server would reject the connection request since the client ID is already in-use. +It is also used to resolve the issue of not having direct client connections to the server. For instance, say that a client crashes without closing the connection. It later restarts with the same client ID. The server will detect that this client ID is already in-use. It will try to contact that known client to its original private inbox. If the server does not receive a response - which would be the case if the client crashed - it will replace the old client with this new one. Otherwise, the server would reject the connection request since the client ID is already in-use. From a331a1143aa80fe57b1c4eb3abf2e59d83bdefd0 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:08:17 -0500 Subject: [PATCH 06/31] Update auto-configuration.md --- nats_streaming/clustering/auto-configuration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nats_streaming/clustering/auto-configuration.md b/nats_streaming/clustering/auto-configuration.md index 2a19c63..bd193e2 100644 --- a/nats_streaming/clustering/auto-configuration.md +++ b/nats_streaming/clustering/auto-configuration.md @@ -1,6 +1,6 @@ # Auto Configuration -We can also bootstrap a NATS Streaming cluster by starting one server as the seed node using the `-cluster_bootstrap` flag. This node will elect itself leader, so it's important to avoid starting multiple servers as seed. Once a seed node is started, other servers will automatically join the cluster. If the server is recovering, it will use the recovered cluster configuration. +We can also bootstrap a NATS Streaming cluster by starting **one server** as the seed node using the `-cluster_bootstrap` flag. This node will elect itself leader, **so it's important to avoid starting multiple servers as seed**. Once a seed node is started, other servers will automatically join the cluster. If the server is recovering, it will use the recovered cluster configuration. Here is an example of starting three servers in a cluster by starting one as the seed and letting the others automatically join: @@ -14,4 +14,4 @@ nats-streaming-server -store file -dir store-c -clustered -nats_server nats://lo For a given cluster ID, if more than one server is started with `cluster_bootstrap` set to true, each server with this parameter will report the misconfiguration and exit. -The very first server that bootstrapped the cluster can be restarted, however, the operator must remove the datastores of the other servers that were incorrectly started with the bootstrap parameter before attempting to restart them. If they are restarted -even without the `-cluster_bootstrap` parameter- but with existing state, they will once again start as a leader. +The very first server that bootstrapped the cluster can be restarted, however, the operator **must remove the datastores** of the other servers that were incorrectly started with the bootstrap parameter before attempting to restart them. If they are restarted -even without the `-cluster_bootstrap` parameter- but with existing state, they will once again start as a leader. From 080089af4bd134048bcb0de4f1b34000fa710229 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:12:27 -0500 Subject: [PATCH 07/31] Update configuration.md --- nats_streaming/clustering/configuration.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/nats_streaming/clustering/configuration.md b/nats_streaming/clustering/configuration.md index 6160294..cd519d7 100644 --- a/nats_streaming/clustering/configuration.md +++ b/nats_streaming/clustering/configuration.md @@ -21,8 +21,7 @@ nats-streaming-server -store file -dir store-d -clustered -nats_server nats://lo The equivalent clustering configurations can be specified in a configuration file under the `cluster` group. See the [Configuring](#configuring) section for more information. Here is an example of a cluster of 3 nodes using the following configuration files. The nodes are running on `host1`, `host2` and `host3` respectively. - -NOTE If you have an existing NATS cluster and want to run NATS Streaming Cluster on top of that, see details at the end of this section. +**NOTE** If you have an existing NATS cluster and want to run NATS Streaming Cluster on top of that, see details at the end of this section. On `host1`, this configuration indicates that the server will accept client connections on port 4222. It will accept route connections on port 6222. It creates 2 routes, to `host2` and `host3` cluster port. @@ -92,4 +91,4 @@ streaming { } ``` -In the example above, the configuration assumes no existing NATS Cluster and therefore configure the NATS routes between each node. Should you want to use an existing NATS cluster, do not include the "NATS specific configuration" section, instead, add `nats_server_url` in the `streaming` section to point to the NATS server you want. \ No newline at end of file +In the example above, the configuration assumes no existing NATS Cluster and therefore configure the NATS routes between each node. Should you want to use an existing NATS cluster, do not include the "NATS specific configuration" section, instead, add `nats_server_url` in the `streaming` section to point to the NATS server you want. From 87ee72ba05c30a1d0fe8294ef29249f63238b85e Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:17:45 -0500 Subject: [PATCH 08/31] Update supported-stores.md --- nats_streaming/clustering/supported-stores.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats_streaming/clustering/supported-stores.md b/nats_streaming/clustering/supported-stores.md index 0fb3a3e..7c0322e 100644 --- a/nats_streaming/clustering/supported-stores.md +++ b/nats_streaming/clustering/supported-stores.md @@ -6,4 +6,4 @@ The NATS Streaming stores server meta information, messages and subscriptions to However, in clustered mode, we use RAFT for leader election. The raft layer uses its own stores which are currently necessarily file based. The location of the RAFT stores defaults to the current directory under a sub-directory named after the cluster ID, or you can configure it using `--cluster_log_path`. -This means that even if you select an SQL Store, there will still be a need for storing data on the file system. +This means that even if you select a SQL Store, there will still be a need for storing data on the file system. From 5d986fe9553e66cd97987391645be0067d098627 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:20:02 -0500 Subject: [PATCH 09/31] Update ft.md --- nats_streaming/fault-tolerance/ft.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nats_streaming/fault-tolerance/ft.md b/nats_streaming/fault-tolerance/ft.md index b3833ce..34bc078 100644 --- a/nats_streaming/fault-tolerance/ft.md +++ b/nats_streaming/fault-tolerance/ft.md @@ -2,7 +2,7 @@ To minimize the single point of failure, NATS Streaming server can be run in Fault Tolerance mode. It works by having a group of servers with one acting as the active server (accessing the store) and handling all communication with clients, and all others acting as standby servers. -It is important to note that is not possible to run Nats Stream as Fault Tolerance mode and Clustering mode at the same time. +It is important to note that is not possible to run Nats Streaming as Fault Tolerance mode and Clustering mode at the same time. To start a server in Fault Tolerance (FT) mode, you specify an FT group name. @@ -12,4 +12,4 @@ Here is an example on how starting 2 servers in FT mode running on the same host nats-streaming-server -store file -dir datastore -ft_group "ft" -cluster nats://localhost:6222 -routes nats://localhost:6223 -p 4222 nats-streaming-server -store file -dir datastore -ft_group "ft" -cluster nats://localhost:6223 -routes nats://localhost:6222 -p 4223 -``` \ No newline at end of file +``` From 71888f8a00f7a0adc87568f6cc03f9fc7cee95f3 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:21:13 -0500 Subject: [PATCH 10/31] Update standby-server.md --- nats_streaming/fault-tolerance/standby-server.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats_streaming/fault-tolerance/standby-server.md b/nats_streaming/fault-tolerance/standby-server.md index da8fff6..79e64f3 100644 --- a/nats_streaming/fault-tolerance/standby-server.md +++ b/nats_streaming/fault-tolerance/standby-server.md @@ -1,3 +1,3 @@ # Standby servers -There can be as many as you want standby servers on the same group. These servers do not access the store and do not receive any data from the streaming clients. They are just running waiting for the detection of the active server failure. +There can be as many standby servers as you want in the same group. These servers do not access the store and do not receive any data from the streaming clients. They are just running waiting for the detection of the active server failure. From 3f54cd78c1830d9c90ceef0f0e3eaa2bccc29f20 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:24:06 -0500 Subject: [PATCH 11/31] Update intro.md --- nats_streaming/gettingstarted/intro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats_streaming/gettingstarted/intro.md b/nats_streaming/gettingstarted/intro.md index 6be1634..48554be 100644 --- a/nats_streaming/gettingstarted/intro.md +++ b/nats_streaming/gettingstarted/intro.md @@ -2,5 +2,5 @@ The best way to get the NATS Streaming Server is to use one of the pre-built release binaries which are available for OSX, Linux (x86-64/ARM), Windows. -Of course you can build the latest version of the server from the master branch. The master branch will always build and pass tests, but may not work correctly in your environment. You will first need Go installed on your machine (version 1.11+ is required) to build the NATS Streaming server. +Of course you can build the latest version of the server from the master branch. The master branch will always build and pass tests, but may not work correctly in your environment. You will first need at least Go version 1.11+ installed on your machine to build the NATS Streaming Server. From d99ac4369ad1cba093ed2eb6f9b06a0b78a832b7 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:24:56 -0500 Subject: [PATCH 12/31] Update intro.md --- nats_streaming/gettingstarted/intro.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/nats_streaming/gettingstarted/intro.md b/nats_streaming/gettingstarted/intro.md index 48554be..0ecab49 100644 --- a/nats_streaming/gettingstarted/intro.md +++ b/nats_streaming/gettingstarted/intro.md @@ -2,5 +2,4 @@ The best way to get the NATS Streaming Server is to use one of the pre-built release binaries which are available for OSX, Linux (x86-64/ARM), Windows. -Of course you can build the latest version of the server from the master branch. The master branch will always build and pass tests, but may not work correctly in your environment. You will first need at least Go version 1.11+ installed on your machine to build the NATS Streaming Server. - +Of course you can build the latest version of the server from the master branch. The master branch will always build and pass tests, but may not work correctly in your environment. First you will need at least Go version 1.11+ installed on your machine to build the NATS Streaming Server. From 6003cccd10c446683bc586593b4e6f76a560a9d9 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:25:40 -0500 Subject: [PATCH 13/31] Update process-signaling.md --- nats_streaming/gettingstarted/process-signaling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats_streaming/gettingstarted/process-signaling.md b/nats_streaming/gettingstarted/process-signaling.md index a3e88b8..ef6237c 100644 --- a/nats_streaming/gettingstarted/process-signaling.md +++ b/nats_streaming/gettingstarted/process-signaling.md @@ -8,7 +8,7 @@ On Unix systems, the NATS Streaming Server responds to the following signals: | SIGINT, SIGTERM | Stops the server gracefully | | SIGUSR1 | Reopens the log file for log rotation | -The `nats-streaming-server` binary can be used to send these signals to running NATS Streaming Servers using the `-sl` flag: +The `nats-streaming-server` binary can be used to send these signals to run NATS Streaming Servers using the `-sl` flag: ```sh # Reopen log file for log rotation From bc35b923b14e851905ff50d5c282268c94d7146d Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:27:26 -0500 Subject: [PATCH 14/31] Update windows-service.md --- nats_streaming/gettingstarted/windows-service.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats_streaming/gettingstarted/windows-service.md b/nats_streaming/gettingstarted/windows-service.md index 6a52b78..62f3ff2 100644 --- a/nats_streaming/gettingstarted/windows-service.md +++ b/nats_streaming/gettingstarted/windows-service.md @@ -1,6 +1,6 @@ # Windows Service -The NATS Streaming Server supports running as a Windows service. There is currently no installer and instead users should use `sc.exe` to install the service: +The NATS Streaming Server supports running as a Windows service. There is currently no installer and instead users should use `sc.exe` to install the service. Here is how to create and start a NATS Streaming Server named `nats-streaming-server`. Note that the server flags should be passed in when creating the service. ```sh From 2e16acdbb29b444e6382e6bc7753002a55f2b1b1 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:29:03 -0500 Subject: [PATCH 15/31] Update enabling.md --- nats_streaming/monitoring/enabling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nats_streaming/monitoring/enabling.md b/nats_streaming/monitoring/enabling.md index 675cc58..4dd94bf 100644 --- a/nats_streaming/monitoring/enabling.md +++ b/nats_streaming/monitoring/enabling.md @@ -7,7 +7,7 @@ Monitoring options -m, --http_port PORT HTTP PORT for monitoring -ms,--https_port PORT Use HTTPS PORT for monitoring (requires TLS cert and key) ``` -To enable monitoring via the configuration file, use `http: "host:port"` or `https: "host:port"` (there is no explicit configuration flag for the monitoring interface). +To enable monitoring via the configuration file, use `http: "host:port"` or `https: "host:port"`. There is no explicit configuration flag for the monitoring interface. For example, after running this: ``` From e6c60d490f4ff4642ba0a500507e5f4457465874 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:32:34 -0500 Subject: [PATCH 16/31] Update partitioning.md --- nats_streaming/partitioning.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nats_streaming/partitioning.md b/nats_streaming/partitioning.md index 1e6ab4c..be63aed 100644 --- a/nats_streaming/partitioning.md +++ b/nats_streaming/partitioning.md @@ -26,7 +26,7 @@ store_limits: { } ``` -When partitioning is enabled, multiple servers with the same cluster ID can coexist on the same NATS network, each server handling its own set of channels. ***Note however that in this mode, state is not replicated as it is in Clustering mode. The only communication between servers is to report if a given channel is handled in more than one serve.r*** +When partitioning is enabled, multiple servers with the same cluster ID can coexist on the same NATS network, each server handling its own set of channels. ***Note however that in this mode, state is not replicated as it is in Clustering mode. The only communication between servers is to report if a given channel is handled in more than one server.*** ## Wildcards @@ -116,4 +116,4 @@ A published message will be received by only the server that has that channel de However, other client requests (such as connection and subscription requests) are received by all servers. For connections, all servers handle them and the client library will receive a response from all servers in the cluster, but use the first one that it received. -For subscriptions, a server receiving the request for a channel that it does not handle will simply ignore the request. Again, if no server handle this channel, the client's subscription request will simply time out. \ No newline at end of file +For subscriptions, a server receiving the request for a channel that it does not handle will simply ignore the request. Again, if no server handle this channel, the client's subscription request will simply time out. From 7c557a3bf9bab70bd52c5a6790f44b42c1c6695a Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:36:54 -0500 Subject: [PATCH 17/31] Update partitioning.md --- nats_streaming/partitioning.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/nats_streaming/partitioning.md b/nats_streaming/partitioning.md index be63aed..db4676b 100644 --- a/nats_streaming/partitioning.md +++ b/nats_streaming/partitioning.md @@ -98,7 +98,7 @@ host1$ nats-streaming-server -store file -dir /nss/datastore/foodata -sc foo.con host2$ nats-streaming-server -store file -dir /nss/datastore/foodata -sc foo.conf -ft_group_name foo -cluster nats://host2:6222 -routes nats://host1:6222,nats://host1:6223 ``` -Notice that each server on each note points to each other (the `-routes` parameter). The reason why we also point to `6223` will be explained later. They both listen for routes connections on their host's `6222` port. +Notice that each server on each node points to each other (the `-routes` parameter). The reason why we also point to `6223` will be explained later. They both listen for routes connections on their host's `6222` port. We now start the FT pair for `bar`. Since we are running from the same machines (we don't have to), we need to use a different port: ``` @@ -106,7 +106,7 @@ host1$ nats-streaming-server -store file -dir /nss/datastore/bardata -sc bar.con host2$ nats-streaming-server -store file -dir /nss/datastore/bardata -sc bar.conf -ft_group_name bar -p 4223 -cluster nats://host2:6223 -routes nats://host1:6222,nats://host1:6223 ``` -You will notice that the `-routes` parameter points to both `6222` and `6223`, this is so that both partitions belong to the same cluster and be view as "one" by a Streaming application connecting to this cluster. Effectively, we have created a full mesh of 4 NATS servers that can all communicate with each other. Two of these servers are backups for servers running on the same FT group. +You will notice that the `-routes` parameter points to both `6222` and `6223`, this is so that both partitions belong to the same cluster and be viewed as "one" by a Streaming application connecting to this cluster. Effectively, we have created a full mesh of 4 NATS servers that can all communicate with each other. Two of these servers are backups for servers running on the same FT group. ## Applications behavior @@ -114,6 +114,6 @@ When an application connects, it specifies a cluster ID. If several servers are A published message will be received by only the server that has that channel defined. If no server is handling this channel, no specific error is returned, instead the publish call will timeout. Same goes for message acknowledgements. Only the server handling the subscription on this channel should receive those. -However, other client requests (such as connection and subscription requests) are received by all servers. For connections, all servers handle them and the client library will receive a response from all servers in the cluster, but use the first one that it received. +However, other client requests (such as connection and subscription requests) are received by all servers. For connections, all servers handle them and the client library will receive a response from all servers in the cluster, but will use the first one that it received. -For subscriptions, a server receiving the request for a channel that it does not handle will simply ignore the request. Again, if no server handle this channel, the client's subscription request will simply time out. +For subscriptions, a server receiving the request for a channel that it does not handle will simply ignore the request. Again, if no server handles this channel, the client's subscription request will simply time out. From e3befbbbc2d5ed6ee7df7e926e2ee7c02aa3b148 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:39:40 -0500 Subject: [PATCH 18/31] Update store-encryption.md --- nats_streaming/store-encryption.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/nats_streaming/store-encryption.md b/nats_streaming/store-encryption.md index 9dbf744..cec4d2e 100644 --- a/nats_streaming/store-encryption.md +++ b/nats_streaming/store-encryption.md @@ -23,11 +23,8 @@ Note that only message payload is encrypted, all other data stored by NATS Strea When running in clustering mode (see below), the server uses RAFT, which uses its own log files. Those will be encrypted too. -Starting a server with `encrypt` against a datastore that was not encrypted may result in failures when it comes to decrypt a message, which may not happen immediately upon startup. Instead, -it will happen when attempting to deliver messages to consumers. However, when possible, the server will detect if the data was not encrypted and return the data without attempting to decrypt it. -The server will also detect which cipher was used to encrypt the data and use the proper cipher to decrypt, even if this is not the currently selected cipher. +Starting a server with `encrypt` against a datastore that was not encrypted may result in failures when it comes to decrypt a message, which may not happen immediately upon startup. Instead, it will happen when attempting to deliver messages to consumers. However, when possible, the server will detect if the data was not encrypted and return the data without attempting to decrypt it. The server will also detect which cipher was used to encrypt the data and use the proper cipher to decrypt, even if this is not the currently selected cipher. If the data is encrypted with a key and the server is restarted with a different key, the server will fail to decrypt messages when attempting to load them from the store. -Performance considerations: As expected, encryption is likely to decrease performance, but by how much is hard to define. In some performance tests on a MacbookPro 2.8 GHz Intel Core i7 with SSD, we have -observed as little as 1% decrease to more than 30%. In addition to CPU cycles required for encryption, the encrypted payload is bigger, which result in more data being stored or read. +Performance considerations: As expected, encryption is likely to decrease performance, but by how much is hard to define. In some performance tests on a MacbookPro 2.8 GHz Intel Core i7 with SSD, we have observed as little as 1% decrease to more than 30%. In addition to CPU cycles required for encryption, the encrypted payload is bigger, which result in more data being stored or read. From ea1f66152c0c401d10e2be40ce68a880ca1326fb Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 13:44:21 -0500 Subject: [PATCH 19/31] Update receiving.md --- developer/streaming/receiving.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/developer/streaming/receiving.md b/developer/streaming/receiving.md index 7841e6c..f09d2c5 100644 --- a/developer/streaming/receiving.md +++ b/developer/streaming/receiving.md @@ -9,7 +9,7 @@ Subscriptions come in several forms: * Queue * Queue/Durable -For more details on the various types, check the [concept](/nats_streaming/channels/subscriptions/subscriptions.md) section. +For more details on the various types, check the [concepts](/nats_streaming/channels/subscriptions/subscriptions.md) section. ***Note: message callbacks are invoked serially, one message at a time. If your application does not care about processing ordering and would prefer the messages to be dispatched concurrently, it is the application responsibility to move them to some internal queue to be picked up by threads/go routines.*** From dbe28dcae7fca781f8827f9b9a61cee9c4c294a5 Mon Sep 17 00:00:00 2001 From: Ivan Kozlovic Date: Thu, 23 May 2019 13:39:16 -0600 Subject: [PATCH 20/31] Updates based on comments Signed-off-by: Ivan Kozlovic --- developer/streaming/README.md | 8 ++++---- developer/streaming/embedding.md | 6 +++--- developer/streaming/protocol.md | 2 +- nats_streaming/channels/subscriptions/queue-group.md | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/developer/streaming/README.md b/developer/streaming/README.md index 8e4de41..7ada759 100644 --- a/developer/streaming/README.md +++ b/developer/streaming/README.md @@ -1,14 +1,14 @@ # NATS Streaming -Where NATS provides at most once quality of service, streaming adds at least once. Streaming is implemented as a request-reply service on top of NATS. Streaming messages are encoded as protocol buffers, the streaming clients use NATS to talk to the streaming server. The streaming server organizes messages in channels and stores them in files or databases. ACKs are used to insure delivery in both directions. +Where NATS provides at most once quality of service, streaming adds at least once. Streaming is implemented as a request-reply service on top of NATS, where streaming messages are encoded as protocol buffers and the streaming clients use NATS to talk to the streaming server. The streaming server organizes messages in channels and stores them in files or databases, using ACKs to ensure delivery in both directions. > Sometimes the maintainers will refer to NATS as "nats core" and streaming as "stan" or "streaming". Messages to the streaming service are opaque byte arrays, just as they are with NATS. However, the streaming server protocol uses protocol buffers to wrap these byte arrays. So if you listen to the NATS traffic the messages will appear as protocol buffers, while the actual data sent and received will simply be byte arrays. -NATS streaming uses the concept of a channel to represent an ordered collection of messages. Clients send to and receive from channels instead of subjects. The subjects used by the streaming libraries and server are managed internally. Channels do not currently support wildcard. Channels aren’t raw subjects. Streaming isn’t raw NATS. The streaming libraries hide some of the differences. +NATS streaming uses the concept of a channel to represent an ordered collection of messages. Clients send to and receive from channels instead of subjects. The subjects used by the streaming libraries and server are managed internally. Channels do not currently support wildcards. Channels aren’t raw subjects. Streaming isn’t raw NATS. The streaming libraries hide some of the differences. -Think of channels as a First In First Out (FIFO) queue. Messages are added until the configured limit is reached. Old messages are removed to make room for new ones. Old messages can expire, based on configuration. Subscriptions don’t affect channel content, that is, when a message is acknowledged, it is not removed from the channel. +Think of channels as a First In First Out (FIFO) queue. Messages are added until the configured limit is reached. Old messages can be set to expire based on configuration, making room for new messages. Subscriptions don’t affect channel content, that is, when a message is acknowledged, it is not removed from the channel. Positions in the channel are specified in multiple ways: @@ -16,7 +16,7 @@ Positions in the channel are specified in multiple ways: * Time * Time delta (converted to time on client) -New subscriptions can also specify last received to indicate they only want new messages. Sequence numbers are persistent, when message #1 goes away the oldest message is message #2. Trying to go to a position before the oldest message will be moved to the oldest message. +New subscriptions can also specify last received to indicate they only want new messages. Sequence numbers are persistent so when message #1 goes away, the oldest message is then message #2. If you try to go to a position before the oldest message, you will be moved to the oldest message. ## Subscription Types diff --git a/developer/streaming/embedding.md b/developer/streaming/embedding.md index 5fc5f4c..3ab441e 100644 --- a/developer/streaming/embedding.md +++ b/developer/streaming/embedding.md @@ -31,7 +31,7 @@ Then get the default options and override some of them: s, err := stand.RunServerWithOpts(opts, nil) ``` -However, since the NATS Streaming Server project vendors NATS Server (that it uses as the communication layer with its clients and other servers in the cluster, there are some limitations. +However, since the NATS Streaming Server project vendors NATS Server that is uses as the communication layer with its clients and other servers in the cluster, there are some limitations. If you were to import `github.com/nats-io/nats-server/server`, instantiate a NATS `Options` structure, configure it and pass it to the second argument of `RunServerWithOpts`, you would get a compiler error. For instance doing this does not work: @@ -67,7 +67,7 @@ To workaround this issue, the NATS Streaming Server package provides a function That will work. -But, if you want to do advanced NATS configuration that requires types or interfaces that belong to the NATS Server package, then this approach won't work. In this case you need to run the NATS Server indepently and have the NATS Streaming Server connects to it. Here is how: +But, if you want to do advanced NATS configuration that requires types or interfaces that belong to the NATS Server package, then this approach won't work. In this case, you need to run the NATS Server independently and have the NATS Streaming Server connect to it. ``` // This configure the NATS Server using natsd package @@ -108,4 +108,4 @@ But, if you want to do advanced NATS configuration that requires types or interf } ``` -The above seem involved, but it really only if you use very advanced NATS Server options. \ No newline at end of file +The above process may seem involved, but only if you use very advanced NATS Server options. \ No newline at end of file diff --git a/developer/streaming/protocol.md b/developer/streaming/protocol.md index a964caa..b9901e4 100644 --- a/developer/streaming/protocol.md +++ b/developer/streaming/protocol.md @@ -1,6 +1,6 @@ # Writing your own client library -You can find a list of all supported client libraries [here](https://nats.io/download/). There is also links to community contributed clients. +You can find a list of all supported client libraries [here](https://nats.io/download/). There are also links to community contributed clients. In the event you would want to write your own NATS Streaming library, you could have a look at existing libraries to understand the flow. But you need to use [Google Protocol Buffers](https://developers.google.com/protocol-buffers/) to exchange protocols between the client and the server. diff --git a/nats_streaming/channels/subscriptions/queue-group.md b/nats_streaming/channels/subscriptions/queue-group.md index 527b8ae..1a026e1 100644 --- a/nats_streaming/channels/subscriptions/queue-group.md +++ b/nats_streaming/channels/subscriptions/queue-group.md @@ -6,6 +6,6 @@ After the first queue member is created, any other member joining the group will When the last member of the group leaves (subscription unsubscribed/closed/or connection closed), the group is removed from the server. The next application creating a subscription with the same name will create a new group, starting at the start position given in the subscription request. -A queue subscription can also be durable. For that, the client needs to provide a queue and durable name. The behavior is, as you would expect, a combination of queue and durable subscription. Unlike a durable subscription, though, the client ID is not part of the queue group name. It makes sense, because since client ID must be unique, it would prevent more than one connection to participate in the queue group. The main difference between a queue subscription and a durable one, is that when the last member leaves the group, the state of the group will be maintained by the server. Later, when a member rejoins the group, the delivery will resume. +A queue subscription can also be durable. For that, the client needs to provide a queue and durable name. The behavior is, as you would expect, a combination of queue and durable subscription. Though unlike a durable subscription, the client ID is not part of the queue group name since the client ID must be unique, and would prevent more than one connection to participate in the queue group. The main difference between a queue subscription and a durable one, is that when the last member leaves the group, the state of the group will be maintained by the server. Later, when a member rejoins the group, the delivery will resume. ***Note: For a durable queue subscription, the last member to * unsubscribe * (not simply close) causes the group to be removed from the server.*** From 38c105ebf3abc42fca4e55b246596c074da0ab4c Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Thu, 23 May 2019 14:28:25 -0600 Subject: [PATCH 21/31] Updates based on comments Signed-off-by: Colin Sullivan --- README.md | 28 +++++++++++++++++-------- whats_new/whats_new_20.md | 44 ++++++++++++++++++++++----------------- 2 files changed, 44 insertions(+), 28 deletions(-) diff --git a/README.md b/README.md index aaaa734..5b046f5 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,21 @@ # Introduction -## Why Messaging is Important +## The Importance of Messaging -Developing and deploying applications that communicate in distributed systems is complex and difficult. A communication infrastructure should provide features to make this easier, including multiple messaging patterns bundled into one technology, location transparency, the decoupling of data producers and consumers, and asynchronous communications to build event driven applications. +Developing and deploying applications that communicate in distributed systems +can be complex and difficult. A communication infrastructure should provide +features to make this easier, including multiple messaging patterns bundled +into one technology, location transparency, the decoupling of data producers +and consumers, and asynchronous communications to build event driven +applications. This can drive services/micro-services based workloads and take +the place of things like a service mesh, and also drive observability, all +from the same technology. ### Distributed Computing Needs of Today -A modern messaging system needs to support multiple communication patterns, be secure by default, support multiple qualities of service, and provide secure multi-tenancy for shared infrastructure. A modern system needs to include: +A modern messaging system needs to support multiple communication patterns, be +secure by default, support multiple qualities of service, and provide secure +multi-tenancy for a truly shared infrastructure. A modern system needs to include: * Secure by default communications for microservices, edge platforms and devices * Secure multi-tenancy in a single distributed communication technology @@ -18,23 +27,24 @@ A modern messaging system needs to support multiple communication patterns, be s ## NATS -NATS was built to meet the distributed computing needs of today. NATS is -simple and secure messaging made for developers and operators who want to -spend more time developing modern applications than worrying about a -distributed communication system. +NATS was built to meet the distributed computing needs of today and tomorrow. +NATS is simple and secure messaging made for developers and operators who want +to spend more time developing modern applications and services than worrying +about a distributed communication system. * Easy to use for developers and operators * High-Performance * Always on and available * Extremely lightweight * At Most Once (NATS) or At Least Once Delivery (NATS Streaming) -* Common Messaging Pattern Support (Services, Streams, Load Balancing) +* Common Messaging Pattern Support (Scalable Services, Event/Data Streams) * Client support for over 30 different programming languages * Cloud Native, a CNCF project with Kubernetes and Prometheus integrations ### Use Cases -NATS can run anywhere, from devices, to edge computers, to cloud. Use cases NATS include: +NATS can run anywhere, from large servers and cloud instances, through edge +gateways and even IoT devices. Use cases NATS include: * Cloud Messaging * Services (microservices, service mesh) diff --git a/whats_new/whats_new_20.md b/whats_new/whats_new_20.md index d26e70a..0544e65 100644 --- a/whats_new/whats_new_20.md +++ b/whats_new/whats_new_20.md @@ -16,9 +16,9 @@ as they scale upward. Problems arise around service discovery, connectivity, scaling for volume, and application onboarding and updates. Disaster recovery is difficult, especially as systems have evolved to operate in silos defined by technology rather than business needs. -As complexity increases, systems become expensive to operate. The become -fragile making it difficult to deploy services and applications hindering -innovation. +As complexity increases, systems become expensive to operate in terms of time +and money. They become fragile making it difficult to deploy services and +applications hindering innovation, time to value, and total cost of ownership. We decided to: @@ -28,7 +28,7 @@ can operate at global scale with simple configuration and a resilient and cloud-native architecture. * __Decrease Time to Value__: As systems scale, _time to value_ increases. Operations resist change due to risk in touching a complex and fragile -system. Providing isolation contexts can mitigate this. +system. Providing isolation contexts can help mitigate this. * __Support manageable large scale deployments__: No data silos defined by software, instead easily managed though software to provide exactly what the business needs. We wanted to provide easy to configure disaster recovery. @@ -47,9 +47,10 @@ business driven use cases, where data silos are created by design, not software limitations. When a client connects, it specifies an account or will default to authentication with a global account. -At least some applications need to share data outside of their account. +At least some services need to share data outside of their account. Data can be securely shared between accounts with secure services and -streams, where only mutual agreement between account owners permit data flow. +streams. Only mutual agreement between account owners permit data flow, +and the import account has complete control over its own subject space. This means within an account, limitations may be set and subjects can be used without worry of collisions with other groups or organizations. Development @@ -62,16 +63,16 @@ autonomy reducing time to value with faster, more agile development practices. ### Service and Streams -Services and streams are mechanisms to share data between accounts. +Services and streams are mechanisms to share messages between accounts. -Think of a service as a RPC endpoint into an account. Behind that account +Think of a service as an RPC endpoint into an account. Behind that account there might be many microservices working in concert to handle requests, but from outside the account there simply one subject exposed. __Services__ definition to share an endpoint: * Export a service to allow other accounts to import -* Import a service to allow requests to be sent and securely and seamlessly to +* Import a service to allow requests to be sent securely and seamlessly to another account Use cases include most applications - anything that accepts a request and returns @@ -132,9 +133,9 @@ for seamless rolling upgrades and scaling up or down. ### Superclusters -Conceptually, superclusters are clusters of NATS server clusters. Create +Conceptually, superclusters are clusters of NATS clusters. Create superclusters to deploy a truly global NATS network. Superclusters use -a novel spline based technology with a unique apporoach to toplology, keeping +a novel spline based technology with a unique apporoach to topology, keeping one hop semantics and optimizing WAN traffic through optimistic sends with interest graph pruning. Superclusters provide transparent, intelligent support for geo-distributed queue subscribers. @@ -157,7 +158,7 @@ clients in US-EAST will begin using services in EU-WEST. Once the Eastern US services have reconnected to US-EAST, those services will immediately begin servicing the Eastern US clients since they're local to -the NATS cluster. This is automatic and entirely transparent to the client. +the NATS cluster. This is automatic and entirely transparent to the client. There is no extra configuration in NATS servers. This is __zero configuration disaster recovery__. @@ -167,9 +168,11 @@ This is __zero configuration disaster recovery__. Leaf nodes are a NATS servers running in a special configuration, allowing hub and spoke topologies to extend superclusters. -Leaf can bridge separate security domains. e.g. IoT, mobile, web. They are -ideal for edge computing, IoT hubs, or data centers that need to be connected -to a global NATS deployment. +Leaf nodes can also bridge separate security domains. e.g. IoT, mobile, web. +They are ideal for edge computing, IoT hubs, or data centers that need to be +connected to a global NATS deployment. Local applications that communicate +using the loopback interface with physical VM or Container security can +leverage leaf nodes as well. Leaf nodes: @@ -189,9 +192,10 @@ within a NATS deployment. * An __Operator__ provides the root of trust for the system, may represent a company or enterprise * Creates __Accounts__ for account administrators. An account represents -an organization with a secure context within the NATS deployment, for example -an IT system monitoring group, a set of microservices, a regional IoT -deployment. Account creation would likely be managed by a central group. +an organization, business unit, or service offering with a secure context +within the NATS deployment, for example an IT system monitoring group, a +set of microservices, or a regional IoT deployment. Account creation +would likely be managed by a central group. * __Accounts__ define limits and may securely expose services and streams. * Account managers create __Users__ with permissions * __Users__ have specific credentials and permissions. @@ -203,7 +207,9 @@ create a hierarchy of Operators, Accounts, and Users creating a scalable and flexible distributed security mechanism. * __Operators__ are represented by a self signed JWT and is the only thing that -is required to be configured in the server. +is required to be configured in the server. This JWT is usually signed by a +master key that is kept offline. The JWT will contain valid signing keys that +can be revoked with the master updating this JWT. * Operators will sign __Account__ JWTs with various signing keys. * __Accounts__ sign __User__ JWTs, again with various signing keys. * Clients or leaf nodes present __User__ credentials and a signed nonce when connecting. From e4b06a416b447b16a20ea776202f9efca36b70fb Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 15:45:22 -0500 Subject: [PATCH 22/31] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5b046f5..587fb81 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ about a distributed communication system. ### Use Cases NATS can run anywhere, from large servers and cloud instances, through edge -gateways and even IoT devices. Use cases NATS include: +gateways and even IoT devices. Use cases for NATS include: * Cloud Messaging * Services (microservices, service mesh) From d28b9333fcdeefe624ca83a4e4630826e9017f20 Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 15:58:23 -0500 Subject: [PATCH 23/31] Update whats_new_20.md --- whats_new/whats_new_20.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/whats_new/whats_new_20.md b/whats_new/whats_new_20.md index 0544e65..86dfe76 100644 --- a/whats_new/whats_new_20.md +++ b/whats_new/whats_new_20.md @@ -30,10 +30,10 @@ and cloud-native architecture. Operations resist change due to risk in touching a complex and fragile system. Providing isolation contexts can help mitigate this. * __Support manageable large scale deployments__: No data silos defined by -software, instead easily managed though software to provide exactly what the +software, instead easily managed through software to provide exactly what the business needs. We wanted to provide easy to configure disaster recovery. * __Decentralize security__: Provide security supporting one -technology end-to-end where organizations may self-manage will make it +technology end-to-end where organizations may self-manage making it easier to support a massive number of endpoints. To achieve this, we added a number of new features that are transparent @@ -216,4 +216,4 @@ can be revoked with the master updating this JWT. * The server uses resolvers to obtain JWTs and verify the client trust chain. This allows for rapid change of permissions, authentication and limits, to a -the secure multi-tenant NATS system. \ No newline at end of file +the secure multi-tenant NATS system. From 5b3bbacd49b5811991ab45f9b8715689426f05ab Mon Sep 17 00:00:00 2001 From: Ginger Collison Date: Thu, 23 May 2019 16:09:27 -0500 Subject: [PATCH 24/31] Update whats_new_20.md --- whats_new/whats_new_20.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/whats_new/whats_new_20.md b/whats_new/whats_new_20.md index 86dfe76..71fd312 100644 --- a/whats_new/whats_new_20.md +++ b/whats_new/whats_new_20.md @@ -67,7 +67,7 @@ Services and streams are mechanisms to share messages between accounts. Think of a service as an RPC endpoint into an account. Behind that account there might be many microservices working in concert to handle requests, but -from outside the account there simply one subject exposed. +from outside the account there is simply one subject exposed. __Services__ definition to share an endpoint: From f2ed46c938e84142b1b35b5626fbfb66fe532422 Mon Sep 17 00:00:00 2001 From: Colin Sullivan Date: Thu, 23 May 2019 16:21:08 -0600 Subject: [PATCH 25/31] Updates based on comments --- whats_new/whats_new_20.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/whats_new/whats_new_20.md b/whats_new/whats_new_20.md index 71fd312..80ff054 100644 --- a/whats_new/whats_new_20.md +++ b/whats_new/whats_new_20.md @@ -18,7 +18,8 @@ Disaster recovery is difficult, especially as systems have evolved to operate in silos defined by technology rather than business needs. As complexity increases, systems become expensive to operate in terms of time and money. They become fragile making it difficult to deploy services and -applications hindering innovation, time to value, and total cost of ownership. +applications hindering innovation, increasing time to value and total cost +of ownership. We decided to: From 4f0f23c730ecc9d172604c3318c919f6315b3402 Mon Sep 17 00:00:00 2001 From: ainsley Date: Fri, 24 May 2019 13:36:32 -0500 Subject: [PATCH 26/31] add in anchor plugin --- book.json | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/book.json b/book.json index b9ab25f..d7cdf3a 100644 --- a/book.json +++ b/book.json @@ -1,19 +1,20 @@ { - "title" : "NATS", - "description": "Administrative, developer and conceptual documentation for the NATS messaging system.", - "author": "The NATS Maintainers", - "plugins" : [ - "prism", - "-highlight", - "include-html", - "toggle-chapters" - ], - "pluginsConfig": { - "prism": { - "lang": { - "ascii": "markup", - "text": "markup" - } - } + "title" : "NATS", + "description": "Administrative, developer and conceptual documentation for the NATS messaging system.", + "author": "The NATS Maintainers", + "plugins" : [ + "prism", + "-highlight", + "include-html", + "toggle-chapters", + "anchors" + ], + "pluginsConfig": { + "prism": { + "lang": { + "ascii": "markup", + "text": "markup" } + } + } } \ No newline at end of file From 6e08f30e9d92afbfba3f1c1e5077e1b7ec6046f7 Mon Sep 17 00:00:00 2001 From: ainsley Date: Fri, 24 May 2019 14:02:34 -0500 Subject: [PATCH 27/31] updating docs --- docs/developer/concepts/acks.html | 2233 +++++++++------ docs/developer/concepts/intro.html | 2235 +++++++++------ docs/developer/concepts/pubsub.html | 2233 +++++++++------ docs/developer/concepts/queue.html | 2233 +++++++++------ docs/developer/concepts/reqreply.html | 2233 +++++++++------ docs/developer/concepts/seq_num.html | 2233 +++++++++------ docs/developer/concepts/subjects.html | 2251 +++++++++------ docs/developer/connecting/cluster.html | 2233 +++++++++------ .../developer/connecting/connect_timeout.html | 2233 +++++++++------ docs/developer/connecting/default_server.html | 2235 +++++++++------ docs/developer/connecting/intro.html | 2233 +++++++++------ docs/developer/connecting/noecho.html | 2235 +++++++++------ docs/developer/connecting/pingpong.html | 2239 +++++++++------ docs/developer/connecting/protocol.html | 2241 +++++++++------ .../developer/connecting/specific_server.html | 2233 +++++++++------ docs/developer/events/events.html | 2237 +++++++++------ docs/developer/events/intro.html | 2233 +++++++++------ docs/developer/events/slow.html | 2237 +++++++++------ docs/developer/index.html | 2237 +++++++++------ docs/developer/receiving/async.html | 2233 +++++++++------ docs/developer/receiving/drain.html | 2233 +++++++++------ docs/developer/receiving/intro.html | 2233 +++++++++------ docs/developer/receiving/queues.html | 2235 +++++++++------ docs/developer/receiving/reply.html | 2233 +++++++++------ docs/developer/receiving/structure.html | 2233 +++++++++------ docs/developer/receiving/sync.html | 2233 +++++++++------ docs/developer/receiving/unsub_after.html | 2233 +++++++++------ docs/developer/receiving/unsubscribing.html | 2233 +++++++++------ docs/developer/receiving/wildcards.html | 2233 +++++++++------ docs/developer/reconnect/buffer.html | 2233 +++++++++------ docs/developer/reconnect/disable.html | 2233 +++++++++------ docs/developer/reconnect/events.html | 2233 +++++++++------ docs/developer/reconnect/intro.html | 2233 +++++++++------ docs/developer/reconnect/max.html | 2233 +++++++++------ docs/developer/reconnect/random.html | 2233 +++++++++------ docs/developer/reconnect/wait.html | 2233 +++++++++------ docs/developer/resources/ring_buffer.png | Bin 25282 -> 0 bytes docs/developer/resources/start_positions.png | Bin 23794 -> 0 bytes docs/developer/security/creds.html | 2233 +++++++++------ docs/developer/security/intro.html | 2233 +++++++++------ docs/developer/security/nkey.html | 2233 +++++++++------ docs/developer/security/tls.html | 2243 +++++++++------ docs/developer/security/token.html | 2237 +++++++++------ docs/developer/security/userpass.html | 2243 +++++++++------ docs/developer/sending/caches.html | 2235 +++++++++------ docs/developer/sending/intro.html | 2233 +++++++++------ docs/developer/sending/replyto.html | 2233 +++++++++------ docs/developer/sending/request_reply.html | 2235 +++++++++------ docs/developer/sending/structure.html | 2233 +++++++++------ docs/developer/streaming/acks.html | 2239 +++++++++------ docs/developer/streaming/connecting.html | 2243 +++++++++------ docs/developer/streaming/durables.html | 2238 +++++++++------ docs/developer/streaming/index.html | 2253 +++++++++------ docs/developer/streaming/publishing.html | 2238 +++++++++------ docs/developer/streaming/queues.html | 2258 +++++++++------ docs/developer/streaming/receiving.html | 2248 +++++++++------ docs/developer/tutorials/custom_dialer.html | 2237 +++++++++------ docs/developer/tutorials/intro.html | 2233 +++++++++------ docs/developer/tutorials/pubsub.html | 2289 +++++++++------ docs/developer/tutorials/queues.html | 2251 +++++++++------ docs/developer/tutorials/reqreply.html | 2247 +++++++++------ docs/gateways/gateway.html | 2237 +++++++++------ docs/gateways/index.html | 2245 +++++++++------ docs/index.html | 2285 +++++++++------ docs/leafnodes/index.html | 2237 +++++++++------ docs/leafnodes/leafnode_conf.html | 2237 +++++++++------ docs/nats_admin/index.html | 2233 +++++++++------ docs/nats_admin/signals.html | 2233 +++++++++------ docs/nats_admin/slow_consumers.html | 2245 +++++++++------ docs/nats_admin/upgrading_cluster.html | 2241 +++++++++------ docs/nats_docker/docker_swarm.html | 2243 +++++++++------ docs/nats_docker/index.html | 2251 +++++++++------ docs/nats_docker/tutorial.html | 2233 +++++++++------ docs/nats_protocol/nats-client-dev.html | 2255 +++++++++------ docs/nats_protocol/nats-protocol-demo.html | 2239 +++++++++------ docs/nats_protocol/nats-protocol.html | 2301 +++++++++------ docs/nats_protocol/nats-server-protocol.html | 2297 +++++++++------ docs/nats_server/auth_intro.html | 2237 +++++++++------ docs/nats_server/auth_timeout.html | 2233 +++++++++------ docs/nats_server/authorization.html | 2239 +++++++++------ docs/nats_server/clients.html | 2241 +++++++++------ docs/nats_server/cluster_config.html | 2233 +++++++++------ docs/nats_server/cluster_tls.html | 2233 +++++++++------ docs/nats_server/clustering.html | 2245 +++++++++------ docs/nats_server/configuration.html | 2243 +++++++++------ docs/nats_server/flags.html | 2245 +++++++++------ docs/nats_server/installation.html | 2245 +++++++++------ docs/nats_server/jwt_auth.html | 2247 +++++++++------ docs/nats_server/logging.html | 2249 +++++++++------ docs/nats_server/monitoring.html | 2247 +++++++++------ docs/nats_server/nkey_auth.html | 2237 +++++++++------ docs/nats_server/running.html | 2237 +++++++++------ docs/nats_server/securing_nats.html | 2233 +++++++++------ docs/nats_server/tls.html | 2235 +++++++++------ docs/nats_server/tls_mutual_auth.html | 2239 +++++++++------ docs/nats_server/tokens.html | 2235 +++++++++------ docs/nats_server/username_password.html | 2237 +++++++++------ docs/nats_server/windows_srv.html | 2233 +++++++++------ .../nats-streaming-install.html | 2397 ---------------- docs/nats_streaming/nats-streaming-intro.html | 2388 ---------------- .../nats-streaming-protocol.html | 2525 ----------------- .../nats-streaming-quickstart.md | 82 - docs/nats_streaming/nats-streaming-swarm.html | 2381 ---------------- docs/nats_streaming/nats-streaming-tls.html | 2354 --------------- docs/nats_tools/mkpasswd.html | 2241 +++++++++------ docs/nats_tools/nas/index.html | 2235 +++++++++------ docs/nats_tools/nas/inspecting_jwts.html | 2233 +++++++++------ docs/nats_tools/nas/mem_resolver.html | 2237 +++++++++------ docs/nats_tools/nas/nas_conf.html | 2251 +++++++++------ docs/nats_tools/nats_top/index.html | 2249 +++++++++------ docs/nats_tools/nats_top/tutorial.html | 2255 +++++++++------ docs/nats_tools/natsbench.html | 2251 +++++++++------ docs/nats_tools/nk.html | 2237 +++++++++------ docs/nats_tools/nsc/index.html | 2239 +++++++++------ docs/nats_tools/nsc/nsc.html | 2245 +++++++++------ docs/nats_tools/nsc/services.html | 2249 +++++++++------ docs/nats_tools/nsc/signing_keys.html | 2233 +++++++++------ docs/nats_tools/nsc/streams.html | 2247 +++++++++------ docs/search_index.json | 2 +- docs/sys_accounts/index.html | 2235 +++++++++------ docs/sys_accounts/sys_accounts.html | 2249 +++++++++------ docs/whats_new/whats_new_20.html | 2459 ++++++++++------ 122 files changed, 156245 insertions(+), 109269 deletions(-) delete mode 100644 docs/developer/resources/ring_buffer.png delete mode 100644 docs/developer/resources/start_positions.png delete mode 100644 docs/nats_streaming/nats-streaming-install.html delete mode 100644 docs/nats_streaming/nats-streaming-intro.html delete mode 100644 docs/nats_streaming/nats-streaming-protocol.html delete mode 100644 docs/nats_streaming/nats-streaming-quickstart.md delete mode 100644 docs/nats_streaming/nats-streaming-swarm.html delete mode 100644 docs/nats_streaming/nats-streaming-tls.html diff --git a/docs/developer/concepts/acks.html b/docs/developer/concepts/acks.html index 45c1c70..1518f97 100644 --- a/docs/developer/concepts/acks.html +++ b/docs/developer/concepts/acks.html @@ -25,6 +25,10 @@ + + + + @@ -126,11 +130,742 @@ +
  • Developing With NATS
    • + + + +
  • + + + + + Introduction + + + + + +
    • + +
  • + + + + + Concepts + + + + + + + +
    • + +
  • + + + + + Connecting + + + + + + + +
    • + +
  • + + + + + Automatic Reconnections + + + + + + + +
    • + +
  • + + + + + Securing Connections + + + + + + + +
    • + +
  • + + + + + Receiving Messages + + + + + + + +
    • + +
  • + + + + + Sending Messages + + + + + + + +
    • + +
  • + + + + + Monitoring the Connection + + + + + + + +
    • + +
  • + + + + + Tutorials + + + + + + + +
    • + + + +
  • NATS Server
    • -
  • +
  • @@ -143,7 +878,7 @@
    • -
  • +
  • @@ -157,7 +892,7 @@