added gateways, leafnodes, nats-account-server, fixed typos etc.

2025-01-18 04:03:23 -08:00 · 2019-05-20 09:15:03 -05:00 · 2019-05-20 09:15:03 -05:00 · b33bd52668
commit b33bd52668
parent e832940daf
19 changed files with 1212 additions and 8 deletions
--- a/SUMMARY.md
+++ b/SUMMARY.md
@ -21,13 +21,19 @@
    * [Authorization](nats_server/authorization.md)
  * [Clustering](nats_server/clustering.md)
    * [TLS Authentication](nats_server/cluster_tls.md)
+  * [Gateways](gateways/README.md)
+    * [Configuration](gateways/gateway.md)
+  * [Leafnodes](leafnodes/README.md)
+    * [Configuration](leafnodes/leafnode_conf.md)
  * [Logging](nats_server/logging.md)
  * [Monitoring](nats_server/monitoring.md)

-### Managing A NATS Server
+### Managing A NATS Server  
  * [Upgrading a Cluster](nats_admin/upgrading_cluster.md)
  * [Slow Consumers](nats_admin/slow_consumers.md)
  * [Signals](nats_admin/signals.md)
+  * [System Accounts](sys_accounts/README.md)
+    * [Configuration](sys_accounts/sysaccounts.md)    

 ### NATS Tools
 * [mkpasswd](nats_tools/mkpasswd.md)
@ -37,6 +43,10 @@
  * [Streams](nats_tools/nsc/streams.md)
  * [Services](nats_tools/nsc/services.md)
  * [Signing Keys](nats_tools/nsc/signing_keys.md)
+* [nats account server](nats_tools/nas/README.md)
+  * [Basics](nats_tools/nas/nas_conf.md)
+  * [Inspecting JWTs](nats_tools/nas/inspecting_jwts.md)
+  * [Memory Resolver](nats_tools/nas/mem_resolver.md)
 * [NATS Top](nats_tools/nats_top/README.md)
  * [Tutorial](nats_tools/nats_top/tutorial.md)
 * [Benchmarking](nats_tools/natsbench.md)
--- a/gateways/README.md
+++ b/gateways/README.md
@ -0,0 +1,76 @@
+## Gateways
+
+Gateways enable connecting one or more clusters together; they allow the formation of super clusters from smaller clusters. Cluster and Gateway protocols listen in different ports. Clustering is used for adjacent servers; gateways are for joining clusters together. Typically all cluster nodes will also be gateway nodes, but this is not a requirement.
+
+Gateway configuration is similar to clustering:
+
+- gateways have a dedicated port where they listen for gateway requests
+- gateways gossip gateway members and remote discovered gateways
+
+Unlike clusters, gateways:
+
+- don't form a full mesh
+- are bound by uni-directional connections
+
+Gateways exist to:
+
+- reduce the number of connections required between servers 
+- optimize the interest graph propagation
+
+## Gateway Connections
+
+A nats-server in a gateway role will specify a port where it will accept gateways connections. If the configuration specifies other _external_ `gateways`, the gateway will create one outbound gateway connection for each gateway in its configuration. It will also gossip other gateways it knows or discovered. 
+
+If the local cluster has three gateway nodes, this means there will be three outbound connections to each external gateway.
+
+![Gateway Connections](simple.svg)
+
+> In the example above cluster _A_ has configured gateway connections for _B_ (solid lines). B has discovered gateway connections to _A_ (dotted lines). Note that the number of outgoing connections always matches the number of gateways with the same name.
+
+![Gateway Discovered Gateways](three_gw.svg)
+
+> In this second example, again configured connections are shown with solid lines and discovered gateway connections are shown using dotted lines. Gateways _A_ and _C_ were both discovered via gossiping; _B_ discovered _A_ and _A_ discovered _C_.
+
+
+A key point in the description above is that each node in the cluster will make a connection to a single node in the remote cluster — a difference from the clustering protocol, where every node is directly connected to all other nodes.
+
+For those mathematically inclined, cluster connections are `N(N-1)/2` where _N_ is the number of nodes in the cluster. On gateway configurations, outbound connections are the summation of `Ni(M-1)` where Ni is the number of nodes in a gateway _i_, and _M_ is the total number of gateways. Inbound connections are the summation of `U-Ni` where U is the sum of all gateway nodes in all gateways, and N is the number of nodes in a gateway _i_. It works out that both inbound and outbound connection counts are the same.
+
+The number of connections required to join clusters using clustering vs. gateways is apparent very quickly. For 3 clusters, with N nodes:
+
+| Nodes per Cluster | Full Mesh Conns | Gateway Conns |
+| ---: | ----: | ----: |
+| 1 | 3 | 6|
+| 2 | 15 | 12 |
+| 3 | 36 | 18 |
+| 4 | 66 | 24 |
+| 5 | 105 | 30 |
+| 30 | 4005 | 180 |
+
+## Interest Propagation
+
+Gateways propagate interest using three different mechanisms:
+
+- Optimistic Mode
+- Interest-only Mode
+- Queue Subscriptions
+
+### Optimistic Mode
+
+When a publisher in _A_ publishes "foo", the _A_ gateway will check if cluster _B_ has registered _no_ interest in "foo". If not, it forwards "foo" to _B_. If upon receiving "foo", _B_ has no subscribers on "foo", _B_ will send a gateway protocol message to _A_ expressing that it has no interest on "foo", preventing future messages on "foo" from being forwarded.
+
+Should a subscriber on _B_ create a subscription to "foo", _B_ knowing that it had previously rejected interest on _foo_, will send a gateway protocol message to cancel its previous _no interest_ on "foo" in _A_.
+
+### Interest-only Mode
+
+When a gateway on _A_ sends many messages on various subjects for which _B_ has no interest. _B_ sends a gateway protocol message for _A_ to stop sending optimistically, and instead send if there's known interest in the subject. As subscriptions come and go on _B_, _B_ will update its subject interest with _A_.
+
+### Queue Subscriptions
+
+When a queue subscriber creates a new subscription, the gateway propagates the subscription interest to other gateways. The subscription interest is only propagated _once_ per _Account_ and subject. When the last queue subscriber is gone, the cluster interest is removed.
+
+Queue subscriptions work on _Interest-only Mode_ to honor NATS' queue semantics across the _Super Cluster_. For each queue group, a message is only delivered to a single queue subscriber. Only when a local queue group member is not found, is a message forwarded to a different interested cluster; gateways will always try to serve local queue subscribers first and only failover when a local queue subscriber is not found.
+
+### Gateway Configuration
+
+The [Gateway Configuration](gateway.md) document describes all the options available to gateways.
--- a/gateways/gateway.md
+++ b/gateways/gateway.md
@ -0,0 +1,100 @@
+
+## Gateway Configuration
+
+The `gateway` configuration block is similar to a `cluster` block:
+
+```yaml
+gateway {
+	name: "A"
+	listen: "localhost:7222"
+	authorization {
+		user: gwu
+		password: gwp
+	}
+
+	gateways: [
+		{name: "A", url: "nats-gateway://gwu:gwp@localhost:7222"},
+		{name: "B", url: "nats-gateway://gwu:gwp@localhost:7333"},
+		{name: "C", url: "nats-gateway://gwu:gwp@localhost:7444"},
+	]
+}
+```
+
+One difference is that instead of `routes` you specify `gateways`. As expected _self-gateway_ connections are ignored, so you can share gateway configurations with minimal fuzz.
+
+Starting a server: 
+```text
+> nats-server -c A.conf
+[85803] 2019/05/07 10:50:55.902474 [INF] Starting nats-server version 2.0.0-RC11
+[85803] 2019/05/07 10:50:55.902547 [INF] Git commit [not set]
+[85803] 2019/05/07 10:50:55.903669 [INF] Gateway name is A
+[85803] 2019/05/07 10:50:55.903684 [INF] Listening for gateways connections on localhost:7222
+[85803] 2019/05/07 10:50:55.903696 [INF] Address for gateway "A" is localhost:7222
+[85803] 2019/05/07 10:50:55.903909 [INF] Listening for client connections on 0.0.0.0:4222
+[85803] 2019/05/07 10:50:55.903914 [INF] Server id is NBHUDBF3TVJSWCDPG2HSKI4I2SBSPDTNYEXEMOFAZUZYXVA2IYRUGPZU
+[85803] 2019/05/07 10:50:55.903917 [INF] Server is ready
+[85803] 2019/05/07 10:50:56.830669 [INF] 127.0.0.1:50892 - gid:2 - Processing inbound gateway connection
+[85803] 2019/05/07 10:50:56.830673 [INF] 127.0.0.1:50891 - gid:1 - Processing inbound gateway connection
+[85803] 2019/05/07 10:50:56.831079 [INF] 127.0.0.1:50892 - gid:2 - Inbound gateway connection from "C" (NBHWDFO3KHANNI6UCEUL27VNWL7NWD2MC4BI4L2C7VVLFBSMZ3CRD7HE) registered
+[85803] 2019/05/07 10:50:56.831211 [INF] 127.0.0.1:50891 - gid:1 - Inbound gateway connection from "B" (ND2UJB3GFUHXOQ2UUMZQGOCL4QVR2LRJODPZH7MIPGLWCQRARJBU27C3) registered
+[85803] 2019/05/07 10:50:56.906103 [INF] Connecting to explicit gateway "B" (localhost:7333) at 127.0.0.1:7333
+[85803] 2019/05/07 10:50:56.906104 [INF] Connecting to explicit gateway "C" (localhost:7444) at 127.0.0.1:7444
+[85803] 2019/05/07 10:50:56.906404 [INF] 127.0.0.1:7333 - gid:3 - Creating outbound gateway connection to "B"
+[85803] 2019/05/07 10:50:56.906444 [INF] 127.0.0.1:7444 - gid:4 - Creating outbound gateway connection to "C"
+[85803] 2019/05/07 10:50:56.906647 [INF] 127.0.0.1:7444 - gid:4 - Outbound gateway connection to "C" (NBHWDFO3KHANNI6UCEUL27VNWL7NWD2MC4BI4L2C7VVLFBSMZ3CRD7HE) registered
+[85803] 2019/05/07 10:50:56.906772 [INF] 127.0.0.1:7333 - gid:3 - Outbound gateway connection to "B" (ND2UJB3GFUHXOQ2UUMZQGOCL4QVR2LRJODPZH7MIPGLWCQRARJBU27C3) registered
+```
+
+Once all the gateways are up, these clusters of one will forward messages as expected:
+```text
+> nats-pub -s localhost:4444 foo bar
+Published [foo] : 'bar'
+
+# On a different session...
+> nats-sub -s localhost:4333 ">"
+Listening on [>]
+[#1] Received on [foo]: 'bar'
+```
+
+### `Gateway` Configuration Block
+
+| Property | Description |
+| :------  | :---- |
+| `advertise` | Hostport `<host>:<port>` to advertise to other gateways. |
+| `authorization` | Authorization block (same as other nats-server `authorization` configuration). |
+| `connect_retries` | Number of times the server will try to connect to a discovered gateway. |
+| `gateways` | List of Gateway entries - see below. |
+| `host` | Interface where the gateway will listen for incomming gateway connections. |
+| `listen` | Combines `host` and `port` as `<host>:<port>` |
+| `name` | Name for this cluster, all gateways belonging to the same cluster, should specify the same name. |
+| `port` | Port where the gateway will listen for incomming gateway connections. |
+| `reject_unknown` | If `true`, gateway will reject connections from gateways that are not configured in `gateways`. |
+| `tls` | TLS configuration block (same as other nats-server `tls` configuration). |
+
+
+
+#### `Gateway` Entry
+
+The `gateways` configuration block is a list of gateway entries with the following properties:
+
+| Property | Description |
+| :------  | :---- |
+| `name` | Gateway name. |
+| `url` | Hostport `<host>:<port>` describing where the remote gateway can be reached. If multiple IPs are returned, one is randomly selected. |
+| `urls` | A list of `url` |
+
+
+
+### `TLS` Configuration Block
+
+| Property | Description |
+| :------  | :---- |
+| `ca_file` | TLS certificate authority file. |
+| `cert_file` | TLS certificate file. |
+| `cipher_suites` | When set, only the specified TLS cipher suites will be allowed. Values must match golang version used to build the server.  |
+| `curve_preferences` | List of TLS cypher curves to use in order. |
+| `insecure` | Skip certificate verfication. |
+| `key_file` | TLS certificate key file. |
+| `timeout` | TLS handshake timeout in fractional seconds. |
+| `verify_and_map` | If `true`, require and verify client certificates and use values map certificate values for authentication purposes. |
+| `verify` | If `true`, require and verify client certificates. |
--- a/gateways/simple.svg
+++ b/gateways/simple.svg
@ -0,0 +1,76 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" version="1.1" viewBox="152 192 358 280" width="358" height="280">
+  <defs>
+    <font-face font-family="Helvetica Neue" font-size="16" panose-1="2 0 5 3 0 0 0 2 0 4" units-per-em="1000" underline-position="-100" underline-thickness="50" slope="0" x-height="517" cap-height="714" ascent="951.9958" descent="-212.99744" font-weight="400">
+      <font-face-src>
+        <font-face-name name="HelveticaNeue"/>
+      </font-face-src>
+    </font-face>
+    <marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" stroke-linejoin="miter" stroke-miterlimit="10" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black">
+      <g>
+        <path d="M 8 0 L 0 -3 L 0 3 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/>
+      </g>
+    </marker>
+  </defs>
+  <metadata> Produced by OmniGraffle 7.10.2 
+    <dc:date>2019-05-07 16:42:18 +0000</dc:date>
+  </metadata>
+  <g id="Canvas_1" fill-opacity="1" stroke="none" stroke-dasharray="none" fill="none" stroke-opacity="1">
+    <title>Canvas 1</title>
+    <rect fill="white" x="152" y="192" width="358" height="280"/>
+    <g id="Canvas_1: Layer 1">
+      <title>Layer 1</title>
+      <g id="Graphic_5">
+        <circle cx="185.75" cy="225.75" r="33.250053130238" fill="white"/>
+        <circle cx="185.75" cy="225.75" r="33.250053130238" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+        <text transform="translate(164.15 215.75)" fill="black">
+          <tspan font-family="Helvetica Neue" font-size="16" font-weight="400" fill="black" x="11.968" y="15">A1</tspan>
+        </text>
+      </g>
+      <g id="Graphic_6">
+        <circle cx="185.75" cy="332.25" r="33.250053130238" fill="white"/>
+        <circle cx="185.75" cy="332.25" r="33.250053130238" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+        <text transform="translate(164.15 322.25)" fill="black">
+          <tspan font-family="Helvetica Neue" font-size="16" font-weight="400" fill="black" x="11.968" y="15">A2</tspan>
+        </text>
+      </g>
+      <g id="Graphic_7">
+        <circle cx="185.75" cy="438.25" r="33.2500531302381" fill="white"/>
+        <circle cx="185.75" cy="438.25" r="33.2500531302381" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+        <text transform="translate(164.15 428.25)" fill="black">
+          <tspan font-family="Helvetica Neue" font-size="16" font-weight="400" fill="black" x="11.968" y="15">A3</tspan>
+        </text>
+      </g>
+      <g id="Graphic_9">
+        <circle cx="476.25" cy="225.75" r="33.250053130238" fill="white"/>
+        <circle cx="476.25" cy="225.75" r="33.250053130238" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+        <text transform="translate(454.65 215.75)" fill="black">
+          <tspan font-family="Helvetica Neue" font-size="16" font-weight="400" fill="black" x="11.672" y="15">B1</tspan>
+        </text>
+      </g>
+      <g id="Graphic_8">
+        <circle cx="476.25" cy="332.25" r="33.250053130238" fill="white"/>
+        <circle cx="476.25" cy="332.25" r="33.250053130238" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+        <text transform="translate(454.65 322.25)" fill="black">
+          <tspan font-family="Helvetica Neue" font-size="16" font-weight="400" fill="black" x="11.672" y="15">B2</tspan>
+        </text>
+      </g>
+      <g id="Line_10">
+        <line x1="219.00002" y1="225.75" x2="433.1" y2="225.75" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+      </g>
+      <g id="Line_11">
+        <line x1="216.9757" y1="320.80236" x2="435.72924" y2="240.6053" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+      </g>
+      <g id="Line_12">
+        <line x1="216.99302" y1="426.8498" x2="435.70677" y2="347.04374" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+      </g>
+      <g id="Line_14">
+        <line x1="449.4098" y1="245.38355" x2="220.58062" y2="412.7715" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="1.0,5.0" stroke-width="1"/>
+      </g>
+      <g id="Line_15">
+        <line x1="443" y1="332.25" x2="228.90002" y2="332.25" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="1.0,5.0" stroke-width="1"/>
+      </g>
+    </g>
+  </g>
+</svg>
--- a/gateways/three_gw.svg
+++ b/gateways/three_gw.svg
@ -0,0 +1,109 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" version="1.1" viewBox="152 140.7461 648.5 356.93095" width="648.5" height="356.93095">
+  <defs>
+    <font-face font-family="Helvetica Neue" font-size="16" panose-1="2 0 5 3 0 0 0 2 0 4" units-per-em="1000" underline-position="-100" underline-thickness="50" slope="0" x-height="517" cap-height="714" ascent="951.9958" descent="-212.99744" font-weight="400">
+      <font-face-src>
+        <font-face-name name="HelveticaNeue"/>
+      </font-face-src>
+    </font-face>
+    <marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" stroke-linejoin="miter" stroke-miterlimit="10" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black">
+      <g>
+        <path d="M 8 0 L 0 -3 L 0 3 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/>
+      </g>
+    </marker>
+    <marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker_2" stroke-linejoin="miter" stroke-miterlimit="10" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black">
+      <g>
+        <path d="M 8 0 L 0 -3 L 0 3 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/>
+      </g>
+    </marker>
+  </defs>
+  <metadata> Produced by OmniGraffle 7.10.2 
+    <dc:date>2019-05-07 16:43:34 +0000</dc:date>
+  </metadata>
+  <g id="Canvas_1" fill-opacity="1" stroke="none" stroke-dasharray="none" fill="none" stroke-opacity="1">
+    <title>Canvas 1</title>
+    <rect fill="white" x="152" y="140.7461" width="648.5" height="356.93095"/>
+    <g id="Canvas_1: Layer 1">
+      <title>Layer 1</title>
+      <g id="Graphic_5">
+        <circle cx="185.75" cy="225.75" r="33.250053130238" fill="white"/>
+        <circle cx="185.75" cy="225.75" r="33.250053130238" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+        <text transform="translate(164.15 215.75)" fill="black">
+          <tspan font-family="Helvetica Neue" font-size="16" font-weight="400" fill="black" x="11.968" y="15">A1</tspan>
+        </text>
+      </g>
+      <g id="Graphic_6">
+        <circle cx="185.75" cy="332.25" r="33.250053130238" fill="white"/>
+        <circle cx="185.75" cy="332.25" r="33.250053130238" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+        <text transform="translate(164.15 322.25)" fill="black">
+          <tspan font-family="Helvetica Neue" font-size="16" font-weight="400" fill="black" x="11.968" y="15">A2</tspan>
+        </text>
+      </g>
+      <g id="Graphic_7">
+        <circle cx="185.75" cy="438.25" r="33.2500531302381" fill="white"/>
+        <circle cx="185.75" cy="438.25" r="33.2500531302381" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+        <text transform="translate(164.15 428.25)" fill="black">
+          <tspan font-family="Helvetica Neue" font-size="16" font-weight="400" fill="black" x="11.968" y="15">A3</tspan>
+        </text>
+      </g>
+      <g id="Graphic_9">
+        <circle cx="476.25" cy="225.75" r="33.250053130238" fill="white"/>
+        <circle cx="476.25" cy="225.75" r="33.250053130238" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+        <text transform="translate(454.65 215.75)" fill="black">
+          <tspan font-family="Helvetica Neue" font-size="16" font-weight="400" fill="black" x="11.672" y="15">B1</tspan>
+        </text>
+      </g>
+      <g id="Graphic_8">
+        <circle cx="476.25" cy="332.25" r="33.250053130238" fill="white"/>
+        <circle cx="476.25" cy="332.25" r="33.250053130238" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+        <text transform="translate(454.65 322.25)" fill="black">
+          <tspan font-family="Helvetica Neue" font-size="16" font-weight="400" fill="black" x="11.672" y="15">B2</tspan>
+        </text>
+      </g>
+      <g id="Line_10">
+        <line x1="219.00002" y1="225.75" x2="433.1" y2="225.75" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+      </g>
+      <g id="Line_11">
+        <line x1="216.9757" y1="320.80236" x2="435.72924" y2="240.6053" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+      </g>
+      <g id="Line_12">
+        <line x1="216.99302" y1="426.8498" x2="435.70677" y2="347.04374" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+      </g>
+      <g id="Line_14">
+        <line x1="449.4098" y1="245.38355" x2="220.58062" y2="412.7715" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="1.0,5.0" stroke-width="1"/>
+      </g>
+      <g id="Line_15">
+        <line x1="443" y1="332.25" x2="228.90002" y2="332.25" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="1.0,5.0" stroke-width="1"/>
+      </g>
+      <g id="Graphic_18">
+        <circle cx="766.75" cy="225.75" r="33.250053130238" fill="white"/>
+        <circle cx="766.75" cy="225.75" r="33.250053130238" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+        <text transform="translate(745.15 215.75)" fill="black">
+          <tspan font-family="Helvetica Neue" font-size="16" font-weight="400" fill="black" x="11.376" y="15">C1</tspan>
+        </text>
+      </g>
+      <g id="Line_19">
+        <line x1="509.5" y1="225.75" x2="723.6" y2="225.75" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+      </g>
+      <g id="Line_24">
+        <path d="M 737.9152 209.17695 C 691.19614 184.57088 592.20205 141.2461 474.34375 141.2461 C 363.9938 141.2461 271.8946 179.22656 223.17214 204.26238" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="1.0,4.0" stroke-width="1"/>
+      </g>
+      <g id="Line_25">
+        <path d="M 215.77826 452.5492 C 273.97692 477.73306 407.3362 523.2367 531.3711 477.8711 C 649.03855 434.8344 718.7598 324.48573 748.9529 265.0568" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="1.0,4.0" stroke-width="1"/>
+      </g>
+      <g id="Line_26">
+        <path d="M 206.45157 358.2732 C 244.25963 400.88577 331.892 478.7072 453.0625 456.4961 C 575.3629 434.07786 689.0991 317.5855 739.4792 259.19245" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="1.0,4.0" stroke-width="1"/>
+      </g>
+      <g id="Line_27">
+        <path d="M 199.5272 256.02128 C 228.04436 312.15083 301.47194 426.71484 425.75 426.71484 C 549.392 426.71484 679.6919 313.32052 736.8824 256.88767" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="1.0,4.0" stroke-width="1"/>
+      </g>
+      <g id="Line_28">
+        <line x1="507.4757" y1="320.80236" x2="726.2292" y2="240.6053" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
+      </g>
+      <g id="Line_30">
+        <path d="M 736.5963 211.71723 C 709.7235 200.97972 667.2586 188 619 188 C 577.0519 188 540.7803 197.8071 515.29633 207.424" marker-end="url(#FilledArrow_Marker_2)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="1.0,4.0" stroke-width="1"/>
+      </g>
+    </g>
+  </g>
+</svg>
--- a/leafnodes/README.md
+++ b/leafnodes/README.md
@ -0,0 +1,208 @@
+## Leaf Nodes
+
+A _Leaf Node_ is a separate authentication domain. Leaf Nodes leverage [Accounts](../nats_server/jwt_auth.md) and JWT to enable a server to connect to another and filter messages as per the leaf node's account User configuration.
+
+This effectively means that the leaf node cluster's with the other server at an Account level:
+
+- Leaf nodes clients authenticate locally (or just connect if authentication is not required)
+- Traffic between the leaf node and the cluster assume the restrictions of the User configuration used to create the leaf connection. 
+	- Subjects that the user is allowed to publish are exported to the cluster. 
+	- Subjects the user is allowed to subscribe to, are imported into the leaf node.
+
+> Leaf Nodes are an important component as a way to bridge traffic between local nats-servers you control and servers that are managed by a third-party. The Synadia's [NATS Global Service (NGS)](https://www.synadia.com/) allows accounts to use leaf nodes, but gain accessibility to the global network to inexpensively connect geographically distributed servers or small clusters.
+
+[LeafNode Configuration Options](leafnode_conf.md)
+
+### LeafNode Configuration Tutorial
+
+Create a new operator called "O":
+```text
+> nsc add operator -n O
+Generated operator key - private key stored "~/.nkeys/O/O.nk"
+Success! - added operator "O"
+```
+
+Create an account called "A":
+```text
+> nsc add account -n A
+Generated account key - private key stored "~/.nkeys/O/accounts/A/A.nk"
+Success! - added account "A"
+```
+
+Create an user called "leaf":
+```text
+> nsc add user -n leaf
+Generated user key - private key stored "~/.nkeys/O/accounts/A/users/leaf.nk"
+Generated user creds file "~/.nkeys/O/accounts/A/users/leaf.creds"
+Success! - added user "leaf" to "A"
+```
+
+Let's create an second user called 'nolimit'
+```text
+> nsc add user -n nolimit
+Generated user key - private key stored "~/.nkeys/O/accounts/A/users/nolimit.nk"
+Generated user creds file "~/.nkeys/O/accounts/A/users/nolimit.creds"
+Success! - added user "nolimit" to "A"
+```
+
+Start a nats-account-server:
+```text
+> nats-account-server -nsc ~/.nsc/nats/O
+```
+
+Create the server configuration file (server.conf) with the following contents:
+```text
+operator: /Users/synadia/.nsc/nats/O/O.jwt
+resolver: URL(http://localhost:9090/jwt/v1/accounts/)
+leafnodes {
+	listen: "127.0.0.1:4000"
+}
+```
+The server configuration naturally requires an `operator` and `resolver` to deal with the JWT authentication and accounts. In addition the `leafnodes` configuration exposes a `listen` where the server will receive leaf nodes. In this case on the localhost on port 4000.
+
+Start the nats-server:
+```text
+> nats-server -c server.conf
+```
+
+Create a subscriber on the server:
+```text
+> nats-sub -creds ~/.nkeys/O/accounts/A/users/nolimit.creds ">"
+Listening on [>]
+```
+
+
+Create the leaf server configuration (leaf.conf) with the following contents:
+```text
+port: 4111
+leafnodes {
+	remotes = [ 
+		{ url: nats-leaf://localhost:4000, 
+		  credentials: /Users/synadia/.nkeys/O/accounts/A/users/leaf.creds 
+		},
+	]
+}
+```
+Note the leaf node configuration lists a number of `remotes`. The `url` specifies the port on the server where leaf node connections are allowed. The `credentials` configuration specifies the path to a user's credentials file.
+
+Create a subscriber on the leaf:
+```text
+> nats-sub -s localhost:4111 ">"
+Listening on [>]
+```
+
+Publish a message on the server:
+```text
+> nats-pub -creds ~/.nkeys/O/accounts/A/users/leaf.creds foo foo
+Published [foo] : 'foo'
+```
+
+Both the server and leaf subscriber print:
+```text
+[#1] Received on [foo]: 'foo'
+```
+
+Publish a message on the leaf:
+```text
+> nats-pub -s localhost:4111 bar bar
+Published [bar] : 'bar'
+```
+Both the server and leaf subscribers print:
+```text
+[#2] Received on [bar]: 'bar'
+```
+
+The leaf forwards all local messages to the server where members of the account are able to receive them. Messages published on the server by the account are forwarded to the leaf where subscribers are able to receive them.
+
+### Leaf Authorization
+
+In some cases you may want to restrict what messages can be exported from the leaf node or imported from the account.  For leaf servers this is simply a user account configuration, as users can have specific permissions on what subjects to publish and/or subscribe to.
+
+Let's put some restrictions on the `leaf` user so that it can only publish to `foo` and subscribe to `bar`:
+
+```text
+> nsc edit user -n leaf --allow-pub foo --allow-sub bar
+Updated user creds file "~/.nkeys/O/accounts/A/users/leaf.creds"
+Success! - edited user "leaf" in account "A"
+
+-----BEGIN NATS ACCOUNT JWT-----
+eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5In0.eyJqdGkiOiJVSk9RTFVSTUVFTVZXQVpVT0E2VlE1UVQ0UEdIV081WktDWlBLVFBJQVpLSldaSTJGNVpRIiwiaWF0IjoxNTU2ODM1MzU4LCJpc3MiOiJBRDU3TUZOQklLTzNBRFU2VktMRkVYQlBVQjdFWlpLU0tVUDdZTzNWVUFJTUlBWUpVNE1EM0NDUiIsIm5hbWUiOiJsZWFmIiwic3ViIjoiVUNEMlpSVUs1UE8yMk02MlNWVTZITzZJS01BVERDUlJYVVVGWDRRU1VTWFdRSDRHU1Y3RDdXVzMiLCJ0eXBlIjoidXNlciIsIm5hdHMiOnsicHViIjp7ImFsbG93IjpbImZvbyJdfSwic3ViIjp7ImFsbG93IjpbImJhciJdfX19.IeqSylTaisMQMH3Ih_0G8LLxoxe0gIClpxTm3B_ys_XwL9TtPIW-M2qdaYQZ_ZmR2glMvYK4EJ6J8RQ1UZdGAg
+------END NATS ACCOUNT JWT------
+
+> nsc describe user -n leaf
+╭───────────────────────────────────────────╮
+│                   User                    │
+├─────────────────┬─────────────────────────┤
+│ Name            │ leaf                    │
+│ User ID         │ UCD2ZRUK5PO2            │
+│ Issuer ID       │ AD57MFNBIKO3            │
+│ Issued          │ 2019-05-02 22:15:58 UTC │
+│ Expires         │                         │
+├─────────────────┼─────────────────────────┤
+│ Pub Allow       │ foo                     │
+│ Sub Allow       │ bar                     │
+├─────────────────┼─────────────────────────┤
+│ Max Messages    │ Unlimited               │
+│ Max Msg Payload │ Unlimited               │
+│ Network Src     │ Any                     │
+│ Time            │ Any                     │
+╰─────────────────┴─────────────────────────╯
+```
+
+As we can see on the inspection of the user, the restrictions have been applied.
+
+Let's repeat the experiment. This time we'll restart the leaf server so that the new user configuration is applied:
+
+```text
+> nats-server -c leaf.conf
+```
+
+You should see a new message on the leaf subscriber:
+```text
+Reconnected [nats://localhost:4111]
+```
+
+Let's publish a message on the leaf:
+```text
+> nats-pub -s localhost:4111 foo foo
+Published [foo] : 'foo'
+```
+
+You should see a new message in all your subscriber windows:
+```text
+[#3] Received on [foo]: 'foo'
+```
+
+Now publish a new message on the leaf, but this time with the subject `bar`:
+```text
+> nats-pub -s localhost:4111 bar bar
+Published [bar] : 'bar'
+```
+
+This time only the leaf subscriber will print `[#4] Received on [bar]: 'bar'`, the account subscriber won't print it because the leaf user doesn't have permissions to publish on 'bar'.
+
+
+Let's try the flow of messages from the server to the leaf node:
+```
+> nats-pub -creds ~/.nkeys/O/accounts/A/users/leaf.creds foo foo
+Published [foo] : 'foo'
+```
+Only the server subscriber will receive the message as expected.
+
+Repeat the publish this time with 'bar':
+
+```
+> nats-pub -creds ~/.nkeys/O/accounts/A/users/leaf.creds bar bar
+Published [bar] : 'bar'
+```
+Both subscribers will receive the message as expected.
+
+As you can see:
+
+- Messages to and from the leaf node to the server are limitted by the user associated with the leaf node connection.
+- Messages within the leaf node are as per the server's authentication and authorization configuration
+
+
+
+
+
--- a/leafnodes/leafnode_conf.md
+++ b/leafnodes/leafnode_conf.md
@ -0,0 +1,37 @@
+## `LeafNode` Configuration Block
+
+| Property | Description |
+| :------  | :---- |
+| `advertise` | Hostport `<host>:<port>` to advertise to other gateways. |
+| `authorization` | Authorization block (same as other nats-server `authorization` configuration). |
+| `host` | Interface where the gateway will listen for incomming gateway connections. |
+| `listen` | Combines `host` and `port` as `<host>:<port>` |
+| `name` | Name for this cluster, all gateways belonging to the same cluster, should specify the same name. |
+| `no_advertise | if `true` the leafnode shouldn't be advertised. |
+| `port` | Port where the gateway will listen for incomming gateway connections. |
+| `remotes` | List of `remote` entries specifying servers where leafnode client connection can be made. |
+| `tls` | TLS configuration block (same as other nats-server `tls` configuration). |
+
+
+### LeafNode `Remote` Entry Block
+
+| Property | Description |
+| :------  | :---- |
+| `url` | Leafnode URL (URL protocol should be `leafnode`). |
+| `account` | Account public key identifying the leafnode. Account must be defined locally. |
+| `credentials` | Credential file for connecting to the leafnode server. |
+| `tls` | A TLS configuration block. Gateway client will use specified TLS certificates when connecting/authenticating. |
+
+### `TLS` Configuration Block
+
+| Property | Description |
+| :------  | :---- |
+| `cert_file` | TLS certificate file. |
+| `key_file` | TLS certificate key file. |
+| `ca_file` | TLS certificate authority file. |
+| `insecure` | Skip certificate verfication. |
+| `verify` | If `true`, require and verify client certificates. |
+| `verify_and_map` | If `true`, require and verify client certificates and use values map certificate values for authentication purposes. |
+| `cipher_suites` | When set, only the specified TLS cipher suites will be allowed. Values must match golang version used to build the server.  |
+| `curve_preferences` | List of TLS cypher curves to use in order. |
+| `timeout` | TLS handshake timeout in fractional seconds. |
--- a/nats_server/configuration.md
+++ b/nats_server/configuration.md
@ -9,8 +9,8 @@ The NATS configuration file supports the following syntax:
 	- Equals sign: `foo = 2`
 	- Colon: `foo: 2`
 	- Whitespace: `foo 2`
- Arrays are enclosed in brackets: `[...]`
- Maps are enclosed in braces: `{...}`
+- Arrays are enclosed in brackets: `["a", "b", "c"]`
+- Maps are enclosed in braces: `{foo: 2}`
 - Maps can be assigned with no key separator
 - Semicolons can be used as terminators

@ -46,7 +46,7 @@ Server configurations can specify variables. Variables allow you to reference a
 Variables:
 - Are block scoped
 - Are referenced with a `$` prefix.
- Can be resolved from the environment variables having the same name
+- Can be resolved from environment variables having the same name

 > If the environment variable value begins with a number you may have trouble resolving it depending on the server version you are running.

--- a/nats_server/flags.md
+++ b/nats_server/flags.md
@ -1,7 +1,7 @@
 ## Flags


-The NATS server has many flags to customize it's behaviour without having to write a configuration file.
+The NATS server has many flags to customize its behaviour without having to write a configuration file.

 The configuration flags revolve around:

--- a/nats_server/tls_mutual_auth.md
+++ b/nats_server/tls_mutual_auth.md
@ -26,7 +26,7 @@ Or via the command line:

 This option simply verifies the client's certificate has been signed by the CA specified in the `ca_file` option. 

-## Mapping Client Certificates To An User
+## Mapping Client Certificates To A User

 In addition to verifying that a client certificate was issued by a specified CA, you can use information encoded in the certificate to authenticate a client. The client wouldn't have to provide or track usernames or passwords.

--- a/nats_server/tokens.md
+++ b/nats_server/tokens.md
@ -42,7 +42,7 @@ authorization {
 }
 ```

-The client will still require the clear-text password to connect:
+The client will still require the clear-text token to connect:

 ```
 nats-sub -s nats://dag0HTXl4RGg7dXdaJwbC8@localhost:4222 ">"
--- a/nats_server/username_password.md
+++ b/nats_server/username_password.md
@ -37,13 +37,14 @@ bcrypt hash: $2a$11$V1qrpBt8/SLfEBr4NJq4T.2mg8chx8.MTblUiTBOLV3MKDeAy.f7u
 ```
 And on the configuration file:

+```
 authorization: {
 	users: [
 		{user: a, password: "$2a$11$V1qrpBt8/SLfEBr4NJq4T.2mg8chx8.MTblUiTBOLV3MKDeAy.f7u"},
 		...	
 	]
 }
-
+```

 ## Reloading a Configuration

--- a/nats_tools/nas/README.md
+++ b/nats_tools/nas/README.md
@ -0,0 +1,16 @@
+## NATS Account Server
+
+The [NATS Account Server](https://github.com/nats-io/nats-account-server) is an HTTP server that hosts and vends JWTs for nats-server 2.0 account authentication. The server supports an number of stores which enable it to serve JWTs from:
+
+- a directory
+- an [NSC](../nsc/nsc.md) directory
+- memory (for testing purposes)
+
+The server can operate in a _READ ONLY_ mode where it serves content from a directory, or in notification mode, where it can notify a NATS server that JWT in the store have been modified, updating the NATS server with the updated JWT.
+
+
+### Memory Resolver
+
+For very simple installations, where JWTs are mostly static, the NATS server also supports a _Memory Resolver_ that can be configured statically in the server's configuration file.
+
+You can learn more about how to configure the [memory resolver here](mem_resolver.md).
--- a/nats_tools/nas/dir_store.md
+++ b/nats_tools/nas/dir_store.md
@ -0,0 +1,24 @@
+## Directory Store
+
+
+### NATS Account Server Configuration
+
+```
+OperatorJWTPath: "/users/synadia/.nsc/nats/Test/Test.jwt",
+http {
+	port: 9090
+},
+store {
+	dir: "/tmp/as_store",
+	readonly: false,
+	shard: true
+}
+```
+
+### To Add/Update a JWT
+
+```
+> curl -i -X POST localhost:9090/jwt/v1/accounts/AC7PO3MREV26U3LFZFP5BN3HAI32X3PKLBRVMPAETLEHWPQEUG7EJY4H --data-binary @/Users/synadia/.nsc/nats/Test/accounts/TestAccount/TestAccount.jwt -H "Content-Type: text/text"
+```
+
+Note that the `@` before the file name is required for curl to read the specified file, and use it as the payload. Otherwise it will simply post the path specified, which will result in an update error.
--- a/nats_tools/nas/inspecting_jwts.md
+++ b/nats_tools/nas/inspecting_jwts.md
@ -0,0 +1,40 @@
+## Inspecting a JWT from the `nats-account-server`
+
+Let’s say that you know the account for a stream that you are interested in, but you don't know all the details for creating an import. If you know and have access to a nats-account-server, you can help yourself. The nats-account-server can decode a JWT and give you human readable values that you can use.
+
+The endpoint for retrieving an account JWT is:
+`/jwt/v1/accounts/<account_id>`. To decode a JWT add the query string `?decode=true`.
+
+
+```json
+> curl http://localhost:9090/jwt/v1/accounts/AC7PO3MREV26U3LFZFP5BN3HAI32X3PKLBRVMPAETLEHWPQEUG7EJY4H\?decode=true
+{
+    "typ": "jwt",
+    "alg": "ed25519"
+}
+{
+    "jti": "5YMRO4KNMYWQDMRAHVTT4KX63CA2L3M6F4VM3S7NNGPMCCATORXQ",
+    "iat": 1556229062 (2019-04-25),
+    "iss": "OAYI3YUZSWDNMERD2IN3HZSIP3JA2E3VDTXSTEVOIII273XL2NABJP64",
+    "name": "TestAccount",
+    "sub": "AC7PO3MREV26U3LFZFP5BN3HAI32X3PKLBRVMPAETLEHWPQEUG7EJY4H",
+    "type": "account",
+    "nats": {
+        "exports": [
+            {
+                "name": "abc",
+                "subject": "a.b.c.>",
+                "type": "stream"
+            }
+        ],
+…
+```
+As you can see from above, the JWT is decoded. The standard JWT claim field abbreviated names may be a little terse, so here's a list of the more important ones:
+
+- `jti` is the _JWT ID_. All JWTs have one and they are unique.
+- `iat` is _Issued At_ - the UNIX date (number of seconds since 1970) when the JWT was issued.
+- `iss` is the _Issuer_. For NATS JWTs it is the public key of the issuer. In the example above the entity is an account, so the issuer will be an operator. Thus the id will always start with the letter `O`.
+- `sub` is the _Subject_ of the claim. In NATS JWTs it is the public key of the entity of the claim is for. In the example above, it is an Account, so the issuer will always start with the letter `A`.
+
+
+On the example above, we see that there is one export in this account, it is public (`token_req` is `false` or not set), and it is a `stream`. So this account exports a public stream. With that information you can create an import on the public stream.
--- a/nats_tools/nas/mem_resolver.md
+++ b/nats_tools/nas/mem_resolver.md
@ -0,0 +1,78 @@
+## Memory Resolver
+
+The `MEMORY` resolver is a built-in resolver for JWTs. It is mostly used by test setups but can be used to test the simplest of environments where there is one or very few accounts, and the account JWTs don’t change often.
+
+The basic configuration for the server requires:
+
+- The operator JWT
+- `resolver` set to `MEMORY`
+- `resolver_preload` set to an object where account public keys are mapped to account JWTs.
+
+
+### Create Required Entities
+
+Let’s create the setup:
+
+```text
+> nsc add operator -n memory
+Generated operator key - private key stored "/Users/synadia/.nkeys/memory/memory.nk"
+Success! - added operator "memory"
+
+> nsc add account --name A
+Generated account key - private key stored "/Users/synadia/.nkeys/memory/accounts/A/A.nk"
+Success! - added account "A"
+
+> nsc describe account -W
+╭──────────────────────────────────────────────────────────────────────────────────────╮
+│                                   Account Details                                    │
+├───────────────────────────┬──────────────────────────────────────────────────────────┤
+│ Name                      │ A                                                        │
+│ Account ID                │ ACSU3Q6LTLBVLGAQUONAGXJHVNWGSKKAUA7IY5TB4Z7PLEKSR5O6JTGR │
+│ Issuer ID                 │ ODWZJ2KAPF76WOWMPCJF6BY4QIPLTUIY4JIBLU4K3YDG3GHIWBVWBHUZ │
+│ Issued                    │ 2019-04-30 20:21:34 UTC                                  │
+│ Expires                   │                                                          │
+├───────────────────────────┼──────────────────────────────────────────────────────────┤
+│ Max Connections           │ Unlimited                                                │
+│ Max Leaf Node Connections │ Unlimited                                                │
+│ Max Data                  │ Unlimited                                                │
+│ Max Exports               │ Unlimited                                                │
+│ Max Imports               │ Unlimited                                                │
+│ Max Msg Payload           │ Unlimited                                                │
+│ Max Subscriptions         │ Unlimited                                                │
+│ Exports Allows Wildcards  │ True                                                     │
+├───────────────────────────┼──────────────────────────────────────────────────────────┤
+│ Imports                   │ None                                                     │
+│ Exports                   │ None                                                     │
+╰───────────────────────────┴──────────────────────────────────────────────────────────╯
+
+> cat /Users/synadia/.nsc/nats/memory/accounts/A/A.jwt 
+eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5In0.eyJqdGkiOiJPRFhJSVI2Wlg1Q1AzMlFJTFczWFBENEtTSDYzUFNNSEZHUkpaT05DR1RLVVBISlRLQ0JBIiwiaWF0IjoxNTU2NjU1Njk0LCJpc3MiOiJPRFdaSjJLQVBGNzZXT1dNUENKRjZCWTRRSVBMVFVJWTRKSUJMVTRLM1lERzNHSElXQlZXQkhVWiIsIm5hbWUiOiJBIiwic3ViIjoiQUNTVTNRNkxUTEJWTEdBUVVPTkFHWEpIVk5XR1NLS0FVQTdJWTVUQjRaN1BMRUtTUjVPNkpUR1IiLCJ0eXBlIjoiYWNjb3VudCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJjb25uIjotMSwibGVhZiI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwiZGF0YSI6LTEsInBheWxvYWQiOi0xLCJ3aWxkY2FyZHMiOnRydWV9fX0._WW5C1triCh8a4jhyBxEZZP8RJ17pINS8qLzz-01o6zbz1uZfTOJGvwSTS6Yv2_849B9iUXSd-8kp1iMXHdoBA
+
+> nsc add user --name TA
+Generated user key - private key stored “/Users/synadia/.nkeys/memory/accounts/A/users/TA.nk"
+Generated user creds file “/Users/synadia/.nkeys/memory/accounts/A/users/TA.creds"
+Success! - added user "TA" to "A"
+```
+
+### Create the Server Config
+
+With the above entries, we can reference the operator JWT and the account JWT in a server configuration. Remember that your configuration will be in `$NSC_HOME/nats/<operator_name>/<operator_name>.jwt` for the operator. The account JWT will be in `$NSC_HOME/nats/<operator_name>/accounts/<account_name>/<account_name>.jwt`
+
+```text
+operator: /Users/synadia/.nsc/nats/memory/memory.jwt
+resolver: MEMORY
+resolver_preload: {
+ACSU3Q6LTLBVLGAQUONAGXJHVNWGSKKAUA7IY5TB4Z7PLEKSR5O6JTGR: eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5In0.eyJqdGkiOiJPRFhJSVI2Wlg1Q1AzMlFJTFczWFBENEtTSDYzUFNNSEZHUkpaT05DR1RLVVBISlRLQ0JBIiwiaWF0IjoxNTU2NjU1Njk0LCJpc3MiOiJPRFdaSjJLQVBGNzZXT1dNUENKRjZCWTRRSVBMVFVJWTRKSUJMVTRLM1lERzNHSElXQlZXQkhVWiIsIm5hbWUiOiJBIiwic3ViIjoiQUNTVTNRNkxUTEJWTEdBUVVPTkFHWEpIVk5XR1NLS0FVQTdJWTVUQjRaN1BMRUtTUjVPNkpUR1IiLCJ0eXBlIjoiYWNjb3VudCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJjb25uIjotMSwibGVhZiI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwiZGF0YSI6LTEsInBheWxvYWQiOi0xLCJ3aWxkY2FyZHMiOnRydWV9fX0._WW5C1triCh8a4jhyBxEZZP8RJ17pINS8qLzz-01o6zbz1uZfTOJGvwSTS6Yv2_849B9iUXSd-8kp1iMXHdoBA
+}
+```
+
+Save the config at server.conf and start the server:
+```text
+> nats-server -c server.conf
+```
+
+Start a subscriber:
+```text
+> nats-sub -creds /Users/synadia/.nkeys/memory/accounts/A/users/TA.creds ">"
+Listening on [>]
+```
--- a/nats_tools/nas/nas_conf.md
+++ b/nats_tools/nas/nas_conf.md
@ -0,0 +1,164 @@
+## Configuration
+
+Basic configuration revolves around 4 settings:
+
+- The store to read JWTs from
+- The HTTP/S configuration
+- NATS (for cases where updates are enabled)
+- Logging
+
+For complete information on please refer to the project's [Github](https://github.com/nats-io/nats-account-server).
+
+
+### `nsc` Configuration
+
+For a basic usage of the server you can specify the `-nsc` flag, and specify the path to an operator in your environment.
+
+> If you have not yet created an operator or accounts, you'll need to do so before continuing. See [NSC](../nsc/README.md)
+
+You can easily locate the path by running `nsc env` to print your `nsc` settings:
+
+```text
+> nsc env
+╭──────────────────────────────────────────╮
+│             NSC Environment              │
+├──────────────────┬─────┬─────────────────┤
+│ Setting          │ Set │ Effective Value │
+├──────────────────┼─────┼─────────────────┤
+│ $NKEYS_PATH      │ No  │ ~/.nkeys        │
+│ $NSC_HOME        │ No  │ ~/.nsc          │
+│ Config           │     │ ~/.nsc/nsc.json │
+├──────────────────┼─────┼─────────────────┤
+│ Stores Dir       │     │ ~/.nsc/nats     │
+│ Default Operator │     │ Test            │
+│ Default Account  │     │ TestAccount     │
+│ Default Cluster  │     │                 │
+╰──────────────────┴─────┴─────────────────╯
+```
+
+The path you are interested in the `Stores Dir`. This is the root of all operators, you'll also need the name of your operator. If your current operator is not listed, you can list all your available operators by doing:
+
+```text
+> nsc list operators
+```
+
+To start the `nats-account-server` with the operator `Test`:
+
+```text
+> nats-account-server -nsc ~/.nsc/nats/Test
+2019/05/10 11:22:41.845148 [INF] starting NATS Account server, version 0.0-dev
+2019/05/10 11:22:41.845241 [INF] server time is Fri May 10 11:22:41 CDT 2019
+2019/05/10 11:22:41.845245 [INF] loading operator from /Users/synadia/.nsc/nats/Test/Test.jwt
+2019/05/10 11:22:41.846367 [INF] creating a read-only store for the NSC folder at /Users/synadia/.nsc/nats/Test
+2019/05/10 11:22:41.846459 [INF] NATS is not configured, server will not fire notifications on update
+2019/05/10 11:22:41.855291 [INF] http listening on port 9090
+2019/05/10 11:22:41.855301 [INF] nats-account-server is running
+2019/05/10 11:22:41.855303 [INF] configure the nats-server with:
+2019/05/10 11:22:41.855306 [INF]   resolver: URL(http://localhost:9090/jwt/v1/accounts/)
+```
+
+By default the server will serve JWTs on the localhost at port 9090. The last line in the shown in the printout is important, that is the resolver URL you'll have to provide on your NATS server configuration. You'll also need the matching operator JWT which is on `~/.nsc/nats/Test/Test.jwt` if you are following the example above. On the server configuration you'll need to expand the `~` as necessary. Here's what my NATS server configuration looks like:
+
+```text
+operator: /Users/synadia/.nsc/nats/Test/Test.jwt
+resolver: URL(http://localhost:9090/jwt/v1/accounts/)
+```
+
+Note that servers you create with the `-nsc` option (or store option) are read-only. This means that the server will not accept POST requests to update the JWT store.
+
+### Directory Configuration
+
+You can start a server using a plain directory. In this case you'll be responsible for adding any JWT that you want resolved. 
+
+> The server looks for account JWTs by using the public key of the account as the file name followed by the extension `.jwt`. The server will not introspect the JWTs, so if you don't name the files correctly, it will fail to find them or serve a JWT that doesn't match the requested account.
+
+```text
+> mkdir /tmp/jwts
+nats-account-server -dir /tmp/jwts
+2019/05/10 11:33:40.501305 [INF] starting NATS Account server, version 0.0-dev
+2019/05/10 11:33:40.501383 [INF] server time is Fri May 10 11:33:40 CDT 2019
+2019/05/10 11:33:40.501404 [INF] creating a store at /tmp/jwts
+2019/05/10 11:33:40.501430 [INF] NATS is not configured, server will not fire notifications on update
+2019/05/10 11:33:40.510273 [INF] http listening on port 9090
+2019/05/10 11:33:40.510283 [INF] nats-account-server is running
+2019/05/10 11:33:40.510285 [INF] configure the nats-server with:
+2019/05/10 11:33:40.510291 [INF]   resolver: URL(http://localhost:9090/jwt/v1/accounts/)
+```
+
+Configuration for the NATS server is the same as in the previous example:
+
+```text
+operator: /Users/synadia/.nsc/nats/Test/Test.jwt
+resolver: URL(http://localhost:9090/jwt/v1/accounts/)
+```
+
+### Configuration File
+
+While the `-nsc` and `-dir` store flags are sufficient for some very simple developer setups, any production or non-read-only server will require a configuration file.
+
+Let's take a look at the configuration options:
+
+#### Configuration Options
+
+| Option | Description |
+| :- | :- |
+| `http` | An `http` configuration block specifying HTTP options. |
+| `logging` | A `logging` configuration block specifying server logging options. |
+| `nats` | A `nats` configuration block specifying NATS connection information for the account server to push JWT changes to a NATS server. |
+| `operatorjwtpath` | The path to an operator JWT. Required for non-read-only servers. Only JWTs signed by the operator (or one of it's signing keys) are accepted. |
+| `store` | A `store` configuration block specifying store options. |
+| `systemaccountjwtpath` | Path to an Account JWT that should be returned as the system account. |
+
+
+#### `store` Configuration
+
+| Option | Description |
+| :- | :- |
+| `dir` | Configures a directory as a store. |
+| `nsc` | Configures an nsc read-only store. The value should be the path to an operator _directory_. Option is mutually exlusive with `dir`. |
+| `readonly` | If `true`, the store will not accept POST requests. Note that to receive requests, the store must also have `operatorjwtpath` specified as a root option. |
+| `shard` | If `true`, JWTs are sharded in the store directory. |
+
+
+
+### `logging` Options
+
+| Option | Description |
+| :- | :- |
+| `time` | If `true`, a timestamp is added to log messages. |
+| `debug` | If `true`, debug messages are logged. |
+| `trace` | If `true`, trace messages are logged. |
+| `colors` | If `true`, messages are logged using ANSI color escape sequences. |
+| `pid` | If `true`, the process id for the server is added to log messages. |
+
+### `http` Options
+
+| Option | Description |
+| :- | :- |
+| `host` | Interface to listen for requests on. |
+| `port` | Port to listen for requests on. |
+| `readtimeout` | Max amount of time in milliseconds to wait for a http read operation to complete. |
+| `writetimeout` | Max amount of time in milliseconds to wait for a http write operation to complete. |
+
+
+### `nats` Options
+
+| Option | Description |
+| :- | :- |
+| `servers` | List of NATS servers for the account server to use when connecting to a NATS server to publish updates. |
+| `connecttimeout` | Max amount of time in milliseconds to wait for a NATS connection. |
+| `reconnecttimewait` | Amount of time in milliseconds to between NATS server reconnect attempts. |
+| `tls` | A `tls` configuration block. |
+| `usercredentials` | A credentials _creds_ file for connecting to the NATS server. Account must be a member of a system account. |
+
+### `tls` Options
+| Option | Description |
+| :- | :- |
+| `root` | filepath to the CA certificate. |
+| `cert` | filepath to the certificate. |
+| `cert` | filepath to the certificate key. |
+
+
+
+
+
--- a/sys_accounts/README.md
+++ b/sys_accounts/README.md
@ -0,0 +1,15 @@
+## System Accounts
+
+NATS servers leverage [Account](nats_server/jwt_auth.md) support and generate events such as:
+
+- account connect/disconnect
+- authentication errors
+- server shutdown
+- server stat summary
+
+In addition the server supports a limitted number of requests that can be used to query for account connections, server stat summaries, and pinging servers in the cluster.
+
+These events are only accepted and visible to system account users. 
+
+### The System Events Tutorial
+You can learn more about System Accounts in the [Tutorial](sys_accounts.md).
--- a/sys_accounts/sys_accounts.md
+++ b/sys_accounts/sys_accounts.md
@ -0,0 +1,250 @@
+## System Account Tutorial
+
+The following is a short tutorial on how you can activate a system account to:
+
+- receive periodic updates from the server
+- send requests to the server
+- send an account update to the server
+
+### Events and Services
+
+The system account publishes messages under well known subject patterns.
+
+Server initiated events:
+
+- `$SYS.ACCOUNT.<id>.CONNECT` (client connects)
+- `$SYS.ACCOUNT.<id>.DISCONNECT` (client disconnects)
+- `$SYS.SERVER.ACCOUNT.<id>.CONNS` (connections for an account changed)
+- `$SYS.SERVER.<id>.CLIENT.AUTH.ERR` (authentication error)
+- `$SYS.ACCOUNT.<id>.LEAFNODE.CONNECT` (leaf node connnects)
+- `$SYS.SERVER.<id>.STATSZ` (stats summary)
+
+In addition other tools with system account privileges, can initiate requests:
+
+- `$SYS.REQ.SERVER.<id>.STATSZ` (request server stat summary)
+- `$SYS.REQ.SERVER.PING` (discover servers - will return multiple messages)
+
+
+Servers like `nats-account-server` publish system account messages when a claim is updated, the nats-server listens for them, and updates its account information accordingly:
+- `$SYS.ACCOUNT.<id>.CLAIMS.UPDATE`
+
+
+With these few messages you can build fairly surprisingly useful monitoring tools:
+
+- health/load of your servers
+- client connects/disconnects
+- account connections
+- authentication errors
+
+### Enabling System Events
+
+To enable and access system events, you'll have to:
+
+- Create an Operator, Account and User
+- Run a NATS Account Server (or Memory Resolver)
+
+
+#### Create an Operator, Account, User
+Let's create an operator, system account and system account user:
+
+```text
+# Create an operator if you 
+> nsc add operator -n SAOP
+Generated operator key - private key stored "~/.nkeys/SAOP/SAOP.nk"
+Success! - added operator "SAOP"
+
+# Add the system account
+> nsc add account -n SYS
+Generated account key - private key stored "~/.nkeys/SAOP/accounts/SYS/SYS.nk"
+Success! - added account "SYS"
+
+# Add a system account user
+> nsc add user -n SYSU
+Generated user key - private key stored "~/.nkeys/SAOP/accounts/SYS/users/SYSU.nk"
+Generated user creds file "~/.nkeys/SAOP/accounts/SYS/users/SYSU.creds"
+Success! - added user "SYSU" to "SYS"
+```
+
+By default, the operator JWT can be found in `~/.nsc/nats/<operator_name>/<operator.name>.jwt`.
+
+#### Nats-Account-Server
+
+To vend the credentials to the nats-server, we'll use a [nats-account-server](/nats_tools/nas/nas.md). Let's start a nats-account-server to serve the JWT credentials:
+```text
+> nats-account-server -nsc ~/.nsc/nats/SAOP
+```
+
+The server will by default vend JWT configurations on the an endpoint at: `http(s)://<server_url>/jwt/v1/accounts/`.
+
+#### NATS Server Configuration
+
+The server configuration will need:
+- The operator JWT - (`~/.nsc/nats/<operator_name>/<operator.name>.jwt`)
+- The URL where the server can resolve accounts (`http://localhost:9090/jwt/v1/accounts/`)
+- The public key of the `system_account`
+
+The only thing we don't have handy is the public key for the system account. We can get it easy enough:
+```text
+> nsc list accounts -W
+╭─────────────────────────────────────────────────────────────────╮
+│                            Accounts                             │
+├──────┬──────────────────────────────────────────────────────────┤
+│ Name │ Public Key                                               │
+├──────┼──────────────────────────────────────────────────────────┤
+│ SYS  │ ADWJVSUSEVC2GHL5GRATN2LOEOQOY2E6Z2VXNU3JEIK6BDGPWNIW3AXF │
+╰──────┴──────────────────────────────────────────────────────────╯
+```
+
+Because the server has additional resolver implementations, you need to enclose the server url like: `URL(<url>)`.
+
+
+Let's create server config with the following contents and save it to `server.conf`:
+```text
+operator: /Users/synadia/.nsc/nats/SAOP/SAOP.jwt
+system_account: ADWJVSUSEVC2GHL5GRATN2LOEOQOY2E6Z2VXNU3JEIK6BDGPWNIW3AXF
+resolver: URL(http://localhost:9090/jwt/v1/accounts/)
+```
+
+
+Let's start the nats-server:
+```text
+> nats-server -c server.conf
+```
+
+### Inspecting Server Events
+
+Let's add a subscriber for all the events published by the system account:
+
+```text
+> nats-sub -creds ~/.nkeys/SAOP/accounts/SYS/users/SYSU.creds ">"
+```
+Very quickly we'll start seeing messages from the server as they are published by the NATS server. As should be expected, the messages are just JSON, so they can easily be inspected even if just using a simple `nats-sub` to read them.
+
+To see an an account update:
+```text
+> nats-pub -creds ~/.nkeys/SAOP/accounts/SYS/users/SYSU.creds foo bar
+```
+
+The subscriber will print the connect and disconnect:
+```text
+[#35] Received on [$SYS.SERVER.ACCOUNT.ADWJVSUSEVC2GHL5GRATN2LOEOQOY2E6Z2VXNU3JEIK6BDGPWNIW3AXF.CONNS]: '{
+  "server": {
+    "host": "0.0.0.0",
+    "id": "NBTGVY3OKDKEAJPUXRHZLKBCRH3LWCKZ6ZXTAJRS2RMYN3PMDRMUZWPR",
+    "ver": "2.0.0-RC5",
+    "seq": 32,
+    "time": "2019-05-03T14:53:15.455266-05:00"
+  },
+  "acc": "ADWJVSUSEVC2GHL5GRATN2LOEOQOY2E6Z2VXNU3JEIK6BDGPWNIW3AXF",
+  "conns": 1,
+  "total_conns": 1
+}'
+[#36] Received on [$SYS.ACCOUNT.ADWJVSUSEVC2GHL5GRATN2LOEOQOY2E6Z2VXNU3JEIK6BDGPWNIW3AXF.DISCONNECT]: '{
+  "server": {
+    "host": "0.0.0.0",
+    "id": "NBTGVY3OKDKEAJPUXRHZLKBCRH3LWCKZ6ZXTAJRS2RMYN3PMDRMUZWPR",
+    "ver": "2.0.0-RC5",
+    "seq": 33,
+    "time": "2019-05-03T14:53:15.455304-05:00"
+  },
+  "client": {
+    "start": "2019-05-03T14:53:15.453824-05:00",
+    "host": "127.0.0.1",
+    "id": 6,
+    "acc": "ADWJVSUSEVC2GHL5GRATN2LOEOQOY2E6Z2VXNU3JEIK6BDGPWNIW3AXF",
+    "user": "UACPEXCAZEYWZK4O52MEGWGK4BH3OSGYM3P3C3F3LF2NGNZUS24IVG36",
+    "name": "NATS Sample Publisher",
+    "lang": "go",
+    "ver": "1.7.0",
+    "stop": "2019-05-03T14:53:15.45526-05:00"
+  },
+  "sent": {
+    "msgs": 1,
+    "bytes": 3
+  },
+  "received": {
+    "msgs": 0,
+    "bytes": 0
+  },
+  "reason": "Client Closed"
+}'
+```
+
+
+### `$SYS.REQ.SERVER.PING` - Discovering Servers
+
+To discover servers in the cluster, and get a small heath summary, publish a request to `$SYS.REQ.SERVER.PING`. Note that while the example below uses `nats-req`, only the first answer for the request will be printed. You can easily modify the example to wait until no additional responses are received for a specific amount of time, thus allowing for all responses to be collected.
+
+```text
+> nats-req -creds ~/.nkeys/SAOP/accounts/SYS/users/SYSU.creds \$SYS.REQ.SERVER.PING ""
+Published [$SYS.REQ.SERVER.PING] : ''
+Received  [_INBOX.G5mbsf0k7l7nb4eWHa7GTT.omklmvnm] : '{
+  "server": {
+    "host": "0.0.0.0",
+    "id": "NCZQDUX77OSSTGN2ESEOCP4X7GISMARX3H4DBGZBY34VLAI4TQEPK6P6",
+    "ver": "2.0.0-RC9",
+    "seq": 47,
+    "time": "2019-05-02T14:02:46.402166-05:00"
+  },
+  "statsz": {
+    "start": "2019-05-02T13:41:01.113179-05:00",
+    "mem": 12922880,
+    "cores": 20,
+    "cpu": 0,
+    "connections": 2,
+    "total_connections": 2,
+    "active_accounts": 1,
+    "subscriptions": 10,
+    "sent": {
+      "msgs": 7,
+      "bytes": 2761
+    },
+    "received": {
+      "msgs": 0,
+      "bytes": 0
+    },
+    "slow_consumers": 0
+  }
+}'
+```
+
+### `$SYS.SERVER.<id>.STATSZ` - Requesting Server Stats Summary
+
+If you know the server id for a particular server (such as from a response to `$SYS.REQ.SERVER.PING`), you can query the specific server for its health information:
+
+```text
+nats-req -creds ~/.nkeys/SAOP/accounts/SYS/users/SYSU.creds \$SYS.REQ.SERVER.NC7AKPQRC6CIZGWRJOTVFIGVSL7VW7WXTQCTUJFNG7HTCMCKQTGE5PUL.STATSZ ""
+Published [$SYS.REQ.SERVER.NC7AKPQRC6CIZGWRJOTVFIGVSL7VW7WXTQCTUJFNG7HTCMCKQTGE5PUL.STATSZ] : ''
+Received  [_INBOX.DQD44ugVt0O4Ur3pWIOOD1.WQOBevoq] : '{
+  "server": {
+    "host": "0.0.0.0",
+    "id": "NC7AKPQRC6CIZGWRJOTVFIGVSL7VW7WXTQCTUJFNG7HTCMCKQTGE5PUL",
+    "ver": "2.0.0-RC5",
+    "seq": 25,
+    "time": "2019-05-03T14:34:02.066077-05:00"
+  },
+  "statsz": {
+    "start": "2019-05-03T14:32:19.969037-05:00",
+    "mem": 11874304,
+    "cores": 20,
+    "cpu": 0,
+    "connections": 2,
+    "total_connections": 4,
+    "active_accounts": 1,
+    "subscriptions": 10,
+    "sent": {
+      "msgs": 26,
+      "bytes": 9096
+    },
+    "received": {
+      "msgs": 2,
+      "bytes": 0
+    },
+    "slow_consumers": 0
+  }
+}'
+
+```
+
+
+