Added information on prometheus/graphana (needs more from Colin)

Added disconnect event from leafnode
More clarifications

leafnodes/README.md

@@ -1,6 +1,8 @@
 ## Leaf Nodes
 
-A _Leaf Node_ is a separate authentication domain. Leaf Nodes leverage [Accounts](../nats_server/jwt_auth.md) and JWT to enable a server to connect to another and filter messages as per the leaf node's account User configuration.
+A _Leaf Node_ allows an extension to a cluter or supercluster that bridges accounts and security domains. This is useful in IoT and Edge scenarios and when the local server trafic should be low RTT and local unless routed to the super cluster.
+
+Leaf Nodes leverage [Accounts](../nats_server/jwt_auth.md) and JWT to enable a server to connect to another and filter messages as per the leaf node's account User configuration.
 
 This effectively means that the leaf node cluster's with the other server at an Account level:
 

nats_server/authorization.md

@@ -27,7 +27,7 @@ The `permission` map provides additional properties for configuring a `permissio
 
 
 
-**Important Note** NATS Authorizations are whitelist only, meaning to not break request/reply patterns you need to add rules as above with Alice and Bob for the `_INBOX.>` pattern. If an unauthorized client publishes or attempts to subscribe to a subject that has not been whitelisted, the action fails and is logged at the server, and an error message is returned to the client.
+**Important Note** NATS Authorizations can be _allow lists_, _deny lists_, or both. It is important to not break request/reply patterns. In some cases (as shown below) you need to add rules as above with Alice and Bob for the `_INBOX.>` pattern. If an unauthorized client publishes or attempts to subscribe to a subject that has not been _allow listed_, the action fails and is logged at the server, and an error message is returned to the client.
 
 ### Example
 
@@ -70,4 +70,4 @@ authorization {
 
 - _other_ has no permissions granted and therefore inherits the default permission set. You set the inherited default permissions by assigning them to the `default_permissions` entry inside of the authorization configuration block.
 
-> Note that in the above example, any client with permissions to subscribe to `_INBOX.>` can receive _all_ responses published. More sensitive installations will want to add or subset the prefix to further limit subjects that a client can subscribe. Alternatively, [_Accounts_](jwt_auth.md) allow complete isolation limiting what members of an account can see.
+> Note that in the above example, any client with permissions to subscribe to `_INBOX.>` can receive _all_ responses published. More sensitive installations will want to add or subset the prefix to further limit subjects that a client can subscribe. Alternatively, [_Accounts_](accounts.md) allow complete isolation limiting what members of an account can see.

nats_server/cluster_tls.md

@@ -2,6 +2,8 @@
 
 When setting up clusters all servers in the cluster, if using TLS, will both verify the connecting endpoints and the server responses. So certificates are checked in both directions. Certificates can be configured only for the server's cluster identity, keeping client and server certificates separate from cluster formation.
 
+TLS Mutual Authentication *is the recommended way* of securing routes.
+
 ```
 cluster {
   listen: 127.0.0.1:4244
@@ -22,4 +24,4 @@ cluster {
     nats-route://127.0.0.1:4246
   ]
 }
-```
+```

nats_server/clustering.md

@@ -9,34 +9,27 @@ Note that NATS clustered servers have a forwarding limit of one hop. This means
 
 In addition to a port for listening for clients, `nats-server` can listen on a "cluster" URL (the `-cluster` option). Additional `nats-server` servers can then add that URL to their `-routes` argument to join the cluster. These options can also be specified in a config file, but only the command-line version is shown in this overview for simplicity.
 
-### Running with No Cluster
+## Running a Simple Cluster
+
+Here is a simple cluster running on the same machine:
 
 ```sh
-nats-server -p 4222
-```
-----
+# Server A - the 'seed server'
+> nats-server -p 4222 -cluster nats://0.0.0.0:5222
 
-### Running a Simple Cluster
+# Server B
+> nats-server -p -1 -cluster nats://0.0.0.0:-1 -routes nats://localhost:5222
+# Check the output of the server for the selected client and route ports.
 
-```sh
-# Server A on 10.10.0.1
-nats-server -p 4222 -cluster nats://10.10.0.1:5222
-
-# Server B on 10.10.0.2
-nats-server -p 4222 -cluster nats://10.10.0.2:5222 -routes nats://10.10.0.1:5222
+# Server C
+> nats-server -p -1 -cluster nats://0.0.0.0:-1 -routes nats://localhost:5222
+# Check the output of the server for the selected client and route ports.
 ```
 
-----
+The _seed server_ simply declares its client and clustering port. All other servers delegate to the nats-server to auto-select a port that is not in use for both clients and cluster connections, and route to the seed server. Because the clustering protocol gossips members of the cluster, all servers are able to discover other server servers in the cluster. When a server is discovered, the discovering server will automatically attempt to connect to it in order to form a _full mesh_. Typically only one instance of the server will run per machine, so you can reuse the client port (4222) and the cluster port (5222), and simply the route to the host/port of the seed server.
 
-```sh
-# Server A on 10.10.0.1
-nats-server -p 4222 -cluster nats://10.10.0.1:5222 -routes nats://10.10.0.2:5222
 
-# Server B on 10.10.0.2
-nats-server -p 4222 -cluster nats://10.10.0.2:5222 -routes nats://10.10.0.1:5222
-```
-
-Clients connecting to any server in the cluster will remain connected to the cluster even if the server it originally connected to is taken down, as long as at least a single server remains.
+Similarly, clients connecting to any server in the cluster will discover other servers in cluster. If connection to the server is interrupted, the client will attempt to connect to all other known servers.
 
 ## Command Line Options
 
@@ -56,7 +49,7 @@ Clustering can also be configured using the server [config file](cluster_config.
 The following example demonstrates how to run a cluster of 3 servers on the same host. We will start with the seed server and use the `-D` command line parameter to produce debug information.
 
 ```sh
-nats-server -p 4222 -cluster nats://localhost:4248 -D
+nats-server -p 4222 -cluster nats://localhost:5222 -D
 ```
 
 Alternatively, you could use a configuration file, let's call it `seed.conf`, with a content similar to this:
@@ -68,7 +61,7 @@ listen: 127.0.0.1:4222
 http: 8222
 
 cluster {
-  listen: 127.0.0.1:4248
+  listen: 127.0.0.1:5222
 }
 ```
 
@@ -182,9 +175,9 @@ nats-pub -s "nats://192.168.59.105:7222" hello world
 
 [#1] Received on [hello] : 'world'
 
-# GNATSD on Node C logs:
+# nats-server on Node C logs:
 [1] 2015/06/23 05:20:31.100032 [TRC] 192.168.59.103:7244 - rid:2 - <<- [MSG hello RSID:8:2 5]
 
-# GNATSD on Node A logs:
+# nats-server on Node A logs:
 [1] 2015/06/23 05:20:31.100600 [TRC] 10.0.2.2:51007 - cid:8 - <<- [MSG hello 2 5]
 ```

nats_server/monitoring.md

@@ -180,3 +180,6 @@ $.getJSON('http://localhost:8222/connz?callback=?', function(data) {
 
 ```
 
+## Monitoring Tools
+
+In addition to writing custom monitoring tools, you can nats-server into Prometheus. The [Prometheus NATS Exporter](https://github.com/nats-io/prometheus-nats-exporter) allows you to configure the metrics you want to observe and store in Prometheous. There's a sample [Graphana](https://graphana.com) dashboard that you can use to visualize the server metrics.

nats_tools/README.md

@@ -7,4 +7,5 @@ The NATS Ecosystem has many tools to support server configuration, enhance monit
 - [nsc](nats_tools/nsc/README.md) - Configure Operators, Accounts and Users
 - [nats account server](nats_tools/nas/README.md) - Serve Account JWTs
 - [nats top](nats_tools/nats_top/README.md) - Monitor NATS Server
-- [nats-bench](nats_tools/natsbench.md) - Benchmark NATS Server
+- [nats-bench](nats_tools/natsbench.md) - Benchmark NATS Server
+- [prometheus-nats-exporter](https://github.com/nats-io/prometheus-nats-exporter) - Export NATS server metrics to [Prometheous](https://prometheus.io/) and a [Graphana](https://graphana.com) dashboard.

sys_accounts/sys_accounts.md

@@ -17,6 +17,7 @@ Server initiated events:
 - `$SYS.SERVER.ACCOUNT.<id>.CONNS` (connections for an account changed)
 - `$SYS.SERVER.<id>.CLIENT.AUTH.ERR` (authentication error)
 - `$SYS.ACCOUNT.<id>.LEAFNODE.CONNECT` (leaf node connnects)
+- `$SYS.ACCOUNT.<id>.LEAFNODE.DISCONNECT` (leaf node disconnects)
 - `$SYS.SERVER.<id>.STATSZ` (stats summary)
 
 In addition other tools with system account privileges, can initiate requests: