Commit Graph

2152 Commits

Author SHA1 Message Date
Ivan Kozlovic
6c4229300a Fixed service import cycle detection that broke with JS clustering
Also added some no-op error handler for some tests to silence the
error report in the log.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-01-14 11:27:36 -07:00
Derek Collison
1b0e740123 Fix for race
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 07:28:14 -08:00
Derek Collison
9d9f0f7099 Fix for race on setting term
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 07:02:20 -08:00
Derek Collison
14e3319ba3 Fix for no raft node
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 06:59:36 -08:00
Derek Collison
ab2a645791 Fix for various flappers
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 06:54:08 -08:00
Derek Collison
b68d7066c4 Remove alpha banner
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 06:05:55 -08:00
Derek Collison
4bfe9d4c24 Fixes to PR.
Add nats to default storage directory
Fix race in raft, change leader notice
Fix test crash on failure

Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 05:56:05 -08:00
Derek Collison
37cf7584bd Merge branch 'master' into jsc 2021-01-14 02:52:35 -07:00
Derek Collison
f0cdf89c61 JetStream Clustering WIP
Signed-off-by: Derek Collison <derek@nats.io>
2021-01-14 01:14:52 -08:00
Matthias Hanel
9c2bf8e4a9 [Added] support for jwt export response threshold
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-14 01:12:35 -08:00
Ivan Kozlovic
23f8e3d5b9 Fixed sublist notification
The insert notification was done based on the creation of a node
during an insert, which was wrong since the node may have already
existed and still the subscription could be all new. For instance,
suppose that there is a subscription on "foo.bar".
We register an notification interest for "foo", which does not
notify, which is normal. Then we create a subscription on "foo".
During the insert, "foo" node already exists so notification would
not be sent, but it should.
Fixed also removed by having removeFromNode() returning a boolean
to indicate if the subscription was the last in that node.
However, it seems that we again check for interest in
chkForRemoveNotification(), so not sure if that is required.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-01-13 20:59:36 -07:00
Ivan Kozlovic
7b116379cb Propose going back to condition variable to notify writeLoop
This is how it was up to v2.1.2 included (changed in v2.1.4 onward).
I added a benchmark that has 3 subscribers running and increase
the number of publishers: 1, 2, 5 and 10. This is the comparison
between the pre-PR and post-PR:

```
benchcmp old.txt new.txt
benchmark                           old ns/op     new ns/op     delta
Benchmark___BumpPubCount_1x3-16     396           385           -2.78%
Benchmark___BumpPubCount_2x3-16     495           406           -17.98%
Benchmark___BumpPubCount_5x3-16     542           395           -27.12%
Benchmark__BumpPubCount_10x3-16     549           515           -6.19%

benchmark                           old MB/s     new MB/s     speedup
Benchmark___BumpPubCount_1x3-16     717.27       737.54       1.03x
Benchmark___BumpPubCount_2x3-16     574.31       699.02       1.22x
Benchmark___BumpPubCount_5x3-16     524.35       718.80       1.37x
Benchmark__BumpPubCount_10x3-16     517.26       551.53       1.07x
```

It is inline with what the user reported, seeing a 20% drop in performance
when going from 1 publisher to 2. But, as we can see, the difference
between go channel and cond variable reduces with the increased number
of publishers after a certain number.

I am not sure of the performance impact on other situations, so this
PR is more of a proposal than a fix.

Resolves #1786

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-01-12 12:24:37 -07:00
Ivan Kozlovic
0d34688c4b Merge pull request #1800 from nats-io/fix_1799
[FIXED] Monitoring endpoint `connz?auth=true` show incorrect user
2021-01-11 14:28:28 -07:00
Ivan Kozlovic
f50c655e75 [FIXED] Monitoring endpoint connz?auth=true show incorrect user
Only the user (from username/password connection method) was reported
in this monitoring endpoint. Will now report proper nkey, public key,
etc..

Resolves #1799

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-01-11 12:59:05 -07:00
Matthias Hanel
431b642cbe Incorporating review comments
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-07 18:19:25 -05:00
Matthias Hanel
0ff6252692 Added tests for cfg/jwt based queue restrictions and updated jwt lib
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-07 17:30:51 -05:00
Matthias Hanel
1149c4cef9 [Added] support for jwt export response threshold
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-05 19:52:52 -05:00
Matthias Hanel
4699779f72 Merge pull request #1790 from nats-io/import-rewrite
[Added] support for wildcard services and import remapping by JWT.
2021-01-05 19:43:06 -05:00
Matthias Hanel
d59b074c64 Incorporating review comments
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-05 18:45:46 -05:00
Matthias Hanel
67ca277142 Incorporating review comment
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-04 20:02:43 -05:00
Matthias Hanel
592a6447a7 [Added] support for wildcard services and import remapping by JWT.
Imports in JWT where extended to contain a new filed LocalSubject.
This Change pulls the new JWT library version in.
It was needed as prefix did not exist in the JWT library and the
original field could not be used. The field To has been deprecated.

When LocalSubject is set, service imports can be configured the same way
they are in config. Meaning, no reversal due to the type.

This change also ensures that wildcard references in transforms are only
set in To/LocalSubject. Before, for services, $1 would have to be set in Subject.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-04 14:11:36 -05:00
Ivan Kozlovic
de1e492858 Merge pull request #1787 from scottf/stream-headers-colon-no-space
stream handle headers with no space after colon
2021-01-04 09:18:02 -07:00
scottf
bd954561bf stream handle headers with no space after colon 2021-01-03 09:47:15 -05:00
Ivan Kozlovic
d24e9b75b3 Fixed GW implicit reconnection
PR #1412 had a fix for races during implicit GW reconnection.
However, the fix was a bit too simplistic in that it was checking
only if there was any inbound gateway to decide to try to reconnect
an implicit disconnected GW. We need to check the name, not only
presence of inbound GW connections.

Related to #1412

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-28 12:28:55 -07:00
Ivan Kozlovic
14aecb2202 Fixed headers support for inbound leafnode connection
The server that solicits a LeafNode connection does not send an
INFO, so the accepting side had no way to know if the remote
supports headers or not. The solicit side will now send the headers
support capability in the CONNECT protocol so that the receiving
side can mark the inbound connection with headers support based
on that and its own support for headers.

Resolves #1781

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-21 11:53:24 -07:00
Alberto Ricart
f09992a889 updated iteration of signing keys (previously a list, now a map). (#1779) 2020-12-17 13:59:18 -07:00
Matthias Hanel
c6daffbfcc [Added] ability to use jwt latency sampling properties headers/share
Signed-off-by: Matthias Hanel <mh@synadia.com>
2020-12-16 14:34:09 -05:00
Matthias Hanel
e4b06cf7da Break test up into shorter ones
Also change ttl to be nanosecond interval and run parallel

Signed-off-by: Matthias Hanel <mh@synadia.com>
2020-12-14 19:48:54 -05:00
Derek Collison
c1c9401011 Merge pull request #1775 from nats-io/idx_short_bug
Fix for https://github.com/nats-io/jetstream/issues/406
2020-12-14 08:22:30 -08:00
Derek Collison
eb403ed4d0 Merge pull request #1773 from nats-io/cycle_wc_bug
Catch condition where a serviceImport response matched the original import.
2020-12-14 08:20:55 -08:00
Derek Collison
be6289be51 Fix for https://github.com/nats-io/jetstream/issues/406
Signed-off-by: Derek Collison <derek@nats.io>
2020-12-13 11:59:50 -08:00
Derek Collison
a3f7e97f9a Catch condition where a serviceImport response matched the original import subject.
Signed-off-by: Derek Collison <derek@nats.io>
2020-12-13 10:17:29 -08:00
Ivan Kozlovic
d5f255b98e Merge pull request #1771 from nats-io/gw_ln_tls_config_reload
[FIXED] Config reload for gateways/leaf remote TLS configurations
2020-12-12 10:51:52 -07:00
Ivan Kozlovic
2d2f85267b Add fix for TestLeafNodeLoop and others
Based on timing, it is possible that the first error is about
connection refused as opposed to "Loop detected". So use a dedicated
logger to notify only when expected error is found.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-11 18:15:49 -07:00
Ivan Kozlovic
ce5f9d6683 Fixed some flappers
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-11 17:30:33 -07:00
Ivan Kozlovic
fc1521636c [FIXED] Config reload for gateways/leaf remote TLS configurations
Presence of TLS config in any remote gateway or leafnode would
cause the config reload to fail (because TLS config internal
content may change which fails the DeepEqual check).

This PR excludes the TLS configs in such case to check for
changes in gateways and leafnodes.

Although GW and LN config reload is technically supported, this
PR updates the internal remotes' TLS configuration so that
changes/updates to TLS certificates would take effect after
a configuration reload.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-11 16:56:25 -07:00
Ivan Kozlovic
1dd485531f Updates to MQTT
- Keep track of published topic to avoid conversion in some case
- Rework tracing of some MQTT protocols
- Rework topic-to-subject conversion
- Update some tests

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-10 10:55:41 -07:00
Ivan Kozlovic
25a5fa62eb Merge pull request #1741 from nats-io/fix-issuer-same-as-account
fixed bad issuer check performed against account issuer instead account subject (Name)
2020-12-08 17:12:53 -07:00
Ivan Kozlovic
c199bec7c3 Bump version
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-07 10:07:30 -07:00
Ivan Kozlovic
b048b6b3de Merge pull request #1754 from nats-io/mqtt
[ADDED] MQTT Support
2020-12-07 09:06:12 -07:00
Ivan Kozlovic
1d7c4712a5 Increase Pub performance
Essentially make publish a zero alloc. Use c.mqtt.pp as the parser
publish packet structure. Messages were initially copied because
MQTT messages don't have CR_LF but was adding it so that it worked
for NATS pub/subs and MQTT pub/subs.
Now an MQTT producer sending to NATS sub will queue CR_LF after
payload.

Here is result of benchcmp for MQTT pub runs only:
```
benchmark                                     old ns/op     new ns/op     delta
BenchmarkMQTT_QoS0_Pub_______0b_Payload-8     157           55.6          -64.59%
BenchmarkMQTT_QoS0_Pub_______8b_Payload-8     167           61.0          -63.47%
BenchmarkMQTT_QoS0_Pub______32b_Payload-8     181           65.3          -63.92%
BenchmarkMQTT_QoS0_Pub_____128b_Payload-8     243           78.1          -67.86%
BenchmarkMQTT_QoS0_Pub_____256b_Payload-8     298           95.0          -68.12%
BenchmarkMQTT_QoS0_Pub_______1K_Payload-8     604           224           -62.91%
BenchmarkMQTT_QoS1_Pub_______0b_Payload-8     1713          1314          -23.29%
BenchmarkMQTT_QoS1_Pub_______8b_Payload-8     1703          1311          -23.02%
BenchmarkMQTT_QoS1_Pub______32b_Payload-8     1722          1345          -21.89%
BenchmarkMQTT_QoS1_Pub_____128b_Payload-8     2105          1432          -31.97%
BenchmarkMQTT_QoS1_Pub_____256b_Payload-8     2148          1503          -30.03%
BenchmarkMQTT_QoS1_Pub_______1K_Payload-8     3024          1889          -37.53%

benchmark                                     old MB/s     new MB/s     speedup
BenchmarkMQTT_QoS0_Pub_______0b_Payload-8     31.76        89.91        2.83x
BenchmarkMQTT_QoS0_Pub_______8b_Payload-8     77.79        213.01       2.74x
BenchmarkMQTT_QoS0_Pub______32b_Payload-8     204.52       566.26       2.77x
BenchmarkMQTT_QoS0_Pub_____128b_Payload-8     550.65       1715.96      3.12x
BenchmarkMQTT_QoS0_Pub_____256b_Payload-8     877.77       2757.16      3.14x
BenchmarkMQTT_QoS0_Pub_______1K_Payload-8     1705.02      4607.72      2.70x
BenchmarkMQTT_QoS1_Pub_______0b_Payload-8     6.42         8.37         1.30x
BenchmarkMQTT_QoS1_Pub_______8b_Payload-8     11.16        14.49        1.30x
BenchmarkMQTT_QoS1_Pub______32b_Payload-8     24.97        31.97        1.28x
BenchmarkMQTT_QoS1_Pub_____128b_Payload-8     66.52        97.74        1.47x
BenchmarkMQTT_QoS1_Pub_____256b_Payload-8     124.78       178.27       1.43x
BenchmarkMQTT_QoS1_Pub_______1K_Payload-8     342.64       548.32       1.60x

benchmark                                     old allocs     new allocs     delta
BenchmarkMQTT_QoS0_Pub_______0b_Payload-8     3              0              -100.00%
BenchmarkMQTT_QoS0_Pub_______8b_Payload-8     3              0              -100.00%
BenchmarkMQTT_QoS0_Pub______32b_Payload-8     3              0              -100.00%
BenchmarkMQTT_QoS0_Pub_____128b_Payload-8     4              0              -100.00%
BenchmarkMQTT_QoS0_Pub_____256b_Payload-8     4              0              -100.00%
BenchmarkMQTT_QoS0_Pub_______1K_Payload-8     4              0              -100.00%
BenchmarkMQTT_QoS1_Pub_______0b_Payload-8     5              2              -60.00%
BenchmarkMQTT_QoS1_Pub_______8b_Payload-8     5              2              -60.00%
BenchmarkMQTT_QoS1_Pub______32b_Payload-8     5              2              -60.00%
BenchmarkMQTT_QoS1_Pub_____128b_Payload-8     7              3              -57.14%
BenchmarkMQTT_QoS1_Pub_____256b_Payload-8     7              3              -57.14%
BenchmarkMQTT_QoS1_Pub_______1K_Payload-8     7              3              -57.14%

benchmark                                     old bytes     new bytes     delta
BenchmarkMQTT_QoS0_Pub_______0b_Payload-8     80            0             -100.00%
BenchmarkMQTT_QoS0_Pub_______8b_Payload-8     88            0             -100.00%
BenchmarkMQTT_QoS0_Pub______32b_Payload-8     120           0             -100.00%
BenchmarkMQTT_QoS0_Pub_____128b_Payload-8     224           0             -100.00%
BenchmarkMQTT_QoS0_Pub_____256b_Payload-8     369           1             -99.73%
BenchmarkMQTT_QoS0_Pub_______1K_Payload-8     1250          31            -97.52%
BenchmarkMQTT_QoS1_Pub_______0b_Payload-8     106           28            -73.58%
BenchmarkMQTT_QoS1_Pub_______8b_Payload-8     122           28            -77.05%
BenchmarkMQTT_QoS1_Pub______32b_Payload-8     154           28            -81.82%
BenchmarkMQTT_QoS1_Pub_____128b_Payload-8     381           157           -58.79%
BenchmarkMQTT_QoS1_Pub_____256b_Payload-8     655           287           -56.18%
BenchmarkMQTT_QoS1_Pub_______1K_Payload-8     2312          1078          -53.37%
```

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-04 14:42:37 -07:00
Matthias Hanel
dc2eebcd85 removing t.Errorf 2020-12-03 21:40:06 -05:00
Matthias Hanel
f5fd5e4f40 fix test timing issue and flapper caused by unnecessary pop/push 2020-12-03 21:14:04 -05:00
Ivan Kozlovic
415a7071a7 Tweaks to mqttProcessConnect()
The test TestMQTTPersistedSession() flapped once on GA. It turns
out that when the server was sending CONNACK the test was immediately
using a NATS publisher to send a message that was not received by
the MQTT subscription for the recovered session.

Sending the CONNACK before restoring subscriptions allowed for a
window where a different connection could publish and messages
would be missed. It is technically ok, I think, and test could
have been easily fixed to ensure that we don't NATS publish before
the session is fully restored.

However, I have changed the order to first restore subscriptions
then send the CONNACK. The way locking happens with MQTT subscriptions
we are sure that the CONNACK will be sent first because even if
there are inflight messages, the MQTT callbacks will have to wait
for the session lock under which the subscriptions are restored
and the CONNACK sent.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-03 17:57:51 -07:00
Derek Collison
7564768027 Added Compact to store interface for WAL functionality
Signed-off-by: Derek Collison <derek@nats.io>
2020-12-03 16:18:58 -08:00
Ivan Kozlovic
035cffae37 Add clientType() which returns NATS/MQTT/WS for CLIENT connections.
It returns NON_CLIENT if invoked from a non CLIENT connection.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-03 14:23:57 -07:00
Derek Collison
a97e84d8b9 Merge pull request #1760 from nats-io/jsbug
[FIXES] https://github.com/nats-io/jetstream/issues/396
2020-12-02 16:29:39 -08:00
Derek Collison
0f7d18d6e8 Fixes https://github.com/nats-io/jetstream/issues/396
Had a deadlock with new preconditions. We need to hold lock across Store() call but that call could call into storeUpdate() such that we may need to acquire the lock. We can enter this callback from the storage layer itself and the lock would not be held so added an atomic.

Signed-off-by: Derek Collison <derek@nats.io>
2020-12-02 16:18:00 -08:00
Ivan Kozlovic
cf9ba928ca Fixed some MQTT tests
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-02 17:00:47 -07:00
Ivan Kozlovic
67425d23c8 Add c.isMqtt() and c.isWebsocket()
This hides the check on "c.mqtt != nil" or "c.ws != nil".
Added some tests.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-02 15:52:06 -07:00