Commit Graph

2143 Commits

Author SHA1 Message Date
Matthias Hanel
2cb5f1b391 Fix flapping unit test and incorporate more review comments
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-14 16:59:57 -05:00
Matthias Hanel
c14076b13f Incorporating review comments
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-14 15:15:20 -05:00
Matthias Hanel
2edd883a6e [added] enforcement and usage of scoped signing keys
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-14 14:52:54 -05:00
Ivan Kozlovic
0d34688c4b Merge pull request #1800 from nats-io/fix_1799
[FIXED] Monitoring endpoint `connz?auth=true` show incorrect user
2021-01-11 14:28:28 -07:00
Ivan Kozlovic
f50c655e75 [FIXED] Monitoring endpoint connz?auth=true show incorrect user
Only the user (from username/password connection method) was reported
in this monitoring endpoint. Will now report proper nkey, public key,
etc..

Resolves #1799

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2021-01-11 12:59:05 -07:00
Matthias Hanel
431b642cbe Incorporating review comments
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-07 18:19:25 -05:00
Matthias Hanel
0ff6252692 Added tests for cfg/jwt based queue restrictions and updated jwt lib
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-07 17:30:51 -05:00
Matthias Hanel
1149c4cef9 [Added] support for jwt export response threshold
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-05 19:52:52 -05:00
Matthias Hanel
4699779f72 Merge pull request #1790 from nats-io/import-rewrite
[Added] support for wildcard services and import remapping by JWT.
2021-01-05 19:43:06 -05:00
Matthias Hanel
d59b074c64 Incorporating review comments
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-05 18:45:46 -05:00
Matthias Hanel
67ca277142 Incorporating review comment
Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-04 20:02:43 -05:00
Matthias Hanel
592a6447a7 [Added] support for wildcard services and import remapping by JWT.
Imports in JWT where extended to contain a new filed LocalSubject.
This Change pulls the new JWT library version in.
It was needed as prefix did not exist in the JWT library and the
original field could not be used. The field To has been deprecated.

When LocalSubject is set, service imports can be configured the same way
they are in config. Meaning, no reversal due to the type.

This change also ensures that wildcard references in transforms are only
set in To/LocalSubject. Before, for services, $1 would have to be set in Subject.

Signed-off-by: Matthias Hanel <mh@synadia.com>
2021-01-04 14:11:36 -05:00
Ivan Kozlovic
de1e492858 Merge pull request #1787 from scottf/stream-headers-colon-no-space
stream handle headers with no space after colon
2021-01-04 09:18:02 -07:00
scottf
bd954561bf stream handle headers with no space after colon 2021-01-03 09:47:15 -05:00
Ivan Kozlovic
d24e9b75b3 Fixed GW implicit reconnection
PR #1412 had a fix for races during implicit GW reconnection.
However, the fix was a bit too simplistic in that it was checking
only if there was any inbound gateway to decide to try to reconnect
an implicit disconnected GW. We need to check the name, not only
presence of inbound GW connections.

Related to #1412

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-28 12:28:55 -07:00
Ivan Kozlovic
14aecb2202 Fixed headers support for inbound leafnode connection
The server that solicits a LeafNode connection does not send an
INFO, so the accepting side had no way to know if the remote
supports headers or not. The solicit side will now send the headers
support capability in the CONNECT protocol so that the receiving
side can mark the inbound connection with headers support based
on that and its own support for headers.

Resolves #1781

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-21 11:53:24 -07:00
Alberto Ricart
f09992a889 updated iteration of signing keys (previously a list, now a map). (#1779) 2020-12-17 13:59:18 -07:00
Matthias Hanel
c6daffbfcc [Added] ability to use jwt latency sampling properties headers/share
Signed-off-by: Matthias Hanel <mh@synadia.com>
2020-12-16 14:34:09 -05:00
Matthias Hanel
e4b06cf7da Break test up into shorter ones
Also change ttl to be nanosecond interval and run parallel

Signed-off-by: Matthias Hanel <mh@synadia.com>
2020-12-14 19:48:54 -05:00
Derek Collison
c1c9401011 Merge pull request #1775 from nats-io/idx_short_bug
Fix for https://github.com/nats-io/jetstream/issues/406
2020-12-14 08:22:30 -08:00
Derek Collison
eb403ed4d0 Merge pull request #1773 from nats-io/cycle_wc_bug
Catch condition where a serviceImport response matched the original import.
2020-12-14 08:20:55 -08:00
Derek Collison
be6289be51 Fix for https://github.com/nats-io/jetstream/issues/406
Signed-off-by: Derek Collison <derek@nats.io>
2020-12-13 11:59:50 -08:00
Derek Collison
a3f7e97f9a Catch condition where a serviceImport response matched the original import subject.
Signed-off-by: Derek Collison <derek@nats.io>
2020-12-13 10:17:29 -08:00
Ivan Kozlovic
d5f255b98e Merge pull request #1771 from nats-io/gw_ln_tls_config_reload
[FIXED] Config reload for gateways/leaf remote TLS configurations
2020-12-12 10:51:52 -07:00
Ivan Kozlovic
2d2f85267b Add fix for TestLeafNodeLoop and others
Based on timing, it is possible that the first error is about
connection refused as opposed to "Loop detected". So use a dedicated
logger to notify only when expected error is found.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-11 18:15:49 -07:00
Ivan Kozlovic
ce5f9d6683 Fixed some flappers
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-11 17:30:33 -07:00
Ivan Kozlovic
fc1521636c [FIXED] Config reload for gateways/leaf remote TLS configurations
Presence of TLS config in any remote gateway or leafnode would
cause the config reload to fail (because TLS config internal
content may change which fails the DeepEqual check).

This PR excludes the TLS configs in such case to check for
changes in gateways and leafnodes.

Although GW and LN config reload is technically supported, this
PR updates the internal remotes' TLS configuration so that
changes/updates to TLS certificates would take effect after
a configuration reload.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-11 16:56:25 -07:00
Ivan Kozlovic
1dd485531f Updates to MQTT
- Keep track of published topic to avoid conversion in some case
- Rework tracing of some MQTT protocols
- Rework topic-to-subject conversion
- Update some tests

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-10 10:55:41 -07:00
Ivan Kozlovic
25a5fa62eb Merge pull request #1741 from nats-io/fix-issuer-same-as-account
fixed bad issuer check performed against account issuer instead account subject (Name)
2020-12-08 17:12:53 -07:00
Ivan Kozlovic
c199bec7c3 Bump version
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-07 10:07:30 -07:00
Ivan Kozlovic
b048b6b3de Merge pull request #1754 from nats-io/mqtt
[ADDED] MQTT Support
2020-12-07 09:06:12 -07:00
Ivan Kozlovic
1d7c4712a5 Increase Pub performance
Essentially make publish a zero alloc. Use c.mqtt.pp as the parser
publish packet structure. Messages were initially copied because
MQTT messages don't have CR_LF but was adding it so that it worked
for NATS pub/subs and MQTT pub/subs.
Now an MQTT producer sending to NATS sub will queue CR_LF after
payload.

Here is result of benchcmp for MQTT pub runs only:
```
benchmark                                     old ns/op     new ns/op     delta
BenchmarkMQTT_QoS0_Pub_______0b_Payload-8     157           55.6          -64.59%
BenchmarkMQTT_QoS0_Pub_______8b_Payload-8     167           61.0          -63.47%
BenchmarkMQTT_QoS0_Pub______32b_Payload-8     181           65.3          -63.92%
BenchmarkMQTT_QoS0_Pub_____128b_Payload-8     243           78.1          -67.86%
BenchmarkMQTT_QoS0_Pub_____256b_Payload-8     298           95.0          -68.12%
BenchmarkMQTT_QoS0_Pub_______1K_Payload-8     604           224           -62.91%
BenchmarkMQTT_QoS1_Pub_______0b_Payload-8     1713          1314          -23.29%
BenchmarkMQTT_QoS1_Pub_______8b_Payload-8     1703          1311          -23.02%
BenchmarkMQTT_QoS1_Pub______32b_Payload-8     1722          1345          -21.89%
BenchmarkMQTT_QoS1_Pub_____128b_Payload-8     2105          1432          -31.97%
BenchmarkMQTT_QoS1_Pub_____256b_Payload-8     2148          1503          -30.03%
BenchmarkMQTT_QoS1_Pub_______1K_Payload-8     3024          1889          -37.53%

benchmark                                     old MB/s     new MB/s     speedup
BenchmarkMQTT_QoS0_Pub_______0b_Payload-8     31.76        89.91        2.83x
BenchmarkMQTT_QoS0_Pub_______8b_Payload-8     77.79        213.01       2.74x
BenchmarkMQTT_QoS0_Pub______32b_Payload-8     204.52       566.26       2.77x
BenchmarkMQTT_QoS0_Pub_____128b_Payload-8     550.65       1715.96      3.12x
BenchmarkMQTT_QoS0_Pub_____256b_Payload-8     877.77       2757.16      3.14x
BenchmarkMQTT_QoS0_Pub_______1K_Payload-8     1705.02      4607.72      2.70x
BenchmarkMQTT_QoS1_Pub_______0b_Payload-8     6.42         8.37         1.30x
BenchmarkMQTT_QoS1_Pub_______8b_Payload-8     11.16        14.49        1.30x
BenchmarkMQTT_QoS1_Pub______32b_Payload-8     24.97        31.97        1.28x
BenchmarkMQTT_QoS1_Pub_____128b_Payload-8     66.52        97.74        1.47x
BenchmarkMQTT_QoS1_Pub_____256b_Payload-8     124.78       178.27       1.43x
BenchmarkMQTT_QoS1_Pub_______1K_Payload-8     342.64       548.32       1.60x

benchmark                                     old allocs     new allocs     delta
BenchmarkMQTT_QoS0_Pub_______0b_Payload-8     3              0              -100.00%
BenchmarkMQTT_QoS0_Pub_______8b_Payload-8     3              0              -100.00%
BenchmarkMQTT_QoS0_Pub______32b_Payload-8     3              0              -100.00%
BenchmarkMQTT_QoS0_Pub_____128b_Payload-8     4              0              -100.00%
BenchmarkMQTT_QoS0_Pub_____256b_Payload-8     4              0              -100.00%
BenchmarkMQTT_QoS0_Pub_______1K_Payload-8     4              0              -100.00%
BenchmarkMQTT_QoS1_Pub_______0b_Payload-8     5              2              -60.00%
BenchmarkMQTT_QoS1_Pub_______8b_Payload-8     5              2              -60.00%
BenchmarkMQTT_QoS1_Pub______32b_Payload-8     5              2              -60.00%
BenchmarkMQTT_QoS1_Pub_____128b_Payload-8     7              3              -57.14%
BenchmarkMQTT_QoS1_Pub_____256b_Payload-8     7              3              -57.14%
BenchmarkMQTT_QoS1_Pub_______1K_Payload-8     7              3              -57.14%

benchmark                                     old bytes     new bytes     delta
BenchmarkMQTT_QoS0_Pub_______0b_Payload-8     80            0             -100.00%
BenchmarkMQTT_QoS0_Pub_______8b_Payload-8     88            0             -100.00%
BenchmarkMQTT_QoS0_Pub______32b_Payload-8     120           0             -100.00%
BenchmarkMQTT_QoS0_Pub_____128b_Payload-8     224           0             -100.00%
BenchmarkMQTT_QoS0_Pub_____256b_Payload-8     369           1             -99.73%
BenchmarkMQTT_QoS0_Pub_______1K_Payload-8     1250          31            -97.52%
BenchmarkMQTT_QoS1_Pub_______0b_Payload-8     106           28            -73.58%
BenchmarkMQTT_QoS1_Pub_______8b_Payload-8     122           28            -77.05%
BenchmarkMQTT_QoS1_Pub______32b_Payload-8     154           28            -81.82%
BenchmarkMQTT_QoS1_Pub_____128b_Payload-8     381           157           -58.79%
BenchmarkMQTT_QoS1_Pub_____256b_Payload-8     655           287           -56.18%
BenchmarkMQTT_QoS1_Pub_______1K_Payload-8     2312          1078          -53.37%
```

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-04 14:42:37 -07:00
Matthias Hanel
dc2eebcd85 removing t.Errorf 2020-12-03 21:40:06 -05:00
Matthias Hanel
f5fd5e4f40 fix test timing issue and flapper caused by unnecessary pop/push 2020-12-03 21:14:04 -05:00
Ivan Kozlovic
415a7071a7 Tweaks to mqttProcessConnect()
The test TestMQTTPersistedSession() flapped once on GA. It turns
out that when the server was sending CONNACK the test was immediately
using a NATS publisher to send a message that was not received by
the MQTT subscription for the recovered session.

Sending the CONNACK before restoring subscriptions allowed for a
window where a different connection could publish and messages
would be missed. It is technically ok, I think, and test could
have been easily fixed to ensure that we don't NATS publish before
the session is fully restored.

However, I have changed the order to first restore subscriptions
then send the CONNACK. The way locking happens with MQTT subscriptions
we are sure that the CONNACK will be sent first because even if
there are inflight messages, the MQTT callbacks will have to wait
for the session lock under which the subscriptions are restored
and the CONNACK sent.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-03 17:57:51 -07:00
Derek Collison
7564768027 Added Compact to store interface for WAL functionality
Signed-off-by: Derek Collison <derek@nats.io>
2020-12-03 16:18:58 -08:00
Ivan Kozlovic
035cffae37 Add clientType() which returns NATS/MQTT/WS for CLIENT connections.
It returns NON_CLIENT if invoked from a non CLIENT connection.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-03 14:23:57 -07:00
Derek Collison
a97e84d8b9 Merge pull request #1760 from nats-io/jsbug
[FIXES] https://github.com/nats-io/jetstream/issues/396
2020-12-02 16:29:39 -08:00
Derek Collison
0f7d18d6e8 Fixes https://github.com/nats-io/jetstream/issues/396
Had a deadlock with new preconditions. We need to hold lock across Store() call but that call could call into storeUpdate() such that we may need to acquire the lock. We can enter this callback from the storage layer itself and the lock would not be held so added an atomic.

Signed-off-by: Derek Collison <derek@nats.io>
2020-12-02 16:18:00 -08:00
Ivan Kozlovic
cf9ba928ca Fixed some MQTT tests
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-02 17:00:47 -07:00
Ivan Kozlovic
67425d23c8 Add c.isMqtt() and c.isWebsocket()
This hides the check on "c.mqtt != nil" or "c.ws != nil".
Added some tests.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-02 15:52:06 -07:00
Ivan Kozlovic
41fac39f8e Split createClient() into versions for normal, WS and MQTT clients.
This duplicate quite a bit of code, but reduces the conditionals
in the createClient() function.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-02 13:50:50 -07:00
Derek Collison
cddf23c200 Limit search depth for account cycles for imports
Signed-off-by: Derek Collison <derek@nats.io>
2020-12-02 11:44:27 -08:00
Derek Collison
9b107c0f4b Merge pull request #1759 from nats-io/acc_cycles
Better implementation to detect various cycles from account imports/exports.
2020-12-02 10:02:24 -08:00
Waldemar Quevedo
a9a6bdc04f Merge pull request #1732 from nats-io/rdn-ordering
Match DNs regardless of order when using TLS auth
2020-12-02 09:25:36 -08:00
Derek Collison
705cc0f5ea Better impl for detecting cycles between accounts
Signed-off-by: Derek Collison <derek@nats.io>
2020-12-02 08:56:19 -08:00
Ivan Kozlovic
4fc04d3f55 Revert changes to processSub()
Based on how the MQTT callback operates, it is safe to finish setup
of the MQTT subscriptions after processSub() returns. So I have
reverted the changes to processSub() which will minimize changes
to non-MQTT related code.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-01 15:38:47 -07:00
Ivan Kozlovic
3e91ef75ab Some updates based on code review
- Added non-public stream and consumer configuration options to
achieve the "no subject" and "no interest" capabilities. Had
to implement custom FileStreamInfo and FileConsumerInfo marshal/
unmarshal methods so that those non public fields can be
persisted/recovered properly.
- Restored some of JS original code (since now can use config
instead of passing booleans to the functions).
- Use RLock for deliveryFormsCycle() check (unrelated to MQTT).
- Removed restriction on creating streams with MQTT prefix.
- Preventing API deletion of internal streams and their consumers.
- Added comment on Sublist's ReverseMatch method.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-12-01 14:05:54 -07:00
Ivan Kozlovic
718c995914 Allow "nats" utility to display internal MQTT streams
MQTT streams are special in that we do not set subjects in the config
since they capture all subjects. Otherwise, we would have been forced
to create a stream on say "MQTT.>" but then all publishes would have
to be prefixed with "MQTT." in order for them to be captured.

However, if one uses the "nats" tool to inspect those streams, the tool
would fail with:

```
server response is not a valid "io.nats.jetstream.api.v1.stream_info_response" message:
(root): Must validate one and only one schema (oneOf)
config: subjects is required
config: Must validate all the schemas (allOf)
```

To solve that, if we detect that user asks for the MQTT streams, we
artificially set the returned config's subject to ">".

Alternatively, we may want to not return those streams at all, although
there may be value to see the info for mqtt streams/consumers.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-11-30 20:08:44 -07:00
Ivan Kozlovic
ac4890acba Fixed flapper
Tests dealing with MQTT "will" needed to wait for the server to
process the MQTT client close of the connection. Only then we
have the guarantee that the server produced the "will" message.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
2020-11-30 20:08:44 -07:00