The bug occures when latency tracking is on, a requestor and responder are not connected to the same server, and the responder sends two responses for a single request.
Signed-off-by: Derek Collison <derek@nats.io>
The server reads data from a client from a go routine. When receiving
messages, it checks for matching subscriptions, and if found, would
send those messages from the producer's readLoop.
A notion of "budget" was used to make sure the server does not spend
too much time sending to clients from the producer's readLoop, however,
regardless of how small the budget was, if one of the subscription's
connection TCP buffer was full, a TCP write would block for as long
as the defined write_deadline (which is now 10 seconds).
We are removing this behavior and therefore clients (like it was the
case for other type of connections) will now always notify the
subscriber's writeLoop that data is ready to be sent, but the send
will not occur in the producer's writeLoop.
Resolves#2679
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
When encountering errors for sequence mismatches that were benign we were returning an error and not processing the rest of the entries.
This would lead to more severe sequence mismatches later on that would cause stream resets.
Also added code to deal with server restarts and the clfs fixup states which should have been reset properly.
Signed-off-by: Derek Collison <derek@nats.io>
Upon server restart a server would set the check expiration to the configured amount vs delta of next to expire.
Signed-off-by: Derek Collison <derek@nats.io>
The error message would now look like this:
```
[8672] 2021/11/01 10:56:50.251985 [ERR] [::1]:59279 - websocket handshake error: invalid value for header 'Upgrade'
```
(without this change the part `[::1]:59279 - ` would not be present)
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
- When detecting duplicate route, it was possible that a server
would lose track of the peer's gateway URL, which would prevent
it from gossiping that URL to inbound gateway connections
- When a server has gateways enabled and has as a remote its
own gateway, the monitoring endpoint `/varz` would include it
but without the "urls" array.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
When a socket is bound for IP stack protocols with port == 0, the kernel picks
a free port in a specific range and binds it; on Linux, the range can be seen
(and modified) with `sysctl net.ipv4.ip_local_port_range` or looking in
`/proc/sys/net/ipv4/ip_local_port_range`. This defaults to 32768:60999.
When binding explicit ports (for tests), don't use a port number from that
range, or there will be flaky tests as periodically that port will already be
in use from another test.
This renumbers all the JS clustering tests I found binding in that range to be
beneath that range; I checked the code to ensure the new port wasn't already in
use.
* Redact URLs before logging or returning in error
This does not affect strings which failed to parse, and in such a scenario
there's a mix of "which evil" to accept; we can't sanely find what should be
redacted in those cases, so we leave them alone for debugging.
The JWT library returns some errors for Operator URLs, but it rejects URLs
which contain userinfo, so there can't be passwords in those and they're safe.
Fixes#2597
* Test the URL redaction auxiliary functions
* End-to-end tests for secrets in debug/trace
Create internal/testhelper and move DummyLogger there, so it can be used from
the test/ sub-dir too.
Let DummyLogger optionally accumulate all log messages, not just retain the
last-seen message.
Confirm no passwords logged by TestLeafNodeBasicAuthFailover.
Change TestNoPasswordsFromConnectTrace to check all trace messages, not just the
most recent.
Validate existing trace redaction in TestRouteToSelf.
* Test for password in solicited route reconnect debug
There was a bug that would erase the sync subject for upper level catchup for streams.
Raft layer repair was ok but if that was compacted it gets kicked up to the upper layers which would fail.
Users would see "Catchup stalled" messages repeatedly and consumers that had their leaders attached to that replica would also stop working.
Changes were put in to repair the corrupt state after the fact as well, regardless of presence of fix.
Signed-off-by: Derek Collison <derek@nats.io>
