Had a deadlock with new preconditions. We need to hold lock across Store() call but that call could call into storeUpdate() such that we may need to acquire the lock. We can enter this callback from the storage layer itself and the lock would not be held so added an atomic.
Signed-off-by: Derek Collison <derek@nats.io>
When a system account was configured and not the default when we did a reload we would lose the JetStream service exports.
Signed-off-by: Derek Collison <derek@nats.io>
When cluster origin code was added, a server may send LS+ with
an origin cluster name in the protocol. Parsing code from a ROUTER
connection was adjusted to understand this LS+ protocol.
However, the server was also sending an LS- with origin but the
parsing code was not able to understand that. When the unsub was
for a queue subscription, this would cause the parser to error out
and close the route connection.
This PR sends an LS- without the origin in this case (so that tracing
makes sense in term of LS+/LS- sent to a route). The receiving side
then traces appropriate LS- but processes as a normal RS-.
Resolves#1751
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
In some cases, the reply of a request message is prefixed when
going over a gateway so that if it comes back to a different
server than when the request originates, it can be routed back.
For system accounts, this routed reply subject was not tracked
so the server would reply to the inbox and may reach a server
that had not yet processed (through the route) the interest
on that inbox. If the reply came with the GW routed info, that
server would know to route it to the original server.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
We were not properly restoring our state for consumers and we also had a bug where we would not properly encode and write redelivered state.
Signed-off-by: Derek Collison <derek@nats.io>
Made several changes based on feedback.
1. Made PubAckResponse only optionally include an ApiError and not force an API type.
2. Allow FilterSubject to be set on a consumer config and cleared if it matches the only stream subject.
3. Remove LookupStream by subject, and add in filters for stream names API.
Signed-off-by: Derek Collison <derek@nats.io>
It would be convenient if we did not need to use cgo to compile for
FreeBSD for common architectures, allowing builds to be generated in
Linux CI flows.
The sysctl interface uses offsets which can vary by build architecture,
so doing this in the general case without cgo is not realistic. But we
can front-load the C work to get the offsets for a given architecture,
then use encoding/binary at run-time.
While doing this, I saw that the existing FreeBSD code was
mis-calculating `%cpu` by converting the `fixpt_t` scaled int straight
to a double without dividing by the scaling factor, so we also fix for
all other architectures by introducing a division by `FSCALE`.
The offsets-emitting code is in `freebsd.txt`, with the filename chosen
to keep the Go toolchain from picking it up and trying to compile.
The result is unsafe-free cgo-free builds for FreeBSD/amd64.
A newly introduced test (TestLeafNodeTwoRemotesBindToSameAccount)
had a server creating two remotes to the same server/account.
This test quite often show the data race:
```
go test -race -v -run=TestLeafNodeTwoRemotesBindToSameAccount ./server -count 100 --failfast
=== RUN TestLeafNodeTwoRemotesBindToSameAccount
==================
WARNING: DATA RACE
Write at 0x00c000168790 by goroutine 34:
github.com/nats-io/nats-server/v2/server.(*client).processLeafNodeConnect()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/leafnode.go:1177 +0x314
github.com/nats-io/nats-server/v2/server.(*client).processConnect()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/client.go:1719 +0x9e4
github.com/nats-io/nats-server/v2/server.(*client).parse()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/parser.go:870 +0xf88
github.com/nats-io/nats-server/v2/server.(*client).readLoop()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/client.go:1052 +0x7a5
github.com/nats-io/nats-server/v2/server.(*Server).createLeafNode.func4()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/leafnode.go:872 +0x52
Previous read at 0x00c000168790 by goroutine 32:
github.com/nats-io/nats-server/v2/server.(*client).remoteCluster()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/leafnode.go:1203 +0x42d
github.com/nats-io/nats-server/v2/server.(*Server).updateLeafNodes()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/leafnode.go:1375 +0x2cf
github.com/nats-io/nats-server/v2/server.(*client).processLeafSub()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/leafnode.go:1619 +0x858
github.com/nats-io/nats-server/v2/server.(*client).parse()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/parser.go:624 +0x5031
github.com/nats-io/nats-server/v2/server.(*client).readLoop()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/client.go:1052 +0x7a5
github.com/nats-io/nats-server/v2/server.(*Server).createLeafNode.func4()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/leafnode.go:872 +0x52
Goroutine 34 (running) created at:
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/server.go:2627 +0xc7
github.com/nats-io/nats-server/v2/server.(*Server).createLeafNode()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/leafnode.go:872 +0xf7a
github.com/nats-io/nats-server/v2/server.(*Server).startLeafNodeAcceptLoop.func1()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/leafnode.go:474 +0x5e
github.com/nats-io/nats-server/v2/server.(*Server).acceptConnections.func1()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/server.go:1784 +0x57
Goroutine 32 (running) created at:
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/server.go:2627 +0xc7
github.com/nats-io/nats-server/v2/server.(*Server).createLeafNode()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/leafnode.go:872 +0xf7a
github.com/nats-io/nats-server/v2/server.(*Server).startLeafNodeAcceptLoop.func1()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/leafnode.go:474 +0x5e
github.com/nats-io/nats-server/v2/server.(*Server).acceptConnections.func1()
/Users/ivan/dev/go/src/github.com/nats-io/nats-server/server/server.go:1784 +0x57
==================
testing.go:965: race detected during execution of test
--- FAIL: TestLeafNodeTwoRemotesBindToSameAccount (0.05s)
```
This is because as soon as a LEAF is registered with the account, it is available
in the account's lleafs map, even before the CONNECT for this connectio is processed.
If another LEAF connection is processing a LSUB, the code goes over all leaf connections
for the account and may find the new connection that is in the process of connecting.
The check accesses c.leaf.remoteCluster unlocked which is also set unlocked during
the CONNECT. The fix is to have the set and check on that particular location using
the client's lock.
Ideally I believe that the connection should not have been in the account's lleafs,
or at least not used until the CONNECT for this leaf connection is fully processed.
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
If a LeafNode message is sent across a route, and the message
does not fit in the buffer, the parser would incorrectly process
the "pub args" as if it was a ROUTED message, not a LEAF message.
This caused clonePubArg() to return an error that would cause
the parser to end with a protocol violation.
Keep track that we are processing an LMSG so that we can pass
that information to clonePubArg() and do proper parsing in split
scenario.
Resolves#1743
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
