gogrlx/nats-server
1
0
Fork 0
mirror of https://github.com/gogrlx/nats-server.git synced 2026-04-02 03:38:42 -07:00
Code Issues Packages Projects Releases Wiki Activity
5,837 Commits 30 Branches 48 Tags
725ea8a5ed8383a48a4d7b9d33a6094b385cbfbf
Commit Graph

5837 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ivan Kozlovic
725ea8a5ed Update go.mod dependencies 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-26 17:16:37 -06:00
Ivan Kozlovic
72d45a046d Bump version to v2.8.5-beta 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-26 17:05:51 -06:00
Ivan Kozlovic
66524ed715 Merge pull request #3152 from nats-io/release_2_8_4 
Release v2.8.4
 2022-05-26 10:55:22 -06:00
Ivan Kozlovic
de52c0ba22 Release v2.8.4 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-26 10:15:52 -06:00
Ivan Kozlovic
a52f12613e Bump version to v2.8.4-beta.2 and fix flapper 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-25 13:25:45 -06:00
Ivan Kozlovic
1c984d40f6 Merge pull request #3150 from nats-io/mqtt_sess_diff_domains 
[FIXED] MQTT: Sessions with same ID in different domains were considered duplicates
 2022-05-25 13:06:32 -06:00
Ivan Kozlovic
b344519176 [FIXED] MQTT: Same session ID in different domains were considered duplicates 
There is a mechanism to detect if a connection somewhere in the
cluster is using the session ID of an existing one, and if so,
close one as a duplicate.
However, when different domains are used, they should not be considered
duplicates.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-25 11:16:51 -06:00
Derek Collison
72ed48d096 Merge pull request #3149 from nats-io/pull_perf_stable 
[FIXED] Spurious pull consumer 408s under load
 2022-05-25 09:44:27 -07:00
Derek Collison
d69394efad Fix spurious 408s under load and move processing of acks to their own Go routine. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-25 09:27:34 -07:00
Derek Collison
752c0adec5 Merge pull request #3148 from nats-io/max_msgs_per_num_pending_bug 
[FIXED] Consumer NumPending incorrect on certain stream setups.
 2022-05-24 15:53:20 -07:00
Derek Collison
46f7f7bfc9 Consumer pending was not correct when stream had max msgs per subject set > 1 and a consumer that filtered out part of the stream was created. 
Also make sure to update stream's config on a stream restore in case of changes.

Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-24 14:44:15 -07:00
Ivan Kozlovic
7bdd6799b7 Bump version to dev v2.8.4-beta 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-23 15:27:40 -06:00
Ivan Kozlovic
7e9b6047a4 Merge pull request #3146 from nats-io/release_2_8_3 
Release v2.8.3
 2022-05-23 15:06:35 -06:00
Ivan Kozlovic
f423f07134 Release v2.8.3 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-23 14:48:19 -06:00
Derek Collison
0ebc1cc797 Merge pull request #3143 from nats-io/js_cons_dlv_new_restart_or_step_down 
[FIXED] JetStream: consumer with deliver new may miss messages
 2022-05-23 11:22:19 -07:00
Ivan Kozlovic
53e3c53d96 [FIXED] JetStream: consumer with deliver new may miss messages 
This could happen when a consumer had not sent anything to the
attached NATS subscription and there was a consumer leader
step down or server restart.

Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-23 12:01:48 -06:00
Ivan Kozlovic
65094c4ee4 Merge pull request #3142 from nats-io/js_routed_api_reqs 
[FIXED] Fast routed JetStream API requests were dropped
 2022-05-23 11:54:54 -06:00
Ivan Kozlovic
47359329b5 Merge pull request #3144 from nats-io/mqtt_del_cons_errors 
[FIXED] MQTT: Errors deleting consumers will now prevent deletion of session
 2022-05-23 11:54:21 -06:00
Ivan Kozlovic
66b1b51182 [FIXED] MQTT: Errors deleting consumers will now prevent deletion of session 
When there was a failure to delete a QoS1 consumer, the session
would still be deleted, which would cause orphaned consumers.

In case of error, the session record will not be deleted, which means
that it is still possible to restart the session and then close
it (with the clean flag).

Relates to #3116

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-23 11:28:18 -06:00
Ivan Kozlovic
4bf81420e2 [FIXED] Fast routed JetStream API requests were dropped 
If a JS API request is received from a non client connection, it
was processed in its own go routine. To reduce the number of
such go routine, we were limiting the number of outstanding routines
to 4096. However, in some situations, it was possible to issue
many requests at the same time that would then cause those requests
to be dropped.

(an example was an MQTT benchmark tool that would create 5000
sessions, each with one QoS1 R1 consumer (with the use of consumer_replicas=1).
On abrupt exit of the tool, the consumers and their sessions needed
to be deleted. Since would cause fast incoming delete consumer requests
which would cause the original code to drop some of them)

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-23 11:15:55 -06:00
Derek Collison
75129609bb Merge pull request #3137 from nats-io/consumer_num_pending 
Consumer's num pending can now rely on the stream's store
 2022-05-21 11:43:44 -07:00
Derek Collison
790d643431 Consumer's num pending can now rely on the stream's store vs trying to maintain furing runtime which could be wrong under certain conditions. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-20 08:45:43 -07:00
Matthias Hanel
114474562c removed redundant republish code (#3138) 
was not removed when moving the code into checkStreamCfg

Signed-off-by: Matthias Hanel <mh@synadia.com>
 2022-05-19 18:36:28 -04:00
Derek Collison
a582e38bc6 Merge pull request #3135 from nats-io/raft-speedup-tests 
Speed up raft for tests
 2022-05-18 16:40:36 -07:00
Derek Collison
ef3eea4d73 Speed up raft for tests 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-18 16:28:58 -07:00
Ivan Kozlovic
1f8d9bbed1 Merge pull request #3134 from nats-io/js_data_race 
[FIXED] JetStream: data race with account's jsLimits
 2022-05-18 17:04:29 -06:00
Ivan Kozlovic
68792b678b Fix new MQTT test flapper 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-18 16:36:07 -06:00
Ivan Kozlovic
8ca3d2f7f5 [FIXED] JetStream: data race with account's jsLimits 
Saw this data race report:
```
=== RUN   TestJetStreamJWTClusteredDeleteTierWithStreamAndMove
==================
WARNING: DATA RACE
Write at 0x00c000d88a60 by goroutine 86:
  github.com/nats-io/nats-server/v2/server.(*Server).updateAccountClaimsWithRefresh()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/accounts.go:3299 +0x4532
  github.com/nats-io/nats-server/v2/server.(*Server).UpdateAccountClaims()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/accounts.go:2932 +0x45
  github.com/nats-io/nats-server/v2/server.(*Server).updateAccountWithClaimJWT()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1490 +0x3b4
  github.com/nats-io/nats-server/v2/server.(*Server).updateAccount()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1463 +0x1f0
  github.com/nats-io/nats-server/v2/server.(*Server).lookupAccount()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1428 +0x168
  github.com/nats-io/nats-server/v2/server.(*Server).LookupAccount()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1448 +0x2b9
  github.com/nats-io/nats-server/v2/server.(*Server).getRequestInfo()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_api.go:834 +0x28d
  github.com/nats-io/nats-server/v2/server.(*Server).jsStreamCreateRequest()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_api.go:1183 +0xca
  github.com/nats-io/nats-server/v2/server.(*Server).jsStreamCreateRequest-fm()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_api.go:1179 +0xcc
  github.com/nats-io/nats-server/v2/server.(*jetStream).apiDispatch.func1()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_api.go:717 +0x125
Previous read at 0x00c000d88a60 by goroutine 60:
  github.com/nats-io/nats-server/v2/server.(*Server).configJetStream()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream.go:638 +0x59
  github.com/nats-io/nats-server/v2/server.(*Server).updateAccountClaimsWithRefresh()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/accounts.go:3332 +0x48b3
  github.com/nats-io/nats-server/v2/server.(*Server).UpdateAccountClaims()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/accounts.go:2932 +0x45
  github.com/nats-io/nats-server/v2/server.(*Server).updateAccountWithClaimJWT()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1490 +0x3b4
  github.com/nats-io/nats-server/v2/server.(*Server).updateAccount()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1463 +0x1f0
  github.com/nats-io/nats-server/v2/server.(*Server).lookupAccount()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1428 +0x168
  github.com/nats-io/nats-server/v2/server.(*Server).LookupAccount()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1448 +0x2c8
  github.com/nats-io/nats-server/v2/server.(*jetStream).processStreamAssignment()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:2422 +0x2b6
  github.com/nats-io/nats-server/v2/server.(*jetStream).applyMetaEntries()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:1416 +0x7e4
  github.com/nats-io/nats-server/v2/server.(*jetStream).monitorCluster()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:896 +0xc75
  github.com/nats-io/nats-server/v2/server.(*jetStream).monitorCluster-fm()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:822 +0x39
Goroutine 86 (running) created at:
  github.com/nats-io/nats-server/v2/server.(*jetStream).apiDispatch()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_api.go:716 +0x8fc
  github.com/nats-io/nats-server/v2/server.(*jetStream).apiDispatch-fm()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_api.go:658 +0xcc
  github.com/nats-io/nats-server/v2/server.(*client).deliverMsg()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:3175 +0xbde
  github.com/nats-io/nats-server/v2/server.(*client).processMsgResults()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:4168 +0xf9e
  github.com/nats-io/nats-server/v2/server.(*client).processInboundRoutedMsg()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/route.go:443 +0x2ce
  github.com/nats-io/nats-server/v2/server.(*client).processInboundMsg()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:3491 +0x79
  github.com/nats-io/nats-server/v2/server.(*client).parse()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/parser.go:497 +0x3886
  github.com/nats-io/nats-server/v2/server.(*client).readLoop()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:1229 +0x1669
  github.com/nats-io/nats-server/v2/server.(*Server).createRoute.func1()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/route.go:1372 +0x37
Goroutine 60 (running) created at:
  github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:3013 +0x86
  github.com/nats-io/nats-server/v2/server.(*jetStream).setupMetaGroup()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:621 +0x108a
  github.com/nats-io/nats-server/v2/server.(*Server).enableJetStreamClustering()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:514 +0x20a
  github.com/nats-io/nats-server/v2/server.(*Server).enableJetStream()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream.go:401 +0x1168
  github.com/nats-io/nats-server/v2/server.(*Server).EnableJetStream()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream.go:207 +0x651
  github.com/nats-io/nats-server/v2/server.(*Server).Start()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1746 +0x1804
  github.com/nats-io/nats-server/v2/server.RunServer·dwrap·3827()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server_test.go:90 +0x39
==================
    testing.go:1152: race detected during execution of test
--- FAIL: TestJetStreamJWTClusteredDeleteTierWithStreamAndMove (6.31s)
```

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-18 16:22:53 -06:00
Ivan Kozlovic
6e4a3c8314 Merge pull request #3130 from nats-io/mqtt_add_stream_replicas 
[ADDED] MQTT: Stream/Consumer replica count override
 2022-05-18 16:05:07 -06:00
Ivan Kozlovic
da256ea15a Added consumer_memory_storage to make consumer memory based 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-18 15:53:23 -06:00
Ivan Kozlovic
1ddc5bd9f6 Added consumer_replicas (similar to stream_replicas) 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-18 15:53:23 -06:00
Ivan Kozlovic
5d3b1743e3 [ADDED] MQTT: Stream/Consumer replica count override 
Ability to override the stream and consumers replica count, which is by default
determined based on the cluster size.

```
mqtt {
  port: 1883
  stream_replicas: 5
  consumer_replicas: 1
}
```

The above would allow *new* MQTT streams to be created with a replicas
factor of 5 (it will be an error if the cluster does not have that
many nodes, and error will occur at runtime when the first client
on a given account connects), and new consumers would be R=1.

The MQTT existing streams/consumers for an account are not modified.

The stream_replicas can also obviously be reduced to 1 for a cluster
of 3 nodes if one desire to have those streams as R=1.

A value of 0 or negative is considered letting the server pick
the value (from 1 to 3 depending on standalone/cluster size).

There is another property that allows the consumers to be created
with memory storage instead of file:
```
mqtt {
  ..
  consumer_memory_storage: true
}
```

Those new settings are global and apply to new streams/consumers
only.

Related to #3116

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>

Update warning

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-18 15:50:23 -06:00
Derek Collison
938caf9963 Merge pull request #3131 from nats-io/consumer-memory-override 
Make sure consumer store is memory based when selected.
 2022-05-18 13:04:57 -07:00
Ivan Kozlovic
d2c32221a0 Merge pull request #3133 from nats-io/routez_add_some_conn_info 
[ADDED] Monitoring: Routez's individual route has now more info
 2022-05-18 13:53:25 -06:00
Derek Collison
41cca8d6c4 Allow proper mix and match of consumer stores and stream stores. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-18 12:51:48 -07:00
Ivan Kozlovic
5261d98781 [ADDED] Monitoring: Routez's individual route has now more info 
Added Start, LastActivity, Uptime and Idle that we normally have
in a Connz for non route connections. This info can be useful
to determine if a route is recent, etc..

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-18 13:18:53 -06:00
Derek Collison
1f4d4cf0ed Merge pull request #3132 from nats-io/raft-stable 
With use cases bringing us more data I wanted to suggest these changes.
 2022-05-18 09:30:04 -07:00
Derek Collison
906eb332fc Make sure consumer store is memory based when selected 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-17 18:48:27 -07:00
Derek Collison
bb9e942208 Merge pull request #3129 from nats-io/jetstream-republish 
Enable republishing of messages once stored in a stream.
 2022-05-17 15:54:22 -07:00
Derek Collison
e3249d8b6c Move cfg check for republish to common func 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-17 15:33:43 -07:00
Derek Collison
c166c9b199 Enable republishing of messages once stored in a stream. 
This enables lightweight distribution of messages to very large number of NATS subscribers.
We add in metadata as headers that allows for gap detection which enables initial value (via JetStream, maybe KV) and realtime NATS core updates but all globally ordered.

Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-17 15:18:54 -07:00
Derek Collison
2d30d12cd2 Merge pull request #3128 from nats-io/consumer_explicit 
Allow explicit consumer configuration of replica count and memory storage
 2022-05-16 19:12:25 -07:00
Derek Collison
50be0a6599 Allow explicit configuration of consumer's replica count and allow a consumer to force memory storage. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-16 19:03:56 -07:00
Derek Collison
77b3e0ce82 Merge pull request #3126 from nats-io/pull_max_bytes 
Support for MaxBytes for pull requests.
 2022-05-16 09:41:02 -07:00
Derek Collison
6bbc5f627c Support for MaxBytes for pull requests. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-16 08:43:33 -07:00
Derek Collison
ccd2290355 With use cases bringing us more data I wanted to suggest these changes. 
With inlining election timeout updates we double the lock contention and most likely introduced head of line issues for routes under heavy load.
Also slowing down heartbeats with so many assets being deployed in our user ecosystem, also moved the normal follower to candidate timing further out, similar to the lost quorum.
Note that the happy path transfer will still be very quick.

Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-15 09:55:22 -07:00
Derek Collison
3aa7965ad7 Bump to 2.8.3-beta.2 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-12 16:03:44 -07:00
Derek Collison
b6ebe34734 Merge pull request #3121 from nats-io/issue-3114 
General improvements to accounting for the filestore.
 2022-05-12 16:01:25 -07:00
Derek Collison
35e373f6e6 Merge pull request #3122 from nats-io/issue-3119 
Fix for #3119
 2022-05-12 16:01:00 -07:00
Derek Collison
bcecae42ac Fix for #3119 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-12 15:45:29 -07:00