gogrlx/nats-server
1
0
Fork 0
mirror of https://github.com/gogrlx/nats-server.git synced 2026-04-15 02:30:40 -07:00
Code Issues Packages Projects Releases Wiki Activity
5,871 Commits 30 Branches 48 Tags
4c8110c3ff562aeaaa7a72abfa925e17a0eefef0
Commit Graph

3904 Commits

Author SHA1 Message Date
Derek Collison
4c8110c3ff Add in support for inactivity thresholds for durable consumers. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-06-14 06:51:00 -07:00
Derek Collison
a830015703 Bump to 2.8.5-beta.4 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-06-12 09:40:41 -07:00
Derek Collison
aba38501f3 Merge pull request #3184 from nats-io/fss_opt 
[IMPROVED] Performance of wildcard filtered consumer with stream with many subjects.
 2022-06-12 09:37:56 -07:00
Derek Collison
af43dd3c74 When internal system messages were destined for a queue subscriber across a route the reply subject would be empty but not nil which caused mangling of the RMSG proto. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-06-11 14:12:05 -07:00
Derek Collison
148877b2f0 In the presence of many subjects in a stream and a wildcard filter subject, fall back to linear scan if too many. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-06-11 10:12:40 -07:00
Derek Collison
0794eafa6f Bump to 2.8.5-beta.2 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-06-07 07:49:03 -07:00
Derek Collison
fb51162e37 Make error on make bytes exceeded on a pull request a 409 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-06-07 06:59:08 -07:00
Derek Collison
301eb11725 Merge pull request #3168 from nats-io/no_fds_imp 
[IMPROVED] Loaded server and low on resources like FDs.
 2022-06-06 06:01:20 -07:00
Derek Collison
b64f3095ce Republish on the republish subject, place original in a header like direct get 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-06-05 16:21:39 -07:00
Derek Collison
f15934d3fb Bump version to 2.8.5-beta.1 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-06-05 15:27:36 -07:00
Derek Collison
dfc74dd4c1 Make sure only stream leader does a republish 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-06-05 15:27:07 -07:00
Derek Collison
fddc31adb5 Merge pull request #3158 from nats-io/kv-direct-get 
[IMPROVED] Fast and Direct access to stream messages.
 2022-06-05 07:04:18 -07:00
Derek Collison
e1c8f9fb55 This improves when a server is under load or low on resources like FDs and a user is trying to delete a stream with lots of consumers. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-06-04 16:49:17 -07:00
R.I.Pienaar
52a1c542f5 export the correct subject transformer subject 
While the TransformSubject function was doing the right
thing it did not match first and so would panic for subjects
that do not match the mapping.

The map function does the right thing so this is a more
appropriate function to export.

This undoes the exporting of unsafe TransformSubject and
exports the safer Match instead.

Signed-off-by: R.I.Pienaar <rip@devco.net>
 2022-06-02 18:26:12 +02:00
Derek Collison
405de254b6 Mark meta recovering state and use to suppress api responses and api audits during restarts. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-06-01 20:06:27 -07:00
Derek Collison
d8e5bcec55 Merge pull request #3159 from nats-io/kv-fs-race-bug 
[FIXED] When stores and load for last for subject where concurrent and compet…
 2022-06-01 12:22:39 -07:00
Derek Collison
0979bce543 Updates based on group feedback. 
1. Do not use original subject since this could use Request() and we want to use muxing.
2  Place original subject and timestamp into headers.

Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-31 19:15:52 -07:00
Derek Collison
1fe1f458d1 Make sure we do not resend responses on restart for certain clustered consumers. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-31 18:57:05 -07:00
Derek Collison
6cc14ff84d When stores and load for last for subject where concurrent and competiting for the same msg, we could fail to retrieve a newly placed message. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-31 18:04:05 -07:00
Derek Collison
c8a730ce55 Stream get for KV was going through API layer, but with popularity needed a more peformant and lighter weight and direct approach. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-30 16:34:54 -07:00
Derek Collison
e08f6d863d Allow for republish to be headers only 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-30 12:05:17 -07:00
Derek Collison
5592315e89 Suppress consumer create and R1 stream update advisories on server restart. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-30 09:58:35 -07:00
Derek Collison
daa4b97eeb Don't do advisories or API stats for a direct get msg from a stream. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-30 09:32:07 -07:00
R.I.Pienaar
0bc1d96f65 Merge pull request #3154 from ripienaar/subject_transformer 
Export the subject transformer
 2022-05-27 22:58:42 +02:00
R.I.Pienaar
dc9d6776f8 Export the subject transformer 
This exports the one key function of the subject transformer
allowing external tools to be written to test mappings are
valid and see how they would interact without the hassle of
configuring a serrver

The APIs are specifically marked as being unsupported and
having kept the transform struct itself unexported one can
not cast from the interface to the real implementation

Signed-off-by: R.I.Pienaar <rip@devco.net>
 2022-05-27 10:33:59 +02:00
Ivan Kozlovic
72d45a046d Bump version to v2.8.5-beta 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-26 17:05:51 -06:00
Ivan Kozlovic
de52c0ba22 Release v2.8.4 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-26 10:15:52 -06:00
Ivan Kozlovic
a52f12613e Bump version to v2.8.4-beta.2 and fix flapper 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-25 13:25:45 -06:00
Ivan Kozlovic
1c984d40f6 Merge pull request #3150 from nats-io/mqtt_sess_diff_domains 
[FIXED] MQTT: Sessions with same ID in different domains were considered duplicates
 2022-05-25 13:06:32 -06:00
Ivan Kozlovic
b344519176 [FIXED] MQTT: Same session ID in different domains were considered duplicates 
There is a mechanism to detect if a connection somewhere in the
cluster is using the session ID of an existing one, and if so,
close one as a duplicate.
However, when different domains are used, they should not be considered
duplicates.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-25 11:16:51 -06:00
Derek Collison
72ed48d096 Merge pull request #3149 from nats-io/pull_perf_stable 
[FIXED] Spurious pull consumer 408s under load
 2022-05-25 09:44:27 -07:00
Derek Collison
d69394efad Fix spurious 408s under load and move processing of acks to their own Go routine. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-25 09:27:34 -07:00
Derek Collison
46f7f7bfc9 Consumer pending was not correct when stream had max msgs per subject set > 1 and a consumer that filtered out part of the stream was created. 
Also make sure to update stream's config on a stream restore in case of changes.

Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-24 14:44:15 -07:00
Ivan Kozlovic
7bdd6799b7 Bump version to dev v2.8.4-beta 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-23 15:27:40 -06:00
Ivan Kozlovic
f423f07134 Release v2.8.3 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-23 14:48:19 -06:00
Ivan Kozlovic
53e3c53d96 [FIXED] JetStream: consumer with deliver new may miss messages 
This could happen when a consumer had not sent anything to the
attached NATS subscription and there was a consumer leader
step down or server restart.

Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-23 12:01:48 -06:00
Ivan Kozlovic
65094c4ee4 Merge pull request #3142 from nats-io/js_routed_api_reqs 
[FIXED] Fast routed JetStream API requests were dropped
 2022-05-23 11:54:54 -06:00
Ivan Kozlovic
66b1b51182 [FIXED] MQTT: Errors deleting consumers will now prevent deletion of session 
When there was a failure to delete a QoS1 consumer, the session
would still be deleted, which would cause orphaned consumers.

In case of error, the session record will not be deleted, which means
that it is still possible to restart the session and then close
it (with the clean flag).

Relates to #3116

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-23 11:28:18 -06:00
Ivan Kozlovic
4bf81420e2 [FIXED] Fast routed JetStream API requests were dropped 
If a JS API request is received from a non client connection, it
was processed in its own go routine. To reduce the number of
such go routine, we were limiting the number of outstanding routines
to 4096. However, in some situations, it was possible to issue
many requests at the same time that would then cause those requests
to be dropped.

(an example was an MQTT benchmark tool that would create 5000
sessions, each with one QoS1 R1 consumer (with the use of consumer_replicas=1).
On abrupt exit of the tool, the consumers and their sessions needed
to be deleted. Since would cause fast incoming delete consumer requests
which would cause the original code to drop some of them)

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-23 11:15:55 -06:00
Derek Collison
75129609bb Merge pull request #3137 from nats-io/consumer_num_pending 
Consumer's num pending can now rely on the stream's store
 2022-05-21 11:43:44 -07:00
Derek Collison
790d643431 Consumer's num pending can now rely on the stream's store vs trying to maintain furing runtime which could be wrong under certain conditions. 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-20 08:45:43 -07:00
Matthias Hanel
114474562c removed redundant republish code (#3138) 
was not removed when moving the code into checkStreamCfg

Signed-off-by: Matthias Hanel <mh@synadia.com>
 2022-05-19 18:36:28 -04:00
Derek Collison
a582e38bc6 Merge pull request #3135 from nats-io/raft-speedup-tests 
Speed up raft for tests
 2022-05-18 16:40:36 -07:00
Derek Collison
ef3eea4d73 Speed up raft for tests 
Signed-off-by: Derek Collison <derek@nats.io>
 2022-05-18 16:28:58 -07:00
Ivan Kozlovic
68792b678b Fix new MQTT test flapper 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-18 16:36:07 -06:00
Ivan Kozlovic
8ca3d2f7f5 [FIXED] JetStream: data race with account's jsLimits 
Saw this data race report:
```
=== RUN   TestJetStreamJWTClusteredDeleteTierWithStreamAndMove
==================
WARNING: DATA RACE
Write at 0x00c000d88a60 by goroutine 86:
  github.com/nats-io/nats-server/v2/server.(*Server).updateAccountClaimsWithRefresh()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/accounts.go:3299 +0x4532
  github.com/nats-io/nats-server/v2/server.(*Server).UpdateAccountClaims()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/accounts.go:2932 +0x45
  github.com/nats-io/nats-server/v2/server.(*Server).updateAccountWithClaimJWT()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1490 +0x3b4
  github.com/nats-io/nats-server/v2/server.(*Server).updateAccount()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1463 +0x1f0
  github.com/nats-io/nats-server/v2/server.(*Server).lookupAccount()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1428 +0x168
  github.com/nats-io/nats-server/v2/server.(*Server).LookupAccount()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1448 +0x2b9
  github.com/nats-io/nats-server/v2/server.(*Server).getRequestInfo()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_api.go:834 +0x28d
  github.com/nats-io/nats-server/v2/server.(*Server).jsStreamCreateRequest()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_api.go:1183 +0xca
  github.com/nats-io/nats-server/v2/server.(*Server).jsStreamCreateRequest-fm()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_api.go:1179 +0xcc
  github.com/nats-io/nats-server/v2/server.(*jetStream).apiDispatch.func1()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_api.go:717 +0x125
Previous read at 0x00c000d88a60 by goroutine 60:
  github.com/nats-io/nats-server/v2/server.(*Server).configJetStream()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream.go:638 +0x59
  github.com/nats-io/nats-server/v2/server.(*Server).updateAccountClaimsWithRefresh()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/accounts.go:3332 +0x48b3
  github.com/nats-io/nats-server/v2/server.(*Server).UpdateAccountClaims()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/accounts.go:2932 +0x45
  github.com/nats-io/nats-server/v2/server.(*Server).updateAccountWithClaimJWT()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1490 +0x3b4
  github.com/nats-io/nats-server/v2/server.(*Server).updateAccount()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1463 +0x1f0
  github.com/nats-io/nats-server/v2/server.(*Server).lookupAccount()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1428 +0x168
  github.com/nats-io/nats-server/v2/server.(*Server).LookupAccount()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1448 +0x2c8
  github.com/nats-io/nats-server/v2/server.(*jetStream).processStreamAssignment()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:2422 +0x2b6
  github.com/nats-io/nats-server/v2/server.(*jetStream).applyMetaEntries()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:1416 +0x7e4
  github.com/nats-io/nats-server/v2/server.(*jetStream).monitorCluster()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:896 +0xc75
  github.com/nats-io/nats-server/v2/server.(*jetStream).monitorCluster-fm()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:822 +0x39
Goroutine 86 (running) created at:
  github.com/nats-io/nats-server/v2/server.(*jetStream).apiDispatch()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_api.go:716 +0x8fc
  github.com/nats-io/nats-server/v2/server.(*jetStream).apiDispatch-fm()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_api.go:658 +0xcc
  github.com/nats-io/nats-server/v2/server.(*client).deliverMsg()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:3175 +0xbde
  github.com/nats-io/nats-server/v2/server.(*client).processMsgResults()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:4168 +0xf9e
  github.com/nats-io/nats-server/v2/server.(*client).processInboundRoutedMsg()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/route.go:443 +0x2ce
  github.com/nats-io/nats-server/v2/server.(*client).processInboundMsg()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:3491 +0x79
  github.com/nats-io/nats-server/v2/server.(*client).parse()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/parser.go:497 +0x3886
  github.com/nats-io/nats-server/v2/server.(*client).readLoop()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/client.go:1229 +0x1669
  github.com/nats-io/nats-server/v2/server.(*Server).createRoute.func1()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/route.go:1372 +0x37
Goroutine 60 (running) created at:
  github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:3013 +0x86
  github.com/nats-io/nats-server/v2/server.(*jetStream).setupMetaGroup()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:621 +0x108a
  github.com/nats-io/nats-server/v2/server.(*Server).enableJetStreamClustering()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:514 +0x20a
  github.com/nats-io/nats-server/v2/server.(*Server).enableJetStream()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream.go:401 +0x1168
  github.com/nats-io/nats-server/v2/server.(*Server).EnableJetStream()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream.go:207 +0x651
  github.com/nats-io/nats-server/v2/server.(*Server).Start()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:1746 +0x1804
  github.com/nats-io/nats-server/v2/server.RunServer·dwrap·3827()
      /home/travis/gopath/src/github.com/nats-io/nats-server/server/server_test.go:90 +0x39
==================
    testing.go:1152: race detected during execution of test
--- FAIL: TestJetStreamJWTClusteredDeleteTierWithStreamAndMove (6.31s)
```

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-18 16:22:53 -06:00
Ivan Kozlovic
da256ea15a Added consumer_memory_storage to make consumer memory based 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-18 15:53:23 -06:00
Ivan Kozlovic
1ddc5bd9f6 Added consumer_replicas (similar to stream_replicas) 
Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-18 15:53:23 -06:00
Ivan Kozlovic
5d3b1743e3 [ADDED] MQTT: Stream/Consumer replica count override 
Ability to override the stream and consumers replica count, which is by default
determined based on the cluster size.

```
mqtt {
  port: 1883
  stream_replicas: 5
  consumer_replicas: 1
}
```

The above would allow *new* MQTT streams to be created with a replicas
factor of 5 (it will be an error if the cluster does not have that
many nodes, and error will occur at runtime when the first client
on a given account connects), and new consumers would be R=1.

The MQTT existing streams/consumers for an account are not modified.

The stream_replicas can also obviously be reduced to 1 for a cluster
of 3 nodes if one desire to have those streams as R=1.

A value of 0 or negative is considered letting the server pick
the value (from 1 to 3 depending on standalone/cluster size).

There is another property that allows the consumers to be created
with memory storage instead of file:
```
mqtt {
  ..
  consumer_memory_storage: true
}
```

Those new settings are global and apply to new streams/consumers
only.

Related to #3116

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>

Update warning

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
 2022-05-18 15:50:23 -06:00
Derek Collison
938caf9963 Merge pull request #3131 from nats-io/consumer-memory-override 
Make sure consumer store is memory based when selected.
 2022-05-18 13:04:57 -07:00