Commit Graph

1654 Commits

Author SHA1 Message Date
Derek Collison
476e339cf9 Race detection fix
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 16:36:08 -07:00
Derek Collison
99d1e56aac Don't send updates to leafnodes before all subs on init
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:56 -07:00
Derek Collison
915e3cd74e Header support for Leafnodes
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:56 -07:00
Derek Collison
0129a7fa09 Header support for GWs
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:56 -07:00
Derek Collison
7b6352c991 Add timestamps to stream state for first and last, addresses #1396
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:06 -07:00
Derek Collison
5b369ad7dc Do not allow streams to overlap with our public API - https://github.com/nats-io/jetstream/issues/137
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:06 -07:00
Derek Collison
67e49148c5 Fix for race in selectMsgBlock
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:06 -07:00
Derek Collison
019c105ca7 Updates based on feedback, more tests, few bug fixes
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:06 -07:00
Derek Collison
f5ceab339a Server support for headers between routes
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:06 -07:00
Derek Collison
d8b475c4b0 Better way to retrieve header support
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:06 -07:00
Derek Collison
55c77d1e4e Added support for delivery of HMSG and support for older clients
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:06 -07:00
Derek Collison
d61f1f5d92 Add in support for client header bool in CONNECT and tests
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:06 -07:00
Derek Collison
d51566881e First pass at headers awareness for server
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:06 -07:00
R.I.Pienaar
859a36e7ac ensures that the streams value is never null
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:33:06 -07:00
R.I.Pienaar
85da492096 initialize api response structs properly
when go json marshal a unset []string it puts
null in the body and not [] which then makes
it an invalid list of strings.

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:33:06 -07:00
R.I.Pienaar
ce26c65174 fix the type hint for msg_get
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:33:06 -07:00
R.I.Pienaar
45a4d929da rename ApiPagedResponse 2020-05-19 14:33:06 -07:00
R.I.Pienaar
70670fabd5 consistency fix on stream template list response
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:33:06 -07:00
R.I.Pienaar
54f45850f9 improve some error reporting around bad JSON and bad requests
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:33:06 -07:00
R.I.Pienaar
7a1ec0ad27 adds API type hints to responses
Also standardise some names of response structures and create
reusable types that can be included into others to construct
our standard API requests and responses.

Fixes some json tags.

Updates the design document of the JSON responses to reflect
the implementation that was done

Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:33:06 -07:00
Derek Collison
9ffaf44d71 Make sure to not turn on cache when it was disable
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:06 -07:00
Derek Collison
e9b9788fbe Various bug fixes, fixes for flappers
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:06 -07:00
Derek Collison
e4c15d8680 Fix for data race
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:33:06 -07:00
Derek Collison
03aacecb81 Changed cache back to simple map.
We were using a sync.Map. This did provide a benefit with massive contention from lots of Go routines. However this is only about 2x in the crazy extremes now and with a normal map and read locks we can assist the RemoveBatch which was a cause for performance issues.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:31:56 -07:00
Derek Collison
d3a447e6e9 Be more efficient when cleaning up all consumers for a stream
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
acc8da8b6e Improve RemoveBatch by disabling cache
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
9a702c2bc7 Removed duplicate init
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
R.I.Pienaar
5fa50392f8 minor fix for json keys on consumer lists
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:27:45 -07:00
Derek Collison
37b8988643 Return 404 codes when resources are not found
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
d36dff7d2f Move list to names, and add list in for detailed info for streams and consumers
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
40900ca1a8 Fixes for Template subject API
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
5256781ba0 Addressed issues raised in https://github.com/nats-io/nats-server/pull/1369
API still only turned on for account info in disabled accounts. Issues with advisories. Plan is still to have all endpoints on in all accounts.
Stream list and Consumer list return names only, page limit increased to 1024.
Stream, Consumer and Template names limited to 256 for now.
Subject API for stream messages, delete and get, not have STREAM.MSG.
Subject API for Durable is now CONSUMER.DURABLE.
Subject API for Templates now STREAM.TEMPLATE.
All subject APIs for list reverted back, so STREAM.LIST, CONSUMER.LIST etc.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
cadd39a01c Major rewrite for NATS JetStream API
API made more consistent. Noun followed by verb.
Name arguments in request subejcts are always at the end now.
Remove enabled call, just use account info.
Getting a message directly from a stream is treated like an admin API and requires JSON request.
Deleting a message directly as well.
StreamList and ConsumerList now include details and support paging.
Streams and Consumers now contain a created field in their info.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
Derek Collison
17aca11002 Small changes to event ids, good approach though with separate lock on account
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:27:45 -07:00
R.I.Pienaar
63845b8577 add type hints to service latency, use time.Time for timestamp
Signed-off-by: R.I.Pienaar <rip@devco.net>
2020-05-19 14:26:46 -07:00
Derek Collison
5ffb500857 Be more forgiving on empty api requests, force json request for delete msg api
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
cd5ec0f2e1 Move enabled to Enabled vs JetStream in response
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
513bff051b Redesign message-based API
Removed usage of +OK and -ERR. All responses are valid json objects now and optionally can include an ApiError which will have Code and Description.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
7f41b5a6ae Added support for DiscardPolicy on streams
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
911e7ef35d Add additional fields to client info for latency
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
36d5421318 Don't do sample ack unless msg was on pending
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
50443ead89 Change to consumer info, removed state, made pending and redelivery counts
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
ae8bc2d22b Move ack to +ACK, +OK still works
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
7d7a9e8f23 Make sure messages are not redelivered if ack'd late (pull-based this can happen)
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
0bf6b2afbc Allow addition of streams and consumers (durables) with same config to be acceptable
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
a5dbc20e94 Fix for test failure
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
a7f1bca534 Additional service latency upgrades.
We now share more information about the responder and the requestor. The requestor information by default is not shared, but can be when declaring the import.

Also fixed bug for error handling on old request style requests that would always result on a 408 response.

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
b8c04c1abf Make all start times UTC for latency tracking
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
ea5e5bd364 Services rewrite #2
This contains a rewrite to the services layer for exporting and importing. The code this merges to already had a first significant rewrite that moved from special interest processing to plain subscriptions.

This code changes the prior version's dealing with reverse mapping which was based mostly on thresholds and manual pruning, with some sporadic timer usage. This version uses the jetstream branch's code that understands interest and failed deliveries. So this code is much more tuned to reacting to interest changes. It also removes thresholds and goes only by interest changes or expirations based around a new service export property, response thresholds. This allows a service provider to provide semantics on how long a response should take at a maximum.

This commit also introduces formal support for service export streamed and chunked response types send an empty message to signify EOF.

This commit also includes additions to the service latency tracking such that errors are now sent, not only successful interactions. We have added a Status field and an optional Error fields to ServiceLatency.

We support the following Status codes, these are directly from HTTP.

400 Bad Request (request did not have a reply subject)
408 Request Timeout (when system detects request interest went away, old request style to make dependable)..
503 Service Unavailable (no service responders running)
504 Service Timeout (The new response threshold expired)

Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:26:46 -07:00
Derek Collison
016331c270 Fix race condition
Signed-off-by: Derek Collison <derek@nats.io>
2020-05-19 14:24:31 -07:00