Add 'chaos' test utility and 2 example tests

'Chaos' is a new a group of test that validates behavior in presence of
random failures.

Overview:
 - Introduce a 'Chaos Monkey' controller which can unleash a monkey
against a test cluster.
 - Introduce a monkey of type 'ClusterBouncer' which stops and restarts
nodes according to some configuration
 - Add 2 example tests, they ensure a cluster can survive some amount of
nodes bouncing
 - Configure the build to skip chaos tests unless explicitly requested
 - Add some test utility functions
This commit is contained in:
Marco Primi
2022-08-01 16:40:09 -07:00
parent b42b0d5a77
commit d83e0e2b25
4 changed files with 255 additions and 1 deletions

View File

@@ -35,7 +35,7 @@ elif [ "$1" = "js_tests" ]; then
# tests by using the `skip_js_cluster_tests` and `skip_js_super_cluster_tests`
# build tags.
go test -race -v -run=TestJetStream ./server -tags=skip_js_cluster_tests,skip_js_super_cluster_tests -count=1 -vet=off -timeout=30m -failfast
go test -race -v -run=TestJetStream ./server -tags=skip_js_cluster_tests,skip_js_super_cluster_tests,skip_js_cluster_chaos_tests -count=1 -vet=off -timeout=30m -failfast
elif [ "$1" = "js_cluster_tests" ]; then
@@ -51,6 +51,13 @@ elif [ "$1" = "js_super_cluster_tests" ]; then
go test -race -v -run=TestJetStreamSuperCluster ./server -count=1 -vet=off -timeout=30m -failfast
elif [ "$1" = "js_chaos_tests" ]; then
# Run JetStream chaos tests. By convention, all JS cluster chaos
# tests with `TestJetStreamChaos`.
go test -race -v -p=1 -run=TestJetStreamChaos ./server -count=1 -vet=off -timeout=30m -failfast
elif [ "$1" = "srv_pkg_non_js_tests" ]; then
# Run all non JetStream tests in the server package. We exclude the