Execute all tests even if some of them fail

[rluetzner]

Is your feature request related to a problem? Please describe.
Whenever the Mint test suite is executed, a single failing test from one of the test packages causes all the remaining tests from that package to be skipped.

Describe the solution you'd like
There should be a way, preferably by setting an environment variable, to execute all tests from all packages independent of failures.

Describe alternatives you've considered
None.

Additional context
None.

[aweisser]

@andrebruch Please start to evaluate this on a "minio gateway nas /data" setup.
In case of questions also @rluetzner can help, if I'm not responding.

[iternity-dotcom]

In order to solve some of the Mint issues I created forks of some of the Minio clients. 
https://github.com/iternity-dotcom/minio-dotnet
https://github.com/iternity-dotcom/minio-java
https://github.com/iternity-dotcom/minio-go
 
 @andrebruch Does the other clients do not suffer from this behavoir? Are they running through all test even if the first test is failing?

[rluetzner]

Most of the tests are already located in the Mint folder, but especially tests relating to MinIO specific client implementations are not part of the test suite. They are instead downloaded from GitHub directly and are located in the projects you forked, @aweisser .

[aweisser]

But do we also need to adjust some of the client tests included in the Minio repo?

[rluetzner]

We should only have to adjust tests that are run as part of the Mint suite and none of the client's unit tests. At least the functional_tests.go file from minio-go already references a Mint specific environment variable that is otherwise unused in the project. We could try to solve this in a similar fashion.

[aweisser]

Sure. Only system tests that are part of Mint. Not the clients unit tests. That’s clear.

I just want to know which of the following parts of the Mint Suite are affected by this change (no matter where they are implemented):

aws-sdk-go
aws-sdk-java
aws-sdk-php
aws-sdk-ruby
awscli
healthcheck
mc
minio-dotnet
minio-go
minio-java
minio-js
minio-py
s3cmd
s3select
security
versioning

[andrebruch]

@aweisser I have only considered the clients that give one failed test when running Mint on the minio gateway nas /data setup:
minio-dotnet: ListenBucketNotificationsAsync_Test1
minio-go: testSSES3EncryptionPutGet()
minio-java: setBucketEncryption()

Mint runs the functional_test.go file from the latest minio-go release:

minio/mint/build/minio-go/install.sh

Line 18 in 922c7b5

MINIO_GO_VERSION=$(curl --retry 10 -Ls -o /dev/null -w "%{url_effective}" https://github.com/minio/minio-go/releases/latest | sed "s/https:\/\/github.com\/minio\/minio-go\/releases\/tag\///")

minio/mint/build/minio-go/install.sh

Lines 24 to 26 in 922c7b5

	test_run_dir="$MINT_RUN_CORE_DIR/minio-go"
	curl -sL -o "${test_run_dir}/main.go" "https://raw.githubusercontent.com/minio/minio-go/${MINIO_GO_VERSION}/functional_tests.go"
	(cd "$test_run_dir" && GO111MODULE=on CGO_ENABLED=0 go build -o minio-go main.go)

and runs the following tests in full mode:
https://github.com/iternity-dotcom/minio-go/blob/92fe50d14294782d96402deb861d442992038109/functional_tests.go#L11218-L11312

[aweisser]

@andrebruch Are you done with this issue? Can we start to create some PRs?

[andrebruch]

@aweisser Yes we can start to create some PRs:
https://github.com/iternity-dotcom/minio/tree/mint/run-on-fail
https://github.com/iternity-dotcom/minio-go/tree/run-on-fail
https://github.com/iternity-dotcom/minio-java/tree/run-on-fail

[iternity-dotcom]

@rluetzner Could you please check if the actualle issue "Execute all tests even if some of them fail" is already solved?

[andrebruch]

@aweisser @rluetzner

There is an open issue in the Mint repo from us:
minio/mint#325

Open PRs
minio/minio-go#1475
minio/minio-java#1180

Merged PR
minio/mint@69e4bf1

A PR still needs to be created:
https://github.com/minio/mint/compare/master...iternity-dotcom:fix-description?expand=1
I sent you @aweisser the prepared PR on Teams.

[rluetzner]

@andrebruch Thanks for linking the issues.