tarantool / tarantool / 24506
83%
master: 88%

LAST BUILD BRANCH: gh-12260-memtx-mvcc-wrong-delete-tuple-clarifying
DEFAULT BRANCH: master
Ran
Jobs 1
Files 309
Run time 37s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 82.793%. Remained the same
24506

push

travis-ci

Totktonada 
test: update test-run

Retry a failed test when it is marked as fragile (and several other
conditions are met, see below).

The test-run already allows to set a list of fragile tests. They are run
one-by-one after all parallel ones in order to eliminate possible
resource starvation and fit timings to ones when the tests pass. See
[1].

In practice this approach does not help much against our problem with
flaky tests. We decided to retry failed tests, when they are known as
flagile. See [2].

The core idea is to split responsibility: known flaky fails will not
deflect attention of a developer, but each fragile test will be marked
explicitly, trackerized and will be analyzed by the quality assurance
team.

The default behaviour is not changed: each test from the fragile list
will be run once after all parallel ones. But now it is possible to set
retries amount.

Beware: the implementation does not allow to just set retries count, it
also requires to provide an md5sum of a failed test output (so called
reject file). The idea here is to ensure that we retry the test only in
case of a known fail: not some other fail within the test.

This approach has the limitation: in case of fail a test may output an
information that varies from run to run or depend of a base directory.
We should always verify the output before put its checksum into the
configuration file.

Despite doubts regarding this approach, it looks simple and we decided
to try and revisit it if there will be a need.

See configuration example in [3].

[1]: https://github.com/tarantool/test-run/issues/187
[2]: https://github.com/tarantool/test-run/issues/189
[3]: https://github.com/tarantool/test-run/pull/217

Part of #5050

(cherry picked from commit 43482eedc)

42490 of 51321 relevant lines covered (82.79%)

975272.9 hits per line

Jobs
ID Job ID Ran Files Coverage
3 24506.3 (TARGET=coverage) 0
82.79
 Travis Job 24506.3
Source Files on build 24506
Detailed source file information is not available for this build.