alphagov / accessible-autocomplete
90%
master: 90%

LAST BUILD BRANCH: bump-deps
DEFAULT BRANCH: master
Repo Added
Files 4
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH fix-saucelabs-tests
branch: fix-saucelabs-tests
CHANGE BRANCH
x

1159

push

travis-ci-com

hannalaakso 
Only run aria-live tests in Chrome, bump timeout

We were seeing the new integration tests fail as they didn't complete
within the set time limit of 2000ms.

This would happen when they were run through Saucelabs as done on CI
or when you have the Saucelabs set up to run locally. When Saucelabs
is not available, the tests are run through Karma/headless Chrome,
which seems to reliably pass.

Solutions 2 and 3 to keep some test coverage, while also trying to ensure that
our PRs don't start failing due to Saucelabs timer issues. With this in
mind, we should keep an eye on the tests on CI going forward.

Setting timeout on
[waitUntil](https://webdriver.io/docs/api/browser/waitUntil.html)
is optional.

However removing it throws
```
Promise was rejected for the following reason: timeout
```

-Chrome on Saucelabs would intermittently fail within 2000ms; Firefox
and IE would always fail
- On 3000ms Firefox and IE11 would still fail.
- On 4000ms IE11 was flaky.
- IE10 needed at least 6000ms to pass, whilst IE9 wouldn't pass even
at 10000ms.
- Additionally, Firefox would fail intermittently regardless of the timeout
limit set.

As Chrome seemed most reliable to run the tests with, we only run these tests
through Saucelabs only in Chrome on Windows 10.

However even then we would occassionally see the tests in Chrome fail within
the 2000ms limit.

We increased the limit to 10000ms, at which point no issues were observed.

Saucelabs runs tests in tests/integration. karma/headless chrome run tests in
tests/functional.

Tests on the original PR would have been run through karma/headless Chrome as
Saucelabs wasn't available. They seem too flaky to run through Saucelabs because
of timer issues so we investigated moving them to /tests/functional.

However we found that trying to create the interactive component for testing
was really difficult - this is probably why these tests were originally in
placed in tests/integration.

Investigate adding a file that has the sa... (continued)

191 of 243 branches covered (78.6%)

245 of 271 relevant lines covered (90.41%)

61.41 hits per line

Relevant lines Covered
271 RELEVANT LINES 245 COVERED LINES
61.41 HITS PER LINE
Source Files on fix-saucelabs-tests
Coverage ∆ File Lines Relevant Covered Missed Hits/Line Branch Hits Branch Misses

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
1159 fix-saucelabs-tests Only run aria-live tests in Chrome, bump timeout We were seeing the new integration tests fail as they didn't complete within the set time limit of 2000ms. This would happen when they were run through Saucelabs as done on CI or when you have the... push hannalaakso travis-ci-com pending completion  
1157 fix-saucelabs-tests Only run aria-live tests in Chrome, bump timeout We were seeing the new integration tests fail as they didn't complete within the set time limit of 2000ms. This would happen when they were run through Saucelabs as done on CI or when you have the... Pull #362 web-flow travis-ci-com pending completion  
1153 fix-saucelabs-tests Only run aria-live tests in Chrome, bump timeout ## Problem We were seeing the new integration tests fail as they didn't complete within the set time limit of 2000ms. This would happen when they were run through Saucelabs as done on CI or when ... push hannalaakso travis-ci-com pending completion  
1148 fix-saucelabs-tests Only run aria-live tests in Chrome, bump timeout ## Problem We were seeing the new integration tests fail as they didn't complete within the set time limit of 2000ms. This would happen when they were run through Saucelabs as done on CI or when ... push hannalaakso travis-ci-com pending completion  
See All Builds (688)