alphagov / accessible-autocomplete / 1157

90%

master: 90%

Pull #362

travis-ci-com

Only run aria-live tests in Chrome, bump timeout

We were seeing the new integration tests fail as they didn't complete
within the set time limit of 2000ms.

This would happen when they were run through Saucelabs as done on CI
or when you have the Saucelabs set up to run locally. When Saucelabs
is not available, the tests are run through Karma/headless Chrome,
which seems to reliably pass.

Solutions 2 and 3 to keep some test coverage, while also trying to ensure that
our PRs don't start failing due to Saucelabs timer issues. With this in
mind, we should keep an eye on the tests on CI going forward.

Setting timeout on
[waitUntil](https://webdriver.io/docs/api/browser/waitUntil.html)
is optional.

However removing it throws
```
Promise was rejected for the following reason: timeout
```

-Chrome on Saucelabs would intermittently fail within 2000ms; Firefox
and IE would always fail
- On 3000ms Firefox and IE11 would still fail.
- On 4000ms IE11 was flaky.
- IE10 needed at least 6000ms to pass, whilst IE9 wouldn't pass even
at 10000ms.
- Additionally, Firefox would fail intermittently regardless of the timeout
limit set.

As Chrome seemed most reliable to run the tests with, we only run these tests
through Saucelabs only in Chrome on Windows 10.

However even then we would occassionally see the tests in Chrome fail within
the 2000ms limit.

We increased the limit to 10000ms, at which point no issues were observed.

Saucelabs runs tests in tests/integration. karma/headless chrome run tests in
tests/functional.

Tests on the original PR would have been run through karma/headless Chrome as
Saucelabs wasn't available. They seem too flaky to run through Saucelabs because
of timer issues so we investigated moving them to /tests/functional.

However we found that trying to create the interactive component for testing
was really difficult - this is probably why these tests were originally in
placed in tests/integration.

Investigate adding a file that has the sa... (continued)

191 of 243 branches covered (78.6%)

245 of 271 relevant lines covered (90.41%)

61.41 hits per line