qunitjs / qunit / 9488493013

90%

Build Type

push

github

Committed by Krinkle

Commit Message

Core: Allow late `runEnd` listener, Fix testresult-display scraping compat

In commit dbeab48c25 (https://github.com/qunitjs/qunit/pull/1760),
the assertion counts are removed from the qunit-testresult-display
element of the HTML Reporter. While this is not in any way a documented
API, it is quite common for QUnit runners that are based on
Selenium/Webdriver to use scraping to obtain a simple summary of
whether the run passed and how many failures there were.

Examples:
* https://github.com/discourse/discourse/blob/v3.2.2/spec/system/theme_qunit_spec.rb#L28
  `success_count = find("#qunit-testresult-display .passed").text`
* https://github.com/jazzband/django-sortedm2m/blob/3.1.1/test_project/qunit-runner.py#L16
  `xpath = '//div[@id="qunit-testresult-display"]/span[@class="failed"]'`
* https://github.com/negstek/frappe/blob/v8.3.6/frappe/tests/ui/test_test_runner.py#L13
  `self.driver.wait_for('#qunit-testresult-display', timeout=60)`

There are numerous downsides to down, including that raw assertion
counts do not reflect whether a test run passed. For example, "todo"
tests are considered "passing" if they contain assertion failures.
For legacy reasons, these raw assertion counts reflect those as-is.
These caveats are documented at https://qunitjs.com/api/callbacks/QUnit.done/
and this is why we promose use of test counts instead of raw assertions.

As mitigation to reduce ecosystem churn as result of QUnit 3.0:

* Intruce memory for `runEnd` event so that anything JS-based
  definitely has a stable and documented way to obtain the result in
  a machine-readable format.

* Restore the `<span class=passed>` and `<span class=failed>` elements
  in a roughly similar meaning. Notable differences include that these
  now represent reliable test counts instead of raw assertion counts.
  This means when you have "todo" tests with known failures, there
  will now be a "0" in span.failed. This is most likely to result in
  these old test runners starting to ... (continued)

Coverage Stats

5 of 0 new or added lines in 0 files covered. (NaN%)

52 existing lines in 3 files now uncovered.

1202 of 2049 relevant lines covered (58.66%)

92.71 hits per line