OpenRefine / OpenRefine / 9066005739 / 1

51%

master: 51%

push

github

Fix encoding guesser for non-UTF-8 BOM-based files. Fixes #6595 (#6596)

* Refactor InputStreamReader handling

DRY up handling of null encoding and our special UTF-8-BOM encoding

* Fix user override of encoding guessing

* Fix encoding guess for UTF-16LE & UTF-16BE with BOM. Fixes #6595

- Don't skip BOM for non-UTF-8 cases so that it is still available for character encoding guesser
- Add tests for UTF-16LE and UTF-16BE with BOMs
- Refactor encoding guesser so that it can be used for a single file
- Enhanced format guessing tests to not assume UTF-8 and instead guess encoding
  using refactored encoding guesser

* Use UTF-8 instead of US-ASCII for test fixtures

6597 of 12101 branches covered (54.52%)

Branch coverage included in aggregate %.

16785 of 25450 relevant lines covered (65.95%)

2.97 hits per line