OpenRefine / OpenRefine / #7634

51%

push

GitHub Actions

Switch to uniVocity CSV parser. Fixes #2268 & #1372  (#6098)

* Add TODO for error reporting and remove redundant qualifiers

* Switch to uniVocity CSV parser - Fixes #2268 Fixes #1372

Switch from ancient private fork of opencsv with our
multi-character delimiter patch to the uniVocity CSV
parser which supports string delimiters.

There are two unit test changes:

- On import, a quote character before a separator character
  is no longer stripped in "ignore quotes" mode. Instead it's
  included in the data field, which seems correct to me.
- On export, CSV/TSV tests now check that the export uses the
  system line delimiter rather than always a Unix newline (\n)

The following changes were made to e2e tests:

- Disable quote processing option before switching from
  CSV to TSV because the test data doesn't have legally
  escaped quotes when the separator isn't a comma

- Force options to be sent by project creation text fixture
  (If a tag name isn't present, no options are sent. Super weird
  side effect!)

- Escape test fixture data which contains quotes before
  sending it to be parsed as a CSV file

* Add uniVocity format guessing

As a starting point use both our separator guesser and the
CSV parsers format guesser and compare the two. We probably
need a wider study to compare the performance of the two.

86 of 86 new or added lines in 3 files covered. (100.0%)

15682 of 28047 relevant lines covered (55.91%)

0.56 hits per line