wikimedia / parsoid / 2682

85%

push

travis-ci

c6c8b294 followup: Emit EOFTk in 'start' rule + tweak 'start_async' rule

* c6c8b294 added an unnecessary check to the start_async rule to decide
  whether to terminate the loop. That check was unnecessary. I removed
  it and documented it.

* As part of c6c8b294, to support generator behavior of the Grammar and
  tokenizer, I had to remove the EOFTk append from the async grammar
  action and move it to the tokenizer. To make the grammar uniform,
  I removed it from the "start" rule as well and added it to
  PegTokenizer::tokenizeSync, but forgot to account for the fact that
  tokenizeSync is used with non-start rules => we were now emitting
  EOFTk where it wasn't expected.

  56e0335a tried to address that but it was considered hacky and it
  turns out that we use the 'tokenizeAs' function in lots of places
  in the codebase which I didn't fix up at all. I was focused on the
  single crasher I ran into while testing.

  This patch reverts the change to the "start" rule and moves the
  EOFTk append from PegTkenizer::tokenizeSync back to the "start" rule.

* The codebase has a number of places where we strip an unneeded EOFTk.
  In the future, we should clean this up. The EOFTk may only be really
  needed by the HTML5TreeBuilder which runs only as part of the
  "text/mediawiki/full" and "tokens/mediawiki/expanded" pipelines.
  So, we could probably inspect pipeline type and/or rule to decide
  whether to insert the EOFTk.

  Or, an alternative would be to add a parser option to let the caller
  decide if it needs an EOFTk signal.

  This cleanup can come in the future.

Change-Id: I3048c5493

9530 of 12004 branches covered (79.39%)

14903 of 17618 relevant lines covered (84.59%)

90406.88 hits per line