jdkato / prose / 30404977531

45%

master: 93%

Build Type

push

github

Committed by jdkato

Commit Message

fix: don't emit whitespace-only tokens

Tokenize advances past a single trailing space, so a run of two or more
left a blank span behind and emitted it as a token. "The  dog  runs."
produced a " " token between the words, which the tagger then labelled
NNP.

That breaks any consumer reasoning about token adjacency: Vale's
`sequence` check stopped matching across double spaces entirely, because
the blank token sat between the tokens the rule expected to be
neighbours.

prose v2 filtered these in addToken; the v3 rewrite dropped the check.
Restored in split, with a regression test covering runs of spaces, tabs,
newlines and mixed whitespace.

The existing fuzz target did not catch this: it verifies the offset
invariant, and a blank token satisfies it perfectly.

Signed-off-by: Joseph Kato <joseph@jdkato.io>

Coverage Stats

9 of 0 new or added lines in 0 files covered. (NaN%)

2 existing lines in 1 file now uncovered.

836 of 1985 relevant lines covered (42.12%)

1776.91 hits per line