jdkato / prose / 30451272144

45%

master: 93%

Build Type

push

github

Committed by jdkato

Commit Message

feat: add a dictionary-driven tagger and select taggers by name

The perceptron reads the words around the one it is tagging, which is
what makes it accurate on ordinary prose and unsteady on text that is
not. A grammar rule only ever runs on a sentence that is already wrong,
and the context there is not evidence for anything: in "they were all
ready aware of the risk" it reads `aware` as a noun.

Lexical tags from a dictionary instead. A word listed one way is
settled without reading the sentence; a word listed several ways is
left to a fallback tagger, bounded to those readings; an unknown word
is the fallback's entirely.

Interface is what both satisfy, and Open builds one by name so a caller
can name a tagger in configuration rather than in code. Tagger keeps
the shape it shipped with, so this adds to the API without changing it.

Signed-off-by: Joseph Kato <joseph@jdkato.io>

Coverage Stats

96 of 107 new or added lines in 3 files covered. (89.72%)

26 existing lines in 1 file now uncovered.

960 of 2116 relevant lines covered (45.37%)

1679.18 hits per line