stanford-oval / genie-toolkit / 157 / 1

56%

master: 56%

push

travis-ci-com

Tokenizer: remove token length from entity tokens

The format of entity tokens returned by almond-tokenizer is:
`QUOTED_STRING_<n>*<t>` where <n> is the progressive ordinal
in the sentence, and <t> is the number of raw tokens that
correspond to this entity.

We don't make use of the token length anywhere though, and it
confuses other layers who do not expect it, so kill it.

746 of 1919 branches covered (38.87%)

Branch coverage included in aggregate %.

2322 of 4087 relevant lines covered (56.81%)

2824.06 hits per line