Esukhia / botok / 405

94%

master: 90%

push

travis-ci

add space_as_punct argument to WordTokenizer.tokenize()

It first puts all transparent chars into separate tokens, then does the rest of the pre-processing. This way, any space or \n always end up in separate tokens, instead of being treated as part of other tokens. This is required by Dakje editor.

14 of 14 new or added lines in 2 files covered. (100.0%)

1786 of 1892 relevant lines covered (94.4%)

0.94 hits per line