anachronauts / jeff65 / 373

88%

master: 88%

push

travis-ci

Improve lexer efficiency for large tokens

Lexer now loads file by blocks, rather than lines. Technically, the
lexer is still O(n^2) on token size, but the coefficient on the
quadratic term is low enough that it shouldn't be an issue unless you
have alarmingly long block comments. (Or strings, but you'll have bigger
things to worry about than lex times then.)

If ReStream is passed an in-memory string stream, it'll bypass the
stream API. This is mainly to get around the fact that StringIO objects
don't have an underlying binary stream to detach.

Fixes a bug where a token which is partially matched at the end of the
file is emitted as if it were a full match. This was masked by the way
that comments are currently parsed, but would have been turned up later
when we switch comment syntax.

Added catchall token regexes for string and comment modes to the
gold-syntax lexer. This was unnecessary with the original line-oriented
matcher, and was then masked by the above bug in the multiline matcher.

Removed the noisy+slow "Produced token" logging line which I forgot to
remove before the last commit.

453 of 527 branches covered (85.96%)

Branch coverage included in aggregate %.

1522 of 1713 relevant lines covered (88.85%)

2.66 hits per line