wkhere / bcl / 9227473635

91%

push

github

fix fake-eof and related behavior in lex/parse

Lexer is now sending explicit tFAIL token after error. That token is
interpreted in the parser accordingly. The assumption is that, contrary
to parse errors, it doesn't make sense to continue processing input
after a lex error, as there's usually a cascade of mess afterwards.
Even if I could do better than this assumption, that's the silent
behavior of previous code, which was misbehaving anyway - see below.

In addition to sending tFAIL token by lexer and interpreting it in the
parser (eg. there are two final tokens now, examined by parser.matchEnd
and parser.checkEnd; the error printing is fixed as well), there is also
a check inside block parsing to return earlier after tFAIL. This can be
extended in the future for other parts of the parser.

Now about a previous subtle bug: tEOF after tERR was read by
parser.advance only because it was zero-value from closed channel.
It could be seen as 'line 1:1: error at end: ...' error message,
even though 1:1 is definitely not end of the stream.
Parser was relying on this behavior, so for example after starting
tokens with iota+1 it went into infinite loop.

29 of 34 new or added lines in 2 files covered. (85.29%)

1545 of 1723 relevant lines covered (89.67%)

0.98 hits per line