wkhere / bcl / 9223935700

91%

push

github

fix fake-eof and related behavior in lex/parse

Lexer is now sending explicit tFAIL after error, interpreted in the parser
accordingly. The assumption is that, contrary to parse errors, it doesn't
make sense to continue processing input after a lex error, as there's
usually a cascase of mess afterwards.
Even if this could be handled better, that's the silent behavior of previous
code, which was misbehaving anyway - see below.

In addition to sending tFAIL token by lexer and interpreting it in the parser
(eg. there are two final tokens now, examined by parser.matchEnd and
parser.checkEnd; the error printing is fixed as well), there is also a check
inside block parsing to return earlier after tFAIL. This can be extended
in the future for other parts of the parser.

Now about a previous subtle bug: tEOF after tERR was read by parser.advance
only because it was zero-value from closed channel.
It could be seen as 'line 1:1: error at end: ...' error message,
even though 1:1 is definitely not end of the stream.
Parser was relying on this behavior, so for example after starting
tokens with iota+1 it went into infinite loop.

29 of 34 new or added lines in 2 files covered. (85.29%)

1545 of 1723 relevant lines covered (89.67%)

0.98 hits per line