wkhere / bcl / 13475630849

91%

push

github

lex: return syntax errors for "sticky tokens"

Sticky token sequence occurs when a token is immediately
followed by a character from a different token but really
there should be a whitespace between them.

Such cases occur for: numbers, strings, keywords and identifiers.
Namely, what is considered an error is:
- number then quote char
- number then alpha char

- keyword or ident then quote char

- quoted string then alpanum

For other tokens, it's ok to be followed by another token without whitespace,
for example:
foo[0]
foo2[1]
foo+1*3

..etc.

Sticky token errors are printed as "invalid syntax `<sequence>`".

Side effect: returning better error when hex number is followed by a dot.
Previously is was "unknown char", but in general this
character is not unknown at all, it's valid in floats.
So, the error becomes "invalid syntax" as well.

28 of 28 new or added lines in 1 file covered. (100.0%)

1569 of 1740 relevant lines covered (90.17%)

0.99 hits per line