gristlabs / asttokens / 29176076671

95%

push

github

Fix IndexError on source mixing lone CR with LF line endings (#105) (#174)

* Treat a lone "\r" as a line boundary in LineNumbers

Fixes #105.

`ASTTokens(src, parse=True)` raised `IndexError: list index out of
range` on valid source that mixes a lone "\r" (old-Mac line ending) with
"\n" and ends with an attribute access, e.g. the minimal reproducer
"\ry.y\n".

`LineNumbers` computed line starts with a `re.M` `^`, which does not
treat a lone "\r" as a line boundary. Python's tokenizer and `ast`
module do, so AST node positions on the line after a "\r" were mapped to
a character offset past the source text and thus onto the ENDMARKER
token. `MarkTokens.handle_attr` then called `next_token` on the
ENDMARKER, walking past the end of the token list.

Compute line starts from the actual end-of-line sequences ("\r\n", "\r"
or "\n") instead, matching how source lines are numbered elsewhere.
"\r\n"- and "\n"-only sources are unaffected.

Add regression tests for the line-offset handling and for marking
tokens on CR/LF-mixed sources.

* Normalize lone carriage returns before tokenizing

* Strengthen mixed line ending tests

227 of 243 branches covered (93.42%)

Branch coverage included in aggregate %.

5 of 5 new or added lines in 2 files covered. (100.0%)

574 of 602 relevant lines covered (95.35%)

16.2 hits per line