romshark / icumsg / 38147566563

98%

push

github

fix: Return buffer unchanged when Tokenize fails

On error, return buffer as passed, without the tokens of the invalid
message. Some of those tokens don't link to their terminators yet, so
Token.String, Options and Analyze misbehave on them, and a caller that
tokenizes many messages into one buffer has to truncate it after each
failure:

    buf, err := tokenizer.Tokenize(language.English, nil,
        "{a, select, other{{n, plural, other{x}")

    Before: err == ErrUnexpectedEOF, len(buf) == 8, and buf[3].String
            panics with "slice bounds out of range [18:0]"
    After:  err == ErrUnexpectedEOF, len(buf) == 0

5 of 5 new or added lines in 1 file covered. (100.0%)

1736 of 1778 relevant lines covered (97.64%)

964.45 hits per line