romshark / icumsg / 38001117047

98%

push

github

fix: Make appended tokens usable on their own

Tokenize appends to buffer, but tokens linked to each other by their
index in the whole buffer. The tokens of a message appended after
another one couldn't be analyzed on their own, and with the whole
buffer, Analyze and Errors walked the other message too:

    first := "{a, plural, one{x} other{y}}"
    second := "{b, select, other{z}}"
    buffer, _ := tokenizer.Tokenize(language.English, nil, first)
    n := len(buffer)
    buffer, _ = tokenizer.Tokenize(language.English, buffer, second)

    Before:
      Analyze(.., second, buffer, ..):     totalChoices=2
      Analyze(.., second, buffer[n:], ..): panic: index out of range

    After:
      Analyze(.., second, buffer[n:], ..): totalChoices=1

Links now count from the first token of their message, so buffer[n:]
equals the tokens of second tokenized on its own. Nothing changes for
a buffer that starts empty. Code that passed the whole buffer to
Token.String or Options for an appended message must pass that
message's tokens instead.

In a buffer holding the tokens of more than one message, links now
point back. Options and Analyze never step back when skipping
contents, so such a buffer can't make them loop forever.

10 of 10 new or added lines in 2 files covered. (100.0%)

1745 of 1818 relevant lines covered (95.98%)

113.21 hits per line