kp-forks / txtai

100%

push

github

Fix translation chunk concatenation when showmodels is enabled (#1052)

* Fix translation chunk concatenation when showmodels is enabled

When translating long text that gets split into multiple chunks by the
tokenizer, the translate() method combines results from the same source
text segment. With showmodels=True, each result is a tuple (path, text).
Using += to combine tuples produces (path, text1, path, text2) instead
of the expected (path, text1+text2). This causes a ValueError when the
caller unpacks with "model, op = outputs[y]" since there are too many
values.

Fix by explicitly concatenating only the text portion of the tuple when
showmodels is enabled.

Signed-off-by: JiangNan <1394485448@qq.com>

* Add unit test for showmodels with chunked translation

Adds testLongTranslationWithShowmodels to verify that when text is long
enough to be chunked by the tokenizer, the showmodels result is still a
proper 3-tuple (translation, language, model) and not a malformed tuple
from incorrect concatenation.

Signed-off-by: JiangNan <1394485448@qq.com>

* Shorten test method name to fix build

Rename testLongTranslationWithShowmodels -> testShowmodelsChunked
as the previous name was too long for the test runner.

Signed-off-by: JiangNan <1394485448@qq.com>

---------

Signed-off-by: JiangNan <1394485448@qq.com>

5 of 5 new or added lines in 1 file covered. (100.0%)

9591 of 9591 relevant lines covered (100.0%)

1.0 hits per line