neuml / txtai
100%

DEFAULT BRANCH: master
Repo Added
Files 250
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH master
branch: SELECT
CHANGE BRANCH
x

coverage: 100.0%. Remained the same
22869048851

push

github

web-flow 
Fix translation chunk concatenation when showmodels is enabled (#1052)

* Fix translation chunk concatenation when showmodels is enabled

When translating long text that gets split into multiple chunks by the
tokenizer, the translate() method combines results from the same source
text segment. With showmodels=True, each result is a tuple (path, text).
Using += to combine tuples produces (path, text1, path, text2) instead
of the expected (path, text1+text2). This causes a ValueError when the
caller unpacks with "model, op = outputs[y]" since there are too many
values.

Fix by explicitly concatenating only the text portion of the tuple when
showmodels is enabled.

Signed-off-by: JiangNan <1394485448@qq.com>

* Add unit test for showmodels with chunked translation

Adds testLongTranslationWithShowmodels to verify that when text is long
enough to be chunked by the tokenizer, the showmodels result is still a
proper 3-tuple (translation, language, model) and not a malformed tuple
from incorrect concatenation.

Signed-off-by: JiangNan <1394485448@qq.com>

* Shorten test method name to fix build

Rename testLongTranslationWithShowmodels -> testShowmodelsChunked
as the previous name was too long for the test runner.

Signed-off-by: JiangNan <1394485448@qq.com>

---------

Signed-off-by: JiangNan <1394485448@qq.com>

5 of 5 new or added lines in 1 file covered. (100.0%)

9591 of 9591 relevant lines covered (100.0%)

1.0 hits per line

Relevant lines Covered
9591 RELEVANT LINES 9591 COVERED LINES
1.0 HITS PER LINE
Source Files on master

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
22869048851 master Fix translation chunk concatenation when showmodels is enabled (#1052) * Fix translation chunk concatenation when showmodels is enabled When translating long text that gets split into multiple chunks by the tokenizer, the translate() method comb... push web-flow github
100.0
22831675295 fix/translation-showmodels-chunk-concat Merge 108befa16 into 69cc7538a Pull #1052 web-flow github
100.0
22818924168 fix/translation-showmodels-chunk-concat Merge b23e3d7c3 into 69cc7538a Pull #1052 web-flow github
99.98
22683590425 master Remove coveralls build script workaround push davidmezzetti github
100.0
22286036944 master Add deterministic setting to database functions closes #1048. Add ability to index database expressions #1049 push davidmezzetti github
100.0
22285690217 master Add deterministic setting to database functions closes #1048. Add ability to index database expressions #1049 push davidmezzetti github
100.0
22256216613 master Add support for custom SQL functions with DuckDB, closes #1047 push davidmezzetti github
100.0
22238458228 master Add BB25 normalization for sparse encoders (#1046) * Add log-odds conjunction fusion for BB25 hybrid search BB25 normalization outputs calibrated probabilities, but the existing hybrid fusion uses convex combination which discards the Bayesian p... push web-flow github
100.0
22229408003 master Merge 4fc556a03 into 89299926b Pull #1046 web-flow github
100.0
22156937843 master Add additional methods push davidmezzetti github
100.0
See All Builds (1235)