23149578349
93%

Ran 16 Mar 2026 02:48PM UTC

Jobs 1

Files 258

Run time 1min

Badge

Embed ▾

Committed 16 Mar 2026 02:44PM UTC coverage: 92.906% (+0.01%) from 92.894%

Build # 23149578349

Build Type

push

github

Committed by

web-flow

Commit Message

fix!: Update BM25 tokenization regex to match single char tokens (#10814)

* fix: correct off-by-one in BM25 average document length calculation

In `write_documents`, the average document length formula used
`len(self._bm25_attr)` after the new document was already inserted,
leading to an incorrect calculation:

  Before (wrong): (new_len + avg * N) / (N + 1)  — where N already includes new doc
  After  (fixed): (new_len + avg * (N - 1)) / N

This caused _avg_doc_len to be systematically underestimated, which
affects BM25 scoring accuracy (the length normalization term `b *
doc_len / avg_doc_len` would be too large, over-penalizing longer
documents).

The delete path was already correct since it adjusts `len` after
popping from `_bm25_attr`.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: update BM25 tokenization regex to match single char tokens

* Add tests

* update BM25 tokenization regex across multiple retriever tests

* add release note

* Update score values in tests

* Update release note

---------

Co-authored-by: gambletan <tan@echooo.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Run Details

15807 of 17014 relevant lines covered (92.91%)

0.93 hits per line

Jobs

ID	Job ID	Ran	Files	Coverage
1	23149578349.1	16 Mar 2026 02:48PM UTC	258	92.91	GitHub Action Run

deepset-ai / haystack / 23149578349
93%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 23149578349

deepset-ai / haystack / 23149578349 93%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 23149578349

deepset-ai / haystack / 23149578349
93%

README BADGES
x