22143565479

Committed 18 Feb 2026 02:22PM UTC coverage: 99.571% (-0.4%) from 100.0%

Build # 22143565479

Build Type

push

github

Committed by

web-flow

Commit Message

Add log-odds conjunction fusion for BB25 hybrid search (#1041)

* Add log-odds conjunction fusion for BB25 hybrid search

BB25 normalization outputs calibrated probabilities, but the existing
hybrid fusion uses convex combination which discards the Bayesian
probability semantics. This causes BB25 to regress on 4/5 BEIR datasets.

Add log-odds conjunction fusion (from "From Bayesian Inference to Neural
Computation") that correctly combines probability signals in logit space
with per-query dynamic calibration for dense cosine scores.

- scoring/normalize.py: Extract Bayesian method check into isbayes()
- scoring/base.py: Add default isbayes() returning False
- scoring/tfidf.py: Add isbayes() delegating to normalizer
- search/base.py: Add logodds(), convex(), rrf() fusion methods;
  dispatch based on isbayes()

BEIR nDCG@10 results (BB25+LogOdds vs Default):
  arguana +2.23, fiqa +2.03, scidocs +0.62, scifact +1.33, nfcorpus -1.96

* Extract Hybrid class for score fusion strategies

Move logodds, convex, and rrf fusion methods from Search into
a dedicated Hybrid class, following the same pattern as Normalize.

* Fix coding convention issues in Hybrid class for CI

- Fix black formatting: remove unnecessary parentheses, remove spaces around **
- Fix pylint too-many-branches: extract calibrate() method from logodds()
- Fix pylint unused-variable: rename score to _ in rrf()

Coverage Stats

39 of 80 new or added lines in 6 files covered. (48.75%)

9510 of 9551 relevant lines covered (99.57%)

1.0 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

97.5

/src/python/txtai/scoring/base.py

"""
Scoring module
"""


class Scoring:
    """
    Base scoring.
    """

    def __init__(self, config=None):
        """
        Creates a new Scoring instance.

        Args:
            config: input configuration
        """

        # Scoring configuration
        self.config = config if config is not None else {}

        # Transform columns
        columns = self.config.get("columns", {})
        self.text = columns.get("text", "text")
        self.object = columns.get("object", "object")

        # Vector model, if available
        self.model = None

    def insert(self, documents, index=None, checkpoint=None):
        """
        Inserts documents into the scoring index.

        Args:
            documents: list of (id, dict|text|tokens, tags)
            index: indexid offset
            checkpoint: optional checkpoint directory, enables indexing restart
        """

        raise NotImplementedError

    def delete(self, ids):
        """
        Deletes documents from scoring index.

        Args:
            ids: list of ids to delete
        """

        raise NotImplementedError

    def index(self, documents=None):
        """
        Indexes a collection of documents using a scoring method.

        Args:
            documents: list of (id, dict|text|tokens, tags)
        """

        # Insert documents
        if documents:
            self.insert(documents)

    def upsert(self, documents=None):
        """
        Convience method for API clarity. Calls index method.

        Args:
            documents: list of (id, dict|text|tokens, tags)
        """

        self.index(documents)

    def weights(self, tokens):
        """
        Builds a weights vector for each token in input tokens.

        Args:
            tokens: input tokens

        Returns:
            list of weights for each token
        """

        raise NotImplementedError

    def search(self, query, limit=3):
        """
        Search index for documents matching query.

        Args:
            query: input query
            limit: maximum results

        Returns:
            list of (id, score) or (data, score) if content is enabled
        """

        raise NotImplementedError

    def batchsearch(self, queries, limit=3, threads=True):
        """
        Search index for documents matching queries.

        Args:
            queries: queries to run
            limit: maximum results
            threads: run as threaded search if True and supported
        """

        raise NotImplementedError

    def count(self):
        """
        Returns the total number of documents indexed.

        Returns:
            total number of documents indexed
        """

        raise NotImplementedError

    def load(self, path):
        """
        Loads a saved Scoring object from path.

        Args:
            path: directory path to load scoring index
        """

        raise NotImplementedError

    def save(self, path):
        """
        Saves a Scoring object to path.

        Args:
            path: directory path to save scoring index
        """

        raise NotImplementedError

    def close(self):
        """
        Closes this Scoring object.
        """

        raise NotImplementedError

    def findmodel(self):
        """
        Returns the associated vector model used by this scoring instance, if any.

        Returns:
            associated vector model
        """

        return self.model

    def issparse(self):
        """
        Check if this scoring instance has an associated sparse keyword or sparse vector index.

        Returns:
            True if this index has an associated sparse index
        """

        raise NotImplementedError

    def isweighted(self):
        """
        Check if this scoring instance is for term weighting (i.e.) it has no associated sparse index.

        Returns:
            True if this index is for term weighting
        """

        return not self.issparse()

    def isnormalized(self):
        """
        Check if this scoring instance returns normalized scores.

        Returns:
            True if normalize is enabled, False otherwise
        """

        raise NotImplementedError

    def isbayes(self):
        """
        Check if this scoring instance uses Bayesian (BB25) normalization.

        Returns:
            True if BB25/Bayesian normalization is active, False otherwise
        """

        return False

1	"""
2	Scoring module
3	"""
4
5
6	class Scoring:	1✔
7	"""
8	Base scoring.
9	"""
10
11	def __init__(self, config=None):	1✔
12	"""
13	Creates a new Scoring instance.
14
15	Args:
16	config: input configuration
17	"""
18
19	# Scoring configuration
20	self.config = config if config is not None else {}	1✔
21
22	# Transform columns
23	columns = self.config.get("columns", {})	1✔
24	self.text = columns.get("text", "text")	1✔
25	self.object = columns.get("object", "object")	1✔
26
27	# Vector model, if available
28	self.model = None	1✔
29
30	def insert(self, documents, index=None, checkpoint=None):	1✔
31	"""
32	Inserts documents into the scoring index.
33
34	Args:
35	documents: list of (id, dict\|text\|tokens, tags)
36	index: indexid offset
37	checkpoint: optional checkpoint directory, enables indexing restart
38	"""
39
40	raise NotImplementedError	1✔
41
42	def delete(self, ids):	1✔
43	"""
44	Deletes documents from scoring index.
45
46	Args:
47	ids: list of ids to delete
48	"""
49
50	raise NotImplementedError	1✔
51
52	def index(self, documents=None):	1✔
53	"""
54	Indexes a collection of documents using a scoring method.
55
56	Args:
57	documents: list of (id, dict\|text\|tokens, tags)
58	"""
59
60	# Insert documents
61	if documents:	1✔
62	self.insert(documents)	1✔
63
64	def upsert(self, documents=None):	1✔
65	"""
66	Convience method for API clarity. Calls index method.
67
68	Args:
69	documents: list of (id, dict\|text\|tokens, tags)
70	"""
71
72	self.index(documents)	1✔
73
74	def weights(self, tokens):	1✔
75	"""
76	Builds a weights vector for each token in input tokens.
77
78	Args:
79	tokens: input tokens
80
81	Returns:
82	list of weights for each token
83	"""
84
85	raise NotImplementedError	1✔
86
87	def search(self, query, limit=3):	1✔
88	"""
89	Search index for documents matching query.
90
91	Args:
92	query: input query
93	limit: maximum results
94
95	Returns:
96	list of (id, score) or (data, score) if content is enabled
97	"""
98
99	raise NotImplementedError	1✔
100
101	def batchsearch(self, queries, limit=3, threads=True):	1✔
102	"""
103	Search index for documents matching queries.
104
105	Args:
106	queries: queries to run
107	limit: maximum results
108	threads: run as threaded search if True and supported
109	"""
110
111	raise NotImplementedError	1✔
112
113	def count(self):	1✔
114	"""
115	Returns the total number of documents indexed.
116
117	Returns:
118	total number of documents indexed
119	"""
120
121	raise NotImplementedError	1✔
122
123	def load(self, path):	1✔
124	"""
125	Loads a saved Scoring object from path.
126
127	Args:
128	path: directory path to load scoring index
129	"""
130
131	raise NotImplementedError	1✔
132
133	def save(self, path):	1✔
134	"""
135	Saves a Scoring object to path.
136
137	Args:
138	path: directory path to save scoring index
139	"""
140
141	raise NotImplementedError	1✔
142
143	def close(self):	1✔
144	"""
145	Closes this Scoring object.
146	"""
147
148	raise NotImplementedError	1✔
149
150	def findmodel(self):	1✔
151	"""
152	Returns the associated vector model used by this scoring instance, if any.
153
154	Returns:
155	associated vector model
156	"""
157
158	return self.model	1✔
159
160	def issparse(self):	1✔
161	"""
162	Check if this scoring instance has an associated sparse keyword or sparse vector index.
163
164	Returns:
165	True if this index has an associated sparse index
166	"""
167
168	raise NotImplementedError	1✔
169
170	def isweighted(self):	1✔
171	"""
172	Check if this scoring instance is for term weighting (i.e.) it has no associated sparse index.
173
174	Returns:
175	True if this index is for term weighting
176	"""
177
178	return not self.issparse()	1✔
179
180	def isnormalized(self):	1✔
181	"""
182	Check if this scoring instance returns normalized scores.
183
184	Returns:
185	True if normalize is enabled, False otherwise
186	"""
187
188	raise NotImplementedError	1✔
189
190	def isbayes(self):	1✔
191	"""
192	Check if this scoring instance uses Bayesian (BB25) normalization.
193
194	Returns:
195	True if BB25/Bayesian normalization is active, False otherwise
196	"""
197
NEW 198	return False	×

neuml / txtai / 22143565479

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous