joaoh82 / rust_sqlite / 25282544111

69%

push

github

feat(fts): 8a — standalone algorithms (tokenizer + BM25 + posting list) (#78)

Phase 8 sub-phase 8a per docs/phase-8-plan.md. Adds the standalone
inverted-index trio (no SQL integration, no persistence) under
src/sql/fts/, mirroring the role Phase 7d.1's src/sql/hnsw.rs played
for vector search:

- tokenizer.rs   — ASCII split + lowercase (Q3: ASCII MVP)
- bm25.rs        — BM25+ scoring, k1=1.5 / b=0.75 fixed (Q4 + Q5: no
                   stemming, no stop list)
- posting_list.rs — in-memory inverted index keyed on i64 rowid;
                   insert / remove / query / matches / score

Public surface is infallible (no Result wrappers, no project-error
import) and depends only on std — direct reuse target for 8b's
fts_match / bm25_score scalar fns and the try_fts_probe optimizer
hook. Single-line wire-up in src/sql/mod.rs (pub mod fts;).

22 new unit tests, all inline #[cfg(test)]:
- tokenizer: empty/punctuation/lowercase/alphanumeric/non-ASCII
- bm25: zero-cases, TF monotonicity, length normalization, IDF
        domination, hand-computed 3-doc reference, query-token
        compounding
- posting_list: empty/insert/remove/reinsert idempotence, multi-term
                any-term semantics, deterministic 1k-doc corpus,
                tie-break by rowid asc

Out of scope (later sub-phases):
- IndexMethod::Fts arm + executor dispatch         → 8b
- fts_match / bm25_score scalar fns                → 8b
- try_fts_probe optimizer hook                     → 8b
- KIND_FTS_POSTING cell + v4→v5 file-format bump   → 8c
- fts cargo feature gate (defer until MCP tool)    → 8e
- docs sweep                                       → 8f

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

337 of 357 new or added lines in 3 files covered. (94.4%)

5802 of 10029 relevant lines covered (57.85%)

1.15 hits per line