joaoh82 / rust_sqlite / 25281794338
69%

DEFAULT BRANCH: main
Ran
Jobs 1
Files 41
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 56.503%. Remained the same
25281794338

push

github

web-flow 
docs(phase-8): plan — full-text search (BM25) + hybrid retrieval (#77)

* docs(phase-8): plan — full-text search (BM25) + hybrid retrieval

Mirrors phase-7-plan.md's shape: TL;DR + sub-phases + open Q1–Q10 +
implementation order + scope estimate + risks + integration map.
Awaiting user sign-off on Q1–Q10 before any code lands.

## Sub-phases

Six chunks, modeled after Phase 7d's HNSW trio:

- **8a** — standalone algorithms (tokenizer + BM25 scorer + posting
  list), no SQL integration. ~250 LOC. Pure-Rust, easy to test.
- **8b** — SQL surface: `CREATE INDEX … USING fts(col)`, `fts_match`,
  `bm25_score`, `try_fts_probe` optimizer hook. ~250 LOC.
- **8c** — persistence: new `KIND_FTS_POSTING` cell tag + dedicated
  page tree + on-demand v4→v5 file-format bump. ~250 LOC.
- **8d** — hybrid retrieval (mostly docs + worked example showing
  BM25 + vector-distance score composition via raw arithmetic).
- **8e** — MCP `bm25_search` tool (mirrors Phase 7h's
  `vector_search`).
- **8f** — final docs sweep.

Total: ~850 engine LOC + ~800 test + ~750 doc.

## Open questions (need user sign-off)

| # | Question | Recommendation |
|---|---|---|
| Q1 | MATCH-operator syntax | Function-call shape: `fts_match(col, 'q')` (no parser changes) |
| Q2 | Multi-column FTS | Single-column per index for MVP |
| Q3 | Tokenizer | ASCII for MVP; Unicode follow-up behind a feature flag |
| Q4 | Stemming | No |
| Q5 | Stop words | No (BM25's IDF handles it) |
| Q6 | Filtered FTS | Pre-filter then scalar-eval the rest (mirrors HNSW) |
| Q7 | DELETE/UPDATE on FTS-indexed tables | needs_rebuild + rebuild at save (mirrors HNSW) |
| Q8 | Hybrid retrieval API | Raw arithmetic; no typed `hybrid_score()` function |
| Q9 | MCP `bm25_search` tool | YES, mirrors `vector_search` |
| Q10 | File-format version bump | On-demand (only when first FTS index is created) |

## Cross-refs

- `docs/_index.md` — points at the plan from the Future-work section
- `docs/roadmap.md` — Phase 8 entry now refe... (continued)

5465 of 9672 relevant lines covered (56.5%)

1.14 hits per line

Jobs
ID Job ID Ran Files Coverage
1 25281794338.1 41
56.5
 GitHub Action Run
Source Files on build 25281794338