joaoh82 / rust_sqlite / 25285323312

69%

push

github

feat(fts): 8c — cell-encoded persistence + on-demand v4→v5 bump (#80)

Phase 8 sub-phase 8c per docs/phase-8-plan.md. Persists FTS posting
lists as cell-encoded pages so save/reopen restores the index
bit-for-bit instead of re-tokenizing rows. Mirrors Phase 7d.3's
HNSW persistence shape across every layer.

User-visible: zero-friction. Existing v4 databases (no FTS) keep
writing v4 headers; the first save with an FTS index attached
promotes the file to v5. v5 readers handle both formats.

New:

- src/sql/pager/cell.rs:
  - KIND_FTS_POSTING (0x06) — cell tag for FTS posting cells.
- src/sql/pager/fts_cell.rs (NEW):
  - FtsPostingCell encode/decode. Either a posting cell
    (term + [(rowid, term_freq)]) or, with empty term, the
    sidecar carrying the per-doc length map. Sidecar preserves
    every indexed doc — including zero-token rows — so total_docs
    stays honest in BM25 post-reopen.
- src/sql/fts/posting_list.rs:
  - serialize_doc_lengths / serialize_postings emit cell payloads
    in deterministic order.
  - from_persisted_postings reconstructs the index without
    tokenization.
- src/sql/pager/header.rs:
  - FORMAT_VERSION_V4 / FORMAT_VERSION_V5 constants
    (FORMAT_VERSION_BASELINE = V4).
  - DbHeader gains format_version field; encode_header writes
    whatever the caller picked, decode_header accepts both
    versions.
- src/sql/pager/mod.rs:
  - stage_fts_btree / stage_fts_leaves stage one FTS index as a
    TableLeaf-shaped B-Tree (sidecar first, then per-term cells).
  - load_fts_postings walks leaves and decodes back into the
    (doc_lengths, postings) shape.
  - rebuild_fts_index gains the rootpage != 0 path (cell load),
    keeping rootpage == 0 as the compatibility replay path for
    v0.1.x→v0.2.0 upgraders.
  - save_database stages each FTS index, writes its rootpage to
    sqlrite_master, and conditionally bumps the header version
    to v5 (preserving v4 when no FTS index is attached).
  - parse_fts_create_index_sql helper, ... (continued)

332 of 368 new or added lines in 4 files covered. (90.22%)

6380 of 10674 relevant lines covered (59.77%)

1.17 hits per line