• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

joaoh82 / rust_sqlite / 25093849524

29 Apr 2026 06:15AM UTC coverage: 69.044% (-1.9%) from 70.927%
25093849524

push

github

web-flow
Phase 7e: JSON column type + path queries (#54)

Adds the JSON storage class and four path-aware query functions, closing
the second of Phase 7's two storage primitives (the first was VECTOR(N)
in 7a). Shape mirrors SQLite's JSON1 extension — JSON values store as
canonical UTF-8 text, validated via `serde_json::from_str` at INSERT and
UPDATE time. Phase 7 plan Q3 originally proposed bincoded `serde_json::
Value`, but bincode was removed from the engine in Phase 3c (cell-based
encoding replaced it); rather than re-add bincode for one column type,
JSON-as-text matches SQLite's choice and reuses the existing Text storage
path. Q3 in `docs/phase-7-plan.md` records the scope correction inline.

Engine surface:

- `DataType::Json` variant alongside `Vector(N)`. `JSONB` parses as an
  alias (Postgres convention; both store as text in our case).
- INSERT/UPDATE on a JSON column runs `serde_json::from_str::<Value>`;
  malformed JSON is rejected with `Type mismatch: expected JSON for
  column 'foo': <serde error>`. NULLs pass through untouched.
- UNIQUE on a JSON column treats the value as raw text (string equality
  on the canonical form).
- `table_to_create_sql` round-trips JSON columns; `build_empty_table`,
  `Row::Text(BTreeMap::new())` storage, and the `clone_datatype` helpers
  in `executor.rs` and `pager/mod.rs` all gained the new arm.

Functions (executor.rs, ~370 LOC):

- `json_extract(json[, path])` — walks the path, returns the resolved
  node coerced to the closest SQL type. Strings → TEXT, numbers →
  INTEGER/REAL, booleans → BOOLEAN, `null` → NULL, composites
  (object/array) → canonical JSON text.
- `json_type(json[, path])` — returns one of `'object'`, `'array'`,
  `'string'`, `'integer'`, `'real'`, `'true'`, `'false'`, `'null'`.
- `json_array_length(json[, path])` — element count; errors if the
  resolved node isn't an array.
- `json_object_keys(json[, path])` — keys as a JSON-array text in
  insertion order (e.g. `'["a","b","c"]'`). Diverges f... (continued)

154 of 210 new or added lines in 5 files covered. (73.33%)

227 existing lines in 3 files now uncovered.

5382 of 7795 relevant lines covered (69.04%)

1.42 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

89.34
/src/sql/pager/mod.rs
1
//! On-disk persistence for a `Database`, using fixed-size paged files.
2
//!
3
//! The file is a sequence of 4 KiB pages. Page 0 holds the header
4
//! (magic, version, page count, schema-root pointer). Every other page carries
5
//! a small per-page header (type tag + next-page pointer + payload length)
6
//! followed by a payload of up to 4089 bytes.
7
//!
8
//! **Storage strategy (format version 2, Phase 3c.5).**
9
//!
10
//! - Each `Table`'s rows live as **cells** in a chain of `TableLeaf` pages.
11
//!   Cell layout and slot directory are in `cell.rs` / `table_page.rs`;
12
//!   cells that exceed the inline threshold spill into an overflow chain
13
//!   via `overflow.rs`.
14
//! - The schema catalog is itself a regular table named `sqlrite_master`,
15
//!   with one row per user table:
16
//!       `(name TEXT PRIMARY KEY, sql TEXT NOT NULL,
17
//!         rootpage INTEGER NOT NULL, last_rowid INTEGER NOT NULL)`
18
//!   This is the SQLite-style approach: the schema of `sqlrite_master`
19
//!   itself is hardcoded into the engine so the open path can bootstrap.
20
//! - Page 0's `schema_root_page` field points at the first leaf of
21
//!   `sqlrite_master`.
22
//!
23
//! **Format version.** Version 2 is not compatible with files produced by
24
//! earlier commits. Opening a v1 file returns a clean error — users on
25
//! old files have to regenerate them from CREATE/INSERT, as there's no
26
//! production data to migrate yet.
27

28
// Data-layer modules. Not every helper in these modules is used by save/open
29
// yet — some exist for tests, some for future maintenance operations.
30
// Module-level #[allow(dead_code)] keeps the build quiet without dotting
31
// the modules with per-item attributes.
32
#[allow(dead_code)]
33
pub mod cell;
34
pub mod file;
35
pub mod header;
36
#[allow(dead_code)]
37
pub mod hnsw_cell;
38
#[allow(dead_code)]
39
pub mod index_cell;
40
#[allow(dead_code)]
41
pub mod interior_page;
42
pub mod overflow;
43
pub mod page;
44
pub mod pager;
45
#[allow(dead_code)]
46
pub mod table_page;
47
#[allow(dead_code)]
48
pub mod varint;
49
#[allow(dead_code)]
50
pub mod wal;
51

52
use std::collections::{BTreeMap, HashMap};
53
use std::path::Path;
54
use std::sync::{Arc, Mutex};
55

56
use sqlparser::dialect::SQLiteDialect;
57
use sqlparser::parser::Parser;
58

59
use crate::error::{Result, SQLRiteError};
60
use crate::sql::db::database::Database;
61
use crate::sql::db::secondary_index::{IndexOrigin, SecondaryIndex};
62
use crate::sql::db::table::{Column, DataType, Row, Table, Value};
63
use crate::sql::pager::cell::Cell;
64
use crate::sql::pager::header::DbHeader;
65
use crate::sql::pager::index_cell::IndexCell;
66
use crate::sql::pager::interior_page::{InteriorCell, InteriorPage};
67
use crate::sql::pager::overflow::{
68
    OVERFLOW_THRESHOLD, OverflowRef, PagedEntry, read_overflow_chain, write_overflow_chain,
69
};
70
use crate::sql::pager::page::{PAGE_HEADER_SIZE, PAGE_SIZE, PAYLOAD_PER_PAGE, PageType};
71
use crate::sql::pager::pager::Pager;
72
use crate::sql::pager::table_page::TablePage;
73
use crate::sql::parser::create::CreateQuery;
74

75
// Re-export so callers can spell `sql::pager::AccessMode` without
76
// reaching into the `pager::pager::pager` submodule path.
77
pub use crate::sql::pager::pager::AccessMode;
78

79
/// Name of the internal catalog table. Reserved — user CREATEs of this
80
/// name must be rejected upstream.
81
pub const MASTER_TABLE_NAME: &str = "sqlrite_master";
82

83
/// Opens a database file in read-write mode. Shorthand for
84
/// [`open_database_with_mode`] with [`AccessMode::ReadWrite`].
85
pub fn open_database(path: &Path, db_name: String) -> Result<Database> {
2✔
86
    open_database_with_mode(path, db_name, AccessMode::ReadWrite)
2✔
87
}
88

89
/// Opens a database file in read-only mode. Acquires a shared OS-level
90
/// advisory lock, so other read-only openers coexist but any writer is
91
/// excluded. Attempts to mutate the returned `Database` (e.g. an
92
/// `INSERT`, or a `save_database` call against it) bottom out in a
93
/// `cannot commit: database is opened read-only` error from the Pager.
94
pub fn open_database_read_only(path: &Path, db_name: String) -> Result<Database> {
1✔
95
    open_database_with_mode(path, db_name, AccessMode::ReadOnly)
1✔
96
}
97

98
/// Opens a database file and reconstructs the in-memory `Database`,
99
/// leaving the long-lived `Pager` attached for subsequent auto-save
100
/// (read-write) or consistent-snapshot reads (read-only).
101
pub fn open_database_with_mode(path: &Path, db_name: String, mode: AccessMode) -> Result<Database> {
2✔
102
    let pager = Pager::open_with_mode(path, mode)?;
5✔
103

104
    // 1. Load sqlrite_master from the tree at header.schema_root_page.
105
    let mut master = build_empty_master_table();
2✔
106
    load_table_rows(&pager, &mut master, pager.header().schema_root_page)?;
4✔
107

108
    // 2. Two passes over master rows: first build every user table, then
109
    //    attach secondary indexes. Indexes need their base table to exist
110
    //    before we can populate them. Auto-indexes are created at table
111
    //    build time so we only have to load explicit indexes from disk
112
    //    (but we also reload the auto-index CONTENT because Table::new
113
    //    built it empty).
114
    let mut db = Database::new(db_name);
2✔
115
    let mut index_rows: Vec<IndexCatalogRow> = Vec::new();
2✔
116

117
    for rowid in master.rowids() {
6✔
118
        let ty = take_text(&master, "type", rowid)?;
4✔
119
        let name = take_text(&master, "name", rowid)?;
4✔
120
        let sql = take_text(&master, "sql", rowid)?;
4✔
121
        let rootpage = take_integer(&master, "rootpage", rowid)? as u32;
4✔
122
        let last_rowid = take_integer(&master, "last_rowid", rowid)?;
2✔
123

124
        match ty.as_str() {
2✔
125
            "table" => {
2✔
126
                let (parsed_name, columns) = parse_create_sql(&sql)?;
4✔
127
                if parsed_name != name {
4✔
128
                    return Err(SQLRiteError::Internal(format!(
×
129
                        "sqlrite_master row '{name}' carries SQL for '{parsed_name}' — corrupt catalog?"
130
                    )));
131
                }
132
                let mut table = build_empty_table(&name, columns, last_rowid);
4✔
133
                if rootpage != 0 {
2✔
134
                    load_table_rows(&pager, &mut table, rootpage)?;
4✔
135
                }
136
                if last_rowid > table.last_rowid {
2✔
137
                    table.last_rowid = last_rowid;
×
138
                }
139
                db.tables.insert(name, table);
4✔
140
            }
141
            "index" => {
4✔
142
                index_rows.push(IndexCatalogRow {
4✔
143
                    name,
2✔
144
                    sql,
2✔
145
                    rootpage,
146
                });
147
            }
148
            other => {
×
149
                return Err(SQLRiteError::Internal(format!(
×
150
                    "sqlrite_master row '{name}' has unknown type '{other}'"
151
                )));
152
            }
153
        }
154
    }
155

156
    // Second pass: attach each index to its table. HNSW indexes
157
    // (Phase 7d.2) take a different code path because their persisted
158
    // form is just the CREATE INDEX SQL — the graph itself isn't
159
    // persisted yet (Phase 7d.3). Detect HNSW via the SQL's USING clause
160
    // and route to a graph-rebuild instead of the B-Tree-cell load.
161
    for row in index_rows {
6✔
162
        if create_index_sql_uses_hnsw(&row.sql) {
4✔
163
            rebuild_hnsw_index(&mut db, &pager, &row)?;
2✔
164
        } else {
165
            attach_index(&mut db, &pager, row)?;
4✔
166
        }
167
    }
168

169
    db.source_path = Some(path.to_path_buf());
2✔
170
    db.pager = Some(pager);
2✔
171
    Ok(db)
2✔
172
}
173

174
/// Catalog row for a secondary index — deferred until after every table is
175
/// loaded so the index's base table exists by the time we populate it.
176
struct IndexCatalogRow {
177
    name: String,
178
    sql: String,
179
    rootpage: u32,
180
}
181

182
/// Persists `db` to disk. Same diff-commit behavior as before: only pages
183
/// whose bytes actually changed get written.
184
pub fn save_database(db: &mut Database, path: &Path) -> Result<()> {
2✔
185
    // Phase 7d.3 — rebuild any HNSW index that DELETE / UPDATE-on-vector
186
    // marked dirty. Done up front under the &mut Database borrow we
187
    // already hold, before the immutable iteration loops below need
188
    // their own borrow.
189
    rebuild_dirty_hnsw_indexes(db);
2✔
190

191
    let same_path = db.source_path.as_deref() == Some(path);
2✔
192
    let mut pager = if same_path {
2✔
193
        match db.pager.take() {
2✔
194
            Some(p) => p,
2✔
195
            None if path.exists() => Pager::open(path)?,
4✔
196
            None => Pager::create(path)?,
2✔
197
        }
198
    } else if path.exists() {
2✔
199
        Pager::open(path)?
×
200
    } else {
201
        Pager::create(path)?
2✔
202
    };
203

204
    pager.clear_staged();
2✔
205

206
    // Page 0 is the header; payload pages start at 1.
207
    let mut next_free_page: u32 = 1;
2✔
208

209
    // 1. Stage each user table's B-Tree, collecting master-row info.
210
    //    `kind` is "table" or "index" — master has one row per each.
211
    let mut master_rows: Vec<CatalogEntry> = Vec::new();
2✔
212

213
    let mut table_names: Vec<&String> = db.tables.keys().collect();
4✔
214
    table_names.sort();
4✔
215
    for name in table_names {
4✔
216
        if name == MASTER_TABLE_NAME {
4✔
217
            return Err(SQLRiteError::Internal(format!(
×
218
                "user table cannot be named '{MASTER_TABLE_NAME}' (reserved)"
219
            )));
220
        }
221
        let table = &db.tables[name];
4✔
222
        let (rootpage, new_next) = stage_table_btree(&mut pager, table, next_free_page)?;
2✔
223
        next_free_page = new_next;
2✔
224
        master_rows.push(CatalogEntry {
2✔
225
            kind: "table".into(),
2✔
226
            name: name.clone(),
2✔
227
            sql: table_to_create_sql(table),
2✔
228
            rootpage,
229
            last_rowid: table.last_rowid,
2✔
230
        });
231
    }
232

233
    // 2. Stage each secondary index's B-Tree. Indexes persist in a
234
    //    deterministic order: sorted by (owning_table, index_name).
235
    let mut index_entries: Vec<(&Table, &SecondaryIndex)> = Vec::new();
2✔
236
    for table in db.tables.values() {
4✔
237
        for idx in &table.secondary_indexes {
4✔
238
            index_entries.push((table, idx));
2✔
239
        }
240
    }
241
    index_entries
2✔
242
        .sort_by(|(ta, ia), (tb, ib)| ta.tb_name.cmp(&tb.tb_name).then(ia.name.cmp(&ib.name)));
4✔
243
    for (_table, idx) in index_entries {
4✔
244
        let (rootpage, new_next) = stage_index_btree(&mut pager, idx, next_free_page)?;
4✔
245
        next_free_page = new_next;
2✔
246
        master_rows.push(CatalogEntry {
2✔
247
            kind: "index".into(),
2✔
248
            name: idx.name.clone(),
2✔
249
            sql: idx.synthesized_sql(),
2✔
250
            rootpage,
251
            last_rowid: 0,
252
        });
253
    }
254

255
    // 2b. Phase 7d.3: persist HNSW indexes as their own cell-encoded
256
    //     page trees, with the rootpage recorded in sqlrite_master.
257
    //     Reopen loads the graph back from cells (fast, exact match)
258
    //     instead of rebuilding from rows.
259
    //
260
    //     Dirty indexes (set by DELETE / UPDATE-on-vector-col) are
261
    //     rebuilt from current rows BEFORE staging, so the on-disk
262
    //     graph reflects the current row set.
263
    let mut hnsw_entries: Vec<(&Table, &crate::sql::db::table::HnswIndexEntry)> = Vec::new();
2✔
264
    for table in db.tables.values() {
4✔
265
        for entry in &table.hnsw_indexes {
4✔
266
            hnsw_entries.push((table, entry));
1✔
267
        }
268
    }
269
    hnsw_entries
2✔
270
        .sort_by(|(ta, ea), (tb, eb)| ta.tb_name.cmp(&tb.tb_name).then(ea.name.cmp(&eb.name)));
2✔
271
    for (table, entry) in hnsw_entries {
4✔
272
        let (rootpage, new_next) = stage_hnsw_btree(&mut pager, &entry.index, next_free_page)?;
2✔
273
        next_free_page = new_next;
1✔
274
        master_rows.push(CatalogEntry {
1✔
275
            kind: "index".into(),
1✔
276
            name: entry.name.clone(),
1✔
277
            sql: format!(
2✔
278
                "CREATE INDEX {} ON {} USING hnsw ({})",
279
                entry.name, table.tb_name, entry.column_name
280
            ),
281
            rootpage,
282
            last_rowid: 0,
283
        });
284
    }
285

286
    // 3. Build an in-memory sqlrite_master with one row per table or index,
287
    //    then stage it via the same tree-build path.
288
    let mut master = build_empty_master_table();
2✔
289
    for (i, entry) in master_rows.into_iter().enumerate() {
8✔
290
        let rowid = (i as i64) + 1;
4✔
291
        master.restore_row(
2✔
292
            rowid,
293
            vec![
4✔
294
                Some(Value::Text(entry.kind)),
2✔
295
                Some(Value::Text(entry.name)),
2✔
296
                Some(Value::Text(entry.sql)),
2✔
297
                Some(Value::Integer(entry.rootpage as i64)),
2✔
298
                Some(Value::Integer(entry.last_rowid)),
2✔
299
            ],
300
        )?;
301
    }
302
    let (master_root, master_next) = stage_table_btree(&mut pager, &master, next_free_page)?;
2✔
303
    next_free_page = master_next;
2✔
304

305
    pager.commit(DbHeader {
2✔
306
        page_count: next_free_page,
2✔
307
        schema_root_page: master_root,
308
    })?;
309

310
    if same_path {
4✔
311
        db.pager = Some(pager);
2✔
312
    }
313
    Ok(())
2✔
314
}
315

316
/// Build material for a single row in sqlrite_master.
317
struct CatalogEntry {
318
    kind: String, // "table" or "index"
319
    name: String,
320
    sql: String,
321
    rootpage: u32,
322
    last_rowid: i64,
323
}
324

325
// -------------------------------------------------------------------------
326
// sqlrite_master — hardcoded catalog table schema
327

328
fn build_empty_master_table() -> Table {
3✔
329
    // Phase 3e: `type` is the first column, matching SQLite's convention.
330
    // It distinguishes `'table'` rows from `'index'` rows.
331
    let columns = vec![
4✔
332
        Column::new("type".into(), "text".into(), false, true, false),
4✔
333
        Column::new("name".into(), "text".into(), true, true, true),
4✔
334
        Column::new("sql".into(), "text".into(), false, true, false),
4✔
335
        Column::new("rootpage".into(), "integer".into(), false, true, false),
4✔
336
        Column::new("last_rowid".into(), "integer".into(), false, true, false),
4✔
337
    ];
338
    build_empty_table(MASTER_TABLE_NAME, columns, 0)
2✔
339
}
340

341
/// Reads a required Text column from a known-good catalog row.
342
fn take_text(table: &Table, col: &str, rowid: i64) -> Result<String> {
2✔
343
    match table.get_value(col, rowid) {
2✔
344
        Some(Value::Text(s)) => Ok(s),
2✔
345
        other => Err(SQLRiteError::Internal(format!(
×
346
            "sqlrite_master column '{col}' at rowid {rowid}: expected Text, got {other:?}"
347
        ))),
348
    }
349
}
350

351
/// Reads a required Integer column from a known-good catalog row.
352
fn take_integer(table: &Table, col: &str, rowid: i64) -> Result<i64> {
3✔
353
    match table.get_value(col, rowid) {
2✔
354
        Some(Value::Integer(v)) => Ok(v),
2✔
355
        other => Err(SQLRiteError::Internal(format!(
×
356
            "sqlrite_master column '{col}' at rowid {rowid}: expected Integer, got {other:?}"
357
        ))),
358
    }
359
}
360

361
// -------------------------------------------------------------------------
362
// CREATE-TABLE SQL synthesis and re-parsing
363

364
/// Synthesizes a CREATE TABLE SQL string that recreates the table's schema.
365
/// Deterministic: same schema → same SQL, so diffing commits stay stable.
366
fn table_to_create_sql(table: &Table) -> String {
2✔
367
    let mut parts = Vec::with_capacity(table.columns.len());
2✔
368
    for c in &table.columns {
4✔
369
        // Render the SQL type literally so the round-trip through
370
        // CREATE TABLE re-parsing recreates the same schema. Vector
371
        // carries its dimension inline.
372
        let ty: String = match &c.datatype {
2✔
373
            DataType::Integer => "INTEGER".to_string(),
4✔
374
            DataType::Text => "TEXT".to_string(),
4✔
375
            DataType::Real => "REAL".to_string(),
×
376
            DataType::Bool => "BOOLEAN".to_string(),
×
377
            DataType::Vector(dim) => format!("VECTOR({dim})"),
2✔
378
            DataType::Json => "JSON".to_string(),
2✔
UNCOV
379
            DataType::None | DataType::Invalid => "TEXT".to_string(),
×
380
        };
381
        let mut piece = format!("{} {}", c.column_name, ty);
4✔
382
        if c.is_pk {
2✔
383
            piece.push_str(" PRIMARY KEY");
4✔
384
        } else {
385
            if c.is_unique {
2✔
386
                piece.push_str(" UNIQUE");
2✔
387
            }
388
            if c.not_null {
2✔
389
                piece.push_str(" NOT NULL");
2✔
390
            }
391
        }
392
        parts.push(piece);
2✔
393
    }
394
    format!("CREATE TABLE {} ({});", table.tb_name, parts.join(", "))
2✔
395
}
396

397
/// Reverses `table_to_create_sql`: feeds the SQL back through `sqlparser`
398
/// and produces our internal column list. Returns `(table_name, columns)`.
399
fn parse_create_sql(sql: &str) -> Result<(String, Vec<Column>)> {
2✔
400
    let dialect = SQLiteDialect {};
2✔
401
    let mut ast = Parser::parse_sql(&dialect, sql).map_err(SQLRiteError::from)?;
2✔
402
    let stmt = ast.pop().ok_or_else(|| {
4✔
403
        SQLRiteError::Internal("sqlrite_master row held an empty SQL string".to_string())
×
404
    })?;
405
    let create = CreateQuery::new(&stmt)?;
4✔
406
    let columns = create
2✔
407
        .columns
408
        .into_iter()
409
        .map(|pc| Column::new(pc.name, pc.datatype, pc.is_pk, pc.not_null, pc.is_unique))
6✔
410
        .collect();
411
    Ok((create.table_name, columns))
2✔
412
}
413

414
// -------------------------------------------------------------------------
415
// In-memory table (re)construction
416

417
/// Builds an empty in-memory `Table` given the declared columns.
418
fn build_empty_table(name: &str, columns: Vec<Column>, last_rowid: i64) -> Table {
2✔
419
    let rows: Arc<Mutex<HashMap<String, Row>>> = Arc::new(Mutex::new(HashMap::new()));
4✔
420
    let mut secondary_indexes: Vec<SecondaryIndex> = Vec::new();
2✔
421
    {
422
        let mut map = rows.lock().expect("rows mutex poisoned");
4✔
423
        for col in &columns {
6✔
424
            // Mirror the dispatch in `Table::new` so the reconstructed
425
            // table has the same shape it'd have if it were built fresh
426
            // from SQL. Phase 7a adds the Vector arm — without it,
427
            // VECTOR columns silently restore as Row::None and every
428
            // restore_row hits a "storage None vs value Some(Vector(...))"
429
            // type mismatch.
430
            let row = match &col.datatype {
2✔
431
                DataType::Integer => Row::Integer(BTreeMap::new()),
4✔
432
                DataType::Text => Row::Text(BTreeMap::new()),
4✔
433
                DataType::Real => Row::Real(BTreeMap::new()),
×
434
                DataType::Bool => Row::Bool(BTreeMap::new()),
×
435
                DataType::Vector(_dim) => Row::Vector(BTreeMap::new()),
2✔
436
                // JSON columns reuse Text storage — see Table::new and
437
                // Phase 7e's scope-correction note.
438
                DataType::Json => Row::Text(BTreeMap::new()),
2✔
UNCOV
439
                DataType::None | DataType::Invalid => Row::None,
×
440
            };
441
            map.insert(col.column_name.clone(), row);
4✔
442

443
            // Auto-create UNIQUE/PK indexes so the restored table has the
444
            // same shape Table::new would have built from fresh SQL.
445
            if (col.is_pk || col.is_unique)
2✔
446
                && matches!(col.datatype, DataType::Integer | DataType::Text)
2✔
447
            {
448
                if let Ok(idx) = SecondaryIndex::new(
449
                    SecondaryIndex::auto_name(name, &col.column_name),
2✔
450
                    name.to_string(),
4✔
451
                    col.column_name.clone(),
2✔
452
                    &col.datatype,
453
                    true,
454
                    IndexOrigin::Auto,
455
                ) {
456
                    secondary_indexes.push(idx);
2✔
457
                }
458
            }
459
        }
460
    }
461

462
    let primary_key = columns
4✔
463
        .iter()
464
        .find(|c| c.is_pk)
6✔
465
        .map(|c| c.column_name.clone())
6✔
466
        .unwrap_or_else(|| "-1".to_string());
2✔
467

468
    Table {
469
        tb_name: name.to_string(),
2✔
470
        columns,
471
        rows,
472
        secondary_indexes,
473
        // HNSW indexes (Phase 7d.2) are reconstructed on open by re-
474
        // executing each `CREATE INDEX … USING hnsw` SQL stored in
475
        // `sqlrite_master`. This builder produces the empty shell;
476
        // `replay_create_index_for_hnsw` (in this same module) walks
477
        // sqlrite_master after every table is loaded and rebuilds the
478
        // graph from current row data. Persistence of the graph itself
479
        // (avoiding the on-open rebuild cost) is Phase 7d.3.
480
        hnsw_indexes: Vec::new(),
2✔
481
        last_rowid,
482
        primary_key,
483
    }
484
}
485

486
// -------------------------------------------------------------------------
487
// Leaf-chain read / write
488

489
/// Walks a table's B-Tree from `root_page`, following the leftmost-child
490
/// chain down to the first leaf, then iterating leaves via their sibling
491
/// `next_page` pointers. Every cell is decoded and replayed into `table`.
492
///
493
/// Open-path note: we eagerly materialize the entire table into `Table`'s
494
/// in-memory maps. Phase 5 will introduce a `Cursor` that hits the pager
495
/// on demand so queries can stream through the tree without a full upfront
496
/// load.
497
/// Re-parses `CREATE INDEX` SQL from sqlrite_master and restores the
498
/// index on its base table by walking the tree of index cells at
499
/// `rootpage`. The base table is expected to already be in `db.tables`.
500
fn attach_index(db: &mut Database, pager: &Pager, row: IndexCatalogRow) -> Result<()> {
2✔
501
    let (table_name, column_name, is_unique) = parse_create_index_sql(&row.sql)?;
4✔
502

503
    let table = db.get_table_mut(table_name.clone()).map_err(|_| {
4✔
504
        SQLRiteError::Internal(format!(
×
505
            "index '{}' references unknown table '{table_name}' (sqlrite_master out of sync?)",
506
            row.name
507
        ))
508
    })?;
509
    let datatype = table
6✔
510
        .columns
511
        .iter()
2✔
512
        .find(|c| c.column_name == column_name)
6✔
513
        .map(|c| clone_datatype(&c.datatype))
6✔
514
        .ok_or_else(|| {
2✔
515
            SQLRiteError::Internal(format!(
×
516
                "index '{}' references unknown column '{column_name}' on '{table_name}'",
517
                row.name
518
            ))
519
        })?;
520

521
    // An auto-index on this column may already exist (built by
522
    // build_empty_table for UNIQUE/PK columns). If the names match, reuse
523
    // the slot instead of adding a duplicate entry.
524
    let existing_slot = table
6✔
525
        .secondary_indexes
526
        .iter()
527
        .position(|i| i.name == row.name);
6✔
528
    let idx = match existing_slot {
2✔
529
        Some(i) => {
2✔
530
            // Drain any entries that may have been populated during table
531
            // restore_row calls — we're about to repopulate from the
532
            // persisted tree.
533
            table.secondary_indexes.remove(i)
4✔
534
        }
535
        None => SecondaryIndex::new(
2✔
536
            row.name.clone(),
1✔
537
            table_name.clone(),
2✔
538
            column_name.clone(),
1✔
539
            &datatype,
540
            is_unique,
541
            IndexOrigin::Explicit,
542
        )?,
543
    };
544
    let mut idx = idx;
2✔
545
    // Wipe any stale entries from the auto path so the load is idempotent.
546
    let is_unique_flag = idx.is_unique;
2✔
547
    let origin = idx.origin;
2✔
548
    idx = SecondaryIndex::new(
6✔
549
        idx.name,
2✔
550
        idx.table_name,
2✔
551
        idx.column_name,
2✔
552
        &datatype,
553
        is_unique_flag,
554
        origin,
555
    )?;
556

557
    // Populate from the index tree's cells.
558
    load_index_rows(pager, &mut idx, row.rootpage)?;
2✔
559

560
    table.secondary_indexes.push(idx);
2✔
561
    Ok(())
2✔
562
}
563

564
/// Walks the leaves of an index B-Tree rooted at `root_page` and inserts
565
/// every `(value, rowid)` pair into `idx`.
566
fn load_index_rows(pager: &Pager, idx: &mut SecondaryIndex, root_page: u32) -> Result<()> {
2✔
567
    if root_page == 0 {
2✔
568
        return Ok(());
×
569
    }
570
    let first_leaf = find_leftmost_leaf(pager, root_page)?;
2✔
571
    let mut current = first_leaf;
2✔
572
    while current != 0 {
2✔
573
        let page_buf = pager
2✔
574
            .read_page(current)
2✔
575
            .ok_or_else(|| SQLRiteError::Internal(format!("missing index leaf page {current}")))?;
2✔
576
        if page_buf[0] != PageType::TableLeaf as u8 {
2✔
577
            return Err(SQLRiteError::Internal(format!(
×
578
                "page {current} tagged {} but expected TableLeaf (index)",
579
                page_buf[0]
580
            )));
581
        }
582
        let next_leaf = u32::from_le_bytes(page_buf[1..5].try_into().unwrap());
2✔
583
        let payload: &[u8; PAYLOAD_PER_PAGE] = (&page_buf[PAGE_HEADER_SIZE..])
4✔
584
            .try_into()
2✔
585
            .map_err(|_| SQLRiteError::Internal("index leaf payload size".to_string()))?;
2✔
586
        let leaf = TablePage::from_bytes(payload);
2✔
587

588
        for slot in 0..leaf.slot_count() {
4✔
589
            // Slots on an index page hold KIND_INDEX cells; decode directly.
590
            let offset = leaf.slot_offset_raw(slot)?;
4✔
591
            let (ic, _) = IndexCell::decode(leaf.as_bytes(), offset)?;
2✔
592
            idx.insert(&ic.value, ic.rowid)?;
4✔
593
        }
594
        current = next_leaf;
2✔
595
    }
596
    Ok(())
2✔
597
}
598

599
/// Minimal recognizer for the synthesized-or-user `CREATE INDEX` SQL we
600
/// store in sqlrite_master. Returns `(table_name, column_name, is_unique)`.
601
///
602
/// Uses sqlparser so user-supplied SQL with extra whitespace, case, etc.
603
/// still works; the only shape we accept is single-column indexes.
604
fn parse_create_index_sql(sql: &str) -> Result<(String, String, bool)> {
2✔
605
    use sqlparser::ast::{CreateIndex, Expr, Statement};
606

607
    let dialect = SQLiteDialect {};
2✔
608
    let mut ast = Parser::parse_sql(&dialect, sql).map_err(SQLRiteError::from)?;
2✔
609
    let Some(Statement::CreateIndex(CreateIndex {
4✔
610
        table_name,
2✔
611
        columns,
2✔
612
        unique,
2✔
613
        ..
614
    })) = ast.pop()
6✔
615
    else {
616
        return Err(SQLRiteError::Internal(format!(
×
617
            "sqlrite_master index row's SQL isn't a CREATE INDEX: {sql}"
618
        )));
619
    };
620
    if columns.len() != 1 {
4✔
621
        return Err(SQLRiteError::NotImplemented(
×
622
            "multi-column indexes aren't supported yet".to_string(),
×
623
        ));
624
    }
625
    let col = match &columns[0].column.expr {
4✔
626
        Expr::Identifier(ident) => ident.value.clone(),
4✔
627
        Expr::CompoundIdentifier(parts) => {
×
628
            parts.last().map(|p| p.value.clone()).unwrap_or_default()
×
629
        }
630
        other => {
×
631
            return Err(SQLRiteError::Internal(format!(
×
632
                "unsupported indexed column expression: {other:?}"
633
            )));
634
        }
635
    };
636
    Ok((table_name.to_string(), col, unique))
4✔
637
}
638

639
/// True iff a CREATE INDEX SQL string uses `USING hnsw` (case-insensitive).
640
/// Used by the open path to route HNSW indexes to the graph-rebuild path
641
/// instead of the standard B-Tree cell-load. Pre-Phase-7d.2 indexes
642
/// don't have a USING clause, so they all return false and continue
643
/// taking the existing path.
644
fn create_index_sql_uses_hnsw(sql: &str) -> bool {
2✔
645
    use sqlparser::ast::{CreateIndex, IndexType, Statement};
646

647
    let dialect = SQLiteDialect {};
2✔
648
    let Ok(mut ast) = Parser::parse_sql(&dialect, sql) else {
4✔
649
        return false;
×
650
    };
651
    let Some(Statement::CreateIndex(CreateIndex { using, .. })) = ast.pop() else {
6✔
652
        return false;
×
653
    };
654
    matches!(using, Some(IndexType::Custom(ident)) if ident.value.eq_ignore_ascii_case("hnsw"))
3✔
655
}
656

657
/// Loads (or rebuilds) an HNSW index on database open. Two paths:
658
///
659
///   - **rootpage != 0** (Phase 7d.3 default): the graph is persisted
660
///     as cell-encoded pages. Read every node directly via
661
///     `load_hnsw_nodes` and reconstruct the index — fast, zero
662
///     algorithm runs, exact bit-for-bit reproduction of what was saved.
663
///
664
///   - **rootpage == 0** (compatibility): no on-disk graph, e.g. for
665
///     files saved by Phase 7d.2 before persistence landed. Replay the
666
///     CREATE INDEX SQL through `execute_create_index`, which walks the
667
///     table's current rows and populates a fresh graph. Slower but
668
///     correctness-equivalent on the first save with the new code.
669
fn rebuild_hnsw_index(db: &mut Database, pager: &Pager, row: &IndexCatalogRow) -> Result<()> {
1✔
670
    use crate::sql::db::table::HnswIndexEntry;
671
    use crate::sql::executor::execute_create_index;
672
    use crate::sql::hnsw::{DistanceMetric, HnswIndex};
673
    use sqlparser::ast::Statement;
674

675
    let dialect = SQLiteDialect {};
1✔
676
    let mut ast = Parser::parse_sql(&dialect, &row.sql).map_err(SQLRiteError::from)?;
1✔
677
    let Some(stmt @ Statement::CreateIndex(_)) = ast.pop() else {
3✔
678
        return Err(SQLRiteError::Internal(format!(
×
679
            "sqlrite_master HNSW row's SQL isn't a CREATE INDEX: {}",
680
            row.sql
681
        )));
682
    };
683

684
    if row.rootpage == 0 {
1✔
685
        // Compatibility path — no persisted graph; walk current rows.
686
        execute_create_index(&stmt, db)?;
×
687
        return Ok(());
×
688
    }
689

690
    // Persistence path — read the cell tree, deserialize.
691
    let nodes = load_hnsw_nodes(pager, row.rootpage)?;
2✔
692
    let index = HnswIndex::from_persisted_nodes(DistanceMetric::L2, 0xC0FFEE, nodes);
2✔
693

694
    // Parse the CREATE INDEX to know which table + column to attach to
695
    // — same shape as the row-walk path; we just don't execute it.
696
    let (tbl_name, col_name) = parse_hnsw_create_index_sql(&row.sql)?;
2✔
697
    let table_mut = db.get_table_mut(tbl_name.clone()).map_err(|_| {
2✔
698
        SQLRiteError::Internal(format!(
×
699
            "HNSW index '{}' references unknown table '{tbl_name}'",
700
            row.name
701
        ))
702
    })?;
703
    table_mut.hnsw_indexes.push(HnswIndexEntry {
2✔
704
        name: row.name.clone(),
1✔
705
        column_name: col_name,
1✔
706
        index,
1✔
707
        needs_rebuild: false,
708
    });
709
    Ok(())
1✔
710
}
711

712
/// Phase 7d.3 — Phase-7d.3-side helper: walk every leaf in the HNSW
713
/// page tree at `root_page` and decode each cell as a node. Returns
714
/// the (node_id, layers) tuples in slot-order (already ascending by
715
/// node_id since they were staged that way). The caller hands them to
716
/// `HnswIndex::from_persisted_nodes`.
717
fn load_hnsw_nodes(pager: &Pager, root_page: u32) -> Result<Vec<(i64, Vec<Vec<i64>>)>> {
1✔
718
    use crate::sql::pager::hnsw_cell::HnswNodeCell;
719

720
    let mut nodes: Vec<(i64, Vec<Vec<i64>>)> = Vec::new();
1✔
721
    let first_leaf = find_leftmost_leaf(pager, root_page)?;
2✔
722
    let mut current = first_leaf;
1✔
723
    while current != 0 {
1✔
724
        let page_buf = pager
1✔
725
            .read_page(current)
1✔
726
            .ok_or_else(|| SQLRiteError::Internal(format!("missing HNSW leaf page {current}")))?;
1✔
727
        if page_buf[0] != PageType::TableLeaf as u8 {
1✔
728
            return Err(SQLRiteError::Internal(format!(
×
729
                "page {current} tagged {} but expected TableLeaf (HNSW)",
730
                page_buf[0]
×
731
            )));
732
        }
733
        let next_leaf = u32::from_le_bytes(page_buf[1..5].try_into().unwrap());
2✔
734
        let payload: &[u8; PAYLOAD_PER_PAGE] = (&page_buf[PAGE_HEADER_SIZE..])
2✔
735
            .try_into()
1✔
736
            .map_err(|_| SQLRiteError::Internal("HNSW leaf payload size".to_string()))?;
1✔
737
        let leaf = TablePage::from_bytes(payload);
1✔
738
        for slot in 0..leaf.slot_count() {
3✔
739
            let offset = leaf.slot_offset_raw(slot)?;
2✔
740
            let (cell, _) = HnswNodeCell::decode(leaf.as_bytes(), offset)?;
1✔
741
            nodes.push((cell.node_id, cell.layers));
1✔
742
        }
743
        current = next_leaf;
1✔
744
    }
745
    Ok(nodes)
1✔
746
}
747

748
/// Pulls (table_name, column_name) out of a `CREATE INDEX … USING hnsw (col)`
749
/// SQL string. Used by the persistence path on open to know where to
750
/// attach the loaded graph. Same shape as `parse_create_index_sql` for
751
/// regular indexes — only the assertion differs (we don't care about
752
/// UNIQUE for HNSW).
753
fn parse_hnsw_create_index_sql(sql: &str) -> Result<(String, String)> {
1✔
754
    use sqlparser::ast::{CreateIndex, Expr, Statement};
755

756
    let dialect = SQLiteDialect {};
1✔
757
    let mut ast = Parser::parse_sql(&dialect, sql).map_err(SQLRiteError::from)?;
1✔
758
    let Some(Statement::CreateIndex(CreateIndex {
2✔
759
        table_name,
1✔
760
        columns,
1✔
761
        ..
762
    })) = ast.pop()
3✔
763
    else {
764
        return Err(SQLRiteError::Internal(format!(
×
765
            "sqlrite_master HNSW row's SQL isn't a CREATE INDEX: {sql}"
766
        )));
767
    };
768
    if columns.len() != 1 {
2✔
769
        return Err(SQLRiteError::NotImplemented(
×
770
            "multi-column HNSW indexes aren't supported yet".to_string(),
×
771
        ));
772
    }
773
    let col = match &columns[0].column.expr {
2✔
774
        Expr::Identifier(ident) => ident.value.clone(),
2✔
775
        Expr::CompoundIdentifier(parts) => {
×
776
            parts.last().map(|p| p.value.clone()).unwrap_or_default()
×
777
        }
778
        other => {
×
779
            return Err(SQLRiteError::Internal(format!(
×
780
                "unsupported HNSW indexed column expression: {other:?}"
781
            )));
782
        }
783
    };
784
    Ok((table_name.to_string(), col))
2✔
785
}
786

787
/// Phase 7d.3 — rebuilds in-place any HnswIndexEntry whose
788
/// `needs_rebuild` flag is set (DELETE / UPDATE-on-vector marked it).
789
/// Walks the table's current Vec<f32> column storage and runs the
790
/// HNSW algorithm fresh. Called at the top of `save_database` before
791
/// any immutable borrows of `db` start.
792
///
793
/// Cost: O(N · ef_construction · log N) per dirty index. Fine for
794
/// small tables, expensive for ≥100k-row tables — matches the
795
/// trade-off SQLite makes for FTS5: dirtying-and-rebuilding is the
796
/// MVP, more sophisticated incremental delete strategies (soft-delete
797
/// + tombstones, neighbor reconnection) are future polish.
798
fn rebuild_dirty_hnsw_indexes(db: &mut Database) {
2✔
799
    use crate::sql::hnsw::{DistanceMetric, HnswIndex};
800

801
    for table in db.tables.values_mut() {
5✔
802
        // Snapshot which (index_name, column) pairs need rebuilding,
803
        // before we go grabbing column data — keeps the borrow
804
        // structure simple.
805
        let dirty: Vec<(String, String)> = table
4✔
806
            .hnsw_indexes
807
            .iter()
808
            .filter(|e| e.needs_rebuild)
4✔
809
            .map(|e| (e.name.clone(), e.column_name.clone()))
4✔
810
            .collect();
811
        if dirty.is_empty() {
4✔
812
            continue;
813
        }
814

815
        for (idx_name, col_name) in dirty {
3✔
816
            // Snapshot every (rowid, vec) for this column.
817
            let mut vectors: Vec<(i64, Vec<f32>)> = Vec::new();
1✔
818
            {
819
                let row_data = table.rows.lock().expect("rows mutex poisoned");
2✔
820
                if let Some(Row::Vector(map)) = row_data.get(&col_name) {
3✔
821
                    for (id, v) in map.iter() {
1✔
822
                        vectors.push((*id, v.clone()));
1✔
823
                    }
824
                }
825
            }
826
            // Pre-build a HashMap for the get_vec closure so we don't
827
            // pay O(N) lookup per insert call.
828
            let snapshot: std::collections::HashMap<i64, Vec<f32>> =
1✔
829
                vectors.iter().cloned().collect();
830

831
            let mut new_idx = HnswIndex::new(DistanceMetric::L2, 0xC0FFEE);
2✔
832
            // Sort by id so the rebuild is deterministic across runs.
833
            vectors.sort_by_key(|(id, _)| *id);
4✔
834
            for (id, v) in &vectors {
1✔
835
                new_idx.insert(*id, v, |q| snapshot.get(&q).cloned().unwrap_or_default());
4✔
836
            }
837

838
            // Replace the entry's index + clear the dirty flag.
839
            if let Some(entry) = table.hnsw_indexes.iter_mut().find(|e| e.name == idx_name) {
4✔
840
                entry.index = new_idx;
1✔
841
                entry.needs_rebuild = false;
1✔
842
            }
843
        }
844
    }
845
}
846

847
/// Cheap clone helper — `DataType` doesn't derive `Clone` elsewhere.
848
fn clone_datatype(dt: &DataType) -> DataType {
2✔
849
    match dt {
2✔
850
        DataType::Integer => DataType::Integer,
2✔
851
        DataType::Text => DataType::Text,
1✔
852
        DataType::Real => DataType::Real,
×
853
        DataType::Bool => DataType::Bool,
×
854
        DataType::Vector(dim) => DataType::Vector(*dim),
×
NEW
855
        DataType::Json => DataType::Json,
×
856
        DataType::None => DataType::None,
×
857
        DataType::Invalid => DataType::Invalid,
×
858
    }
859
}
860

861
/// Stages an index's B-Tree at `start_page`. Each leaf cell is a
862
/// `KIND_INDEX` entry carrying `(original_rowid, value)`. Returns
863
/// `(root_page, next_free_page)`.
864
///
865
/// The tree's shape matches a regular table's — leaves chained via
866
/// `next_page`, optional interior layer above. `Cell::peek_rowid` works
867
/// uniformly for index cells (same prefix as local cells), so the
868
/// existing slot directory and binary search carry over.
869
fn stage_index_btree(
2✔
870
    pager: &mut Pager,
871
    idx: &SecondaryIndex,
872
    start_page: u32,
873
) -> Result<(u32, u32)> {
874
    // Build the leaves.
875
    let (leaves, mut next_free_page) = stage_index_leaves(pager, idx, start_page)?;
2✔
876
    if leaves.len() == 1 {
4✔
877
        return Ok((leaves[0].0, next_free_page));
4✔
878
    }
879
    let mut level: Vec<(u32, i64)> = leaves;
1✔
880
    while level.len() > 1 {
4✔
881
        let (next_level, new_next_free) = stage_interior_level(pager, &level, next_free_page)?;
2✔
882
        next_free_page = new_next_free;
1✔
883
        level = next_level;
2✔
884
    }
885
    Ok((level[0].0, next_free_page))
2✔
886
}
887

888
/// Packs the index's (value, rowid) entries into a sibling-chained run
889
/// of `TableLeaf` pages. Iteration order matches `SecondaryIndex::iter_entries`
890
/// (ascending value; rowids in insertion order within a value), which is
891
/// also ascending by the "cell rowid" carried in each IndexCell (the
892
/// original row's rowid) — so Cell::peek_rowid + the slot directory's
893
/// rowid ordering stays consistent.
894
fn stage_index_leaves(
2✔
895
    pager: &mut Pager,
896
    idx: &SecondaryIndex,
897
    start_page: u32,
898
) -> Result<(Vec<(u32, i64)>, u32)> {
899
    let mut leaves: Vec<(u32, i64)> = Vec::new();
2✔
900
    let mut current_leaf = TablePage::empty();
4✔
901
    let mut current_leaf_page = start_page;
2✔
902
    let mut current_max_rowid: Option<i64> = None;
2✔
903
    let mut next_free_page = start_page + 1;
2✔
904

905
    // Sort the entries by original rowid so the in-page slot directory,
906
    // which binary-searches by rowid, stays valid. (iter_entries orders by
907
    // value; we reorder here for B-Tree correctness.)
908
    let mut entries: Vec<(Value, i64)> = idx.iter_entries().collect();
4✔
909
    entries.sort_by_key(|(_, r)| *r);
6✔
910

911
    for (value, rowid) in entries {
4✔
912
        let cell = IndexCell::new(rowid, value);
2✔
913
        let entry_bytes = cell.encode()?;
4✔
914

915
        if !current_leaf.would_fit(entry_bytes.len()) {
4✔
916
            let next_leaf_page_num = next_free_page;
1✔
917
            emit_leaf(pager, current_leaf_page, &current_leaf, next_leaf_page_num);
1✔
918
            leaves.push((current_leaf_page, current_max_rowid.unwrap_or(i64::MIN)));
1✔
919
            current_leaf = TablePage::empty();
1✔
920
            current_leaf_page = next_leaf_page_num;
1✔
921
            next_free_page += 1;
1✔
922

923
            if !current_leaf.would_fit(entry_bytes.len()) {
2✔
924
                return Err(SQLRiteError::Internal(format!(
×
925
                    "index entry of {} bytes exceeds empty-page capacity {}",
926
                    entry_bytes.len(),
×
927
                    current_leaf.free_space()
×
928
                )));
929
            }
930
        }
931
        current_leaf.insert_entry(rowid, &entry_bytes)?;
4✔
932
        current_max_rowid = Some(rowid);
2✔
933
    }
934

935
    emit_leaf(pager, current_leaf_page, &current_leaf, 0);
2✔
936
    leaves.push((current_leaf_page, current_max_rowid.unwrap_or(i64::MIN)));
2✔
937
    Ok((leaves, next_free_page))
2✔
938
}
939

940
/// Phase 7d.3 — stages an HNSW index's page tree at `start_page`.
941
/// Each leaf cell is a `KIND_HNSW` entry carrying one node's
942
/// (node_id, layers). Returns `(root_page, next_free_page)`.
943
///
944
/// Tree shape is identical to `stage_index_btree` — chained leaves +
945
/// optional interior layers. The slot directory binary-searches by
946
/// node_id (which is the cell's "rowid" in `Cell::peek_rowid` terms),
947
/// so reads can locate any node in O(log N) once 7d.4-or-later
948
/// optimizes the load path to lazy-fetch instead of read-all.
949
/// Today, `load_hnsw_nodes` reads the entire tree on open.
950
fn stage_hnsw_btree(
1✔
951
    pager: &mut Pager,
952
    idx: &crate::sql::hnsw::HnswIndex,
953
    start_page: u32,
954
) -> Result<(u32, u32)> {
955
    let (leaves, mut next_free_page) = stage_hnsw_leaves(pager, idx, start_page)?;
1✔
956
    if leaves.len() == 1 {
2✔
957
        return Ok((leaves[0].0, next_free_page));
2✔
958
    }
959
    let mut level: Vec<(u32, i64)> = leaves;
×
960
    while level.len() > 1 {
×
961
        let (next_level, new_next_free) = stage_interior_level(pager, &level, next_free_page)?;
×
962
        next_free_page = new_next_free;
×
963
        level = next_level;
×
964
    }
965
    Ok((level[0].0, next_free_page))
×
966
}
967

968
/// Packs HNSW nodes into a sibling-chained run of `TableLeaf` pages.
969
/// `serialize_nodes` already returns nodes in ascending node_id order,
970
/// so the slot directory's rowid ordering stays valid.
971
fn stage_hnsw_leaves(
1✔
972
    pager: &mut Pager,
973
    idx: &crate::sql::hnsw::HnswIndex,
974
    start_page: u32,
975
) -> Result<(Vec<(u32, i64)>, u32)> {
976
    use crate::sql::pager::hnsw_cell::HnswNodeCell;
977

978
    let mut leaves: Vec<(u32, i64)> = Vec::new();
1✔
979
    let mut current_leaf = TablePage::empty();
2✔
980
    let mut current_leaf_page = start_page;
1✔
981
    let mut current_max_rowid: Option<i64> = None;
1✔
982
    let mut next_free_page = start_page + 1;
1✔
983

984
    let serialized = idx.serialize_nodes();
1✔
985

986
    // Empty index → emit a single empty leaf page so the rootpage
987
    // pointer in sqlrite_master stays nonzero (== "graph is persisted,
988
    // it just happens to be empty"). load_hnsw_nodes is fine with an
989
    // empty leaf — slot_count() returns 0.
990
    for (node_id, layers) in serialized {
2✔
991
        let cell = HnswNodeCell::new(node_id, layers);
1✔
992
        let entry_bytes = cell.encode()?;
2✔
993

994
        if !current_leaf.would_fit(entry_bytes.len()) {
2✔
995
            let next_leaf_page_num = next_free_page;
×
996
            emit_leaf(pager, current_leaf_page, &current_leaf, next_leaf_page_num);
×
997
            leaves.push((current_leaf_page, current_max_rowid.unwrap_or(i64::MIN)));
×
998
            current_leaf = TablePage::empty();
×
999
            current_leaf_page = next_leaf_page_num;
×
1000
            next_free_page += 1;
×
1001

1002
            if !current_leaf.would_fit(entry_bytes.len()) {
×
1003
                return Err(SQLRiteError::Internal(format!(
×
1004
                    "HNSW node {node_id} cell of {} bytes exceeds empty-page capacity {}",
1005
                    entry_bytes.len(),
×
1006
                    current_leaf.free_space()
×
1007
                )));
1008
            }
1009
        }
1010
        current_leaf.insert_entry(node_id, &entry_bytes)?;
2✔
1011
        current_max_rowid = Some(node_id);
1✔
1012
    }
1013

1014
    emit_leaf(pager, current_leaf_page, &current_leaf, 0);
1✔
1015
    leaves.push((current_leaf_page, current_max_rowid.unwrap_or(i64::MIN)));
1✔
1016
    Ok((leaves, next_free_page))
1✔
1017
}
1018

1019
fn load_table_rows(pager: &Pager, table: &mut Table, root_page: u32) -> Result<()> {
2✔
1020
    let first_leaf = find_leftmost_leaf(pager, root_page)?;
2✔
1021
    let mut current = first_leaf;
2✔
1022
    while current != 0 {
2✔
1023
        let page_buf = pager
2✔
1024
            .read_page(current)
2✔
1025
            .ok_or_else(|| SQLRiteError::Internal(format!("missing leaf page {current}")))?;
2✔
1026
        if page_buf[0] != PageType::TableLeaf as u8 {
2✔
1027
            return Err(SQLRiteError::Internal(format!(
×
1028
                "page {current} tagged {} but expected TableLeaf",
1029
                page_buf[0]
1030
            )));
1031
        }
1032
        let next_leaf = u32::from_le_bytes(page_buf[1..5].try_into().unwrap());
2✔
1033
        let payload: &[u8; PAYLOAD_PER_PAGE] = (&page_buf[PAGE_HEADER_SIZE..])
4✔
1034
            .try_into()
2✔
1035
            .map_err(|_| SQLRiteError::Internal("leaf payload slice size".to_string()))?;
2✔
1036
        let leaf = TablePage::from_bytes(payload);
2✔
1037

1038
        for slot in 0..leaf.slot_count() {
6✔
1039
            let entry = leaf.entry_at(slot)?;
4✔
1040
            let cell = match entry {
2✔
1041
                PagedEntry::Local(c) => c,
2✔
1042
                PagedEntry::Overflow(r) => {
1✔
1043
                    let body_bytes =
2✔
1044
                        read_overflow_chain(pager, r.first_overflow_page, r.total_body_len)?;
1045
                    let (c, _) = Cell::decode(&body_bytes, 0)?;
2✔
1046
                    c
1✔
1047
                }
1048
            };
1049
            table.restore_row(cell.rowid, cell.values)?;
4✔
1050
        }
1051
        current = next_leaf;
2✔
1052
    }
1053
    Ok(())
2✔
1054
}
1055

1056
/// Descends from `root_page` through `InteriorNode` pages, always taking
1057
/// the leftmost child, until a `TableLeaf` is reached. Returns that leaf's
1058
/// page number. A root that's already a leaf is returned as-is.
1059
fn find_leftmost_leaf(pager: &Pager, root_page: u32) -> Result<u32> {
2✔
1060
    let mut current = root_page;
2✔
1061
    loop {
1062
        let page_buf = pager.read_page(current).ok_or_else(|| {
2✔
1063
            SQLRiteError::Internal(format!("missing page {current} during tree descent"))
×
1064
        })?;
1065
        match page_buf[0] {
1066
            t if t == PageType::TableLeaf as u8 => return Ok(current),
4✔
1067
            t if t == PageType::InteriorNode as u8 => {
2✔
1068
                let payload: &[u8; PAYLOAD_PER_PAGE] =
1✔
1069
                    (&page_buf[PAGE_HEADER_SIZE..]).try_into().map_err(|_| {
1070
                        SQLRiteError::Internal("interior payload slice size".to_string())
×
1071
                    })?;
1072
                let interior = InteriorPage::from_bytes(payload);
1✔
1073
                current = interior.leftmost_child()?;
2✔
1074
            }
1075
            other => {
×
1076
                return Err(SQLRiteError::Internal(format!(
×
1077
                    "unexpected page type {other} during tree descent at page {current}"
1078
                )));
1079
            }
1080
        }
1081
    }
1082
}
1083

1084
/// Stages a table's B-Tree starting at `start_page`. Returns
1085
/// `(root_page, next_free_page)`. Builds bottom-up:
1086
///
1087
/// 1. Pack all row cells into `TableLeaf` pages, chaining them via each
1088
///    leaf's `next_page` sibling pointer (for fast sequential scans).
1089
/// 2. If the table fits in a single leaf, that leaf is the root.
1090
/// 3. Otherwise, group leaves into `InteriorNode` pages; recurse up the
1091
///    tree until one root remains.
1092
///
1093
/// Deterministic: same in-memory rows → same pages at same offsets, so
1094
/// the Pager's diff commit still skips unchanged tables.
1095
fn stage_table_btree(pager: &mut Pager, table: &Table, start_page: u32) -> Result<(u32, u32)> {
2✔
1096
    let (leaves, mut next_free_page) = stage_leaves(pager, table, start_page)?;
2✔
1097
    if leaves.len() == 1 {
4✔
1098
        return Ok((leaves[0].0, next_free_page));
4✔
1099
    }
1100
    let mut level: Vec<(u32, i64)> = leaves;
1✔
1101
    while level.len() > 1 {
4✔
1102
        let (next_level, new_next_free) = stage_interior_level(pager, &level, next_free_page)?;
2✔
1103
        next_free_page = new_next_free;
1✔
1104
        level = next_level;
2✔
1105
    }
1106
    Ok((level[0].0, next_free_page))
2✔
1107
}
1108

1109
/// Packs the table's rows into a sibling-linked chain of `TableLeaf` pages.
1110
/// Returns each leaf's `(page_number, max_rowid)` (used by the next level
1111
/// up to build divider cells) and the first free page after the chain
1112
/// including any overflow pages allocated for oversized cells.
1113
fn stage_leaves(
2✔
1114
    pager: &mut Pager,
1115
    table: &Table,
1116
    start_page: u32,
1117
) -> Result<(Vec<(u32, i64)>, u32)> {
1118
    let mut leaves: Vec<(u32, i64)> = Vec::new();
2✔
1119
    let mut current_leaf = TablePage::empty();
4✔
1120
    let mut current_leaf_page = start_page;
2✔
1121
    let mut current_max_rowid: Option<i64> = None;
2✔
1122
    let mut next_free_page = start_page + 1;
2✔
1123

1124
    for rowid in table.rowids() {
6✔
1125
        let entry_bytes = build_row_entry(pager, table, rowid, &mut next_free_page)?;
4✔
1126

1127
        if !current_leaf.would_fit(entry_bytes.len()) {
4✔
1128
            // Commit the current leaf. Its sibling next_page is the page
1129
            // number where the new leaf will go — which is next_free_page
1130
            // right now (no overflow pages have been allocated between
1131
            // this decision and the new leaf's allocation below).
1132
            let next_leaf_page_num = next_free_page;
1✔
1133
            emit_leaf(pager, current_leaf_page, &current_leaf, next_leaf_page_num);
1✔
1134
            leaves.push((current_leaf_page, current_max_rowid.unwrap_or(i64::MIN)));
1✔
1135
            current_leaf = TablePage::empty();
1✔
1136
            current_leaf_page = next_leaf_page_num;
1✔
1137
            next_free_page += 1;
1✔
1138
            // current_max_rowid is reassigned by the insert below; no need
1139
            // to zero it out here.
1140

1141
            if !current_leaf.would_fit(entry_bytes.len()) {
2✔
1142
                return Err(SQLRiteError::Internal(format!(
×
1143
                    "entry of {} bytes exceeds empty-page capacity {}",
1144
                    entry_bytes.len(),
×
1145
                    current_leaf.free_space()
×
1146
                )));
1147
            }
1148
        }
1149
        current_leaf.insert_entry(rowid, &entry_bytes)?;
4✔
1150
        current_max_rowid = Some(rowid);
2✔
1151
    }
1152

1153
    // Final leaf: sibling next_page = 0 (end of chain).
1154
    emit_leaf(pager, current_leaf_page, &current_leaf, 0);
2✔
1155
    leaves.push((current_leaf_page, current_max_rowid.unwrap_or(i64::MIN)));
2✔
1156
    Ok((leaves, next_free_page))
2✔
1157
}
1158

1159
/// Encodes a single row's on-leaf entry — either the local cell bytes, or
1160
/// an `OverflowRef` pointing at a freshly-allocated overflow chain if the
1161
/// encoded cell exceeded the inline threshold. Advances `next_free_page`
1162
/// past any overflow pages used.
1163
fn build_row_entry(
2✔
1164
    pager: &mut Pager,
1165
    table: &Table,
1166
    rowid: i64,
1167
    next_free_page: &mut u32,
1168
) -> Result<Vec<u8>> {
1169
    let values = table.extract_row(rowid);
2✔
1170
    let local_cell = Cell::new(rowid, values);
2✔
1171
    let local_bytes = local_cell.encode()?;
4✔
1172
    if local_bytes.len() > OVERFLOW_THRESHOLD {
7✔
1173
        let overflow_start = *next_free_page;
1✔
1174
        *next_free_page = write_overflow_chain(pager, &local_bytes, overflow_start)?;
2✔
1175
        Ok(OverflowRef {
2✔
1176
            rowid,
1177
            total_body_len: local_bytes.len() as u64,
1✔
1178
            first_overflow_page: overflow_start,
1179
        }
1180
        .encode())
1✔
1181
    } else {
1182
        Ok(local_bytes)
2✔
1183
    }
1184
}
1185

1186
/// Builds one level of `InteriorNode` pages above the given children.
1187
/// Each interior packs as many dividers as will fit; the last child
1188
/// assigned to an interior becomes its `rightmost_child`. Returns the
1189
/// emitted interior pages as `(page_number, max_rowid_in_subtree)` so the
1190
/// next level can build on top of them.
1191
fn stage_interior_level(
1✔
1192
    pager: &mut Pager,
1193
    children: &[(u32, i64)],
1194
    start_page: u32,
1195
) -> Result<(Vec<(u32, i64)>, u32)> {
1196
    let mut next_level: Vec<(u32, i64)> = Vec::new();
1✔
1197
    let mut next_free_page = start_page;
1✔
1198
    let mut idx = 0usize;
1✔
1199

1200
    while idx < children.len() {
1✔
1201
        let interior_page_num = next_free_page;
1✔
1202
        next_free_page += 1;
2✔
1203

1204
        // Seed the interior with the first unassigned child as its
1205
        // rightmost. As we add more children, the previous rightmost
1206
        // graduates to being a divider and the new arrival takes over
1207
        // as rightmost.
1208
        let (mut rightmost_child_page, mut rightmost_child_max) = children[idx];
2✔
1209
        idx += 1;
2✔
1210
        let mut interior = InteriorPage::empty(rightmost_child_page);
2✔
1211

1212
        while idx < children.len() {
1✔
1213
            let new_divider_cell = InteriorCell {
1214
                divider_rowid: rightmost_child_max,
1215
                child_page: rightmost_child_page,
1216
            };
1217
            let new_divider_bytes = new_divider_cell.encode();
1✔
1218
            if !interior.would_fit(new_divider_bytes.len()) {
2✔
1219
                break;
1220
            }
1221
            interior.insert_divider(rightmost_child_max, rightmost_child_page)?;
2✔
1222
            let (next_child_page, next_child_max) = children[idx];
1✔
1223
            interior.set_rightmost_child(next_child_page);
1✔
1224
            rightmost_child_page = next_child_page;
1✔
1225
            rightmost_child_max = next_child_max;
1✔
1226
            idx += 1;
1✔
1227
        }
1228

1229
        emit_interior(pager, interior_page_num, &interior);
1✔
1230
        next_level.push((interior_page_num, rightmost_child_max));
1✔
1231
    }
1232

1233
    Ok((next_level, next_free_page))
1✔
1234
}
1235

1236
/// Wraps a `TablePage` in the 7-byte page header and hands it to the pager.
1237
fn emit_leaf(pager: &mut Pager, page_num: u32, leaf: &TablePage, next_leaf: u32) {
2✔
1238
    let mut buf = [0u8; PAGE_SIZE];
2✔
1239
    buf[0] = PageType::TableLeaf as u8;
2✔
1240
    buf[1..5].copy_from_slice(&next_leaf.to_le_bytes());
2✔
1241
    // For leaf pages the legacy `payload_len` field isn't used — the slot
1242
    // directory self-describes. Zero it by convention.
1243
    buf[5..7].copy_from_slice(&0u16.to_le_bytes());
2✔
1244
    buf[PAGE_HEADER_SIZE..].copy_from_slice(leaf.as_bytes());
2✔
1245
    pager.stage_page(page_num, buf);
2✔
1246
}
1247

1248
/// Wraps an `InteriorPage` in the 7-byte page header. Interior pages
1249
/// don't use `next_page` (there's no sibling chain between interiors);
1250
/// `payload_len` is also unused (the slot directory self-describes).
1251
fn emit_interior(pager: &mut Pager, page_num: u32, interior: &InteriorPage) {
1✔
1252
    let mut buf = [0u8; PAGE_SIZE];
1✔
1253
    buf[0] = PageType::InteriorNode as u8;
1✔
1254
    buf[1..5].copy_from_slice(&0u32.to_le_bytes());
1✔
1255
    buf[5..7].copy_from_slice(&0u16.to_le_bytes());
1✔
1256
    buf[PAGE_HEADER_SIZE..].copy_from_slice(interior.as_bytes());
1✔
1257
    pager.stage_page(page_num, buf);
1✔
1258
}
1259

1260
#[cfg(test)]
1261
mod tests {
1262
    use super::*;
1263
    use crate::sql::process_command;
1264

1265
    fn seed_db() -> Database {
1✔
1266
        let mut db = Database::new("test".to_string());
1✔
1267
        process_command(
1268
            "CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT NOT NULL UNIQUE, age INTEGER);",
1269
            &mut db,
1270
        )
1271
        .unwrap();
1272
        process_command(
1273
            "INSERT INTO users (name, age) VALUES ('alice', 30);",
1274
            &mut db,
1275
        )
1276
        .unwrap();
1277
        process_command("INSERT INTO users (name, age) VALUES ('bob', 25);", &mut db).unwrap();
1✔
1278
        process_command(
1279
            "CREATE TABLE notes (id INTEGER PRIMARY KEY, body TEXT);",
1280
            &mut db,
1281
        )
1282
        .unwrap();
1283
        process_command("INSERT INTO notes (body) VALUES ('hello');", &mut db).unwrap();
1✔
1284
        db
1✔
1285
    }
1286

1287
    fn tmp_path(name: &str) -> std::path::PathBuf {
1✔
1288
        let mut p = std::env::temp_dir();
1✔
1289
        let pid = std::process::id();
2✔
1290
        let nanos = std::time::SystemTime::now()
2✔
1291
            .duration_since(std::time::UNIX_EPOCH)
1✔
1292
            .map(|d| d.as_nanos())
3✔
1293
            .unwrap_or(0);
1294
        p.push(format!("sqlrite-{pid}-{nanos}-{name}.sqlrite"));
1✔
1295
        p
1✔
1296
    }
1297

1298
    /// Phase 4c: every .sqlrite has a `-wal` sidecar now. Delete both so
1299
    /// `/tmp` doesn't accumulate orphan WALs across test runs.
1300
    fn cleanup(path: &std::path::Path) {
1✔
1301
        let _ = std::fs::remove_file(path);
1✔
1302
        let mut wal = path.as_os_str().to_owned();
1✔
1303
        wal.push("-wal");
1✔
1304
        let _ = std::fs::remove_file(std::path::PathBuf::from(wal));
1✔
1305
    }
1306

1307
    #[test]
1308
    fn round_trip_preserves_schema_and_data() {
4✔
1309
        let path = tmp_path("roundtrip");
1✔
1310
        let mut db = seed_db();
1✔
1311
        save_database(&mut db, &path).expect("save");
2✔
1312

1313
        let loaded = open_database(&path, "test".to_string()).expect("open");
1✔
1314
        assert_eq!(loaded.tables.len(), 2);
2✔
1315

1316
        let users = loaded.get_table("users".to_string()).expect("users table");
1✔
1317
        assert_eq!(users.columns.len(), 3);
1✔
1318
        let rowids = users.rowids();
1✔
1319
        assert_eq!(rowids.len(), 2);
2✔
1320
        let names: Vec<String> = rowids
1✔
1321
            .iter()
1322
            .filter_map(|r| match users.get_value("name", *r) {
3✔
1323
                Some(Value::Text(s)) => Some(s),
1✔
1324
                _ => None,
×
1325
            })
1326
            .collect();
1327
        assert!(names.contains(&"alice".to_string()));
2✔
1328
        assert!(names.contains(&"bob".to_string()));
1✔
1329

1330
        let notes = loaded.get_table("notes".to_string()).expect("notes table");
1✔
1331
        assert_eq!(notes.rowids().len(), 1);
1✔
1332

1333
        cleanup(&path);
1✔
1334
    }
1335

1336
    // -----------------------------------------------------------------
1337
    // Phase 7a — VECTOR(N) save / reopen round-trip
1338
    // -----------------------------------------------------------------
1339

1340
    #[test]
1341
    fn round_trip_preserves_vector_column() {
3✔
1342
        let path = tmp_path("vec_roundtrip");
1✔
1343

1344
        // Build, populate, save.
1345
        {
1346
            let mut db = Database::new("test".to_string());
2✔
1347
            process_command(
1348
                "CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(3));",
1349
                &mut db,
1350
            )
1351
            .unwrap();
1352
            process_command(
1353
                "INSERT INTO docs (embedding) VALUES ([0.1, 0.2, 0.3]);",
1354
                &mut db,
1355
            )
1356
            .unwrap();
1357
            process_command(
1358
                "INSERT INTO docs (embedding) VALUES ([1.5, -2.0, 3.5]);",
1359
                &mut db,
1360
            )
1361
            .unwrap();
1362
            save_database(&mut db, &path).expect("save");
1✔
1363
        } // db drops → its exclusive lock releases before reopen.
1✔
1364

1365
        // Reopen and verify schema + data both round-tripped.
1366
        let loaded = open_database(&path, "test".to_string()).expect("open");
1✔
1367
        let docs = loaded.get_table("docs".to_string()).expect("docs table");
2✔
1368

1369
        // Schema preserved: column is still VECTOR(3).
1370
        let embedding_col = docs
3✔
1371
            .columns
1372
            .iter()
1373
            .find(|c| c.column_name == "embedding")
3✔
1374
            .expect("embedding column");
1375
        assert!(
×
1376
            matches!(embedding_col.datatype, DataType::Vector(3)),
1✔
1377
            "expected DataType::Vector(3) after round-trip, got {:?}",
1378
            embedding_col.datatype
1379
        );
1380

1381
        // Data preserved: both vectors still readable bit-for-bit.
1382
        let mut rows: Vec<Vec<f32>> = docs
1✔
1383
            .rowids()
1384
            .iter()
1385
            .filter_map(|r| match docs.get_value("embedding", *r) {
3✔
1386
                Some(Value::Vector(v)) => Some(v),
1✔
1387
                _ => None,
×
1388
            })
1389
            .collect();
1390
        rows.sort_by(|a, b| a[0].partial_cmp(&b[0]).unwrap());
3✔
1391
        assert_eq!(rows.len(), 2);
1✔
1392
        assert_eq!(rows[0], vec![0.1f32, 0.2, 0.3]);
1✔
1393
        assert_eq!(rows[1], vec![1.5f32, -2.0, 3.5]);
1✔
1394

1395
        cleanup(&path);
1✔
1396
    }
1397

1398
    #[test]
1399
    fn round_trip_preserves_json_column() {
4✔
1400
        // Phase 7e — JSON columns are stored as Text under the hood with
1401
        // INSERT-time validation. Save + reopen should preserve the
1402
        // schema (DataType::Json) and the underlying text bytes; a
1403
        // post-reopen json_extract should still resolve paths correctly.
1404
        let path = tmp_path("json_roundtrip");
1✔
1405

1406
        {
1407
            let mut db = Database::new("test".to_string());
2✔
1408
            process_command(
1409
                "CREATE TABLE docs (id INTEGER PRIMARY KEY, payload JSON);",
1410
                &mut db,
1411
            )
1412
            .unwrap();
1413
            process_command(
1414
                r#"INSERT INTO docs (payload) VALUES ('{"name": "alice", "tags": ["rust","sql"]}');"#,
1415
                &mut db,
1416
            )
1417
            .unwrap();
1418
            save_database(&mut db, &path).expect("save");
1✔
1419
        }
1420

1421
        let mut loaded = open_database(&path, "test".to_string()).expect("open");
1✔
1422
        let docs = loaded.get_table("docs".to_string()).expect("docs");
2✔
1423

1424
        // Schema: column declared as JSON, restored with the same type.
1425
        let payload_col = docs
3✔
1426
            .columns
1427
            .iter()
1428
            .find(|c| c.column_name == "payload")
3✔
1429
            .unwrap();
NEW
1430
        assert!(
×
1431
            matches!(payload_col.datatype, DataType::Json),
1✔
1432
            "expected DataType::Json, got {:?}",
1433
            payload_col.datatype
1434
        );
1435

1436
        // json_extract works against the reopened data — exercises the
1437
        // full Text-storage + serde_json::from_str path post-reopen.
1438
        let resp = process_command(
1439
            r#"SELECT id FROM docs WHERE json_extract(payload, '$.name') = 'alice';"#,
1440
            &mut loaded,
1441
        )
1442
        .expect("select via json_extract after reopen");
1443
        assert!(resp.contains("1 row returned"), "got: {resp}");
2✔
1444

1445
        cleanup(&path);
2✔
1446
    }
1447

1448
    #[test]
1449
    fn round_trip_rebuilds_hnsw_index_from_create_sql() {
3✔
1450
        // Phase 7d.3: HNSW indexes now persist their graph as cell-encoded
1451
        // pages. After save+reopen the index entry reattaches with the
1452
        // same column + same node count, loaded directly from disk
1453
        // instead of re-walking rows.
1454
        let path = tmp_path("hnsw_roundtrip");
1✔
1455

1456
        // Build, populate, index, save.
1457
        {
1458
            let mut db = Database::new("test".to_string());
2✔
1459
            process_command(
1460
                "CREATE TABLE docs (id INTEGER PRIMARY KEY, e VECTOR(2));",
1461
                &mut db,
1462
            )
1463
            .unwrap();
1464
            for v in &[
1✔
1465
                "[1.0, 0.0]",
1466
                "[2.0, 0.0]",
1467
                "[0.0, 3.0]",
1468
                "[1.0, 4.0]",
1469
                "[10.0, 10.0]",
1470
            ] {
1471
                process_command(&format!("INSERT INTO docs (e) VALUES ({v});"), &mut db).unwrap();
2✔
1472
            }
1473
            process_command("CREATE INDEX ix_e ON docs USING hnsw (e);", &mut db).unwrap();
1✔
1474
            save_database(&mut db, &path).expect("save");
1✔
1475
        } // db drops → exclusive lock releases.
1✔
1476

1477
        // Reopen and verify the index reattached, with the same name +
1478
        // column + populated graph.
1479
        let mut loaded = open_database(&path, "test".to_string()).expect("open");
1✔
1480
        {
1481
            let table = loaded.get_table("docs".to_string()).expect("docs");
2✔
1482
            assert_eq!(table.hnsw_indexes.len(), 1, "HNSW index should reattach");
1✔
1483
            let entry = &table.hnsw_indexes[0];
2✔
1484
            assert_eq!(entry.name, "ix_e");
1✔
1485
            assert_eq!(entry.column_name, "e");
1✔
1486
            assert_eq!(entry.index.len(), 5, "loaded graph should hold all 5 rows");
1✔
1487
            assert!(
×
1488
                !entry.needs_rebuild,
1✔
1489
                "fresh load should not be marked dirty"
1490
            );
1491
        }
1492

1493
        // Quick functional check: KNN query through the loaded index
1494
        // returns results.
1495
        let resp = process_command(
1496
            "SELECT id FROM docs ORDER BY vec_distance_l2(e, [1.0, 0.0]) ASC LIMIT 3;",
1497
            &mut loaded,
1498
        )
1499
        .unwrap();
1500
        assert!(resp.contains("3 rows returned"), "got: {resp}");
2✔
1501

1502
        cleanup(&path);
2✔
1503
    }
1504

1505
    #[test]
1506
    fn delete_then_save_then_reopen_excludes_deleted_node_from_hnsw() {
3✔
1507
        // Phase 7d.3 — DELETE marks HNSW dirty; save rebuilds it from
1508
        // current rows + serializes; reopen loads the post-delete graph.
1509
        // After all that, the deleted rowid must NOT come back from a
1510
        // KNN query.
1511
        let path = tmp_path("hnsw_delete_rebuild");
1✔
1512
        let mut db = Database::new("test".to_string());
2✔
1513
        process_command(
1514
            "CREATE TABLE docs (id INTEGER PRIMARY KEY, e VECTOR(2));",
1515
            &mut db,
1516
        )
1517
        .unwrap();
1518
        for v in &["[1.0, 0.0]", "[2.0, 0.0]", "[3.0, 0.0]", "[4.0, 0.0]"] {
1✔
1519
            process_command(&format!("INSERT INTO docs (e) VALUES ({v});"), &mut db).unwrap();
2✔
1520
        }
1521
        process_command("CREATE INDEX ix_e ON docs USING hnsw (e);", &mut db).unwrap();
1✔
1522

1523
        // Delete row 1 (the closest match to [0.5, 0.0]).
1524
        process_command("DELETE FROM docs WHERE id = 1;", &mut db).unwrap();
1✔
1525
        // Confirm it marked dirty.
1526
        let dirty_before_save = db.tables["docs"].hnsw_indexes[0].needs_rebuild;
1✔
1527
        assert!(dirty_before_save, "DELETE should mark dirty");
1✔
1528

1529
        save_database(&mut db, &path).expect("save");
2✔
1530
        // Confirm save cleared the dirty flag.
1531
        let dirty_after_save = db.tables["docs"].hnsw_indexes[0].needs_rebuild;
1✔
1532
        assert!(!dirty_after_save, "save should clear dirty");
1✔
1533
        drop(db);
1✔
1534

1535
        // Reopen, query for the closest match. Row 1 is gone; row 2
1536
        // (id=2, vector [2.0, 0.0]) should now be the nearest.
1537
        let loaded = open_database(&path, "test".to_string()).expect("open");
1✔
1538
        let docs = loaded.get_table("docs".to_string()).expect("docs");
2✔
1539

1540
        // Row 1 must not appear in any storage anymore.
1541
        assert!(
1✔
1542
            !docs.rowids().contains(&1),
2✔
1543
            "deleted row 1 should not be in row storage"
1544
        );
1545
        assert_eq!(docs.rowids().len(), 3, "should have 3 surviving rows");
1✔
1546

1547
        // The HNSW index must also have shed the deleted node.
1548
        assert_eq!(
1✔
1549
            docs.hnsw_indexes[0].index.len(),
1✔
1550
            3,
1551
            "HNSW graph should have shed the deleted node"
1552
        );
1553

1554
        cleanup(&path);
2✔
1555
    }
1556

1557
    #[test]
1558
    fn round_trip_survives_writes_after_load() {
3✔
1559
        let path = tmp_path("after_load");
1✔
1560
        save_database(&mut seed_db(), &path).unwrap();
2✔
1561

1562
        {
1563
            let mut db = open_database(&path, "test".to_string()).unwrap();
1✔
1564
            process_command(
1565
                "INSERT INTO users (name, age) VALUES ('carol', 40);",
1566
                &mut db,
1567
            )
1568
            .unwrap();
1569
            save_database(&mut db, &path).unwrap();
1✔
1570
        } // db drops → its exclusive lock releases before we reopen below.
1✔
1571

1572
        let db2 = open_database(&path, "test".to_string()).unwrap();
1✔
1573
        let users = db2.get_table("users".to_string()).unwrap();
2✔
1574
        assert_eq!(users.rowids().len(), 3);
1✔
1575

1576
        cleanup(&path);
1✔
1577
    }
1578

1579
    #[test]
1580
    fn open_rejects_garbage_file() {
3✔
1581
        let path = tmp_path("bad");
1✔
1582
        std::fs::write(&path, b"not a sqlrite database, just bytes").unwrap();
2✔
1583
        let result = open_database(&path, "x".to_string());
1✔
1584
        assert!(result.is_err());
2✔
1585
        cleanup(&path);
1✔
1586
    }
1587

1588
    #[test]
1589
    fn many_small_rows_spread_across_leaves() {
3✔
1590
        let path = tmp_path("many_rows");
1✔
1591
        let mut db = Database::new("big".to_string());
2✔
1592
        process_command(
1593
            "CREATE TABLE things (id INTEGER PRIMARY KEY, data TEXT);",
1594
            &mut db,
1595
        )
1596
        .unwrap();
1597
        for i in 0..200 {
1✔
1598
            let body = "x".repeat(200);
1✔
1599
            let q = format!("INSERT INTO things (data) VALUES ('row-{i}-{body}');");
2✔
1600
            process_command(&q, &mut db).unwrap();
2✔
1601
        }
1602
        save_database(&mut db, &path).unwrap();
1✔
1603
        let loaded = open_database(&path, "big".to_string()).unwrap();
1✔
1604
        let things = loaded.get_table("things".to_string()).unwrap();
2✔
1605
        assert_eq!(things.rowids().len(), 200);
1✔
1606
        cleanup(&path);
1✔
1607
    }
1608

1609
    #[test]
1610
    fn huge_row_goes_through_overflow() {
3✔
1611
        let path = tmp_path("overflow_row");
1✔
1612
        let mut db = Database::new("big".to_string());
2✔
1613
        process_command(
1614
            "CREATE TABLE docs (id INTEGER PRIMARY KEY, body TEXT);",
1615
            &mut db,
1616
        )
1617
        .unwrap();
1618
        let body = "A".repeat(10_000);
1✔
1619
        process_command(
1620
            &format!("INSERT INTO docs (body) VALUES ('{body}');"),
2✔
1621
            &mut db,
1622
        )
1623
        .unwrap();
1624
        save_database(&mut db, &path).unwrap();
1✔
1625

1626
        let loaded = open_database(&path, "big".to_string()).unwrap();
1✔
1627
        let docs = loaded.get_table("docs".to_string()).unwrap();
2✔
1628
        let rowids = docs.rowids();
1✔
1629
        assert_eq!(rowids.len(), 1);
2✔
1630
        let stored = docs.get_value("body", rowids[0]);
1✔
1631
        match stored {
1✔
1632
            Some(Value::Text(s)) => assert_eq!(s.len(), 10_000),
1✔
1633
            other => panic!("expected Text, got {other:?}"),
×
1634
        }
1635
        cleanup(&path);
1✔
1636
    }
1637

1638
    #[test]
1639
    fn create_sql_synthesis_round_trips() {
3✔
1640
        // Build a table via CREATE, then verify table_to_create_sql +
1641
        // parse_create_sql reproduce an equivalent column list.
1642
        let mut db = Database::new("x".to_string());
1✔
1643
        process_command(
1644
            "CREATE TABLE t (id INTEGER PRIMARY KEY, tag TEXT UNIQUE, note TEXT NOT NULL);",
1645
            &mut db,
1646
        )
1647
        .unwrap();
1648
        let t = db.get_table("t".to_string()).unwrap();
1✔
1649
        let sql = table_to_create_sql(t);
1✔
1650
        let (name, cols) = parse_create_sql(&sql).unwrap();
2✔
1651
        assert_eq!(name, "t");
2✔
1652
        assert_eq!(cols.len(), 3);
1✔
1653
        assert!(cols[0].is_pk);
1✔
1654
        assert!(cols[1].is_unique);
1✔
1655
        assert!(cols[2].not_null);
1✔
1656
    }
1657

1658
    #[test]
1659
    fn sqlrite_master_is_not_exposed_as_a_user_table() {
3✔
1660
        // After open, the public db.tables map should not list the master.
1661
        let path = tmp_path("no_master");
1✔
1662
        save_database(&mut seed_db(), &path).unwrap();
2✔
1663
        let loaded = open_database(&path, "x".to_string()).unwrap();
1✔
1664
        assert!(!loaded.tables.contains_key(MASTER_TABLE_NAME));
2✔
1665
        cleanup(&path);
2✔
1666
    }
1667

1668
    #[test]
1669
    fn multi_leaf_table_produces_an_interior_root() {
3✔
1670
        // 200 fat rows force the table into multiple leaves, which means
1671
        // save_database must build at least one InteriorNode above them.
1672
        // The test verifies the round-trip works and confirms the root is
1673
        // indeed an interior page (not a leaf) by reading the page type
1674
        // directly out of the open pager.
1675
        let path = tmp_path("multi_leaf_interior");
1✔
1676
        let mut db = Database::new("big".to_string());
2✔
1677
        process_command(
1678
            "CREATE TABLE things (id INTEGER PRIMARY KEY, data TEXT);",
1679
            &mut db,
1680
        )
1681
        .unwrap();
1682
        for i in 0..200 {
1✔
1683
            let body = "x".repeat(200);
1✔
1684
            let q = format!("INSERT INTO things (data) VALUES ('row-{i}-{body}');");
2✔
1685
            process_command(&q, &mut db).unwrap();
2✔
1686
        }
1687
        save_database(&mut db, &path).unwrap();
1✔
1688

1689
        // Confirm the round-trip preserved all 200 rows.
1690
        let loaded = open_database(&path, "big".to_string()).unwrap();
1✔
1691
        let things = loaded.get_table("things".to_string()).unwrap();
2✔
1692
        assert_eq!(things.rowids().len(), 200);
1✔
1693

1694
        // Peek at `things`'s root page via the pager attached to the
1695
        // loaded DB and check it's an InteriorNode, not a leaf.
1696
        let pager = loaded
2✔
1697
            .pager
1698
            .as_ref()
1699
            .expect("loaded DB should have a pager");
1700
        // sqlrite_master's row for `things` holds its root page. Easiest
1701
        // way to find it: walk the leaf chain by using find_leftmost_leaf
1702
        // and then hop one level up. Simpler: read the master, scan for
1703
        // the "things" row, look up rootpage.
1704
        let mut master = build_empty_master_table();
1✔
1705
        load_table_rows(pager, &mut master, pager.header().schema_root_page).unwrap();
2✔
1706
        let things_root = master
1✔
1707
            .rowids()
1708
            .into_iter()
1709
            .find_map(|r| match master.get_value("name", r) {
3✔
1710
                Some(Value::Text(s)) if s == "things" => match master.get_value("rootpage", r) {
3✔
1711
                    Some(Value::Integer(p)) => Some(p as u32),
1✔
1712
                    _ => None,
×
1713
                },
1714
                _ => None,
×
1715
            })
1716
            .expect("things should appear in sqlrite_master");
1717
        let root_buf = pager.read_page(things_root).unwrap();
1✔
1718
        assert_eq!(
1✔
1719
            root_buf[0],
1720
            PageType::InteriorNode as u8,
1721
            "expected a multi-leaf table to have an interior root, got tag {}",
1722
            root_buf[0]
×
1723
        );
1724

1725
        cleanup(&path);
2✔
1726
    }
1727

1728
    #[test]
1729
    fn explicit_index_persists_across_save_and_open() {
3✔
1730
        let path = tmp_path("idx_persist");
1✔
1731
        let mut db = Database::new("idx".to_string());
2✔
1732
        process_command(
1733
            "CREATE TABLE users (id INTEGER PRIMARY KEY, tag TEXT);",
1734
            &mut db,
1735
        )
1736
        .unwrap();
1737
        for i in 1..=5 {
1✔
1738
            let tag = if i % 2 == 0 { "odd" } else { "even" };
2✔
1739
            process_command(
1740
                &format!("INSERT INTO users (tag) VALUES ('{tag}');"),
1✔
1741
                &mut db,
1742
            )
1743
            .unwrap();
1744
        }
1745
        process_command("CREATE INDEX users_tag_idx ON users (tag);", &mut db).unwrap();
1✔
1746
        save_database(&mut db, &path).unwrap();
1✔
1747

1748
        let loaded = open_database(&path, "idx".to_string()).unwrap();
1✔
1749
        let users = loaded.get_table("users".to_string()).unwrap();
2✔
1750
        let idx = users
1✔
1751
            .index_by_name("users_tag_idx")
1752
            .expect("explicit index should survive save/open");
1753
        assert_eq!(idx.column_name, "tag");
1✔
1754
        assert!(!idx.is_unique);
1✔
1755
        // 5 rows: rowids 2, 4 are "odd" (i % 2 == 0 when i is 2 or 4) — 2 entries;
1756
        // rowids 1, 3, 5 are "even" (i % 2 != 0) — 3 entries.
1757
        let even_rowids = idx.lookup(&Value::Text("even".into()));
2✔
1758
        let odd_rowids = idx.lookup(&Value::Text("odd".into()));
1✔
1759
        assert_eq!(even_rowids.len(), 3);
1✔
1760
        assert_eq!(odd_rowids.len(), 2);
1✔
1761

1762
        cleanup(&path);
1✔
1763
    }
1764

1765
    #[test]
1766
    fn auto_indexes_for_unique_columns_survive_save_open() {
3✔
1767
        let path = tmp_path("auto_idx_persist");
1✔
1768
        let mut db = Database::new("a".to_string());
2✔
1769
        process_command(
1770
            "CREATE TABLE users (id INTEGER PRIMARY KEY, email TEXT NOT NULL UNIQUE);",
1771
            &mut db,
1772
        )
1773
        .unwrap();
1774
        process_command("INSERT INTO users (email) VALUES ('a@x');", &mut db).unwrap();
1✔
1775
        process_command("INSERT INTO users (email) VALUES ('b@x');", &mut db).unwrap();
1✔
1776
        save_database(&mut db, &path).unwrap();
1✔
1777

1778
        let loaded = open_database(&path, "a".to_string()).unwrap();
1✔
1779
        let users = loaded.get_table("users".to_string()).unwrap();
2✔
1780
        // Every UNIQUE column auto-creates an index; the load path populated
1781
        // it from the persisted entries.
1782
        let auto_name = SecondaryIndex::auto_name("users", "email");
1✔
1783
        let idx = users
1✔
1784
            .index_by_name(&auto_name)
2✔
1785
            .expect("auto index should be restored");
1786
        assert!(idx.is_unique);
1✔
1787
        assert_eq!(idx.lookup(&Value::Text("a@x".into())).len(), 1);
1✔
1788
        assert_eq!(idx.lookup(&Value::Text("b@x".into())).len(), 1);
1✔
1789

1790
        cleanup(&path);
1✔
1791
    }
1792

1793
    #[test]
1794
    fn deep_tree_round_trips() {
3✔
1795
        // Force a 3-level tree by bypassing process_command (which prints
1796
        // the full table on every INSERT, making large bulk loads O(N^2)
1797
        // in I/O). We build the Table directly via restore_row.
1798
        use crate::sql::db::table::Column as TableColumn;
1799

1800
        let path = tmp_path("deep_tree");
1✔
1801
        let mut db = Database::new("deep".to_string());
2✔
1802
        let columns = vec![
3✔
1803
            TableColumn::new("id".into(), "integer".into(), true, true, true),
2✔
1804
            TableColumn::new("s".into(), "text".into(), false, true, false),
2✔
1805
        ];
1806
        let mut table = build_empty_table("t", columns, 0);
1✔
1807
        // ~900-byte rows → ~4 rows per leaf. 6000 rows → ~1500 leaves,
1808
        // which with interior fanout ~400 needs 2 interior levels (3-level
1809
        // tree total, counting leaves).
1810
        for i in 1..=6_000i64 {
2✔
1811
            let body = "q".repeat(900);
1✔
1812
            table
1✔
1813
                .restore_row(
1814
                    i,
1815
                    vec![
3✔
1816
                        Some(Value::Integer(i)),
1✔
1817
                        Some(Value::Text(format!("r-{i}-{body}"))),
2✔
1818
                    ],
1819
                )
1820
                .unwrap();
1821
        }
1822
        db.tables.insert("t".to_string(), table);
1✔
1823
        save_database(&mut db, &path).unwrap();
1✔
1824

1825
        let loaded = open_database(&path, "deep".to_string()).unwrap();
1✔
1826
        let t = loaded.get_table("t".to_string()).unwrap();
2✔
1827
        assert_eq!(t.rowids().len(), 6_000);
1✔
1828

1829
        // Confirm the tree actually grew past 2 levels — i.e., the root's
1830
        // leftmost child is itself an interior page, not a leaf.
1831
        let pager = loaded.pager.as_ref().unwrap();
1✔
1832
        let mut master = build_empty_master_table();
1✔
1833
        load_table_rows(pager, &mut master, pager.header().schema_root_page).unwrap();
2✔
1834
        let t_root = master
1✔
1835
            .rowids()
1836
            .into_iter()
1837
            .find_map(|r| match master.get_value("name", r) {
3✔
1838
                Some(Value::Text(s)) if s == "t" => match master.get_value("rootpage", r) {
3✔
1839
                    Some(Value::Integer(p)) => Some(p as u32),
1✔
1840
                    _ => None,
×
1841
                },
1842
                _ => None,
×
1843
            })
1844
            .expect("t in sqlrite_master");
1845
        let root_buf = pager.read_page(t_root).unwrap();
1✔
1846
        assert_eq!(root_buf[0], PageType::InteriorNode as u8);
1✔
1847
        let root_payload: &[u8; PAYLOAD_PER_PAGE] =
1✔
1848
            (&root_buf[PAGE_HEADER_SIZE..]).try_into().unwrap();
1849
        let root_interior = InteriorPage::from_bytes(root_payload);
1✔
1850
        let child = root_interior.leftmost_child().unwrap();
2✔
1851
        let child_buf = pager.read_page(child).unwrap();
1✔
1852
        assert_eq!(
1✔
1853
            child_buf[0],
1854
            PageType::InteriorNode as u8,
1855
            "expected 3-level tree: root's leftmost child should also be InteriorNode",
1856
        );
1857

1858
        cleanup(&path);
2✔
1859
    }
1860
}
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2026 Coveralls, Inc