joaoh82 / rust_sqlite / 24983496245

69%

Build Type

push

github

Committed by web-flow

Commit Message

Phase 7a: VECTOR(N) column type (storage only) (#42)

* Phase 7a: VECTOR(N) column type — storage only

First sub-phase of the AI-era extensions. Adds a fixed-dimension
dense f32 array as a first-class SQL data type, with the cell
encoding + parser plumbing it needs. No distance functions or
similarity search yet — those land in 7b/7c/7d on top of this.

What works:

  CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(384));
  INSERT INTO docs (embedding) VALUES ([0.1, 0.2, ..., 0.0]);
  SELECT * FROM docs;

Roundtrips through save / open. Wrong-dimension INSERTs are rejected
with a clean typed error. Empty / non-positive / non-numeric
dimensions in CREATE TABLE are rejected with descriptive messages.

**File format bumped to v4** (see docs/file-format.md). Per the
Phase 7 plan's Q8 decision, all Phase 7 storage additions
(VECTOR + JSON + HNSW indexes coming next) live inside this
single v4 envelope — no v5 mid-Phase-7. Old v3 files reject on
open with the standard "unsupported format version" error.

**Engine plumbing** (~300 LOC + 22 new tests, 184 total now passing):

  - DataType::Vector(usize) + parser handlers in DataType::new
    that round-trip through the existing string-based ParsedColumn
    pipeline (encoded as `vector(N)` in the wire string).
  - Value::Vector(Vec<f32>) at the runtime layer.
  - Row::Vector(BTreeMap<i64, Vec<f32>>) at the storage layer.
  - All 6 existing match arms in table.rs / cell.rs / executor.rs
    / pager/mod.rs extended with Vector cases.
  - Cell value tag 0x04 = VECTOR. Wire layout: tag (1 byte) +
    dim (varint) + dim×4 bytes f32 little-endian. Self-describing
    so decode_value works without schema context.

**Parser** (Q6 + Q7 decisions baked in):

  - CREATE TABLE: sqlparser parses `VECTOR(N)` as DataType::Custom
    with a Vec<String> of args. New is_vector_type() + parse_vector_dim()
    helpers in parser/create.rs translate to internal `vector(N)`
    string.
  - INSERT VALUES: sqlparser sees... (continued)

Coverage Stats

246 of 313 new or added lines in 7 files covered. (78.59%)

7 existing lines in 1 file now uncovered.

4247 of 6285 relevant lines covered (67.57%)

1.24 hits per line