24983496245

Committed 27 Apr 2026 08:01AM UTC coverage: 67.574% (+0.7%) from 66.922%

Build # 24983496245

Build Type

push

github

Committed by

web-flow

Commit Message

Phase 7a: VECTOR(N) column type (storage only) (#42)

* Phase 7a: VECTOR(N) column type — storage only

First sub-phase of the AI-era extensions. Adds a fixed-dimension
dense f32 array as a first-class SQL data type, with the cell
encoding + parser plumbing it needs. No distance functions or
similarity search yet — those land in 7b/7c/7d on top of this.

What works:

  CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(384));
  INSERT INTO docs (embedding) VALUES ([0.1, 0.2, ..., 0.0]);
  SELECT * FROM docs;

Roundtrips through save / open. Wrong-dimension INSERTs are rejected
with a clean typed error. Empty / non-positive / non-numeric
dimensions in CREATE TABLE are rejected with descriptive messages.

**File format bumped to v4** (see docs/file-format.md). Per the
Phase 7 plan's Q8 decision, all Phase 7 storage additions
(VECTOR + JSON + HNSW indexes coming next) live inside this
single v4 envelope — no v5 mid-Phase-7. Old v3 files reject on
open with the standard "unsupported format version" error.

**Engine plumbing** (~300 LOC + 22 new tests, 184 total now passing):

  - DataType::Vector(usize) + parser handlers in DataType::new
    that round-trip through the existing string-based ParsedColumn
    pipeline (encoded as `vector(N)` in the wire string).
  - Value::Vector(Vec<f32>) at the runtime layer.
  - Row::Vector(BTreeMap<i64, Vec<f32>>) at the storage layer.
  - All 6 existing match arms in table.rs / cell.rs / executor.rs
    / pager/mod.rs extended with Vector cases.
  - Cell value tag 0x04 = VECTOR. Wire layout: tag (1 byte) +
    dim (varint) + dim×4 bytes f32 little-endian. Self-describing
    so decode_value works without schema context.

**Parser** (Q6 + Q7 decisions baked in):

  - CREATE TABLE: sqlparser parses `VECTOR(N)` as DataType::Custom
    with a Vec<String> of args. New is_vector_type() + parse_vector_dim()
    helpers in parser/create.rs translate to internal `vector(N)`
    string.
  - INSERT VALUES: sqlparser sees... (continued)

Coverage Stats

246 of 313 new or added lines in 7 files covered. (78.59%)

7 existing lines in 1 file now uncovered.

4247 of 6285 relevant lines covered (67.57%)

1.24 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

60.0

/src/sql/parser/insert.rs

use sqlparser::ast::{Expr, Insert, SetExpr, Statement, Value, Values};

use crate::error::{Result, SQLRiteError};

/// The following structure represents a INSERT query already parsed
/// and broken down into `table_name` a `Vec<String>` representing the `Columns`
/// and `Vec<Vec<String>>` representing the list of `Rows` to be inserted
#[derive(Debug)]
pub struct InsertQuery {
    pub table_name: String,
    pub columns: Vec<String>,
    pub rows: Vec<Vec<String>>,
}

impl InsertQuery {
    pub fn new(statement: &Statement) -> Result<InsertQuery> {
        let tname: Option<String>;
        let mut columns: Vec<String> = vec![];
        let mut all_values: Vec<Vec<String>> = vec![];

        match statement {
            Statement::Insert(Insert {
                table,
                columns: cols,
                source,
                ..
            }) => {
                tname = Some(table.to_string());
                for col in cols {
                    columns.push(col.to_string());
                }

                let source = source.as_ref().ok_or_else(|| {
                    SQLRiteError::Internal(
                        "INSERT statement is missing a source expression".to_string(),
                    )
                })?;

                if let SetExpr::Values(Values { rows, .. }) = source.body.as_ref() {
                    for row in rows {
                        let mut value_set: Vec<String> = vec![];
                        for e in row {
                            match e {
                                Expr::Value(v) => match &v.value {
                                    Value::Number(n, _) => {
                                        value_set.push(n.to_string());
                                    }
                                    Value::Boolean(b) => {
                                        if *b {
                                            value_set.push("true".to_string());
                                        } else {
                                            value_set.push("false".to_string());
                                        }
                                    }
                                    Value::SingleQuotedString(sqs) => {
                                        value_set.push(sqs.to_string());
                                    }
                                    Value::Null => {
                                        value_set.push("Null".to_string());
                                    }
                                    _ => {}
                                },
                                Expr::Identifier(i) => {
                                    // Phase 7a — sqlparser parses bracket-array
                                    // literals like `[0.1, 0.2, 0.3]` as
                                    // bracket-quoted identifiers (it inherits
                                    // MSSQL-style `[name]` quoting). Detect
                                    // that by `quote_style == Some('[')` and
                                    // re-wrap with brackets so the
                                    // `parse_vector_literal` helper at
                                    // insert_row time can recognize and parse
                                    // it. Regular unquoted identifiers (column
                                    // refs, which don't make sense in INSERT
                                    // VALUES anyway) keep the existing
                                    // pass-through-as-string behavior.
                                    if i.quote_style == Some('[') {
                                        value_set.push(format!("[{}]", i.value));
                                    } else {
                                        value_set.push(i.to_string());
                                    }
                                }
                                _ => {}
                            }
                        }
                        all_values.push(value_set);
                    }
                }
            }
            _ => {
                return Err(SQLRiteError::Internal(
                    "Error parsing insert query".to_string(),
                ));
            }
        }

        match tname {
            Some(t) => Ok(InsertQuery {
                table_name: t,
                columns,
                rows: all_values,
            }),
            None => Err(SQLRiteError::Internal(
                "Error parsing insert query".to_string(),
            )),
        }
    }
}

1	use sqlparser::ast::{Expr, Insert, SetExpr, Statement, Value, Values};
2
3	use crate::error::{Result, SQLRiteError};
4
5	/// The following structure represents a INSERT query already parsed
6	/// and broken down into `table_name` a `Vec<String>` representing the `Columns`
7	/// and `Vec<Vec<String>>` representing the list of `Rows` to be inserted
8	#[derive(Debug)]
9	pub struct InsertQuery {
10	pub table_name: String,
11	pub columns: Vec<String>,
12	pub rows: Vec<Vec<String>>,
13	}
14
15	impl InsertQuery {
16	pub fn new(statement: &Statement) -> Result<InsertQuery> {	2✔
17	let tname: Option<String>;	2✔
18	let mut columns: Vec<String> = vec![];	2✔
19	let mut all_values: Vec<Vec<String>> = vec![];	2✔
20
21	match statement {	2✔
22	Statement::Insert(Insert {	×
23	table,	2✔
24	columns: cols,	2✔
25	source,	2✔
26	..	×
27	}) => {	×
28	tname = Some(table.to_string());	4✔
29	for col in cols {	2✔
30	columns.push(col.to_string());	4✔
31	}
32
33	let source = source.as_ref().ok_or_else(\|\| {	2✔
34	SQLRiteError::Internal(	×
35	"INSERT statement is missing a source expression".to_string(),	×
36	)
37	})?;
38
39	if let SetExpr::Values(Values { rows, .. }) = source.body.as_ref() {	2✔
40	for row in rows {	6✔
41	let mut value_set: Vec<String> = vec![];	2✔
42	for e in row {	4✔
43	match e {	2✔
44	Expr::Value(v) => match &v.value {	2✔
45	Value::Number(n, _) => {	1✔
46	value_set.push(n.to_string());	2✔
47	}
48	Value::Boolean(b) => {	×
49	if *b {	×
50	value_set.push("true".to_string());	×
51	} else {
52	value_set.push("false".to_string());	×
53	}
54	}
55	Value::SingleQuotedString(sqs) => {	2✔
56	value_set.push(sqs.to_string());	4✔
57	}
58	Value::Null => {	×
59	value_set.push("Null".to_string());	×
60	}
61	_ => {}	×
62	},
63	Expr::Identifier(i) => {	1✔
64	// Phase 7a — sqlparser parses bracket-array
65	// literals like `[0.1, 0.2, 0.3]` as
66	// bracket-quoted identifiers (it inherits
67	// MSSQL-style `[name]` quoting). Detect
68	// that by `quote_style == Some('[')` and
69	// re-wrap with brackets so the
70	// `parse_vector_literal` helper at
71	// insert_row time can recognize and parse
72	// it. Regular unquoted identifiers (column
73	// refs, which don't make sense in INSERT
74	// VALUES anyway) keep the existing
75	// pass-through-as-string behavior.
76	if i.quote_style == Some('[') {	2✔
77	value_set.push(format!("[{}]", i.value));	2✔
78	} else {
NEW 79	value_set.push(i.to_string());	×
80	}
81	}
82	_ => {}	×
83	}
84	}
85	all_values.push(value_set);	2✔
86	}
87	}
88	}
89	_ => {	×
90	return Err(SQLRiteError::Internal(	×
91	"Error parsing insert query".to_string(),	×
92	));
93	}
94	}
95
96	match tname {	2✔
97	Some(t) => Ok(InsertQuery {	4✔
98	table_name: t,	×
99	columns,	2✔
100	rows: all_values,	2✔
101	}),
102	None => Err(SQLRiteError::Internal(	×
103	"Error parsing insert query".to_string(),	×
104	)),
105	}
106	}
107	}

joaoh82 / rust_sqlite / 24983496245

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous