16906445831

Committed 12 Aug 2025 10:41AM UTC coverage: 87.043% (+0.4%) from 86.68%

Build # 16906445831

Build Type

push

github

Committed by

web-flow

Commit Message

chore: improve vortex-dtype test coverage (#4204)

# Summary of Changes in vortex-dtype

## 1. **DecimalDType Safety Improvements** ✅
- Added `try_new()` fallible constructor that returns `Result`
- Added `MIN_SCALE` constant (-76) validation 
- `new()` now calls `try_new()` internally with panic on error
- Validates precision ≤ MAX_PRECISION (76)
- Validates scale within [MIN_SCALE, MAX_SCALE] range
- Updated all serialization code to use `try_new()`:
  - `arrow.rs`: Arrow type conversions
  - `serde/flatbuffers/mod.rs`: FlatBuffer deserialization  
  - `serde/proto.rs`: Protobuf deserialization

## 2. **StructFields API Enhancement** ✅
- Added `try_without_field()` that returns `Result` for safe field
removal
- Deprecated `without_field()` which panics on invalid index
- Proper bounds checking with descriptive error messages

## 3. **Comprehensive Test Coverage** ✅
### decimal.rs (160 new lines)
- Boundary testing for MAX/MIN precision and scale
- Negative scale validation
- Bit width calculations
- Type conversion tests

### field_mask.rs (219 new lines)  
- All three mask types (All, Prefix, Exact)
- `step_into()` transitions
- `starting_field()` behavior
- Helper functions for test construction
- Removed TODO comment about API improvements

### nullability.rs (67 new lines)
- BitOr operator combinations
- Bool conversions and round-trips
- Chained operations

### serde/flatbuffers/project.rs (172 new lines)
- Field resolution by name
- Field extraction from structs
- Projection and deserialization
- Error cases for missing fields

### serde/proto.rs (249 new lines)
- Round-trip tests for all DType variants
- PType conversions
- FieldPath serialization
- Error cases for invalid data

## 4. **Other Crates Updated** ✅
- `vortex-duckdb`: Uses `DecimalDType::new()` (still safe due to
internal try_new)
- `vortex-python`: Uses `DecimalDType::new()` (still safe)
- Both maintain backwards compatibility

## Summary Statistics
- **Files changed**: 12
- **In... (continued)

Run Details

650 of 655 new or added lines in 12 files covered. (99.24%)

1 existing line in 1 file now uncovered.

55118 of 63323 relevant lines covered (87.04%)

535056.59 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

89.64

/vortex-array/src/arrays/struct_/mod.rs

// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright the Vortex contributors

use std::fmt::Debug;
use std::iter::once;

use itertools::Itertools;
use vortex_dtype::{DType, FieldName, FieldNames, StructFields};
use vortex_error::{VortexExpect, VortexResult, vortex_bail, vortex_err};
use vortex_scalar::Scalar;

use crate::stats::{ArrayStats, StatsSetRef};
use crate::validity::Validity;
use crate::vtable::{
    ArrayVTable, CanonicalVTable, NotSupported, OperationsVTable, VTable, ValidityHelper,
    ValidityVTableFromValidityHelper,
};
use crate::{Array, ArrayRef, Canonical, EncodingId, EncodingRef, IntoArray, vtable};

mod compute;
mod serde;

vtable!(Struct);

impl VTable for StructVTable {
    type Array = StructArray;
    type Encoding = StructEncoding;

    type ArrayVTable = Self;
    type CanonicalVTable = Self;
    type OperationsVTable = Self;
    type ValidityVTable = ValidityVTableFromValidityHelper;
    type VisitorVTable = Self;
    type ComputeVTable = NotSupported;
    type EncodeVTable = NotSupported;
    type SerdeVTable = Self;

    fn id(_encoding: &Self::Encoding) -> EncodingId {
        EncodingId::new_ref("vortex.struct")
    }

    fn encoding(_array: &Self::Array) -> EncodingRef {
        EncodingRef::new_ref(StructEncoding.as_ref())
    }
}

/// A struct array that stores multiple named fields as columns, similar to a database row.
///
/// This mirrors the Apache Arrow Struct array encoding and provides a columnar representation
/// of structured data where each row contains multiple named fields of potentially different types.
///
/// ## Data Layout
///
/// The struct array uses a columnar layout where:
/// - Each field is stored as a separate child array
/// - All fields must have the same length (number of rows)
/// - Field names and types are defined in the struct's dtype
/// - An optional validity mask indicates which entire rows are null
///
/// ## Row-level nulls
///
/// The StructArray contains its own top-level nulls, which are superimposed on top of the
/// field-level validity values. This can be the case even if the fields themselves are non-nullable,
/// accessing a particular row can yield nulls even if all children are valid at that position.
///
/// ```
/// use vortex_array::arrays::{StructArray, BoolArray};
/// use vortex_array::validity::Validity;
/// use vortex_array::IntoArray;
/// use vortex_dtype::FieldNames;
/// use vortex_buffer::buffer;
///
/// // Create struct with all non-null fields but struct-level nulls
/// let struct_array = StructArray::try_new(
///     FieldNames::from(["a", "b", "c"]),
///     vec![
///         buffer![1i32, 2i32].into_array(),  // non-null field a
///         buffer![10i32, 20i32].into_array(), // non-null field b  
///         buffer![100i32, 200i32].into_array(), // non-null field c
///     ],
///     2,
///     Validity::Array(BoolArray::from_iter([true, false]).into_array()), // row 1 is null
/// ).unwrap();
///
/// // Row 0 is valid - returns a struct scalar with field values
/// let row0 = struct_array.scalar_at(0).unwrap();
/// assert!(!row0.is_null());
///
/// // Row 1 is null at struct level - returns null even though fields have values
/// let row1 = struct_array.scalar_at(1).unwrap();
/// assert!(row1.is_null());
/// ```
///
/// ## Name uniqueness
///
/// It is valid for a StructArray to have multiple child columns that have the same name. In this
/// case, any accessors that use column names will find the first column in sequence with the name.
///
/// ```
/// use vortex_array::arrays::StructArray;
/// use vortex_array::validity::Validity;
/// use vortex_array::IntoArray;
/// use vortex_dtype::FieldNames;
/// use vortex_buffer::buffer;
///
/// // Create struct with duplicate "data" field names
/// let struct_array = StructArray::try_new(
///     FieldNames::from(["data", "data"]),
///     vec![
///         buffer![1i32, 2i32].into_array(),   // first "data"
///         buffer![3i32, 4i32].into_array(),   // second "data"
///     ],
///     2,
///     Validity::NonNullable,
/// ).unwrap();
///
/// // field_by_name returns the FIRST "data" field
/// let first_data = struct_array.field_by_name("data").unwrap();
/// assert_eq!(first_data.scalar_at(0).unwrap(), 1i32.into());
/// ```
///
/// ## Field Operations
///
/// Struct arrays support efficient column operations:
/// - **Projection**: Select/reorder fields without copying data
/// - **Field access**: Get columns by name or index
/// - **Column addition**: Add new fields to create extended structs
/// - **Column removal**: Remove fields to create narrower structs
///
/// ## Validity Semantics
///
/// - Row-level nulls are tracked in the struct's validity child
/// - Individual field nulls are tracked in each field's own validity
/// - A null struct row means all fields in that row are conceptually null
/// - Field-level nulls can exist independently of struct-level nulls
///
/// # Examples
///
/// ```
/// use vortex_array::arrays::{StructArray, PrimitiveArray};
/// use vortex_array::validity::Validity;
/// use vortex_array::IntoArray;
/// use vortex_dtype::FieldNames;
/// use vortex_buffer::buffer;
///
/// // Create arrays for each field
/// let ids = PrimitiveArray::new(buffer![1i32, 2, 3], Validity::NonNullable);
/// let names = PrimitiveArray::new(buffer![100u64, 200, 300], Validity::NonNullable);
///
/// // Create struct array with named fields
/// let struct_array = StructArray::try_new(
///     FieldNames::from(["id", "score"]),
///     vec![ids.into_array(), names.into_array()],
///     3,
///     Validity::NonNullable,
/// ).unwrap();
///
/// assert_eq!(struct_array.len(), 3);
/// assert_eq!(struct_array.names().len(), 2);
///
/// // Access field by name
/// let id_field = struct_array.field_by_name("id").unwrap();
/// assert_eq!(id_field.len(), 3);
/// ```
#[derive(Clone, Debug)]
pub struct StructArray {
    len: usize,
    dtype: DType,
    fields: Vec<ArrayRef>,
    validity: Validity,
    stats_set: ArrayStats,
}

#[derive(Clone, Debug)]
pub struct StructEncoding;

impl StructArray {
    pub fn fields(&self) -> &[ArrayRef] {
        &self.fields
    }

    pub fn field_by_name(&self, name: impl AsRef<str>) -> VortexResult<&ArrayRef> {
        let name = name.as_ref();
        self.field_by_name_opt(name).ok_or_else(|| {
            vortex_err!(
                "Field {name} not found in struct array with names {:?}",
                self.names()
            )
        })
    }

    pub fn field_by_name_opt(&self, name: impl AsRef<str>) -> Option<&ArrayRef> {
        let name = name.as_ref();
        self.names()
            .iter()
            .position(|field_name| field_name.as_ref() == name)
            .map(|idx| &self.fields[idx])
    }

    pub fn names(&self) -> &FieldNames {
        self.struct_fields().names()
    }

    pub fn struct_fields(&self) -> &StructFields {
        let Some(struct_dtype) = &self.dtype.as_struct() else {
            unreachable!(
                "struct arrays must have be a DType::Struct, this is likely an internal bug."
            )
        };
        struct_dtype
    }

    /// Create a new `StructArray` with the given length, but without any fields.
    pub fn new_with_len(len: usize) -> Self {
        Self::try_new(
            FieldNames::default(),
            Vec::new(),
            len,
            Validity::NonNullable,
        )
        .vortex_expect("StructArray::new_with_len should not fail")
    }

    pub fn try_new(
        names: FieldNames,
        fields: Vec<ArrayRef>,
        length: usize,
        validity: Validity,
    ) -> VortexResult<Self> {
        let nullability = validity.nullability();

        if names.len() != fields.len() {
            vortex_bail!("Got {} names and {} fields", names.len(), fields.len());
        }

        for field in fields.iter() {
            if field.len() != length {
                vortex_bail!(
                    "Expected all struct fields to have length {length}, found {}",
                    fields.iter().map(|f| f.len()).format(","),
                );
            }
        }

        let field_dtypes: Vec<_> = fields.iter().map(|d| d.dtype()).cloned().collect();
        let dtype = DType::Struct(StructFields::new(names, field_dtypes), nullability);

        if length != validity.maybe_len().unwrap_or(length) {
            vortex_bail!(
                "array length {} and validity length must match {}",
                length,
                validity
                    .maybe_len()
                    .vortex_expect("can only fail if maybe is some")
            )
        }

        Ok(Self {
            len: length,
            dtype,
            fields,
            validity,
            stats_set: Default::default(),
        })
    }

    pub fn try_new_with_dtype(
        fields: Vec<ArrayRef>,
        dtype: StructFields,
        length: usize,
        validity: Validity,
    ) -> VortexResult<Self> {
        for (field, struct_dt) in fields.iter().zip(dtype.fields()) {
            if field.len() != length {
                vortex_bail!(
                    "Expected all struct fields to have length {length}, found {}",
                    field.len()
                );
            }

            if &struct_dt != field.dtype() {
                vortex_bail!(
                    "Expected all struct fields to have dtype {}, found {}",
                    struct_dt,
                    field.dtype()
                );
            }
        }

        Ok(Self {
            len: length,
            dtype: DType::Struct(dtype, validity.nullability()),
            fields,
            validity,
            stats_set: Default::default(),
        })
    }

    pub fn from_fields<N: AsRef<str>>(items: &[(N, ArrayRef)]) -> VortexResult<Self> {
        Self::try_from_iter(items.iter().map(|(a, b)| (a, b.to_array())))
    }

    pub fn try_from_iter_with_validity<
        N: AsRef<str>,
        A: IntoArray,
        T: IntoIterator<Item = (N, A)>,
    >(
        iter: T,
        validity: Validity,
    ) -> VortexResult<Self> {
        let (names, fields): (Vec<FieldName>, Vec<ArrayRef>) = iter
            .into_iter()
            .map(|(name, fields)| (FieldName::from(name.as_ref()), fields.into_array()))
            .unzip();
        let len = fields
            .first()
            .map(|f| f.len())
            .ok_or_else(|| vortex_err!("StructArray cannot be constructed from an empty slice of arrays because the length is unspecified"))?;

        Self::try_new(FieldNames::from_iter(names), fields, len, validity)
    }

    pub fn try_from_iter<N: AsRef<str>, A: IntoArray, T: IntoIterator<Item = (N, A)>>(
        iter: T,
    ) -> VortexResult<Self> {
        Self::try_from_iter_with_validity(iter, Validity::NonNullable)
    }

    // TODO(aduffy): Add equivalent function to support field masks for nested column access.
    /// Return a new StructArray with the given projection applied.
    ///
    /// Projection does not copy data arrays. Projection is defined by an ordinal array slice
    /// which specifies the new ordering of columns in the struct. The projection can be used to
    /// perform column re-ordering, deletion, or duplication at a logical level, without any data
    /// copying.
    #[allow(clippy::same_name_method)]
    pub fn project(&self, projection: &[FieldName]) -> VortexResult<Self> {
        let mut children = Vec::with_capacity(projection.len());
        let mut names = Vec::with_capacity(projection.len());

        for f_name in projection.iter() {
            let idx = self
                .names()
                .iter()
                .position(|name| name == f_name)
                .ok_or_else(|| vortex_err!("Unknown field {f_name}"))?;

            names.push(self.names()[idx].clone());
            children.push(self.fields()[idx].clone());
        }

        StructArray::try_new(
            FieldNames::from(names.as_slice()),
            children,
            self.len(),
            self.validity().clone(),
        )
    }

    /// Removes and returns a column from the struct array by name.
    /// If the column does not exist, returns `None`.
    pub fn remove_column(&mut self, name: impl Into<FieldName>) -> Option<ArrayRef> {
        let name = name.into();

        let struct_dtype = self.struct_fields().clone();

        let position = struct_dtype
            .names()
            .iter()
            .position(|field_name| field_name.as_ref() == name.as_ref())?;

        let field = self.fields.remove(position);

        if let Ok(new_dtype) = struct_dtype.without_field(position) {
            self.dtype = DType::Struct(new_dtype, self.dtype.nullability());
            return Some(field);
        }
        None
    }

    /// Create a new StructArray by appending a new column onto the existing array.
    pub fn with_column(&self, name: impl Into<FieldName>, array: ArrayRef) -> VortexResult<Self> {
        let name = name.into();
        let struct_dtype = self.struct_fields().clone();

        let names = struct_dtype.names().iter().cloned().chain(once(name));
        let types = struct_dtype.fields().chain(once(array.dtype().clone()));
        let new_fields = StructFields::new(names.collect(), types.collect());

        let mut children = self.fields.clone();
        children.push(array);

        Self::try_new_with_dtype(children, new_fields, self.len, self.validity.clone())
    }
}

impl ValidityHelper for StructArray {
    fn validity(&self) -> &Validity {
        &self.validity
    }
}

impl ArrayVTable<StructVTable> for StructVTable {
    fn len(array: &StructArray) -> usize {
        array.len
    }

    fn dtype(array: &StructArray) -> &DType {
        &array.dtype
    }

    fn stats(array: &StructArray) -> StatsSetRef<'_> {
        array.stats_set.to_ref(array.as_ref())
    }
}

impl CanonicalVTable<StructVTable> for StructVTable {
    fn canonicalize(array: &StructArray) -> VortexResult<Canonical> {
        Ok(Canonical::Struct(array.clone()))
    }
}

impl OperationsVTable<StructVTable> for StructVTable {
    fn slice(array: &StructArray, start: usize, stop: usize) -> VortexResult<ArrayRef> {
        let fields = array
            .fields()
            .iter()
            .map(|field| field.slice(start, stop))
            .try_collect()?;
        StructArray::try_new_with_dtype(
            fields,
            array.struct_fields().clone(),
            stop - start,
            array.validity().slice(start, stop)?,
        )
        .map(|a| a.into_array())
    }

    fn scalar_at(array: &StructArray, index: usize) -> VortexResult<Scalar> {
        if array.is_valid(index)? {
            Ok(Scalar::struct_(
                array.dtype().clone(),
                array
                    .fields()
                    .iter()
                    .map(|field| field.scalar_at(index))
                    .try_collect()?,
            ))
        } else {
            Ok(Scalar::null(array.dtype().clone()))
        }
    }
}

#[cfg(test)]
mod test {
    use vortex_buffer::buffer;
    use vortex_dtype::{DType, FieldName, FieldNames, Nullability, PType};

    use crate::IntoArray;
    use crate::arrays::primitive::PrimitiveArray;
    use crate::arrays::struct_::StructArray;
    use crate::arrays::varbin::VarBinArray;
    use crate::arrays::{BoolArray, BoolVTable, PrimitiveVTable};
    use crate::validity::Validity;

    #[test]
    fn test_project() {
        let xs = PrimitiveArray::new(buffer![0i64, 1, 2, 3, 4], Validity::NonNullable);
        let ys = VarBinArray::from_vec(
            vec!["a", "b", "c", "d", "e"],
            DType::Utf8(Nullability::NonNullable),
        );
        let zs = BoolArray::from_iter([true, true, true, false, false]);

        let struct_a = StructArray::try_new(
            FieldNames::from(["xs", "ys", "zs"]),
            vec![xs.into_array(), ys.into_array(), zs.into_array()],
            5,
            Validity::NonNullable,
        )
        .unwrap();

        let struct_b = struct_a
            .project(&[FieldName::from("zs"), FieldName::from("xs")])
            .unwrap();
        assert_eq!(
            struct_b.names().as_ref(),
            [FieldName::from("zs"), FieldName::from("xs")],
        );

        assert_eq!(struct_b.len(), 5);

        let bools = &struct_b.fields[0];
        assert_eq!(
            bools
                .as_::<BoolVTable>()
                .boolean_buffer()
                .iter()
                .collect::<Vec<_>>(),
            vec![true, true, true, false, false]
        );

        let prims = &struct_b.fields[1];
        assert_eq!(
            prims.as_::<PrimitiveVTable>().as_slice::<i64>(),
            [0i64, 1, 2, 3, 4]
        );
    }

    #[test]
    fn test_remove_column() {
        let xs = PrimitiveArray::new(buffer![0i64, 1, 2, 3, 4], Validity::NonNullable);
        let ys = PrimitiveArray::new(buffer![4u64, 5, 6, 7, 8], Validity::NonNullable);

        let mut struct_a = StructArray::try_new(
            FieldNames::from(["xs", "ys"]),
            vec![xs.into_array(), ys.into_array()],
            5,
            Validity::NonNullable,
        )
        .unwrap();

        let removed = struct_a.remove_column("xs").unwrap();
        assert_eq!(
            removed.dtype(),
            &DType::Primitive(PType::I64, Nullability::NonNullable)
        );
        assert_eq!(
            removed.as_::<PrimitiveVTable>().as_slice::<i64>(),
            [0i64, 1, 2, 3, 4]
        );

        assert_eq!(struct_a.names(), &[FieldName::from("ys")].into());
        assert_eq!(struct_a.fields.len(), 1);
        assert_eq!(struct_a.len(), 5);
        assert_eq!(
            struct_a.fields[0].dtype(),
            &DType::Primitive(PType::U64, Nullability::NonNullable)
        );
        assert_eq!(
            struct_a.fields[0]
                .as_::<PrimitiveVTable>()
                .as_slice::<u64>(),
            [4u64, 5, 6, 7, 8]
        );

        let empty = struct_a.remove_column("non_existent");
        assert!(
            empty.is_none(),
            "Expected None when removing non-existent column"
        );
        assert_eq!(struct_a.names(), &[FieldName::from("ys")].into());
    }

    #[test]
    fn test_duplicate_field_names() {
        // Test that StructArray allows duplicate field names and returns the first match
        let field1 = buffer![1i32, 2, 3].into_array();
        let field2 = buffer![10i32, 20, 30].into_array();
        let field3 = buffer![100i32, 200, 300].into_array();

        // Create struct with duplicate field names - "value" appears twice
        let struct_array = StructArray::try_new(
            FieldNames::from(["value", "other", "value"]),
            vec![field1, field2, field3],
            3,
            Validity::NonNullable,
        )
        .unwrap();

        // field_by_name should return the first field with the matching name
        let first_value_field = struct_array.field_by_name("value").unwrap();
        assert_eq!(
            first_value_field.as_::<PrimitiveVTable>().as_slice::<i32>(),
            [1i32, 2, 3] // This is field1, not field3
        );

        // Verify field_by_name_opt also returns the first match
        let opt_field = struct_array.field_by_name_opt("value").unwrap();
        assert_eq!(
            opt_field.as_::<PrimitiveVTable>().as_slice::<i32>(),
            [1i32, 2, 3] // First "value" field
        );

        // Verify the third field (second "value") can be accessed by index
        let third_field = &struct_array.fields()[2];
        assert_eq!(
            third_field.as_::<PrimitiveVTable>().as_slice::<i32>(),
            [100i32, 200, 300]
        );
    }
}

1	// SPDX-License-Identifier: Apache-2.0
2	// SPDX-FileCopyrightText: Copyright the Vortex contributors
3
4	use std::fmt::Debug;
5	use std::iter::once;
6
7	use itertools::Itertools;
8	use vortex_dtype::{DType, FieldName, FieldNames, StructFields};
9	use vortex_error::{VortexExpect, VortexResult, vortex_bail, vortex_err};
10	use vortex_scalar::Scalar;
11
12	use crate::stats::{ArrayStats, StatsSetRef};
13	use crate::validity::Validity;
14	use crate::vtable::{
15	ArrayVTable, CanonicalVTable, NotSupported, OperationsVTable, VTable, ValidityHelper,
16	ValidityVTableFromValidityHelper,
17	};
18	use crate::{Array, ArrayRef, Canonical, EncodingId, EncodingRef, IntoArray, vtable};
19
20	mod compute;
21	mod serde;
22
23	vtable!(Struct);
24
25	impl VTable for StructVTable {
26	type Array = StructArray;
27	type Encoding = StructEncoding;
28
29	type ArrayVTable = Self;
30	type CanonicalVTable = Self;
31	type OperationsVTable = Self;
32	type ValidityVTable = ValidityVTableFromValidityHelper;
33	type VisitorVTable = Self;
34	type ComputeVTable = NotSupported;
35	type EncodeVTable = NotSupported;
36	type SerdeVTable = Self;
37
38	fn id(_encoding: &Self::Encoding) -> EncodingId {	537,838✔
39	EncodingId::new_ref("vortex.struct")	537,838✔
40	}	537,838✔
41
42	fn encoding(_array: &Self::Array) -> EncodingRef {	12,448✔
43	EncodingRef::new_ref(StructEncoding.as_ref())	12,448✔
44	}	12,448✔
45	}
46
47	/// A struct array that stores multiple named fields as columns, similar to a database row.
48	///
49	/// This mirrors the Apache Arrow Struct array encoding and provides a columnar representation
50	/// of structured data where each row contains multiple named fields of potentially different types.
51	///
52	/// ## Data Layout
53	///
54	/// The struct array uses a columnar layout where:
55	/// - Each field is stored as a separate child array
56	/// - All fields must have the same length (number of rows)
57	/// - Field names and types are defined in the struct's dtype
58	/// - An optional validity mask indicates which entire rows are null
59	///
60	/// ## Row-level nulls
61	///
62	/// The StructArray contains its own top-level nulls, which are superimposed on top of the
63	/// field-level validity values. This can be the case even if the fields themselves are non-nullable,
64	/// accessing a particular row can yield nulls even if all children are valid at that position.
65	///
66	/// ```
67	/// use vortex_array::arrays::{StructArray, BoolArray};
68	/// use vortex_array::validity::Validity;
69	/// use vortex_array::IntoArray;
70	/// use vortex_dtype::FieldNames;
71	/// use vortex_buffer::buffer;
72	///
73	/// // Create struct with all non-null fields but struct-level nulls
74	/// let struct_array = StructArray::try_new(
75	/// FieldNames::from(["a", "b", "c"]),
76	/// vec![
77	/// buffer![1i32, 2i32].into_array(), // non-null field a
78	/// buffer![10i32, 20i32].into_array(), // non-null field b
79	/// buffer![100i32, 200i32].into_array(), // non-null field c
80	/// ],
81	/// 2,
82	/// Validity::Array(BoolArray::from_iter([true, false]).into_array()), // row 1 is null
83	/// ).unwrap();
84	///
85	/// // Row 0 is valid - returns a struct scalar with field values
86	/// let row0 = struct_array.scalar_at(0).unwrap();
87	/// assert!(!row0.is_null());
88	///
89	/// // Row 1 is null at struct level - returns null even though fields have values
90	/// let row1 = struct_array.scalar_at(1).unwrap();
91	/// assert!(row1.is_null());
92	/// ```
93	///
94	/// ## Name uniqueness
95	///
96	/// It is valid for a StructArray to have multiple child columns that have the same name. In this
97	/// case, any accessors that use column names will find the first column in sequence with the name.
98	///
99	/// ```
100	/// use vortex_array::arrays::StructArray;
101	/// use vortex_array::validity::Validity;
102	/// use vortex_array::IntoArray;
103	/// use vortex_dtype::FieldNames;
104	/// use vortex_buffer::buffer;
105	///
106	/// // Create struct with duplicate "data" field names
107	/// let struct_array = StructArray::try_new(
108	/// FieldNames::from(["data", "data"]),
109	/// vec![
110	/// buffer![1i32, 2i32].into_array(), // first "data"
111	/// buffer![3i32, 4i32].into_array(), // second "data"
112	/// ],
113	/// 2,
114	/// Validity::NonNullable,
115	/// ).unwrap();
116	///
117	/// // field_by_name returns the FIRST "data" field
118	/// let first_data = struct_array.field_by_name("data").unwrap();
119	/// assert_eq!(first_data.scalar_at(0).unwrap(), 1i32.into());
120	/// ```
121	///
122	/// ## Field Operations
123	///
124	/// Struct arrays support efficient column operations:
125	/// - Projection: Select/reorder fields without copying data
126	/// - Field access: Get columns by name or index
127	/// - Column addition: Add new fields to create extended structs
128	/// - Column removal: Remove fields to create narrower structs
129	///
130	/// ## Validity Semantics
131	///
132	/// - Row-level nulls are tracked in the struct's validity child
133	/// - Individual field nulls are tracked in each field's own validity
134	/// - A null struct row means all fields in that row are conceptually null
135	/// - Field-level nulls can exist independently of struct-level nulls
136	///
137	/// # Examples
138	///
139	/// ```
140	/// use vortex_array::arrays::{StructArray, PrimitiveArray};
141	/// use vortex_array::validity::Validity;
142	/// use vortex_array::IntoArray;
143	/// use vortex_dtype::FieldNames;
144	/// use vortex_buffer::buffer;
145	///
146	/// // Create arrays for each field
147	/// let ids = PrimitiveArray::new(buffer![1i32, 2, 3], Validity::NonNullable);
148	/// let names = PrimitiveArray::new(buffer![100u64, 200, 300], Validity::NonNullable);
149	///
150	/// // Create struct array with named fields
151	/// let struct_array = StructArray::try_new(
152	/// FieldNames::from(["id", "score"]),
153	/// vec![ids.into_array(), names.into_array()],
154	/// 3,
155	/// Validity::NonNullable,
156	/// ).unwrap();
157	///
158	/// assert_eq!(struct_array.len(), 3);
159	/// assert_eq!(struct_array.names().len(), 2);
160	///
161	/// // Access field by name
162	/// let id_field = struct_array.field_by_name("id").unwrap();
163	/// assert_eq!(id_field.len(), 3);
164	/// ```
165	#[derive(Clone, Debug)]
166	pub struct StructArray {
167	len: usize,
168	dtype: DType,
169	fields: Vec<ArrayRef>,
170	validity: Validity,
171	stats_set: ArrayStats,
172	}
173
174	#[derive(Clone, Debug)]
175	pub struct StructEncoding;
176
177	impl StructArray {
178	pub fn fields(&self) -> &[ArrayRef] {	79,998✔
179	&self.fields	79,998✔
180	}	79,998✔
181
182	pub fn field_by_name(&self, name: impl AsRef<str>) -> VortexResult<&ArrayRef> {	20,472✔
183	let name = name.as_ref();	20,472✔
184	self.field_by_name_opt(name).ok_or_else(\|\| {	20,472✔
185	vortex_err!(	1✔
186	"Field {name} not found in struct array with names {:?}",	1✔
187	self.names()	1✔
188	)
189	})	1✔
190	}	20,472✔
191
192	pub fn field_by_name_opt(&self, name: impl AsRef<str>) -> Option<&ArrayRef> {	28,438✔
193	let name = name.as_ref();	28,438✔
194	self.names()	28,438✔
195	.iter()	28,438✔
196	.position(\|field_name\| field_name.as_ref() == name)	66,233✔
197	.map(\|idx\| &self.fields[idx])	28,438✔
198	}	28,438✔
199
200	pub fn names(&self) -> &FieldNames {	55,743✔
201	self.struct_fields().names()	55,743✔
202	}	55,743✔
203
204	pub fn struct_fields(&self) -> &StructFields {	57,681✔
205	let Some(struct_dtype) = &self.dtype.as_struct() else {	57,681✔
206	unreachable!(	×
207	"struct arrays must have be a DType::Struct, this is likely an internal bug."
208	)
209	};
210	struct_dtype	57,681✔
211	}	57,681✔
212
213	/// Create a new `StructArray` with the given length, but without any fields.
214	pub fn new_with_len(len: usize) -> Self {	40✔
215	Self::try_new(	40✔
216	FieldNames::default(),	40✔
217	Vec::new(),	40✔
218	len,	40✔
219	Validity::NonNullable,	40✔
220	)
221	.vortex_expect("StructArray::new_with_len should not fail")	40✔
222	}	40✔
223
224	pub fn try_new(	28,669✔
225	names: FieldNames,	28,669✔
226	fields: Vec<ArrayRef>,	28,669✔
227	length: usize,	28,669✔
228	validity: Validity,	28,669✔
229	) -> VortexResult<Self> {	28,669✔
230	let nullability = validity.nullability();	28,669✔
231
232	if names.len() != fields.len() {	28,669✔
233	vortex_bail!("Got {} names and {} fields", names.len(), fields.len());	×
234	}	28,669✔
235
236	for field in fields.iter() {	81,227✔
237	if field.len() != length {	81,227✔
238	vortex_bail!(	×
239	"Expected all struct fields to have length {length}, found {}",	×
240	fields.iter().map(\|f\| f.len()).format(","),	×
241	);
242	}	81,227✔
243	}
244
245	let field_dtypes: Vec<_> = fields.iter().map(\|d\| d.dtype()).cloned().collect();	81,227✔
246	let dtype = DType::Struct(StructFields::new(names, field_dtypes), nullability);	28,669✔
247
248	if length != validity.maybe_len().unwrap_or(length) {	28,669✔
249	vortex_bail!(	×
250	"array length {} and validity length must match {}",	×
251	length,
252	validity	×
253	.maybe_len()	×
254	.vortex_expect("can only fail if maybe is some")	×
255	)
256	}	28,669✔
257
258	Ok(Self {	28,669✔
259	len: length,	28,669✔
260	dtype,	28,669✔
261	fields,	28,669✔
262	validity,	28,669✔
263	stats_set: Default::default(),	28,669✔
264	})	28,669✔
265	}	28,669✔
266
267	pub fn try_new_with_dtype(	1,471✔
268	fields: Vec<ArrayRef>,	1,471✔
269	dtype: StructFields,	1,471✔
270	length: usize,	1,471✔
271	validity: Validity,	1,471✔
272	) -> VortexResult<Self> {	1,471✔
273	for (field, struct_dt) in fields.iter().zip(dtype.fields()) {	4,869✔
274	if field.len() != length {	4,869✔
275	vortex_bail!(	×
276	"Expected all struct fields to have length {length}, found {}",	×
277	field.len()	×
278	);
279	}	4,869✔
280
281	if &struct_dt != field.dtype() {	4,869✔
282	vortex_bail!(	×
283	"Expected all struct fields to have dtype {}, found {}",	×
284	struct_dt,
285	field.dtype()	×
286	);
287	}	4,869✔
288	}
289
290	Ok(Self {	1,471✔
291	len: length,	1,471✔
292	dtype: DType::Struct(dtype, validity.nullability()),	1,471✔
293	fields,	1,471✔
294	validity,	1,471✔
295	stats_set: Default::default(),	1,471✔
296	})	1,471✔
297	}	1,471✔
298
299	pub fn from_fields<N: AsRef<str>>(items: &[(N, ArrayRef)]) -> VortexResult<Self> {	329✔
300	Self::try_from_iter(items.iter().map(\|(a, b)\| (a, b.to_array())))	797✔
301	}	329✔
302
303	pub fn try_from_iter_with_validity<	342✔
304	N: AsRef<str>,	342✔
305	A: IntoArray,	342✔
306	T: IntoIterator<Item = (N, A)>,	342✔
307	>(	342✔
308	iter: T,	342✔
309	validity: Validity,	342✔
310	) -> VortexResult<Self> {	342✔
311	let (names, fields): (Vec<FieldName>, Vec<ArrayRef>) = iter	342✔
312	.into_iter()	342✔
313	.map(\|(name, fields)\| (FieldName::from(name.as_ref()), fields.into_array()))	815✔
314	.unzip();	342✔
315	let len = fields	342✔
316	.first()	342✔
317	.map(\|f\| f.len())	342✔
318	.ok_or_else(\|\| vortex_err!("StructArray cannot be constructed from an empty slice of arrays because the length is unspecified"))?;	342✔
319
320	Self::try_new(FieldNames::from_iter(names), fields, len, validity)	342✔
321	}	342✔
322
323	pub fn try_from_iter<N: AsRef<str>, A: IntoArray, T: IntoIterator<Item = (N, A)>>(	339✔
324	iter: T,	339✔
325	) -> VortexResult<Self> {	339✔
326	Self::try_from_iter_with_validity(iter, Validity::NonNullable)	339✔
327	}	339✔
328
329	// TODO(aduffy): Add equivalent function to support field masks for nested column access.
330	/// Return a new StructArray with the given projection applied.
331	///
332	/// Projection does not copy data arrays. Projection is defined by an ordinal array slice
333	/// which specifies the new ordering of columns in the struct. The projection can be used to
334	/// perform column re-ordering, deletion, or duplication at a logical level, without any data
335	/// copying.
336	#[allow(clippy::same_name_method)]
337	pub fn project(&self, projection: &[FieldName]) -> VortexResult<Self> {	79✔
338	let mut children = Vec::with_capacity(projection.len());	79✔
339	let mut names = Vec::with_capacity(projection.len());	79✔
340
341	for f_name in projection.iter() {	80✔
342	let idx = self	80✔
343	.names()	80✔
344	.iter()	80✔
345	.position(\|name\| name == f_name)	121✔
346	.ok_or_else(\|\| vortex_err!("Unknown field {f_name}"))?;	80✔
347
348	names.push(self.names()[idx].clone());	80✔
349	children.push(self.fields()[idx].clone());	80✔
350	}
351
352	StructArray::try_new(	79✔
353	FieldNames::from(names.as_slice()),	79✔
354	children,	79✔
355	self.len(),	79✔
356	self.validity().clone(),	79✔
357	)
358	}	79✔
359
360	/// Removes and returns a column from the struct array by name.
361	/// If the column does not exist, returns `None`.
362	pub fn remove_column(&mut self, name: impl Into<FieldName>) -> Option<ArrayRef> {	2✔
363	let name = name.into();	2✔
364
365	let struct_dtype = self.struct_fields().clone();	2✔
366
367	let position = struct_dtype	2✔
368	.names()	2✔
369	.iter()	2✔
370	.position(\|field_name\| field_name.as_ref() == name.as_ref())?;	2✔
371
372	let field = self.fields.remove(position);	1✔
373
374	if let Ok(new_dtype) = struct_dtype.without_field(position) {	1✔
375	self.dtype = DType::Struct(new_dtype, self.dtype.nullability());	1✔
376	return Some(field);	1✔
NEW 377	}	×
NEW 378	None	×
379	}	2✔
380
381	/// Create a new StructArray by appending a new column onto the existing array.
382	pub fn with_column(&self, name: impl Into<FieldName>, array: ArrayRef) -> VortexResult<Self> {	×
383	let name = name.into();	×
384	let struct_dtype = self.struct_fields().clone();	×
385
386	let names = struct_dtype.names().iter().cloned().chain(once(name));	×
387	let types = struct_dtype.fields().chain(once(array.dtype().clone()));	×
388	let new_fields = StructFields::new(names.collect(), types.collect());	×
389
390	let mut children = self.fields.clone();	×
391	children.push(array);	×
392
393	Self::try_new_with_dtype(children, new_fields, self.len, self.validity.clone())	×
394	}	×
395	}
396
397	impl ValidityHelper for StructArray {
398	fn validity(&self) -> &Validity {	28,516✔
399	&self.validity	28,516✔
400	}	28,516✔
401	}
402
403	impl ArrayVTable<StructVTable> for StructVTable {
404	fn len(array: &StructArray) -> usize {	169,474✔
405	array.len	169,474✔
406	}	169,474✔
407
408	fn dtype(array: &StructArray) -> &DType {	237,164✔
409	&array.dtype	237,164✔
410	}	237,164✔
411
412	fn stats(array: &StructArray) -> StatsSetRef<'_> {	112,741✔
413	array.stats_set.to_ref(array.as_ref())	112,741✔
414	}	112,741✔
415	}
416
417	impl CanonicalVTable<StructVTable> for StructVTable {
418	fn canonicalize(array: &StructArray) -> VortexResult<Canonical> {	40,238✔
419	Ok(Canonical::Struct(array.clone()))	40,238✔
420	}	40,238✔
421	}
422
423	impl OperationsVTable<StructVTable> for StructVTable {
424	fn slice(array: &StructArray, start: usize, stop: usize) -> VortexResult<ArrayRef> {	24✔
425	let fields = array	24✔
426	.fields()	24✔
427	.iter()	24✔
428	.map(\|field\| field.slice(start, stop))	36✔
429	.try_collect()?;	24✔
430	StructArray::try_new_with_dtype(	24✔
431	fields,	24✔
432	array.struct_fields().clone(),	24✔
433	stop - start,	24✔
434	array.validity().slice(start, stop)?,	24✔
435	)
436	.map(\|a\| a.into_array())	24✔
437	}	24✔
438
439	fn scalar_at(array: &StructArray, index: usize) -> VortexResult<Scalar> {	3,078✔
440	if array.is_valid(index)? {	3,078✔
441	Ok(Scalar::struct_(	3,078✔
442	array.dtype().clone(),	3,078✔
443	array	3,078✔
444	.fields()	3,078✔
445	.iter()	3,078✔
446	.map(\|field\| field.scalar_at(index))	6,363✔
447	.try_collect()?,	3,078✔
448	))
449	} else {
450	Ok(Scalar::null(array.dtype().clone()))	×
451	}
452	}	3,078✔
453	}
454
455	#[cfg(test)]
456	mod test {
457	use vortex_buffer::buffer;
458	use vortex_dtype::{DType, FieldName, FieldNames, Nullability, PType};
459
460	use crate::IntoArray;
461	use crate::arrays::primitive::PrimitiveArray;
462	use crate::arrays::struct_::StructArray;
463	use crate::arrays::varbin::VarBinArray;
464	use crate::arrays::{BoolArray, BoolVTable, PrimitiveVTable};
465	use crate::validity::Validity;
466
467	#[test]
468	fn test_project() {	1✔
469	let xs = PrimitiveArray::new(buffer![0i64, 1, 2, 3, 4], Validity::NonNullable);	1✔
470	let ys = VarBinArray::from_vec(	1✔
471	vec!["a", "b", "c", "d", "e"],	1✔
472	DType::Utf8(Nullability::NonNullable),	1✔
473	);
474	let zs = BoolArray::from_iter([true, true, true, false, false]);	1✔
475
476	let struct_a = StructArray::try_new(	1✔
477	FieldNames::from(["xs", "ys", "zs"]),	1✔
478	vec![xs.into_array(), ys.into_array(), zs.into_array()],	1✔
479	5,
480	Validity::NonNullable,	1✔
481	)
482	.unwrap();	1✔
483
484	let struct_b = struct_a	1✔
485	.project(&[FieldName::from("zs"), FieldName::from("xs")])	1✔
486	.unwrap();	1✔
487	assert_eq!(	1✔
488	struct_b.names().as_ref(),	1✔
489	[FieldName::from("zs"), FieldName::from("xs")],	1✔
490	);
491
492	assert_eq!(struct_b.len(), 5);	1✔
493
494	let bools = &struct_b.fields[0];	1✔
495	assert_eq!(	1✔
496	bools	1✔
497	.as_::<BoolVTable>()	1✔
498	.boolean_buffer()	1✔
499	.iter()	1✔
500	.collect::<Vec<_>>(),	1✔
501	vec![true, true, true, false, false]	1✔
502	);
503
504	let prims = &struct_b.fields[1];	1✔
505	assert_eq!(	1✔
506	prims.as_::<PrimitiveVTable>().as_slice::<i64>(),	1✔
507	[0i64, 1, 2, 3, 4]
508	);
509	}	1✔
510
511	#[test]
512	fn test_remove_column() {	1✔
513	let xs = PrimitiveArray::new(buffer![0i64, 1, 2, 3, 4], Validity::NonNullable);	1✔
514	let ys = PrimitiveArray::new(buffer![4u64, 5, 6, 7, 8], Validity::NonNullable);	1✔
515
516	let mut struct_a = StructArray::try_new(	1✔
517	FieldNames::from(["xs", "ys"]),	1✔
518	vec![xs.into_array(), ys.into_array()],	1✔
519	5,
520	Validity::NonNullable,	1✔
521	)
522	.unwrap();	1✔
523
524	let removed = struct_a.remove_column("xs").unwrap();	1✔
525	assert_eq!(	1✔
526	removed.dtype(),	1✔
527	&DType::Primitive(PType::I64, Nullability::NonNullable)
528	);
529	assert_eq!(	1✔
530	removed.as_::<PrimitiveVTable>().as_slice::<i64>(),	1✔
531	[0i64, 1, 2, 3, 4]
532	);
533
534	assert_eq!(struct_a.names(), &[FieldName::from("ys")].into());	1✔
535	assert_eq!(struct_a.fields.len(), 1);	1✔
536	assert_eq!(struct_a.len(), 5);	1✔
537	assert_eq!(	1✔
538	struct_a.fields[0].dtype(),	1✔
539	&DType::Primitive(PType::U64, Nullability::NonNullable)
540	);
541	assert_eq!(	1✔
542	struct_a.fields[0]	1✔
543	.as_::<PrimitiveVTable>()	1✔
544	.as_slice::<u64>(),	1✔
545	[4u64, 5, 6, 7, 8]
546	);
547
548	let empty = struct_a.remove_column("non_existent");	1✔
549	assert!(	1✔
550	empty.is_none(),	1✔
551	"Expected None when removing non-existent column"
552	);
553	assert_eq!(struct_a.names(), &[FieldName::from("ys")].into());	1✔
554	}	1✔
555
556	#[test]
557	fn test_duplicate_field_names() {	1✔
558	// Test that StructArray allows duplicate field names and returns the first match
559	let field1 = buffer![1i32, 2, 3].into_array();	1✔
560	let field2 = buffer![10i32, 20, 30].into_array();	1✔
561	let field3 = buffer![100i32, 200, 300].into_array();	1✔
562
563	// Create struct with duplicate field names - "value" appears twice
564	let struct_array = StructArray::try_new(	1✔
565	FieldNames::from(["value", "other", "value"]),	1✔
566	vec![field1, field2, field3],	1✔
567	3,
568	Validity::NonNullable,	1✔
569	)
570	.unwrap();	1✔
571
572	// field_by_name should return the first field with the matching name
573	let first_value_field = struct_array.field_by_name("value").unwrap();	1✔
574	assert_eq!(	1✔
575	first_value_field.as_::<PrimitiveVTable>().as_slice::<i32>(),	1✔
576	[1i32, 2, 3] // This is field1, not field3
577	);
578
579	// Verify field_by_name_opt also returns the first match
580	let opt_field = struct_array.field_by_name_opt("value").unwrap();	1✔
581	assert_eq!(	1✔
582	opt_field.as_::<PrimitiveVTable>().as_slice::<i32>(),	1✔
583	[1i32, 2, 3] // First "value" field
584	);
585
586	// Verify the third field (second "value") can be accessed by index
587	let third_field = &struct_array.fields()[2];	1✔
588	assert_eq!(	1✔
589	third_field.as_::<PrimitiveVTable>().as_slice::<i32>(),	1✔
590	[100i32, 200, 300]
591	);
592	}	1✔
593	}

vortex-data / vortex / 16906445831

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous