vortex-data / vortex / 16629653619

88%

Build Type

push

github

Committed by web-flow

Commit Message

[chore] cross-operation consistency tests (#4055)

⏺ Summary of Consistency Test Improvements

### Overview

We've made comprehensive improvements to Vortex array compute
conformance tests, fixing critical bugs and adding extensive test
coverage across 20+ encoding types.

  Key Changes

  1. Fixed ALP Array Double Mask Bug
- Issue: mask(mask(array, mask1), mask2) ≠ mask(array, mask1 | mask2)
for ALP arrays with patches
- Root cause: Patches were using filter (adjusting indices) instead of
mask (preserving indices)
- Solution: Implemented Patches::mask() method that preserves indices
while masking values
  2. Added Consistency Tests for All Major Encodings
    - Added test_array_consistency() test suites to 20+ encodings:
- Core arrays: Bool, Primitive, Constant, Null, Struct, List, Extension
      - Variable-length: VarBin, VarBinView
      - Compressed: ALP, ALP-RD, BitPacked, FoR, FSST, Dict, RunEnd
- Specialized: Sparse, Chunked, Sequence, DecimalByteParts,
DateTimeParts
    - Each encoding now has 5-15 test cases covering different scenarios
  3. Implemented Missing Compute Kernels
- DecimalArray: Added CastKernel for nullable/non-nullable conversions
- ExtensionArray: Added MaskKernel to avoid Arrow conversion failures
    - DateTimePartsArray: Added MaskKernel
    - DecimalByteParts: Added TakeKernel and MaskKernel
  4. Fixed Consistency Test Framework
    - Made test_array_consistency() public for use across crates
    - Fixed mask combination logic: changed from AND to OR operation
- Fixed sparse array tests to use nullable arrays when using null fill
values
    - Removed string test from RunEnd (doesn't support strings)
  5. Architectural Improvements
- Moved patch masking logic from ALP-specific code to reusable
Patches::mask()
    - Added comprehensive tests for Patches struct operations (17 tests)
- Improved cast implementations to use validity.cast_nullability() for
cleaner code

  Impact

  - All consistency tests now pass across the en... (continued)

Run Details

490 of 547 new or added lines in 23 files covered. (89.58%)

8 existing lines in 1 file now uncovered.

45902 of 55282 relevant lines covered (83.03%)

352110.25 hits per line