vortex-data / vortex / 16914825851

88%

push

github

chore: improve test coverage in vortex-scan (#4206)

## Overview

This PR significantly improves the `vortex-scan` crate by adding
exhaustive concurrency testing with Loom, comprehensive unit test
coverage, and complete documentation. The changes ensure the
work-stealing scanner is provably correct under all possible thread
interleavings while maintaining excellent performance.

## Major Changes

### 1. Loom Concurrency Testing Infrastructure

Added comprehensive Loom tests that exhaustively verify all concurrent
code paths:

- **13 specialized Loom tests** covering work-stealing, filter
expressions, and multi-scan operations
- **Tests run by default** - no feature flags required
- **Fast execution** - complete in ~6 seconds in release mode
- **Dedicated CI job** for Loom tests running on optimized Ubuntu 24
runners
- **`disable_loom` cfg flag** for environments incompatible with Loom
(e.g., address sanitizer)

#### Key Concurrent Behaviors Tested:
- Memory ordering between `num_factories_constructed` and task pushing
- TOCTOU (time-of-check-time-of-use) race prevention
- Work-stealing retry logic with multiple concurrent stealers
- Factory error recovery with proper counter increments
- Complex worker termination conditions
- Concurrent filter ordering updates with multiple readers/writers

### 2. Expanded Unit Test Coverage

Added comprehensive unit tests for previously untested components:

#### Arrow Integration (`arrow.rs`)
- RecordBatch conversion from Vortex arrays
- Error propagation in Arrow conversions
- Iterator adapter with mixed success/error scenarios
- Null value handling and schema validation

#### Filter Module (`filter.rs`)
- Concurrent selectivity reporting
- Conjunct reordering based on selectivity statistics
- Thread-safe histogram updates with RwLock patterns
- Edge cases with single and multiple conjuncts

#### Work Queue (`work_queue.rs`)
- Work stealing with empty factories
- Clone semantics for iterators
- Factory constructio... (continued)

1038 of 1058 new or added lines in 9 files covered. (98.11%)

56223 of 64403 relevant lines covered (87.3%)

627471.13 hits per line