fogfish / iq / 20696400416

31%

Build Type

push

github

Committed by web-flow

Commit Message

feat: Implement skip-if-exists flag for incremental processing (ADR-0… (#86)

* feat: Implement skip-if-exists flag for incremental processing (ADR-0006 Phase 3)

This commit implements the --skip-if-exists CLI flag to enable incremental
processing and recovery from failures. The flag checks if anchor output
exists before processing each document, allowing expensive LLM operations
to be skipped when results already exist.

Implementation Details:

1. CLI Flag Integration (cmd/opts.go):
   - Added skipIfExists field to optsAgent struct
   - Registered --skip-if-exists flag with description
   - Integrated skip logic into agent build pipeline
   - Validates that --output-dir is specified when using skip-if-exists

2. Anchor Key Computation (internal/blueprint/compiler/anchor.go):
   - Created AnchorKeyComputer to calculate expected output keys
   - Supports all step types: AgentStep, RouterStep, ForeachStep, RunStep
   - Handles emit attribute to compute prefixed output paths
   - For foreach steps, anchor is the array file (not individual elements)
   - Defaults to input key when no emit is specified

3. Skip Checking Logic (internal/blueprint/compiler/skip.go):
   - Created SkipChecker using Storage interface to check file existence
   - Uses AnchorKeyComputer to determine expected output location
   - Reports skipped documents via progress reporter
   - Returns early if anchor file already exists

4. Pipeline Integration (internal/service/worker/worker.go):
   - Added workflow field to Builder to store compiled workflow
   - Created SkipIfExists() builder method
   - Integrates storage, anchor computer, and progress reporter
   - Adds skip processor to pipeline before agent execution

5. Processor Implementation (internal/iosystem/processor/skipifexists.go):
   - Created SkipIfExists processor implementing Processor interface
   - Filters documents based on anchor existence check
   - Passes through EOF markers and empty document sets
   - Implements... (continued)

Run Details

15 of 216 new or added lines in 7 files covered. (6.94%)

1 existing line in 1 file now uncovered.

1578 of 4016 relevant lines covered (39.29%)

0.43 hits per line