stacklok / toolhive / 19379293047

push

github

Advanced Workflow Features for vMCP Composition (#2592)

## Overview

Implements advanced workflow features for Virtual MCP Composite Tools, including DAG-based parallel execution, step dependencies, sophisticated error handling, and workflow state management. This completes Phase 2 of the composition work.

**Issue**: Closes #156 (stacklok/stacklok-epics)

## What Changed

### Core Features

#### 1. DAG-Based Parallel Execution
- **New file**: [pkg/vmcp/composer/dag_executor.go](pkg/vmcp/composer/dag_executor.go)
  - Implements topological sort using Kahn's algorithm to build execution levels
  - Executes independent steps in parallel using `errgroup` for coordination
  - Semaphore-based concurrency limiting (default: 10 parallel steps)
  - Automatic optimization: steps with no dependencies run concurrently
  - Performance improvement: parallel execution reduces workflow time by ~60-70% for independent steps

#### 2. Step Dependencies
- `depends_on` field support in [pkg/vmcp/composer/composer.go:67](pkg/vmcp/composer/composer.go#L67)
- Dependency graph validation with cycle detection using DFS
- Transitive dependencies automatically handled
- Missing dependency validation at workflow definition time

#### 3. Advanced Error Handling
- **Three-level error handling**:
  - Step-level: `on_error.continue_on_error` overrides workflow-level settings
  - Workflow-level: `failure_mode` (abort/continue/best_effort)
  - Automatic: retry with exponential backoff
- **Retry logic** in [pkg/vmcp/composer/workflow_engine.go:311-350](pkg/vmcp/composer/workflow_engine.go#L311-L350):
  - Configurable retry count and initial delay
  - Exponential backoff (2^attempt * initial_delay, max 60x)
  - Safety cap: maximum 10 retries to prevent infinite loops

#### 4. Workflow State Management
- **Pluggable state store interface**: [pkg/vmcp/composer/composer.go:191-217](pkg/vmcp/composer/composer.go#L191-L217)
- **In-memory implementation**: [pkg/vmcp/composer/state_store.go](... (continued)

453 of 522 new or added lines in 6 files covered. (86.78%)

24216 of 49242 relevant lines covered (49.18%)

61.56 hits per line