redwoodjs__local-ci/.docs/rust-execution-parity-plan.md

Rust Execution Parity Plan

This plan covers the remaining work required before Local CI can remove the TypeScript CLI implementation and run entirely through the native Rust binary.

Current status

• Rust has broad module coverage for parsing, state, DTU APIs, Docker config, runner images, macOS VM helpers, packaging, and validation scaffolding.
• Rust local-ci run can execute Docker-backed Linux jobs through the Rust DTU behind the opt-in Rust path.
• The TypeScript CLI remains the default execution path until the remaining parity phases pass.
• RUST-084 is blocked until this plan is complete.

Acceptance criteria

Rust is at full parity when all of the following are true:

• target/debug/local-ci run --workflow <workflow> can execute real jobs end-to-end.
• Rust can run Docker-backed Linux jobs with caches, artifacts, services, outputs, and matrix expansion.
• Rust can pause, retry, and abort failed jobs with the same behavior as TypeScript.
• Rust can execute or correctly skip/degrade macOS VM jobs with matching messages.
• pnpm rust:smoke:parity expects successful execution instead of the current discovery-only gap.
• pnpm local-ci-dev run --all -q -p --json passes with the Rust path enabled.
• The TypeScript fallback can be removed without losing any documented capability.

Phase 0 — Guardrails

• RXP-000: Keep TypeScript fallback until parity is proven

  • Keep LOCAL_CI_FORCE_TYPESCRIPT=1 documented.
  • Keep npm launcher fallback enabled.
  • Do not remove TypeScript execution code before RXP-080 passes.
  • Test: pnpm golden:cli

• RXP-001: Add an explicit Rust execution feature flag

  • Add LOCAL_CI_FORCE_RUST=1 or equivalent for local parity testing.
  • Make default behavior conservative while Rust execution is incomplete.
  • Ensure unsupported Rust execution falls back or fails with a clear message.
  • Test: launcher unit tests + pnpm golden:cli

• RXP-002: Convert current discovery-gap smoke script into a gate

  • Keep current discovery-only assertions while work is incomplete.
  • Add a mode that expects real execution success.
  • Document how/when to flip the default expectation.
  • Test: pnpm rust:smoke:parity

Phase 1 — Rust run orchestrator

• RXP-010: Introduce Rust run orchestrator

  • Replace run_command.rs discovery-only flow with an orchestration layer.
  • Keep discovery as a pure planning phase.
  • Return typed run/job plans before execution.
  • Test: Rust unit tests for workflow-to-run-plan conversion.

• RXP-011: Job scheduler parity

  • Port job dependency ordering.
  • Port needs result propagation.
  • Port if evaluation for jobs.
  • Preserve always(), success(), failure(), and skipped-job behavior.
  • Test: Rust scheduler tests mirroring TypeScript scheduler fixtures; revalidated for schedule-keyed matrix leg results, aggregate needs.<job>.result, and cyclic dependency errors.

• RXP-012: Matrix orchestration parity

  • Wire expanded matrix jobs into the scheduler.
  • Preserve runner names and strategy metadata.
  • Preserve --no-matrix collapse behavior.
  • Test: Rust matrix smoke plus .github/workflows/smoke-matrix.yml; revalidated with the shared matrix-needs-aggregate fixture and schedule-keyed needs propagation.

• RXP-013: NDJSON event stream parity

  • Emit run.start, job.start, step.start, step.finish, job.finish, run.finish.
  • Preserve schema version and event shapes.
  • Preserve quiet/agent mode behavior.
  • Test: Rust golden NDJSON tests against TypeScript output.

Phase 2 — Real Docker runtime

• RXP-020: Add concrete Docker client implementation

  • Implement container create/start/wait/remove.
  • Implement network create/remove.
  • Implement log streaming.
  • Reuse existing Rust Docker config builders.
  • Test: Docker integration test behind an opt-in env flag.

• RXP-021: Runner container execution

  • Start official runner containers from Rust.
  • Register runner with Rust DTU.
  • Seed jobs and wait for runner completion.
  • Collect exit status and logs.
  • Test: Minimal one-step workflow executes through Rust.

• RXP-022: Workspace and git sync parity

  • Port working-tree snapshot behavior.
  • Include uncommitted tracked and untracked changes.
  • Preserve ignored-file exclusions.
  • Preserve dirty SHA semantics.
  • Test: Rust dirty-worktree execution tests.

• RXP-023: Cache/toolcache mount parity

  • Mount npm/pnpm/yarn/bun caches.
  • Mount Playwright and Cypress caches safely.
  • Mount hosted toolcache.
  • Preserve permission fixes and hints.
  • Test: cache smoke workflows through Rust.

• RXP-024: Docker socket and Docker-in-Docker parity

  • Mount Docker socket correctly.
  • Preserve host gateway handling.
  • Support docker buildx smoke workflow.
  • Test: .github/workflows/smoke-docker-buildx.yml through Rust.

Phase 3 — DTU execution integration

• RXP-030: Wire Rust DTU into top-level run

  • Start DTU per run.
  • Expose CLI URL and container URL.
  • Seed jobs from orchestrator.
  • Stop DTU after run or keep alive during pause.
  • Test: one-job Rust execution with official runner.

• RXP-031: Cache API parity under real runner

  • Validate actions/cache restore miss/hit/save flows.
  • Preserve virtual cache pattern behavior.
  • Preserve local bind-mount fast path.
  • Test: cache smoke workflows through Rust.

• RXP-032: Artifact API parity under real runner

  • Validate upload/download artifact actions.
  • Preserve REST and Twirp/block-blob behavior.
  • Preserve artifact paths and metadata.
  • Test: .github/workflows/smoke-artifacts.yml through Rust.

• RXP-033: Action download parity

  • Serve local action tarballs.
  • Serve pinned remote action tarballs from cache/download path.
  • Preserve private reusable workflow auth behavior.
  • Test: checkout/setup/cache/action smoke workflows through Rust.

Phase 4 — Steps, env, outputs, and services

• RXP-040: Step env parity

  • Evaluate workflow/job/step env.
  • Inject GitHub, runner, strategy, matrix, secrets, vars, and inputs contexts.
  • Preserve shell env precedence.
  • Test: .github/workflows/smoke-env-*.yml through Rust.

• RXP-041: Step condition parity

  • Port step-level if behavior into Rust execution.
  • Preserve skipped-step timeline status.
  • Preserve always() handling.
  • Test: .github/workflows/smoke-step-if.yml through Rust.

• RXP-042: Step outputs and job outputs parity

  • Capture $GITHUB_OUTPUT writes.
  • Resolve step outputs into job outputs.
  • Propagate outputs through needs.
  • Test: .github/workflows/smoke-outputs.yml through Rust.

• RXP-043: Service container parity

  • Start services before runner job.
  • Attach services to the correct network.
  • Preserve health checks, ports, env, and cleanup.
  • Test: service smoke workflows through Rust.

• RXP-044: Container options parity

  • Apply workflow container options.
  • Preserve env, labels, volumes, and network settings.
  • Test: .github/workflows/smoke-container-options.yml through Rust.

Phase 5 — Pause, retry, abort

• RXP-050: Pause-on-failure parity

  • Keep runner container and DTU alive after a failed step.
  • Write signal files compatible with current retry/abort commands.
  • Emit run.paused and exit code 77 from launcher path.
  • Test: .github/workflows/smoke-pause-pipe.yml through Rust.

• RXP-051: Retry failed step parity

  • Resume paused container.
  • Retry only failed step by default.
  • Preserve attempts and logs.
  • Test: retry proof smoke workflow through Rust.

• RXP-052: Retry from step/from start parity

  • Implement --from-step rewind behavior.
  • Implement --from-start behavior.
  • Preserve skipped earlier steps.
  • Test: retry unit/integration tests through Rust.

• RXP-053: Abort parity

  • Tear down paused runner and services.
  • Remove signal dirs.
  • Preserve user-facing messages.
  • Test: abort integration test through Rust.

Phase 6 — macOS VM integration

• RXP-060: Wire macOS VM jobs into scheduler

  • Route runs-on: macos-* jobs to Rust macOS VM execution.
  • Preserve Docker/Linux scheduling for other jobs.
  • Test: macOS routing unit tests.

• RXP-061: macOS VM run parity

  • Pull/clone/start Tart VM.
  • Wait for IP and SSH.
  • Sync workspace and runner binary.
  • Run job and sync logs back.
  • Test: opt-in macOS integration test on Apple Silicon (target/debug/local-ci run --workflow .github/workflows/macos.yml -q --json on Apple Silicon Tart host).

• RXP-062: macOS skip/degraded parity

  • Preserve unsupported-host skip reasons.
  • Preserve install hints for Tart and sshpass.
  • Preserve behavior on Linux/Intel hosts.
  • Test: host capability unit tests and smoke skip checks (PATH=/usr/bin:/bin target/debug/local-ci run --workflow .github/workflows/macos.yml -q --json).

Phase 7 — Result writing and reporting

• RXP-070: Run result JSON parity

  • Match TypeScript run-result.json shape.
  • Include debug logs, step logs, outputs, durations, and status.
  • Preserve branch/repo path layout.
  • Test: state golden/unit coverage plus live Rust result smoke with step logPath and non-zero durationMs.

• RXP-071: Human reporter parity

  • Match summary/failure output.
  • Preserve hints for missing tools and toolcache permissions.
  • Preserve quiet and JSON modes.
  • Test: reporter unit tests plus live failing Rust workflow smoke.

• RXP-072: Cleanup parity

  • Remove containers/networks/services on success/failure.
  • Preserve paused resources only when requested.
  • Preserve log pruning behavior.
  • Test: Docker cleanup unit/integration checks (failed_start_cleans_up_services_and_network, live failure smoke, docker ps -a cleanup check).

Phase 8 — Full parity validation

• RXP-080: Flip Rust smoke parity to execution success

  • Update scripts/rust-smoke-parity.mjs to expect exit code 0.
  • Remove the expected discovery-gap assertion.
  • Keep the smoke workflow list expanding over time.
  • Test: pnpm rust:smoke:parity

• RXP-081: Run core smoke workflows through Rust

  • smoke-binary.yml
  • smoke-expressions.yml
  • smoke-matrix.yml
  • smoke-artifacts.yml
  • smoke-docker-buildx.yml
  • smoke-pause-pipe.yml
  • Test: pnpm rust:smoke:parity covers artifacts, docker-buildx, and pause-pipe in CI.

• RXP-081b: Port job-wave concurrency

  • Rust preserves dependency-wave ordering and runs jobs inside each wave concurrently.
  • The native CLI honors --jobs as the per-wave concurrency limit.
  • macOS VM execution is capped independently with LOCAL_CI_MACOS_VM_CONCURRENCY (default: 2).
  • Test: Rust CLI parsing tests plus smoke parity/full dev validation exercise the native limiter path.

• RXP-082: Run all in-repo workflows through Rust

  • Enable Rust path for the full dev validation.
  • Fix any remaining parity failures.
  • Record unsupported intentional skips.
  • Test: LOCAL_CI_FORCE_RUST=1 pnpm local-ci-dev run --all -q -p --json

• RXP-083: Release-mode benchmarks

  • Benchmark release Rust binary startup.
  • Benchmark parse/discovery.
  • Benchmark orchestration overhead.
  • Update .docs/rust-performance.md.
  • Test: pnpm rust:perf

Phase 9 — Default switch and TypeScript removal

See also rust-crate-split-and-fixtures RFC for the long-term fixture contract layer.

• RXP-090: Make Rust execution default

  • Route npm launcher to native Rust binary by default.
  • Keep TypeScript fallback for one release.
  • Document fallback/rollback.
  • Test: full pnpm local-ci-dev run --all -q -p --json with default settings.

• RXP-091: Remove TypeScript execution fallback

  • Remove TypeScript CLI execution modules.
  • Remove fallback launcher path.
  • Keep only necessary JS packaging shim if npm still needs it.
  • Test: full pnpm local-ci-dev run --all -q -p --json.

• RXP-092: Remove unused dependencies

  • Remove Docker/runner/workflow TS dependencies no longer used.
  • Keep only package/build dependencies required for npm distribution.
  • Update lockfile.
  • Test: pnpm install --lockfile-only && pnpm check.

• RXP-093: Final docs update

  • Update README to describe native Local CI.
  • Update release docs.
  • Update migration/rollback notes.
  • Test: docs links and full dev validation.

Required validation cadence

For each checked-off implementation task:

1. Run targeted unit/integration tests for that task.

2. Run Rust checks:

   cargo fmt --all
   cargo check --workspace --all-targets
   cargo clippy --workspace --all-targets -- -D warnings
   cargo test --workspace --all-targets
   

3. Run Node checks when package/launcher/workflow behavior changes:

   pnpm check
   pnpm golden:cli
   

4. Before reporting the overall work complete, run:

   pnpm local-ci-dev run --all -q -p --json