Update the agent-facing skill surfaces to match the recent pause-on-failure
fixes and `--json` event stream:
- skills/agent-ci/SKILL.md (published) and packages/cli/SKILL.md: document
`--json` / `AGENT_CI_JSON=1`, list the NDJSON event types, and note that
pipes/redirects are safe — pause-on-failure exits 77 in non-TTY mode.
- .claude/commands/agent-ci-dev.md and .pi/skills/agent-ci-dev/SKILL.md:
switch the monitor pattern from grepping plaintext "Step failed" to
matching `"event":"(run.paused|run.finish)"`, and call out exit-77 as a
pause signal alongside the `run.paused` event.
- experiments/skill-evals/variants/v2-run-first.md and v3-no-mangling.md:
remove stale "no pipes / no redirects" warnings — those scenarios are now
the supported path via the detached launcher.
Refs #289, #315.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Passing the raw URI as `host` to dockerode left docker-modem with
`host = "ssh://user@remote-host"`, which ssh2 then tried to DNS-resolve,
failing with `getaddrinfo ENOTFOUND` (or `Invalid username` if no user
was configured elsewhere). Parse the URI and pass `host`, `username`,
`port`, and `sshOptions.agent` separately — matching what docker-modem
does internally when it parses `DOCKER_HOST` itself.
Adds an end-to-end test that asserts on the constructed Docker client's
modem state (`docker.modem.host === "remote-host"`), reproducing the bug
without mocking dockerode.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move per-run log artifacts out of the runner's tmp working dir into a
stable, agent-ci-owned directory next to the per-branch checks JSON
(default `<stateDir>/logs/`, override via AGENT_CI_LOG_DIR). Log paths
recorded in the run-result JSON now survive OS-level pruning of
`os.tmpdir()`.
Add an opportunistic, throttled cleanup at the start of `agent-ci run`
plus an explicit `agent-ci clean` subcommand. Policy: never delete a
run dir referenced by any branch's checks JSON, then keep newest N
runs, then drop runs older than M days. Knobs:
AGENT_CI_LOG_RETAIN_DAYS=7, AGENT_CI_LOG_RETAIN_RUNS=20,
AGENT_CI_LOG_PRUNE=0 to disable. O_EXCL lockfile + last-prune stamp
guard against concurrent runs and per-job fan-out.
Also fix `buildStepEnv` in workflow-parser.ts to thread an envContext
through `expandExpressions`, completing PR #320's env.* expansion
support — `${{ env.JOB_KEY }}` inside another step's `env:` value now
resolves to the workflow- or job-level value instead of an empty
string. This made smoke-env-context.yml pass under agent-ci-dev.
Closes#312.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): unblock --pause-on-failure in non-TTY pipes (#315)
Spawn the run as a detached worker when stdout isn't a TTY (and we're
not under an LLM harness), then exit 77 the moment the worker emits a
`run.paused` NDJSON event — freeing the caller's pipe while the worker
keeps the container + DTU alive for `agent-ci retry`. Retry tails the
same log so a re-failure surfaces as another exit-77.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): NDJSON event stream behind --json (#289)
Extends the existing pause/finish sentinel emitter into a full lifecycle
event stream. New `--json` flag (and `AGENT_CI_JSON=1` env var) emits
`run.start`, `job.start`/`finish`, `step.start`/`finish`, `run.paused`,
`run.finish`, `diagnostic` events on stdout — one JSON object per line —
each carrying an `event` discriminator and a `schemaVersion: 1` on
`run.start`.
`--json` is decoupled from `--quiet` so existing `-q` callers keep silent
stdout; the diff renderer is auto-suppressed under `--json` so ANSI
sequences don't collide with the stream. Detached-worker mode (#315)
keeps emitting only sentinels — the launcher filters non-sentinel events
anyway, so non-JSON callers don't pay the cost.
Renames `run.completed` -> `run.finish` to match the issue's naming.
Closes#289.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): suppress printSummary under --json
The ━━━ FAILURES ━━━ / ━━━ SUMMARY ━━━ blocks write to stdout
unconditionally, which interleaves with the NDJSON event stream and
breaks line-based parsers. Skip them when isJsonMode() — consumers get
the same information from job.finish + run.finish events.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): AGENT_CI_DETACHED=1 forces detached launcher on TTY
Overload AGENT_CI_DETACHED so it can serve both roles:
- `=<absolute path>` → set by the launcher on its worker child;
`isDetachedWorker()` returns true.
- `=1` (or any non-absolute value) → caller-side opt-in: force the
launcher path even when stdout is a TTY. Useful for manual
verification of the #315 pause-then-exit-77 flow without having to
construct a non-TTY pipe.
The two are distinguished by whether the value starts with `/`. New
`isForceDetachedRequested()` helper drives a `forceDetached` arg into
`shouldLaunchDetached`, which still respects `alreadyWorker`,
`agentMode`, and the `pauseOnFailure` requirement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(ci): smoke for --json NDJSON event stream (#289)
scripts/smoke-289.sh creates a tmp repo with a 2-step passing workflow,
runs `agent-ci run --json`, and asserts:
- run.start carries schemaVersion=1
- step.start + step.finish for each declared step
- job.finish status=passed
- run.finish status=passed
- every JSON-shaped line on stdout has a recognized event field
.github/workflows/smoke-json-events.yml runs the script in CI on every
PR (gated by the same safe-to-run / collaborator predicate as the other
smokes).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ci): stop smoke-resource-mismatch from hanging on real GitHub runners
The fixture in .github/workflows/smoke-resource-mismatch.yml declares
`runs-on: ubuntu-latest-999-cores` so agent-ci's resource-fidelity
classifier deterministically tags the job as `degraded` on any host.
With its previous `pull_request_target` trigger, GitHub Actions also
picked it up on every PR and queued a job that no real runner can
satisfy — five queued runs piled up on PR #323 alone, each timing out
after hours.
Switch the workflow to `workflow_dispatch:`-only so GitHub never
auto-dispatches it, and teach `isWorkflowRelevant` to treat
`workflow_dispatch:`-only workflows as relevant for `agent-ci run --all`
so the local smoke keeps running. The opt-in is gated to the sole-event
case to avoid overriding `pull_request` / `push` path/branch filters on
workflows that legitimately combine triggers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(workflow): support expansion of env context references in expressions
`${{ env.* }}` references in workflow expressions silently resolved to empty
string because `resolveExprAtom` had no `env.` handler and no env context was
threaded through the evaluator. The fix adds initial support for expanding these
references in `run:` scripts, `with:` inputs, `name:`, and `env:` value
expressions — matching GitHub Actions behavior. `if:` conditions remain
unsupported (noted in `compatibility.md`).
* chore(smoke): add smoke-env-context.yml as proof for env.* row
Adds a smoke workflow that exercises ${{ env.* }} expansion in `run:`
bodies, in another step's `env:` value, in `with:` inputs (the original
Rails CI bug surface), and verifies the unknown-key → empty-string
contract. Also wires it as the `proof` reference on the env.* row in
compatibility.json.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Peter Pistorius <peter.pistorius@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): surface degraded local runs for resource mismatches
Add resource fidelity classification that compares local host resources
against workflow-declared runner/resource hints. When the local machine
is smaller than what the workflow requests, jobs are marked as degraded
and a warning is shown before execution starts.
Features:
- Parse runs-on labels and NODE_OPTIONS from workflows
- Classify jobs as faithful or degraded based on CPU/memory hints
- Show [degraded] tag and actionable warnings in CLI output
- Add smoke test for degraded classification
- Update documentation with degraded mode guidance
Complements PR #226 which adds memory-aware concurrency limiting.
Together they replace the approach in closed PR #78.
* refactor(cli): use runner specs instead of NODE_OPTIONS for resource classification
Replace NODE_OPTIONS parsing with GitHub runner label heuristics.
This is more accurate and works for all workloads (not just Node).
Changes:
- Add RUNNER_SPECS mapping for known GitHub runner sizes
- Add parseRunnerSpecs() to look up full {cpu, memory} from labels
- Remove parseNodeMaxOldSpaceMb() and NODE_OPTIONS dependency
- Update tests to use runner specs
- Simplify smoke test fixture
Runner specs used:
- ubuntu-latest: 2 CPU, 7GB RAM
- ubuntu-latest-4-cores: 4 CPU, 16GB RAM
- ubuntu-latest-8-cores: 8 CPU, 32GB RAM
- ubuntu-latest-16-cores: 16 CPU, 64GB RAM
* feat(cli): warn when using unknown runner labels
When a workflow uses an unknown runner label (e.g., 'self-hosted' or custom
labels not in RUNNER_SPECS), the job is now marked as degraded with a
warning that explains we're assuming local host capacity.
This makes the behavior transparent - users know we're making an assumption
about the runner size rather than silently defaulting to faithful mode.
* Address PR review comments: remove smoke tests, resource warnings, and writeStdoutLine refactor
* Remove remote Docker daemon suggestion from degraded resource warnings
* chore(compat): add resource-mismatch smoke and document degraded mode
- Add smoke-resource-mismatch.yml fixture that uses ubuntu-latest-999-cores
to deterministically trigger the degraded classification on any host.
- Extend the runs-on row in compatibility.json with a sentence describing
the degraded warning and reference the smoke as proof.
- Regenerate compatibility.md.
- Add changeset.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(classifier): only flag unknown labels for the ubuntu family
The "unknown runner label" reason fired for any label missing from
RUNNER_SPECS, including macos-* / windows-* / self-hosted — labels that
runs-on-compat already handles and that the resource classifier has no
useful opinion on. Restrict the warning to labels that look like a
hosted ubuntu variant (`ubuntu` or `ubuntu-*`), which is the family
RUNNER_SPECS actually covers; this still catches drift when GitHub
adds a new ubuntu label without us noticing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(smoke): de-flake bun smoke by pre-fetching the binary
The smoke depended on oven-sh/setup-bun@v2 pulling bun-linux-aarch64.zip
from github.com/oven-sh/bun/releases inside the nested runner. The
upstream action has no per-request timeout or retry, so a stalled CDN
socket presented as a hung step with one log line ("Downloading a new
version of Bun: …") and nothing else — pure flake, no signal about
agent-ci.
Pre-fetch the binary on the outer host with curl --retry, ship it into
the inner workspace as a fixture, and replace the setup-bun call in the
inner workflow with two narrower steps that exercise the actual #212
failure modes: an actions/cache@v4 probe against a randomized `bun-…`
key (must miss — guards against the virtualCachePatterns substring
fake-hit) and a manual `mkdir -p $HOME/.bun/bin` install (exercises the
bind-mount permissions path).
Local run: 16s, deterministic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Peter Pistorius <peter.pistorius@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Revert "feat(runner): smolvm backend foundation (WIP, blocked on smolvm#177) (#287)"
This reverts commit 813036b543.
* chore: add changeset for smolvm backend revert
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(runner): add opt-in smolvm backend foundation
WIP — blocked on smol-machines/smolvm#177 (TSI idle keep-alive drop).
Pure-shell + checkout workflows run end-to-end; anything using
actions/setup-* deadlocks once libkrun kills the idle listener
connection. Foundation (CLI wrapper, host-capability probe, image
mapping, pack cache, dispatch flag, workspace + shim mounts) is in
place and useful regardless of the networking resolution.
Refs #284.
* chore(runner): bump smolvm CLI-surface marker to 0.5.20
Verified end-to-end against smolvm 0.5.20: the previously-deadlocking
actions/setup-node@v4 smoke now passes in ~3 min, confirming the TSI
idle keep-alive fix (smolvm PR #184 / libkrun abd07c8) lands the
closure for smol-machines/smolvm#177.
The three 0.5.19-observed workaround comments in smolvm-job.ts are
left intact — SQLite single-writer, repeated-pull DNS drop, and
virtiofs small-file hang were not mentioned in the 0.5.20 release
notes and were not re-tested here.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
parseWorkflowSteps never extracted `if:` from the workflow, so the DTU
server fell back to `condition: "success()"` for every step. Custom
gates — `contains(...)`, `always()`, even literal `${{ false }}` — were
silently ignored and every step ran regardless. A new `parseStepIf`
helper strips the optional `${{ }}` wrapper and threads the naked
expression through to the runner's EvaluateStepIf, so step gates now
behave as they do on real GitHub Actions. Adds unit coverage for the
helper and a smoke workflow that fails loudly if any gate leaks.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): persist latest run result per worktree to a stable JSON path
Lets external consumers (tmux panes, status bars, editor integrations) read
the current branch's CI status without re-running agent-ci or scraping human
output. Written atomically after every `agent-ci run` / `run --all`, keyed by
`<branch>.<worktree-hash>.json` under `<org>/<repo>/` so two worktrees on the
same branch don't stomp each other. Includes `headSha` so consumers can detect
stale results themselves.
Closes#288
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): include steps[] and per-step log paths in run-result JSON
Each job entry in the persisted run-result JSON now carries:
- `debugLogPath` for the whole job
- `failingStepLogPath` when a step failed
- `steps[]` with name, status, and `logPath` for each step
Path fields are only emitted when the file still exists at write time,
so consumers get an honest view after passing runs clean up their logs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(cli): drop job-level failingStepLogPath from run-result JSON
The failing step's log path is already discoverable in `steps[]` via the
per-step `logPath` on the entry whose `status == "failed"`, so the job-level
field is redundant. Keep `failingStep` (the name) as a top-level convenience.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DOCKER_HOST collides with the shell's Docker CLI expectation and can't
live in .env.agent-ci without leaking to every docker command. Rename
to AGENT_CI_DOCKER_HOST and hard-exit if the legacy DOCKER_HOST is set.
At bootstrap, apply AGENT_CI_*-prefixed keys from .env.agent-ci to
process.env so Docker/network config persists without shell exports.
Closes#308.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each overloaded row becomes one row per documented feature (or a tighter
note when the sub-features behave identically). No behaviour changes.
* github.* split into sha / repository family / everything-else
* runner.* new row for unsupported siblings
* contains etc. three rows, one per function
* success family downgraded partial (cancelled() always false locally)
* on (other) note enumerates ~20 events
Closes the "overloaded row" bucket on #296.
Refs #296.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds rows for features GitHub Actions documents that our compat table
said nothing about. Each status was picked per code inspection of the
existing parser / runner paths — no behaviour changes.
Closes the "missing rows" bucket on #296.
Refs #296.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The YAML parser extracted `jobs.<id>.container.options` into job.container
but the docker createContainer call never consumed it, so `options: --env
FOO=bar` silently produced a container without FOO.
parseContainerOptions now pulls --env/-e and --label/-l out of the options
string and merges them into the container's Env array and Labels map. All
other Docker flags in options: are still ignored by design — --privileged,
--user, --network, --cap-add, --workdir tend to clash with agent-ci's own
container orchestration, and silently dropping them is safer than a partial
honoring that breaks the runner's invariants. Documented in the
compatibility.json note for jobs.<id>.container.
Also updated two nearby notes while running through #296:
* actions/cache — documents the lack of ref-based isolation (branches
share cache keys locally, unlike GitHub's per-ref scoping)
* GITHUB_TOKEN — documents that OIDC id-token issuance is not wired up,
so permissions: id-token: write → federated cloud auth does not work
Refs #296.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two small expression-engine fixes surfaced by extending smoke-expressions
with assertions for #296's "questionable claim" rows:
1. toJSON now emits 2-space-indented JSON (was compact), so hashFiles
keys that hash a toJSON(x) match GitHub. Parses rawValue first so
toJSON(fromJSON(x)) round-trips cleanly.
2. compareValues() no longer bails out of the numeric path when either
side is empty. `''`, `null`, and numeric strings all coerce to
numbers per the spec; non-numeric strings become NaN (string compare
path, always false).
Five new assertions (contains-on-array, toJSON pretty-print, coercion,
hashFiles multi-pattern, hashFiles negation) all pass. hashFiles
multi-pattern and negation were never broken — cleared off #296 with
proof rather than a fix.
Refs #296.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(workflow): hashFiles descends into dotted directories
findFiles was skipping any directory starting with `.`, so a pattern
like hashFiles('.github/workflows/*.yml') never matched a file and
returned the 40-zero placeholder. Skip only node_modules now.
The smoke assertion is tightened back to 64 chars (SHA-256) since the
real digest now appears.
Closes#294.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(workflow): resolve steps.*.outputs.* to empty at parse time
The parser was returning a sentinel so that `${{ steps.X.outputs.Y }}`
expressions would be preserved verbatim for the runner to evaluate at
runtime. The runner does not re-evaluate expressions inside run-script
bodies, so the literal `${{ }}` reached bash as "bad substitution".
Per the long-standing compatibility contract, these expressions now
resolve to an empty string at parse time. Cross-job values still work
through needs.*.outputs.*.
Closes#295.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The runner executes every run: step with bash regardless of inputs.shell,
so setting shell: sh / python / pwsh had no effect. Wrap the expanded
script at parse time — `<interpreter> <flags> <<'EOF' ... EOF` — so the
requested shell actually runs the user's code. Workflow, job, and step
scopes compose via the existing resolveStepRunDefault helper.
Adds smoke-shell-defaults.yml (removed in an earlier bug-filing cut)
covering all three scopes with a BASH_VERSION assertion.
Closes#293.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(compat): make compatibility.json the source of truth
Flip the compatibility matrix so packages/cli/compatibility.json is
authoritative. scripts/gen-compatibility-md.mjs renders the .md from it,
pnpm check runs a drift check, and the website imports the JSON directly
and renders inline markdown in keys/notes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: add compat smoke coverage for secrets, working-dir defaults, runner image
Fill three audit gaps identified against packages/cli/compatibility.json:
- smoke-secrets-env-file: secrets resolved from .env.agent-ci and from
the shell environment fallback
- smoke-defaults-workdir: workflow- and job-level
defaults.run.working-directory plus a step-level override
- smoke-runner-image: .github/agent-ci.Dockerfile is picked up and the
resulting image is used at job start
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(workflow): propagate defaults.run.working-directory to steps
Parse defaults.run.working-directory at workflow and job level and merge
it into each run-step's inputs with step > job > workflow precedence.
Step-level working-directory already worked; the defaults blocks were
silently ignored.
Closes#290.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(compat): add proof field to compatibility.json rows
Every `supported`/`partial` row now lists the smoke workflows that
exercise it end-to-end. `pnpm compat:gen`/`compat:check` fails if a
listed path does not resolve, catching renames.
Rows with an empty `proof: []` are the audit gaps tracked in #292.
Refs #292.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(dtu): attach step env per-step so it does not leak to siblings
Step-level env was being aggregated into the job-wide EnvironmentVariables
map (last-wins), so a later step's `env: FOO=x` would override an earlier
step's effective FOO at runtime. Emit step.Env as the mapped step's own
`environment` instead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(compat): fill audit gaps #292 — env, expressions, local action
Three smokes covering eight of the ten gaps listed in #292:
- smoke-env-scoping: workflow + job + step `env:` precedence
- smoke-expressions: hashFiles, format, join, runner.arch, type
literals, startsWith/endsWith
- smoke-local-action: `steps[*].uses: ./.github/actions/...`
(verified via marker file rather than steps.<id>.outputs.*)
Proof fields updated for every row these smokes cover. hashFiles
downgraded to ⚠️ partial — agent-ci emits SHA-1, GitHub Actions emits
SHA-256 (tracked by #294).
Two gaps remain open as filed issues:
- #293 defaults.run.shell / steps[*].shell not applied
- #295 ${{ steps.<id>.outputs.<name> }} leaks literal text in run
scripts
Refs #292.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Docker Desktop 4.x ships with /var/run/docker.sock disabled by default,
so a fresh install hits 'agent-ci couldn't use a Docker socket' even
with Docker Desktop running. Two small fixes:
1. docs/docker-socket.md — replace the stale 'starts automatically'
line with the actual Settings → Advanced toggle steps.
2. docker-socket.ts — when the resolver throws, detect Docker Desktop's
user-side socket (~/.docker/run/docker.sock) and append a focused
hint pointing at the toggle, so users don't have to read the whole
doc to find the one paragraph that matters.
Refs #253. The underlying permission failure from #253 was already
fixed by #274 (chmod 1777 on _diag/_work); this PR addresses the
remaining onboarding paper-cut.
When two `agent-ci-dev` runs execute in parallel, each nested CLI
process starts in its own isolated filesystem and allocates the same
sibling container name (`agent-ci-1`). The pre-spawn
`docker rm -f <name>` then kills the other nested run's sibling.
Include the outer container's hostname in the name prefix when
`/.dockerenv` is present so sibling names stay unique across
concurrent nested runs.
Also update smoke-docker-buildx.yml to look for nested-run logs under
both `/tmp/agent-ci` (GitHub-hosted runner) and
`$GITHUB_WORKSPACE/packages/cli/.agent-ci` (nested under
agent-ci-dev), since the nested working dir differs in each case.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GitHub Actions evaluates ${{ runner.os }} and ${{ runner.arch }} per-job
based on the job's runs-on label. Prior to this change we hardcoded
Linux/X64 in the expression expander regardless of runs-on, which broke
tart-backed macOS VM jobs — any step gated on `runner.os == 'macOS'`
would evaluate as false inside the VM.
Thread an optional RunnerContext through resolveExprAtom /
evaluateExprValue / expandExpressions / buildStepEnv, and derive it in
parseWorkflowSteps from `parseJobRunsOn`. Unknown/absent labels keep
the previous Linux/X64 default so existing self-hosted configurations
are unaffected. macOS is mapped to ARM64 because agent-ci's macOS
backend (tart) only runs Apple Silicon VMs.
* fix(runner): chmod /home/runner/_diag and _work for Colima/Docker Desktop (#263)
On macOS with Colima or Docker Desktop the bind-mounted _diag and _work
directories surface as root:root 0755 inside the container (host perms
don't translate through the VM mount layer), so the runner user (uid
1001) crashes with UnauthorizedAccessException when writing diag logs
and hits write failures in the workspace. Add MAYBE_SUDO chmod 1777 on
both mount points during container boot, mirroring the existing fix
for /home/runner/.cache (#234). OrbStack and native Linux Docker are
unaffected — the chmod is a no-op there.
* fix(cli): require /var/run/docker.sock, link to per-provider setup doc
Today resolveDockerSocket falls back through a list of well-known
provider sockets (~/.orbstack/..., ~/.docker/..., ~/.colima/<profile>/...)
whenever /var/run/docker.sock isn't usable. On macOS + Colima, those
provider paths are not bind-mountable through the provider's VM, and
Docker rejects the later container launch with a confusing
error while creating mount source path: operation not supported
Stop guessing. Require /var/run/docker.sock to exist (or an explicit
DOCKER_HOST) and fail fast with a short error that links to the new
packages/cli/docs/docker-socket.md — which walks users through the
per-provider setup (Docker Desktop / OrbStack / Colima / native Linux
/ rootless).
The bind-mount invariant comment in docker-socket.ts was already
implicitly assuming this rule; this change just enforces it at the
resolver boundary instead of deferring the failure to container
launch, and gives users a doc URL with the recipe they need.
* feat(cli): run macos-* jobs in real macOS VMs via tart
When the host is darwin/arm64 with tart + sshpass installed, jobs
targeting runs-on: macos-* boot an ephemeral cirruslabs macOS VM,
rsync the macOS actions-runner binary in, and connect it to the
in-process DTU via the host bridge.
- New runner/macos-vm/ module (tart wrapper, image mapping, host
capability probe, runner binary fetch/cache, semaphore, job driver).
- cli.ts routes macOS jobs to the VM executor when the host supports
it; otherwise falls through to the existing skip-with-warning path.
- DTU generators.ts accepts runnerWorkDir/runnerOs/runnerArch so the
seed payload reflects where the runner actually materializes _work
(sibling of run.sh, not a caller-chosen path).
- runner-credentials.ts extracted from local-job.ts so the macOS path
can reuse the same deterministic .runner/.credentials generation.
Concurrency capped at 2 VMs by default (AGENT_CI_MACOS_VM_CONCURRENCY).
Image override via AGENT_CI_MACOS_VM_IMAGE.
Closes#258.
* feat(cli): say *why* macOS jobs skip on incapable hosts
Before: the skip warning for macos-* jobs on a host that can't run the
VM just pointed at #258 and parroted the pre-Phase-3 line about only
running in a Linux container — which is no longer accurate now that
macOS jobs *can* run locally when the host supports it.
Now: formatUnsupportedOSWarning accepts the HostCapability result and
prints the specific reason + install hint when available — e.g.
'tart not installed' with a brew command, or 'requires Apple Silicon
(got x64)'.
* refactor(cli): deduplicate runs-on classification + warning body
- cli.ts: extract classifyJob(ej) + canRunMacosHere(kind) so the
skip path and the executor-routing path share one source of truth
for 'should this job run in the macOS VM?' — the two previously
duplicated the parseJobRunsOn + classifyRunsOn + capability check.
- runs-on-compat.ts: flatten formatUnsupportedOSWarning to a single
body array per kind, filter undefined, then uniformly indent —
drops the optional-hint if-branch and the per-push indent literals.
Pure refactor; all 14 runs-on-compat tests still pass, warning output
is byte-identical across all 4 permutations (macOS with/without
capability, with/without hint, windows).
* docs: document macOS VM runner support
- README: add tart + sshpass to 'Optional' prerequisites and a new
'macOS jobs' section (image table, env vars, caveats).
- compatibility.md: unstale the three entries that claimed Linux-only
(jobs.<id>.runs-on, runner.os, runner.arch). runs-on now reads as a
partial (⚠️) that explains linux/macos/windows separately.
Leaves apps/website/CompatibilityMatrix.tsx alone — it has the same
stale rows but is in an unrelated mid-merge state on this branch.
* chore: oxfmt packages/cli/README.md and compatibility.md
Jobs with runs-on: macos-* or windows-* were silently routed to the
Linux runner container and failed at the first OS-specific step.
Parse runs-on (string, array, or {group,labels} form), classify the
OS, and skip the job with a clear stderr warning when it is not Linux.
Linux jobs are unchanged. macOS support lands later via tart (#258).
* test: smoke reproducer for #257 docker/setup-buildx-action
* chore: retrigger CI
* ci: switch smoke to pull_request so it triggers from the PR branch
* ci(smoke): dump agent-ci step logs + upload as artifact on failure
* fix: use MAYBE_SUDO to chmod bind-mounted docker socket (#257)
On native Linux Docker the socket is owned root:docker 0660. The runner
user isn't in the host's docker group, so the existing plain 'chmod 666
/var/run/docker.sock' silently failed — leaving every subsequent Docker
client call (docker/setup-buildx-action@v4, docker login, compose, buildx)
to fail with 'permission denied while trying to connect to the docker
API'. Escalating via MAYBE_SUDO matches the surrounding entrypoint ops.
* ci(smoke): assert against Verify-buildx.log, not top-level agent-ci stdout
---------
Co-authored-by: Peter Pistorius <peter@redwoodjs.com>
The workflow parser only read env: declared directly on a step.
Workflow-level and job-level env: blocks were silently ignored, so
any workflow that relied on them — including the smoke-vars-preflight
smoke test that uses a job-level env with a vars.X reference — saw
empty values at runtime.
The parser now merges env: from all three levels per the GitHub
Actions spec: workflow → job → step, step-most-specific wins.
Expressions (${{ vars.X }}, ${{ secrets.X }}, etc.) are expanded on
each value with the full context, so var/secret references resolve
identically regardless of which level declares the env.
Implementation:
* New internal buildStepEnv() helper in workflow-parser.ts that
merges the three levels and runs expandExpressions per value.
* Both the run: and uses: step branches now call it, dropping
the duplicated inline Env-construction logic.
* 5 new unit tests covering job-only, workflow-only, three-level
precedence, vars expansion in job env, and the no-env case.
Fixes the Case 3 failure in .github/workflows/smoke-vars-preflight.yml
that #270 uncovered — Case 3 had been silently passing via env leak
from the outer runner process, not via actual var substitution.
* kindling: start task [skip ci]
* harness: save uncommitted work from Analyst (cycle 1) [skip ci]
* Fix misleading 401/403 error hint when GitHub token is already provided
When fetching a remote workflow from a private repo with a token supplied,
a 401 or 403 from GitHub now says the token may be invalid, expired, or
lack the required 'repo' scope — instead of repeating instructions to use
--github-token that the user already followed.
Added two tests covering the with-token 401 and 403 paths, which were
previously uncovered.
Fixes#255
* harness: save uncommitted work from Developer (cycle 2) [skip ci]
* harness: save uncommitted work from Reviewer (cycle 3) [skip ci]
* harness: pre-DONE sweep [skip ci]
* kindling: ready for CI
* harness: save uncommitted work from Analyst (cycle 1) [skip ci]
* harness: save uncommitted work from Developer (cycle 2) [skip ci]
* harness: save uncommitted work from Reviewer (cycle 3) [skip ci]
* harness: save uncommitted work from Developer (cycle 4) [skip ci]
* harness: save uncommitted work from Reviewer (cycle 5) [skip ci]
* harness: pre-DONE sweep [skip ci]
* fix: handle 404 on remote workflow fetch as private-repo auth failure
GitHub's Contents API returns HTTP 404 (not 401/403) when a caller lacks
access to a private repository — this is intentional to avoid leaking repo
existence. The previous error hint only covered 401/403, so users hitting
the most common private-repo failure modes (missing token, classic PAT
without 'repo' scope, fine-grained PAT without 'contents: read', unauthorized
SSO PAT) saw a bare "HTTP 404" with no guidance.
Unify hint generation into buildAuthHint() covering 401/403/404 × with/
without token. Hints now include explicit gh auth login instructions and
spell out the token-insufficient scenarios.
Verified end-to-end against a private repro repo (peterp/agent-ci-private):
no-token → 404 path now emits full auth guidance, invalid-token → 401 path
emits token-insufficient hint, valid-token → resolves.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: smoke test for private-repo remote workflow fetch (#261)
Adds .github/workflows/smoke-remote-private-workflow.yml — the end-to-end
verification workflow used to reproduce and prove the #261 fix, now preserved
as a regression-guard smoke test.
It references a reusable workflow in a private repo
(peterp/agent-ci-private/.github/workflows/lint.yml@main). Under agent-ci this
exercises prefetchRemoteWorkflows against the GitHub Contents API with the
supplied token, covering the success path (HTTP 200 from a private repo when
a valid token is attached) that the unit tests in
packages/cli/src/workflow/remote-workflow-fetch.test.ts don't cover.
Verified locally with 'pnpm agent-ci-dev run --workflow … --github-token':
Status: ✓ 2 passed (2 total)
Duration: 9s
Without a token, the same smoke surfaces the improved #261 hint chain verbatim
(HTTP 404 → 'The repository or ref was not found …' → auth instructions).
---------
Co-authored-by: kindling <kindling@harness>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* kindling: start task [skip ci]
* Add behavioral specs and tests for vars context support (SPEC-V-001 through SPEC-V-015)
Tests cover loading vars from dotenv file, env-var allowlist fallback, file-over-env
precedence, extracting ${{ vars.X }} refs from workflow YAML (whole-file and job-scoped),
validating that all referenced vars are supplied (error names missing keys and config file
path), and that workflows with no vars refs run without any config present. Tests mirror
the existing secrets test conventions and will pass once the implementation (Track A) is
in place.
* harness: save uncommitted work from Developer+QA (cycle 1) [skip ci]
* harness: save uncommitted work from Reviewer (cycle 2) [skip ci]
* harness: save uncommitted work from Developer (cycle 3) [skip ci]
* harness: save uncommitted work from Reviewer (cycle 4) [skip ci]
* Add vars context loading, extraction, validation, and expression substitution
- loadMachineVars in config.ts reads .env.agent-ci.vars with the same
KEY=VALUE parsing and envFallbackKeys allowlist as loadMachineSecrets
- extractVarRefs in workflow-parser.ts scans workflow YAML for
${{ vars.NAME }} references using the same pattern as extractSecretRefs
- validateVars throws a descriptive error listing missing var names and
the expected file path, mirroring validateSecrets
- resolveExprAtom gains a vars? 7th param and a vars. prefix case after
the secrets. case; all 11 internal evaluateExprValue calls for
fromJSON/toJSON/format/contains/startsWith/endsWith/join updated
- evaluateExprValue gains vars? 7th param threaded through all 6
recursive calls (||, &&, !, comparisons, parens) and the
resolveExprAtom leaf at line 606
- expandExpressions and parseWorkflowSteps each gain vars? as optional
7th param; all internal expandExpressions call sites updated
* Wire vars context into all three CLI call sites
- Import loadMachineVars, extractVarRefs, validateVars from their
respective modules
- Single-job path: extract refs, load vars, validate, thread vars
through expandExpressions (inputsContext expansion) and
parseWorkflowSteps
- buildJob: extract refs, load vars, validate (no parseWorkflowSteps
call here, consistent with how secrets are handled)
- runJob: extract refs, load vars, validate, thread vars through
resolveInputsForJob and parseWorkflowSteps
- resolveInputsForJob: accepts vars? as 4th optional param and threads
it through its expandExpressions call
* Add Vars section to README and minor-bump changeset
README documents .env.agent-ci.vars file and shell env fallback,
explaining that vars are non-sensitive and can be committed.
Changeset bumps both @redwoodjs/agent-ci and dtu-github-actions minor
as they are in the same fixed versioning group.
* Fix loadMachineVars test helper to write .env.agent-ci.vars
The QA-authored loadMachineVars tests used .env.agent-ci as the
fixture file name, but loadMachineVars reads from .env.agent-ci.vars
per the approved RFC. Updated the writeEnvFile helper and test
description in the loadMachineVars block to match the correct file.
* harness: save uncommitted work from Developer (cycle 5) [skip ci]
* Replace file+env vars sourcing with --var CLI flag
Address PR feedback: workflow vars (${{ vars.FOO }}) are now supplied
exclusively via the repeatable --var KEY=VALUE CLI flag. Removes the
.env.agent-ci.vars file and the shell-env fallback so workflow vars
don't overload shell env vars. Missing vars fail fast with an error
listing the --var flags needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Pre-flight missing vars across all workflows before setup
Scan every workflow for ${{ vars.* }} refs at the start of the run
and fail with one combined message listing every missing var across
every file. Happens before Docker prefetch and workspace setup, so
users find out immediately instead of waiting through bootstrap.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add smoke test for --var pre-flight
Covers three cases against a temp workflow referencing vars.API_URL
and vars.DEPLOY_ENV:
1. No --var flags → exit 1, error lists both vars with --var syntax
2. Partial fill → error lists only the still-missing var
3. All vars supplied → run proceeds and the job succeeds
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kindling <kindling@harness>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: setup-node "Bad credentials" + stale-toolcache hint (#171, #249)
Route setup-node's versions-manifest fetch through the DTU so it no
longer 401s against real api.github.com, and surface an actionable
hint when a stale /opt/hostedtoolcache bind mount blocks tar extract.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: unit + smoke coverage for setup-node manifest mock
Adds unit tests for rewriteSetupNodeTarball and for the cached tree/blob
route paths, plus a nested-agent-ci smoke workflow that fails if
setup-node logs "Bad credentials" (the symptom of the manifest mock
breaking and falling back to nodejs.org).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: promote packages/cli/README as the main README
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: symlink root README to packages/cli/README
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: reframe Why section — remote CI is the gatekeeper, agent-ci is the pre-flight check
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: note Docker Desktop is also supported on Linux
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First-time users saw a frozen spinner or "No such image" error because
the runner image pull was silent and failures were only debug-logged.
Now the CLI detects a missing image, shows download/extraction progress,
and surfaces pull errors visibly.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Actions like dorny/paths-filter read github.event.repository.default_branch
to determine the base branch. The DTU was already passing this value in
Resources.Repositories but omitting it from the event context.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: always show workflows and jobs in --all mode, fix duplicate matrix jobs
Two problems fixed:
1. Jobs would disappear. When running --all, jobs under a workflow
(like install-a, install-b, install-c) would only show up while
the workflow was running. Before it started or after it finished,
they'd vanish. Now they always stay visible.
2. Matrix jobs showed up twice. When a workflow had matrix shards
(like 3 copies of the same job), the display would show 5 entries
instead of 3. This happened because each job got two different
names — one when we reserved its spot, and a different one when
it actually started running. The second name didn't match the
first, so the system thought they were different jobs. Now both
use the same name.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: reduce spinner flicker in --all mode with line-diff renderer
Replace log-update's full erase-and-rewrite approach with a line-level
diff renderer that only sends ANSI escape codes for changed lines. For a
30-line tree where only spinner characters change, this skips ~90% of
terminal I/O per frame.
Also make job cleanup non-blocking: convert fs.rmSync to fs.promises.rm
and store.save() to async fs operations so completing runners don't
freeze the event loop (and stall all other spinners).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: unblock event loop during workspace prep to keep spinners smooth
Convert initFakeGitRepo (7 sequential execSync calls) and the
git-archive path in prepareWorkspace to use promisified exec. Each
await yields to the event loop between git process spawns, letting
the render timer fire instead of freezing all spinners for ~200ms.
Also convert the chmod -R 777 call to async exec, and wrap diff-renderer
updates in DEC synchronized output markers so terminals that support it
batch the entire frame into a single repaint.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: return Promise from save() and update tests to await it
The async save() was fire-and-forget, so tests checking the file
immediately after save() failed. Return the Promise so callers
(including tests) can await completion when needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: add smoke test reproducing Cypress cache permission failure (#234)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: make /home/runner/.cache world-writable so Cypress can install (#234)
The Playwright bind mount causes Docker to create /home/runner/.cache as
root, so Cypress postinstall fails with EACCES when it tries to mkdir
/home/runner/.cache/Cypress. chmod 1777 in the entrypoint fixes it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: skip orphan cleanup when running inside a Docker container
When agent-ci runs nested inside a container (e.g. smoke-bun-setup step 8),
it shares the host's Docker socket. killOrphanedContainers() checks
agent-ci.pid labels using process.kill(pid, 0), but host PIDs don't
exist in the container's PID namespace — so every container looks like
an orphan. This caused the nested agent-ci to force-remove its own
parent container (exit 137).
Both pruneOrphanedDockerResources() and killOrphanedContainers() now
bail out when /.dockerenv exists.
Fixes#228
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: ignore inherited AGENT_CI_DTU_HOST inside containers
When agent-ci runs nested inside a container (e.g. smoke-bun-setup),
the outer container's AGENT_CI_DTU_HOST=host.docker.internal is
inherited. The nested agent-ci used this to tell the inner container
where the DTU is, but the nested DTU is inside the outer container,
not on the host. The inner container got "Connection refused" trying
to reach host.docker.internal:PORT.
Now when /.dockerenv exists, AGENT_CI_DTU_HOST is ignored and the
container's own IP (via hostname -I) is used instead, so sibling
containers can reach the DTU.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: move /.dockerenv guard to call site so tests still work
The /.dockerenv check inside killOrphanedContainers() broke the
shutdown integration tests — they run inside a Docker container (the
test runner) and need the function to work. Moved the guard to the
session bootstrap in cli.ts where the nested agent-ci problem occurs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add global concurrency limiter across all workflows
In --all mode, each workflow created its own per-wave limiter, allowing
20+ containers to run simultaneously and exhaust available memory (OOM
kills with exit 137). Now a single global limiter (CPU/2) is created in
the orchestrator and shared by all handleWorkflow calls, including
single-job workflows and cold-cache first jobs.
Closes#225
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: factor Docker VM memory into concurrency limit
CPU-based limit alone (cpuCount/2 = 7) still causes OOM kills. Now
reads MemAvailable from /proc/meminfo inside the Docker VM and caps
at floor(available / 3GB per container). Falls back to docker info
with a 4GB reserve if busybox isn't available.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: guard orphan cleanup in the function, not just the call site
The /.dockerenv guard was only in cli.ts, but killOrphanedContainers
is also called directly by the Docker integration tests (via a spawned
tsx subprocess). Inside a container, the test's subprocess would see
the parent container's PID label, misidentify it as an orphan, and
kill the container running the tests.
Put the guard back in both functions AND keep the cli.ts guard.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: increase per-container memory estimate to 4 GB
3 GB per container wasn't enough — tests.yml (vitest + Docker
integration tests) and smoke-binary.yml (full build) both got killed
at 3 concurrent containers. 4 GB gives 2 concurrent containers on a
12 GB Docker VM, with ~5.4 GB each.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: mock fs.existsSync in shutdown unit tests for /.dockerenv guard
The unit tests mock execSync but not fs. Inside a runner container,
the real fs.existsSync("/.dockerenv") returns true, causing
killOrphanedContainers to bail before reaching the mocked execSync.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: skip Docker integration tests for orphan cleanup inside containers
The integration tests spawn a real killOrphanedContainers() subprocess.
Inside a container, the function bails due to /.dockerenv (PID namespace
mismatch), so the tests can't exercise it. Skip them with insideDocker().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add --jobs flag and document concurrency
Adds -j/--jobs <n> to override the auto-detected concurrency limit.
Adds a Concurrency section to the README explaining how the limit is
calculated and how to override it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: catch any missing tool in runner image hint, add Nix recipe and docs
Closes#229
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Revert "feat: catch any missing tool in runner image hint, add Nix recipe and docs"
This reverts commit ca261639e9.
* feat: generic missing-tool detection with Nix/xz recipe
Replace the hardcoded per-tool pattern list with a single generic
"command not found" regex that catches any missing tool. Add detection
for the Nix installer's "you do not have 'xz' installed" message.
Update hint to point to runner-image.md docs instead of inlining a
Dockerfile snippet. Add Nix recipe to runner-image.md and a runner
image section to the README.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
github-actions[bot]