Update the agent-facing skill surfaces to match the recent pause-on-failure
fixes and `--json` event stream:
- skills/agent-ci/SKILL.md (published) and packages/cli/SKILL.md: document
`--json` / `AGENT_CI_JSON=1`, list the NDJSON event types, and note that
pipes/redirects are safe — pause-on-failure exits 77 in non-TTY mode.
- .claude/commands/agent-ci-dev.md and .pi/skills/agent-ci-dev/SKILL.md:
switch the monitor pattern from grepping plaintext "Step failed" to
matching `"event":"(run.paused|run.finish)"`, and call out exit-77 as a
pause signal alongside the `run.paused` event.
- experiments/skill-evals/variants/v2-run-first.md and v3-no-mangling.md:
remove stale "no pipes / no redirects" warnings — those scenarios are now
the supported path via the detached launcher.
Refs #289, #315.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Passing the raw URI as `host` to dockerode left docker-modem with
`host = "ssh://user@remote-host"`, which ssh2 then tried to DNS-resolve,
failing with `getaddrinfo ENOTFOUND` (or `Invalid username` if no user
was configured elsewhere). Parse the URI and pass `host`, `username`,
`port`, and `sshOptions.agent` separately — matching what docker-modem
does internally when it parses `DOCKER_HOST` itself.
Adds an end-to-end test that asserts on the constructed Docker client's
modem state (`docker.modem.host === "remote-host"`), reproducing the bug
without mocking dockerode.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move per-run log artifacts out of the runner's tmp working dir into a
stable, agent-ci-owned directory next to the per-branch checks JSON
(default `<stateDir>/logs/`, override via AGENT_CI_LOG_DIR). Log paths
recorded in the run-result JSON now survive OS-level pruning of
`os.tmpdir()`.
Add an opportunistic, throttled cleanup at the start of `agent-ci run`
plus an explicit `agent-ci clean` subcommand. Policy: never delete a
run dir referenced by any branch's checks JSON, then keep newest N
runs, then drop runs older than M days. Knobs:
AGENT_CI_LOG_RETAIN_DAYS=7, AGENT_CI_LOG_RETAIN_RUNS=20,
AGENT_CI_LOG_PRUNE=0 to disable. O_EXCL lockfile + last-prune stamp
guard against concurrent runs and per-job fan-out.
Also fix `buildStepEnv` in workflow-parser.ts to thread an envContext
through `expandExpressions`, completing PR #320's env.* expansion
support — `${{ env.JOB_KEY }}` inside another step's `env:` value now
resolves to the workflow- or job-level value instead of an empty
string. This made smoke-env-context.yml pass under agent-ci-dev.
Closes#312.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): unblock --pause-on-failure in non-TTY pipes (#315)
Spawn the run as a detached worker when stdout isn't a TTY (and we're
not under an LLM harness), then exit 77 the moment the worker emits a
`run.paused` NDJSON event — freeing the caller's pipe while the worker
keeps the container + DTU alive for `agent-ci retry`. Retry tails the
same log so a re-failure surfaces as another exit-77.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): NDJSON event stream behind --json (#289)
Extends the existing pause/finish sentinel emitter into a full lifecycle
event stream. New `--json` flag (and `AGENT_CI_JSON=1` env var) emits
`run.start`, `job.start`/`finish`, `step.start`/`finish`, `run.paused`,
`run.finish`, `diagnostic` events on stdout — one JSON object per line —
each carrying an `event` discriminator and a `schemaVersion: 1` on
`run.start`.
`--json` is decoupled from `--quiet` so existing `-q` callers keep silent
stdout; the diff renderer is auto-suppressed under `--json` so ANSI
sequences don't collide with the stream. Detached-worker mode (#315)
keeps emitting only sentinels — the launcher filters non-sentinel events
anyway, so non-JSON callers don't pay the cost.
Renames `run.completed` -> `run.finish` to match the issue's naming.
Closes#289.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): suppress printSummary under --json
The ━━━ FAILURES ━━━ / ━━━ SUMMARY ━━━ blocks write to stdout
unconditionally, which interleaves with the NDJSON event stream and
breaks line-based parsers. Skip them when isJsonMode() — consumers get
the same information from job.finish + run.finish events.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): AGENT_CI_DETACHED=1 forces detached launcher on TTY
Overload AGENT_CI_DETACHED so it can serve both roles:
- `=<absolute path>` → set by the launcher on its worker child;
`isDetachedWorker()` returns true.
- `=1` (or any non-absolute value) → caller-side opt-in: force the
launcher path even when stdout is a TTY. Useful for manual
verification of the #315 pause-then-exit-77 flow without having to
construct a non-TTY pipe.
The two are distinguished by whether the value starts with `/`. New
`isForceDetachedRequested()` helper drives a `forceDetached` arg into
`shouldLaunchDetached`, which still respects `alreadyWorker`,
`agentMode`, and the `pauseOnFailure` requirement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(ci): smoke for --json NDJSON event stream (#289)
scripts/smoke-289.sh creates a tmp repo with a 2-step passing workflow,
runs `agent-ci run --json`, and asserts:
- run.start carries schemaVersion=1
- step.start + step.finish for each declared step
- job.finish status=passed
- run.finish status=passed
- every JSON-shaped line on stdout has a recognized event field
.github/workflows/smoke-json-events.yml runs the script in CI on every
PR (gated by the same safe-to-run / collaborator predicate as the other
smokes).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ci): stop smoke-resource-mismatch from hanging on real GitHub runners
The fixture in .github/workflows/smoke-resource-mismatch.yml declares
`runs-on: ubuntu-latest-999-cores` so agent-ci's resource-fidelity
classifier deterministically tags the job as `degraded` on any host.
With its previous `pull_request_target` trigger, GitHub Actions also
picked it up on every PR and queued a job that no real runner can
satisfy — five queued runs piled up on PR #323 alone, each timing out
after hours.
Switch the workflow to `workflow_dispatch:`-only so GitHub never
auto-dispatches it, and teach `isWorkflowRelevant` to treat
`workflow_dispatch:`-only workflows as relevant for `agent-ci run --all`
so the local smoke keeps running. The opt-in is gated to the sole-event
case to avoid overriding `pull_request` / `push` path/branch filters on
workflows that legitimately combine triggers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(workflow): support expansion of env context references in expressions
`${{ env.* }}` references in workflow expressions silently resolved to empty
string because `resolveExprAtom` had no `env.` handler and no env context was
threaded through the evaluator. The fix adds initial support for expanding these
references in `run:` scripts, `with:` inputs, `name:`, and `env:` value
expressions — matching GitHub Actions behavior. `if:` conditions remain
unsupported (noted in `compatibility.md`).
* chore(smoke): add smoke-env-context.yml as proof for env.* row
Adds a smoke workflow that exercises ${{ env.* }} expansion in `run:`
bodies, in another step's `env:` value, in `with:` inputs (the original
Rails CI bug surface), and verifies the unknown-key → empty-string
contract. Also wires it as the `proof` reference on the env.* row in
compatibility.json.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Peter Pistorius <peter.pistorius@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): surface degraded local runs for resource mismatches
Add resource fidelity classification that compares local host resources
against workflow-declared runner/resource hints. When the local machine
is smaller than what the workflow requests, jobs are marked as degraded
and a warning is shown before execution starts.
Features:
- Parse runs-on labels and NODE_OPTIONS from workflows
- Classify jobs as faithful or degraded based on CPU/memory hints
- Show [degraded] tag and actionable warnings in CLI output
- Add smoke test for degraded classification
- Update documentation with degraded mode guidance
Complements PR #226 which adds memory-aware concurrency limiting.
Together they replace the approach in closed PR #78.
* refactor(cli): use runner specs instead of NODE_OPTIONS for resource classification
Replace NODE_OPTIONS parsing with GitHub runner label heuristics.
This is more accurate and works for all workloads (not just Node).
Changes:
- Add RUNNER_SPECS mapping for known GitHub runner sizes
- Add parseRunnerSpecs() to look up full {cpu, memory} from labels
- Remove parseNodeMaxOldSpaceMb() and NODE_OPTIONS dependency
- Update tests to use runner specs
- Simplify smoke test fixture
Runner specs used:
- ubuntu-latest: 2 CPU, 7GB RAM
- ubuntu-latest-4-cores: 4 CPU, 16GB RAM
- ubuntu-latest-8-cores: 8 CPU, 32GB RAM
- ubuntu-latest-16-cores: 16 CPU, 64GB RAM
* feat(cli): warn when using unknown runner labels
When a workflow uses an unknown runner label (e.g., 'self-hosted' or custom
labels not in RUNNER_SPECS), the job is now marked as degraded with a
warning that explains we're assuming local host capacity.
This makes the behavior transparent - users know we're making an assumption
about the runner size rather than silently defaulting to faithful mode.
* Address PR review comments: remove smoke tests, resource warnings, and writeStdoutLine refactor
* Remove remote Docker daemon suggestion from degraded resource warnings
* chore(compat): add resource-mismatch smoke and document degraded mode
- Add smoke-resource-mismatch.yml fixture that uses ubuntu-latest-999-cores
to deterministically trigger the degraded classification on any host.
- Extend the runs-on row in compatibility.json with a sentence describing
the degraded warning and reference the smoke as proof.
- Regenerate compatibility.md.
- Add changeset.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(classifier): only flag unknown labels for the ubuntu family
The "unknown runner label" reason fired for any label missing from
RUNNER_SPECS, including macos-* / windows-* / self-hosted — labels that
runs-on-compat already handles and that the resource classifier has no
useful opinion on. Restrict the warning to labels that look like a
hosted ubuntu variant (`ubuntu` or `ubuntu-*`), which is the family
RUNNER_SPECS actually covers; this still catches drift when GitHub
adds a new ubuntu label without us noticing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(smoke): de-flake bun smoke by pre-fetching the binary
The smoke depended on oven-sh/setup-bun@v2 pulling bun-linux-aarch64.zip
from github.com/oven-sh/bun/releases inside the nested runner. The
upstream action has no per-request timeout or retry, so a stalled CDN
socket presented as a hung step with one log line ("Downloading a new
version of Bun: …") and nothing else — pure flake, no signal about
agent-ci.
Pre-fetch the binary on the outer host with curl --retry, ship it into
the inner workspace as a fixture, and replace the setup-bun call in the
inner workflow with two narrower steps that exercise the actual #212
failure modes: an actions/cache@v4 probe against a randomized `bun-…`
key (must miss — guards against the virtualCachePatterns substring
fake-hit) and a manual `mkdir -p $HOME/.bun/bin` install (exercises the
bind-mount permissions path).
Local run: 16s, deterministic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Peter Pistorius <peter.pistorius@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Revert "feat(runner): smolvm backend foundation (WIP, blocked on smolvm#177) (#287)"
This reverts commit 813036b543.
* chore: add changeset for smolvm backend revert
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds experiments/skill-evals/, a matrix eval that runs headless Claude Code
against broken-repo fixtures with candidate SKILL.md wordings appended to
CLAUDE.md, scores tool calls + file edits against per-fixture rubric regexes,
and aggregates across N trials per (variant, fixture) cell.
Includes:
- run.mjs + rescore.mjs (live eval + replay saved transcripts against a
changed rubric without re-running the agent)
- 3 fixtures: lint-curly-x18, missing-changeset, noisy-failure
- 3 variants: v1-current (baseline), v2-run-first, v3-no-mangling
- Stub @redwoodjs/agent-ci per fixture — no Docker required for the eval
Fixture repos are deliberately broken (syntax errors, missing changesets,
etc.); .prettierignore and .oxlintrc ignorePatterns exclude them so the
pre-commit hooks stay green.
Scoped to experiments/, no changes to published packages, no changeset.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(runner): add opt-in smolvm backend foundation
WIP — blocked on smol-machines/smolvm#177 (TSI idle keep-alive drop).
Pure-shell + checkout workflows run end-to-end; anything using
actions/setup-* deadlocks once libkrun kills the idle listener
connection. Foundation (CLI wrapper, host-capability probe, image
mapping, pack cache, dispatch flag, workspace + shim mounts) is in
place and useful regardless of the networking resolution.
Refs #284.
* chore(runner): bump smolvm CLI-surface marker to 0.5.20
Verified end-to-end against smolvm 0.5.20: the previously-deadlocking
actions/setup-node@v4 smoke now passes in ~3 min, confirming the TSI
idle keep-alive fix (smolvm PR #184 / libkrun abd07c8) lands the
closure for smol-machines/smolvm#177.
The three 0.5.19-observed workaround comments in smolvm-job.ts are
left intact — SQLite single-writer, repeated-pull DNS drop, and
virtiofs small-file hang were not mentioned in the 0.5.20 release
notes and were not re-tested here.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
parseWorkflowSteps never extracted `if:` from the workflow, so the DTU
server fell back to `condition: "success()"` for every step. Custom
gates — `contains(...)`, `always()`, even literal `${{ false }}` — were
silently ignored and every step ran regardless. A new `parseStepIf`
helper strips the optional `${{ }}` wrapper and threads the naked
expression through to the runner's EvaluateStepIf, so step gates now
behave as they do on real GitHub Actions. Adds unit coverage for the
helper and a smoke workflow that fails loudly if any gate leaks.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): persist latest run result per worktree to a stable JSON path
Lets external consumers (tmux panes, status bars, editor integrations) read
the current branch's CI status without re-running agent-ci or scraping human
output. Written atomically after every `agent-ci run` / `run --all`, keyed by
`<branch>.<worktree-hash>.json` under `<org>/<repo>/` so two worktrees on the
same branch don't stomp each other. Includes `headSha` so consumers can detect
stale results themselves.
Closes#288
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): include steps[] and per-step log paths in run-result JSON
Each job entry in the persisted run-result JSON now carries:
- `debugLogPath` for the whole job
- `failingStepLogPath` when a step failed
- `steps[]` with name, status, and `logPath` for each step
Path fields are only emitted when the file still exists at write time,
so consumers get an honest view after passing runs clean up their logs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(cli): drop job-level failingStepLogPath from run-result JSON
The failing step's log path is already discoverable in `steps[]` via the
per-step `logPath` on the entry whose `status == "failed"`, so the job-level
field is redundant. Keep `failingStep` (the name) as a top-level convenience.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Target tmux panes by captured pane ID instead of window:pane indices so
the script works regardless of the user's tmux base-index. Invoke
scripts/agent-ci-dev.sh directly instead of going through
pnpm --dir "$REPO_ROOT", so the CLI inherits DEMO_DIR as its cwd and
resolves the demo's own .github/workflows/ci.yml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DOCKER_HOST collides with the shell's Docker CLI expectation and can't
live in .env.agent-ci without leaking to every docker command. Rename
to AGENT_CI_DOCKER_HOST and hard-exit if the legacy DOCKER_HOST is set.
At bootstrap, apply AGENT_CI_*-prefixed keys from .env.agent-ci to
process.env so Docker/network config persists without shell exports.
Closes#308.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#297. CI/scripts-only — no packages/ change, so no changeset required
per .claude/rules/changeset-required.md.
- scripts/diff-parse.ts: walks raw YAML at workflow + job scopes (in addition
to the existing step scope) and grades every recognized key against a
preservation predicate or a documented expected-drop. Reuses focused parsers
(parseJobIf, parseJobOutputDefs, parseWorkflowContainer, parseWorkflowServices,
parseMatrixDef, parseFailFast, parseJobDependencies, getWorkflowTemplate).
defaults.run.shell preservation detects the heredoc-wrap marker emitted by
wrapScriptForShell.
- third-party-workflows/: vendored scan-only corpus — turborepo-lint.yml,
playwright-infra.yml, docker-validate.yml — each with attribution header.
Picked up automatically by collectWorkflowFiles.
- package.json: parse:diff added to the parallel pnpm check pipeline so
regressions fail standard CI.
Result: 37 workflows scanned (.github/workflows + third-party-workflows),
0 drifts. pnpm check exits 0; pnpm agent-ci-dev run --all reports 39/39.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each overloaded row becomes one row per documented feature (or a tighter
note when the sub-features behave identically). No behaviour changes.
* github.* split into sha / repository family / everything-else
* runner.* new row for unsupported siblings
* contains etc. three rows, one per function
* success family downgraded partial (cancelled() always false locally)
* on (other) note enumerates ~20 events
Closes the "overloaded row" bucket on #296.
Refs #296.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds rows for features GitHub Actions documents that our compat table
said nothing about. Each status was picked per code inspection of the
existing parser / runner paths — no behaviour changes.
Closes the "missing rows" bucket on #296.
Refs #296.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The YAML parser extracted `jobs.<id>.container.options` into job.container
but the docker createContainer call never consumed it, so `options: --env
FOO=bar` silently produced a container without FOO.
parseContainerOptions now pulls --env/-e and --label/-l out of the options
string and merges them into the container's Env array and Labels map. All
other Docker flags in options: are still ignored by design — --privileged,
--user, --network, --cap-add, --workdir tend to clash with agent-ci's own
container orchestration, and silently dropping them is safer than a partial
honoring that breaks the runner's invariants. Documented in the
compatibility.json note for jobs.<id>.container.
Also updated two nearby notes while running through #296:
* actions/cache — documents the lack of ref-based isolation (branches
share cache keys locally, unlike GitHub's per-ref scoping)
* GITHUB_TOKEN — documents that OIDC id-token issuance is not wired up,
so permissions: id-token: write → federated cloud auth does not work
Refs #296.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two small expression-engine fixes surfaced by extending smoke-expressions
with assertions for #296's "questionable claim" rows:
1. toJSON now emits 2-space-indented JSON (was compact), so hashFiles
keys that hash a toJSON(x) match GitHub. Parses rawValue first so
toJSON(fromJSON(x)) round-trips cleanly.
2. compareValues() no longer bails out of the numeric path when either
side is empty. `''`, `null`, and numeric strings all coerce to
numbers per the spec; non-numeric strings become NaN (string compare
path, always false).
Five new assertions (contains-on-array, toJSON pretty-print, coercion,
hashFiles multi-pattern, hashFiles negation) all pass. hashFiles
multi-pattern and negation were never broken — cleared off #296 with
proof rather than a fix.
Refs #296.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(workflow): hashFiles descends into dotted directories
findFiles was skipping any directory starting with `.`, so a pattern
like hashFiles('.github/workflows/*.yml') never matched a file and
returned the 40-zero placeholder. Skip only node_modules now.
The smoke assertion is tightened back to 64 chars (SHA-256) since the
real digest now appears.
Closes#294.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(workflow): resolve steps.*.outputs.* to empty at parse time
The parser was returning a sentinel so that `${{ steps.X.outputs.Y }}`
expressions would be preserved verbatim for the runner to evaluate at
runtime. The runner does not re-evaluate expressions inside run-script
bodies, so the literal `${{ }}` reached bash as "bad substitution".
Per the long-standing compatibility contract, these expressions now
resolve to an empty string at parse time. Cross-job values still work
through needs.*.outputs.*.
Closes#295.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The runner executes every run: step with bash regardless of inputs.shell,
so setting shell: sh / python / pwsh had no effect. Wrap the expanded
script at parse time — `<interpreter> <flags> <<'EOF' ... EOF` — so the
requested shell actually runs the user's code. Workflow, job, and step
scopes compose via the existing resolveStepRunDefault helper.
Adds smoke-shell-defaults.yml (removed in an earlier bug-filing cut)
covering all three scopes with a BASH_VERSION assertion.
Closes#293.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
For each .github/workflows/*.yml, read the raw YAML and compare against
what parseWorkflowSteps emits, flagging any step-level key our parser
silently drops without a documented expected-drop policy.
First run over all 26 workflows: 0 drifts. Job-level and workflow-level
scans are tracked as follow-ups in #297.
Refs #297.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(compat): make compatibility.json the source of truth
Flip the compatibility matrix so packages/cli/compatibility.json is
authoritative. scripts/gen-compatibility-md.mjs renders the .md from it,
pnpm check runs a drift check, and the website imports the JSON directly
and renders inline markdown in keys/notes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: add compat smoke coverage for secrets, working-dir defaults, runner image
Fill three audit gaps identified against packages/cli/compatibility.json:
- smoke-secrets-env-file: secrets resolved from .env.agent-ci and from
the shell environment fallback
- smoke-defaults-workdir: workflow- and job-level
defaults.run.working-directory plus a step-level override
- smoke-runner-image: .github/agent-ci.Dockerfile is picked up and the
resulting image is used at job start
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(workflow): propagate defaults.run.working-directory to steps
Parse defaults.run.working-directory at workflow and job level and merge
it into each run-step's inputs with step > job > workflow precedence.
Step-level working-directory already worked; the defaults blocks were
silently ignored.
Closes#290.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(compat): add proof field to compatibility.json rows
Every `supported`/`partial` row now lists the smoke workflows that
exercise it end-to-end. `pnpm compat:gen`/`compat:check` fails if a
listed path does not resolve, catching renames.
Rows with an empty `proof: []` are the audit gaps tracked in #292.
Refs #292.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(dtu): attach step env per-step so it does not leak to siblings
Step-level env was being aggregated into the job-wide EnvironmentVariables
map (last-wins), so a later step's `env: FOO=x` would override an earlier
step's effective FOO at runtime. Emit step.Env as the mapped step's own
`environment` instead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(compat): fill audit gaps #292 — env, expressions, local action
Three smokes covering eight of the ten gaps listed in #292:
- smoke-env-scoping: workflow + job + step `env:` precedence
- smoke-expressions: hashFiles, format, join, runner.arch, type
literals, startsWith/endsWith
- smoke-local-action: `steps[*].uses: ./.github/actions/...`
(verified via marker file rather than steps.<id>.outputs.*)
Proof fields updated for every row these smokes cover. hashFiles
downgraded to ⚠️ partial — agent-ci emits SHA-1, GitHub Actions emits
SHA-256 (tracked by #294).
Two gaps remain open as filed issues:
- #293 defaults.run.shell / steps[*].shell not applied
- #295 ${{ steps.<id>.outputs.<name> }} leaks literal text in run
scripts
Refs #292.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Docker Desktop 4.x ships with /var/run/docker.sock disabled by default,
so a fresh install hits 'agent-ci couldn't use a Docker socket' even
with Docker Desktop running. Two small fixes:
1. docs/docker-socket.md — replace the stale 'starts automatically'
line with the actual Settings → Advanced toggle steps.
2. docker-socket.ts — when the resolver throws, detect Docker Desktop's
user-side socket (~/.docker/run/docker.sock) and append a focused
hint pointing at the toggle, so users don't have to read the whole
doc to find the one paragraph that matters.
Refs #253. The underlying permission failure from #253 was already
fixed by #274 (chmod 1777 on _diag/_work); this PR addresses the
remaining onboarding paper-cut.
Single job on macos-14, workflow_dispatch-only. Confirms the
GitHub-hosted macOS runner is healthy: OS info, toolchain sanity.
CI-only change — no changeset needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When two `agent-ci-dev` runs execute in parallel, each nested CLI
process starts in its own isolated filesystem and allocates the same
sibling container name (`agent-ci-1`). The pre-spawn
`docker rm -f <name>` then kills the other nested run's sibling.
Include the outer container's hostname in the name prefix when
`/.dockerenv` is present so sibling names stay unique across
concurrent nested runs.
Also update smoke-docker-buildx.yml to look for nested-run logs under
both `/tmp/agent-ci` (GitHub-hosted runner) and
`$GITHUB_WORKSPACE/packages/cli/.agent-ci` (nested under
agent-ci-dev), since the nested working dir differs in each case.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GitHub Actions evaluates ${{ runner.os }} and ${{ runner.arch }} per-job
based on the job's runs-on label. Prior to this change we hardcoded
Linux/X64 in the expression expander regardless of runs-on, which broke
tart-backed macOS VM jobs — any step gated on `runner.os == 'macOS'`
would evaluate as false inside the VM.
Thread an optional RunnerContext through resolveExprAtom /
evaluateExprValue / expandExpressions / buildStepEnv, and derive it in
parseWorkflowSteps from `parseJobRunsOn`. Unknown/absent labels keep
the previous Linux/X64 default so existing self-hosted configurations
are unaffected. macOS is mapped to ARM64 because agent-ci's macOS
backend (tart) only runs Apple Silicon VMs.
During pnpm run version, scripts/release-closes.mjs captures
Closes|Fixes|Resolves #N references from pending changesets into
.release-closes.json, pairs each with the PR that added the
changeset, and rewrites the keywords to Refs #N so the 'Version
Packages' PR does not close them on merge. After the release
workflow publishes, a new step closes each captured issue with
the comment 'Closes Issue #N via PR #M.'
* fix(runner): chmod /home/runner/_diag and _work for Colima/Docker Desktop (#263)
On macOS with Colima or Docker Desktop the bind-mounted _diag and _work
directories surface as root:root 0755 inside the container (host perms
don't translate through the VM mount layer), so the runner user (uid
1001) crashes with UnauthorizedAccessException when writing diag logs
and hits write failures in the workspace. Add MAYBE_SUDO chmod 1777 on
both mount points during container boot, mirroring the existing fix
for /home/runner/.cache (#234). OrbStack and native Linux Docker are
unaffected — the chmod is a no-op there.
* fix(cli): require /var/run/docker.sock, link to per-provider setup doc
Today resolveDockerSocket falls back through a list of well-known
provider sockets (~/.orbstack/..., ~/.docker/..., ~/.colima/<profile>/...)
whenever /var/run/docker.sock isn't usable. On macOS + Colima, those
provider paths are not bind-mountable through the provider's VM, and
Docker rejects the later container launch with a confusing
error while creating mount source path: operation not supported
Stop guessing. Require /var/run/docker.sock to exist (or an explicit
DOCKER_HOST) and fail fast with a short error that links to the new
packages/cli/docs/docker-socket.md — which walks users through the
per-provider setup (Docker Desktop / OrbStack / Colima / native Linux
/ rootless).
The bind-mount invariant comment in docker-socket.ts was already
implicitly assuming this rule; this change just enforces it at the
resolver boundary instead of deferring the failure to container
launch, and gives users a doc URL with the recipe they need.
* feat(cli): run macos-* jobs in real macOS VMs via tart
When the host is darwin/arm64 with tart + sshpass installed, jobs
targeting runs-on: macos-* boot an ephemeral cirruslabs macOS VM,
rsync the macOS actions-runner binary in, and connect it to the
in-process DTU via the host bridge.
- New runner/macos-vm/ module (tart wrapper, image mapping, host
capability probe, runner binary fetch/cache, semaphore, job driver).
- cli.ts routes macOS jobs to the VM executor when the host supports
it; otherwise falls through to the existing skip-with-warning path.
- DTU generators.ts accepts runnerWorkDir/runnerOs/runnerArch so the
seed payload reflects where the runner actually materializes _work
(sibling of run.sh, not a caller-chosen path).
- runner-credentials.ts extracted from local-job.ts so the macOS path
can reuse the same deterministic .runner/.credentials generation.
Concurrency capped at 2 VMs by default (AGENT_CI_MACOS_VM_CONCURRENCY).
Image override via AGENT_CI_MACOS_VM_IMAGE.
Closes#258.
* feat(cli): say *why* macOS jobs skip on incapable hosts
Before: the skip warning for macos-* jobs on a host that can't run the
VM just pointed at #258 and parroted the pre-Phase-3 line about only
running in a Linux container — which is no longer accurate now that
macOS jobs *can* run locally when the host supports it.
Now: formatUnsupportedOSWarning accepts the HostCapability result and
prints the specific reason + install hint when available — e.g.
'tart not installed' with a brew command, or 'requires Apple Silicon
(got x64)'.
* refactor(cli): deduplicate runs-on classification + warning body
- cli.ts: extract classifyJob(ej) + canRunMacosHere(kind) so the
skip path and the executor-routing path share one source of truth
for 'should this job run in the macOS VM?' — the two previously
duplicated the parseJobRunsOn + classifyRunsOn + capability check.
- runs-on-compat.ts: flatten formatUnsupportedOSWarning to a single
body array per kind, filter undefined, then uniformly indent —
drops the optional-hint if-branch and the per-push indent literals.
Pure refactor; all 14 runs-on-compat tests still pass, warning output
is byte-identical across all 4 permutations (macOS with/without
capability, with/without hint, windows).
* docs: document macOS VM runner support
- README: add tart + sshpass to 'Optional' prerequisites and a new
'macOS jobs' section (image table, env vars, caveats).
- compatibility.md: unstale the three entries that claimed Linux-only
(jobs.<id>.runs-on, runner.os, runner.arch). runs-on now reads as a
partial (⚠️) that explains linux/macos/windows separately.
Leaves apps/website/CompatibilityMatrix.tsx alone — it has the same
stale rows but is in an unrelated mid-merge state on this branch.
* chore: oxfmt packages/cli/README.md and compatibility.md
Jobs with runs-on: macos-* or windows-* were silently routed to the
Linux runner container and failed at the first OS-specific step.
Parse runs-on (string, array, or {group,labels} form), classify the
OS, and skip the job with a clear stderr warning when it is not Linux.
Linux jobs are unchanged. macOS support lands later via tart (#258).
* test: smoke reproducer for #257 docker/setup-buildx-action
* chore: retrigger CI
* ci: switch smoke to pull_request so it triggers from the PR branch
* ci(smoke): dump agent-ci step logs + upload as artifact on failure
* fix: use MAYBE_SUDO to chmod bind-mounted docker socket (#257)
On native Linux Docker the socket is owned root:docker 0660. The runner
user isn't in the host's docker group, so the existing plain 'chmod 666
/var/run/docker.sock' silently failed — leaving every subsequent Docker
client call (docker/setup-buildx-action@v4, docker login, compose, buildx)
to fail with 'permission denied while trying to connect to the docker
API'. Escalating via MAYBE_SUDO matches the surrounding entrypoint ops.
* ci(smoke): assert against Verify-buildx.log, not top-level agent-ci stdout
---------
Co-authored-by: Peter Pistorius <peter@redwoodjs.com>
* Add pi-native validate skill mirroring .claude/commands/validate.md
Pi (the coding agent harness) doesn't read Claude Code's commands/
directory, so contributors using pi didn't have access to /validate.
This adds .pi/skills/validate/SKILL.md with the same background-run +
monitor + retry workflow, invokable as /skill:validate.
Tooling-only change: no code in packages/, no changeset needed.
* Symlink AGENTS.md -> CLAUDE.md
Pi and other non-Claude agent harnesses (Codex, etc.) look for
AGENTS.md. Symlinking keeps a single source of truth in CLAUDE.md
while making the same instructions discoverable to every harness.
* Rename pi validate skill to agent-ci-dev
Matches the underlying command (pnpm agent-ci-dev) and disambiguates
from the published agent-ci skill, which wraps npx @redwoodjs/agent-ci
for downstream users. Description now explicitly calls out the
distinction.
* Generalize CLAUDE.md for AGENTS.md symlink; reference agent-ci-dev pi skill
- Retitle and add a preamble noting the file is shared via AGENTS.md
symlink, so it's now the single source of truth for all harnesses.
- CI section now references both the Claude Code /validate command
and the pi /skill:agent-ci-dev skill, so pi contributors follow the
same background-monitor-retry workflow Claude Code contributors do.
* Rename Claude Code /validate to /agent-ci-dev
Aligns the Claude Code slash command name with the pi skill
(/skill:agent-ci-dev) and the underlying package command
(pnpm agent-ci-dev). Backward compat preserved via the existing
aliases: [validate] frontmatter, so /validate still resolves.
* Add pi extension: bash_background + monitor_wait tools
Pi's native bash is synchronous-only, and pi doesn't ship a Monitor
tool. Previously the agent-ci-dev skill assumed Claude Code's
run_in_background Bash + streaming Monitor, which don't exist in pi.
This adds .pi/extensions/background-shell.ts, a pi extension that
registers two tools:
- bash_background spawn a shell command; return taskId + outputFile
+ pid immediately. stdout+stderr tee'd to a log.
- monitor_wait block until a regex matches new log output, OR
the task exits, OR timeoutMs elapses. Tracks
per-task byte offsets across calls.
The skill is rewritten to drive these tools. Zero npm dependencies at
runtime (Node built-ins only, ExtensionAPI imported type-only, JSON
Schema written inline), so no npm install step is needed after clone.
Tooling-only change, no changeset required.
The workflow parser only read env: declared directly on a step.
Workflow-level and job-level env: blocks were silently ignored, so
any workflow that relied on them — including the smoke-vars-preflight
smoke test that uses a job-level env with a vars.X reference — saw
empty values at runtime.
The parser now merges env: from all three levels per the GitHub
Actions spec: workflow → job → step, step-most-specific wins.
Expressions (${{ vars.X }}, ${{ secrets.X }}, etc.) are expanded on
each value with the full context, so var/secret references resolve
identically regardless of which level declares the env.
Implementation:
* New internal buildStepEnv() helper in workflow-parser.ts that
merges the three levels and runs expandExpressions per value.
* Both the run: and uses: step branches now call it, dropping
the duplicated inline Env-construction logic.
* 5 new unit tests covering job-only, workflow-only, three-level
precedence, vars expansion in job env, and the no-env case.
Fixes the Case 3 failure in .github/workflows/smoke-vars-preflight.yml
that #270 uncovered — Case 3 had been silently passing via env leak
from the outer runner process, not via actual var substitution.
Two bugs in the smoke test, both making it fail every run since #260:
1. The header comment contained `${{ vars.FOO }}` inside backticks.
extractVarRefs() is a text-level regex scan and doesn't understand
YAML comments, so the outer preflight falsely required `--var FOO`
before the workflow would even start. Rephrase the comment to just
`vars.FOO`.
2. The heredoc that writes the inner test.yml used
`env: API_URL: ${{ vars.API_URL }}`. GitHub Actions' template engine
processes `${{ ... }}` in every `run:` script regardless of shell
heredoc quoting (`<< 'WF'` doesn't protect against it), so the
expression was being substituted at outer-run time — stripping the
vars.X references the inner agent-ci needs to exercise preflight.
Result: every Case 1 (no --var flags) run saw test.yml with no
refs, preflight passed, the workflow ran with empty env, and the
assertion against 'https://api.example.com' failed.
Fix: build the literal '${{' at shell runtime from non-triggering parts
(D='$' + "{{") so the outer template engine never sees the pattern in
source. Switch heredoc to unquoted (<<WF) so the shell variable expands,
and escape \/\ to keep them literal for the inner
job's shell.
Validated via `pnpm agent-ci-dev run --workflow smoke-vars-preflight.yml
--var API_URL=https://api.example.com --var DEPLOY_ENV=production`:
all three cases now pass (14s).
No changeset: workflow-only change.
#266 merged an edit to home.md that didn't match oxfmt's canonical
markdown table formatting, leaving `pnpm format:check` (and therefore
`pnpm check`) red on main. Run `pnpm exec oxfmt` to repair.
No changeset: apps/website is private.
Adds a 'Releasing the Website' section to RELEASING.md covering the
exact command used to deploy agent-ci.dev:
cd apps/website
CLOUDFLARE_ACCOUNT_ID=... CLOUDFLARE_ENV=production pnpm release
The env vars are required because the Wrangler OAuth user has access
to multiple Cloudflare accounts, and the deploy script runs
non-interactively.
No changeset: docs-only, does not touch packages/.
Implements the 'Markdown for Agents' agent-readiness goal for '/' without
relying on any Cloudflare dashboard feature flag.
- apps/website/src/app/pages/home.md — hand-authored markdown version of
the homepage. Same content as home.tsx (hero, principles, architecture
comparison, quick start, agent instructions, testimonials) but without
the React-only decorative bits (CRT effects, sticky nav, avatars).
Lives next to home.tsx so they stay in sync through review.
- apps/website/src/app/lib/markdown-negotiation.ts — Accept-header parser
and route middleware factory. Returns markdown only when the client
explicitly lists 'text/markdown' and its q-value beats text/html
(RFC 9110 §12.5.1). Sets Vary: Accept so caches key correctly. Bare
'*/*' is treated as browser default -> HTML, matching Cloudflare's
'Markdown for Agents' semantics.
- apps/website/src/worker.tsx — wire the middleware into route('/') via
the array form: [serveMarkdownIfPreferred(homeMarkdown), Home]. The
React component renders normally unless the middleware short-circuits.
- apps/website/types/vite.d.ts — declare *?raw module so the markdown
import type-checks.
Verified against 'npm run preview':
Accept: text/html, ... */*;q=0.8 -> text/html
Accept: text/markdown -> text/markdown; charset=utf-8
Accept: text/markdown;q=0.9, ... q=.8 -> text/markdown
Accept: text/html, text/markdown;q=.5 -> text/html
Accept: */* -> text/html
No changeset: change is entirely under apps/website/; does not touch packages/.
* kindling: start task [skip ci]
* harness: save uncommitted work from Analyst (cycle 1) [skip ci]
* Fix misleading 401/403 error hint when GitHub token is already provided
When fetching a remote workflow from a private repo with a token supplied,
a 401 or 403 from GitHub now says the token may be invalid, expired, or
lack the required 'repo' scope — instead of repeating instructions to use
--github-token that the user already followed.
Added two tests covering the with-token 401 and 403 paths, which were
previously uncovered.
Fixes#255
* harness: save uncommitted work from Developer (cycle 2) [skip ci]
* harness: save uncommitted work from Reviewer (cycle 3) [skip ci]
* harness: pre-DONE sweep [skip ci]
* kindling: ready for CI
* harness: save uncommitted work from Analyst (cycle 1) [skip ci]
* harness: save uncommitted work from Developer (cycle 2) [skip ci]
* harness: save uncommitted work from Reviewer (cycle 3) [skip ci]
* harness: save uncommitted work from Developer (cycle 4) [skip ci]
* harness: save uncommitted work from Reviewer (cycle 5) [skip ci]
* harness: pre-DONE sweep [skip ci]
* fix: handle 404 on remote workflow fetch as private-repo auth failure
GitHub's Contents API returns HTTP 404 (not 401/403) when a caller lacks
access to a private repository — this is intentional to avoid leaking repo
existence. The previous error hint only covered 401/403, so users hitting
the most common private-repo failure modes (missing token, classic PAT
without 'repo' scope, fine-grained PAT without 'contents: read', unauthorized
SSO PAT) saw a bare "HTTP 404" with no guidance.
Unify hint generation into buildAuthHint() covering 401/403/404 × with/
without token. Hints now include explicit gh auth login instructions and
spell out the token-insufficient scenarios.
Verified end-to-end against a private repro repo (peterp/agent-ci-private):
no-token → 404 path now emits full auth guidance, invalid-token → 401 path
emits token-insufficient hint, valid-token → resolves.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: smoke test for private-repo remote workflow fetch (#261)
Adds .github/workflows/smoke-remote-private-workflow.yml — the end-to-end
verification workflow used to reproduce and prove the #261 fix, now preserved
as a regression-guard smoke test.
It references a reusable workflow in a private repo
(peterp/agent-ci-private/.github/workflows/lint.yml@main). Under agent-ci this
exercises prefetchRemoteWorkflows against the GitHub Contents API with the
supplied token, covering the success path (HTTP 200 from a private repo when
a valid token is attached) that the unit tests in
packages/cli/src/workflow/remote-workflow-fetch.test.ts don't cover.
Verified locally with 'pnpm agent-ci-dev run --workflow … --github-token':
Status: ✓ 2 passed (2 total)
Duration: 9s
Without a token, the same smoke surfaces the improved #261 hint chain verbatim
(HTTP 404 → 'The repository or ref was not found …' → auth instructions).
---------
Co-authored-by: kindling <kindling@harness>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address agent-readiness findings from isitagentready.com:
- apps/website/public/robots.txt — RFC 9309 crawl rules with explicit
entries for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web,
anthropic-ai, Google-Extended, PerplexityBot, Applebot-Extended, CCBot,
Meta-ExternalAgent, Bytespider, DuckAssistBot (and standard search bots),
a Sitemap reference, and Content Signals
(search=yes, ai-input=yes, ai-train=no).
- apps/website/src/app/pages/sitemap.ts — dynamic /sitemap.xml handler
registered outside render(Document, ...) so it returns application/xml.
Enumerates static routes, the copied CLI docs, and blog posts from
content-collections with lastmod.
- apps/website/src/app/headers.ts — adds a RFC 8288 Link response header
on all routes: sitemap, agent-skills (extension), service-doc.
- apps/website/scripts/copy-docs.mjs — sync script that copies the public
CLI docs (README.md, compatibility.md, runner-image.md, SKILL.md) into
apps/website/public/docs/ and regenerates
apps/website/public/.well-known/agent-skills/index.json with fresh
sha256 digests. Hooked into predev/prebuild so the website's static
tree cannot drift from the CLI.
The generated public/docs/ and public/.well-known/agent-skills/ trees are
gitignored.
Verified end-to-end against 'npm run preview': all endpoints return the
expected status, content-type, and body, and the Link header appears on
both HTML and XML responses.
No changeset: this change is entirely under apps/website/ and does not
affect published packages (per .claude/rules/changeset-required.md).
github-actions[bot]