Replace mutable tag refs (@v4, @v5, @v1, @main) with 40-char commit
SHAs and a trailing `# vX.Y.Z` comment, per GitHub's hardening guidance.
Covers .github/workflows/, .docs/marketing/demo-recording/, and the
skill-evals fixture repos. Embedded YAML inside heredocs is also pinned.
No packages/ code changes, so no changeset.
#351 bumped the published packages' engines.node to >=24 along with
the dev-side switch to Node's native TypeScript support. End users
never run our .ts source files, only the compiled dist/cli.js, which
targets ES2020 and works fine on Node 22.
- @redwoodjs/agent-ci and dtu-github-actions: engines.node >=22
- repo-root package.json: gain engines.node >=24 to signal that the
dev scripts (which run .ts files directly) need Node 24.
The first thing `agent-ci run` does is ask git for several pieces of
information: the current branch, the head commit SHA, the changed
files, the remote slug, and (when the tree is dirty) an ephemeral
commit that captures the working-tree state. Each call shelled out
through execSync, blocking the event loop for ~50-200ms.
This change converts each of those helpers to use execFile via
promisify, so they return promises. handleWorkflow then runs the
startup git/remote calls concurrently via Promise.all instead of one
at a time.
Functions converted:
- getFirstRemoteUrl, resolveRepoSlug (config.ts)
- computeDirtySha (runner/dirty-sha.ts)
- getChangedFiles (workflow/workflow-parser.ts)
- resolveHeadSha, resolveBaseSha, persistRunResult (commands/run.ts)
Switching from execSync(command-string) to execFile("git", [args])
also removes a shell escaping step on every call — args are passed
as an array.
Refs #334.
Verified:
- pnpm check: clean
- Tests: 754/754 pass (vitest run, excluding the npx-affected
shutdown.test.ts which is a separate pre-existing issue)
- Smoke suite (pnpm agent-ci-dev run --all): 45/45 jobs pass after
the known nested-secrets retry
* chore: require Node 24 and drop tsx
Node 24 ships native TypeScript stripping as a stable feature. We no
longer need tsx to execute .ts files in development scripts.
Changes:
- `engines.node` bumped from `>=22` to `>=24` in each package.
- `tsx` removed from every workspace's devDependencies.
- Each `tsx foo.ts` invocation in package scripts becomes `node foo.ts`.
Includes `scripts/agent-ci-dev.sh` (the dev wrapper) and the demo
package.
- tsconfigs gain `allowImportingTsExtensions`, `rewriteRelativeImportExtensions`,
and `verbatimModuleSyntax` so source uses `.ts` extensions while tsc
still emits `.js` paths in dist.
- All relative imports rewritten from `from "./foo.js"` to `from "./foo.ts"`
across 72 source files. Dynamic imports updated too.
- Type-only imports (`Job`, `Polka`, MessageResponse, …) converted to
`import type` to satisfy `verbatimModuleSyntax`.
- The json-loader hook registration in workflow-parser.ts picks `.ts`
when the source file exists (dev) and `.js` otherwise (compiled dist).
- Root `package.json` adds `"type": "module"` (no existing .js files
affected) and `yaml` as a devDep (already used by scripts/diff-parse.ts;
previously hoisted through cli's deps).
- `tests.yml` CI bumps from Node 22 to Node 24. Smoke fixtures that set
`node-version: 22` are left alone — they exercise `actions/setup-node`
with various versions deliberately.
Verified:
- pnpm check: clean
- Node 24 dev run: `node packages/cli/src/cli.ts --help` works
- Full smoke suite (`pnpm agent-ci-dev run --all`): 45/45 jobs pass
* fix(shutdown.test): use `node --input-type=module` instead of `npx tsx`
The shutdown integration test spawns a fresh process to sidestep
vitest's mock system. It was using `npx tsx`, but `tsx` is no longer
a dependency. Node 24 reads TypeScript files natively, so we ask Node
directly via `--input-type=module`.
executeLocalJob had a ~190-line updateStoreFromTimeline closure that
ran every 100ms during a job, reading timeline.json and the paused-
signal file and updating the RunStateStore. It captured six mutable
let-variables defined just above it. fallow flagged it as cognitive
70 — the biggest remaining hotspot after the previous round.
Lift it to module scope as two helpers:
- syncTimelineToStore(state, ctx) — drives one poll tick (cognitive 22)
- buildStepsFromTimeline(steps, state) — folds the raw records into
the StepState[] shape the renderer expects (cognitive 45)
The mutable state moves into an explicit TimelineSyncState object the
polling loop and the rest of executeLocalJob now share. A read-only
TimelineSyncContext carries the paths, store reference, and an
`onNewPause` callback to wire up stdin-retry.
Also drops `padW` / `totalSteps` from the old closure — they were
computed but never used (legacy padding logic).
Smoke suite: 45/45 pass with zero retries.
fallow flagged three runtime deps that are declared but never imported:
- log-update in packages/cli (diff-renderer replaced it; only stale
code comments remain)
- jsonc-parser in packages/dtu-github-actions
- yaml in packages/ts-runner (cli still uses yaml; ts-runner does not)
Smaller node_modules, smaller published tarballs, one fewer thing to
keep up to date.
Before:
- runCmd ~228 lines, cognitive 91
- handleWorkflow ~717 lines, cognitive 136
After:
- runCmd cognitive 9
- handleWorkflow cognitive 61
- new helpers split out: parseRunArgs (and friends), discoverRelevantWorkflows,
resolveWorkflowArgPath, finalizeRun, expandJobs, classifyJobsResources,
runWaveJobs
ExpandedJob is lifted to module scope so the new helpers can take it.
No runtime behaviour change. `pnpm check` clean. Full smoke suite
(`pnpm agent-ci-dev run --all`) passes: 45 of 45 jobs.
Six identifiers were exported but used only inside their own file:
- `log-prune.ts`: DEFAULT_RETAIN_DAYS, DEFAULT_RETAIN_RUNS,
DEFAULT_THROTTLE_MS (constants used only by `pruneLogs`).
- `generators.ts`: toContextData, toTemplateTokenMapping (helpers
called only by `createJobResponse` and friends in the same file).
Plus one wholly-unused function — `toContainerTemplateToken` in
`generators.ts` — that nothing called internally or externally.
It was leftover utility code from an earlier design; deleted.
No runtime behaviour change. fallow's unused-exports report drops
by six items as a result.
executeLocalJob was ~860 lines, cognitive 73 (the top complexity hotspot
in the cli package after the previous refactor round). Three self-contained
blocks moved to module-scope helpers:
- pullContainerImageWithProgress (the ~100-line dockerode pull with
per-layer progress reporting)
- seedRunnerBinaryToHost (one-time extraction of the actions-runner binary
from SEED_IMAGE in direct-container mode)
- waitForContainerExit (Promise.race with timeout, force-stop on timeout)
executeLocalJob is now ~715 lines, cognitive 56. The full smoke suite
passes 45/45.
The default fallow scan flagged 85 issues, ~57 of which were false
positives caused by code paths it can't see statically:
- `apps/website/**` is discovered dynamically by rwsdk's route convention.
- `experiments/**` and `.github/fixtures/**` are not workspaces and only
loaded at run time by test fixtures.
- `packages/cli/src/hooks/json-loader.ts` is registered through Node's
ESM loader hook from workflow-parser.ts.
After the config, the report drops to 28 real issues, which the team
can act on instead of skimming past noise.
Docs/tooling-only change; no published packages touched, no changeset.
Replaces the eight optional arguments threaded through `resolveExprAtom` and
`evaluateExprValue` with a single `ExprContext` object, dispatches built-in
functions through a name->handler table, and moves context-variable lookups
into a dedicated resolver with constant tables.
`expandExpressions` keeps its positional public signature so no caller has
to change. No behaviour change; all workflow-parser tests still pass and
the full smoke suite (`pnpm agent-ci-dev run --all`) is green.
Measured complexity drop (fallow health):
- resolveExprAtom: cognitive 124 -> off the high-complexity list
- evaluateExprValue: cognitive 57 -> 28
- file length: 1913 -> 1736 lines
Also adds `fallow` as a workspace-root dev dependency and ignores the
locally generated `.fallow-reports/` directory.
Extracts `run`, `retry`/`abort`, and `clean` into separate modules loaded via
`await import()` from `cli.ts`. The dispatcher itself now only loads what the
invoked command actually needs.
Measured impact on `agent-ci --help`:
- Cold start: 240 ms → 20 ms
- Peak RSS: 88 MB → 42 MB
`--help` / unknown commands no longer load dockerode, @grpc/grpc-js, protobufjs,
ssh2, the runner graph, or the workflow parser. Every command behaves the same
otherwise. `--help`/`-h` now exits 0 (previously 1 — it had been falling through
the else branch).
Refs #334.
On cold boot the VM hasn't grabbed a DHCP lease, so `tart ip` hangs until
runCommand's 5s timeout fires and rejects. getIp didn't catch — that
rejection killed waitForIp on its very first iteration, well before its
90s budget. Wrap the call in `.catch(() => null)` so transient timeouts
look the same as a non-zero exit and waitForIp keeps polling.
Closes#329.
Update the agent-facing skill surfaces to match the recent pause-on-failure
fixes and `--json` event stream:
- skills/agent-ci/SKILL.md (published) and packages/cli/SKILL.md: document
`--json` / `AGENT_CI_JSON=1`, list the NDJSON event types, and note that
pipes/redirects are safe — pause-on-failure exits 77 in non-TTY mode.
- .claude/commands/agent-ci-dev.md and .pi/skills/agent-ci-dev/SKILL.md:
switch the monitor pattern from grepping plaintext "Step failed" to
matching `"event":"(run.paused|run.finish)"`, and call out exit-77 as a
pause signal alongside the `run.paused` event.
- experiments/skill-evals/variants/v2-run-first.md and v3-no-mangling.md:
remove stale "no pipes / no redirects" warnings — those scenarios are now
the supported path via the detached launcher.
Refs #289, #315.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Passing the raw URI as `host` to dockerode left docker-modem with
`host = "ssh://user@remote-host"`, which ssh2 then tried to DNS-resolve,
failing with `getaddrinfo ENOTFOUND` (or `Invalid username` if no user
was configured elsewhere). Parse the URI and pass `host`, `username`,
`port`, and `sshOptions.agent` separately — matching what docker-modem
does internally when it parses `DOCKER_HOST` itself.
Adds an end-to-end test that asserts on the constructed Docker client's
modem state (`docker.modem.host === "remote-host"`), reproducing the bug
without mocking dockerode.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move per-run log artifacts out of the runner's tmp working dir into a
stable, agent-ci-owned directory next to the per-branch checks JSON
(default `<stateDir>/logs/`, override via AGENT_CI_LOG_DIR). Log paths
recorded in the run-result JSON now survive OS-level pruning of
`os.tmpdir()`.
Add an opportunistic, throttled cleanup at the start of `agent-ci run`
plus an explicit `agent-ci clean` subcommand. Policy: never delete a
run dir referenced by any branch's checks JSON, then keep newest N
runs, then drop runs older than M days. Knobs:
AGENT_CI_LOG_RETAIN_DAYS=7, AGENT_CI_LOG_RETAIN_RUNS=20,
AGENT_CI_LOG_PRUNE=0 to disable. O_EXCL lockfile + last-prune stamp
guard against concurrent runs and per-job fan-out.
Also fix `buildStepEnv` in workflow-parser.ts to thread an envContext
through `expandExpressions`, completing PR #320's env.* expansion
support — `${{ env.JOB_KEY }}` inside another step's `env:` value now
resolves to the workflow- or job-level value instead of an empty
string. This made smoke-env-context.yml pass under agent-ci-dev.
Closes#312.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): unblock --pause-on-failure in non-TTY pipes (#315)
Spawn the run as a detached worker when stdout isn't a TTY (and we're
not under an LLM harness), then exit 77 the moment the worker emits a
`run.paused` NDJSON event — freeing the caller's pipe while the worker
keeps the container + DTU alive for `agent-ci retry`. Retry tails the
same log so a re-failure surfaces as another exit-77.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): NDJSON event stream behind --json (#289)
Extends the existing pause/finish sentinel emitter into a full lifecycle
event stream. New `--json` flag (and `AGENT_CI_JSON=1` env var) emits
`run.start`, `job.start`/`finish`, `step.start`/`finish`, `run.paused`,
`run.finish`, `diagnostic` events on stdout — one JSON object per line —
each carrying an `event` discriminator and a `schemaVersion: 1` on
`run.start`.
`--json` is decoupled from `--quiet` so existing `-q` callers keep silent
stdout; the diff renderer is auto-suppressed under `--json` so ANSI
sequences don't collide with the stream. Detached-worker mode (#315)
keeps emitting only sentinels — the launcher filters non-sentinel events
anyway, so non-JSON callers don't pay the cost.
Renames `run.completed` -> `run.finish` to match the issue's naming.
Closes#289.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): suppress printSummary under --json
The ━━━ FAILURES ━━━ / ━━━ SUMMARY ━━━ blocks write to stdout
unconditionally, which interleaves with the NDJSON event stream and
breaks line-based parsers. Skip them when isJsonMode() — consumers get
the same information from job.finish + run.finish events.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): AGENT_CI_DETACHED=1 forces detached launcher on TTY
Overload AGENT_CI_DETACHED so it can serve both roles:
- `=<absolute path>` → set by the launcher on its worker child;
`isDetachedWorker()` returns true.
- `=1` (or any non-absolute value) → caller-side opt-in: force the
launcher path even when stdout is a TTY. Useful for manual
verification of the #315 pause-then-exit-77 flow without having to
construct a non-TTY pipe.
The two are distinguished by whether the value starts with `/`. New
`isForceDetachedRequested()` helper drives a `forceDetached` arg into
`shouldLaunchDetached`, which still respects `alreadyWorker`,
`agentMode`, and the `pauseOnFailure` requirement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(ci): smoke for --json NDJSON event stream (#289)
scripts/smoke-289.sh creates a tmp repo with a 2-step passing workflow,
runs `agent-ci run --json`, and asserts:
- run.start carries schemaVersion=1
- step.start + step.finish for each declared step
- job.finish status=passed
- run.finish status=passed
- every JSON-shaped line on stdout has a recognized event field
.github/workflows/smoke-json-events.yml runs the script in CI on every
PR (gated by the same safe-to-run / collaborator predicate as the other
smokes).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ci): stop smoke-resource-mismatch from hanging on real GitHub runners
The fixture in .github/workflows/smoke-resource-mismatch.yml declares
`runs-on: ubuntu-latest-999-cores` so agent-ci's resource-fidelity
classifier deterministically tags the job as `degraded` on any host.
With its previous `pull_request_target` trigger, GitHub Actions also
picked it up on every PR and queued a job that no real runner can
satisfy — five queued runs piled up on PR #323 alone, each timing out
after hours.
Switch the workflow to `workflow_dispatch:`-only so GitHub never
auto-dispatches it, and teach `isWorkflowRelevant` to treat
`workflow_dispatch:`-only workflows as relevant for `agent-ci run --all`
so the local smoke keeps running. The opt-in is gated to the sole-event
case to avoid overriding `pull_request` / `push` path/branch filters on
workflows that legitimately combine triggers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(workflow): support expansion of env context references in expressions
`${{ env.* }}` references in workflow expressions silently resolved to empty
string because `resolveExprAtom` had no `env.` handler and no env context was
threaded through the evaluator. The fix adds initial support for expanding these
references in `run:` scripts, `with:` inputs, `name:`, and `env:` value
expressions — matching GitHub Actions behavior. `if:` conditions remain
unsupported (noted in `compatibility.md`).
* chore(smoke): add smoke-env-context.yml as proof for env.* row
Adds a smoke workflow that exercises ${{ env.* }} expansion in `run:`
bodies, in another step's `env:` value, in `with:` inputs (the original
Rails CI bug surface), and verifies the unknown-key → empty-string
contract. Also wires it as the `proof` reference on the env.* row in
compatibility.json.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Peter Pistorius <peter.pistorius@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): surface degraded local runs for resource mismatches
Add resource fidelity classification that compares local host resources
against workflow-declared runner/resource hints. When the local machine
is smaller than what the workflow requests, jobs are marked as degraded
and a warning is shown before execution starts.
Features:
- Parse runs-on labels and NODE_OPTIONS from workflows
- Classify jobs as faithful or degraded based on CPU/memory hints
- Show [degraded] tag and actionable warnings in CLI output
- Add smoke test for degraded classification
- Update documentation with degraded mode guidance
Complements PR #226 which adds memory-aware concurrency limiting.
Together they replace the approach in closed PR #78.
* refactor(cli): use runner specs instead of NODE_OPTIONS for resource classification
Replace NODE_OPTIONS parsing with GitHub runner label heuristics.
This is more accurate and works for all workloads (not just Node).
Changes:
- Add RUNNER_SPECS mapping for known GitHub runner sizes
- Add parseRunnerSpecs() to look up full {cpu, memory} from labels
- Remove parseNodeMaxOldSpaceMb() and NODE_OPTIONS dependency
- Update tests to use runner specs
- Simplify smoke test fixture
Runner specs used:
- ubuntu-latest: 2 CPU, 7GB RAM
- ubuntu-latest-4-cores: 4 CPU, 16GB RAM
- ubuntu-latest-8-cores: 8 CPU, 32GB RAM
- ubuntu-latest-16-cores: 16 CPU, 64GB RAM
* feat(cli): warn when using unknown runner labels
When a workflow uses an unknown runner label (e.g., 'self-hosted' or custom
labels not in RUNNER_SPECS), the job is now marked as degraded with a
warning that explains we're assuming local host capacity.
This makes the behavior transparent - users know we're making an assumption
about the runner size rather than silently defaulting to faithful mode.
* Address PR review comments: remove smoke tests, resource warnings, and writeStdoutLine refactor
* Remove remote Docker daemon suggestion from degraded resource warnings
* chore(compat): add resource-mismatch smoke and document degraded mode
- Add smoke-resource-mismatch.yml fixture that uses ubuntu-latest-999-cores
to deterministically trigger the degraded classification on any host.
- Extend the runs-on row in compatibility.json with a sentence describing
the degraded warning and reference the smoke as proof.
- Regenerate compatibility.md.
- Add changeset.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(classifier): only flag unknown labels for the ubuntu family
The "unknown runner label" reason fired for any label missing from
RUNNER_SPECS, including macos-* / windows-* / self-hosted — labels that
runs-on-compat already handles and that the resource classifier has no
useful opinion on. Restrict the warning to labels that look like a
hosted ubuntu variant (`ubuntu` or `ubuntu-*`), which is the family
RUNNER_SPECS actually covers; this still catches drift when GitHub
adds a new ubuntu label without us noticing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(smoke): de-flake bun smoke by pre-fetching the binary
The smoke depended on oven-sh/setup-bun@v2 pulling bun-linux-aarch64.zip
from github.com/oven-sh/bun/releases inside the nested runner. The
upstream action has no per-request timeout or retry, so a stalled CDN
socket presented as a hung step with one log line ("Downloading a new
version of Bun: …") and nothing else — pure flake, no signal about
agent-ci.
Pre-fetch the binary on the outer host with curl --retry, ship it into
the inner workspace as a fixture, and replace the setup-bun call in the
inner workflow with two narrower steps that exercise the actual #212
failure modes: an actions/cache@v4 probe against a randomized `bun-…`
key (must miss — guards against the virtualCachePatterns substring
fake-hit) and a manual `mkdir -p $HOME/.bun/bin` install (exercises the
bind-mount permissions path).
Local run: 16s, deterministic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Peter Pistorius <peter.pistorius@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Revert "feat(runner): smolvm backend foundation (WIP, blocked on smolvm#177) (#287)"
This reverts commit 813036b543.
* chore: add changeset for smolvm backend revert
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds experiments/skill-evals/, a matrix eval that runs headless Claude Code
against broken-repo fixtures with candidate SKILL.md wordings appended to
CLAUDE.md, scores tool calls + file edits against per-fixture rubric regexes,
and aggregates across N trials per (variant, fixture) cell.
Includes:
- run.mjs + rescore.mjs (live eval + replay saved transcripts against a
changed rubric without re-running the agent)
- 3 fixtures: lint-curly-x18, missing-changeset, noisy-failure
- 3 variants: v1-current (baseline), v2-run-first, v3-no-mangling
- Stub @redwoodjs/agent-ci per fixture — no Docker required for the eval
Fixture repos are deliberately broken (syntax errors, missing changesets,
etc.); .prettierignore and .oxlintrc ignorePatterns exclude them so the
pre-commit hooks stay green.
Scoped to experiments/, no changes to published packages, no changeset.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(runner): add opt-in smolvm backend foundation
WIP — blocked on smol-machines/smolvm#177 (TSI idle keep-alive drop).
Pure-shell + checkout workflows run end-to-end; anything using
actions/setup-* deadlocks once libkrun kills the idle listener
connection. Foundation (CLI wrapper, host-capability probe, image
mapping, pack cache, dispatch flag, workspace + shim mounts) is in
place and useful regardless of the networking resolution.
Refs #284.
* chore(runner): bump smolvm CLI-surface marker to 0.5.20
Verified end-to-end against smolvm 0.5.20: the previously-deadlocking
actions/setup-node@v4 smoke now passes in ~3 min, confirming the TSI
idle keep-alive fix (smolvm PR #184 / libkrun abd07c8) lands the
closure for smol-machines/smolvm#177.
The three 0.5.19-observed workaround comments in smolvm-job.ts are
left intact — SQLite single-writer, repeated-pull DNS drop, and
virtiofs small-file hang were not mentioned in the 0.5.20 release
notes and were not re-tested here.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
parseWorkflowSteps never extracted `if:` from the workflow, so the DTU
server fell back to `condition: "success()"` for every step. Custom
gates — `contains(...)`, `always()`, even literal `${{ false }}` — were
silently ignored and every step ran regardless. A new `parseStepIf`
helper strips the optional `${{ }}` wrapper and threads the naked
expression through to the runner's EvaluateStepIf, so step gates now
behave as they do on real GitHub Actions. Adds unit coverage for the
helper and a smoke workflow that fails loudly if any gate leaks.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): persist latest run result per worktree to a stable JSON path
Lets external consumers (tmux panes, status bars, editor integrations) read
the current branch's CI status without re-running agent-ci or scraping human
output. Written atomically after every `agent-ci run` / `run --all`, keyed by
`<branch>.<worktree-hash>.json` under `<org>/<repo>/` so two worktrees on the
same branch don't stomp each other. Includes `headSha` so consumers can detect
stale results themselves.
Closes#288
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): include steps[] and per-step log paths in run-result JSON
Each job entry in the persisted run-result JSON now carries:
- `debugLogPath` for the whole job
- `failingStepLogPath` when a step failed
- `steps[]` with name, status, and `logPath` for each step
Path fields are only emitted when the file still exists at write time,
so consumers get an honest view after passing runs clean up their logs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(cli): drop job-level failingStepLogPath from run-result JSON
The failing step's log path is already discoverable in `steps[]` via the
per-step `logPath` on the entry whose `status == "failed"`, so the job-level
field is redundant. Keep `failingStep` (the name) as a top-level convenience.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Target tmux panes by captured pane ID instead of window:pane indices so
the script works regardless of the user's tmux base-index. Invoke
scripts/agent-ci-dev.sh directly instead of going through
pnpm --dir "$REPO_ROOT", so the CLI inherits DEMO_DIR as its cwd and
resolves the demo's own .github/workflows/ci.yml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DOCKER_HOST collides with the shell's Docker CLI expectation and can't
live in .env.agent-ci without leaking to every docker command. Rename
to AGENT_CI_DOCKER_HOST and hard-exit if the legacy DOCKER_HOST is set.
At bootstrap, apply AGENT_CI_*-prefixed keys from .env.agent-ci to
process.env so Docker/network config persists without shell exports.
Closes#308.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#297. CI/scripts-only — no packages/ change, so no changeset required
per .claude/rules/changeset-required.md.
- scripts/diff-parse.ts: walks raw YAML at workflow + job scopes (in addition
to the existing step scope) and grades every recognized key against a
preservation predicate or a documented expected-drop. Reuses focused parsers
(parseJobIf, parseJobOutputDefs, parseWorkflowContainer, parseWorkflowServices,
parseMatrixDef, parseFailFast, parseJobDependencies, getWorkflowTemplate).
defaults.run.shell preservation detects the heredoc-wrap marker emitted by
wrapScriptForShell.
- third-party-workflows/: vendored scan-only corpus — turborepo-lint.yml,
playwright-infra.yml, docker-validate.yml — each with attribution header.
Picked up automatically by collectWorkflowFiles.
- package.json: parse:diff added to the parallel pnpm check pipeline so
regressions fail standard CI.
Result: 37 workflows scanned (.github/workflows + third-party-workflows),
0 drifts. pnpm check exits 0; pnpm agent-ci-dev run --all reports 39/39.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each overloaded row becomes one row per documented feature (or a tighter
note when the sub-features behave identically). No behaviour changes.
* github.* split into sha / repository family / everything-else
* runner.* new row for unsupported siblings
* contains etc. three rows, one per function
* success family downgraded partial (cancelled() always false locally)
* on (other) note enumerates ~20 events
Closes the "overloaded row" bucket on #296.
Refs #296.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds rows for features GitHub Actions documents that our compat table
said nothing about. Each status was picked per code inspection of the
existing parser / runner paths — no behaviour changes.
Closes the "missing rows" bucket on #296.
Refs #296.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The YAML parser extracted `jobs.<id>.container.options` into job.container
but the docker createContainer call never consumed it, so `options: --env
FOO=bar` silently produced a container without FOO.
parseContainerOptions now pulls --env/-e and --label/-l out of the options
string and merges them into the container's Env array and Labels map. All
other Docker flags in options: are still ignored by design — --privileged,
--user, --network, --cap-add, --workdir tend to clash with agent-ci's own
container orchestration, and silently dropping them is safer than a partial
honoring that breaks the runner's invariants. Documented in the
compatibility.json note for jobs.<id>.container.
Also updated two nearby notes while running through #296:
* actions/cache — documents the lack of ref-based isolation (branches
share cache keys locally, unlike GitHub's per-ref scoping)
* GITHUB_TOKEN — documents that OIDC id-token issuance is not wired up,
so permissions: id-token: write → federated cloud auth does not work
Refs #296.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two small expression-engine fixes surfaced by extending smoke-expressions
with assertions for #296's "questionable claim" rows:
1. toJSON now emits 2-space-indented JSON (was compact), so hashFiles
keys that hash a toJSON(x) match GitHub. Parses rawValue first so
toJSON(fromJSON(x)) round-trips cleanly.
2. compareValues() no longer bails out of the numeric path when either
side is empty. `''`, `null`, and numeric strings all coerce to
numbers per the spec; non-numeric strings become NaN (string compare
path, always false).
Five new assertions (contains-on-array, toJSON pretty-print, coercion,
hashFiles multi-pattern, hashFiles negation) all pass. hashFiles
multi-pattern and negation were never broken — cleared off #296 with
proof rather than a fix.
Refs #296.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(workflow): hashFiles descends into dotted directories
findFiles was skipping any directory starting with `.`, so a pattern
like hashFiles('.github/workflows/*.yml') never matched a file and
returned the 40-zero placeholder. Skip only node_modules now.
The smoke assertion is tightened back to 64 chars (SHA-256) since the
real digest now appears.
Closes#294.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(workflow): resolve steps.*.outputs.* to empty at parse time
The parser was returning a sentinel so that `${{ steps.X.outputs.Y }}`
expressions would be preserved verbatim for the runner to evaluate at
runtime. The runner does not re-evaluate expressions inside run-script
bodies, so the literal `${{ }}` reached bash as "bad substitution".
Per the long-standing compatibility contract, these expressions now
resolve to an empty string at parse time. Cross-job values still work
through needs.*.outputs.*.
Closes#295.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The runner executes every run: step with bash regardless of inputs.shell,
so setting shell: sh / python / pwsh had no effect. Wrap the expanded
script at parse time — `<interpreter> <flags> <<'EOF' ... EOF` — so the
requested shell actually runs the user's code. Workflow, job, and step
scopes compose via the existing resolveStepRunDefault helper.
Adds smoke-shell-defaults.yml (removed in an earlier bug-filing cut)
covering all three scopes with a BASH_VERSION assertion.
Closes#293.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
For each .github/workflows/*.yml, read the raw YAML and compare against
what parseWorkflowSteps emits, flagging any step-level key our parser
silently drops without a documented expected-drop policy.
First run over all 26 workflows: 0 drifts. Job-level and workflow-level
scans are tracked as follow-ups in #297.
Refs #297.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(compat): make compatibility.json the source of truth
Flip the compatibility matrix so packages/cli/compatibility.json is
authoritative. scripts/gen-compatibility-md.mjs renders the .md from it,
pnpm check runs a drift check, and the website imports the JSON directly
and renders inline markdown in keys/notes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: add compat smoke coverage for secrets, working-dir defaults, runner image
Fill three audit gaps identified against packages/cli/compatibility.json:
- smoke-secrets-env-file: secrets resolved from .env.agent-ci and from
the shell environment fallback
- smoke-defaults-workdir: workflow- and job-level
defaults.run.working-directory plus a step-level override
- smoke-runner-image: .github/agent-ci.Dockerfile is picked up and the
resulting image is used at job start
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(workflow): propagate defaults.run.working-directory to steps
Parse defaults.run.working-directory at workflow and job level and merge
it into each run-step's inputs with step > job > workflow precedence.
Step-level working-directory already worked; the defaults blocks were
silently ignored.
Closes#290.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(compat): add proof field to compatibility.json rows
Every `supported`/`partial` row now lists the smoke workflows that
exercise it end-to-end. `pnpm compat:gen`/`compat:check` fails if a
listed path does not resolve, catching renames.
Rows with an empty `proof: []` are the audit gaps tracked in #292.
Refs #292.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(dtu): attach step env per-step so it does not leak to siblings
Step-level env was being aggregated into the job-wide EnvironmentVariables
map (last-wins), so a later step's `env: FOO=x` would override an earlier
step's effective FOO at runtime. Emit step.Env as the mapped step's own
`environment` instead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(compat): fill audit gaps #292 — env, expressions, local action
Three smokes covering eight of the ten gaps listed in #292:
- smoke-env-scoping: workflow + job + step `env:` precedence
- smoke-expressions: hashFiles, format, join, runner.arch, type
literals, startsWith/endsWith
- smoke-local-action: `steps[*].uses: ./.github/actions/...`
(verified via marker file rather than steps.<id>.outputs.*)
Proof fields updated for every row these smokes cover. hashFiles
downgraded to ⚠️ partial — agent-ci emits SHA-1, GitHub Actions emits
SHA-256 (tracked by #294).
Two gaps remain open as filed issues:
- #293 defaults.run.shell / steps[*].shell not applied
- #295 ${{ steps.<id>.outputs.<name> }} leaks literal text in run
scripts
Refs #292.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Docker Desktop 4.x ships with /var/run/docker.sock disabled by default,
so a fresh install hits 'agent-ci couldn't use a Docker socket' even
with Docker Desktop running. Two small fixes:
1. docs/docker-socket.md — replace the stale 'starts automatically'
line with the actual Settings → Advanced toggle steps.
2. docker-socket.ts — when the resolver throws, detect Docker Desktop's
user-side socket (~/.docker/run/docker.sock) and append a focused
hint pointing at the toggle, so users don't have to read the whole
doc to find the one paragraph that matters.
Refs #253. The underlying permission failure from #253 was already
fixed by #274 (chmod 1777 on _diag/_work); this PR addresses the
remaining onboarding paper-cut.
Single job on macos-14, workflow_dispatch-only. Confirms the
GitHub-hosted macOS runner is healthy: OS info, toolchain sanity.
CI-only change — no changeset needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
github-actions[bot]