Update the agent-facing skill surfaces to match the recent pause-on-failure
fixes and `--json` event stream:
- skills/agent-ci/SKILL.md (published) and packages/cli/SKILL.md: document
`--json` / `AGENT_CI_JSON=1`, list the NDJSON event types, and note that
pipes/redirects are safe — pause-on-failure exits 77 in non-TTY mode.
- .claude/commands/agent-ci-dev.md and .pi/skills/agent-ci-dev/SKILL.md:
switch the monitor pattern from grepping plaintext "Step failed" to
matching `"event":"(run.paused|run.finish)"`, and call out exit-77 as a
pause signal alongside the `run.paused` event.
- experiments/skill-evals/variants/v2-run-first.md and v3-no-mangling.md:
remove stale "no pipes / no redirects" warnings — those scenarios are now
the supported path via the detached launcher.
Refs #289, #315.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds experiments/skill-evals/, a matrix eval that runs headless Claude Code
against broken-repo fixtures with candidate SKILL.md wordings appended to
CLAUDE.md, scores tool calls + file edits against per-fixture rubric regexes,
and aggregates across N trials per (variant, fixture) cell.
Includes:
- run.mjs + rescore.mjs (live eval + replay saved transcripts against a
changed rubric without re-running the agent)
- 3 fixtures: lint-curly-x18, missing-changeset, noisy-failure
- 3 variants: v1-current (baseline), v2-run-first, v3-no-mangling
- Stub @redwoodjs/agent-ci per fixture — no Docker required for the eval
Fixture repos are deliberately broken (syntax errors, missing changesets,
etc.); .prettierignore and .oxlintrc ignorePatterns exclude them so the
pre-commit hooks stay green.
Scoped to experiments/, no changes to published packages, no changeset.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces the packages/ts-runner package skeleton with build config,
planning docs (PLAN.md, SPEC.md), and a research document exploring
RivetKit Agent OS integration. No runtime code yet — this sets up the
foundation for subsequent PRs.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Rename package from 'machinen' to 'agent-ci' across all source, tests, scripts, configs, and docs
- Rename CLI binary from 'machinen' to 'agent-ci'
- Rename .machinen → .agent-ci, .env.machinen → .env.agent-ci, MACHINEN_* → AGENT_CI_*
- Rename --runner flag to --name/-n (--runner still accepted for compat)
- agent-ci-dev script now builds DTU before running CLI
- Extract YAML compatibility tables to cli/compatibility.md
- Expand CLI usage docs with per-command flag tables
- Document DOCKER_HOST support for remote Docker
When a step pauses on failure, the tree UI now shows:
- Last 20 lines of step output below the tree (assertion errors,
code context, etc.) matching the end-of-job summary quality
- Press Enter to retry directly from the paused terminal
- Ctrl+C to abort
The step log is resolved via sanitized name with fallback to the
most recently modified .log file (the DTU writes logs under feed
UUIDs that differ from timeline record IDs).
Allows rewinding to an earlier step when retrying a paused runner:
- --from-step <N>: skip steps before N, re-run from step N onwards
- --from-start: re-run all run: steps from the beginning
Each wrapped step embeds a 1-based index matching the tree UI and
checks a from-step signal file on entry. Steps below the target
exit 0 (skip), the matching step deletes the signal and continues.
Closes#35
Minimal vitest test that reads a fixture file and asserts its value,
designed to validate the pause-on-failure retry mechanism end-to-end.
- .github/workflows/retry-proof.yml
- experiments/retry-proof/ (package, test, fixture)
- Added retry-proof to pnpm workspace
- Extend computeLockfileHash() to detect package-lock.json, yarn.lock, bun.lock
- Replace PNPM-only .modules.yaml sentinel with universal hasInstallSentinel()
that checks .package-lock.json (NPM), .yarn-integrity (Yarn), .cache (Bun)
- Add NPM and Bun global cache bind-mounts into containers
- Extend virtualCachePatterns to skip tar for all PM cache keys
- Add benchmark scripts in experiments/pkg-cache-bench/
- 176 tests pass, TypeScript clean
Peter Pistorius