Update the agent-facing skill surfaces to match the recent pause-on-failure
fixes and `--json` event stream:
- skills/agent-ci/SKILL.md (published) and packages/cli/SKILL.md: document
`--json` / `AGENT_CI_JSON=1`, list the NDJSON event types, and note that
pipes/redirects are safe — pause-on-failure exits 77 in non-TTY mode.
- .claude/commands/agent-ci-dev.md and .pi/skills/agent-ci-dev/SKILL.md:
switch the monitor pattern from grepping plaintext "Step failed" to
matching `"event":"(run.paused|run.finish)"`, and call out exit-77 as a
pause signal alongside the `run.paused` event.
- experiments/skill-evals/variants/v2-run-first.md and v3-no-mangling.md:
remove stale "no pipes / no redirects" warnings — those scenarios are now
the supported path via the detached launcher.
Refs #289, #315.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds experiments/skill-evals/, a matrix eval that runs headless Claude Code
against broken-repo fixtures with candidate SKILL.md wordings appended to
CLAUDE.md, scores tool calls + file edits against per-fixture rubric regexes,
and aggregates across N trials per (variant, fixture) cell.
Includes:
- run.mjs + rescore.mjs (live eval + replay saved transcripts against a
changed rubric without re-running the agent)
- 3 fixtures: lint-curly-x18, missing-changeset, noisy-failure
- 3 variants: v1-current (baseline), v2-run-first, v3-no-mangling
- Stub @redwoodjs/agent-ci per fixture — no Docker required for the eval
Fixture repos are deliberately broken (syntax errors, missing changesets,
etc.); .prettierignore and .oxlintrc ignorePatterns exclude them so the
pre-commit hooks stay green.
Scoped to experiments/, no changes to published packages, no changeset.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Peter Pistorius