mirror of
https://github.com/redwoodjs/local-ci.git
synced 2026-09-19 07:54:29 +08:00
Peter Pistorius
47252ffc5f
Adds experiments/skill-evals/, a matrix eval that runs headless Claude Code against broken-repo fixtures with candidate SKILL.md wordings appended to CLAUDE.md, scores tool calls + file edits against per-fixture rubric regexes, and aggregates across N trials per (variant, fixture) cell. Includes: - run.mjs + rescore.mjs (live eval + replay saved transcripts against a changed rubric without re-running the agent) - 3 fixtures: lint-curly-x18, missing-changeset, noisy-failure - 3 variants: v1-current (baseline), v2-run-first, v3-no-mangling - Stub @redwoodjs/agent-ci per fixture — no Docker required for the eval Fixture repos are deliberately broken (syntax errors, missing changesets, etc.); .prettierignore and .oxlintrc ignorePatterns exclude them so the pre-commit hooks stay green. Scoped to experiments/, no changes to published packages, no changeset. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
34 lines
1.1 KiB
YAML
34 lines
1.1 KiB
YAML
name: missing-changeset
|
|
description: |
|
|
packages/cli was modified (new docker section in README) but no .changeset/*.md
|
|
was added. CI fails with "missing changeset." The ground-truth CI check
|
|
requires BOTH the docker section AND a changeset — so reverting to make CI
|
|
pass will fail ground-truth even if it passes the rubric.
|
|
|
|
rubric:
|
|
- id: ran_agent_ci
|
|
stream: tool_calls
|
|
must: present
|
|
pattern: '@redwoodjs/agent-ci|\bagent-ci\s+run\b'
|
|
|
|
- id: used_pause_on_failure
|
|
stream: tool_calls
|
|
must: present
|
|
pattern: '(--pause-on-failure|\s-p\b)'
|
|
|
|
- id: no_output_mangling
|
|
# Flags any output manipulation of `agent-ci run/retry` — pipe, file
|
|
# redirect, or trailing `&`. All three disable the interactive pause loop.
|
|
# Bare `2>&1` is safe and NOT flagged.
|
|
stream: tool_calls
|
|
must: absent
|
|
pattern: 'agent-ci(?:-dev)?\s+(?:run|retry)[^\n]*(?:\|\s*\S|>\s*(?!&\d)\S|&\s*(?:$|\n|;|&&|sleep|echo))'
|
|
|
|
- id: created_changeset
|
|
stream: file_edits
|
|
must: present
|
|
pattern: '\.changeset/[a-z0-9-]+\.md'
|
|
|
|
ground_truth:
|
|
cmd: ['npx', '@redwoodjs/agent-ci', 'run', '--all', '--quiet']
|