redwoodjs__local-ci/experiments/skill-evals/package.json at fix/immutable-node-modules-snapshots - redwoodjs__local-ci - GetSkill.work

skills/redwoodjs__local-ci

mirror of https://github.com/redwoodjs/local-ci.git synced 2026-09-19 07:54:29 +08:00

Files

T

Peter Pistorius 47252ffc5f chore(skill-evals): add harness for measuring SKILL.md variants (#316 )

Adds experiments/skill-evals/, a matrix eval that runs headless Claude Code
against broken-repo fixtures with candidate SKILL.md wordings appended to
CLAUDE.md, scores tool calls + file edits against per-fixture rubric regexes,
and aggregates across N trials per (variant, fixture) cell.

Includes:
- run.mjs + rescore.mjs (live eval + replay saved transcripts against a
  changed rubric without re-running the agent)
- 3 fixtures: lint-curly-x18, missing-changeset, noisy-failure
- 3 variants: v1-current (baseline), v2-run-first, v3-no-mangling
- Stub @redwoodjs/agent-ci per fixture — no Docker required for the eval

Fixture repos are deliberately broken (syntax errors, missing changesets,
etc.); .prettierignore and .oxlintrc ignorePatterns exclude them so the
pre-commit hooks stay green.

Scoped to experiments/, no changes to published packages, no changeset.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-23 13:20:37 +02:00

14 lines

218 B

JSON

Raw Permalink Blame History

 {
   "name": "skill-evals",
   "private": true,
   "type": "module",
   "scripts": {
     "eval": "node run.mjs",
     "rescore": "node rescore.mjs"
   },
   "dependencies": {
     "execa": "^9.5.1",
     "yaml": "^2.6.1"
   }
 }