mirror of
https://github.com/getsentry/skills.git
synced 2026-09-14 20:00:03 +08:00
David Cramer
5a64b36c62
This standardizes skill evaluation on AXIS, using its built-in Codex adapter so evals run through the real `codex exec --json` harness instead of custom runner scripts. The new maintainer path for `skill-writer` is: ```bash npx @netlify/axis run --config skills/skill-writer/evals/axis.config.json npx @netlify/axis run --config skills/skill-writer/evals/axis.config.json --scenario small-inline-workflow npx @netlify/axis reports latest --config skills/skill-writer/evals/axis.config.json --html ``` Each AXIS scenario copies the repo into an isolated temp workspace, excludes local agent/report state, installs the local skill under test with `skills: ["./.."]`, runs Codex, and captures transcript/judge/artifact evidence. Generated `.axis/` report directories are now ignored so local eval runs do not pollute future diffs. The initial `skill-writer` evals cover three behavior classes: - `small-inline-workflow`: catches simple skills being overbuilt with references, scripts, or broad source research. - `reference-backed-integration`: checks that complex/integration skills keep `SKILL.md` as a router and split optional depth into focused references. - `iteration-from-bad-output`: checks that `skill-writer` can improve an existing bloated skill by narrowing triggers and removing unnecessary files. The runtime `references/skill-evals.md` teaches future skills the same pattern: add `EVAL.md`, `evals/axis.config.json`, and focused AXIS scenarios with observable judge checks. It explicitly keeps AXIS as the prescribed skill-eval framework and treats Promptfoo/custom Codex scripts as out of scope for this repo's skill evals. `pr-writer` gets a proof eval focused on actual behavior rather than eval plumbing: ```bash npx @netlify/axis run --config skills/pr-writer/evals/axis.config.json --scenario concise-docs-pr ``` That scenario creates a docs-only skill change and verifies the generated PR title/body stay concise, use an appropriate conventional title, avoid template headings, and omit validation/tool boilerplate. The renamed `concise-docs-pr` case is meant to be a pattern for behavior-focused eval cases, not framework smoke tests. I exercised the new paths locally: `small-inline-workflow` completed through AXIS/Codex with score 94, and `concise-docs-pr` completed with score 99. Structural validation also passed for both `skill-writer` and `pr-writer`. Co-authored-by: GPT-5 Codex <codex@openai.com>
15 lines
139 B
Plaintext
15 lines
139 B
Plaintext
# Claude Code local settings
|
|
.claude/settings.local.json
|
|
|
|
# AXIS reports
|
|
.axis/
|
|
**/.axis/
|
|
|
|
# OS
|
|
.DS_Store
|
|
|
|
# Editors
|
|
.idea/
|
|
.vscode/
|
|
*.swp
|