Auree Aubert
57254032cf
* Record which skills a session actually loaded Routing is the thing this bundle turns on, and it has been argued rather than measured. evals/routing.jsonl now scores 100% on the real-model tier — a ceiling effect, not a victory: every prompt in it was written by someone looking at the descriptions they were grading, and several were added to pin behavior that had just been implemented. A suite that always passes has stopped discovering anything. Prompts nobody wrote to pass have to come from real sessions. - hooks/skill-loads.sh reads a Claude Code transcript and emits a marker naming the skills invoked and the recipes/playbooks/guides opened: `[cargo-skills: cargo-gtm | docs: build-tam,waterfall]`. The session hooks append it to the row's summary, so it is queryable today over workspace_management.sessions with no schema change. - Wired into both session-checkpoint.sh and session-end.sh. Recomputed from the whole transcript each time rather than accumulated, so a session that dies abruptly still carries it. Emits nothing at all for sessions that never touched Cargo. - evals/README.md documents the three questions the data answers (which skills never load, which sessions loaded two before settling, which load with no doc read after) and how each turns into an eval case. - Self-test wired into skills-lint, because a regression here corrupts telemetry silently instead of failing loudly. Two bugs the fixtures passed and real transcripts caught: - **Prose mentions counted as reads.** The doc regex matched a bare path anywhere in the transcript, and a skill's own catalog table names every recipe and playbook — so one cargo-gtm load reported all 43 playbooks as opened. Now anchored to a `"file_path":` tool input. Fixture 7 is that exact case. - **Dangling prefix** when docs were read without a Skill call, giving `[cargo-skills: | docs: …]`. Both halves are now independently optional. Also: the doc list caps at 12, and says `+N more` rather than looking complete. PRIVACY: skill and document names only, all of them already public in this repository. Never a prompt, file contents, an argument, a record, or anything a user typed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix Bugbot issues: preserve marker in sync finalize, retain skill load order 1. End hook drops skill marker (high severity): The sync finalize now includes the skill marker in the placeholder summary, so it persists even when refine() fails, times out, or is skipped. The marker is fast (no LLM call, just grep/sed). 2. Skill order lost to sorting (medium severity): Replaced 'sort -u' with 'awk !seen[$0]++' to preserve load order. The first skill in a multi-skill session is the misroute candidate, so order matters for harvest question 2. Added test case 10 to verify load order preservation. Co-authored-by: Auree Aubert <aureeaubert@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Auree Aubert <aureeaubert@users.noreply.github.com>
3.5 KiB
Routing evals — and where the next cases come from
An agent sees exactly one thing before deciding whether to load a skill: its description. Seventeen of them compete for every prompt. These evals are the regression test for that decision.
Running them
node .github/scripts/routing-eval.ts # structural + lexical (gates CI)
node .github/scripts/routing-eval.ts --verbose # per-case scores
node .github/scripts/routing-eval.ts --llm # real model (needs ANTHROPIC_API_KEY)
node .github/scripts/routing-eval.ts --strict # gate on the hard tier too
The --llm tier runs in its own workflow (.github/workflows/routing-evals-llm.yml) on PRs touching a SKILL.md, weekly, and on demand. It reports to the run summary and fails below --llm-min=93.
The case file
routing.jsonl, one JSON object per line:
{"prompt": "…", "expect": "cargo-gtm", "why": "what this guards", "tier": "hard"}
expect— the skill that should win.why— optional, printed on failure. Write one whenever the case is non-obvious.tier—core(default) gates CI;hardis a deep paraphrase reported but not gated, because the offline lexical ranker cannot judge those fairly. The--llmtier judges both.
Add a case whenever you change a description. That is the whole contract: the change is the hypothesis, the case is the test.
The ceiling problem, and the fix
The suite currently scores 100% on the model tier. That is a ceiling effect, not a victory — every prompt in here was written by someone who could see the descriptions they were grading, and several were added specifically to pin behavior that had just been implemented. A suite that always passes has stopped discovering anything.
Prompts nobody wrote to pass have to come from real sessions. That is what hooks/skill-loads.sh is for.
Harvesting real cases
The session hooks append a marker to each Cargo session's summary naming the skills and reference docs that session actually loaded:
[cargo-skills: cargo-gtm | docs: build-tam,waterfall]
Sessions live in workspace_management.sessions, so the marker is queryable:
# Which skills are actually loading, and how often?
cargo-ai storage query execute "
SELECT title, summary
FROM workspace_management.sessions
WHERE summary LIKE '%cargo-skills:%'
ORDER BY created_at DESC
LIMIT 50"
Three questions worth asking of that data, each of which produces eval cases:
- Which skills never load? A skill with real usage everywhere else and zero loads has a description problem, not a capability problem. Its absence is the finding.
- Which sessions loaded two or more skills before settling? The first one loaded is a candidate misroute — the title tells you the prompt, and that pairing is a
hardcase with a known-correct answer. - Which skills load without any doc read following? Either the skill answered on its own, or the agent bounced. Compare against sessions where a recipe was opened.
Turn what you find into cases here, with a why that names the session it came from. Cases sourced from real transcripts are worth more than a dozen written from the descriptions, because they are the only ones that can surprise us.
What the marker does not contain
Skill and document names only — all of them already public in this repository. Never a prompt, a file's contents, an argument, a record, or anything a user typed. See the header of hooks/skill-loads.sh for the exact matching rules.