mirror of
https://github.com/getcargohq/cargo-skills.git
synced 2026-09-14 19:17:15 +08:00
Auree Aubert
57254032cf
* Record which skills a session actually loaded Routing is the thing this bundle turns on, and it has been argued rather than measured. evals/routing.jsonl now scores 100% on the real-model tier — a ceiling effect, not a victory: every prompt in it was written by someone looking at the descriptions they were grading, and several were added to pin behavior that had just been implemented. A suite that always passes has stopped discovering anything. Prompts nobody wrote to pass have to come from real sessions. - hooks/skill-loads.sh reads a Claude Code transcript and emits a marker naming the skills invoked and the recipes/playbooks/guides opened: `[cargo-skills: cargo-gtm | docs: build-tam,waterfall]`. The session hooks append it to the row's summary, so it is queryable today over workspace_management.sessions with no schema change. - Wired into both session-checkpoint.sh and session-end.sh. Recomputed from the whole transcript each time rather than accumulated, so a session that dies abruptly still carries it. Emits nothing at all for sessions that never touched Cargo. - evals/README.md documents the three questions the data answers (which skills never load, which sessions loaded two before settling, which load with no doc read after) and how each turns into an eval case. - Self-test wired into skills-lint, because a regression here corrupts telemetry silently instead of failing loudly. Two bugs the fixtures passed and real transcripts caught: - **Prose mentions counted as reads.** The doc regex matched a bare path anywhere in the transcript, and a skill's own catalog table names every recipe and playbook — so one cargo-gtm load reported all 43 playbooks as opened. Now anchored to a `"file_path":` tool input. Fixture 7 is that exact case. - **Dangling prefix** when docs were read without a Skill call, giving `[cargo-skills: | docs: …]`. Both halves are now independently optional. Also: the doc list caps at 12, and says `+N more` rather than looking complete. PRIVACY: skill and document names only, all of them already public in this repository. Never a prompt, file contents, an argument, a record, or anything a user typed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix Bugbot issues: preserve marker in sync finalize, retain skill load order 1. End hook drops skill marker (high severity): The sync finalize now includes the skill marker in the placeholder summary, so it persists even when refine() fails, times out, or is skipped. The marker is fast (no LLM call, just grep/sed). 2. Skill order lost to sorting (medium severity): Replaced 'sort -u' with 'awk !seen[$0]++' to preserve load order. The first skill in a multi-skill session is the misroute candidate, so order matters for harvest question 2. Added test case 10 to verify load order preservation. Co-authored-by: Auree Aubert <aureeaubert@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Auree Aubert <aureeaubert@users.noreply.github.com>