1 Commits

Author	SHA1	Message	Date
Auree Aubert	3708f8c955	Agent discoverability: trigger-shaped descriptions, routing evals, standalone bootstrap (#77) ... * Agent discoverability: trigger-shaped descriptions, routing evals, standalone bootstrap The `description` field is the only text an agent weighs before deciding whether to load a skill. Everything else in the bundle is invisible until that decision is already made — and twelve of the seventeen descriptions read as "Manage <CLI nouns> using the Cargo CLI", which matches a user who is already thinking in Cargo's object model rather than one stating a job. This rebuilds that layer and, for the first time, tests it. Routing - Every description rewritten to a four-part template: job in the user's words -> literal quoted trigger phrases -> proper nouns -> an explicit `Skip when:` pointing at the sibling skill that owns the adjacent case. - Integration and provider names are generated, not hand-maintained. sync-trigger-slugs.ts fills cargo-gtm's `Providers:` list from the playbook tree and cargo-connection's `Integrations:` list from a committed catalog snapshot (138 integrations, 50 credits-based). `--refresh` re-pulls from a logged-in CLI; `--check` gates CI. - evals/routing.jsonl (92 prompts) + routing-eval.ts, wired into CI. Structural tier asserts the template and that each skill keeps >=5 terms unique to it; lexical tier requires the expected skill to rank first; an opt-in --llm tier asks a real model to route from descriptions alone. Deep paraphrases are tagged "hard" and reported rather than gated — a lexical proxy cannot judge them fairly. Standalone installs - `skills add --skill <one>` is a supported path and the registry lists each skill separately, but thirteen skills' only install instructions were a relative link to ../cargo/references/prerequisites.md, which does not exist in a single-skill install. Every SKILL.md now has a self-contained Bootstrap section, enforced by the linter. Coverage - New capability skill cargo-segmentation — the segmentation CLI domain, previously uncovered while segments were load-bearing in most recipes. Verified against the live CLI, including the flag traps. - Three cargo-gtm recipes: source-planning (probe and cost candidate sources per *hit* before any fan-out), ads-audience-activation (Google Customer Match / LinkedIn Matched Audiences), review-and-iterate (human review of judgment output, kept as an eval set). - Router gains a "these skills vs a workspace MCP server" section: Cargo has no first-party MCP server — a workspace builds one and serves it over `cargo-ai mcp` — so the routing rule is scale and reproducibility. Index and hygiene - llms.txt now lists every recipe and provider playbook with a generated summary: 17 entries -> ~80 job-named addressable URLs. - Router trimmed ~80 lines of per-skill summaries that restated each skill's own description; critical rules kept in full. - Linter: bootstrap check, skill-count claims validated against the tree, `doctor` and `mcp` added to the known CLI domains. - CONTRIBUTING.md, keyword-dense README opening, updated repo metadata. Also carries the aiArk/apolloio priority-stack promotion authored in a parallel session on this branch (cargo-gtm 1.12.0) — the two touch the same SKILL.md files and could not be split cleanly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Surface the 100-credit free tier where an agent meets it There is no purchase gate between installing Cargo and producing a real deliverable — a new account starts with 100 free credits and no card — and nothing in the bundle said so. An agent that doesn't know the budget can't tell a first-time user whether the demo is a look around or a commitment, and defaults to hedging. Now stated at every point an agent encounters the question: - README opening and the generated llms.txt install block: an agent can install, sign the user up, and hand back a list in one turn. - Router install section and prerequisites.md, with anchors for what 100 credits buys (~5,000 sourced leads, ~1,000 verified-email enrichments, ~50 fully enriched contacts). - cargo-quickstart: the demo spends ~0.5 of the 100, and the skill now says so before the first paid call — that reframes the moment from a purchase decision into a look around, which is the job of a quickstart. - cargo-billing: a "what 100 credits buys" table, plus two new trigger phrases so free-tier questions route here at all. - cost-discipline.md: on a new account, receipts and next-step proposals must be framed against the free tier. "87.6 of your 100 free credits left" is a number the user can act on; "87.6 remaining" is not. Also warns that phone lookups (6-7/call) are the fastest way to burn a tier a first session should end with something left to try. Two routing eval cases added for free-tier questions (79/79 core). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Run the routing evals against a real model in CI The lexical tier passes 79/79, but it is a deterministic proxy written alongside the descriptions it grades — it cannot fail in a way that surprises us, which is what makes it weak evidence. The only honest scoreboard for the description rewrite is whether a real model, shown nothing but the 17 descriptions, routes the eval prompts correctly. That tier existed but had never run. Now it does, in its own workflow. - New .github/workflows/routing-evals-llm.yml, using the repo's ANTHROPIC_API_KEY. Deliberately not part of skills-lint: it costs money and is non-deterministic, so it must never stand between a typo fix and a merge. Triggers on PRs and pushes that touch a SKILL.md, the eval cases, or the runner; weekly on a schedule so model drift surfaces even when nobody edits a description; and on demand with a model override. - runLlm reworked for CI: 6-way concurrency (92 sequential calls is a slow job), retry with backoff on 429/5xx, tolerant answer parsing, and a core/hard/overall breakdown instead of one number. - Results are written to GITHUB_STEP_SUMMARY as a table, including every misroute with what the model picked instead — so a regression is readable on the run page without opening logs. - Optional --llm-min=<percent> floor, wired to the exit code. Left unset until the first runs establish a baseline; a floor guessed before the data would either never fire or block on noise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Act on the first real-model routing run: 92/94, two fixes Baseline from the model tier (claude-haiku-4-5, judging from the 17 descriptions alone): 92/94 overall — core 79/79, hard 13/15. The lexical proxy's 3/15 on the hard tier was noise about the proxy, not the descriptions, which is what that tier existed to establish. Both misroutes were worth having: - "Our outbound is going to the wrong people — help me figure out who actually buys from us" routed to cargo-context instead of cargo-gtm. A real collision: cargo-context's triggers include "who do we sell to", which reads as ICP ownership. It owns *writing the conclusion down*; deriving it from won/lost data is cargo-gtm. Added that boundary to cargo-context's Skip when, and the two phrasings to cargo-gtm. - "Give the assistant access to our sales collateral so it stops making things up" routed to cargo-content instead of cargo-ai — and the model was right. As written the first step is getting the collateral into the workspace. Corrected the expectation and added the other side of the boundary as its own case ("attach the library we already uploaded to the agent"), which the lexical tier then failed, correctly: cargo-ai never said "library" or "release resources". Added both, so the content/ai split is now legible to a ranker and a model. Set --llm-min=93 now that a baseline exists. Deliberately below the observed 98%, not at it: a model tier that fails on one non-deterministic flip trains people to ignore it. Lexical 80/80 core. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Document cargo-ai mcp — the bridge that was written down nowhere Cargo reaches coding agents through two doors. Skills reach agents that implement skills.sh; a workspace MCP server reaches everything else — Claude Desktop, ChatGPT, and whatever ships next. The second door was fully built and documented in exactly one place: `cargo-ai mcp --help`. Worse, cargo-ai/SKILL.md taught `ai mcp-server create` (building a server) without ever mentioning the command that serves it, so an agent could construct one and have no idea how to connect it — and the section conflated publishing a server with consuming someone else's, which are unrelated surfaces. - MCP section rewritten around the two directions, with a table: publish (`ai mcp-server`, called by any MCP client, wired via `cargo-ai mcp`) vs consume (`ai mcp-client`, called by your agents, wired via `release update-draft --mcp-clients`). States plainly that there is no first-party Cargo MCP server — a workspace builds its own. - Documents `cargo-ai mcp` with the registration one-liner, the CARGO_MCP_SERVER_UUID fallback, and the stdout-is-protocol constraint. - Details verified against the live API rather than --help: actions take kind "tool" *or* "agent" (an agent is exposable as a callable MCP tool), resources take model or file, and `update` replaces the arrays wholesale instead of merging — a real footgun. - Routing rule for which door to use, pointing at the router section. - Two trigger phrases ("expose our tools as an MCP server", "use Cargo from Claude Desktop or ChatGPT") and two eval cases, so the publish direction is reachable from a description at all. It previously wasn't. Lexical 81/81 core. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-12 10:18:32 -07:00