zernie
9413061d9b
* refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com>
4.9 KiB
untested-subagent
Flag a subagent (agents/*.md) that ships without a test or eval. One of the
per-kind surface-coverage rules alongside untested-skill
and untested-hook.
An agent's contract is different from a skill's: it's dispatched by name (via the
Task tool) and carries a tool contract (tools: frontmatter). The test that
matters is "does it honor its contract and produce the right result?" — not "does
its description fire", which is the skill question. An agent with no test is an
unmeasured surface in the deterministic layer.
Which tier answers what
The two tiers ask the same question of an agent and differ in who plays the model — so a passing deterministic test does not make the eval tier redundant, and this is the split the coverage nudge now reflects:
| tier | model | what it establishes |
|---|---|---|
deterministic (runHarnessTest) |
scripted mock | the contract parses and the rail holds — subagent(...), notTool, assertAgentOk |
eval (runEval / measure) |
real, paid | a real model, handed this agent's prompt, stays inside the contract |
Keep the same assertions across both; only the model changes. The typed-outcome helpers are model-free by design — see railway subagents.
measureTriggerRate does not apply to an agent
Measured against this build, no model spent:
packageSkillsDir("<repo>/agents") → THREW: No <name>/SKILL.md
skills found under <repo>/agents
measureTriggerRateWith({ pluginDir: <agents-only → rate = 0 | competitors = 0
plugin> }, fakeRunner)
It installs <name>/SKILL.md skills and decides "fired" by skill
selection (skillResolved), so an agent is not a surface it can address. The
loose-skills form throws; the plugin form returns a number for a surface it
never installed, which is worse than an error. "Does Claude pick this subagent
when it should?" is a real question — vigiles has no API for it yet.
⚠️ runEval / measure / measureArms drive Claude Code and take no
evalDriver (only measureTriggerRate has that seam, and runEvalWith is not
exported from vigiles/eval). On a harness other than Claude Code the eval
tier has no public dispatch for a subagent yet — Codex has no subagent dir at all
(agentDir: ""), so in practice this concerns OpenCode.
Configuration
{ "rules": { "untested-subagent": "warn" } }
Severity
| Value | Behavior |
|---|---|
"error" |
vigiles lint exits non-zero when an agent is untested |
"warn" (default) |
Prints a warning, exits 0 — a nudge, not a gate |
false |
Skip agent coverage entirely |
Options (testGlobs, exclude) are shared with the other untested-* rules —
see untested-skill.
Scope
Scans agents/*.md and .claude/agents/*.md.
What counts as "tested"
A recorded run first (.vigiles/coverage.json), then colocation as the
fallback — agents/bar.harness.mjs next to agents/bar.md. A test elsewhere
that merely NAMES the agent does not count (that tier was removed 2026-08-11).
See
untested-skill for the shared
mechanics.
What earns an agent a recorded run (added 2026-08-12): a dispatch in the
transcript — a tool_use whose input carries a subagent_type, which is what a
passing subagent("bar", …) check is asserting on. It is keyed on the input
field rather than the tool name, because the dispatch tool is named Agent on
the live CLI and Task in older docs. An errored dispatch does not count (the
tool was reached and the agent was not), and neither does a call merely named
Task with no subagent_type.
Before that, no probe could resolve to an agent at all, so an agent proven to run
by a passing subagent(...) was still reported here as untested. If the dispatch
is namespaced (plugin:bar, which is what --plugin-dir reports) the namespace
must be this repo's own plugin name — another plugin's bar is not yours.
Exemptions
Every agent is held to this; the only opt-out is an explicit
<!-- vigiles:ignore-test --> marker in the agent file, reported as exempt.
Why
Same as the skill rule: the second layer tests the assembled harness. A subagent
is a high-risk surface (it acts with tools), so leaving it unmeasured is exactly
the gap this family closes. Warning-by-default; flip to "error" to gate CI.