zernie
9413061d9b
* refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com>
11 KiB
Harnesses — which one vigiles targets, and how to pick
An agent is Model + Harness. The harness is the machine around the model: the CLI, the hook protocol, the plugin layout, the instruction-file dialect. vigiles keeps its core harness-agnostic — harness-specific pieces live behind a swappable adapter.
✅ Supported today: Claude Code and Codex.
vigiles/codexis auto-detected by the CLI. It verifies references, compiles Codex-shaped instructions (AGENTS.md) and skills (minimalSKILL.md), and runs Test-layer harness tests against the realcodexbinary (keyless, OpenAI-Responses mock) viarunHarnessTest({ adapter: codexAdapter }). Subagents are a deliberate boundary — see the matrix below. Adding a harness never touches the core.
You pick the harness by which subpath you import
Select the adapter at author time, by your import — not by a config setting. Two surfaces:
// Core — harness-agnostic. The stable API you write tests/evals against.
import { paid_runEval } from "vigiles/eval";
import { runHarnessTest, assertHookBlocked } from "vigiles";
// Adapter — the Claude Code-specific pieces, named explicitly.
import { loadPlugin, scriptModel } from "vigiles/claude-code";
The vigiles root (plus vigiles/eval for the model-calling half) is the part that never changes when the harness changes. It includes runHarnessTest, paid_runEval, runHook, the Trace predicates, and all assertions.
vigiles/claude-code is the adapter. It reads real Claude Code layouts (.claude-plugin/plugin.json, .claude/settings.json, ${CLAUDE_PLUGIN_ROOT}, skills/, agents/) and provides the scriptable Anthropic Messages mock you point claude at.
A second harness sits beside the first — same core, a different import. Codex already ships this way:
import { startCodexMock, codexAdapter } from "vigiles/codex";
Nothing in vigiles or vigiles/eval changes, and unused adapters tree-shake out. Importing the adapter (rather than a harness: config key) means the bundle carries only what you use, and the choice is type-checked where you write it.
The CLI auto-detects (or reads project config)
The CLI detects the harness from the repo automatically — a .claude-plugin/, an AGENTS.md, etc. — and vigiles compile, vigiles audit, and vigiles lint all work with zero config.
When detection is ambiguous, or you want a committed, deterministic choice, override it two ways:
harnesskey in.vigilesrc.json—"codex", or["claude-code", "codex"]for a multi-harness repo. Written byvigiles init.--harness=<name>flag — wins over the config file.
Precedence: --harness= → config harness → auto-detect. A multi-harness or ambiguous match prints a loud notice instead of silently guessing. A repo declaring several harnesses also gets a byte-identical CLAUDE.md⇄AGENTS.md mirror on compile when no sync tool already fans it out.
What's actually harness-specific (the two axes)
Not everything labelled "Claude Code" is coupled the same way. There are two independent axes, and a new harness might share one but not the other:
| Axis | What it covers | Example difference for Codex |
|---|---|---|
| Format / dialect | the instruction-file format and plugin layout: CLAUDE.md vs AGENTS.md, SKILL.md frontmatter, the tools: contract, ${…_PLUGIN_ROOT} |
emits/verifies AGENTS.md instead of CLAUDE.md |
| Runtime / transport | the binary that runs the model, the hook event protocol, the model mock | spawns codex with a different hook JSON shape |
The reference-verification engine — does this linter rule exist and is it enabled, does this path/script/symbol resolve — is the same across harnesses. Only the format it reads/writes and the runtime it drives differ.
Adapter capability matrix
Not every harness can reach every tier. Pretending otherwise produces adapters that hang or no-op. Each adapter declares an AdapterCapabilities descriptor (src/core/adapter.ts). The conformance kit requires a transport port only for the capability the adapter claims — so a reference-only harness is a first-class adapter, not a broken one.
| Capability | Descriptor flag | Needs ports | What it unlocks |
|---|---|---|---|
| Lint (layer 1) — reference verification | referenceVerification (always true) |
dialect, layout |
compile / scan / lint — verify refs, tools, paths |
| Test (layer 2) — harness testing | harnessTesting |
runtime + modelMock |
runHarnessTest / runEval (spawn binary, mock model) |
| Shell-hook tier | shellHooks |
hookProtocol |
the runHook unit tier (hooks as shell processes) |
| Subagents | subagents |
— | the subagent lint rules (subagent-tool-contract, …); n/a where false |
Where the harnesses land (✅ shipped · 🧪 internal prototype · ⛔ blocked, with why):
| Harness | Lint | Test (mockable) | Shell hooks | Status |
|---|---|---|---|---|
| Claude Code | ✅ | ✅ Anthropic SSE | ✅ exit 2 / decision | ✅ shipped — the reference adapter |
| Codex | ✅¹ | ✅ Responses SSE | ✅ veto (exit 2) | ✅ shipped (vigiles/codex) — Lint & Test² (subagents differ by design) |
| OpenCode | ✅ | 🚧 openai-compat³ | ⛔ code-module hooks | 🧪 prototype (src/adapters/opencode/) — shellHooks:false |
| Cursor | ✅ | ⛔ closed, no BYOM | ⛔ | not built — Lint-only at best |
| Devin / Amp / Amazon Q | ✅ | ⛔ un-mockable | varies | not built — Lint-only (closed backend) |
¹ Codex Lint is format-correct for the surfaces that map. References are verified against the Codex dialect, and compile emits Codex-shaped output — AGENTS.md (plain markdown) and minimal SKILL.md (name/description only, driven by dialect.skillFrontmatter), with Claude Code output byte-identical (the dogfood integrity hash is the guardrail).
Subagents are a deliberate boundary, not a gap. A Codex subagent is an [agents.<name>] TOML concurrency table (max_threads/max_depth), not a tool-contract file. vigiles's agent() doesn't map onto it, so it isn't compiled to Codex (it is still verified). The loader reads Codex's TOML manifest format-aware, so its [mcp_servers] table is detected like CC's JSON mcpServers.
² Codex is shipped: registered in ADAPTERS (the CLI auto-detects a .codex/config.toml or AGENTS.md repo) and exported as vigiles/codex. Test is full and usable through the public API — runHarnessTest(spec, { adapter: codexAdapter }) drives real codex exec against the OpenAI Responses mock (src/adapters/codex/mock-model.ts), keylessly. A gated harness-test.test.ts runs the real-codex turn (request shape + SSE captured from live traffic, not guessed). Trigger-rate IS usable through the public API — measureTriggerRate(spec, { evalDriver: codexEvalDriver }), with codexEvalDriver exported from vigiles/codex — but the number is experimental (CODEX_TRIGGER_RATE_EXPERIMENTAL, printed as a loud caveat on every run): Codex has no skill-selection event, so firing is inferred from a SKILL.md read, which is wrong in both directions (a cached skill is not re-read → false negative; an exploratory read is not a fire → false positive). Treat it as directional, not a measurement. runEval still has no public dispatch: runEval(spec) always drives Claude Code, and while the Codex runner (codexEvalAgentRunner) is exported, the runEvalWith(spec, runner) seam that would take it is not.
³ OpenCode's mockable tier is declared but not yet built: it needs the openai-compat (Chat Completions) SSE renderer, and the opencode binary isn't wired here yet. The Codex path proves the same work generalizes.
The key discriminator is Test, not Lint. Everyone converges on Lint (AGENTS.md is becoming universal). The gate is mockability. OpenCode is the case that splits the matrix mid-row: mockable (Test is reachable) but with in-process plugin hooks instead of shell processes (shellHooks:false, no hookProtocol).
Full capability inventory. This matrix is the brief version. The exhaustive record of every harness-specific capability (CC vs Codex) — and, per capability, whether vigiles verifies it, tests it, or records-only (acknowledged but deliberately not supported). Testing every special capability is an explicit non-goal.
How this is kept honest
The core staying harness-agnostic isn't a convention you have to remember — it's enforced. eslint-plugin-boundaries classifies modules into verify-core (the domain) and cc-harness (the Claude Code adapter) and forbids the core from importing the adapter (eslint.config.mjs, rule boundaries/dependencies). The dependency only points one way: adapter → core, never core → adapter.
vigiles also dogfoods that rule: CLAUDE.md.spec.ts carries enforce("boundaries/dependencies"), so vigiles compile checks the boundary rule is present and enabled. The architecture invariant is a verified reference, not a comment.
See also
docs/harness-testing.md— the harness-agnostic test tiers that ride onvigiles/vigiles/eval. Per-harness specifics:harness-testing-claude-code.md·harness-testing-codex.md.docs/non-js-harnesses.md— using vigiles on a Kotlin/Go/Java repo with nopackage.jsonor Node toolchain (which checks work with zero setup, JVM/Go linter cross-referencing).