mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
zernie
9413061d9b
* refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com>
23 lines
1.2 KiB
TypeScript
23 lines
1.2 KiB
TypeScript
/**
|
|
* `vigiles/codex` — the OpenAI Codex harness adapter. Sits beside
|
|
* `vigiles/claude-code`: same harness-agnostic `vigiles` testing core, a
|
|
* different transport (the `codex` binary + the OpenAI **Responses** SSE mock).
|
|
*
|
|
* Pillar 2 (harness testing) is proven here — `startCodexMock` serves the
|
|
* Responses SSE that real `codex exec` completes a turn against, keylessly (see
|
|
* `codexMockArgs`/`codexMockEnv` for the wiring). Pillar 1 instruction/skill
|
|
* *renderers* still emit the Claude-Code shape until the format-axis renderers
|
|
* land (see `research/code-adapter-architecture.md`).
|
|
*/
|
|
export * from "./adapters/codex/dialect.js";
|
|
export * from "./adapters/codex/layout.js";
|
|
export * from "./adapters/codex/runtime.js";
|
|
export * from "./adapters/codex/hook-protocol.js";
|
|
export * from "./adapters/codex/model-mock.js";
|
|
export * from "./adapters/codex/mock-model.js";
|
|
export * from "./adapters/codex/driver.js";
|
|
export * from "./adapters/codex/adapter.js";
|
|
// Eval-tier transport (increment 2 — scaffold, pending live-binary validation):
|
|
// parseCodexEvalRun (the ModelOutputParser for `codex exec --json`) + codexEvalRunner.
|
|
export * from "./adapters/codex/eval.js";
|