mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
9413061d9b
* refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com>
127 lines
4.2 KiB
TypeScript
127 lines
4.2 KiB
TypeScript
/**
|
|
* audit read-vs-run decision tests — the pure `decideExecute` that makes a plain
|
|
* `audit` a deterministic READ and gates ALL execution (safety battery + live MCP
|
|
* + trigger-rate) behind one consent: ask at a TTY (remembered), `--measure` is
|
|
* the headless yes, headless stays a read + a loud nudge. Plus the model-access
|
|
* helpers (disclosure wording) and the prompts scaffold.
|
|
*/
|
|
import { describe, it, expect } from "vitest";
|
|
import {
|
|
hasModelAccess,
|
|
isMeteredAccess,
|
|
decideExecute,
|
|
formatExecuteSkip,
|
|
scaffoldTriggerPrompts,
|
|
type ExecuteEnv,
|
|
} from "./scan-trigger-suggest.js";
|
|
|
|
describe("hasModelAccess", () => {
|
|
it("true for a metered API key", () => {
|
|
expect(hasModelAccess({ ANTHROPIC_API_KEY: "sk-x" })).toBe(true);
|
|
});
|
|
it("true inside an authenticated Claude Code session (no key)", () => {
|
|
expect(hasModelAccess({ CLAUDECODE: "1" })).toBe(true);
|
|
expect(hasModelAccess({ CLAUDE_CODE_ENTRYPOINT: "remote" })).toBe(true);
|
|
});
|
|
it("false with nothing set", () => {
|
|
expect(hasModelAccess({})).toBe(false);
|
|
expect(hasModelAccess({ CLAUDECODE: "0" })).toBe(false);
|
|
});
|
|
});
|
|
|
|
describe("isMeteredAccess", () => {
|
|
it("metered iff a paid API key is set (subscription is not metered)", () => {
|
|
expect(isMeteredAccess({ ANTHROPIC_API_KEY: "sk-x" })).toBe(true);
|
|
expect(isMeteredAccess({ CLAUDECODE: "1" })).toBe(false);
|
|
expect(isMeteredAccess({})).toBe(false);
|
|
});
|
|
});
|
|
|
|
describe("decideExecute", () => {
|
|
// The default: an interactive human, something executable present, no sticky.
|
|
const base: ExecuteEnv = {
|
|
hasExecutable: true,
|
|
isTTY: true,
|
|
json: false,
|
|
noInteractive: false,
|
|
};
|
|
|
|
it("ASKS once at a TTY when there's something to run and no sticky choice", () => {
|
|
expect(decideExecute(base)).toEqual({ kind: "ask" });
|
|
});
|
|
|
|
it("nothing executable → skip 'nothing' (a clean read, no nudge)", () => {
|
|
expect(decideExecute({ ...base, hasExecutable: false })).toEqual({
|
|
kind: "skip",
|
|
reason: "nothing",
|
|
});
|
|
});
|
|
|
|
it("headless (non-TTY / --json / --no-interactive) stays a read → skip 'headless'", () => {
|
|
expect(decideExecute({ ...base, isTTY: false })).toEqual({
|
|
kind: "skip",
|
|
reason: "headless",
|
|
});
|
|
expect(decideExecute({ ...base, json: true })).toEqual({
|
|
kind: "skip",
|
|
reason: "headless",
|
|
});
|
|
expect(decideExecute({ ...base, noInteractive: true })).toEqual({
|
|
kind: "skip",
|
|
reason: "headless",
|
|
});
|
|
});
|
|
|
|
it("headless never executes — not even a remembered yes (audit is a local report, not CI)", () => {
|
|
expect(decideExecute({ ...base, isTTY: false, remembered: true })).toEqual({
|
|
kind: "skip",
|
|
reason: "headless",
|
|
});
|
|
});
|
|
|
|
it("interactive honors the sticky choice (no re-ask)", () => {
|
|
expect(decideExecute({ ...base, remembered: true })).toEqual({
|
|
kind: "run",
|
|
});
|
|
expect(decideExecute({ ...base, remembered: false })).toEqual({
|
|
kind: "skip",
|
|
reason: "remembered-no",
|
|
});
|
|
});
|
|
});
|
|
|
|
describe("formatExecuteSkip", () => {
|
|
it("'nothing' → no nudge (a clean read)", () => {
|
|
expect(formatExecuteSkip("nothing")).toBeNull();
|
|
});
|
|
|
|
it("headless points at interactive + the testing API (no execution flag exists)", () => {
|
|
const note = formatExecuteSkip("headless");
|
|
expect(note).toContain("skipped");
|
|
expect(note).toContain("interactively");
|
|
expect(note).toContain("`vigiles` testing API");
|
|
expect(note).not.toContain("--measure");
|
|
});
|
|
|
|
it("remembered-no points at the config, not a flag", () => {
|
|
const note = formatExecuteSkip("remembered-no");
|
|
expect(note).toContain("audit.measure");
|
|
expect(note).not.toContain("--measure`");
|
|
});
|
|
});
|
|
|
|
describe("scaffoldTriggerPrompts", () => {
|
|
it("emits the real TriggerPromptSet shape (name → {prompts, irrelevant})", () => {
|
|
const json = scaffoldTriggerPrompts(["alpha", "beta"]);
|
|
const parsed = JSON.parse(json) as Record<
|
|
string,
|
|
{ prompts: string[]; irrelevant: string[] }
|
|
>;
|
|
expect(Object.keys(parsed)).toEqual(["alpha", "beta"]);
|
|
expect(Array.isArray(parsed.alpha.prompts)).toBe(true);
|
|
expect(parsed.alpha.prompts.length).toBeGreaterThan(0);
|
|
expect(Array.isArray(parsed.alpha.irrelevant)).toBe(true);
|
|
expect(parsed.beta.prompts[0]).toContain("beta");
|
|
});
|
|
});
|