zernie__vigiles/api-surface/vigiles-eval.api.md at v22

mirror of https://github.com/zernie/vigiles.git synced 2026-09-14 20:53:57 +08:00

Files

T

zernie 9413061d9b feat!: split the public exports by COST — vigiles/test (free) vs vigiles/eval (spends money) (#149 )

* refactor(exports)!: split the testing surface on COST, and name the cost

The exports map split the testing API by TEST TIER — `vigiles/testing`,
`vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had
a home: the vitest projects, keyed on test-file NAMES. The exports map was a
second, competing implementation of it, and zero of 19 comparable packages
publish `./unit` / `./integration` / `./e2e` subpaths.

It splits on COST now, on two axes at once — the PATH and the NAME.

  "."             -> ./dist/test.js          (was ./dist/core/spec.js)
  "./spec"        -> ./dist/core/spec.js     unchanged
  "./eval"        -> ./dist/eval-surface.js  NEW
  ./linting ./hook ./claude-code ./codex ./adapter ./experimental
  ./vitest ./jest                            unchanged

DELETED: ./testing, ./unit, ./integration, ./e2e.  14 subpaths -> 11.

No `./test` subpath: the bare package name IS the testing surface. Two names for
one door is the duplication being removed (`.` + `./spec` used to be exactly
that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with
`vigiles eval`, the free half answers to the plain package name. Accepted;
`@playwright/test` publishes no subpath naming a test type at all.

THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`:
paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate,
paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top
of a file; the name warns EVERY time, at the call site. Reading
`await judged(trace, "did it refuse?")` on line 140, the import is long out of
view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined
path with an `experimental_` prefix; this applies the device to a second axis.

TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something,
and a type is never called. `paid_EvalReport` would be noise — and since the 29
report types are re-exported from BOTH barrels, prefixing them would put `paid_`
names on the FREE surface.

⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN.
`paid_judged(rubric, { judge: myFn })` calls your function and spends nothing;
the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path
bills (`paid_judge` is the one where `paid_` is exact — no injection seam).
`metered_` would be precise but reads a beat slower, and a warning not absorbed
at a glance is not a warning. Clarity won; the imprecision is written into the
module docstring, each symbol's JSDoc, and docs/testing-api.md.

ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime`
sat unprefixed among six `experimental_` siblings on `./experimental`. One
missing prefix is worse than none: the six that comply teach the reader that an
unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`.

BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose
buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the
TypeScript compiler (so it can tell a value from a type instead of grepping) and
FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check`
and a CI step beside `docs:check`.

ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts`
(12 cases, colocated) fires on planted violations AND is silent on the legitimate
surface next door. Four mutations, each failing on its OWN assertion:

  drop alias resolution        -> 2 fail  (the vacuous-pass: `./eval` is built
                                  entirely from `export { x as paid_x }`, and an
                                  alias reports Alias, not Value — a check that
                                  skipped resolution would read every export as a
                                  type and pass on a fully unprefixed barrel)
  exit(1) -> exit(0)           -> 1 fail  (gate degrades to nudge)
  delete the ./experimental
    entry from SURFACES        -> 1 fail  (see below)
  never push a violation       -> 4 fail

🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check.
An entire axis could stop being policed with nothing turning red: the axis test
passes its own surface list, the "real barrels" test is satisfied by a SHORTER
list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting
SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the
same way (its fixture lacked `./experimental`, so it exited 1 either way) and now
runs a clean twin that must exit 0.

NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols.
New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported
from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH
directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a
superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus
decideHook / parseHookOutput) existed only on `unit` and are on the root now.

TWO HONEST COSTS, in the docstrings rather than hidden:
1. The type coupling crosses the boundary and cannot be removed. ~27 helpers
   (assertImproves, compareArms, diffReports, cost/latency/tokens,
   formatEvalReport, …) read an eval RESULT and spend nothing, so they are free
   and unprefixed — but their argument types are defined on the paid side. Hence
   the 29 duplicated types, so nobody imports `./eval` for a type alone.
2. Free is not fast. The old "real CLI, no API key" tier collapses into the root,
   so the import no longer signals that `runHarnessTest` spawns a real claude
   under bubblewrap for ~40s. Duration lives in the runner config now.

`.` moving off `core/spec` means every doc example importing a spec builder from
bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught
those.

Entry points are enumerated in four places independently of package.json; all
updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is
the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs
(AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation
to still fire on the new filenames), vitest.config.mjs comment.

Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files)
· npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier
--check . · npm run api:check (11 entries, no drift) · npm run build · plus
npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also
`node dist/cli.js lint` exits 0 with every compiled-markdown hash valid.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd

* ci: закрыть класс «сгенерированный файл протух, а CI зелёный»

Шаг «Check generated types compile» перегенерирует файл и компилирует результат.
Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке
НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на
314 строк / ~147 записей, и CI всё это время была зелёной.

Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они
актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает.

Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine
08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift
in a generated file is invisible unless something regenerates it with --check.
Upstream vigiles tracks its own copy and it IS stale on main for exactly that
reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает
строка в CI.

Проверено обеими половинами: на актуальном файле проходит, на подложенном
дрейфе (одна строка в конец) падает.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd

---------

Co-authored-by: Claude <noreply@anthropic.com>

2026-08-16 22:46:26 +05:00

9.8 KiB

Raw Permalink Blame History

API Report File for "vigiles-eval"

Do not edit this file. It is a report generated by API Extractor.


// @public
export interface AgentRunArgs {
    // (undocumented)
    readonly cwd: string;
    readonly env?: Record<string, string>;
    // (undocumented)
    readonly hasSettings: boolean;
    // (undocumented)
    readonly model: string;
    // (undocumented)
    readonly pluginDir: string | undefined;
    readonly replaceEnv?: boolean;
    // (undocumented)
    readonly task: string;
    // (undocumented)
    readonly timeoutMs: number;
    // (undocumented)
    readonly tools: readonly string[];
}

// @public
export type AgentRunner = (args: AgentRunArgs) => Promise<RunOut>;

// @public (undocumented)
export interface ArmReport {
    readonly metrics: Record<string, number>;
    // (undocumented)
    readonly runs: number;
    readonly stats: Record<string, MetricStat>;
    readonly usage: ArmUsage;
}

// @public
export interface ArmsCheckReport {
    // (undocumented)
    readonly arms: Record<string, CheckReport>;
}

// @public
export interface ArmsMeasureSpec {
    // (undocumented)
    readonly allowedTools?: readonly string[];
    readonly arms: Record<string, EvalArm>;
    // (undocumented)
    readonly checks: readonly Check<RunContext>[];
    // (undocumented)
    readonly fixture?: Record<string, string>;
    // (undocumented)
    readonly model?: string;
    // (undocumented)
    readonly spacingSec?: number;
    readonly stubSkillBodies?: boolean;
    // (undocumented)
    readonly task: string;
    // (undocumented)
    readonly timeoutMs?: number;
    // (undocumented)
    readonly trials?: number;
}

// @public
export interface ArmUsage {
    // (undocumented)
    readonly meanCostUsd: number;
    // (undocumented)
    readonly meanDurationMs: number;
    // (undocumented)
    readonly totalCacheCreationTokens: number;
    // (undocumented)
    readonly totalCacheReadTokens: number;
    // (undocumented)
    readonly totalCostUsd: number;
    // (undocumented)
    readonly totalInputTokens: number;
    // (undocumented)
    readonly totalOutputTokens: number;
}

// @public
export interface Check<T> {
    eval(target: T): CheckResult;
    readonly kind: string;
    toJSON(): CheckJSON;
}

// @public
export interface CheckRate {
    // (undocumented)
    readonly check: CheckJSON;
    readonly n: number;
    readonly passK: number;
    readonly rate: number;
    readonly se: number;
}

// @public (undocumented)
export interface CheckReport {
    // (undocumented)
    readonly n: number;
    // (undocumented)
    readonly perCheck: readonly CheckRate[];
    readonly usage: ArmUsage;
}

// @public
export interface CheckResult {
    readonly message: string;
    readonly pass: boolean;
    readonly score: number;
}

// @public
export interface EvalArm {
    readonly files?: Record<string, string>;
    readonly interceptTools?: readonly ToolIntercept[];
    readonly model?: string;
    readonly plugin?: string;
    readonly pluginDir?: string;
    readonly settings?: unknown;
}

// @public
export interface EvalDriver {
    readonly experimental?: string;
    readonly harness?: string;
    // (undocumented)
    readonly parse: ModelOutputParser;
    // (undocumented)
    readonly runError?: (out: RunOut) => string | null;
    // (undocumented)
    readonly runner: AgentRunner;
}

// @public (undocumented)
export interface EvalReport {
    readonly aborted: boolean;
    // (undocumented)
    readonly arms: Record<string, ArmReport>;
    // (undocumented)
    readonly name: string;
    readonly totalCostUsd: number;
    // (undocumented)
    readonly trials: number;
}

// @public (undocumented)
export interface EvalSpec<M extends Metrics> {
    readonly allowedTools?: readonly string[];
    readonly arms: Record<string, EvalArm>;
    readonly cache?: CacheMode;
    readonly cacheDir?: string;
    readonly concurrency?: number;
    readonly ephemeralEnv?: boolean;
    readonly fixture?: Record<string, string>;
    readonly lock?: EvalLockOptions;
    readonly maxCostUsd?: number;
    readonly measure: (ctx: RunContext) => M;
    readonly model?: string;
    // (undocumented)
    readonly name?: string;
    readonly rateLimitRetries?: number;
    readonly retryBackoffMs?: number;
    readonly spacingSec?: number;
    readonly stubs?: readonly ToolStub[];
    readonly task: string;
    readonly timeoutMs?: number;
    readonly trials?: number;
}

// @public
export interface EvalUsage {
    readonly cacheCreationTokens: number;
    readonly cacheReadTokens: number;
    readonly costUsd: number;
    readonly durationMs: number;
    readonly inputTokens: number;
    // (undocumented)
    readonly outputTokens: number;
}

// @public
export type JudgeFn = (opts: {
    output: string;
    rubric: string;
    threshold?: number;
}) => {
    score: number;
    pass: boolean;
    reason?: string;
};

// @public
export interface MeasureSpec {
    readonly allowedTools?: readonly string[];
    readonly checks: readonly Check<RunContext>[];
    readonly fixture?: Record<string, string>;
    readonly interceptTools?: readonly ToolIntercept[];
    readonly model?: string;
    readonly plugin?: string;
    readonly pluginDir?: string;
    readonly settings?: unknown;
    readonly spacingSec?: number;
    readonly stubSkillBodies?: boolean;
    readonly task: string;
    readonly timeoutMs?: number;
    readonly trials?: number;
}

// @public (undocumented)
export type Metrics = Record<string, number | boolean>;

// @public
export interface MetricStat {
    readonly mean: number;
    readonly n: number;
    readonly passK: number;
    readonly se: number;
    readonly std: number;
}

// @public (undocumented)
export type ModelOutputParser = (out: RunOut) => ParsedModelRun;

// @public
export const paid_claudeEvalDriver: EvalDriver;

// @public
export function paid_judge(opts: JudgeOptions): JudgeResult;

// @public
export function paid_judged(rubric: string, opts?: {
    min?: number;
    judge?: JudgeFn;
}): Check<Trace>;

// @public
export function paid_measure(spec: MeasureSpec): Promise<CheckReport>;

// @public
export function paid_measureArms(spec: ArmsMeasureSpec): Promise<ArmsCheckReport>;

// @public
export function paid_measureTriggerRate(spec: TriggerRateSpec, opts?: {
    evalDriver?: EvalDriver;
}): Promise<TriggerRateReport>;

// @public (undocumented)
export function paid_runEval<M extends Metrics>(spec: EvalSpec<M>): Promise<EvalReport>;

// @public
export interface ParsedModelRun {
    // (undocumented)
    readonly hooks: ReturnType<typeof parseHooks>;
    // (undocumented)
    readonly output: string;
    // (undocumented)
    readonly subagents: ReturnType<typeof parseSubagents>;
    // (undocumented)
    readonly toolCalls: ReturnType<typeof parseToolCalls>;
    // (undocumented)
    readonly turns: number;
    // (undocumented)
    readonly usage: EvalUsage;
}

// @public (undocumented)
export interface PromptDiversityIssue {
    // (undocumented)
    readonly kind: "too-few" | "too-similar";
    // (undocumented)
    readonly message: string;
}

// @public
export interface PromptTriggerStat {
    // (undocumented)
    readonly fired: number;
    // (undocumented)
    readonly prompt: string;
    readonly rate: number;
    // (undocumented)
    readonly trials: number;
}

// @public
export interface RunContext extends Trace {
    // (undocumented)
    readonly cwd: string;
    // (undocumented)
    readonly exitCode: number;
    sh(command: string): string;
    // (undocumented)
    readonly stdout: string;
    readonly toolCalls: readonly ToolCall[];
    readonly turns: number;
    readonly usage: EvalUsage;
}

// @public
export interface RunOut {
    // (undocumented)
    code: number;
    stderr?: string;
    // (undocumented)
    stdout: string;
}

// @public
export interface SelectionTrialResult {
    // (undocumented)
    readonly errored: boolean;
    // (undocumented)
    readonly fired: readonly string[];
}

// @public
export interface Trace {
    file(path: string): string | null;
    readonly hooks: readonly HookFire[];
    readonly modelRequests: readonly ModelRequest[];
    readonly output: string;
    readonly subagents?: readonly SubagentTrace[];
    readonly toolCalls: readonly ToolCall[];
    readonly turns: number;
}

// @public (undocumented)
export interface TriggerRateReport {
    readonly competitors: number;
    readonly errored?: number;
    readonly experimental?: string;
    readonly falsePositiveRate?: number;
    readonly n: number;
    readonly perIrrelevant?: readonly PromptTriggerStat[];
    // (undocumented)
    readonly perPrompt: readonly PromptTriggerStat[];
    readonly precision?: number;
    readonly rate: number;
    readonly usage: ArmUsage;
}

// @public
export interface TriggerRateSpec {
    readonly allowedTools?: readonly string[];
    readonly concurrency?: number;
    readonly fired: (trace: Trace) => boolean;
    readonly fixture?: Record<string, string>;
    readonly installSet?: readonly string[];
    readonly irrelevantPrompts?: readonly string[];
    readonly lock?: EvalLockOptions;
    readonly minDistance?: number;
    readonly minModel?: string;
    readonly minPrompts?: number;
    readonly model?: string;
    readonly name?: string;
    readonly pluginDir?: string;
    readonly prompts: readonly string[];
    readonly skillsDir?: string;
    readonly spacingSec?: number;
    readonly stubSkillBodies?: boolean;
    readonly timeoutMs?: number;
    readonly trials?: number;
}

// (No @packageDocumentation comment for this package)

9.8 KiB Raw Permalink Blame History

API Report File for "vigiles-eval"

9.8 KiB

Raw Permalink Blame History