mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
zernie
4f0e6e8037
`claude --effort` sets the reasoning budget per turn. vigiles could not set it, so an A/B could not reproduce a benchmark that pins one: the arm you ran was not the arm the author measured (reported in #189, with a real case — a skill whose published results pin "effort low"). Effort is a MEASUREMENT, not a run knob: it moves the output distribution, not the sample size. So it lives on the spec beside `model` (never an env), it is an arm in its own right, and it is hashed into BOTH the eval lock and the local cache — the reporter named only the lock; the cache key carried `model` without effort in exactly the same way, and fixing one would have left half the hole. Three things that are not in the reported shape: - THE ENV PIN. Effort has three inputs — the flag, the `effortLevel` settings key, and CLAUDE_CODE_EFFORT_LEVEL — and the env var outranks the flag, while EPHEMERAL_ALLOW_PREFIXES passes `CLAUDE_*` through by design. So an ambient value survives even the scrubbed ephemeral env. Hashing effort WITHOUT pinning it would make the lock confidently wrong: `low` recorded over a run that executed at the shell's `max` — the defect this feature exists to prevent, reintroduced by the fix for it. Declared sets the var; OMITTED deletes an inherited one, so "omit" means the harness default rather than whatever the machine exported. - NOT A LITERAL UNION. The binary is case-insensitive, keeps an alias map, and accepts an integer budget; its valid set already MOVED between builds (2.1.42 had no `xhigh`). A hard-coded union would reject a valid level after any upstream addition. The silent fallback a union was meant to prevent is not silent — the binary warns on stderr, which is already captured — so the wrong value is caught at runtime by `effortRejection`, and a rejected value THROWS rather than counting as `runError`: an unusable value is deterministic, so every trial fails it and the report would be a rate over zero samples. - `harness-test.ts` IS NOT A SITE. That tier runs a scripted mock with a hard-coded model; effort has no observable effect there. The issue names it; adding the flag would have been dead argv. Codex refuses a declared effort loudly instead of dropping it — no mapping has been measured, and silently recording a budget the run did not use is worse than refusing. `audit` deliberately gets no CLI flag: it is a local report, not a reproducibility surface. `buildAgentArgs` is extracted so the flag is provable at all — the argv was built inside the `v8 ignore`d spawn, where nothing could assert it. Measured: `npm run check` 18/18 in 73s; 3462 unit tests, 0 failures. Eight mutations, each proven to have LANDED. Seven fail on their own assertion. The eighth stays green and that is a finding about the TEST, recorded in place: effort reaches the lock hash through two fills (the `withEvalLock` chokepoint and the per-arm inputs), so removing either alone is green and removing BOTH fails — overlap across two populations, not a dead line. One regression caught by the full suite and fixed here: `withEffortGuard` was written `async`, which converts the wrapped runner's SYNCHRONOUS refusals into rejected promises — downgrading the foreign-test-runner guard that stops a paid eval from billing. Now pinned by a test. Claude-Session: https://claude.ai/code/session_01PrS34j2M8XR12P2QHaBer4 Co-authored-by: Claude <noreply@anthropic.com>
Documentation — index
How-to and reference docs for using vigiles. New here? Start with the README for the pitch and a 5-minute quick start, or try the live demo at vigiles.sh — grade any repo in your browser.
The docs are grouped by what you're trying to do:
- Guides — step-by-step, task-first ("help me do X").
- Reference — exact flags, symbols, rules.
- Explanation — the reasoning and trade-offs.
Guides — "help me do X"
Verify your instruction files (the Lint layer)
verifying-instruction-files.md— the master guide: the markdown → typed-spec ladder, the three rule types (enforce/guidance/guard), verified references, and the before/after tables. Holds the full validation-rules matrix.markdown-mode.md— the no-spec on-ramp: verify rules in plain markdown with inline<!-- vigiles:enforce -->comments, no TypeScript.skills-monorepo.md— adopt vigiles in a CI-tested skill library or a plain.claude/repo (noplugin.json).
Test & measure your harness (the Test + Eval layers)
harness-testing.md— task-first how-to: pick what you want to test (hook / wiring / skill firing / behaviour) and the tier that answers it, with a copy-paste first test and CI.harness-testing-claude-code.md— Claude Code specifics:scriptModel,${CLAUDE_PLUGIN_ROOT}/pluginDir/ theSkilltool, the bubblewrap sandbox.harness-testing-codex.md— Codex specifics:runHarnessTest({ adapter: codexAdapter })against realcodex exec, the Responses mock, what maps and what doesn't.
measuring-skills.md— A/B a skill, plugin, model, or rule change on real coding tasks: the metric triple (bill / target / blast-radius), the worked example, and why it's affordable on your subscription.migrating-from-promptfoo.md— move existing skill evals onto the subscription: the concept + assertion mapping, a worked side-by-side, and the honest gaps.
Author & ship
skills.md— authoring a SKILL.md across the three on-ramps; the prose-vs-gates split.compiled-hooks.md— author a hook as a pure typed function against the closedvigiles/hookvocabulary and compile it, making whole classes of hook bugs unrepresentable (false confidence, matcher bypass, capability creep).railway-subagents.md— the typedResultsubagent contract: declare a typed outcome, compose flat workers, and assert the outcome deterministically (no model judge).for-plugin-authors.md— the plugin-author journey end to end: scan a draft, fix what it flags, make your skills fire, rank against a marketplace, gate it in CI.github-action.md— run vigiles in CI: the composite Action, every input, the sticky PR comment, versioning.
Harnesses, adapters & agents
harnesses.md— which harness vigiles targets and how you pick one (by import), plus the capability matrix.authoring-an-adapter.md— teach vigiles a new harness: the five ports, a worked skeleton, validating with the conformance kit.non-js-harnesses.md— running vigiles on a harness whose project is Kotlin, Go or anything else non-JS: what needs Node and what does not.agent-setup.md— agent setup & workflows in one guide: whatinitdoes, per-agent recipes (Claude Code / Codex / multi-agent / Cursor), non-interactive setup + fallback hooks, and CI.
Reference — "the exact flag, symbol, or rule"
cli.md— the full CLI: every verb and flag, the Claude Code plugin,lintvsaudit.commands-and-how-they-relate.md— the mental model: howaudit/lint/test/eval/initfit together, and why measuring "do my skills fire?" usesinit, notaudit.testing-matrix.md— every use case of the harness-testing API mapped to the tier that tests it, and what each tier costs.testing-api.md— the full harness-testing API: every predicate, assertion,check, matcher, and option (measureTriggerRate/runEval/ significance).spec-format.md— the typed.spec.tsformat (target, sections, rules, verified references) — the source of truth.linter-support.md— the 11 linters +generate-types/generate-schema.adapter-api.md— the adapter API reference: every port field, the conformance functions, the registry API.- Validation rules: the full matrix lives in the linting guide; each rule has a doc under
rules/. - Library entry points (grouped by concern so a future harness can sit beside the current one):
vigiles/linting— the compiler + reference verification (compileClaude,compileSkill, …).vigiles/spec— the spec builders (claude,enforce,guidance,file,cmd,symbol, …) and the module-augmentation target for generated types.vigiles(the package root) — the free testing surface: the harness/hook runners, thecheckvocabulary and the runner-agnostic assertions.vigiles/eval— everything that can spend money, every symbol prefixedpaid_:paid_runEval,paid_measure,paid_measureArms,paid_measureTriggerRate,paid_judge,paid_judged,paid_claudeEvalDriver.defineEvalis on the free root, not here, and that is the point: a*.eval.mjsfile DESCRIBES its eval andvigiles evalruns it, so importing one cannot spend. See eval files describe their eval.vigiles/claude-code,vigiles/codex— the per-harness adapters.vigiles/adapter— the adapter-authoring kit.
- API reference (generated) → — every exported symbol across all entry points, generated from the source. The hand-written guides here are the human-facing layer; this is the exhaustive symbol-level reference.
Explanation — "why it's built this way"
what-vigiles-catches.md— the taxonomy of problems vigiles handles: the prevented / caught / measured model, biggest-problem-first.comparison.md— before/after tables, the determinism breakdown, the flow diagram, and what vigiles composes with rather than replaces.sandboxing.md— what the sandbox isolates vs records (honestly): IO /rm -rf, the three network modes, tiers and limits.emit-channel.md—experimental_emitTool: a skill returns a typed result by calling a tool instead of ending its turn, which is how an INLINE skill can carry an output contract at all. Includes the measurements, and when to prefer the stableoutput:instead.faq.md— the front-door FAQ across all four layers.