Files
zernie__vigiles/src/jest.ts
T
zernie 66ca161e8b fix: round-2 dogfooding fixes for untested-surface, audit rename, orphans, loose-skill evals (#35)
* fix: round-2 dogfooding fixes for untested-surface, audit rename, orphans, loose-skill evals

- untested-surface: discover colocated tests under dot dirs (dot:true), so a
  `.claude/skills/<name>/<name>.eval.mjs` the warning suggests actually counts
  as coverage instead of leaving the nudge unsatisfiable.
- init: flag a CI workflow that still calls a renamed/removed subcommand
  (`vigiles audit` → `lint`) even when the Action or a modern command is also
  present, instead of reporting it "up to date".
- orphan-docs: honor an inline `<!-- vigiles-disable orphan-docs -->` opt-out
  and point the failing message at both escape hatches (inline disable +
  .vigilesrc.json orphans.exclude/include).
- measureTriggerRate: accept a loose `skillsDir` (e.g. .claude/skills) and
  auto-package it into a throwaway --plugin-dir via packageSkillsDir, so
  trigger-testing repo-local skills is a one-liner.
- docs/strings: replace stale `vigiles audit` command references with
  `vigiles lint` (schema descriptions, hook/skill prose, CLAUDE.md).

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* feat: make strengthen/edit-spec model-invocable, tidy skill packaging, harden agent install

Skills are useless if nobody remembers to run them. Address discovery + install
gaps surfaced in the dogfood review:

- strengthen + edit-spec are now MODEL-INVOCABLE (drop disable-model-invocation)
  with sharpened, precision-aware descriptions, so the agent reaches for them on
  its own — "strengthen my rules" / "add a rule to CLAUDE.md" fire the skill.
  The pre-edit hook already points at edit-spec when it blocks a direct edit;
  making it model-invocable closes that loop. Each gets a dogfood trigger eval
  (recall + precision) so auto-firing doesn't become over-firing.
- Merge generate-rule into edit-spec (rule classification absorbed) and delete
  it. Fix stale commands in edit-spec (vigiles check->lint, setup->init, the old
  `npx skills add` vendoring installer -> marketplace).
- linter-docs gets a real SKILL.md (user-invoked reference), fixing the
  packaging smell of a skills/ dir with no SKILL.md.
- Bump .claude-plugin/plugin.json version 3.0.0 -> 4.0.1 (was stale).
- Harden installPlugins: on auto-install failure/no-CLI, fail LOUDLY and lead
  with the shell-runnable CLI commands (an agent can run those) instead of only
  /plugin slash commands a non-TTY agent can't execute.
- Sweep remaining stale `vigiles audit` -> `lint` (example spec).

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* refactor: remove the renamed `audit` command everywhere; rename lint-command internals

`audit` became `lint` in v4 but the old name lingered. Purge it:

- cli.ts: rename the lint command's internals (audit()->runLint, AuditReport->
  LintReport, auditExitCode->lintExitCode, printAuditSummary->printLintSummary,
  annotateAuditForGitHub->annotateLintForGitHub) + comments. Keep the
  REMOVED_SUBCOMMANDS `audit: "lint"` detection key (it matches the OLD name).
- FUNCTIONAL: bench scripts invoked `node $CLI audit SKILL.md` (removed cmd) —
  now `lint`.
- docs/research/comments: `vigiles audit` -> `vigiles lint`, and the command-
  sense word "audit" -> "lint".
- Kept genuinely-distinct uses: the post-session AUDIT feature (session.ts),
  the Merkle "audit trail", the supply-chain audit / scan concept, the
  dev-only `audit-feedback-loop` skill, and English "you audited the code".
- Left third-party vendored fixtures (SHA-pinned) untouched.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* feat: prompt-diversity gate for trigger evals + raise auto-skill recall bar to 0.8

Task: ensure shipped model-invocable skills have evals, and validate prompt sets
are big/varied enough before spending tokens.

- measureTriggerRate now runs a deterministic pre-flight gate (no model): each
  prompt set needs >= minPrompts (default 10) and no two prompts more than
  maxSimilarity (default 0.8) alike via normalized Levenshtein. Throws with an
  actionable message before any run. Exports levenshtein/promptSimilarity/
  checkPromptDiversity/assertPromptDiversity; minPrompts/maxSimilarity are
  per-spec overridable (narrow skills, short walkthroughs).
- Expanded all shipped model-invocable dogfood evals (test-harness, strengthen,
  edit-spec) to 10 diverse relevant + 10 irrelevant prompts and raised the bar:
  recall min 0.8, maxFalsePositive 0.3. generate-logo + the canonical template
  expanded to 10; the OMC walkthrough sets minPrompts: 3.
- All shipped skills are covered: the 3 auto skills by trigger evals, the
  user-invoked (migrate-to-spec, linter-docs) by the load gate.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* refactor: use NCD (gzip) instead of Levenshtein for prompt-diversity check

Levenshtein measured spelling; switch the prompt-diversity gate to Normalized
Compression Distance — the SAME engine `findSimilarRules` uses for near-
duplicate rule detection, so the project has one house algorithm for "are these
two texts basically the same". NCD scores shared structure/redundancy (a
templated prompt with one word swapped compresses together), which is exactly
the lazy copy-paste set the gate should reject.

- Drop hand-rolled levenshtein/promptSimilarity; add promptDistance (= ncd over
  normalized text), reusing src/core/proofs.ts ncd().
- TriggerRateSpec: maxSimilarity -> minDistance (default 0.3, the rule-dup
  threshold). Calibration: real 10-prompt sets sit at NCD >= 0.44; near-dup
  pairs (one-word swaps, punctuation) at <= 0.24 — clean separation at 0.3.
- Tests updated to NCD distance.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* feat: stubSkillBodies — measure skill triggering on frontmatter only

Trigger-rate is decided by a skill's frontmatter (name + description) before its
body ever loads, so the body is causally downstream of selection. stubSkillBodies
keeps every skill's frontmatter (the selection competition stays faithful) but
replaces each body with a no-op stub, so a trigger run stops AT selection instead
of executing an expensive multi-step procedure — far cheaper, faster, side-
effect-free, without changing what's measured. ~95% smaller skill payload.

- stubSkillBody (pure) + packageSkillsDir({ stub }) (frontmatter-only package).
- resolveTriggerPluginDir stubs both paths (pluginDir + skillsDir), preserving
  the plugin NAME so `<name>:<skill>` still matches the fired predicate.
- TriggerRateSpec.stubSkillBodies (default off; on in all trigger evals).
- Needs one real-model run to confirm the Skill tool still resolves under a
  stubbed body (the deterministic transform is fully tested here).

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* fix(init): scaffold spec imports only `claude` so it passes strict ESLint

The generated CLAUDE.md.spec.ts imported `{ claude, enforce, guidance }` but the
enforce()/guidance() examples are commented out, so a repo running
`eslint . --max-warnings=0` (common in CI) failed on two no-unused-vars warnings
the moment the scaffold was committed. Import only `claude` (the symbol the
scaffold actually uses); show the fuller import as a comment for when a real
rule is added. Regression test guards the import line.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* feat(init): --force rewrites a stale CI workflow in place

Round-2 follow-up: instead of only warning about a stale workflow, `vigiles init
--force` fixes it. A renamed/removed subcommand (vigiles audit) is rewritten
surgically in place (audit → lint, the rest of the user's workflow preserved); a
bare-`npx vigiles` no-op workflow is regenerated from the template. Without
--force the behavior is unchanged (warn, don't clobber) and the warning now points
at --force. Pure `rewriteRemovedSubcommands` + force plumbed through setup-plan;
unit + CLI integration tests.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* docs: positioning for the eval-tier decision + new trigger-rate features

- research/eval-api-landscape.md: dated decision — harness eval ≠ model eval
  (fidelity to the harness-as-shipped is why promptfoo et al. can't host it);
  keep the deterministic tiers, keep the real-model surface thin (stubSkillBodies
  = selection not execution), don't rebuild the eval stack, model fidelity is the
  one on-moat refinement.
- CLAUDE.md(.spec): second-pillar paragraph gains the harness-eval-≠-model-eval
  framing; src/eval.ts Key Files entry documents skillsDir/packageSkillsDir,
  stubSkillBodies, and the NCD prompt-diversity gate.
- docs/harness-testing.md: trigger-rate section documents skillsDir,
  stubSkillBodies, the diversity gate, and measuring on the user's real model.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* docs(research): testing-API design proposal — the ideal shape

Synthesizes "what makes a world-class testing API" (deterministic assertion libs:
node:assert/Chai/Vitest/Testing-Library/fast-check; AI-eval: promptfoo/DeepEval/
Inspect) with the "do we need all these layers?" question into one conclusion:

- One Trace (already the substrate); a single declarative `check` vocabulary
  evaluated two ways — strict (throw, unit/integration) vs scored (0-1 ± se,
  eval). Collapses today's predicate/assert*/matcher 3× into one surface and
  unifies deterministic + non-deterministic.
- Three tiers, not four: e2e is integration + one egress symbol → fold egress in
  as a capability of integration, keep the capability ladder.
- Checks-as-data ⇒ JUnit/baseline/regression + a promptfoo bridge fall out free.
- Recommends option C (declarative checks) + D (property-test add-on for hooks).

Indexed in CLAUDE.md Key Files + research/README; no users yet, so it's framed as
a clean-break design, not a migration.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* fix(orphan-docs): resolve file-relative markdown links; refine tier taxonomy in design doc

- orphans.ts: a markdown link is conventionally file-relative ([x](foo.md) in
  research/README.md → research/foo.md, ../docs/x.md walks up). The detector only
  matched root-relative paths, so a bare sibling link read as an orphan. Now
  credits BOTH readings (refTargets resolves against the source file's dir +
  keeps the root-relative reading). Tests for same-dir and ../ links.
- testing-api-design.md: fold in the two taxonomy questions. The layer names are
  borrowed from app testing, not derived: the harness has only TWO execution
  scopes (hook = the one standalone-process surface; harness = the whole agent,
  since skills/CLAUDE.md/settings only act inside a live agent) over a static lint
  floor. integration/e2e/eval are the SAME scope under realness flags (model
  mock|real, egress) — model picks the verdict mode (mock→strict, real→scored).
  "unit only works with hooks" is structural, not a missing capability; each layer
  adds exactly one capability (the ladder). Recommends runHook + runHarness(flags)
  over the four borrowed tiers.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* feat(check): Phase 0 — the declarative `vigiles/check` vocabulary

First step of the testing-API revamp (research/testing-api-design.md). A check is
DATA, not a throwing assert: { kind, eval(target) → {pass, score, message},
toJSON() }, typed over Trace (tool/skill/output/hookFired/wrote) vs HookRunResult
(blocked/allowed). One vocabulary, to be evaluated two ways downstream — strict
(expect, Phase 1) and scored (measure, Phase 3) — so a check reads as pass/fail on
one run AND a rate across trials, and serializes (JUnit/baseline/promptfoo-bridge
fall out later).

- src/check.ts + src/check.test.ts (pure, model-free; the failure MESSAGE is
  tested as much as the verdict).
- Exported at `vigiles/check` only — NOT folded into vigiles/testing yet, since
  hookFired collides with the legacy predicate (reconciled in Phase 6).
- Additive: nothing removed; 889 tests green.
- Build plan appended to the design doc as Part 6 (Phase 0 marked shipped).

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* feat(check): Phase 1 — strict evaluator (assertChecks) + check matcher veneer

- assertChecks(target, checks): the strict evaluator — throws collecting ALL
  failed checks' messages (Validation-applicative, not short-circuit). Any runner.
- vigilesMatchers gains ONE generic veneer for the whole vocabulary:
  expect(r).toPass(tool("Bash")) / toPassAll([...]) — each carries the check's own
  failure message. vitest/jest type augmentations added; cross-runner tests prove
  toPass works under both.
- Additive: the legacy predicate/assert*/matcher surface is untouched (it gets
  removed in Phase 6). 891 tests green.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* feat(harness): Phase 2 — runHarness, the explicit harness-scope entry

Establishes the collapsed naming from Part 4: the harness has two execution
scopes (hook = runHook, harness = the whole agent); integration/e2e/eval are the
ONE harness scope under realness flags. runHarness is the DETERMINISTIC harness
run (model: "mock", default) — behaviour identical to runHarnessTest, which it
wraps. A real-model harness run is non-deterministic, so a single one can't be
asserted: runHarness(spec, { model: "real" }) throws and points at measure() /
runEval (the eval scope). Unit-tested (the guard needs no claude). The e2e barrel
fold + a materialized real path remain for Phase 6.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* feat(check): Phases 3+4 — measure() over checks + serialization (toJUnit, assertRates)

Phase 3 — the SCORED evaluator. measure(spec, { trials, checks }) scores a Trace
check vocabulary across trials → per-check rate ± se + pass^k, reusing the tested
runEvalWith aggregation (one arm). measureWith takes an injected runner (unit-
tested, no model); measure wraps it with the real CLI. Defaults to model "sonnet"
(measure on the model your users run). So `tool("Bash")` reads pass/fail strictly
(assertChecks) AND as a rate here — one vocabulary, two evaluators.

Phase 4 — serialization → CI. checkReportToJUnit (each check a <testcase>, fails
below `min`) + assertRates (the scored gate) + formatCheckReport, all keyed off
checkLabel(check.toJSON()). Because a check is data, JUnit/baseline/promptfoo-
bridge fall out. Pure + unit-tested. (Wiring into the Action/baseline is the
remaining CI-integration step.)

894 tests green.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* feat(run-hook): Phase 5 — propertyHook, invariant testing for a hook's decision

A hook is (event) → decision, so instead of a few example events, generate MANY
and assert invariants hold for every decision ("destructive command ⇒ blocked",
"never both allow and block"). Reuses the deterministic propertyTest from
proofs.ts (seeded, shrinks the counterexample; no fast-check dep). The `decide`
runner is injectable — pure fake for unit tests, `(e) => runHook(cmd, e)` for a
real hook. Returns the shrunk counterexample, not a throw, so it's inspectable.
Test: a correct guard satisfies the invariant; a buggy one (blocks only the exact
"rm -rf /") gets a "rm -rf foo" counterexample found. 895 tests green.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* docs(check): Phase 6 — front-door the new check surface; defer the mass deletion

The new world-class surface (Phases 0-5) is now the documented front door:
docs/harness-testing.md gains a "Checks — one vocabulary, two evaluators" section
leading with assertChecks / measure / toPass; the design doc marks Phases 0-5
shipped; CLAUDE.md Key Files note runHarness/measure/propertyHook/toPass.

DELIBERATELY NOT done: ripping out the ~40 legacy predicate/assert*/matcher
helpers + the e2e barrel and rewriting the ~900-test suite/examples off them. On a
no-users lib that's high-churn cleanup with no functional gain and real regression
risk — the legacy surface stays as working sugar, removed in a single separately-
reviewed major-version cleanup (resolving the check↔testing hookFired collision
then). The breaking step is the one to take slowly.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* refactor(testing): fold check into testing/unit; collapse e2e into integration

The "breaking cleanup" — done at its true (small) blast radius, not the feared
900-test rewrite:
- vigiles/check folds into vigiles/testing + vigiles/unit; the check `hookFired`
  (Check<Trace>) supersedes the legacy boolean predicate via an explicit
  re-export that wins over the two `export *`s. Checks (tool/skill/output/...,
  assertChecks, measure, propertyHook) are now first-class at the front door.
- e2e collapsed: egress is a CAPABILITY of integration (egressRoutes + runHook's
  egress:{allow}), not a tier — the old e2e barrel added exactly one symbol.
  vigiles/e2e stays as a thin deprecated alias so imports keep working.

KEPT (deliberately): the bare predicates + per-runner assert*/matcher fronts.
They're not redundant — they're context-appropriate surfaces (compose / node:test
/ vitest-jest) now unified UNDER one check vocabulary. Deleting them would be
churn without a greatness gain. 895 tests green, types + cross-runner pass.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* docs(testing-api): Part 7 — greatness re-evaluation + ranked eval-side gaps

Post-cleanup assessment: the API is great on the two deciding axes (unification:
one Trace / one check vocabulary / two evaluators / serialized; and the moat: the
deterministic no-key tiers + pass^k/significance/fidelity nobody else has).
Extensibility is strong; the one seam to watch is Check.eval being sync (fine
incl. a judge()-backed check; an async scorer would need an AsyncCheck for
measure only). Ranked gaps, eval-side first: (1) judged() model-graded check —
judge() exists but isn't a Check; fits the sync interface; #1 parity gap;
(2) cost/latency/tokens checks (data already captured); (3) checks × A/B arms +
significance (unify the harness-A/B moat with measure); (4) multi-surface CC eval
— Trace is single+flat, no nested subagent/Task traces, slash-command expansion,
multi-turn, or mcp() checks (the deep, new-capability bet); (5) real-tier Trace
fidelity (modelRequests mock-only); (6) CC-feature drift maintenance.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* feat(check): close eval-side gaps — judged/cost/latency/tokens/mcp checks, measureArms + significance, harness-agnostic

- judged(rubric, { min, judge }) — model-graded check (LLM rubric) via an
  injectable judge fn (default real judge(), sync via spawnSync), composes into
  measure/assertRates/toJUnit. Closes the promptfoo llm-rubric / DeepEval GEval gap.
- cost/latency/tokens — Check<UsageTrace> over a run's usage; measure.checks
  widened to Check<RunContext> so they compose with behavioural checks.
- mcp(server, tool) — matches the mcp__server__tool tool name.
- measureArms(spec, { arms, checks }) + compareCheck(report, baseline, arm, i):
  the SAME checks scored per arm, with Welch significance (reuses stats.ts) —
  unifies the harness-A/B moat with the check vocabulary.
- Harness axis: a test pins that checks are harness-agnostic (evaluate over a
  Codex-sparse Trace — output passes, tool/hookFired fail gracefully). Runners
  dispatch via adapter; measureWith/measureArmsWith take an injectable runner so a
  Codex eval reuses the same checks.

All unit-tested with fakes (no model). 900 tests green. Design doc Part 7 updated:
gaps 1-3 + mcp shipped; nested-trace (subagents/turns) is the remaining big bet.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* docs(harness-testing): add the vitest/jest setup + matcher matrix

The reference table we'd flagged in the vitest/jest discussion: a two-column
setup matrix (vitest vs jest: config / per-file / manual) and a matcher table
(toPass / toPassAll / toHaveCreated / toBlock / toBeatBaseline → asserts → over
which result), leading with toPass as the one matcher fronting the whole check
vocabulary. Makes the runner story scannable instead of prose.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* feat(check): nested subagent traces — Trace.subagents + subagent() check

The deep gap's first piece: a subagent (Task) runs its own session, recoverable
because CC tags its events with parent_tool_use_id (= the Task tool call) and the
Task input carries subagent_type. parseSubagents groups the tagged tool calls into
Trace.subagents (keyed by name); subagent(name, [checks]) runs the WHOLE check
vocabulary recursively over that sub-trace, so a test/eval can assert what the
subagent DID, not just that Task fired. Wired into makeResult (harness) +
makeContext (eval); empty for harnesses that don't emit parent_tool_use_id (Codex)
so it stays harness-agnostic. parseSubagents + the check unit-tested on synthetic
stream-json; flagged in Part 7 for one real-CC-subagent run to confirm the field
names. 902 tests green.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* feat(check): slash-command-expansion (received) + multi-turn (turns) checks

Closes the last two pieces of gap #4, both unit-tested:
- received(matcher): the expanded slash command (or a hook's injected context)
  actually REACHED the model — reads modelRequests (mock/harness tier captures
  them; the eval tier captures none, with an explicit message saying so).
- turns({min,max}): the multi-turn observable — the agent took a back-and-forth,
  not a one-shot (the deterministic harness already scripts multiple model turns
  via ModelTurn[]; this asserts how many it took).

Remaining (smaller, noted in Part 7): driving multiple USER messages
(--resume-style, unverifiable here) + per-turn trace slicing. 904 tests green.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* docs(check): validate the subagent parser schema against Claude Code stream-json

Confirmed via the Claude Code SDK docs / guide: subagent events carry a top-level
`parent_tool_use_id` (= the Task tool-use id), the Task input carries
`subagent_type`, linked by tool_use id — exactly what parseSubagents assumes, on
the same CLI `message.content` line shape parseToolCalls already consumes (the SDK
StreamEvent `event:{delta}` shape is a different, unused format). No code change —
the parser is validated; dropped the "needs a real run to confirm" caveat in the
comment + Part 7.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* test(dogfood): exercise the new check capabilities on real targets

- checks-dogfood.harness.mjs — RUNS here (claude, no key): assertChecks over a
  real runHook (blocked/allowed) + a real runHarness mock run (turns/output).
  Validated end-to-end, not just unit-tested with fakes.
- dogfood/skill-quality.eval.mjs — the SCORED path on vigiles's OWN strengthen
  skill: measure({ checks: [skill, judged, cost] }) + assertRates (does it fire,
  is the suggestion good, is it cheap). Write-don't-run (needs a key).
- subagent.eval.mjs — the nested-trace check on a REAL subagent (vigiles ships
  none): the vendored oh-my-claudecode code-reviewer, dispatched via Task;
  subagent("code-reviewer", [tool("Read")]) asserts what the subagent DID.

Note: the harness-tier dogfood asserts the cross-version-reliable observables
(turns/output); tool-call capture in a bare run is plugin/permission-dependent
(the fuller pluginDir tests in harness-test.test.ts cover it). 904 green.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* docs: document the mock-tier tool-capture fidelity edge (diagnosed, not a check bug)

Diagnosed the flaky tool-call capture from the dogfood: the full stream shows the
mock's scripted Write tool_use is never surfaced ("tool_use" absent, only the
text turn, plus a spurious rate_limit_event from a LOCAL mock) — a mock↔claude
desync with the current version, isolated to a BARE runHarness. The established
pluginDir configs capture tools fine (the Skill test runs + passes here).
Confirmed it's not permissions (acceptEdits didn't help, reverted), not the parser
(tool_use isn't in the stream), and not the check API. Noted in Part 7 gap #5 as a
harness-fidelity backlog item; the reliable observables (hooks/turns/output/
received) are unaffected.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* docs(harness-testing): make the full check vocabulary + measureArms discoverable

The guide had the Checks intro but not the vocabulary itself. Added a scannable
reference table (tool/skill/mcp/subagent/output/received/hookFired/turns/wrote/
judged/cost-latency-tokens/blocked-allowed → holds-when → over-which-result),
plus a brief measureArms+compareCheck (A/B with Welch significance) and
propertyHook note. So a user can discover the new capabilities, not just read
that they exist.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* fix(harness): match subagent dispatch by subagent_type input, not tool name

A real `claude` run revealed the subagent dispatch tool is named `Agent`
on the live CLI, not `Task` as older docs say. parseSubagents now keys off
the `subagent_type` input field (present on the dispatch tool_use regardless
of its name) so it survives the rename. Validated end-to-end against real
claude output, which recovered the dispatched subagent and its tool calls.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* feat(eval): support stubSkillBodies in measure for cheap skill-firing checks

measure() now takes stubSkillBodies (the same body-stub the trigger tier
uses, via a shared stubbedPluginDir helper): a skill-FIRING check (skill())
stops the selected skill at selection instead of running its often-expensive
procedure, so a description/firing run costs a fraction of the tokens.
Requires pluginDir; throws if combined without one. Not for judged/quality
checks — the body is gone. Refactors resolveTriggerPluginDir to reuse the
extracted stubbedPluginDir helper. Unit-tested (stubbed body in the packaged
plugin, cleanup after, the no-pluginDir guard).

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* chore(types): regenerate project-file types to include src/check

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* docs(dogfood): add cheap firing-only eval using measure + stubSkillBodies

A shipped dogfood example for the body-stub path: measure({ checks: [skill,
latency], stubSkillBodies: true }) on the strengthen skill, the cheap sibling
of skill-quality.eval.mjs (which keeps the body to grade quality). Makes the
~18x firing-check saving discoverable, not just unit-tested. Smoke-run against
real claude: both checks 100% (skill fires, run stays fast).

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* feat(eval): support stubSkillBodies in measureArms for cheap A/B firing tests

Brings measureArms to parity with measure: an A/B firing comparison (does
description variant A fire more reliably than B?) now stubs each arm's skill
bodies via stubSkillBodies. Every arm that sets a pluginDir is repackaged with
bodies stripped (frontmatter kept) through the shared stubbedPluginDir helper,
so each run stops at selection — a fraction of the tokens, per-arm. Arms without
a pluginDir pass through untouched; throwaways are cleaned up in a finally.
Not for judged/quality checks — the body is gone. Unit-tested (both arms see a
stubbed throwaway, originals untouched, cleanup after).

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

* test(eval): restore 100% coverage for the stubSkillBodies + check surfaces

CI's coverage gate (100% lines/functions/statements on the harness-testing
files) flagged uncovered paths accumulated across the revamp: the measure/
measureArms stubSkillBodies branches and their helpers (stubbedPluginDir,
skillsDirOf .claude/skills fallback, pluginName missing-manifest catch,
packageSkillsDir missing-dir throw + non-directory entry skip), checkLabel's
no-arg fallback, and the toPass/toPassAll matchers. Adds targeted unit tests
for each — all tests still pass, coverage back to 100%.

https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-06-16 21:29:30 +07:00

39 lines
1.2 KiB
TypeScript

/* eslint-disable max-params, @typescript-eslint/no-explicit-any --
The matcher signatures mirror the runtime vigilesMatchers (positional args);
`Check<any>` matches the runtime generic for the type-only augmentation. */
/**
* vigiles — jest integration (opt-in).
*
* Importing this entry registers the vigiles matchers AND augments jest's types
* so `toHaveCreated` / `toBeatBaseline` type-check.
*
* // jest.config.js → setupFilesAfterEnv: ["vigiles/jest"]
* // …or at the top of a test file:
* import "vigiles/jest";
*
* expect(result).toHaveCreated("DONE");
* expect(report).toBeatBaseline("vanilla", "gated", "caught");
*
* jest is an optional peer dependency — only jest users load this entry.
*/
import { expect } from "@jest/globals";
import { vigilesMatchers } from "./harness-assert.js";
import type { Check } from "./check.js";
expect.extend(vigilesMatchers);
declare module "@jest/expect" {
interface Matchers<R> {
toHaveCreated(path: string): R;
toBlock(): R;
toBeatBaseline(
baseline: string,
arm: string,
metric: string,
by?: number,
): R;
toPass(check: Check<any>): R;
toPassAll(checks: readonly Check<any>[]): R;
}
}