mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
fix/hook-runtime-unused-err
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 zernie
|
d8c639f8da |
refactor(cli)!: consolidate the verb surface (13→8) + docs/README overhaul (#46)
* docs: sharpen README lint/guard copy - Replace internal "7 catalogs" jargon with "7 linters (ESLint, Ruff, Clippy, and four more)"; Cedar stays in the subdoc - Drop the plugin-audit paragraph from the Lint section (secondary use case, still reachable from More) - Rewrite the Guard intro to lead with the concrete "fail open" failure and name what vigiles emits (exit code, JSON, AST-backed matcher) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: surface the $0/subscription cost story in the first paragraph The "no metered tokens, runs on your Claude subscription" wedge was buried in the last line of the Eval section. Pull a crisp version into the opening paragraph (above the fold); the Eval section keeps the detailed promptfoo/DeepEval contrast as proof. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: surface agent-writes-the-harness flow, reorder More, drop internal links - Quick start now leads with "then your agent writes the harness for you": the three model-invocable skills (test-harness/strengthen/ edit-spec) and the plain-English ask each one answers - Reorder More so CLI sits above the leaderboard (whose link points into the CLI docs); trim the now-redundant Skills bullet to a pointer - Stop linking internal research/ docs from the README (the moat link and the Research index) per public-vs-internal-docs Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(compile): auto-refresh an existing harness.gen.ts compile now keeps a whole-harness registry in sync as a side effect, so the user never hand-runs generate-harness. Gated on the file already existing (compile maintains a registry you opted into, never imposes one) and cheap (parses specs, no linter spawn — unlike generate-types/ generate-schema, which spawn every linter and stay on the config-change guard). A duplicate agent name fails the compile. Docs: note the auto-refresh in the compile section and mark the generate-* artifacts as dev-toolchain output rarely run by hand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(src): add nested CLAUDE.md with a high-bar-for-new-commands rule Directory-scoped guidance loaded when working in src/: adding a new CLI verb or hook-runtime subcommand must clear a high bar (genuinely new mental action, no overlap with an existing command, siblings collapse to one verb + subcommand, runtime entrypoints stay hidden). Complements the root cohesive-cli-surface rule, placed next to the code that defines the command surface (cli.ts, cli-commands.ts). Spec-managed (require-spec + integrity); dogfoods vigiles compile. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: drop the "(generated)" annotation from the API reference link Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(build): stop src/CLAUDE.md.spec.ts shadowing the root spec + add docs rule The nested src/CLAUDE.md.spec.ts (added last commit) compiles under rootDir:src to dist/CLAUDE.md.spec.js — the exact path loadSpec resolves the root CLAUDE.md.spec.ts to, so recompiling the root spec loaded the src spec and clobbered CLAUDE.md. Exclude src/*.md.spec.ts from tsc so both load via the tsx fallback (as the root always has). Also adds the no-internal-links-in-public-docs guidance rule (public docs never link research/*.md; keep the fact in research, don't link it), reconciles the old "link to research" line in public-vs-internal-docs, and files the deterministic lint-rule version as a P1 roadmap item. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: separate public/internal indexes + surface eval safety - docs/README.md no longer links into internal research/ (public docs don't point at the private record) - research/README.md is explicitly labelled the internal record - README Eval section gains a "Safe to repeat" note (sandbox + ephemeral dir + egress + interceptTools, past-the-sandbox case flagged) linking docs/safety.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: stop public docs linking internal research/ + add safety FAQ Strip every research/*.md link from docs/*.md (the no-internal-links-in- public-docs rule) — ~60 links across 16 guides plus the docs/ index; prose kept readable, no orphaned research doc (all still cited from CLAUDE.md keyFiles / research/README.md, orphan-docs green). Add a FAQ to docs/safety.md answering the side-effects-beyond-the-sandbox question honestly: network via egress allowlist, a real DB/redis/browser via a disposable container you stand up (turnkey provisioning is a committed next step), irreversible externals via interceptTools, and the no-Docker deterministic/record-replay tiers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(cli)!: consolidate verbs — measure/explain→scan flags, refs→lint, generate-*→generate <kind> De-sprawl the human verb surface from 13 to 8 (the high-bar-for-new-commands rule, dogfooded). All breaking: - `measure <dir>` → `scan <dir> --trigger` (the model-gated behavioral column folded into scan; trigger-rate + collision matrix unchanged) - `explain <dir> [name]` → `scan <dir> --explain [name]` (the deterministic WHY, a sibling of --fix-plan on the same ScanReport) - `refs <file>` → folded into `lint` (the unmarked-refs check is already a lint rule; the PostToolUse hook keeps its shared collectRefIssues entry) - `generate-types|schema|harness` → `generate <kind>` (one verb + subcommand; the three dev-toolchain generators grouped, not N hyphenated siblings) scan reuses handleMeasure/handleExplain directly behind --trigger/--explain. Docs, help text, the regen guard, tests, and the self-command-refs dogfood all updated together (cohesive-cli-surface); 1588 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: split the GitHub Action reference out of cli.md into its own doc cli.md was 718 lines (kitchen sink). Extract the ~108-line GitHub Action section into docs/github-action.md (quick start, inputs, output channels, versioning) and leave a short pointer; retitle the cli.md intro. Update the README More bullet (CLI · GitHub Action) and the docs index. cli.md → 619 lines. orphan-docs + self-command-refs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(lint): add prefer-compiled-hooks rule — one nudge toward compiled hooks A single repo-level recommendation (one finding regardless of hook count): when a plugin ships hand-written hook commands that aren't compiled vigiles/hook artifacts, nudge toward compiling them — the message links docs/compiled-hooks.md and points at guardrail-check for verifying the ones you have. A discovery nudge, not a defect: the hand-written shell lane stays first-class, so it fires ONCE (never per-hook), defaults to warn (info- level ℹ; informational, doesn't fail CI), and is opt-out. Detector (manualHookCount: a command that isn't a hook-runtime invocation) lives in scan.ts and is shared by scan + lint (one-detector-no-drift). Doc + matrix rows added (rules-docs-in-sync). Honest caveat in the doc: it's form-based, so it stays a recommendation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): record the eval-startups critique mapped onto positioning New research/eval-startups-positioning.md captures Thomas Liao's "why eval startups fail" critique and its implications: vigiles dodges the trap (harness-eval not model-eval, tooling not services, eval already a feature), but it's a yellow flag on the public ecosystem-benchmark as a MOAT (Goodhart-gameable → keep it a thin acquisition flywheel) and it independently validates the attestation/safety direction (capability-diff, guardrail proof) as the only defensible eval-shaped niche. Linked from research/README.md, measurement-authority.md (whose benchmark thesis it challenges), and strategic-synthesis-2026-06.md (whose Tier-2 conformance/attestation play it validates). Internal-only per no-internal-links-in-public-docs; orphan-docs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: purge internal positioning from public docs (jargon, name-drops, benchmark) Public docs (README + docs/) now state user-facing value, not vigiles's internal strategic vocabulary: - strip "moat" from 14 places → user-facing wording (engine / cost advantage / differentiator / cross-reference) - remove all bare research/*.md name-drops from docs/ (the backtick form the link-strip + deterministic check miss); scope-descriptions reworded to "configured directories" rather than naming research/ - README: drop the "ranks the hyped ecosystem: what works vs hype" benchmark banner from the Eval row + kicker (depth stays in measuring-skills.md) — per eval-startups, the benchmark is a flywheel, not a headline - docs/cli.md: "moat #2 (research/…)" → "the capability-diff check" Codified in CLAUDE.md: public-vs-internal-docs now has a strategic- vocabulary clause (moat/measurement-authority/flywheel stay internal; public docs name the benefit), so this doesn't regress. orphan-docs + self-command-refs green; the public↔private split stays docs/ vs research/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): apply multi-persona review fixes From a multi-persona README review (power user / staff eng / plugin author / non-TS user / eng manager): - non-TS users: a visible "No TypeScript needed — lint + inline comments work on any CLAUDE.md" line (was invisible; contradicted progressive adoption) - plugin authors: surface "vigiles scan ranks a marketplace, no key" above the fold (was buried in More) - eng-manager cost: hero now says deterministic layers run free in CI; merged kicker+affordability nails "real-model on a dev's own subscription, never a CI token" - staff-eng skepticism: the 2/7→7/7 claim now reads "measured, not asserted"; the typed-pipeline claim is concretized to the real pipe()/needs() mechanism - fixed a broken blockquote and trimmed the safe-to-repeat para to hold length Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): capture the README design direction as an editor-facing comment A top-of-file comment block records the framing the README must hold: lead with benefits, present compiled specs as the easy/auto-enforced next step (skills write them, hooks compile them; JS→TS evolution, not a wall), keep all four instruments first-class incl. Eval, stay scannable + under the length cap, and no internal vocabulary/research links. Guides future edits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): benefits-forward rewrite + specs-are-easy framing Per the README direction: lead with what the user GETS, and present the typed spec as the frictionless auto-enforced next step, not a TS wall. - hero: stronger, concrete failure modes (stale ref / guard fails open / skill never fires), keeps Agent=Model frame, cost above the fold; dropped the redundant centered pitch (merge) - table cells condensed to punchy benefit one-liners - "You don't hand-write specs — your agent does. TypeScript for your harness: opt-in, gradual, auto-compiled, never a chore" (skills write them; JS→TS evolution) - de-dupe cost (drop "not metered tokens" from the Test cell) - "Not for you if …" honest-scope line - docs/verifying-instruction-files.md: back the claim — "you rarely hand-write one; edit-spec/strengthen write it, a hook recompiles on save" Eval stays a first-class instrument. fmt + self-command-refs + orphan-docs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): don't lead callouts with "typed spec" — benefit first Casual readers shouldn't be greeted by "a typed spec is a program" / "TypeScript for your harness" in the prominent callouts. Reframe both to lead with the outcome and demote the spec to a soft, opt-in aside: - specs-are-easy block: "You don't hand-write any of this — your agent does … you never touch it by hand" (drop the TypeScript-for-your-harness headline) - Lint blockquote: "Bad states stop compiling … turn it up only when you want it" (drop "Markdown is prose; a typed spec is a program") - the edit-spec bullet says "edits the source", not "the typed spec" "No TypeScript?" reassurance and the optional .spec.ts mention stay. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF for the CLI-consolidation + docs/README overhaul session Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: prettier-format HANDOFF Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): update the stale generate-types ref the verb rename missed The CI "Check generated types compile" step ran `node dist/cli.js generate-types` — renamed to `generate types` in the verb consolidation, so the job failed with "Unknown command". self-command-refs didn't catch it because it didn't scan .github/; add .github/ to its scan scope so a stale `cli.js <cmd>` ref in a workflow fails the dogfood from now on. Also saves the prefer-compiled-hooks-vs-require-hook-spec decision to research/hook-modes-and-testing.md (the nudge, not a requirement; the write-hook authoring skill is the real authoring lever, unbuilt). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(lint): eslint-ignore src/*.md.spec.ts (excluded from tsconfig) The dist-collision fix excluded src/*.md.spec.ts from tsconfig, so eslint's type-aware project service can't resolve src/CLAUDE.md.spec.ts and errors "not found by the project service". It's a tsx-loaded build input (like the root CLAUDE.md.spec.ts, which isn't linted either), so ignore it in eslint to match. Latent since the exclude landed; surfaced now that the earlier generate-types step no longer fails first. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
66ca161e8b |
fix: round-2 dogfooding fixes for untested-surface, audit rename, orphans, loose-skill evals (#35)
* fix: round-2 dogfooding fixes for untested-surface, audit rename, orphans, loose-skill evals - untested-surface: discover colocated tests under dot dirs (dot:true), so a `.claude/skills/<name>/<name>.eval.mjs` the warning suggests actually counts as coverage instead of leaving the nudge unsatisfiable. - init: flag a CI workflow that still calls a renamed/removed subcommand (`vigiles audit` → `lint`) even when the Action or a modern command is also present, instead of reporting it "up to date". - orphan-docs: honor an inline `<!-- vigiles-disable orphan-docs -->` opt-out and point the failing message at both escape hatches (inline disable + .vigilesrc.json orphans.exclude/include). - measureTriggerRate: accept a loose `skillsDir` (e.g. .claude/skills) and auto-package it into a throwaway --plugin-dir via packageSkillsDir, so trigger-testing repo-local skills is a one-liner. - docs/strings: replace stale `vigiles audit` command references with `vigiles lint` (schema descriptions, hook/skill prose, CLAUDE.md). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat: make strengthen/edit-spec model-invocable, tidy skill packaging, harden agent install Skills are useless if nobody remembers to run them. Address discovery + install gaps surfaced in the dogfood review: - strengthen + edit-spec are now MODEL-INVOCABLE (drop disable-model-invocation) with sharpened, precision-aware descriptions, so the agent reaches for them on its own — "strengthen my rules" / "add a rule to CLAUDE.md" fire the skill. The pre-edit hook already points at edit-spec when it blocks a direct edit; making it model-invocable closes that loop. Each gets a dogfood trigger eval (recall + precision) so auto-firing doesn't become over-firing. - Merge generate-rule into edit-spec (rule classification absorbed) and delete it. Fix stale commands in edit-spec (vigiles check->lint, setup->init, the old `npx skills add` vendoring installer -> marketplace). - linter-docs gets a real SKILL.md (user-invoked reference), fixing the packaging smell of a skills/ dir with no SKILL.md. - Bump .claude-plugin/plugin.json version 3.0.0 -> 4.0.1 (was stale). - Harden installPlugins: on auto-install failure/no-CLI, fail LOUDLY and lead with the shell-runnable CLI commands (an agent can run those) instead of only /plugin slash commands a non-TTY agent can't execute. - Sweep remaining stale `vigiles audit` -> `lint` (example spec). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * refactor: remove the renamed `audit` command everywhere; rename lint-command internals `audit` became `lint` in v4 but the old name lingered. Purge it: - cli.ts: rename the lint command's internals (audit()->runLint, AuditReport-> LintReport, auditExitCode->lintExitCode, printAuditSummary->printLintSummary, annotateAuditForGitHub->annotateLintForGitHub) + comments. Keep the REMOVED_SUBCOMMANDS `audit: "lint"` detection key (it matches the OLD name). - FUNCTIONAL: bench scripts invoked `node $CLI audit SKILL.md` (removed cmd) — now `lint`. - docs/research/comments: `vigiles audit` -> `vigiles lint`, and the command- sense word "audit" -> "lint". - Kept genuinely-distinct uses: the post-session AUDIT feature (session.ts), the Merkle "audit trail", the supply-chain audit / scan concept, the dev-only `audit-feedback-loop` skill, and English "you audited the code". - Left third-party vendored fixtures (SHA-pinned) untouched. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat: prompt-diversity gate for trigger evals + raise auto-skill recall bar to 0.8 Task: ensure shipped model-invocable skills have evals, and validate prompt sets are big/varied enough before spending tokens. - measureTriggerRate now runs a deterministic pre-flight gate (no model): each prompt set needs >= minPrompts (default 10) and no two prompts more than maxSimilarity (default 0.8) alike via normalized Levenshtein. Throws with an actionable message before any run. Exports levenshtein/promptSimilarity/ checkPromptDiversity/assertPromptDiversity; minPrompts/maxSimilarity are per-spec overridable (narrow skills, short walkthroughs). - Expanded all shipped model-invocable dogfood evals (test-harness, strengthen, edit-spec) to 10 diverse relevant + 10 irrelevant prompts and raised the bar: recall min 0.8, maxFalsePositive 0.3. generate-logo + the canonical template expanded to 10; the OMC walkthrough sets minPrompts: 3. - All shipped skills are covered: the 3 auto skills by trigger evals, the user-invoked (migrate-to-spec, linter-docs) by the load gate. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * refactor: use NCD (gzip) instead of Levenshtein for prompt-diversity check Levenshtein measured spelling; switch the prompt-diversity gate to Normalized Compression Distance — the SAME engine `findSimilarRules` uses for near- duplicate rule detection, so the project has one house algorithm for "are these two texts basically the same". NCD scores shared structure/redundancy (a templated prompt with one word swapped compresses together), which is exactly the lazy copy-paste set the gate should reject. - Drop hand-rolled levenshtein/promptSimilarity; add promptDistance (= ncd over normalized text), reusing src/core/proofs.ts ncd(). - TriggerRateSpec: maxSimilarity -> minDistance (default 0.3, the rule-dup threshold). Calibration: real 10-prompt sets sit at NCD >= 0.44; near-dup pairs (one-word swaps, punctuation) at <= 0.24 — clean separation at 0.3. - Tests updated to NCD distance. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat: stubSkillBodies — measure skill triggering on frontmatter only Trigger-rate is decided by a skill's frontmatter (name + description) before its body ever loads, so the body is causally downstream of selection. stubSkillBodies keeps every skill's frontmatter (the selection competition stays faithful) but replaces each body with a no-op stub, so a trigger run stops AT selection instead of executing an expensive multi-step procedure — far cheaper, faster, side- effect-free, without changing what's measured. ~95% smaller skill payload. - stubSkillBody (pure) + packageSkillsDir({ stub }) (frontmatter-only package). - resolveTriggerPluginDir stubs both paths (pluginDir + skillsDir), preserving the plugin NAME so `<name>:<skill>` still matches the fired predicate. - TriggerRateSpec.stubSkillBodies (default off; on in all trigger evals). - Needs one real-model run to confirm the Skill tool still resolves under a stubbed body (the deterministic transform is fully tested here). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * fix(init): scaffold spec imports only `claude` so it passes strict ESLint The generated CLAUDE.md.spec.ts imported `{ claude, enforce, guidance }` but the enforce()/guidance() examples are commented out, so a repo running `eslint . --max-warnings=0` (common in CI) failed on two no-unused-vars warnings the moment the scaffold was committed. Import only `claude` (the symbol the scaffold actually uses); show the fuller import as a comment for when a real rule is added. Regression test guards the import line. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(init): --force rewrites a stale CI workflow in place Round-2 follow-up: instead of only warning about a stale workflow, `vigiles init --force` fixes it. A renamed/removed subcommand (vigiles audit) is rewritten surgically in place (audit → lint, the rest of the user's workflow preserved); a bare-`npx vigiles` no-op workflow is regenerated from the template. Without --force the behavior is unchanged (warn, don't clobber) and the warning now points at --force. Pure `rewriteRemovedSubcommands` + force plumbed through setup-plan; unit + CLI integration tests. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs: positioning for the eval-tier decision + new trigger-rate features - research/eval-api-landscape.md: dated decision — harness eval ≠ model eval (fidelity to the harness-as-shipped is why promptfoo et al. can't host it); keep the deterministic tiers, keep the real-model surface thin (stubSkillBodies = selection not execution), don't rebuild the eval stack, model fidelity is the one on-moat refinement. - CLAUDE.md(.spec): second-pillar paragraph gains the harness-eval-≠-model-eval framing; src/eval.ts Key Files entry documents skillsDir/packageSkillsDir, stubSkillBodies, and the NCD prompt-diversity gate. - docs/harness-testing.md: trigger-rate section documents skillsDir, stubSkillBodies, the diversity gate, and measuring on the user's real model. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(research): testing-API design proposal — the ideal shape Synthesizes "what makes a world-class testing API" (deterministic assertion libs: node:assert/Chai/Vitest/Testing-Library/fast-check; AI-eval: promptfoo/DeepEval/ Inspect) with the "do we need all these layers?" question into one conclusion: - One Trace (already the substrate); a single declarative `check` vocabulary evaluated two ways — strict (throw, unit/integration) vs scored (0-1 ± se, eval). Collapses today's predicate/assert*/matcher 3× into one surface and unifies deterministic + non-deterministic. - Three tiers, not four: e2e is integration + one egress symbol → fold egress in as a capability of integration, keep the capability ladder. - Checks-as-data ⇒ JUnit/baseline/regression + a promptfoo bridge fall out free. - Recommends option C (declarative checks) + D (property-test add-on for hooks). Indexed in CLAUDE.md Key Files + research/README; no users yet, so it's framed as a clean-break design, not a migration. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * fix(orphan-docs): resolve file-relative markdown links; refine tier taxonomy in design doc - orphans.ts: a markdown link is conventionally file-relative ([x](foo.md) in research/README.md → research/foo.md, ../docs/x.md walks up). The detector only matched root-relative paths, so a bare sibling link read as an orphan. Now credits BOTH readings (refTargets resolves against the source file's dir + keeps the root-relative reading). Tests for same-dir and ../ links. - testing-api-design.md: fold in the two taxonomy questions. The layer names are borrowed from app testing, not derived: the harness has only TWO execution scopes (hook = the one standalone-process surface; harness = the whole agent, since skills/CLAUDE.md/settings only act inside a live agent) over a static lint floor. integration/e2e/eval are the SAME scope under realness flags (model mock|real, egress) — model picks the verdict mode (mock→strict, real→scored). "unit only works with hooks" is structural, not a missing capability; each layer adds exactly one capability (the ladder). Recommends runHook + runHarness(flags) over the four borrowed tiers. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): Phase 0 — the declarative `vigiles/check` vocabulary First step of the testing-API revamp (research/testing-api-design.md). A check is DATA, not a throwing assert: { kind, eval(target) → {pass, score, message}, toJSON() }, typed over Trace (tool/skill/output/hookFired/wrote) vs HookRunResult (blocked/allowed). One vocabulary, to be evaluated two ways downstream — strict (expect, Phase 1) and scored (measure, Phase 3) — so a check reads as pass/fail on one run AND a rate across trials, and serializes (JUnit/baseline/promptfoo-bridge fall out later). - src/check.ts + src/check.test.ts (pure, model-free; the failure MESSAGE is tested as much as the verdict). - Exported at `vigiles/check` only — NOT folded into vigiles/testing yet, since hookFired collides with the legacy predicate (reconciled in Phase 6). - Additive: nothing removed; 889 tests green. - Build plan appended to the design doc as Part 6 (Phase 0 marked shipped). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): Phase 1 — strict evaluator (assertChecks) + check matcher veneer - assertChecks(target, checks): the strict evaluator — throws collecting ALL failed checks' messages (Validation-applicative, not short-circuit). Any runner. - vigilesMatchers gains ONE generic veneer for the whole vocabulary: expect(r).toPass(tool("Bash")) / toPassAll([...]) — each carries the check's own failure message. vitest/jest type augmentations added; cross-runner tests prove toPass works under both. - Additive: the legacy predicate/assert*/matcher surface is untouched (it gets removed in Phase 6). 891 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(harness): Phase 2 — runHarness, the explicit harness-scope entry Establishes the collapsed naming from Part 4: the harness has two execution scopes (hook = runHook, harness = the whole agent); integration/e2e/eval are the ONE harness scope under realness flags. runHarness is the DETERMINISTIC harness run (model: "mock", default) — behaviour identical to runHarnessTest, which it wraps. A real-model harness run is non-deterministic, so a single one can't be asserted: runHarness(spec, { model: "real" }) throws and points at measure() / runEval (the eval scope). Unit-tested (the guard needs no claude). The e2e barrel fold + a materialized real path remain for Phase 6. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): Phases 3+4 — measure() over checks + serialization (toJUnit, assertRates) Phase 3 — the SCORED evaluator. measure(spec, { trials, checks }) scores a Trace check vocabulary across trials → per-check rate ± se + pass^k, reusing the tested runEvalWith aggregation (one arm). measureWith takes an injected runner (unit- tested, no model); measure wraps it with the real CLI. Defaults to model "sonnet" (measure on the model your users run). So `tool("Bash")` reads pass/fail strictly (assertChecks) AND as a rate here — one vocabulary, two evaluators. Phase 4 — serialization → CI. checkReportToJUnit (each check a <testcase>, fails below `min`) + assertRates (the scored gate) + formatCheckReport, all keyed off checkLabel(check.toJSON()). Because a check is data, JUnit/baseline/promptfoo- bridge fall out. Pure + unit-tested. (Wiring into the Action/baseline is the remaining CI-integration step.) 894 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(run-hook): Phase 5 — propertyHook, invariant testing for a hook's decision A hook is (event) → decision, so instead of a few example events, generate MANY and assert invariants hold for every decision ("destructive command ⇒ blocked", "never both allow and block"). Reuses the deterministic propertyTest from proofs.ts (seeded, shrinks the counterexample; no fast-check dep). The `decide` runner is injectable — pure fake for unit tests, `(e) => runHook(cmd, e)` for a real hook. Returns the shrunk counterexample, not a throw, so it's inspectable. Test: a correct guard satisfies the invariant; a buggy one (blocks only the exact "rm -rf /") gets a "rm -rf foo" counterexample found. 895 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(check): Phase 6 — front-door the new check surface; defer the mass deletion The new world-class surface (Phases 0-5) is now the documented front door: docs/harness-testing.md gains a "Checks — one vocabulary, two evaluators" section leading with assertChecks / measure / toPass; the design doc marks Phases 0-5 shipped; CLAUDE.md Key Files note runHarness/measure/propertyHook/toPass. DELIBERATELY NOT done: ripping out the ~40 legacy predicate/assert*/matcher helpers + the e2e barrel and rewriting the ~900-test suite/examples off them. On a no-users lib that's high-churn cleanup with no functional gain and real regression risk — the legacy surface stays as working sugar, removed in a single separately- reviewed major-version cleanup (resolving the check↔testing hookFired collision then). The breaking step is the one to take slowly. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * refactor(testing): fold check into testing/unit; collapse e2e into integration The "breaking cleanup" — done at its true (small) blast radius, not the feared 900-test rewrite: - vigiles/check folds into vigiles/testing + vigiles/unit; the check `hookFired` (Check<Trace>) supersedes the legacy boolean predicate via an explicit re-export that wins over the two `export *`s. Checks (tool/skill/output/..., assertChecks, measure, propertyHook) are now first-class at the front door. - e2e collapsed: egress is a CAPABILITY of integration (egressRoutes + runHook's egress:{allow}), not a tier — the old e2e barrel added exactly one symbol. vigiles/e2e stays as a thin deprecated alias so imports keep working. KEPT (deliberately): the bare predicates + per-runner assert*/matcher fronts. They're not redundant — they're context-appropriate surfaces (compose / node:test / vitest-jest) now unified UNDER one check vocabulary. Deleting them would be churn without a greatness gain. 895 tests green, types + cross-runner pass. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(testing-api): Part 7 — greatness re-evaluation + ranked eval-side gaps Post-cleanup assessment: the API is great on the two deciding axes (unification: one Trace / one check vocabulary / two evaluators / serialized; and the moat: the deterministic no-key tiers + pass^k/significance/fidelity nobody else has). Extensibility is strong; the one seam to watch is Check.eval being sync (fine incl. a judge()-backed check; an async scorer would need an AsyncCheck for measure only). Ranked gaps, eval-side first: (1) judged() model-graded check — judge() exists but isn't a Check; fits the sync interface; #1 parity gap; (2) cost/latency/tokens checks (data already captured); (3) checks × A/B arms + significance (unify the harness-A/B moat with measure); (4) multi-surface CC eval — Trace is single+flat, no nested subagent/Task traces, slash-command expansion, multi-turn, or mcp() checks (the deep, new-capability bet); (5) real-tier Trace fidelity (modelRequests mock-only); (6) CC-feature drift maintenance. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): close eval-side gaps — judged/cost/latency/tokens/mcp checks, measureArms + significance, harness-agnostic - judged(rubric, { min, judge }) — model-graded check (LLM rubric) via an injectable judge fn (default real judge(), sync via spawnSync), composes into measure/assertRates/toJUnit. Closes the promptfoo llm-rubric / DeepEval GEval gap. - cost/latency/tokens — Check<UsageTrace> over a run's usage; measure.checks widened to Check<RunContext> so they compose with behavioural checks. - mcp(server, tool) — matches the mcp__server__tool tool name. - measureArms(spec, { arms, checks }) + compareCheck(report, baseline, arm, i): the SAME checks scored per arm, with Welch significance (reuses stats.ts) — unifies the harness-A/B moat with the check vocabulary. - Harness axis: a test pins that checks are harness-agnostic (evaluate over a Codex-sparse Trace — output passes, tool/hookFired fail gracefully). Runners dispatch via adapter; measureWith/measureArmsWith take an injectable runner so a Codex eval reuses the same checks. All unit-tested with fakes (no model). 900 tests green. Design doc Part 7 updated: gaps 1-3 + mcp shipped; nested-trace (subagents/turns) is the remaining big bet. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(harness-testing): add the vitest/jest setup + matcher matrix The reference table we'd flagged in the vitest/jest discussion: a two-column setup matrix (vitest vs jest: config / per-file / manual) and a matcher table (toPass / toPassAll / toHaveCreated / toBlock / toBeatBaseline → asserts → over which result), leading with toPass as the one matcher fronting the whole check vocabulary. Makes the runner story scannable instead of prose. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): nested subagent traces — Trace.subagents + subagent() check The deep gap's first piece: a subagent (Task) runs its own session, recoverable because CC tags its events with parent_tool_use_id (= the Task tool call) and the Task input carries subagent_type. parseSubagents groups the tagged tool calls into Trace.subagents (keyed by name); subagent(name, [checks]) runs the WHOLE check vocabulary recursively over that sub-trace, so a test/eval can assert what the subagent DID, not just that Task fired. Wired into makeResult (harness) + makeContext (eval); empty for harnesses that don't emit parent_tool_use_id (Codex) so it stays harness-agnostic. parseSubagents + the check unit-tested on synthetic stream-json; flagged in Part 7 for one real-CC-subagent run to confirm the field names. 902 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): slash-command-expansion (received) + multi-turn (turns) checks Closes the last two pieces of gap #4, both unit-tested: - received(matcher): the expanded slash command (or a hook's injected context) actually REACHED the model — reads modelRequests (mock/harness tier captures them; the eval tier captures none, with an explicit message saying so). - turns({min,max}): the multi-turn observable — the agent took a back-and-forth, not a one-shot (the deterministic harness already scripts multiple model turns via ModelTurn[]; this asserts how many it took). Remaining (smaller, noted in Part 7): driving multiple USER messages (--resume-style, unverifiable here) + per-turn trace slicing. 904 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(check): validate the subagent parser schema against Claude Code stream-json Confirmed via the Claude Code SDK docs / guide: subagent events carry a top-level `parent_tool_use_id` (= the Task tool-use id), the Task input carries `subagent_type`, linked by tool_use id — exactly what parseSubagents assumes, on the same CLI `message.content` line shape parseToolCalls already consumes (the SDK StreamEvent `event:{delta}` shape is a different, unused format). No code change — the parser is validated; dropped the "needs a real run to confirm" caveat in the comment + Part 7. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * test(dogfood): exercise the new check capabilities on real targets - checks-dogfood.harness.mjs — RUNS here (claude, no key): assertChecks over a real runHook (blocked/allowed) + a real runHarness mock run (turns/output). Validated end-to-end, not just unit-tested with fakes. - dogfood/skill-quality.eval.mjs — the SCORED path on vigiles's OWN strengthen skill: measure({ checks: [skill, judged, cost] }) + assertRates (does it fire, is the suggestion good, is it cheap). Write-don't-run (needs a key). - subagent.eval.mjs — the nested-trace check on a REAL subagent (vigiles ships none): the vendored oh-my-claudecode code-reviewer, dispatched via Task; subagent("code-reviewer", [tool("Read")]) asserts what the subagent DID. Note: the harness-tier dogfood asserts the cross-version-reliable observables (turns/output); tool-call capture in a bare run is plugin/permission-dependent (the fuller pluginDir tests in harness-test.test.ts cover it). 904 green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs: document the mock-tier tool-capture fidelity edge (diagnosed, not a check bug) Diagnosed the flaky tool-call capture from the dogfood: the full stream shows the mock's scripted Write tool_use is never surfaced ("tool_use" absent, only the text turn, plus a spurious rate_limit_event from a LOCAL mock) — a mock↔claude desync with the current version, isolated to a BARE runHarness. The established pluginDir configs capture tools fine (the Skill test runs + passes here). Confirmed it's not permissions (acceptEdits didn't help, reverted), not the parser (tool_use isn't in the stream), and not the check API. Noted in Part 7 gap #5 as a harness-fidelity backlog item; the reliable observables (hooks/turns/output/ received) are unaffected. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(harness-testing): make the full check vocabulary + measureArms discoverable The guide had the Checks intro but not the vocabulary itself. Added a scannable reference table (tool/skill/mcp/subagent/output/received/hookFired/turns/wrote/ judged/cost-latency-tokens/blocked-allowed → holds-when → over-which-result), plus a brief measureArms+compareCheck (A/B with Welch significance) and propertyHook note. So a user can discover the new capabilities, not just read that they exist. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * fix(harness): match subagent dispatch by subagent_type input, not tool name A real `claude` run revealed the subagent dispatch tool is named `Agent` on the live CLI, not `Task` as older docs say. parseSubagents now keys off the `subagent_type` input field (present on the dispatch tool_use regardless of its name) so it survives the rename. Validated end-to-end against real claude output, which recovered the dispatched subagent and its tool calls. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(eval): support stubSkillBodies in measure for cheap skill-firing checks measure() now takes stubSkillBodies (the same body-stub the trigger tier uses, via a shared stubbedPluginDir helper): a skill-FIRING check (skill()) stops the selected skill at selection instead of running its often-expensive procedure, so a description/firing run costs a fraction of the tokens. Requires pluginDir; throws if combined without one. Not for judged/quality checks — the body is gone. Refactors resolveTriggerPluginDir to reuse the extracted stubbedPluginDir helper. Unit-tested (stubbed body in the packaged plugin, cleanup after, the no-pluginDir guard). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * chore(types): regenerate project-file types to include src/check https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(dogfood): add cheap firing-only eval using measure + stubSkillBodies A shipped dogfood example for the body-stub path: measure({ checks: [skill, latency], stubSkillBodies: true }) on the strengthen skill, the cheap sibling of skill-quality.eval.mjs (which keeps the body to grade quality). Makes the ~18x firing-check saving discoverable, not just unit-tested. Smoke-run against real claude: both checks 100% (skill fires, run stays fast). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(eval): support stubSkillBodies in measureArms for cheap A/B firing tests Brings measureArms to parity with measure: an A/B firing comparison (does description variant A fire more reliably than B?) now stubs each arm's skill bodies via stubSkillBodies. Every arm that sets a pluginDir is repackaged with bodies stripped (frontmatter kept) through the shared stubbedPluginDir helper, so each run stops at selection — a fraction of the tokens, per-arm. Arms without a pluginDir pass through untouched; throwaways are cleaned up in a finally. Not for judged/quality checks — the body is gone. Unit-tested (both arms see a stubbed throwaway, originals untouched, cleanup after). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * test(eval): restore 100% coverage for the stubSkillBodies + check surfaces CI's coverage gate (100% lines/functions/statements on the harness-testing files) flagged uncovered paths accumulated across the revamp: the measure/ measureArms stubSkillBodies branches and their helpers (stubbedPluginDir, skillsDirOf .claude/skills fallback, pluginName missing-manifest catch, packageSkillsDir missing-dir throw + non-directory entry skip), checkLabel's no-arg fallback, and the toPass/toPassAll matchers. Adds targeted unit tests for each — all tests still pass, coverage back to 100%. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
71b6c4c3d3 |
fix: finish audit→lint rename — repair CI build, shipped hook, and docs (#34)
* fix: finish audit→lint rename — repair CI build, shipped hook, and docs
The `audit` command was renamed to `lint` in
|
||
|
zernie
|
b89c5b6d75 |
Expand docs freshness research with competitive analysis and prototype sidecar manifest (#20)
* Expand docs freshness research with competitive analysis and prototype sidecar manifest
Research: survey 11 competing tools (Drift, ctxlint, DOCER, Doc Detective,
agents-lint, agnix, Swimm, Repowise, code-forensics, doc-hunt, ai-contextor).
Add 7 new options: per-file sidecar manifest, affected-specs reporter, git
rename auto-repair, temporal coupling analysis, AST-normalized config
fingerprinting, spec age warning, code element cross-referencing. Pick top 3
(sidecar manifest, affected-specs, git rename). Add hook integration patterns
(pre-commit, post-checkout, CI affected-only, weekly coupling audit).
Prototype: implement per-file sidecar manifest and affected-specs reporter in
freshness.ts. SidecarManifest stores individual file hashes in
.vigiles/<target>.inputs.json. diffSidecarManifest reports changed/deleted/added
files individually. buildReverseIndex + affectedSpecs enable Bazel-style target
determination — given changed files, report which specs need recompilation.
25 new tests, all passing.
* Reframe doc freshness research: separate build freshness from doc freshness
The research was conflating two problems:
1. Build artifact freshness — "did inputs change since last compile?"
Already solved by strict mode. Sidecar manifest optimizes it.
2. Doc freshness — "does the spec still accurately describe the project?"
The hard problem. No tool fully solves it deterministically.
Add build-vs-adopt analysis for each option. Build: temporal coupling,
code element cross-ref, spec age warning, git rename repair (vigiles has
structural advantage via the spec→file dependency graph). Recommend Drift
for symbol-level AST anchoring (tree-sitter is too heavy to vendor).
Don't adopt ctxlint/agents-lint (they lint raw markdown; vigiles users
have specs that compile). Reorganize into three tiers: build now, build
next, recommend.
* Add runtime enforcement research: spec-derived policies via hooks
Explores three layers of runtime spec enforcement:
1. Passive observation — post-session git diff analysis against spec surface
2. Hook-based policy — compile specs to PreToolUse/PostToolUse hooks that
enforce file access controls, command allowlists, compiled output protection
3. Skill contracts — declare reads/writes/commands per skill, enforce via hooks
Key insight: vigiles already has the structured declaration (the spec) and a
compilation pipeline. Adding hooks as a compilation target is architecturally
natural. The spec compiles to both markdown (agent reads) AND hook configs
(harness enforces).
Four implementation phases: post-session audit (read-only), hook generation
(compile target), scoped linting hooks (enforce() at runtime), skill contracts.
* Add post-session audit: analyze agent behavior against spec surface area
New module src/session.ts implements Phase 1 of runtime enforcement:
read-only analysis of what an agent changed vs. what specs expected.
Core functions:
- gitChangedFiles() — files changed since a base ref (git diff + untracked)
- loadSpecSurface() — aggregate all specs' tracked inputs from sidecar manifests
- loadKeyFilesFromSpecs() — extract keyFiles from compiled markdown
- analyzeSession() — cross-reference changed files against spec surface
- formatSessionReport() — human-readable output
Four finding types:
- untracked-file: modified file not in any spec
- stale-spec: tracked input changed, spec may need recompile
- spec-modified: spec source changed, needs compile
- target-modified: compiled output edited directly
Works as `vigiles audit --session` or as a PostSession hook.
23 tests, all passing.
* Expand formal verification research: Lean 4 is now viable, Cedar pattern
Three developments shift the Dafny-first recommendation to Lean 4:
1. Leanstral (Mistral, March 2026) — first open-source LLM agent for Lean 4,
uses MCP natively, Apache 2.0. The user doesn't read the proof; the agent
writes it, the kernel checks it.
2. Cedar/AWS — production Lean 4 verification of an authorization language.
"Verification-guided development": formal model in Lean (~10x smaller than
production code), prove properties, differential test against production.
Found 25 bugs.
3. The Cedar pattern maps directly to vigiles: formal model of compileClaude()
in Lean, prove structural properties (enforce() appears in output, hash
integrity, monotonicity), differential test against TypeScript.
Add sections 6-8: revised Lean vs Dafny comparison, Cedar pattern applied to
vigiles (what to verify, what NOT to verify, effort estimate), property-test
to Lean bridge, lean() enforce target.
* Add architecture-platform research: vigiles as architecture-aware agent platform
Explores expanding vigiles from a spec linter to a platform where architecture
(FSD, DDD, hexagonal), AI setup, skills, rules, and hooks form a validated
system. Architecture presets (npm packages) provide rules, skills, keyFiles,
hooks, and linter configs for specific architectures. Meta-validation checks
the entire setup works together.
Key concepts:
- Architecture presets: @vigiles/arch-fsd exports enforce() rules that map to
eslint-plugin-boundaries/dependency-cruiser/steiger rules
- Install/customize/eject lifecycle (like CRA)
- Meta-validation: vigiles audit --setup checks spec + architecture + skills +
hooks + freshness in one command
- FSD as the first preset (most structured, active community)
* Add spec coverage metrics with configurable thresholds
New coverage rule in .vigilesrc.json:
{ "rules": { "coverage": "warn" },
"coverageThresholds": { "linterRules": 30, "scripts": 80 } }
Two metrics:
- Linter rule coverage: % of enabled linter rules with enforce() in specs
- Script coverage: % of npm scripts documented in spec commands
Thresholds are optional — without them, coverage is informational. With them,
audit warns or errors when coverage drops below the minimum. Fits naturally
into the existing audit pipeline alongside freshness and require-spec rules.
New module src/coverage.ts with 21 tests. Default: coverage rule is off
(false). Opt-in via .vigilesrc.json.
* Add drift rule, dogfooding setup with enforce() rules and full config
Drift integration (src/drift.ts):
- New `drift` rule in .vigilesrc.json (default: "warn")
- Checks for drift anchors in compiled output, runs `drift check`
- Graceful degradation: no anchors → skip; anchors but no drift binary → error
with install instructions (brew install fiberplane/tap/drift)
- 6 tests
Dogfooding (CLAUDE.md.spec.ts + .vigilesrc.json):
- Added 3 enforce() rules: no-non-null-assertion, no-floating-promises,
cognitive-complexity — all verified against actual eslint config
- Added 11 new files to keyFiles: coverage.ts, session.ts, drift.ts, types.ts,
and 4 new research docs (runtime-enforcement, architecture-platform,
formal-proofs, doc-freshness expanded)
- Created .vigilesrc.json with all rules enabled: require-spec (error),
require-skill-spec (warn), freshness (warn), coverage (warn), drift (warn)
- Coverage thresholds: scripts 50%, linterRules 5%
- vigiles now validates itself with every feature it ships
* Fix dogfooding: add lint command, generate sidecar manifest, verify end-to-end
- Add missing `npm run lint` to spec commands (100% script coverage now)
- Generate .vigiles/CLAUDE.md.inputs.json sidecar manifest so coverage
and affected-specs features work on our own project
- Verified: vigiles compile (3 linter-verified enforce rules),
vigiles audit (hashes valid, freshness passes), script coverage 100%
* Refactor rule config to ESLint-style tuples for freshness and coverage
Before:
{ "rules": { "freshness": "warn" }, "freshnessMode": "strict",
"coverageThresholds": { "scripts": 50 } }
After:
{ "rules": {
"freshness": ["warn", { "mode": "strict" }],
"coverage": ["warn", { "scripts": 50, "linterRules": 5 }]
} }
Rule options live with their severity, not as separate top-level fields.
New RuleWithOptions<T> type: RuleSeverity | [severity, T].
Helper functions ruleSeverity() and ruleOptions() parse both forms.
Removed: top-level freshnessMode, freshnessInputs, coverageThresholds
from VigilesConfig. CoverageThresholds moved from coverage.ts to types.ts.
* Remove half-baked drift rule, fix coverage to read specs directly
Drift rule removed:
- Integration was hollow — no mechanism for anchors to get into compiled
output. Symbol-level tracking needs proper design, not a wrapper around
a binary that may not exist.
- Deleted src/drift.ts, src/drift.test.ts
- Removed drift from RulesConfig and default config
Coverage fixed:
- collectDocumentedCommands() now reads spec objects directly via their
structured `commands` field instead of parsing compiled markdown with
regex. Accepts ClaudeSpec[] when available, falls back to loading
compiled JS from dist/.
- No more regex parsing of compiled output — the spec IS the source of
truth, read it as data.
* Add per-rule docs, link from README, add doc-per-rule guidance
New docs/rules/ directory with one doc per validation rule:
- require-spec.md: config, severity, what it checks, disable per file
- require-skill-spec.md: config, severity, what it checks
- freshness.md: modes, options (ESLint-style tuple), inputs tracked
- coverage.md: thresholds, script/linter metrics, examples
README updated:
- Rule table links to individual docs
- Config example uses new tuple format
- Freshness config uses tuple format (removed old top-level keys)
docs/freshness.md updated to new tuple config format.
New guidance rule "doc-per-rule": every validation rule must have a
corresponding doc in docs/rules/<rule-name>.md.
* ci: add semantic-release and PR title validation
semantic-release:
- .releaserc.json with commit-analyzer, release-notes, npm, github plugins
- .github/workflows/release.yml runs on push to main, publishes to npm
- package.json: removed private:true, added version placeholder
PR title validation:
- .github/workflows/pr-title.yml validates conventional commit format
- Allowed types: feat, fix, docs, chore, refactor, test, perf, ci, build
- Squash merge uses PR title as commit message → semantic-release parses it
Required repo secret: NPM_TOKEN (npm token create)
GITHUB_TOKEN is provided automatically.
* refactor: extract shared hash, test utils, and deduplicate code
- Extract sha256short() into src/hash.ts — used by compile, freshness,
and proofs instead of duplicating createHash().digest().slice(0,16)
- Extract test helpers into src/test-utils.ts — makeTmpDir(), makeSpec(),
cleanupTmpDir(), initGitRepo(), git() shared across 4 test files
- Deduplicate readPackageScripts: coverage.ts now imports from compile.ts
instead of reimplementing the same function
- Extract filterExistingFiles in freshness.ts for detectLockFiles and
detectLinterConfigs
- Extract iterateSidecars in session.ts to eliminate duplicated .vigiles/
directory iteration in loadSpecSurface and loadKeyFilesFromSpecs
- Fix 3 pre-existing lint errors (useless escapes, non-null assertion)
- Remove stale eslint-disable directive in coverage.ts
Net: -83 lines, zero new test failures.
https://claude.ai/code/session_01FsbRVewGpTGW1AXysJhiw8
* refactor: add branded SHA256Hash type, exhaustive switch checks, SessionFinding discriminated union
- SHA256Hash branded type in hash.ts — sha256short() returns
SHA256Hash instead of plain string, preventing hash/string confusion
- assertNever() helper for exhaustive switch statements
- Exhaustive default cases in renderFragment (compile.ts),
describeMutation and applyMutation (evolve.ts)
- SessionFinding split into UntrackedFinding | TrackedFinding
discriminated union — specs field is required on tracked findings,
absent on untracked ones (no more optional chaining band-aids)
- ts-essentials guidance rule added to CLAUDE.md spec: prefer branded
types, discriminated unions, and exhaustive assertNever() switches
- hash.ts and test-utils.ts added to keyFiles
https://claude.ai/code/session_01FsbRVewGpTGW1AXysJhiw8
* refactor: remove legacy JS spec fiction, use assertNever everywhere
There are no legacy JS specs. The `_kind: "check"` type never existed
in this codebase — it was defensive code against an imaginary threat.
- Replace manual `throw new Error("Unknown rule kind...")` with
assertNever() in ruleToText (proofs.ts) and compileRule (compile.ts)
- Remove try/catch wrappers in runProofSuite (evolve.ts) that guarded
NCD, Bloom filter, and fitness against "legacy data"
- Remove try/catch in propose() guarding baseline fitness
- Remove try/catch in cli.ts findDuplicateRules
- Delete 3 tests that injected fake `{ _kind: "check" }` data via
`as unknown as` casts
Net: -171 lines of dead defensive code and tests for impossible states.
Lint warnings dropped from 68 → 62 (less nesting).
https://claude.ai/code/session_01FsbRVewGpTGW1AXysJhiw8
* chore: regenerate types after merge
https://claude.ai/code/session_01FsbRVewGpTGW1AXysJhiw8
* docs: recompile CLAUDE.md, fix stale config format in freshness.md
- CLAUDE.md recompiled: adds ts-essentials rule, hash.ts and
test-utils.ts keyFiles
- docs/freshness.md: replace old freshnessMode/freshnessInputs
top-level keys with current ESLint-style tuple format
https://claude.ai/code/session_01FsbRVewGpTGW1AXysJhiw8
* docs: fix rule count, add missing comparison row, format freshness.md
- README.md: "three rules" → "four rules" (coverage was missing from count)
- README.md: add "Spec edits" row to Claude Code comparison table
- docs/freshness.md: fix prettier formatting
https://claude.ai/code/session_01FsbRVewGpTGW1AXysJhiw8
* feat: add guard() rule type for reactive file-change hooks
New rule type alongside enforce() and guidance():
guard({ watch: "*.spec.ts", run: "npx vigiles compile" },
"Recompile on spec change")
guard() declares reactive behavior: when watched files change, run a
command. At compile time, renders as **Guard:** in markdown. Participates
in monotonicity (strength 1, same as enforce — both mechanically
enforced). Supports strengthen (guidance → guard not yet, use add),
weaken (guard → guidance), and reword mutations.
Dogfooding: 3 guard rules added to CLAUDE.md spec:
- recompile-on-spec-change: *.spec.ts → vigiles compile
- regen-types-on-config-change: eslint/package/pyproject → generate-types
- format-check: **/*.ts → fmt:check
Hook emission (compiling guard rules to .husky/ and .claude/settings.json)
is the next step.
https://claude.ai/code/session_01FsbRVewGpTGW1AXysJhiw8
* docs(readme): document guard() rule type
Renamed "Two Rule Types" → "Three Rule Types". Added guard() section
with two examples and the key value prop: one declaration emits hooks
for every supported system (Claude Code, husky, CI), eliminating
copy-paste across .claude/settings.json, .husky/, and CI configs.
https://claude.ai/code/session_01FsbRVewGpTGW1AXysJhiw8
* docs(rules): explain what freshness IS before diving into config
The freshness rule doc went straight into severity tables and detection
modes without ever defining what "freshness" means. Added an intro that:
- Defines fresh vs stale (recompiling produces identical output)
- Lists the 6 concrete things that cause staleness
- Explains the actual harm (agent reads lies from stale claims)
- Shows what audit output looks like
Removed the redundant "Why" section at the bottom.
https://claude.ai/code/session_01FsbRVewGpTGW1AXysJhiw8
* refactor!: replace freshness rule with focused integrity rule
The freshness rule had three modes (strict, input-hash, output-hash) and
fought a problem already solved elsewhere:
- "spec changed but compile not run" → now solved by guard() rules
- "linter config disabled" → enforce() catches at compile time
- "file/script removed" → file() / cmd() catch at compile time
- "hand-edit of compiled markdown" → SHA-256 hash check (kept)
Only the last is genuinely useful and not duplicated. Reframe the rule
to do that one thing well:
"rules": { "integrity": "warn" }
No more modes, no more options, no more 100-line audit function. The
rule is "did anyone tamper with a compiled file?" — answered in <1ms
by comparing the embedded SHA-256 to the body's actual hash.
Code changes:
- New: src/integrity.ts (~50 lines, single function)
- New: src/sidecar.ts (sidecar manifests, used by session audit)
- New: src/sidecar.test.ts
- Deleted: src/freshness.ts, src/freshness.test.ts (~1300 lines)
- Deleted: docs/freshness.md, docs/rules/freshness.md, research/doc-freshness.md
- New: docs/rules/integrity.md (clear scope statement)
- cli.ts: 100-line checkFreshness → 30-line checkIntegrityForFiles
- types.ts: removed FreshnessMode, FreshnessOptions
- README: rule table updated, freshness section removed
BREAKING: .vigilesrc.json users must rename `freshness` → `integrity`.
The mode/extraInputs options no longer exist.
Lint warnings dropped 62 → 56 (less complex code).
https://claude.ai/code/session_01FsbRVewGpTGW1AXysJhiw8
* fix: address PR review feedback (security, sidecar nesting, coverage gating)
Codex review caught several real issues. Fixed:
P1 — security/correctness
- Shell injection in gitChangedFiles: baseRef was interpolated into a
shell command; switched to execFileSync with argv array
- Sidecar parent dirs: nested targets (e.g. .github/copilot-instructions.md)
failed with ENOENT because writeSidecarManifest only mkdirp'd .vigiles/
itself; now creates the full parent path
- iterateSidecars didn't recurse: nested manifests were skipped, so
session analysis silently dropped directory-scoped targets
- analyzeSession dropped *.md.spec.ts edits when sidecars hadn't been
written yet (first compile after init); now classifies by extension
- Coverage rule had thresholds in config but auditExitCode ignored them;
now gates CI when severity is "error"
- fitness() coverage only counted enforce(), so guard() rules looked like
reduced coverage and got rejected by the proof suite; now counts both
(anything with STRENGTH ≥ 1)
P2 — robustness
- gitChangedFiles swallowed git errors as []; now throws so a typoed
baseRef doesn't masquerade as "no changes"
- keyFiles parser broke on CRLF line endings (Windows checkouts);
normalizes to LF before matching
- coverage.test.js and session.test.js weren't in the test script, so
npm test silently skipped them; added (361 tests now, was 315)
Outdated/rejected (no action):
- Comments referencing freshness.ts / drift.ts (files deleted)
- "Wrap in try/catch for legacy data" suggestions — we deliberately
removed those because legacy JS specs don't exist in this codebase
- "Persist sidecars during compile" — integrity rule doesn't need them
https://claude.ai/code/session_01FsbRVewGpTGW1AXysJhiw8
* fix: address codex review on coverage rule (P2)
Two P2 comments on the new coverage threshold flow:
1. computeScriptCoverage accepted bare script names (e.g. "build") as
covered, but cmd() in compile.ts only validates "npm run X" / "npm X"
forms. Removed the bare-name check so coverage matches what compile
actually verifies — stale entries can no longer satisfy the threshold.
2. checkCoverageThresholds called computeScriptCoverage without passing
specs, which falls back to require()'ing dist/<spec>.js. CI runs
`vigiles audit` without a build step, so dist/ was empty and coverage
reported 0% even when commands were documented. Now loads claude
specs via loadSpec() (which has tsx fallback) and passes them in.
Verified locally: scripts coverage went from 0% (broken) → 100%
(correctly counts npm run build, npm test, npm run fmt, npm run lint).
https://claude.ai/code/session_01FsbRVewGpTGW1AXysJhiw8
---------
Co-authored-by: Claude <noreply@anthropic.com>
|