mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
fix/hook-runtime-unused-err
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 zernie
|
7be092ce19 |
refactor(linters): LinterAdapter port + JVM/Go doc & test parity (#116)
LinterAdapter registry (Record<BuiltinLinter, LinterAdapter>) makes a missing linter a tsc error; generate-types iterates the registry; conformance test set-matches registry vs BUILTIN_LINTERS + docs + site chip list. CI installs detekt/ktlint/checkstyle/golangci-lint so their gated tests run (no silent skips). Docs/site catalogs->linters parity. add-a-linter contributor skill. Report-view teaser cards lightened + linter footnote de-staled. Stage 4 (dispatch-map collapse) tracked as a follow-up. |
||
|
zernie
|
3017f70ef5 |
feat: JVM/Go linters + audit/init/compile hardening + adoption UX (#114)
Closes #107, #109, #110, #111, #113. Vlad's 5 issues (skill allowed-tools round-trip, detekt/ktlint/checkstyle/golangci-lint catalogs, 2 scan false-positives, non-JS harness guide, testGlobs docs), a 14-bug dogfood batch, adoption/demo UX, and prevention refactors (one markdown-it fence oracle replacing 5 regex toggles; one js-yaml detekt-config parser replacing 3). LinterAdapter port added to the roadmap as P0. |
||
|
zernie
|
18d7cfcae4 |
docs: honest versioning, README scannability, and docs restructure (#67)
Fix the version claim (v12, not 0.x; the 0.0.0-semantically-released placeholder was the source), reframe the eval section around cost measurement, trim the README for scannability (Diátaxis link buckets, tighter reconciliation + proofs), reorganize the docs index into Guides/Reference/Explanation and add six missing docs, and consolidate overlapping docs (related-tools→comparison, inline-mode→markdown-mode, agent-setup+agent-workflows). Relocate the eval-architecture design-of-record ADR from docs/ to research/ per the doc-tiers rule, and fix the library entry points (vigiles/linting is the compiler surface) plus the Codex setup path. |
||
|
zernie
|
a7a2c53fad |
feat!: audit --serve, scoring/FP fixes, README refresh (disables frontmatter lint mode) (#51)
Audit/adoption/lint-coherence wave + audit eval-thickening + ecosystem-benchmark pilots. BREAKING CHANGE: frontmatter lint mode is disabled (FRONTMATTER_MODE_ENABLED=false) — kept in code and flag-reversible, but inert in lint. The on-ramp ladder collapses to inline-comments + typed-spec. Highlights: audit --serve (hardened loopback one-click adoption); inherit-all → advisory (stops crying wolf); audit's executing tier now runs selection-collision + adversarial-gate evals beside trigger-rate; ecosystem compression-cluster + headroom benchmarks; README refresh + real hero. See the PR body for the full list. |
||
|
zernie
|
e8fa3b6b78 |
refactor!: pre-release polish — surface freeze, spec-first defaults, auto-adopt (#48)
* chore: refresh HANDOFF — PR #47 merged, surface-freeze still open Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * refactor!: freeze pre-release public surface + collapse markdown ladder Pre-release surface freeze (research/pre-release-focus.md): - Mark the experimental typed-composition cluster @internal in vigiles/spec (pipe/pipeStep/needs/start/andThen + Supplies/Handoff/KnownAgentName/ Pipeline/PipeStep/NeedsContract) and effect()/EffectRegion. They stay exported at runtime (no removal) but are excluded from the published API docs and flagged unstable in the committed surface reports, so a later change to them is not a breaking change to a documented-stable API. - Note evolve.ts as @internal/research-stage (already off every public entry). - Markdown cut: drop the "Level 0/1/2" ladder framing across public docs; collapse to two on-ramps (plain markdown via inline enforce comments -> typed spec). Frontmatter mode is KEPT, still works, and its full docs + the old ladder are PARKED via an HTML comment in docs/markdown-mode.md (matching the README's parked-section convention) — nothing deleted, trivially un-parked post-launch. - Ship STABILITY.md (0.x: CLI stable; library API evolving; experimental surfaces marked) + a one-line README pointer. - Regenerate etc/*.api.md to reflect the @internal tags (api:check green). BREAKING CHANGE: public symbols in vigiles/spec are re-tagged @internal (typed-composition combinators, effect()). They remain importable but are no longer part of the frozen, supported surface. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: surface the author-time-vs-runtime wedge + citeable authorities - README hero: state the differentiator positively — vigiles proves the harness "before the agent ever runs" (author-time / pre-run), ending on verify/test/measure (no negative opener, per the README direction rules). - pre-release-focus.md Positioning lock: fold in the canonical one-line wedge (author-time vs runtime, public-safe phrasing, no competitor names) and the citeable harness-engineering authorities (Karpathy "automate what you can verify"; OpenAI "the harness is hard"; arXiv AHE structure-beats- prose = enforce > guidance) for the launch article. The AHE prediction-loop note already lives on the roadmap A2 optimizer item. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: lead the README hero with the concrete pain (untested harness) Reframe the hero to go straight to the developer pain: you installed plugins and wrote skills but don't know if they work — a library with no tests. Replaces the "100x coder / 1x verifier" lead (kept as a backup hook in the positioning lock) with the concrete plugins/skills situation + silent-fail triad, ending on the verify/test/measure benefit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: lead the guide openings with the concrete pain, not the mechanics Apply the README's pain-first hook to the subdoc openings: - verifying-instruction-files.md: open with the CLAUDE.md-drifts pain instead of "vigiles's linting layer ... checks every reference". - for-plugin-authors.md: add the installer pain hook (do your skills fire for the people who install it?) above the audience-routing intro. - harness-testing.md: sharpen the lead to the concrete "does it actually block? does your skill fire?" question. Each keeps its README uplink + "what this doc is" line (docs-quality). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: codify the pain-first principle + fix a stale doc link - README direction comment: add rule 1c — lead with the concrete pain in the reader's own situation (the specific silent failure + the "library with no tests" anchor), for the hero AND every subdoc opening; reconcile with 1b (a pain about the reader's situation is a hook, not an apology). - Fix a stale link: src/vendor.test.ts -> src/adapters/claude-code/vendor.test.ts (moved in the adapter relayout). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * chore: refresh HANDOFF — surface freeze + launch polish shipped on branch Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs(positioning): not a linter — spec-first + eject, don't fight on structure-lint Lock the decision (Positioning lock + roadmap launch-readiness): vigiles is NOT a markdown/structure linter and won't contest that crowded, commoditized lane. The bold differentiated play is author-time typed specs the agent writes + evals you can afford. Markdown demotes from on-ramp to floor / eject-target. Names the enabling build — a one-command `vigiles eject` (strip the integrity header -> clean owned markdown) so "always ejectable" is true and the two spec objections (TypeScript friction, lock-in) dissolve. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat: add `vigiles eject` + reframe spec-first (managed, but ejectable) The escape hatch that makes "adopt a typed spec, you can always eject" true, and the positioning shift that leans on it. - `vigiles eject [file]`: the inverse of compile. Strips the vigiles:sha256 integrity header -> plain hand-owned markdown, removes the managing spec (--keep-spec to keep it), and adds a `vigiles-disable require-spec` marker so lint stays quiet. Pure helpers (parseIntegrityHeader/ejectMarkdown in core/integrity.ts) + handler + dispatch + help + docs/cli.md. New verb in cli-commands.ts (additive). Unit + e2e tests. - Positioning reframe: don't contest the structure-lint crown. README hero sub-line + spec mentions lead with "your agent writes the spec, and you can eject anytime"; markdown demoted to floor/eject-target (still the zero-TS on-ramp). README direction rule 1c codifies the pain-first hook. - adopt-spec skill: faithful-by-default + eject note; dropped the Level 0/1 ladder pitch; fixed stale commands (check->lint, generate-schema->generate schema, Action @main command:check -> @v1 command:lint). - Closed the scan gap: skills/ now in the self-command-refs dogfood, which caught two rotted `vigiles generate-types` refs (now `generate types`). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat(scan): nudge toward the real-model trigger tier; agent-safe by default When a plain `scan` finds model-invocable skills AND a model is reachable (ANTHROPIC_API_KEY, or an authenticated Claude Code session via CLAUDECODE / CLAUDE_CODE_ENTRYPOINT), surface the real-model `--trigger` tier that measures whether those skills actually FIRE: - A human at a TTY is OFFERED setup (scaffold a trigger-prompts.json, or run an existing one — the real-model run is always an explicit yes). - An agent / CI run (non-TTY / --json / --no-interactive / --yes) gets a one-line, non-blocking hint and never a prompt (great-agent-flow). Pure core (hasModelAccess / decideTriggerSuggestion / formatTriggerHint / scaffoldTriggerPrompts) + unit tests; CLI wiring + env-gated e2e in scan-cli.test.ts; docs in docs/cli.md. Also records (HANDOFF) that real-model tiers ARE runnable in Claude Code web: the claude CLI is authenticated via session OAuth (no API key), proven by `claude -p` and a live `scan --trigger` run (test-harness recall 100%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat: --strict gates broken surfaces (CI red); fix `lint <dir>` crash The actual "edit a skill/subagent/hook badly → CI goes red" payoff. - `vigiles init --strict` now promotes the HIGH-PRECISION, FP-safe structural rules to `error` (not just require-spec): subagent-tool-contract, subagent-frontmatter, hook-events, hook-script-exists, mcp-config, mcp-tool-resolves, mcp-hook-target-resolves, disallowed-tools-contract, description-overlap. So a typo'd tool, a dead hook script, a misconfigured MCP server, or two colliding skill descriptions FAIL `vigiles lint` (exit 2). Never clobbers a user-set severity; leaves the lenient/recommendation rules (frontmatter-valid, skill-frontmatter, prefer-compiled-hooks, untested-*) at warn. Honest limit documented: CC itself loads a name-less / broken-YAML skill, so vigiles gates the subagent/hook/MCP defects + skill collisions that genuinely break, not skill content that still works. Proven: clean subagent exit 0, typo'd tool exit 2. - FIX: `vigiles lint .` (a directory arg) crashed with EISDIR (it read `.` as a file). findInstructionFiles now expands a directory arg to the instruction files inside it; a file arg and a non-existent arg pass through unchanged. Tests: setup-plan (strict gate set), cli (lint-dir no-crash + strict exit-2 regression). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat: init gates broken surfaces BY DEFAULT (no --strict needed) Fold the structural gating into a plain `init` — no confusing "strict mode" to remember. A default `init` now writes the 9 HIGH-PRECISION, FP-safe structural rules as `error` (subagent-tool-contract, subagent-frontmatter, hook-events, hook-script-exists, mcp-config, mcp-tool-resolves, mcp-hook-target-resolves, disallowed-tools-contract, description-overlap), so a broken subagent / hook / MCP ref or a skill collision FAILS `vigiles lint` out of the box — while a well-formed plugin stays green (never cries wolf). `--strict` is reframed to the WORKFLOW-FORCING tier a clean repo can still fail because the work isn't done yet: require-spec, untested-skill/subagent/hook, frontmatter-valid, skill-frontmatter. Opt-in by design (progressive adoption), so a first CI run isn't red just for not having written a spec yet. Never clobbers a severity the user already set. vigiles dogfoods it — the 9 gate rules added to its own .vigilesrc.json (self-lint stays green). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat(init): interactive init offers the workflow tier (recommended, opt-out) Per the chosen default model: structural gating is always on; the WORKFLOW tier (require specs + a test per surface) is the recommended default a human opts OUT of, never forced on a silent run. - Interactive `init` now asks "Also enforce specs + a test per surface (recommended)? [Y/n]" (default yes) → sets strict. SetupAnswers/applyAnswers carry it; resolvePlan honors it. - A bare NON-interactive `init` (agent/CI, no one to ask) stays structural-only unless `--strict` — so an automated setup never silently turns an existing repo's CI red just for not having written a spec yet. - README agentic-install prompt + docs/cli.md updated: the agent asks before enforcing the workflow tier, structural gating is the safe default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * chore: refresh HANDOFF — spec-first defaults + eject + scan nudge; PR is the next step Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * test(init): make the interactive setup Q&A unit-testable (inject ask) The interactive init prompts were the one untested-and-fragile bit of the setup flow (readline can't run in CI's non-TTY env, so the questions/defaults/ answer-mapping had zero coverage). Extract the Q&A into a pure collectSetupAnswers(ask) with an injected AskFn; promptSetup is now just the readline IO shell. Unit-tested with a fake ask: all-defaults, lint/test pillar selection, and declining CI/plugin/strict — deterministic, no TTY, no model. Now the init flow is covered end to end: pure plan logic + the interactive Q&A (unit) + non-interactive init (e2e in cli.test.ts). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: install-enforcement model — rule groups (not presets) + rationale Capture the reasoning, not just the outcome, before implementing. - New `install-enforcement-model` rule in CLAUDE.md (via the spec): rules are organized into GROUPS by confidence (structural / workflow / nudge), NOT a relaxed/standard/strict preset MENU — the linter best practice (Clippy/Ruff/ Biome; "presets are a poor emulation of rule groups"). default = structural at error; --strict adds workflow; --report-only is an orthogonal severity dial. Installation is EXPLICIT ABOUT TRADEOFFS (one source reused across the CLI prompt / agent AskUserQuestion / summary). - Reconcile smooth-adoption + progressive-adoption to the new stance: "permissive = doesn't force specs/TS + doesn't cry wolf, NOT ignores breakage." - research/install-enforcement-dx.md: the full design record — the linter landscape (verified), why we rejected a preset menu, the group→rule mapping, the agentic-install DX (no prior art), and the philosophy reconciliation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * refactor!: rename require-spec → require-instructions-spec; rule groups; init auto-adopt Rename + narrow the rule, name the install rule-groups, and make `init` faithfully adopt existing instruction files so the rule is satisfied by construction. - require-spec → require-instructions-spec, NARROWED: only a `.spec.ts` satisfies it — inline `vigiles:enforce` / `vigiles:` frontmatter no longer do. Disable marker is now `<!-- vigiles-disable require-instructions-spec -->`. - Name the rule groups in src/setup-plan.ts (STRUCTURAL_RULES / WORKFLOW_RULES / NUDGE_RULES) and add `--report-only` (writes the whole gate at warn). - prefer-compiled-hooks defaults off; require-skill-spec un-deprecated (the consistent require-<surface>-spec parallel, default off). - New src/core/adopt.ts: adoptMarkdown/adoptToSpec faithfully convert an existing CLAUDE.md/AGENTS.md into a claude() spec (heading → verbatim section, no rule inferred, always compiles). Wired into init()/setupPillar1; round-trip tested (byte-identical below the integrity header). - Sweep stale Level-0/1/2 markdown-mode comments; docs + CLAUDE.md recompiled. BREAKING CHANGE: the `require-spec` lint rule and config key are renamed to `require-instructions-spec` and narrowed — only a `.spec.ts` satisfies it (inline comments / `vigiles:` frontmatter no longer do), and the per-file disable marker is now `<!-- vigiles-disable require-instructions-spec -->`. No automatic alias; update `.vigilesrc.json` and any disable markers. * fix(init,eject): honor interactive strict; don't delete a shared spec on eject Addresses two Codex review findings on #48 + the fmt:check CI failure. - setup(): write `.vigilesrc.json` and the summary from the RESOLVED plan (`plan.strict`), not the raw `--strict` flag — so an interactive "yes" to the workflow tier is actually persisted instead of silently dropped. - eject(): only delete the header's `compiled from` spec when it EXCLUSIVELY belongs to the file being ejected (`<file>.spec.ts`). A mirrored secondary target (e.g. AGENTS.md compiled from CLAUDE.md.spec.ts) names the shared source; deleting it would break the primary's recompile. Keep it + warn instead. Regression test added. - Format HANDOFF.md (the fmt:check failure). * fix(init): non-destructive adopt, deferred compile, test-only & strict config gating Install-DX hardening from PR #48 review + a self-audit across user/project types. - Auto-adopt is now NON-DESTRUCTIVE: `init` writes the spec but never overwrites your CLAUDE.md/AGENTS.md. Run `vigiles compile` to opt the file into spec management (byte-faithful; review the diff; `eject` reverses it). - Defer the compile when `vigiles` isn't resolvable yet (a fresh repo hasn't run `npm install`) — print the next step instead of a "failed to load" error. - Test-only setup (`init --test` / `--no-lint`) records the harness but writes NO lint rules — honors the positive-flag contract (Codex review). - setup() writes config from the RESOLVED plan, so an interactive "yes" to the workflow tier persists (was dropped). - Next steps reordered (install → compile → strengthen → test); `--report-only` in `init --help`; README + docs surface non-destructive adopt. New e2e tests. * fix(adopt): don't drop intro text when a literal ## Overview already exists Codex review (P2): a file with intro prose before the first heading AND an explicit `## Overview` section produced two sections keyed "Overview" — building the sections object overwrote the synthesized intro, silently dropping it (and a later `compile` would remove it from the file). Extract a shared `allocKey` dedup helper and use it for the synthesized Overview too, so it becomes "Overview (2)" on collision instead of clobbering. Regression test added. * fix(eject): preserve frontmatter on skill/agent eject; keep shared multi-target specs Two Codex review P2s on the eject feature. - ejectMarkdown no longer prepends the require-instructions-spec disable marker before a body that leads with YAML frontmatter (a compiled SKILL.md / subagent): doing so displaced the frontmatter and lost the surface's name/description/tools. The marker is instruction-file-only, so a frontmatter-led body ejects verbatim. - eject deletes the header's spec only when NO other compiled output still references it (a directory scan), instead of a basename heuristic. A multi-target spec (`target: ["CLAUDE.md","AGENTS.md"]`) or mirror now keeps its spec until the LAST output is ejected — ejecting either the primary or a secondary no longer orphans the others. The trailing marker hint prints only when a marker was added. Tests: frontmatter-preserved eject (unit) + eject primary/secondary/last-output of a multi-target spec (e2e). Docs note the two safety behaviors. * fix(eject,scan): path-safety on spec deletion; gate scan prompt on stdin TTY Two Codex review P2s. - eject: the `compiled from <path>` header is untrusted — a hand-edited/forged header naming `package.json` or a `../` path would make rmSync delete an unrelated file. Only remove a `.spec.ts` that resolves INSIDE the project; otherwise keep it and warn. Regression test added. - scan --trigger nudge: prompt only when BOTH stdout AND stdin are a terminal — `askOnce` reads stdin, so a TTY stdout with piped/redirected stdin (agents, pipelines) must take the non-blocking hint path instead of blocking on a read. * docs(readme): add FAQ + docs/faq.md, tighten install prompt, lift adoption line 5-star onboarding pass (1/N). - New docs/faq.md: the objection-handling answers (why a typed spec & is it optional, do I write TS, does init touch my files, CC+Codex, cost/key, non-JS repo, safe to run, vs promptfoo). - README: compact FAQ section linking it; install prompt rewritten to plain English (names adoption, tells the agent to finish: install + compile, no flag jargon); the non-destructive adoption reassurance lifted above the fold. * fix(init): honest guidance for non-JS repos (no package.json) 5-star onboarding pass (2/N). A Python/Rust/Go repo with a CLAUDE.md but no package.json can't resolve the vigiles npm package, so the typed-spec compile path isn't available there. Previously init told them to run `npm install` (which wouldn't help). Now it points at the real paths: - the deferred-compile notice and the adopted next-step both detect a missing package.json and recommend `npx vigiles lint` (verifies with zero install) plus `npm init -y && npm i -D vigiles` to enable specs. (Dropped the post-init lint "aha" idea — a freshly-adopted file is guidance-only with no marks to verify, so it would show nothing until `/strengthen` runs.) Test: init on a no-package.json repo guides to lint, not a bare npm install. * docs: add scripts/demo.sh — self-contained asciinema demo of the Lint layer 5-star onboarding pass (3/N). A zero-dependency, reproducible script for recording an asciinema cast / GIF: it builds a throwaway repo whose CLAUDE.md points (via inline marks) at a file that moved and a script that was renamed, then runs `vigiles lint` to show the ✗/✓. One command to record: asciinema rec --command "bash scripts/demo.sh" Tested end-to-end (real ✗ "File not found" / "Script not found" output). Consistency spot-check: public docs carry no stale adopt/compile language and all README-linked guides open with a README uplink. * docs(readme): final 5-star pass — tighten spec blurb, wire demo GIF slot 5-star onboarding pass (4/4). - Tighten the above-fold spec/adopt/eject paragraph (the new FAQ now carries the "do I write TS / will it touch my files" detail), and point new readers to it. - Wire a DEMO GIF slot at the prime "see it work" spot with the exact asciinema/agg recording command (scripts/demo.sh) + the embed snippet. README rendered content ≈152 non-blank lines (well within the ~200 cap); all internal doc links resolve; command refs pass the self-command-refs dogfood. * docs(handoff): record the 5-star onboarding-polish pass * docs(readme): add the demo GIF — vigiles lint catching stale references Animated terminal of `vigiles lint` flagging a moved file + a renamed script, at the README's prime "see it work" spot. The output lines are verbatim from the real CLI (nothing fabricated). Generated headlessly with Pillow — no recorder needed — so it's reproducible: `python3 scripts/make-demo-gif.py` (generator committed); scripts/demo.sh remains for a live asciinema recording. * docs(rules): sync rule docs with this session's default changes doc-per-rule drift caught auditing the session: - prefer-compiled-hooks.md: default is now OFF (was documented "warn") — a recommendation that shouldn't fire unasked. - require-skill-spec.md: drop the "deprecated" framing — it's the consistent require-<surface>-spec parallel (kept, default off), matching the code + matrix. * fix(eject): scan the whole project tree for shared-spec references, matched by resolved path The shared-spec guard scanned only the ejected file's directory, so a multi-target spec emitting a secondary output in a subdirectory (target: ["CLAUDE.md", "docs/AGENTS.md"]) was missed — eject would delete a spec still feeding another compiled file. Walk the whole project tree (skipping node_modules/.git/dist/build/coverage/.next/out) and match on the resolved spec path, not the basename, so two unrelated CLAUDE.md.spec.ts in different packages don't false-positive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
d8c639f8da |
refactor(cli)!: consolidate the verb surface (13→8) + docs/README overhaul (#46)
* docs: sharpen README lint/guard copy - Replace internal "7 catalogs" jargon with "7 linters (ESLint, Ruff, Clippy, and four more)"; Cedar stays in the subdoc - Drop the plugin-audit paragraph from the Lint section (secondary use case, still reachable from More) - Rewrite the Guard intro to lead with the concrete "fail open" failure and name what vigiles emits (exit code, JSON, AST-backed matcher) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: surface the $0/subscription cost story in the first paragraph The "no metered tokens, runs on your Claude subscription" wedge was buried in the last line of the Eval section. Pull a crisp version into the opening paragraph (above the fold); the Eval section keeps the detailed promptfoo/DeepEval contrast as proof. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: surface agent-writes-the-harness flow, reorder More, drop internal links - Quick start now leads with "then your agent writes the harness for you": the three model-invocable skills (test-harness/strengthen/ edit-spec) and the plain-English ask each one answers - Reorder More so CLI sits above the leaderboard (whose link points into the CLI docs); trim the now-redundant Skills bullet to a pointer - Stop linking internal research/ docs from the README (the moat link and the Research index) per public-vs-internal-docs Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(compile): auto-refresh an existing harness.gen.ts compile now keeps a whole-harness registry in sync as a side effect, so the user never hand-runs generate-harness. Gated on the file already existing (compile maintains a registry you opted into, never imposes one) and cheap (parses specs, no linter spawn — unlike generate-types/ generate-schema, which spawn every linter and stay on the config-change guard). A duplicate agent name fails the compile. Docs: note the auto-refresh in the compile section and mark the generate-* artifacts as dev-toolchain output rarely run by hand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(src): add nested CLAUDE.md with a high-bar-for-new-commands rule Directory-scoped guidance loaded when working in src/: adding a new CLI verb or hook-runtime subcommand must clear a high bar (genuinely new mental action, no overlap with an existing command, siblings collapse to one verb + subcommand, runtime entrypoints stay hidden). Complements the root cohesive-cli-surface rule, placed next to the code that defines the command surface (cli.ts, cli-commands.ts). Spec-managed (require-spec + integrity); dogfoods vigiles compile. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: drop the "(generated)" annotation from the API reference link Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(build): stop src/CLAUDE.md.spec.ts shadowing the root spec + add docs rule The nested src/CLAUDE.md.spec.ts (added last commit) compiles under rootDir:src to dist/CLAUDE.md.spec.js — the exact path loadSpec resolves the root CLAUDE.md.spec.ts to, so recompiling the root spec loaded the src spec and clobbered CLAUDE.md. Exclude src/*.md.spec.ts from tsc so both load via the tsx fallback (as the root always has). Also adds the no-internal-links-in-public-docs guidance rule (public docs never link research/*.md; keep the fact in research, don't link it), reconciles the old "link to research" line in public-vs-internal-docs, and files the deterministic lint-rule version as a P1 roadmap item. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: separate public/internal indexes + surface eval safety - docs/README.md no longer links into internal research/ (public docs don't point at the private record) - research/README.md is explicitly labelled the internal record - README Eval section gains a "Safe to repeat" note (sandbox + ephemeral dir + egress + interceptTools, past-the-sandbox case flagged) linking docs/safety.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: stop public docs linking internal research/ + add safety FAQ Strip every research/*.md link from docs/*.md (the no-internal-links-in- public-docs rule) — ~60 links across 16 guides plus the docs/ index; prose kept readable, no orphaned research doc (all still cited from CLAUDE.md keyFiles / research/README.md, orphan-docs green). Add a FAQ to docs/safety.md answering the side-effects-beyond-the-sandbox question honestly: network via egress allowlist, a real DB/redis/browser via a disposable container you stand up (turnkey provisioning is a committed next step), irreversible externals via interceptTools, and the no-Docker deterministic/record-replay tiers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(cli)!: consolidate verbs — measure/explain→scan flags, refs→lint, generate-*→generate <kind> De-sprawl the human verb surface from 13 to 8 (the high-bar-for-new-commands rule, dogfooded). All breaking: - `measure <dir>` → `scan <dir> --trigger` (the model-gated behavioral column folded into scan; trigger-rate + collision matrix unchanged) - `explain <dir> [name]` → `scan <dir> --explain [name]` (the deterministic WHY, a sibling of --fix-plan on the same ScanReport) - `refs <file>` → folded into `lint` (the unmarked-refs check is already a lint rule; the PostToolUse hook keeps its shared collectRefIssues entry) - `generate-types|schema|harness` → `generate <kind>` (one verb + subcommand; the three dev-toolchain generators grouped, not N hyphenated siblings) scan reuses handleMeasure/handleExplain directly behind --trigger/--explain. Docs, help text, the regen guard, tests, and the self-command-refs dogfood all updated together (cohesive-cli-surface); 1588 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: split the GitHub Action reference out of cli.md into its own doc cli.md was 718 lines (kitchen sink). Extract the ~108-line GitHub Action section into docs/github-action.md (quick start, inputs, output channels, versioning) and leave a short pointer; retitle the cli.md intro. Update the README More bullet (CLI · GitHub Action) and the docs index. cli.md → 619 lines. orphan-docs + self-command-refs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(lint): add prefer-compiled-hooks rule — one nudge toward compiled hooks A single repo-level recommendation (one finding regardless of hook count): when a plugin ships hand-written hook commands that aren't compiled vigiles/hook artifacts, nudge toward compiling them — the message links docs/compiled-hooks.md and points at guardrail-check for verifying the ones you have. A discovery nudge, not a defect: the hand-written shell lane stays first-class, so it fires ONCE (never per-hook), defaults to warn (info- level ℹ; informational, doesn't fail CI), and is opt-out. Detector (manualHookCount: a command that isn't a hook-runtime invocation) lives in scan.ts and is shared by scan + lint (one-detector-no-drift). Doc + matrix rows added (rules-docs-in-sync). Honest caveat in the doc: it's form-based, so it stays a recommendation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): record the eval-startups critique mapped onto positioning New research/eval-startups-positioning.md captures Thomas Liao's "why eval startups fail" critique and its implications: vigiles dodges the trap (harness-eval not model-eval, tooling not services, eval already a feature), but it's a yellow flag on the public ecosystem-benchmark as a MOAT (Goodhart-gameable → keep it a thin acquisition flywheel) and it independently validates the attestation/safety direction (capability-diff, guardrail proof) as the only defensible eval-shaped niche. Linked from research/README.md, measurement-authority.md (whose benchmark thesis it challenges), and strategic-synthesis-2026-06.md (whose Tier-2 conformance/attestation play it validates). Internal-only per no-internal-links-in-public-docs; orphan-docs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: purge internal positioning from public docs (jargon, name-drops, benchmark) Public docs (README + docs/) now state user-facing value, not vigiles's internal strategic vocabulary: - strip "moat" from 14 places → user-facing wording (engine / cost advantage / differentiator / cross-reference) - remove all bare research/*.md name-drops from docs/ (the backtick form the link-strip + deterministic check miss); scope-descriptions reworded to "configured directories" rather than naming research/ - README: drop the "ranks the hyped ecosystem: what works vs hype" benchmark banner from the Eval row + kicker (depth stays in measuring-skills.md) — per eval-startups, the benchmark is a flywheel, not a headline - docs/cli.md: "moat #2 (research/…)" → "the capability-diff check" Codified in CLAUDE.md: public-vs-internal-docs now has a strategic- vocabulary clause (moat/measurement-authority/flywheel stay internal; public docs name the benefit), so this doesn't regress. orphan-docs + self-command-refs green; the public↔private split stays docs/ vs research/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): apply multi-persona review fixes From a multi-persona README review (power user / staff eng / plugin author / non-TS user / eng manager): - non-TS users: a visible "No TypeScript needed — lint + inline comments work on any CLAUDE.md" line (was invisible; contradicted progressive adoption) - plugin authors: surface "vigiles scan ranks a marketplace, no key" above the fold (was buried in More) - eng-manager cost: hero now says deterministic layers run free in CI; merged kicker+affordability nails "real-model on a dev's own subscription, never a CI token" - staff-eng skepticism: the 2/7→7/7 claim now reads "measured, not asserted"; the typed-pipeline claim is concretized to the real pipe()/needs() mechanism - fixed a broken blockquote and trimmed the safe-to-repeat para to hold length Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): capture the README design direction as an editor-facing comment A top-of-file comment block records the framing the README must hold: lead with benefits, present compiled specs as the easy/auto-enforced next step (skills write them, hooks compile them; JS→TS evolution, not a wall), keep all four instruments first-class incl. Eval, stay scannable + under the length cap, and no internal vocabulary/research links. Guides future edits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): benefits-forward rewrite + specs-are-easy framing Per the README direction: lead with what the user GETS, and present the typed spec as the frictionless auto-enforced next step, not a TS wall. - hero: stronger, concrete failure modes (stale ref / guard fails open / skill never fires), keeps Agent=Model frame, cost above the fold; dropped the redundant centered pitch (merge) - table cells condensed to punchy benefit one-liners - "You don't hand-write specs — your agent does. TypeScript for your harness: opt-in, gradual, auto-compiled, never a chore" (skills write them; JS→TS evolution) - de-dupe cost (drop "not metered tokens" from the Test cell) - "Not for you if …" honest-scope line - docs/verifying-instruction-files.md: back the claim — "you rarely hand-write one; edit-spec/strengthen write it, a hook recompiles on save" Eval stays a first-class instrument. fmt + self-command-refs + orphan-docs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): don't lead callouts with "typed spec" — benefit first Casual readers shouldn't be greeted by "a typed spec is a program" / "TypeScript for your harness" in the prominent callouts. Reframe both to lead with the outcome and demote the spec to a soft, opt-in aside: - specs-are-easy block: "You don't hand-write any of this — your agent does … you never touch it by hand" (drop the TypeScript-for-your-harness headline) - Lint blockquote: "Bad states stop compiling … turn it up only when you want it" (drop "Markdown is prose; a typed spec is a program") - the edit-spec bullet says "edits the source", not "the typed spec" "No TypeScript?" reassurance and the optional .spec.ts mention stay. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF for the CLI-consolidation + docs/README overhaul session Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: prettier-format HANDOFF Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): update the stale generate-types ref the verb rename missed The CI "Check generated types compile" step ran `node dist/cli.js generate-types` — renamed to `generate types` in the verb consolidation, so the job failed with "Unknown command". self-command-refs didn't catch it because it didn't scan .github/; add .github/ to its scan scope so a stale `cli.js <cmd>` ref in a workflow fails the dogfood from now on. Also saves the prefer-compiled-hooks-vs-require-hook-spec decision to research/hook-modes-and-testing.md (the nudge, not a requirement; the write-hook authoring skill is the real authoring lever, unbuilt). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(lint): eslint-ignore src/*.md.spec.ts (excluded from tsconfig) The dist-collision fix excluded src/*.md.spec.ts from tsconfig, so eslint's type-aware project service can't resolve src/CLAUDE.md.spec.ts and errors "not found by the project service". It's a tsx-loaded build input (like the root CLAUDE.md.spec.ts, which isn't linted either), so ignore it in eslint to match. Latent since the exclude landed; surfaced now that the earlier generate-types step no longer fails first. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
66ca161e8b |
fix: round-2 dogfooding fixes for untested-surface, audit rename, orphans, loose-skill evals (#35)
* fix: round-2 dogfooding fixes for untested-surface, audit rename, orphans, loose-skill evals - untested-surface: discover colocated tests under dot dirs (dot:true), so a `.claude/skills/<name>/<name>.eval.mjs` the warning suggests actually counts as coverage instead of leaving the nudge unsatisfiable. - init: flag a CI workflow that still calls a renamed/removed subcommand (`vigiles audit` → `lint`) even when the Action or a modern command is also present, instead of reporting it "up to date". - orphan-docs: honor an inline `<!-- vigiles-disable orphan-docs -->` opt-out and point the failing message at both escape hatches (inline disable + .vigilesrc.json orphans.exclude/include). - measureTriggerRate: accept a loose `skillsDir` (e.g. .claude/skills) and auto-package it into a throwaway --plugin-dir via packageSkillsDir, so trigger-testing repo-local skills is a one-liner. - docs/strings: replace stale `vigiles audit` command references with `vigiles lint` (schema descriptions, hook/skill prose, CLAUDE.md). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat: make strengthen/edit-spec model-invocable, tidy skill packaging, harden agent install Skills are useless if nobody remembers to run them. Address discovery + install gaps surfaced in the dogfood review: - strengthen + edit-spec are now MODEL-INVOCABLE (drop disable-model-invocation) with sharpened, precision-aware descriptions, so the agent reaches for them on its own — "strengthen my rules" / "add a rule to CLAUDE.md" fire the skill. The pre-edit hook already points at edit-spec when it blocks a direct edit; making it model-invocable closes that loop. Each gets a dogfood trigger eval (recall + precision) so auto-firing doesn't become over-firing. - Merge generate-rule into edit-spec (rule classification absorbed) and delete it. Fix stale commands in edit-spec (vigiles check->lint, setup->init, the old `npx skills add` vendoring installer -> marketplace). - linter-docs gets a real SKILL.md (user-invoked reference), fixing the packaging smell of a skills/ dir with no SKILL.md. - Bump .claude-plugin/plugin.json version 3.0.0 -> 4.0.1 (was stale). - Harden installPlugins: on auto-install failure/no-CLI, fail LOUDLY and lead with the shell-runnable CLI commands (an agent can run those) instead of only /plugin slash commands a non-TTY agent can't execute. - Sweep remaining stale `vigiles audit` -> `lint` (example spec). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * refactor: remove the renamed `audit` command everywhere; rename lint-command internals `audit` became `lint` in v4 but the old name lingered. Purge it: - cli.ts: rename the lint command's internals (audit()->runLint, AuditReport-> LintReport, auditExitCode->lintExitCode, printAuditSummary->printLintSummary, annotateAuditForGitHub->annotateLintForGitHub) + comments. Keep the REMOVED_SUBCOMMANDS `audit: "lint"` detection key (it matches the OLD name). - FUNCTIONAL: bench scripts invoked `node $CLI audit SKILL.md` (removed cmd) — now `lint`. - docs/research/comments: `vigiles audit` -> `vigiles lint`, and the command- sense word "audit" -> "lint". - Kept genuinely-distinct uses: the post-session AUDIT feature (session.ts), the Merkle "audit trail", the supply-chain audit / scan concept, the dev-only `audit-feedback-loop` skill, and English "you audited the code". - Left third-party vendored fixtures (SHA-pinned) untouched. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat: prompt-diversity gate for trigger evals + raise auto-skill recall bar to 0.8 Task: ensure shipped model-invocable skills have evals, and validate prompt sets are big/varied enough before spending tokens. - measureTriggerRate now runs a deterministic pre-flight gate (no model): each prompt set needs >= minPrompts (default 10) and no two prompts more than maxSimilarity (default 0.8) alike via normalized Levenshtein. Throws with an actionable message before any run. Exports levenshtein/promptSimilarity/ checkPromptDiversity/assertPromptDiversity; minPrompts/maxSimilarity are per-spec overridable (narrow skills, short walkthroughs). - Expanded all shipped model-invocable dogfood evals (test-harness, strengthen, edit-spec) to 10 diverse relevant + 10 irrelevant prompts and raised the bar: recall min 0.8, maxFalsePositive 0.3. generate-logo + the canonical template expanded to 10; the OMC walkthrough sets minPrompts: 3. - All shipped skills are covered: the 3 auto skills by trigger evals, the user-invoked (migrate-to-spec, linter-docs) by the load gate. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * refactor: use NCD (gzip) instead of Levenshtein for prompt-diversity check Levenshtein measured spelling; switch the prompt-diversity gate to Normalized Compression Distance — the SAME engine `findSimilarRules` uses for near- duplicate rule detection, so the project has one house algorithm for "are these two texts basically the same". NCD scores shared structure/redundancy (a templated prompt with one word swapped compresses together), which is exactly the lazy copy-paste set the gate should reject. - Drop hand-rolled levenshtein/promptSimilarity; add promptDistance (= ncd over normalized text), reusing src/core/proofs.ts ncd(). - TriggerRateSpec: maxSimilarity -> minDistance (default 0.3, the rule-dup threshold). Calibration: real 10-prompt sets sit at NCD >= 0.44; near-dup pairs (one-word swaps, punctuation) at <= 0.24 — clean separation at 0.3. - Tests updated to NCD distance. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat: stubSkillBodies — measure skill triggering on frontmatter only Trigger-rate is decided by a skill's frontmatter (name + description) before its body ever loads, so the body is causally downstream of selection. stubSkillBodies keeps every skill's frontmatter (the selection competition stays faithful) but replaces each body with a no-op stub, so a trigger run stops AT selection instead of executing an expensive multi-step procedure — far cheaper, faster, side- effect-free, without changing what's measured. ~95% smaller skill payload. - stubSkillBody (pure) + packageSkillsDir({ stub }) (frontmatter-only package). - resolveTriggerPluginDir stubs both paths (pluginDir + skillsDir), preserving the plugin NAME so `<name>:<skill>` still matches the fired predicate. - TriggerRateSpec.stubSkillBodies (default off; on in all trigger evals). - Needs one real-model run to confirm the Skill tool still resolves under a stubbed body (the deterministic transform is fully tested here). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * fix(init): scaffold spec imports only `claude` so it passes strict ESLint The generated CLAUDE.md.spec.ts imported `{ claude, enforce, guidance }` but the enforce()/guidance() examples are commented out, so a repo running `eslint . --max-warnings=0` (common in CI) failed on two no-unused-vars warnings the moment the scaffold was committed. Import only `claude` (the symbol the scaffold actually uses); show the fuller import as a comment for when a real rule is added. Regression test guards the import line. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(init): --force rewrites a stale CI workflow in place Round-2 follow-up: instead of only warning about a stale workflow, `vigiles init --force` fixes it. A renamed/removed subcommand (vigiles audit) is rewritten surgically in place (audit → lint, the rest of the user's workflow preserved); a bare-`npx vigiles` no-op workflow is regenerated from the template. Without --force the behavior is unchanged (warn, don't clobber) and the warning now points at --force. Pure `rewriteRemovedSubcommands` + force plumbed through setup-plan; unit + CLI integration tests. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs: positioning for the eval-tier decision + new trigger-rate features - research/eval-api-landscape.md: dated decision — harness eval ≠ model eval (fidelity to the harness-as-shipped is why promptfoo et al. can't host it); keep the deterministic tiers, keep the real-model surface thin (stubSkillBodies = selection not execution), don't rebuild the eval stack, model fidelity is the one on-moat refinement. - CLAUDE.md(.spec): second-pillar paragraph gains the harness-eval-≠-model-eval framing; src/eval.ts Key Files entry documents skillsDir/packageSkillsDir, stubSkillBodies, and the NCD prompt-diversity gate. - docs/harness-testing.md: trigger-rate section documents skillsDir, stubSkillBodies, the diversity gate, and measuring on the user's real model. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(research): testing-API design proposal — the ideal shape Synthesizes "what makes a world-class testing API" (deterministic assertion libs: node:assert/Chai/Vitest/Testing-Library/fast-check; AI-eval: promptfoo/DeepEval/ Inspect) with the "do we need all these layers?" question into one conclusion: - One Trace (already the substrate); a single declarative `check` vocabulary evaluated two ways — strict (throw, unit/integration) vs scored (0-1 ± se, eval). Collapses today's predicate/assert*/matcher 3× into one surface and unifies deterministic + non-deterministic. - Three tiers, not four: e2e is integration + one egress symbol → fold egress in as a capability of integration, keep the capability ladder. - Checks-as-data ⇒ JUnit/baseline/regression + a promptfoo bridge fall out free. - Recommends option C (declarative checks) + D (property-test add-on for hooks). Indexed in CLAUDE.md Key Files + research/README; no users yet, so it's framed as a clean-break design, not a migration. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * fix(orphan-docs): resolve file-relative markdown links; refine tier taxonomy in design doc - orphans.ts: a markdown link is conventionally file-relative ([x](foo.md) in research/README.md → research/foo.md, ../docs/x.md walks up). The detector only matched root-relative paths, so a bare sibling link read as an orphan. Now credits BOTH readings (refTargets resolves against the source file's dir + keeps the root-relative reading). Tests for same-dir and ../ links. - testing-api-design.md: fold in the two taxonomy questions. The layer names are borrowed from app testing, not derived: the harness has only TWO execution scopes (hook = the one standalone-process surface; harness = the whole agent, since skills/CLAUDE.md/settings only act inside a live agent) over a static lint floor. integration/e2e/eval are the SAME scope under realness flags (model mock|real, egress) — model picks the verdict mode (mock→strict, real→scored). "unit only works with hooks" is structural, not a missing capability; each layer adds exactly one capability (the ladder). Recommends runHook + runHarness(flags) over the four borrowed tiers. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): Phase 0 — the declarative `vigiles/check` vocabulary First step of the testing-API revamp (research/testing-api-design.md). A check is DATA, not a throwing assert: { kind, eval(target) → {pass, score, message}, toJSON() }, typed over Trace (tool/skill/output/hookFired/wrote) vs HookRunResult (blocked/allowed). One vocabulary, to be evaluated two ways downstream — strict (expect, Phase 1) and scored (measure, Phase 3) — so a check reads as pass/fail on one run AND a rate across trials, and serializes (JUnit/baseline/promptfoo-bridge fall out later). - src/check.ts + src/check.test.ts (pure, model-free; the failure MESSAGE is tested as much as the verdict). - Exported at `vigiles/check` only — NOT folded into vigiles/testing yet, since hookFired collides with the legacy predicate (reconciled in Phase 6). - Additive: nothing removed; 889 tests green. - Build plan appended to the design doc as Part 6 (Phase 0 marked shipped). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): Phase 1 — strict evaluator (assertChecks) + check matcher veneer - assertChecks(target, checks): the strict evaluator — throws collecting ALL failed checks' messages (Validation-applicative, not short-circuit). Any runner. - vigilesMatchers gains ONE generic veneer for the whole vocabulary: expect(r).toPass(tool("Bash")) / toPassAll([...]) — each carries the check's own failure message. vitest/jest type augmentations added; cross-runner tests prove toPass works under both. - Additive: the legacy predicate/assert*/matcher surface is untouched (it gets removed in Phase 6). 891 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(harness): Phase 2 — runHarness, the explicit harness-scope entry Establishes the collapsed naming from Part 4: the harness has two execution scopes (hook = runHook, harness = the whole agent); integration/e2e/eval are the ONE harness scope under realness flags. runHarness is the DETERMINISTIC harness run (model: "mock", default) — behaviour identical to runHarnessTest, which it wraps. A real-model harness run is non-deterministic, so a single one can't be asserted: runHarness(spec, { model: "real" }) throws and points at measure() / runEval (the eval scope). Unit-tested (the guard needs no claude). The e2e barrel fold + a materialized real path remain for Phase 6. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): Phases 3+4 — measure() over checks + serialization (toJUnit, assertRates) Phase 3 — the SCORED evaluator. measure(spec, { trials, checks }) scores a Trace check vocabulary across trials → per-check rate ± se + pass^k, reusing the tested runEvalWith aggregation (one arm). measureWith takes an injected runner (unit- tested, no model); measure wraps it with the real CLI. Defaults to model "sonnet" (measure on the model your users run). So `tool("Bash")` reads pass/fail strictly (assertChecks) AND as a rate here — one vocabulary, two evaluators. Phase 4 — serialization → CI. checkReportToJUnit (each check a <testcase>, fails below `min`) + assertRates (the scored gate) + formatCheckReport, all keyed off checkLabel(check.toJSON()). Because a check is data, JUnit/baseline/promptfoo- bridge fall out. Pure + unit-tested. (Wiring into the Action/baseline is the remaining CI-integration step.) 894 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(run-hook): Phase 5 — propertyHook, invariant testing for a hook's decision A hook is (event) → decision, so instead of a few example events, generate MANY and assert invariants hold for every decision ("destructive command ⇒ blocked", "never both allow and block"). Reuses the deterministic propertyTest from proofs.ts (seeded, shrinks the counterexample; no fast-check dep). The `decide` runner is injectable — pure fake for unit tests, `(e) => runHook(cmd, e)` for a real hook. Returns the shrunk counterexample, not a throw, so it's inspectable. Test: a correct guard satisfies the invariant; a buggy one (blocks only the exact "rm -rf /") gets a "rm -rf foo" counterexample found. 895 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(check): Phase 6 — front-door the new check surface; defer the mass deletion The new world-class surface (Phases 0-5) is now the documented front door: docs/harness-testing.md gains a "Checks — one vocabulary, two evaluators" section leading with assertChecks / measure / toPass; the design doc marks Phases 0-5 shipped; CLAUDE.md Key Files note runHarness/measure/propertyHook/toPass. DELIBERATELY NOT done: ripping out the ~40 legacy predicate/assert*/matcher helpers + the e2e barrel and rewriting the ~900-test suite/examples off them. On a no-users lib that's high-churn cleanup with no functional gain and real regression risk — the legacy surface stays as working sugar, removed in a single separately- reviewed major-version cleanup (resolving the check↔testing hookFired collision then). The breaking step is the one to take slowly. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * refactor(testing): fold check into testing/unit; collapse e2e into integration The "breaking cleanup" — done at its true (small) blast radius, not the feared 900-test rewrite: - vigiles/check folds into vigiles/testing + vigiles/unit; the check `hookFired` (Check<Trace>) supersedes the legacy boolean predicate via an explicit re-export that wins over the two `export *`s. Checks (tool/skill/output/..., assertChecks, measure, propertyHook) are now first-class at the front door. - e2e collapsed: egress is a CAPABILITY of integration (egressRoutes + runHook's egress:{allow}), not a tier — the old e2e barrel added exactly one symbol. vigiles/e2e stays as a thin deprecated alias so imports keep working. KEPT (deliberately): the bare predicates + per-runner assert*/matcher fronts. They're not redundant — they're context-appropriate surfaces (compose / node:test / vitest-jest) now unified UNDER one check vocabulary. Deleting them would be churn without a greatness gain. 895 tests green, types + cross-runner pass. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(testing-api): Part 7 — greatness re-evaluation + ranked eval-side gaps Post-cleanup assessment: the API is great on the two deciding axes (unification: one Trace / one check vocabulary / two evaluators / serialized; and the moat: the deterministic no-key tiers + pass^k/significance/fidelity nobody else has). Extensibility is strong; the one seam to watch is Check.eval being sync (fine incl. a judge()-backed check; an async scorer would need an AsyncCheck for measure only). Ranked gaps, eval-side first: (1) judged() model-graded check — judge() exists but isn't a Check; fits the sync interface; #1 parity gap; (2) cost/latency/tokens checks (data already captured); (3) checks × A/B arms + significance (unify the harness-A/B moat with measure); (4) multi-surface CC eval — Trace is single+flat, no nested subagent/Task traces, slash-command expansion, multi-turn, or mcp() checks (the deep, new-capability bet); (5) real-tier Trace fidelity (modelRequests mock-only); (6) CC-feature drift maintenance. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): close eval-side gaps — judged/cost/latency/tokens/mcp checks, measureArms + significance, harness-agnostic - judged(rubric, { min, judge }) — model-graded check (LLM rubric) via an injectable judge fn (default real judge(), sync via spawnSync), composes into measure/assertRates/toJUnit. Closes the promptfoo llm-rubric / DeepEval GEval gap. - cost/latency/tokens — Check<UsageTrace> over a run's usage; measure.checks widened to Check<RunContext> so they compose with behavioural checks. - mcp(server, tool) — matches the mcp__server__tool tool name. - measureArms(spec, { arms, checks }) + compareCheck(report, baseline, arm, i): the SAME checks scored per arm, with Welch significance (reuses stats.ts) — unifies the harness-A/B moat with the check vocabulary. - Harness axis: a test pins that checks are harness-agnostic (evaluate over a Codex-sparse Trace — output passes, tool/hookFired fail gracefully). Runners dispatch via adapter; measureWith/measureArmsWith take an injectable runner so a Codex eval reuses the same checks. All unit-tested with fakes (no model). 900 tests green. Design doc Part 7 updated: gaps 1-3 + mcp shipped; nested-trace (subagents/turns) is the remaining big bet. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(harness-testing): add the vitest/jest setup + matcher matrix The reference table we'd flagged in the vitest/jest discussion: a two-column setup matrix (vitest vs jest: config / per-file / manual) and a matcher table (toPass / toPassAll / toHaveCreated / toBlock / toBeatBaseline → asserts → over which result), leading with toPass as the one matcher fronting the whole check vocabulary. Makes the runner story scannable instead of prose. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): nested subagent traces — Trace.subagents + subagent() check The deep gap's first piece: a subagent (Task) runs its own session, recoverable because CC tags its events with parent_tool_use_id (= the Task tool call) and the Task input carries subagent_type. parseSubagents groups the tagged tool calls into Trace.subagents (keyed by name); subagent(name, [checks]) runs the WHOLE check vocabulary recursively over that sub-trace, so a test/eval can assert what the subagent DID, not just that Task fired. Wired into makeResult (harness) + makeContext (eval); empty for harnesses that don't emit parent_tool_use_id (Codex) so it stays harness-agnostic. parseSubagents + the check unit-tested on synthetic stream-json; flagged in Part 7 for one real-CC-subagent run to confirm the field names. 902 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): slash-command-expansion (received) + multi-turn (turns) checks Closes the last two pieces of gap #4, both unit-tested: - received(matcher): the expanded slash command (or a hook's injected context) actually REACHED the model — reads modelRequests (mock/harness tier captures them; the eval tier captures none, with an explicit message saying so). - turns({min,max}): the multi-turn observable — the agent took a back-and-forth, not a one-shot (the deterministic harness already scripts multiple model turns via ModelTurn[]; this asserts how many it took). Remaining (smaller, noted in Part 7): driving multiple USER messages (--resume-style, unverifiable here) + per-turn trace slicing. 904 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(check): validate the subagent parser schema against Claude Code stream-json Confirmed via the Claude Code SDK docs / guide: subagent events carry a top-level `parent_tool_use_id` (= the Task tool-use id), the Task input carries `subagent_type`, linked by tool_use id — exactly what parseSubagents assumes, on the same CLI `message.content` line shape parseToolCalls already consumes (the SDK StreamEvent `event:{delta}` shape is a different, unused format). No code change — the parser is validated; dropped the "needs a real run to confirm" caveat in the comment + Part 7. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * test(dogfood): exercise the new check capabilities on real targets - checks-dogfood.harness.mjs — RUNS here (claude, no key): assertChecks over a real runHook (blocked/allowed) + a real runHarness mock run (turns/output). Validated end-to-end, not just unit-tested with fakes. - dogfood/skill-quality.eval.mjs — the SCORED path on vigiles's OWN strengthen skill: measure({ checks: [skill, judged, cost] }) + assertRates (does it fire, is the suggestion good, is it cheap). Write-don't-run (needs a key). - subagent.eval.mjs — the nested-trace check on a REAL subagent (vigiles ships none): the vendored oh-my-claudecode code-reviewer, dispatched via Task; subagent("code-reviewer", [tool("Read")]) asserts what the subagent DID. Note: the harness-tier dogfood asserts the cross-version-reliable observables (turns/output); tool-call capture in a bare run is plugin/permission-dependent (the fuller pluginDir tests in harness-test.test.ts cover it). 904 green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs: document the mock-tier tool-capture fidelity edge (diagnosed, not a check bug) Diagnosed the flaky tool-call capture from the dogfood: the full stream shows the mock's scripted Write tool_use is never surfaced ("tool_use" absent, only the text turn, plus a spurious rate_limit_event from a LOCAL mock) — a mock↔claude desync with the current version, isolated to a BARE runHarness. The established pluginDir configs capture tools fine (the Skill test runs + passes here). Confirmed it's not permissions (acceptEdits didn't help, reverted), not the parser (tool_use isn't in the stream), and not the check API. Noted in Part 7 gap #5 as a harness-fidelity backlog item; the reliable observables (hooks/turns/output/ received) are unaffected. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(harness-testing): make the full check vocabulary + measureArms discoverable The guide had the Checks intro but not the vocabulary itself. Added a scannable reference table (tool/skill/mcp/subagent/output/received/hookFired/turns/wrote/ judged/cost-latency-tokens/blocked-allowed → holds-when → over-which-result), plus a brief measureArms+compareCheck (A/B with Welch significance) and propertyHook note. So a user can discover the new capabilities, not just read that they exist. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * fix(harness): match subagent dispatch by subagent_type input, not tool name A real `claude` run revealed the subagent dispatch tool is named `Agent` on the live CLI, not `Task` as older docs say. parseSubagents now keys off the `subagent_type` input field (present on the dispatch tool_use regardless of its name) so it survives the rename. Validated end-to-end against real claude output, which recovered the dispatched subagent and its tool calls. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(eval): support stubSkillBodies in measure for cheap skill-firing checks measure() now takes stubSkillBodies (the same body-stub the trigger tier uses, via a shared stubbedPluginDir helper): a skill-FIRING check (skill()) stops the selected skill at selection instead of running its often-expensive procedure, so a description/firing run costs a fraction of the tokens. Requires pluginDir; throws if combined without one. Not for judged/quality checks — the body is gone. Refactors resolveTriggerPluginDir to reuse the extracted stubbedPluginDir helper. Unit-tested (stubbed body in the packaged plugin, cleanup after, the no-pluginDir guard). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * chore(types): regenerate project-file types to include src/check https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(dogfood): add cheap firing-only eval using measure + stubSkillBodies A shipped dogfood example for the body-stub path: measure({ checks: [skill, latency], stubSkillBodies: true }) on the strengthen skill, the cheap sibling of skill-quality.eval.mjs (which keeps the body to grade quality). Makes the ~18x firing-check saving discoverable, not just unit-tested. Smoke-run against real claude: both checks 100% (skill fires, run stays fast). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(eval): support stubSkillBodies in measureArms for cheap A/B firing tests Brings measureArms to parity with measure: an A/B firing comparison (does description variant A fire more reliably than B?) now stubs each arm's skill bodies via stubSkillBodies. Every arm that sets a pluginDir is repackaged with bodies stripped (frontmatter kept) through the shared stubbedPluginDir helper, so each run stops at selection — a fraction of the tokens, per-arm. Arms without a pluginDir pass through untouched; throwaways are cleaned up in a finally. Not for judged/quality checks — the body is gone. Unit-tested (both arms see a stubbed throwaway, originals untouched, cleanup after). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * test(eval): restore 100% coverage for the stubSkillBodies + check surfaces CI's coverage gate (100% lines/functions/statements on the harness-testing files) flagged uncovered paths accumulated across the revamp: the measure/ measureArms stubSkillBodies branches and their helpers (stubbedPluginDir, skillsDirOf .claude/skills fallback, pluginName missing-manifest catch, packageSkillsDir missing-dir throw + non-directory entry skip), checkLabel's no-arg fallback, and the toPass/toPassAll matchers. Adds targeted unit tests for each — all tests still pass, coverage back to 100%. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
71b6c4c3d3 |
fix: finish audit→lint rename — repair CI build, shipped hook, and docs (#34)
* fix: finish audit→lint rename — repair CI build, shipped hook, and docs
The `audit` command was renamed to `lint` in
|
||
|
zernie
|
c1aa8fd081 |
feat: markdown-first adoption — YAML frontmatter rules + JSON Schema (#22)
Add Level 1 markdown adoption: vigiles: YAML frontmatter rules verified by audit, generate-schema for YAML-LSP autocomplete, and a markdown-first repositioning of the README and docs. Spec.ts (Level 2) and inline mode (Level 0) unchanged. |