mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
main
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
848538d888 |
feat(site): sell the test and eval tiers with measured OSS findings (#218)
Render the test and eval tiers as measured findings rather than feature bullets: a real vendored OSS guard's own verdicts through runHook, and measureTriggerRate on superpowers' TDD skill at 83% over 30 runs. Both render from committed fixtures, one regenerated from the engine and pinned by a browser test, the other copied from the lock so CI spends no model quota. Adds a /comparison page rendered from a probe snapshot, retires four unmeasured claims plus two shipped-but-"planned" rows, and type-checks rendered snippets against the real .d.ts instead of scanning them with a regex. Committing this repo's first eval lock also revealed that the CI eval-check step had never once executed — it short-circuited on anyLocksCommitted and exited 0. Two defects behind it are fixed: a quota consent gate that refused a model-free verify, and a dogfood step that now names the one eval holding a lock. |
||
|
|
195c1e14b7 |
feat: launch blockers, a hook-state test API, and the site's missing proof (#210)
Ten launch blockers, all sharing one cause: a first reader gets a false claim or an embarrassing failure in their first ten minutes. Fixes: ruff enabled-state was never read on a real repo; compile was unreachable without node_modules/vigiles (every Python/Rust/Go repo); a hooks-only plugin graded F/0 while its own report listed inlineHooks: 2; skill-resource findings printed a body-relative line number; the README quoted a caveman number its own FINDINGS.md had invalidated; STABILITY.md claimed v21 against 26.0.1; the Action's floating v1 tag was never created; a react's notice() reached nobody. Features: experimental_hookState, so a stateful compiled hook can be tested by backdating the fact it throttles on; the site now leads with the proof (compiled hooks, the disaster battery, 2/7) and its OG card is rendered rather than hand-drawn, with a generator that fails loud if the page fetches anything off file:. Also: this repository's first compiled hook, which is what exposed the notice-delivery defect; a floor under the harness CI job, which was green on zero files; and a smoke test of the PUBLISHED package, since every gate until now ran dist/. |
||
|
|
41169e379a |
fix(audit,testing): split Tested/Evaluated, make coverage evidence honest, and stop three silent-success paths (#129)
* feat(audit): split Tested into Tested + Evaluated, with a `not measured` state `Tested` collapsed deterministic harnesses and real-model evals into one number, and its finding read "N surfaces with no vigiles test/eval" — one sentence for two prescriptions three orders of magnitude apart in cost. A harness (`*.harness.mjs`, `*.test.*`) is free, runs in milliseconds on every push, and asks whether a gate still catches what it claims. An eval (`*.eval.mjs`) spends real model calls, runs on a schedule, and asks the one question a deterministic read cannot: does this skill fire at all? Folded together, a repo with complete deterministic coverage and no evals scored identically to a repo with neither. - test-coverage: one discovery pass, partitioned by suffix into a harness tier and an eval tier; the union `covered`/`untested` fields are unchanged. Mirrored in the browser-safe twin. - scan: carries `untestedHarness` / `unevaluated` / `evaluable` (optional and additive; a producer predating the split falls back to the union). - audit-score: `Tested` (deterministic) and `Evaluated` (real-model), both advisory. `Evaluated` has THREE states — a number, a measured `0`, and `not measured` when the executing checks were skipped and no eval exists. Scoring 0 there would report the absence of a check as its result, which is the pattern this tool exists to name elsewhere. In that state the ring names the command that would measure it. - cli: threads the existing execute-skip signal into the ring. - report/HTML/JSON: both rings render; the inventory carries both counts. Grade weighting is unchanged — both rings are advisory and neither enters `reportDeductions`. Audited on this repo before and after: A (100/100) both times; only the diagnostic rings move (Tested 67 → Tested 58 + Evaluated 22). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * docs(site): the audit grades six categories, not five Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * fix(lint): a clean union must not read as 'firing verified' Every surface can have a deterministic harness while nothing has ever measured that a skill fires. The gate is unchanged (still the union), but the ✓ now says so instead of implying the question was asked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * test(site): regenerate the browser-parity fixture for the Evaluated ring The audit now emits a sixth category, so the Node-generated expected report and the in-browser one differ until the fixture is regenerated — which is what the test's own header prescribes for an intentional engine change. Regenerated with `node site/scripts/gen-parity-expected.mjs`. The diff adds the Evaluated ring and its two inventory counts and nothing else: grade and overall are untouched, which is the check that the split did not move the weighting. * fix(coverage): `Tested` measured a STRING, not a test DEFECT. A surface counted as covered when its path or namespace appeared as a SUBSTRING anywhere in any discovered test file (`content-reference`, in src/test-coverage.ts). Two halves, and the second was worse. REPRODUCTION (a real repo: 37 skills, 14 hooks). Appending ONE line to an existing harness file: // probe: skills/argument-arc moved the audit's untested count 33 -> 32. A comment. Not a test. Reverting the line put it back. So `Tested` was trivially gameable and the gaming was indistinguishable from real work. The false NEGATIVE: that repo's `.claude/pipeline/skills.harness.mjs` genuinely asserts over all 21 of its pipeline skills, but builds each path at runtime (`join(root, ".claude", "skills", name, "SKILL.md")`), so it contains ZERO literal `skills/<name>` strings (grep: 0 matches) and covered NOTHING. Generality was PENALISED: 21 trivial files each naming their own subject would have scored better than the one harness that actually tests all 21. This is the exact substitution vigiles names in other people's repos — presence of a surface taken for presence of the property, the way `noExplicitAny` in a config is taken for the rule being enforced. The tool was committing it in its own scoring. FIX, three parts (new src/coverage-evidence.ts — pure and browser-safe, so the disk detector and its in-browser twin cannot drift on what decides a grade): 1. DECLARATION. A test file may carry `vigiles:covers <surface> ...`, mirroring the existing `vigiles:ignore-test` marker convention on the other side. A runtime-path harness can now say what it covers; an accidental mention cannot. Declaration beats inference — the same move as declaring `allowed-tools` instead of inferring capability. 2. PROVENANCE. Every coverage decision now carries HOW it was reached — declared / colocated / mention — and the counts are printed by `formatUntestedReport`, by `formatScanReport`, and carried in `AuditReport.inventory.coverageEvidence` (additive; schema version unchanged). A repo whose coverage rests entirely on substring matches can finally see that about itself; "28 covered" and "28 names that happen to appear in a file" used to print identically. 3. The bare substring detector is KEPT, deliberately, and said so in the code: it is the zero-config path that makes an ordinary `foo.test.ts` naming `skills/foo` count without anyone learning a marker, and dropping it would flip honestly-covered surfaces to "untested" overnight. What it no longer reads is COMMENTS. A path in a comment is prose about a test; a path in executable code is the test reaching for the surface. Prose isn't policy, including ours. It is also now visibly the weakest tier in the report, and when it is the ONLY evidence the report says so in words. Grade weighting is untouched. TESTS prove both halves, each watched failing first: with the fix reverted, "a surface named ONLY in a comment is NOT covered", "a runtime-path harness that DECLARES its surfaces is counted" and "a declaration OUTRANKS a mention" fail (3 failed | 20 passed); with it, 23 passed. Also guarded: a mention in CODE still counts, and a `"https://..."` string literal survives comment-stripping. Docs (untested-skill/-subagent/-hook) updated — untested-skill.md previously recommended "add the skill's path in a comment", which was advice to exploit the defect. * fix(site): key the browser-test prebundle to dist, so a stale cache can't lie DEFECT. `site/src/demo/browser-parity.browser.test.ts` guards node:zlib vs pako divergence by comparing an in-browser AuditReport against one generated in Node. `site/vitest.config.ts` pre-bundles the engine (`optimizeDeps.include` on `@engine/scan-files` / `@engine/audit-report` / `pako`) and Vite caches the result in `site/node_modules/.vite`. That cache SURVIVES `npm run build` — Vite invalidates on config/lockfile changes, not on the mtimes of an aliased path outside node_modules. So the browser side can compute against a PREVIOUS engine while Node computes against the current one. REPRODUCTION (observed 2026-08-07, twice — once by accident, once deliberately). Accidentally: an engine change added a report field; the parity test failed showing the field present on the Node side and absent in the browser. I read that aloud as a genuine engine divergence and started hunting a pako bug that does not exist. `rm -rf site/node_modules/.vite` -> 42/42 pass. Deliberately, to prove the fix: plant an extra `inventory` field in src/audit-report.ts, `npm run build`, regenerate the expected fixture, run the browser suite with a warm cache. - ORIGINAL config: `1 failed | 41 passed`, diff showing `"probeStaleCache": 1` on one side only — a field-level diff that reads exactly like a real divergence and says nothing about the cache. - THIS config: `42 passed`, cache dir rotated `.vite-engine-8baf73f9e373` -> `.vite-engine-a52a27e0137e`. Reverting the plant returns the fixture to its committed bytes. FIX. Key the prebundle cache DIRECTORY on a fingerprint of `dist/` (`site/vite.engine-stamp.ts`; `cacheDir: node_modules/.vite-engine-<stamp>`), sweeping the directories keyed to other fingerprints so they don't pile up. WHY a key and not the alternatives. Comparing the cache's mtime against `dist/` and forcing a re-bundle, or clearing `.vite` in the test script, both work and both leave the bad state REACHABLE — a comparison someone can get wrong, or a script step bypassed by running `vitest` directly. Keying the directory removes the state instead: a different `dist/` is a different path, so "prebundle built from another engine" cannot be reached, and an unchanged `dist/` still gets a warm cache. By construction rather than by check — the argument this project makes everywhere else about removing a capability rather than policing its use. The test's header instruction ("pako and node:zlib disagree — do NOT paper over it") was correct for one of the two states and actively misleading for the other; it now says which state is no longer reachable and why, and the assertion carries the same note in its failure message. * fix(site): make the parity-fixture generator emit prettier-clean JSON DEFECT. `site/scripts/gen-parity-expected.mjs` wrote the fixture with `JSON.stringify(x, null, 2)`, which is NOT what prettier prints for JSON — it keeps short arrays exploded across three lines where prettier collapses them. CI gates on `prettier --check .`, so every regeneration of the fixture committed a file that reddened CI later and far from the cause. It just did: branch head |
||
|
|
9de2e6905e |
feat(site): demo UX redesign — repo autocomplete, per-check pages, share links, softer palette
Repo-name autocomplete (no org needed, live stars); dedicated /checks/<slug> explainer pages linked from every finding (Vite MPA); stronger trigger-rate tease + graded leaderboard chips; shareable ?repo= grade links + selling copy; plain-language lethal-trifecta (ships in the CLI report too); softer material-oceanic palette across the site and the local HTML report. Deterministic report changes flow through the shared @vigiles/report-view. Includes 14 Codex review fixes (autocomplete request races + stale-result guards, honest empty-fixes/clean state, safety-card gating, GitHub-API disclosure, Claude-Code-only trigger-rate CTA). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
8fa0f236cb |
feat(site): live in-browser "Grade any repo" audit demo + README-parity content (#100)
* docs: refresh HANDOFF — demo shipped (#99); PR 3 live engine in progress via subagents * feat(scan): browser-safe audit engine (scanFiles) with CLI-parity gate Add src/scan-files.ts — scanFiles(files: Record<path,content>) runs the deterministic audit over an in-memory file map with NO node:fs/child_process, so the in-browser grade is byte-identical to the CLI's `audit --json`. Reconstructs the LoadedPlugin shape from the map, REUSES scan.ts's pure per-surface detectors, and reimplements only the ~6 fs touchpoints as map-backed pure logic. src/test-coverage-files.ts is the browser-safe twin of findUntestedSurfaces (globSync → Object.keys filter). scan.ts: export the pure detectors the engine reuses and thread a backwards-compatible injectable `exists` (defaults to existsSync) through resolveScript/scanHooks + `existsSync?` on SkillScanContext — no disk-path behaviour change. src/scan-files.test.ts is the correctness firewall: for every SHA-pinned vendored plugin, scanFiles(map) must produce an AuditReport byte-identical to scanPlugin(dir) (both the buildAuditReport output and the raw ScanReport), normalizing only the checkout path (BROWSER_ROOT). Foundation for the in-browser "grade any repo" demo. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(scan): import editDistance from edit-distance.js not linters.js Drops the whole linters.ts module (glob + node:module + child_process import-time weight) from scan.ts's transitive graph — it was pulled in only for editDistance, already extracted to core/edit-distance.js. Shrinks the browser-bundle graph for the in-browser audit engine; no behaviour change (same editDistance). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(core): extract NCD to node-free core/ncd.ts Move ncd() + its gzip helper out of proofs.ts (which imports node:crypto via hash.js) into a node-free leaf so the description-overlap detector reaches it without dragging crypto into the in-browser audit bundle. proofs.ts re-exports ncd for its existing consumers; description-overlap.ts imports it from the leaf. Swaps Buffer.from(s,'utf-8') → TextEncoder so the byte input is identical in Node and the browser and the node-only Buffer global is gone (the browser build aliases node:zlib → pako). No behaviour change — parity + proofs + overlap tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): record node-free engine decision (module-split, not stubs) The Stage-2 build plan in report-view-and-browser-demo.md prescribed the bundler-stub approach that was rejected. Rewrite it to the decision of record: module-splitting over stubs (silent-divergence risk) and over dynamic-imports (dead code needs no lazy selection; sync engine), the dist/ require-graph tracing method (scan.js is the single bridge), the splits (editDistance/ncd/ scan-core + required-IO + mcp-message leaf + posix helper), the one real zlib->pako alias, DI as the runtime adapter, and the browser-mode parity test that closes the pako-vs-node gap. Plus the site<->engine integration (site consumes built dist CJS) and this-branch status. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(scan): node-free the in-browser audit engine (scan-core split) Extract the pure audit detectors into node-free leaf modules so the browser engine (scan-files.ts / test-coverage-files.ts) reaches NONE of node:fs, node:crypto, node:child_process, node:path, @ast-grep/napi, or glob. The dist/ require-graph for the browser engine is now: 29 files, builtins = only node:zlib (→ pako in the Vite build), pkgs = @iarna/toml, js-yaml, mvdan-sh (all pure-JS, browser-bundleable). Module-splitting (the core ⊄ adapter hexagonal pattern), NOT bundler stubs or dynamic imports — the node deps are transitive-dead on the browser path, so the clean kill is removing them from the static graph, with zero silent-divergence risk. Tracing the compiled dist/ require-graph showed scan.js is the single runtime bridge, so one scan-core extraction removes the whole node subtree. - src/scan-core.ts: the moved pure detectors (preserves the \x00 matcher delimiter byte-for-byte). scan.ts re-exports via so every existing consumer is unchanged. - src/posix-path.ts: node-free POSIX path ops (verbatim path.posix port) — drops node:path from the engine entirely. - src/core/mcp-contract-message.ts, src/core/assert-never.ts, src/adapters/claude-code/agent-tools.ts: pure leaves extracted from mcp.ts / hash.ts / agent-runtime.ts (each re-exports the leaf, so no public surface changes — api gate confirms no drift). - hook-block-ineffective.ts / plugin-dir-layout.ts / skill-resources.ts: the node:fs default made required-injected IO (only callers are the two engines; the browser injects map-backed impls). Firewall held: src/scan-files.test.ts (byte-identical CLI parity on 4 vendored plugins) unchanged + green; full unit suite green at 100% coverage; api no drift. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style(research): prettier-format the browser-demo doc * docs: refresh HANDOFF — node-free engine pushed; demo wiring + site content additions next * refactor(scan): node-free buildAuditReport via score-core leaf buildAuditReport re-tainted the browser graph (@ast-grep/napi native binary + node:fs/crypto/child_process) through audit-score/optimize/audit-verdict → leaderboard.ts → scan.ts. leaderboard.ts eagerly imports scanPlugin (used only in the CLI-only rankPlugins), so importing its pure scoring functions dragged the whole native chain in. Extract the pure scoring core into node-free leaf src/score-core.ts (PluginScore, W_* weights, Deduction, gradeFor, reportDeductions, isEmptyMachine, computeIntegrityScore, scoreReport — same pattern as core/ncd.ts / scan-core.ts). leaderboard.ts re-exports them and keeps its node-only scanPlugin/rankPlugins; audit-score/optimize/audit-verdict/audit-report import from score-core. Now BOTH browser entry points are node-free: the dist require-graph union of scan-files.js + audit-report.js has builtins = only node:zlib (→ pako in the site build), pkgs = @iarna/toml/js-yaml/mvdan-sh. audit-report.js went 41 files (+ ast-grep) → 6 files, zero node deps. Internal-only (no api drift); coverage-neutral; parity firewall + full unit suite (100% cov) green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(site): live in-browser 'Grade any repo' audit demo Type any public GitHub repo and get a real, live-computed audit report rendered by the SAME @vigiles/report-view components the CLI uses — the browser runs the literally-same compiled engine (dist/ CJS) with one node:zlib→pako swap, so the grade is byte-identical to the CLI. - Vite wiring: @engine/* aliases → built dist/scan-files.js + dist/audit-report.js (node-free), node:zlib → src/lib/zlib-shim.ts (pako.gzip), commonjs transform of the dist graph. - fetchRepo.ts: GitHub Trees API (recursive, once) + raw.githubusercontent.com for contents (not rate-limited), harness paths only, size-capped, honest progress events; typed handling of 404 (private) / 403 rate-limit / no-harness. - runAudit.ts: scanFiles → buildAuditReport; DemoAudit.tsx rewritten to the Fable live-typing UX — input → honest promise-driven 3-step loading → discriminated report|empty|notfound|ratelimit|error states, ?repo= deep-link + auto-run, baked chips kept (instant), edge states carry their own in-frame CTA. - Tests (3 layers): browser-parity (real Chromium + pako, asserts runAudit === the Node-generated AuditReport — closes the pako-vs-node-zlib gap), interaction (fetch mocked, real engine), one Playwright e2e (built bundle, network mocked). Site build has zero node-builtin/native leaks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(site): add how-it-works, adoption, and FAQ sections Close the gap between the landing page and the README — the site had a strong top (hook + problem + demo) but was missing the middle (what vigiles does across its verbs + how you adopt it) and the objection-handling. - VerbMap ('One tool. Four questions.'): the audit/lint/test/eval verb map as a type+divider table (stacks on mobile), the 'one engine, two doors' line, and a compact 'your rules → enforced' card (the prose→enforcement payoff). - Adoption ('One command, then your agent does the rest.'): the three shipped skills (test-harness/strengthen/edit-spec), the in-loop hooks, and a copyable paste-into-Claude-Code/Codex setup prompt (wraps on mobile). - FAQ: the framework? / markdown-linter? / TypeScript? / stability / non-JS objections, as a type-only list (no boxes). - Debunk: fold in the promptfoo/DeepEval 'bill per token every run' cost contrast. Order: Hero → Wedge → VerbMap → Debunk → DemoAudit → Adoption → FAQ → CTA. Each held to the landing-site subtraction bar (type + space over stacked boxes, no cross-section command duplication); verified desktop + 390px mobile. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(site): apply Fable design cuts to the new sections Fresh design pass flagged tail-end restatement (not crowding). Cuts: - FAQ: drop 'framework?' + 'markdown linter?' — they restated VerbMap's 'two doors' and Wedge's 'valid != true'; keep the three net-new objections (TypeScript / stability / non-JS). Full FAQ link carries the rest. - Demo: remove the 'Now grade yours' tail command block — it was the 4th npx-vigiles-audit surface (hero + sticky nav + final CTA already carry it). - VerbMap: make the right column self-explanatory ('no model · read-only · anytime' … 'needs a model · your subscription · on demand'). - Adoption: shrink the hooks paragraph to one line. - Demo chips: wrap + center on every width so a chip never clips at the mobile edge (four chips → two tidy rows). - .prettierignore: ignore site Playwright/vitest-browser run artifacts. fmt clean; site build green; browser tests 9/9. Verified desktop + 390px. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF — node-free engine + live demo + site content shipped * fix(demo,scan): address PR review — hook-script fetch + dir-aware browser exists Two browser-vs-CLI divergences the Codex review flagged (both weaken the byte-identical-to-the-CLI guarantee): - fetchRepo (P1): a hook command can reference a script OUTSIDE the harness dirs, e.g. ${CLAUDE_PLUGIN_ROOT}/scripts/guard.sh. The harness-path filter dropped it, so the browser scan reported the hook as missing and applied the graded penalty even though the tree showed the file exists. Add a bounded second fetch pass: collectReferencedPaths() scans fetched harness files for plugin-root/project-dir token refs and fetches any that actually exist in the tree (size-capped), matching the CLI's whole-repo read. - scan-files (P2): mapExists mirrored fs.existsSync but returned false for a directory (file keys only), so pluginDirLayoutIssues' exists(dir) && isDirectory(dir) silently missed the layout finding the disk scan reports. Make mapExists faithful to fs.existsSync — true for a file OR a directory. Parity firewall unchanged + green; site build 0 leaks; browser tests 9/9. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(pages): build root dist/ before the site build The Pages deploy built site/ right after npm ci with no root build, but the landing's live audit demo now imports the compiled engine (dist/scan-files.js + dist/audit-report.js) via the site's @engine/* Vite aliases, and dist/ is a gitignored build artifact. A clean Pages deploy would fail to resolve those imports (not caught by PR CI, which builds dist/ in its own jobs — only on merge-to-main deploy). Add a root 'npm run build' step first. Validated by running the pages sequence against a clean dist/. * fix(demo): scope the live demo to Claude Code (drop unused Codex inputs) fetchRepo pulled AGENTS.md / .codex but runAudit always scans with the Claude Code layout, so a Codex-only repo was graded as CC — its Codex surfaces ignored, yielding a misleading near-empty report. The engine's Codex audit is full-parity, but the demo doesn't wire the Codex layout/dialect yet (a follow-up). Make the demo honestly Claude-Code-only: drop AGENTS.md/.codex from the harness fetch so a Codex-only repo lands in the no-harness state that points at the CLI, not a wrong-layout report. Empty-state copy + the two edge-state test assertions updated to match. (Do not re-add Codex inputs without wiring the Codex adapter into runAudit.) Browser tests 9/9, e2e 2/2, fmt clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(demo): harden repo/harness detection (PR review round 2) Three more browser-fetch correctness issues from the Codex review: - isHarnessPath matched a harness word as ANY path segment, so an ordinary repo with src/hooks/useThing.ts (a plain React app) was fetched as a harness and graded an empty machine instead of showing no-harness. Match harness dirs only as the TOP-LEVEL segment (or a .claude/.claude-plugin root). - collectReferencedPaths only caught ${CLAUDE_*} token refs, so a hook using a RELATIVE script path outside the harness dirs (scripts/guard.sh, ./bin/x.sh — a form scanHooks resolves against the plugin root) was still dropped → false missing-hook penalty. Also collect relative dir-qualified script paths. - The recursive Git tree is truncated for very large repos (>100k entries), so the needed harness files can fall outside the returned slice → a misreported no-harness or partial grade. Read tree.truncated and bail honestly to a new 'too-large — grade it locally' in-frame state. Adds fetchRepo.browser.test.ts guarding isHarnessPath (nested src/hooks excluded) and collectReferencedPaths (token + relative script refs, prose ignored). Site build + browser tests 14/14 + e2e 2/2, fmt clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(demo): require a real harness marker + fetch manifest-declared hooks Two more browser-fetch/CLI divergences from the review: - A plain repo with a top-level hooks/ (git hooks — husky, pre-commit) or other bare surface dir passed the harness filter, so fetchRepo returned ok and the demo graded an empty machine instead of showing no-harness. Add a GATE: the fetched set must contain a definitive marker (isHarnessMarker) — CLAUDE.md, .mcp.json, anything under .claude/.claude-plugin/, hooks/hooks.json, or a real top-level surface FILE (skills/<x>/SKILL.md, agents/<x>.md, commands/<x>.md). A bare hooks/ of scripts with no declaration is not a harness (matches the loader, which only loads declared hooks). - A plugin manifest can point its hooks field at a file outside the harness dirs ("hooks": "config/hooks.json"); the scan's readHooksJsonFile then found nothing and dropped every hook. The 2nd fetch pass now also pulls the manifest-declared hook-config path. Tests cover isHarnessMarker (git-hooks/ordinary files rejected, real markers accepted). Build + browser 16/16 + e2e 2/2, fmt clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF — PR #100 open + auto-merging with review fixes * docs: HANDOFF — mark PR #100 as the vehicle (consistency) * fix(demo): root SKILL.md harness + chained hook-config script refs Two more browser-fetch/CLI divergences from the review: - A single-skill repo (a root SKILL.md — loadPluginFromFiles' 'single-skill' case) wasn't in the root allowlist, so its only surface was never fetched and it fell through to no-harness. Add SKILL.md to the fetched root files AND the marker gate, so a public single-skill repo is graded. - The manifest-hooks fetch was single-shot: a manifest-declared config/hooks.json was fetched AFTER collectReferencedPaths ran, so that config's OWN out-of-harness script refs (manifest -> config/hooks.json -> scripts/guard.sh) were never scanned. Make the second pass a bounded fixpoint (re-scan newly-fetched files, <=3 rounds, capped at MAX_FILES) so realistic reference chains resolve. Test covers the root-SKILL.md marker. Build + browser 16/16 + e2e green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(site): plainer report copy, marketplace state, progress bar Live UX-review fixes on the "grade any repo" demo: - Report copy: replace the jargon "maximal trifecta blast radius" with plain "no limits on what it can read, fetch, or run" for the inherits-all safety finding (engine copy in audit-score.ts — ships in the CLI report too). Baked demo fixtures + the parity expected fixture regenerated to match. - Marketplace repos: a repo whose only .claude-plugin surface is a marketplace.json is a COLLECTION, not one gradeable harness — the browser demo would have rendered a bogus zero-surface F (the CLI expands members via inspectMarketplace, a path the demo doesn't run). Route it to an honest "this is a plugin marketplace — grade members with the CLI" state. A repo that also ships plugin.json is a self-publishing single plugin and still grades. - Loading: add a real progress bar under the "reading harness files" step. - A-grade note: drop the vague "the row below is your next step"; state the next step self-containedly (a browser can't ask a model, your CLI does). - Report footer: gate "Generated by vigiles …" behind a showFooter prop (default on for the standalone HTML report; off in the in-page demo, where the page already brands). Tests: audit-score + scanFiles parity green; site browser tests 17/17 incl. a new marketplace edge-state case. * fix(site): name the lethal trifecta, stop caching transient errors, drop analytics slug More live UX-review fixes on the "grade any repo" demo: - Report copy (round 2): the inherits-all safety finding now NAMES the concept and explains it — "the lethal trifecta (reads data, reaches the web, runs commands), so a prompt injection could exfiltrate secrets" — instead of the vague "no limits …". Engine copy (audit-score.ts), so the CLI report gets it too; fixtures regenerated. - Transient failures stay retryable: only STABLE outcomes are cached now (isCacheable) — a dropped-connection error / rate-limit is no longer cached, so the frame's own "try again" actually re-fetches instead of replaying the cached error until a reload. - Analytics never sees the typed slug: track() now sends the outcome KIND only, never owner/repo — the demo promises nothing leaves the browser but GitHub requests, and a typed slug would leak a private repo name (→404) to a future Plausible. (track() is a no-op shim today; this is future-proofing.) - Root SKILL.md name: a nameless single-skill repo is labeled after the repo (threaded into scanFiles → the CLI's audited-dir-basename behavior) instead of the synthetic "__vigiles_repo__". New parity-firewall test guards it. Tests: scanFiles parity + single-skill naming green; site browser 17/17; engine audit-score green; tsc + fmt + lint clean. * feat(site): persistent grade-cache (idb-keyval) with honest age + re-grade A shared ?repo= deep-link, a reload, or a revisit now renders a previously computed grade INSTANTLY and with ZERO GitHub requests (dodging the anonymous 60-req/hr limit) — without ever showing a silently-stale grade. Design (Fable pass): idb-keyval was the ADOPT over the reactive useLocalStorage hooks (@react-hookz/web / usehooks-ts / react-use) — the cache is accessed imperatively under a dynamic key (the slug) and needs TTL + versioned invalidation + LRU, none of which a fixed-key storage hook provides; idb-keyval's structured-clone + per-entry keys fit exactly (no JSON-parse bug class, no whole-object multi-tab race). - site/src/demo/gradeCache.ts — TTL 24h, version-namespaced key (schemaVersion.engineVersion), 30-entry LRU, parse-don't-validate boundary that rejects corrupt/wrong-schema/future/expired entries; never throws (storage-disabled browsers run cache-off). - Version namespace = AUDIT_SCHEMA_VERSION + the build-time git SHA (__ENGINE_V__ via a Vite define; the root package.json version is a release-time placeholder in-repo, so it can't serve here). A deploy that changes the engine invalidates every prior grade — over-invalidation is safe; a stale grade is the only hazard. - Persist only report/empty — never notfound (a private repo name on disk + a trap after it goes public) or a flippable/transient state. - DemoAudit: L1 in-memory Map (all stable outcomes) layered over L2 gradeCache (report/empty); run() falls through L1 → L2 → network, runId-guarded; a "graded N ago · ↻ re-grade" header badge makes every cache hit honest and one-click refreshable; sweepGrades() on mount drops old-namespace/expired keys. Tests (real Chromium + real IndexedDB): gradeCache round-trip / TTL expiry / corrupt-reject / wrong-namespace sweep / LRU cap; DemoAudit deep-link zero-fetch + forced re-grade bypasses cache. Site browser suite 24/24; tsc + fmt clean. * fix(demo): fetch root single-skill bundled resources before grading A root single-skill repo (`SKILL.md` at the top level) that references a bundled resource — `references/api.md`, `assets/example.json`, a `[guide](assets/x.md)` link — had those files DROPPED by the harness-path filter (only script extensions were collected in the second pass, and non-script resources at the repo root aren't under a harness dir). `scanFiles` then ran `skillResourceIssues` against a partial tree and reported a false missing-resource penalty, where `vigiles audit` on disk reads the whole skill dir. Extend `collectReferencedPaths` to also collect bundle-dir paths (`scripts`/`references`/`assets` with any extension) and markdown-link relative targets, so the tree-present ones are fetched. (Nested-skill resources under `skills/**` were already fetched via the harness dir.) Over-fetch is harmless — only tree-present, size/count-capped paths are pulled. Newly relevant since root SKILL.md repos now grade. Browser tests +2 (bundled resources caught, external links ignored). * docs: refresh HANDOFF — live UX-review round (copy, marketplace, grade-cache) * ci: run the site browser suite (parity + interaction + grade-cache) Add a `site` job to ci.yml — build the root dist/, install Chromium, run `npm run test:browser` (the browser-mode vitest suite: real Chromium + real pako parity, the DemoAudit interaction states, and the idb-keyval grade-cache over real IndexedDB). Closes the gap where a demo/engine-parity regression was only caught locally. Playwright `test:e2e` stays manual/browser-gated. * fix(demo): never grade partial data (fetch failure / file-cap truncation) Two ways the demo could publish a grade computed over an INCOMPLETE harness — both now bail honestly instead of grading partial data: - A raw.githubusercontent.com fetch that fails or returns non-OK (a transient 500 / CORS drop on a harness surface like SKILL.md or .claude/settings.json) used to be skipped silently, leaving the file absent and the grade partial. Now first-pass surface fetches are failure-counted; any failure → the retryable `error` state (not cached, so "try again" re-fetches). Second-pass referenced files stay best-effort (a missing one is a soft advisory, not a hard fail). - A repo with more harness surfaces than the browser reads fully (> MAX_FILES) used to silently slice off the overflow — dropping skills/agents/hooks, or even the definitive marker (→ false no-harness). Now the marker gate runs on the FULL harness set, and an over-cap harness → the `too-large` state (bail to CLI), same as a truncated tree. Tests: 3 fetchRepo integration cases over a stubbed GitHub (too-large on 301 surfaces, error on a failed surface fetch, ok when all fetch). Site browser 29/29. * fix(demo): error on required second-pass fetch failures; fetch coverage tests Two more browser-vs-CLI parity gaps in the fetch layer: - REQUIRED second-pass files that fail to fetch now error (were silently OK). A manifest-declared hook config (config/hooks.json), a referenced hook script, or a skill bundled resource all feed GRADED findings — a transient 500/CORS drop on one left the grade partial. The first-pass failure rule now applies to the second pass too (a `countFailure` flag on fetchBlob/drain distinguishes required from advisory fetches). - COVERAGE test files outside the harness dirs are now fetched (best-effort). The untested-surface check reads tests that live in a top-level tests/ or a colocated *.eval.mjs — the harness filter dropped the non-colocated ones, so the browser showed a false `untested` advisory the CLI's whole-repo read wouldn't. New isTestPath collects them, budget-bounded; a failure does NOT error (untested is excluded from the overall grade — cosmetic, not a wrong grade). Tests +4: required second-pass failure → error, a coverage test outside harness dirs is fetched, a coverage-test failure stays ok, isTestPath matching. Site browser 33/33; tsc + fmt clean. * fix(demo): second-pass cap → too-large; build engine before e2e - Complete the "never grade partial data" invariant: the first-pass harness cap already bailed to too-large, but the SECOND pass (required referenced files — hook configs, scripts, bundled resources, all graded) still silently sliced to the remaining budget. Now if the eligible referenced set exceeds the budget it returns too-large too, instead of grading a partial repo. Test: 300 surfaces + a referenced resource with no budget left → too-large. - e2e: the site's test:e2e webServer built only the site, but the @engine/* Vite aliases resolve to the gitignored root ../dist/ — so a clean checkout failed before preview. Prepend the root engine build (npm --prefix .. run build). Site browser 34/34; tsc + fmt clean. * docs: document the browser demo's bounded-fetch limitation Codex keeps flagging "file X outside the harness dirs isn't fetched" on fetchRepo — all one root cause: the demo does a BOUNDED, SELECTIVE fetch (to respect GitHub's 60-req/hr anonymous limit), not the CLI's whole-repo read. So document it as the intentional, safe-by-invariant design it is, rather than chase an infinite tail: - fetchRepo.ts MODULE HEADER (where Codex + contributors read): the NEVER-GRADE-PARTIAL-DATA invariant (too-large/error bail-outs; advisory data best-effort), what IS fetched to match the CLI, the documented non-goals, and the "unless it's a wrong grade not already covered, it's the accepted cost" framing. - research/browser-demo-fetch-limits.md: the canonical record + the edge-case ledger (every divergence Codex found on PR #100 → closed or a documented non-goal, incl. extensionless relative scripts). Indexed in the research spec (dogfood green) so the next session finds it. - HANDOFF pointer so the next session doesn't re-litigate the tail. No behavior change — documentation only. * fix(demo): collect bundled-resource refs only from SKILL.md (regression) The bundled-resource collector (BUNDLE_RES / MD_LINK_REL) ran over ALL fetched files, but the engine only grades those refs from a SKILL.md body (skillResourceIssues). So a normal CLAUDE.md with ordinary doc links like `[guide](docs/x.md)` had those docs treated as REQUIRED second-pass files — enough of them → a false `too-large`, or one failed fetch → a false `error`, refusing to grade a repo the CLI grades fine. Restrict BUNDLE_RES/MD_LINK_REL to SKILL.md files (hook-script refs stay all-files, since a hook command lives in a manifest/settings/config). Test: a CLAUDE.md's md-links are dropped, a SKILL.md's kept, a hook script kept. Also document (research/browser-demo-fetch-limits.md) the narrow oversized-bundled -resource edge (a > 256 KiB SKILL.md resource skipped by the size cap → a possible false-missing) as an accepted, deferred approximation. Site browser 35/35; tsc + fmt clean. --------- Co-authored-by: Claude <noreply@anthropic.com> |