5 Commits

Author SHA1 Message Date
zernie 848538d888 feat(site): sell the test and eval tiers with measured OSS findings (#218)
Render the test and eval tiers as measured findings rather than feature
bullets: a real vendored OSS guard's own verdicts through runHook, and
measureTriggerRate on superpowers' TDD skill at 83% over 30 runs. Both render
from committed fixtures, one regenerated from the engine and pinned by a
browser test, the other copied from the lock so CI spends no model quota.

Adds a /comparison page rendered from a probe snapshot, retires four unmeasured
claims plus two shipped-but-"planned" rows, and type-checks rendered snippets
against the real .d.ts instead of scanning them with a regex.

Committing this repo's first eval lock also revealed that the CI eval-check
step had never once executed — it short-circuited on anyLocksCommitted and
exited 0. Two defects behind it are fixed: a quota consent gate that refused a
model-free verify, and a dogfood step that now names the one eval holding a
lock.
2026-09-09 19:31:45 +05:00
zernie 195c1e14b7 feat: launch blockers, a hook-state test API, and the site's missing proof (#210)
Ten launch blockers, all sharing one cause: a first reader gets a false claim or an embarrassing failure in their first ten minutes.

Fixes: ruff enabled-state was never read on a real repo; compile was unreachable without node_modules/vigiles (every Python/Rust/Go repo); a hooks-only plugin graded F/0 while its own report listed inlineHooks: 2; skill-resource findings printed a body-relative line number; the README quoted a caveman number its own FINDINGS.md had invalidated; STABILITY.md claimed v21 against 26.0.1; the Action's floating v1 tag was never created; a react's notice() reached nobody.

Features: experimental_hookState, so a stateful compiled hook can be tested by backdating the fact it throttles on; the site now leads with the proof (compiled hooks, the disaster battery, 2/7) and its OG card is rendered rather than hand-drawn, with a generator that fails loud if the page fetches anything off file:.

Also: this repository's first compiled hook, which is what exposed the notice-delivery defect; a floor under the harness CI job, which was green on zero files; and a smoke test of the PUBLISHED package, since every gate until now ran dist/.
2026-09-08 00:52:20 +05:00
zernie 41169e379a fix(audit,testing): split Tested/Evaluated, make coverage evidence honest, and stop three silent-success paths (#129)
* feat(audit): split Tested into Tested + Evaluated, with a `not measured` state

`Tested` collapsed deterministic harnesses and real-model evals into one
number, and its finding read "N surfaces with no vigiles test/eval" — one
sentence for two prescriptions three orders of magnitude apart in cost.

A harness (`*.harness.mjs`, `*.test.*`) is free, runs in milliseconds on
every push, and asks whether a gate still catches what it claims. An eval
(`*.eval.mjs`) spends real model calls, runs on a schedule, and asks the one
question a deterministic read cannot: does this skill fire at all? Folded
together, a repo with complete deterministic coverage and no evals scored
identically to a repo with neither.

- test-coverage: one discovery pass, partitioned by suffix into a harness
  tier and an eval tier; the union `covered`/`untested` fields are unchanged.
  Mirrored in the browser-safe twin.
- scan: carries `untestedHarness` / `unevaluated` / `evaluable` (optional and
  additive; a producer predating the split falls back to the union).
- audit-score: `Tested` (deterministic) and `Evaluated` (real-model), both
  advisory. `Evaluated` has THREE states — a number, a measured `0`, and
  `not measured` when the executing checks were skipped and no eval exists.
  Scoring 0 there would report the absence of a check as its result, which is
  the pattern this tool exists to name elsewhere. In that state the ring
  names the command that would measure it.
- cli: threads the existing execute-skip signal into the ring.
- report/HTML/JSON: both rings render; the inventory carries both counts.

Grade weighting is unchanged — both rings are advisory and neither enters
`reportDeductions`. Audited on this repo before and after: A (100/100) both
times; only the diagnostic rings move (Tested 67 → Tested 58 + Evaluated 22).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u

* docs(site): the audit grades six categories, not five

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u

* fix(lint): a clean union must not read as 'firing verified'

Every surface can have a deterministic harness while nothing has ever
measured that a skill fires. The gate is unchanged (still the union), but
the ✓ now says so instead of implying the question was asked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u

* test(site): regenerate the browser-parity fixture for the Evaluated ring

The audit now emits a sixth category, so the Node-generated expected report and
the in-browser one differ until the fixture is regenerated — which is what the
test's own header prescribes for an intentional engine change.

Regenerated with `node site/scripts/gen-parity-expected.mjs`. The diff adds the
Evaluated ring and its two inventory counts and nothing else: grade and overall
are untouched, which is the check that the split did not move the weighting.

* fix(coverage): `Tested` measured a STRING, not a test

DEFECT. A surface counted as covered when its path or namespace appeared as a
SUBSTRING anywhere in any discovered test file (`content-reference`, in
src/test-coverage.ts). Two halves, and the second was worse.

REPRODUCTION (a real repo: 37 skills, 14 hooks). Appending ONE line to an
existing harness file:

    // probe: skills/argument-arc

moved the audit's untested count 33 -> 32. A comment. Not a test. Reverting the
line put it back. So `Tested` was trivially gameable and the gaming was
indistinguishable from real work.

The false NEGATIVE: that repo's `.claude/pipeline/skills.harness.mjs` genuinely
asserts over all 21 of its pipeline skills, but builds each path at runtime
(`join(root, ".claude", "skills", name, "SKILL.md")`), so it contains ZERO
literal `skills/<name>` strings (grep: 0 matches) and covered NOTHING.
Generality was PENALISED: 21 trivial files each naming their own subject would
have scored better than the one harness that actually tests all 21.

This is the exact substitution vigiles names in other people's repos — presence
of a surface taken for presence of the property, the way `noExplicitAny` in a
config is taken for the rule being enforced. The tool was committing it in its
own scoring.

FIX, three parts (new src/coverage-evidence.ts — pure and browser-safe, so the
disk detector and its in-browser twin cannot drift on what decides a grade):

1. DECLARATION. A test file may carry `vigiles:covers <surface> ...`, mirroring
   the existing `vigiles:ignore-test` marker convention on the other side. A
   runtime-path harness can now say what it covers; an accidental mention
   cannot. Declaration beats inference — the same move as declaring
   `allowed-tools` instead of inferring capability.

2. PROVENANCE. Every coverage decision now carries HOW it was reached —
   declared / colocated / mention — and the counts are printed by
   `formatUntestedReport`, by `formatScanReport`, and carried in
   `AuditReport.inventory.coverageEvidence` (additive; schema version
   unchanged). A repo whose coverage rests entirely on substring matches can
   finally see that about itself; "28 covered" and "28 names that happen to
   appear in a file" used to print identically.

3. The bare substring detector is KEPT, deliberately, and said so in the code:
   it is the zero-config path that makes an ordinary `foo.test.ts` naming
   `skills/foo` count without anyone learning a marker, and dropping it would
   flip honestly-covered surfaces to "untested" overnight. What it no longer
   reads is COMMENTS. A path in a comment is prose about a test; a path in
   executable code is the test reaching for the surface. Prose isn't policy,
   including ours. It is also now visibly the weakest tier in the report, and
   when it is the ONLY evidence the report says so in words.

Grade weighting is untouched.

TESTS prove both halves, each watched failing first: with the fix reverted, "a
surface named ONLY in a comment is NOT covered", "a runtime-path harness that
DECLARES its surfaces is counted" and "a declaration OUTRANKS a mention" fail
(3 failed | 20 passed); with it, 23 passed. Also guarded: a mention in CODE
still counts, and a `"https://..."` string literal survives comment-stripping.

Docs (untested-skill/-subagent/-hook) updated — untested-skill.md previously
recommended "add the skill's path in a comment", which was advice to exploit
the defect.

* fix(site): key the browser-test prebundle to dist, so a stale cache can't lie

DEFECT. `site/src/demo/browser-parity.browser.test.ts` guards node:zlib vs pako
divergence by comparing an in-browser AuditReport against one generated in Node.
`site/vitest.config.ts` pre-bundles the engine (`optimizeDeps.include` on
`@engine/scan-files` / `@engine/audit-report` / `pako`) and Vite caches the
result in `site/node_modules/.vite`. That cache SURVIVES `npm run build` — Vite
invalidates on config/lockfile changes, not on the mtimes of an aliased path
outside node_modules. So the browser side can compute against a PREVIOUS engine
while Node computes against the current one.

REPRODUCTION (observed 2026-08-07, twice — once by accident, once deliberately).
Accidentally: an engine change added a report field; the parity test failed
showing the field present on the Node side and absent in the browser. I read
that aloud as a genuine engine divergence and started hunting a pako bug that
does not exist. `rm -rf site/node_modules/.vite` -> 42/42 pass.

Deliberately, to prove the fix: plant an extra `inventory` field in
src/audit-report.ts, `npm run build`, regenerate the expected fixture, run the
browser suite with a warm cache.
  - ORIGINAL config: `1 failed | 41 passed`, diff showing `"probeStaleCache": 1`
    on one side only — a field-level diff that reads exactly like a real
    divergence and says nothing about the cache.
  - THIS config: `42 passed`, cache dir rotated `.vite-engine-8baf73f9e373` ->
    `.vite-engine-a52a27e0137e`.
Reverting the plant returns the fixture to its committed bytes.

FIX. Key the prebundle cache DIRECTORY on a fingerprint of `dist/`
(`site/vite.engine-stamp.ts`; `cacheDir: node_modules/.vite-engine-<stamp>`),
sweeping the directories keyed to other fingerprints so they don't pile up.

WHY a key and not the alternatives. Comparing the cache's mtime against `dist/`
and forcing a re-bundle, or clearing `.vite` in the test script, both work and
both leave the bad state REACHABLE — a comparison someone can get wrong, or a
script step bypassed by running `vitest` directly. Keying the directory removes
the state instead: a different `dist/` is a different path, so "prebundle built
from another engine" cannot be reached, and an unchanged `dist/` still gets a
warm cache. By construction rather than by check — the argument this project
makes everywhere else about removing a capability rather than policing its use.

The test's header instruction ("pako and node:zlib disagree — do NOT paper over
it") was correct for one of the two states and actively misleading for the
other; it now says which state is no longer reachable and why, and the
assertion carries the same note in its failure message.

* fix(site): make the parity-fixture generator emit prettier-clean JSON

DEFECT. `site/scripts/gen-parity-expected.mjs` wrote the fixture with
`JSON.stringify(x, null, 2)`, which is NOT what prettier prints for JSON — it
keeps short arrays exploded across three lines where prettier collapses them.
CI gates on `prettier --check .`, so every regeneration of the fixture committed
a file that reddened CI later and far from the cause. It just did: branch head
dec140a failed the `test` job on exactly this file after the Evaluated-ring
change required a regeneration.

REPRODUCTION: `node site/scripts/gen-parity-expected.mjs` then
`npx prettier --check site/src/demo/__fixtures__/sample-repo.expected.json` →
exit 1, six array literals reflowed.

FIX: the generator formats through prettier before writing. Chosen over adding
the file to `.prettierignore` because the fixture is committed and reviewed, so
its diffs should look like every other file's — and because ignoring it would
silence the alarm while leaving the state that trips it. Same shape as the
stale-prebundle defect this fixture's own test hit: an artifact whose
regeneration leaves a bad state is fixed at the regeneration, not downstream.

Also adds `site/checks/` to `.prettierignore` — generated per-rule pages from
the site prebuild, gitignored by `site/.gitignore`, but not ignored by prettier,
so building the site locally reddened `fmt:check` on files nobody commits. Same
class as the `site/test-results/` entry already there.

`npm run fmt:check` is clean.

* fix(cli): refuse unknown flags; `<verb> --help`; nothing-to-audit ≠ grade F

DEFECT, in three parts. Every verb parsed its own flags with
`args.includes("--x")` and ignored everything else.

1. AN UNKNOWN FLAG WAS SWALLOWED. Measured:

       $ npx vigiles audit --this-flag-does-not-exist
       Detected harness: claude-code
       Harness audit …
       EXIT=0

   A full, confident audit as if the flag had never been typed. Not cosmetic:
   audit's flags govern what LEAVES the machine — `--no-html`, `--no-json`,
   `--out=<dir>`, `--serve`. `--no-htlm` wrote the HTML report while the author
   believed they had suppressed it. For a tool whose thesis is "prose isn't
   policy", accepting an argument it did not understand and continuing is the
   same class it names elsewhere: the argument is PRESENT, therefore the
   property is assumed enforced.

   The hole was on EVERY verb, not just audit — checked and closed uniformly
   (`lint`, `test`, `eval`, `compile`, `eject`, `init`, `generate`).
   `hook-runtime` is deliberately exempt: its argv is emitted into hooks configs
   by the harness, never typed by a human, so validating it would break the
   runtime contract.

2. `vigiles audit --help` RAN AN AUDIT. Help existed only on the bare
   `vigiles --help`. Asking what a flag is called executed the command. Now each
   verb answers `--help` with its own entry plus its complete flag list, and
   `printUsage` renders from the same table, so the banner and the per-verb help
   cannot drift.

3. "FOUND NOTHING" WAS INDISTINGUISHABLE FROM "FOUND SOMETHING BAD". How it
   bit: `npx vigiles audit --json /some/path`. `--json` is real but takes no
   value, so the PATH fell through to the positional scan directory. The audit
   ran over a directory containing no harness and reported "No loadable harness
   surface — nothing to grade yet", `skills: 0`, grade F, EXIT 0 — byte-wise
   indistinguishable from a genuine audit of a repo that scores badly. A
   confident measurement OF THE WRONG OBJECT, in silence. An operator who did
   not already know the repo had 37 skills would take it as fact.

   `score.empty` (no instruction file AND no surface) is now a distinct, loud
   outcome on stderr that says what was scanned, that nothing was measurable,
   and that a flag may have swallowed the path. No grade is printed for it.
   Note the flag check does NOT catch that invocation — `--json ./dir` is
   legitimate — so this branch is what makes it visible, and it covers every
   other way of pointing at the wrong place.

EXIT CODE, chosen against the CLI's existing convention rather than invented: 1
already means "I ran, and the thing you asked about is bad" (lint findings, a
failing harness script); 2 already means "I could not do what you asked" (the
top-level error handler, `eval`'s refusal-to-fan-out, `generate` with an unknown
kind). Both new outcomes are the second. A script that greps for a grade needs
"unhealthy" and "I was pointed at the wrong directory" to differ, and the second
has no grade at all. `--json` still emits the report for an empty target (it
self-describes with `score.empty: true`), so the machine contract is kept.

TESTS, each watched failing first. With both guards reverted, 8 of
src/scan-cli.test.ts fail (unknown flag on audit, the typo suggestion, the
per-verb sweep, `audit --help`, every-verb `--help`, the empty-target outcome,
its `--json` form, and the codex-config case below); with them, 48 pass. Plus 7
pure tests in src/cli-flag-check.test.ts, including the over-rejection guard
that asserts EVERY flag in the registry is accepted by its verb, and a dogfood
test that drives the real `vigiles --help` and fails if it advertises a flag the
registry doesn't know.

TWO EXISTING TESTS CHANGED, both because the behaviour they pinned WAS the
defect:
  - cli.test.ts "old --pillars / --verify flags no longer scope a pillar"
    asserted that a REMOVED flag is silently ignored and `init` proceeds with
    defaults. It now asserts the removed flag is rejected — a cleaner break, and
    the exact "silently do something other than what was asked" this fixes.
  - scan-cli.test.ts "honors the .vigilesrc.json `harness` key" audits a
    CLAUDE.md-only fixture AS CODEX, which finds no AGENTS.md and no codex
    surface. That is now exit 2 rather than a silent grade F at exit 0; the
    harness-resolution assertion the test exists for is unchanged.

Both `--help`/unknown-flag sweeps run from a THROWAWAY cwd: while proving the
fix by reverting it, `init --definitely-not-a-flag` ran a real `vigiles init`
and wrote 21 specs, a workflow and a config into the repo. The test now cannot
do that to whatever directory it runs in.

docs/cli.md documents the rejection, the `=`-only value form, and the exit-code
table.

* fix(report-view): mirror the split inventory fields, and let CI see the project

The Evaluated ring added `untestedHarness` and `unevaluated` to the engine's
inventory and not to the view's mirror, so `sample.ts` set two fields the type
did not have. `tsc -p site/tsconfig.json` failed the whole time.

It went unnoticed because CI runs three typecheck steps — root, `test/types`,
and generated types — and none of them covers site/ or packages/report-view.
Three green checks, none over the project that broke. That is the same shape as
every defect in this PR: a check that exists, passes, and does not cover the
thing that failed.

So both halves. The fields are declared optional, because a report produced
before the split carries neither and the view must still render an older
artifact. And CI now typechecks that project, so the next field added on one
side of the mirror fails where someone will see it instead of waiting for
whoever next runs tsc by hand.

Found by the agent that fixed #10-#12 while reporting what it had NOT fixed.

* feat(audit): say so when a repo's vigiles skills are unreachable

vigiles ships six user-facing skills, and `test-harness` alone answers the
question the testing docs are otherwise organized around ("which tier do I
want?"). They are wired by the GLOBAL plugin install; `npm install vigiles`
ALSO drops them at `node_modules/vigiles/skills/`, which Claude Code never
scans. A repo that took the dependency but never ran the plugin install has
all six present on disk, unselectable, and silent — observed for a full day in
a consumer repo whose author was writing harness tests by hand.

`vigiles audit` now checks reachability and warns, following the
`dialect-drift` shape: pure parsers + a best-effort local read that never
throws + a formatter that returns null when there is nothing to say. No new
verb, no new flag. Advisory only, never scored: reachability is a property of
the machine, not the repo, and a score that differed between a laptop and CI
for identical source would be a lie.

The subtle part, and the reason this was misdiagnosed as an npm packaging bug:
the authoritative record of `claude plugin install` is the global
~/.claude/plugins/installed_plugins.json, NOT the repo's .claude/settings.json
(which carries PROJECT-level enabledPlugins, where a correctly-installed
user-scope plugin does not appear). Judged from settings.json alone, a working
install reads as broken. Verified live: with this box's real HOME the check is
silent because vigiles@vigiles genuinely IS installed; with a fresh HOME the
same repo warns.

Also pins the trap that started this. A hand-rolled execFileSync runner returns
stdout ALONE on success, so advisory output — including vigiles's own compiled
hook notice() — vanishes and a healthy react hook reports as dead (hit three
times in one repo). runHook already carries both streams and is the right tier
for a plain helper script; test-harness now says so, including the honest limit
that filesWritten is recorded only under confinement, so assertNoWrite passes
vacuously on an unconfined run. Three tests hold that contract.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(audit): a project plugin declaration is not an install

Investigating whether `init` should wire the plugin per-repo turned up the
opposite of the premise, and a bug in the check shipped in a138826.

Claude Code's team-marketplace docs, as of CC v2.1.195: "A plugin that only the
project's .claude/settings.json enables, and that comes from an external source
such as a GitHub repository or npm package, doesn't load until the team member
installs it." vigiles ships from a GitHub marketplace, so that is our case.
Committing extraKnownMarketplaces + enabledPlugins makes Claude Code PROMPT a
collaborator to install; it does not install, and a fresh clone or CI job still
has no plugin.

Confirmed empirically on one machine: a repo whose committed settings.json
declares an external plugin project-level had NO marketplace entry, NO cache
dir and NO install record for it in ~/.claude/plugins/, while a plugin
installed the normal way on that same machine had all three. Network was not
the difference — the second install succeeded there, commit SHA recorded.

The bug: the check treated a project `enabledPlugins` entry as proof of
reachability, so it would fall silent for a repo that declares the plugin and
has never installed it — precisely the state it exists to surface, and
precisely the state observed. Project declaration is now a THIRD state,
`declaredNotInstalled`, which still warns but with the correct fix line: the
install is per machine, and the repo cannot perform it on the collaborator's
behalf.

Four tests were watched failing against the previous implementation before the
fix. Advisory-and-unscored stands, and for a stronger reason than before: this
is machine state that no repo-committed file can determine, by deliberate
security design — plugins execute arbitrary code with user privileges.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(testing)!: split runScript out as the primitive under runHook

`runHook` already WAS a general program runner — arbitrary shell command, stdin
fed, exit code + both streams + writes + egress back — with a thin hook layer on
top. The specialized name owned the general machinery, so someone testing an
ordinary helper script had no name to look for. One did: he hand-rolled an
execFileSync runner and hit the same bug three times, because execFileSync
returns stdout ALONE on success while advisory output (including vigiles's own
compiled-hook notice()) goes to stderr, so healthy react hooks reported as dead.

The layering is now the right way up:

    runHook  =  runScript  +  event-to-stdin  +  exit-code-to-decision

src/run-script.ts owns spawning, confinement, egress and write recording; none
of that is about hooks. src/run-hook.ts is 250 lines of hook protocol.

These are two questions, not two ways to ask one: a HOOK has a decision, a
SCRIPT has effects. So ScriptRunResult carries no `decision` field rather than
an always-undefined one, which would teach the reader the field means nothing.
A test asserts its absence, so re-adding it fails.

Also fixes a vacuous assertion, which is the same defect class this repo exists
to catch in other people's harnesses. `filesWritten` is recorded by diffing the
work dir, which only a confined run does — so an unconfined run reported `[]`
and `assertNoWrite` returned GREEN having inspected nothing. "Nobody looked" and
"nothing happened" are now structurally distinct: `undefined` vs `[]`.
assertNoWrite/assertWroteOnly throw on the former and name the fix
(`sandbox: "auto"`). Documenting this was too weak; the code distinguishes it
now instead of the reader having to remember.

BREAKING CHANGE: `HookRunResult.filesWritten` is optional
(`readonly string[] | undefined`) — `undefined` when writes were never recorded,
rather than a misleading `[]`. Code reading it directly needs `?? []`; code using
assertNoWrite/assertWroteOnly gets a real assertion instead of a vacuous pass.
Everything else is additive or structurally identical: HookRunResult now extends
ScriptRunResult with the same members, and RunHookOptions is
Omit<RunScriptOptions, "stdin">. runHook's signature and behaviour are unchanged.

Watched failing first: 3 vacuity tests (assert.throws saw no throw — the vacuous
pass itself), then 3 deliberate breaks — dropping stderr fails 3 tests, re-adding
a decision field fails 1, collapsing unrecorded writes to [] fails 2.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(init): declare the plugin in the repo, turning silent absence into a prompt

`vigiles init` now merges a two-key declaration into the repo's
.claude/settings.json: an `extraKnownMarketplaces.vigiles` source and
`enabledPlugins["vigiles@vigiles"]`.

WHAT THIS BUYS, precisely: a collaborator who clones and never runs `init`
currently gets SILENCE — the npm package is in node_modules, its six skills are
unreachable, and nothing says so. That silence cost a full day. With the
declaration, Claude Code tells them the project wants this plugin and prints the
install command.

WHAT IT DOES NOT BUY: the plugin. An external-source plugin declared
project-level does not load until each person installs it on their own machine
(CC docs, v2.1.195+), and the boundary is deliberate — plugins execute arbitrary
code with the user's privileges, so a repo may not install one on your behalf.
A fresh clone and a CI job still have no plugin. The README, agent-setup docs
and the CLI output all say this in those terms; none of them claim it wires
anything up.

Nothing is vendored: the entry is a reference. Plugin content stays in the
global cache, one copy shared across repos.

MERGE, never clobber — that file holds the user's hooks, permissions and other
plugins. The merge rules are pure and unit-tested: unrelated keys survive
byte-identical; an existing `vigiles` marketplace entry is left alone (it may
point at a fork); an explicit `false` is respected as a deliberate disable; the
input object is never mutated; a second run is a no-op. Verified end to end
against a synthetic repo whose settings.json carries hooks, permissions, a
$schema and another marketplace — the diff was exactly the two additions.
Invalid JSON is a loud skip, never a rewrite. Gated by the existing
`--no-plugin`; skipped in the vigiles repo itself.

`removeVigilesDeclaration` ships tested alongside it, deliberately not wired to
`eject`: `eject [file]` is the per-file inverse of `compile`, and repo-scoped
plugin wiring is not a property of any one file. Reasoning and the two-key
manual removal are documented.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-08 14:30:39 +05:00
zernie 9de2e6905e feat(site): demo UX redesign — repo autocomplete, per-check pages, share links, softer palette
Repo-name autocomplete (no org needed, live stars); dedicated /checks/<slug> explainer pages linked from every finding (Vite MPA); stronger trigger-rate tease + graded leaderboard chips; shareable ?repo= grade links + selling copy; plain-language lethal-trifecta (ships in the CLI report too); softer material-oceanic palette across the site and the local HTML report.

Deterministic report changes flow through the shared @vigiles/report-view. Includes 14 Codex review fixes (autocomplete request races + stale-result guards, honest empty-fixes/clean state, safety-card gating, GitHub-API disclosure, Claude-Code-only trigger-rate CTA).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 13:07:10 +05:00
zernie 8fa0f236cb feat(site): live in-browser "Grade any repo" audit demo + README-parity content (#100)
* docs: refresh HANDOFF — demo shipped (#99); PR 3 live engine in progress via subagents

* feat(scan): browser-safe audit engine (scanFiles) with CLI-parity gate

Add src/scan-files.ts — scanFiles(files: Record<path,content>) runs the
deterministic audit over an in-memory file map with NO node:fs/child_process,
so the in-browser grade is byte-identical to the CLI's `audit --json`.
Reconstructs the LoadedPlugin shape from the map, REUSES scan.ts's pure
per-surface detectors, and reimplements only the ~6 fs touchpoints as
map-backed pure logic. src/test-coverage-files.ts is the browser-safe twin of
findUntestedSurfaces (globSync → Object.keys filter).

scan.ts: export the pure detectors the engine reuses and thread a
backwards-compatible injectable `exists` (defaults to existsSync) through
resolveScript/scanHooks + `existsSync?` on SkillScanContext — no disk-path
behaviour change.

src/scan-files.test.ts is the correctness firewall: for every SHA-pinned
vendored plugin, scanFiles(map) must produce an AuditReport byte-identical to
scanPlugin(dir) (both the buildAuditReport output and the raw ScanReport),
normalizing only the checkout path (BROWSER_ROOT). Foundation for the
in-browser "grade any repo" demo.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(scan): import editDistance from edit-distance.js not linters.js

Drops the whole linters.ts module (glob + node:module + child_process
import-time weight) from scan.ts's transitive graph — it was pulled in only
for editDistance, already extracted to core/edit-distance.js. Shrinks the
browser-bundle graph for the in-browser audit engine; no behaviour change
(same editDistance).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(core): extract NCD to node-free core/ncd.ts

Move ncd() + its gzip helper out of proofs.ts (which imports node:crypto via
hash.js) into a node-free leaf so the description-overlap detector reaches it
without dragging crypto into the in-browser audit bundle. proofs.ts re-exports
ncd for its existing consumers; description-overlap.ts imports it from the leaf.
Swaps Buffer.from(s,'utf-8') → TextEncoder so the byte input is identical in
Node and the browser and the node-only Buffer global is gone (the browser build
aliases node:zlib → pako). No behaviour change — parity + proofs + overlap
tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(research): record node-free engine decision (module-split, not stubs)

The Stage-2 build plan in report-view-and-browser-demo.md prescribed the
bundler-stub approach that was rejected. Rewrite it to the decision of record:
module-splitting over stubs (silent-divergence risk) and over dynamic-imports
(dead code needs no lazy selection; sync engine), the dist/ require-graph
tracing method (scan.js is the single bridge), the splits (editDistance/ncd/
scan-core + required-IO + mcp-message leaf + posix helper), the one real
zlib->pako alias, DI as the runtime adapter, and the browser-mode parity test
that closes the pako-vs-node gap. Plus the site<->engine integration (site
consumes built dist CJS) and this-branch status.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(scan): node-free the in-browser audit engine (scan-core split)

Extract the pure audit detectors into node-free leaf modules so the browser
engine (scan-files.ts / test-coverage-files.ts) reaches NONE of node:fs,
node:crypto, node:child_process, node:path, @ast-grep/napi, or glob. The
dist/ require-graph for the browser engine is now: 29 files, builtins = only
node:zlib (→ pako in the Vite build), pkgs = @iarna/toml, js-yaml, mvdan-sh
(all pure-JS, browser-bundleable).

Module-splitting (the core ⊄ adapter hexagonal pattern), NOT bundler stubs or
dynamic imports — the node deps are transitive-dead on the browser path, so the
clean kill is removing them from the static graph, with zero silent-divergence
risk. Tracing the compiled dist/ require-graph showed scan.js is the single
runtime bridge, so one scan-core extraction removes the whole node subtree.

- src/scan-core.ts: the moved pure detectors (preserves the \x00 matcher
  delimiter byte-for-byte). scan.ts re-exports via  so every existing
  consumer is unchanged.
- src/posix-path.ts: node-free POSIX path ops (verbatim path.posix port) — drops
  node:path from the engine entirely.
- src/core/mcp-contract-message.ts, src/core/assert-never.ts,
  src/adapters/claude-code/agent-tools.ts: pure leaves extracted from mcp.ts /
  hash.ts / agent-runtime.ts (each re-exports the leaf, so no public surface
  changes — api gate confirms no drift).
- hook-block-ineffective.ts / plugin-dir-layout.ts / skill-resources.ts: the
  node:fs default made required-injected IO (only callers are the two engines;
  the browser injects map-backed impls).

Firewall held: src/scan-files.test.ts (byte-identical CLI parity on 4 vendored
plugins) unchanged + green; full unit suite green at 100% coverage; api no drift.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* style(research): prettier-format the browser-demo doc

* docs: refresh HANDOFF — node-free engine pushed; demo wiring + site content additions next

* refactor(scan): node-free buildAuditReport via score-core leaf

buildAuditReport re-tainted the browser graph (@ast-grep/napi native binary +
node:fs/crypto/child_process) through audit-score/optimize/audit-verdict →
leaderboard.ts → scan.ts. leaderboard.ts eagerly imports scanPlugin (used only
in the CLI-only rankPlugins), so importing its pure scoring functions dragged
the whole native chain in.

Extract the pure scoring core into node-free leaf src/score-core.ts (PluginScore,
W_* weights, Deduction, gradeFor, reportDeductions, isEmptyMachine,
computeIntegrityScore, scoreReport — same pattern as core/ncd.ts / scan-core.ts).
leaderboard.ts re-exports them and keeps its node-only scanPlugin/rankPlugins;
audit-score/optimize/audit-verdict/audit-report import from score-core.

Now BOTH browser entry points are node-free: the dist require-graph union of
scan-files.js + audit-report.js has builtins = only node:zlib (→ pako in the
site build), pkgs = @iarna/toml/js-yaml/mvdan-sh. audit-report.js went 41 files
(+ ast-grep) → 6 files, zero node deps. Internal-only (no api drift);
coverage-neutral; parity firewall + full unit suite (100% cov) green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(site): live in-browser 'Grade any repo' audit demo

Type any public GitHub repo and get a real, live-computed audit report rendered
by the SAME @vigiles/report-view components the CLI uses — the browser runs the
literally-same compiled engine (dist/ CJS) with one node:zlib→pako swap, so the
grade is byte-identical to the CLI.

- Vite wiring: @engine/* aliases → built dist/scan-files.js + dist/audit-report.js
  (node-free), node:zlib → src/lib/zlib-shim.ts (pako.gzip), commonjs transform of
  the dist graph.
- fetchRepo.ts: GitHub Trees API (recursive, once) + raw.githubusercontent.com for
  contents (not rate-limited), harness paths only, size-capped, honest progress
  events; typed handling of 404 (private) / 403 rate-limit / no-harness.
- runAudit.ts: scanFiles → buildAuditReport; DemoAudit.tsx rewritten to the Fable
  live-typing UX — input → honest promise-driven 3-step loading → discriminated
  report|empty|notfound|ratelimit|error states, ?repo= deep-link + auto-run, baked
  chips kept (instant), edge states carry their own in-frame CTA.
- Tests (3 layers): browser-parity (real Chromium + pako, asserts runAudit ===
  the Node-generated AuditReport — closes the pako-vs-node-zlib gap), interaction
  (fetch mocked, real engine), one Playwright e2e (built bundle, network mocked).
  Site build has zero node-builtin/native leaks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(site): add how-it-works, adoption, and FAQ sections

Close the gap between the landing page and the README — the site had a strong
top (hook + problem + demo) but was missing the middle (what vigiles does across
its verbs + how you adopt it) and the objection-handling.

- VerbMap ('One tool. Four questions.'): the audit/lint/test/eval verb map as a
  type+divider table (stacks on mobile), the 'one engine, two doors' line, and a
  compact 'your rules → enforced' card (the prose→enforcement payoff).
- Adoption ('One command, then your agent does the rest.'): the three shipped
  skills (test-harness/strengthen/edit-spec), the in-loop hooks, and a copyable
  paste-into-Claude-Code/Codex setup prompt (wraps on mobile).
- FAQ: the framework? / markdown-linter? / TypeScript? / stability / non-JS
  objections, as a type-only list (no boxes).
- Debunk: fold in the promptfoo/DeepEval 'bill per token every run' cost contrast.

Order: Hero → Wedge → VerbMap → Debunk → DemoAudit → Adoption → FAQ → CTA. Each
held to the landing-site subtraction bar (type + space over stacked boxes, no
cross-section command duplication); verified desktop + 390px mobile.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(site): apply Fable design cuts to the new sections

Fresh design pass flagged tail-end restatement (not crowding). Cuts:
- FAQ: drop 'framework?' + 'markdown linter?' — they restated VerbMap's 'two
  doors' and Wedge's 'valid != true'; keep the three net-new objections
  (TypeScript / stability / non-JS). Full FAQ link carries the rest.
- Demo: remove the 'Now grade yours' tail command block — it was the 4th
  npx-vigiles-audit surface (hero + sticky nav + final CTA already carry it).
- VerbMap: make the right column self-explanatory ('no model · read-only ·
  anytime' … 'needs a model · your subscription · on demand').
- Adoption: shrink the hooks paragraph to one line.
- Demo chips: wrap + center on every width so a chip never clips at the mobile
  edge (four chips → two tidy rows).
- .prettierignore: ignore site Playwright/vitest-browser run artifacts.

fmt clean; site build green; browser tests 9/9. Verified desktop + 390px.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: refresh HANDOFF — node-free engine + live demo + site content shipped

* fix(demo,scan): address PR review — hook-script fetch + dir-aware browser exists

Two browser-vs-CLI divergences the Codex review flagged (both weaken the
byte-identical-to-the-CLI guarantee):

- fetchRepo (P1): a hook command can reference a script OUTSIDE the harness dirs,
  e.g. ${CLAUDE_PLUGIN_ROOT}/scripts/guard.sh. The harness-path filter dropped it,
  so the browser scan reported the hook as missing and applied the graded penalty
  even though the tree showed the file exists. Add a bounded second fetch pass:
  collectReferencedPaths() scans fetched harness files for plugin-root/project-dir
  token refs and fetches any that actually exist in the tree (size-capped), matching
  the CLI's whole-repo read.

- scan-files (P2): mapExists mirrored fs.existsSync but returned false for a
  directory (file keys only), so pluginDirLayoutIssues' exists(dir) && isDirectory(dir)
  silently missed the layout finding the disk scan reports. Make mapExists faithful
  to fs.existsSync — true for a file OR a directory.

Parity firewall unchanged + green; site build 0 leaks; browser tests 9/9.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* ci(pages): build root dist/ before the site build

The Pages deploy built site/ right after npm ci with no root build, but the
landing's live audit demo now imports the compiled engine (dist/scan-files.js +
dist/audit-report.js) via the site's @engine/* Vite aliases, and dist/ is a
gitignored build artifact. A clean Pages deploy would fail to resolve those
imports (not caught by PR CI, which builds dist/ in its own jobs — only on
merge-to-main deploy). Add a root 'npm run build' step first. Validated by
running the pages sequence against a clean dist/.

* fix(demo): scope the live demo to Claude Code (drop unused Codex inputs)

fetchRepo pulled AGENTS.md / .codex but runAudit always scans with the Claude
Code layout, so a Codex-only repo was graded as CC — its Codex surfaces ignored,
yielding a misleading near-empty report. The engine's Codex audit is full-parity,
but the demo doesn't wire the Codex layout/dialect yet (a follow-up).

Make the demo honestly Claude-Code-only: drop AGENTS.md/.codex from the harness
fetch so a Codex-only repo lands in the no-harness state that points at the CLI,
not a wrong-layout report. Empty-state copy + the two edge-state test assertions
updated to match. (Do not re-add Codex inputs without wiring the Codex adapter
into runAudit.)

Browser tests 9/9, e2e 2/2, fmt clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(demo): harden repo/harness detection (PR review round 2)

Three more browser-fetch correctness issues from the Codex review:

- isHarnessPath matched a harness word as ANY path segment, so an ordinary repo
  with src/hooks/useThing.ts (a plain React app) was fetched as a harness and
  graded an empty machine instead of showing no-harness. Match harness dirs only
  as the TOP-LEVEL segment (or a .claude/.claude-plugin root).

- collectReferencedPaths only caught ${CLAUDE_*} token refs, so a hook using a
  RELATIVE script path outside the harness dirs (scripts/guard.sh, ./bin/x.sh —
  a form scanHooks resolves against the plugin root) was still dropped → false
  missing-hook penalty. Also collect relative dir-qualified script paths.

- The recursive Git tree is truncated for very large repos (>100k entries), so
  the needed harness files can fall outside the returned slice → a misreported
  no-harness or partial grade. Read tree.truncated and bail honestly to a new
  'too-large — grade it locally' in-frame state.

Adds fetchRepo.browser.test.ts guarding isHarnessPath (nested src/hooks excluded)
and collectReferencedPaths (token + relative script refs, prose ignored). Site
build + browser tests 14/14 + e2e 2/2, fmt clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(demo): require a real harness marker + fetch manifest-declared hooks

Two more browser-fetch/CLI divergences from the review:

- A plain repo with a top-level hooks/ (git hooks — husky, pre-commit) or other
  bare surface dir passed the harness filter, so fetchRepo returned ok and the
  demo graded an empty machine instead of showing no-harness. Add a GATE: the
  fetched set must contain a definitive marker (isHarnessMarker) — CLAUDE.md,
  .mcp.json, anything under .claude/.claude-plugin/, hooks/hooks.json, or a real
  top-level surface FILE (skills/<x>/SKILL.md, agents/<x>.md, commands/<x>.md).
  A bare hooks/ of scripts with no declaration is not a harness (matches the
  loader, which only loads declared hooks).

- A plugin manifest can point its hooks field at a file outside the harness dirs
  ("hooks": "config/hooks.json"); the scan's readHooksJsonFile then found nothing
  and dropped every hook. The 2nd fetch pass now also pulls the manifest-declared
  hook-config path.

Tests cover isHarnessMarker (git-hooks/ordinary files rejected, real markers
accepted). Build + browser 16/16 + e2e 2/2, fmt clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: refresh HANDOFF — PR #100 open + auto-merging with review fixes

* docs: HANDOFF — mark PR #100 as the vehicle (consistency)

* fix(demo): root SKILL.md harness + chained hook-config script refs

Two more browser-fetch/CLI divergences from the review:

- A single-skill repo (a root SKILL.md — loadPluginFromFiles' 'single-skill'
  case) wasn't in the root allowlist, so its only surface was never fetched and
  it fell through to no-harness. Add SKILL.md to the fetched root files AND the
  marker gate, so a public single-skill repo is graded.

- The manifest-hooks fetch was single-shot: a manifest-declared config/hooks.json
  was fetched AFTER collectReferencedPaths ran, so that config's OWN out-of-harness
  script refs (manifest -> config/hooks.json -> scripts/guard.sh) were never
  scanned. Make the second pass a bounded fixpoint (re-scan newly-fetched files,
  <=3 rounds, capped at MAX_FILES) so realistic reference chains resolve.

Test covers the root-SKILL.md marker. Build + browser 16/16 + e2e green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(site): plainer report copy, marketplace state, progress bar

Live UX-review fixes on the "grade any repo" demo:

- Report copy: replace the jargon "maximal trifecta blast radius" with plain
  "no limits on what it can read, fetch, or run" for the inherits-all safety
  finding (engine copy in audit-score.ts — ships in the CLI report too). Baked
  demo fixtures + the parity expected fixture regenerated to match.
- Marketplace repos: a repo whose only .claude-plugin surface is a
  marketplace.json is a COLLECTION, not one gradeable harness — the browser
  demo would have rendered a bogus zero-surface F (the CLI expands members via
  inspectMarketplace, a path the demo doesn't run). Route it to an honest
  "this is a plugin marketplace — grade members with the CLI" state. A repo that
  also ships plugin.json is a self-publishing single plugin and still grades.
- Loading: add a real progress bar under the "reading harness files" step.
- A-grade note: drop the vague "the row below is your next step"; state the
  next step self-containedly (a browser can't ask a model, your CLI does).
- Report footer: gate "Generated by vigiles …" behind a showFooter prop
  (default on for the standalone HTML report; off in the in-page demo, where the
  page already brands).

Tests: audit-score + scanFiles parity green; site browser tests 17/17 incl. a
new marketplace edge-state case.

* fix(site): name the lethal trifecta, stop caching transient errors, drop analytics slug

More live UX-review fixes on the "grade any repo" demo:

- Report copy (round 2): the inherits-all safety finding now NAMES the concept
  and explains it — "the lethal trifecta (reads data, reaches the web, runs
  commands), so a prompt injection could exfiltrate secrets" — instead of the
  vague "no limits …". Engine copy (audit-score.ts), so the CLI report gets it
  too; fixtures regenerated.
- Transient failures stay retryable: only STABLE outcomes are cached now
  (isCacheable) — a dropped-connection error / rate-limit is no longer cached,
  so the frame's own "try again" actually re-fetches instead of replaying the
  cached error until a reload.
- Analytics never sees the typed slug: track() now sends the outcome KIND only,
  never owner/repo — the demo promises nothing leaves the browser but GitHub
  requests, and a typed slug would leak a private repo name (→404) to a future
  Plausible. (track() is a no-op shim today; this is future-proofing.)
- Root SKILL.md name: a nameless single-skill repo is labeled after the repo
  (threaded into scanFiles → the CLI's audited-dir-basename behavior) instead of
  the synthetic "__vigiles_repo__". New parity-firewall test guards it.

Tests: scanFiles parity + single-skill naming green; site browser 17/17; engine
audit-score green; tsc + fmt + lint clean.

* feat(site): persistent grade-cache (idb-keyval) with honest age + re-grade

A shared ?repo= deep-link, a reload, or a revisit now renders a previously
computed grade INSTANTLY and with ZERO GitHub requests (dodging the anonymous
60-req/hr limit) — without ever showing a silently-stale grade.

Design (Fable pass): idb-keyval was the ADOPT over the reactive useLocalStorage
hooks (@react-hookz/web / usehooks-ts / react-use) — the cache is accessed
imperatively under a dynamic key (the slug) and needs TTL + versioned
invalidation + LRU, none of which a fixed-key storage hook provides; idb-keyval's
structured-clone + per-entry keys fit exactly (no JSON-parse bug class, no
whole-object multi-tab race).

- site/src/demo/gradeCache.ts — TTL 24h, version-namespaced key
  (schemaVersion.engineVersion), 30-entry LRU, parse-don't-validate boundary
  that rejects corrupt/wrong-schema/future/expired entries; never throws
  (storage-disabled browsers run cache-off).
- Version namespace = AUDIT_SCHEMA_VERSION + the build-time git SHA (__ENGINE_V__
  via a Vite define; the root package.json version is a release-time placeholder
  in-repo, so it can't serve here). A deploy that changes the engine invalidates
  every prior grade — over-invalidation is safe; a stale grade is the only hazard.
- Persist only report/empty — never notfound (a private repo name on disk + a
  trap after it goes public) or a flippable/transient state.
- DemoAudit: L1 in-memory Map (all stable outcomes) layered over L2 gradeCache
  (report/empty); run() falls through L1 → L2 → network, runId-guarded; a
  "graded N ago · ↻ re-grade" header badge makes every cache hit honest and
  one-click refreshable; sweepGrades() on mount drops old-namespace/expired keys.

Tests (real Chromium + real IndexedDB): gradeCache round-trip / TTL expiry /
corrupt-reject / wrong-namespace sweep / LRU cap; DemoAudit deep-link zero-fetch
+ forced re-grade bypasses cache. Site browser suite 24/24; tsc + fmt clean.

* fix(demo): fetch root single-skill bundled resources before grading

A root single-skill repo (`SKILL.md` at the top level) that references a bundled
resource — `references/api.md`, `assets/example.json`, a `[guide](assets/x.md)`
link — had those files DROPPED by the harness-path filter (only script
extensions were collected in the second pass, and non-script resources at the
repo root aren't under a harness dir). `scanFiles` then ran `skillResourceIssues`
against a partial tree and reported a false missing-resource penalty, where
`vigiles audit` on disk reads the whole skill dir.

Extend `collectReferencedPaths` to also collect bundle-dir paths
(`scripts`/`references`/`assets` with any extension) and markdown-link relative
targets, so the tree-present ones are fetched. (Nested-skill resources under
`skills/**` were already fetched via the harness dir.) Over-fetch is harmless —
only tree-present, size/count-capped paths are pulled. Newly relevant since root
SKILL.md repos now grade. Browser tests +2 (bundled resources caught, external
links ignored).

* docs: refresh HANDOFF — live UX-review round (copy, marketplace, grade-cache)

* ci: run the site browser suite (parity + interaction + grade-cache)

Add a `site` job to ci.yml — build the root dist/, install Chromium, run
`npm run test:browser` (the browser-mode vitest suite: real Chromium + real pako
parity, the DemoAudit interaction states, and the idb-keyval grade-cache over
real IndexedDB). Closes the gap where a demo/engine-parity regression was only
caught locally. Playwright `test:e2e` stays manual/browser-gated.

* fix(demo): never grade partial data (fetch failure / file-cap truncation)

Two ways the demo could publish a grade computed over an INCOMPLETE harness —
both now bail honestly instead of grading partial data:

- A raw.githubusercontent.com fetch that fails or returns non-OK (a transient
  500 / CORS drop on a harness surface like SKILL.md or .claude/settings.json)
  used to be skipped silently, leaving the file absent and the grade partial.
  Now first-pass surface fetches are failure-counted; any failure → the retryable
  `error` state (not cached, so "try again" re-fetches). Second-pass referenced
  files stay best-effort (a missing one is a soft advisory, not a hard fail).
- A repo with more harness surfaces than the browser reads fully (> MAX_FILES)
  used to silently slice off the overflow — dropping skills/agents/hooks, or even
  the definitive marker (→ false no-harness). Now the marker gate runs on the
  FULL harness set, and an over-cap harness → the `too-large` state (bail to CLI),
  same as a truncated tree.

Tests: 3 fetchRepo integration cases over a stubbed GitHub (too-large on 301
surfaces, error on a failed surface fetch, ok when all fetch). Site browser 29/29.

* fix(demo): error on required second-pass fetch failures; fetch coverage tests

Two more browser-vs-CLI parity gaps in the fetch layer:

- REQUIRED second-pass files that fail to fetch now error (were silently OK). A
  manifest-declared hook config (config/hooks.json), a referenced hook script, or
  a skill bundled resource all feed GRADED findings — a transient 500/CORS drop on
  one left the grade partial. The first-pass failure rule now applies to the
  second pass too (a `countFailure` flag on fetchBlob/drain distinguishes required
  from advisory fetches).

- COVERAGE test files outside the harness dirs are now fetched (best-effort). The
  untested-surface check reads tests that live in a top-level tests/ or a colocated
  *.eval.mjs — the harness filter dropped the non-colocated ones, so the browser
  showed a false `untested` advisory the CLI's whole-repo read wouldn't. New
  isTestPath collects them, budget-bounded; a failure does NOT error (untested is
  excluded from the overall grade — cosmetic, not a wrong grade).

Tests +4: required second-pass failure → error, a coverage test outside harness
dirs is fetched, a coverage-test failure stays ok, isTestPath matching. Site
browser 33/33; tsc + fmt clean.

* fix(demo): second-pass cap → too-large; build engine before e2e

- Complete the "never grade partial data" invariant: the first-pass harness cap
  already bailed to too-large, but the SECOND pass (required referenced files —
  hook configs, scripts, bundled resources, all graded) still silently sliced to
  the remaining budget. Now if the eligible referenced set exceeds the budget it
  returns too-large too, instead of grading a partial repo. Test: 300 surfaces +
  a referenced resource with no budget left → too-large.
- e2e: the site's test:e2e webServer built only the site, but the @engine/* Vite
  aliases resolve to the gitignored root ../dist/ — so a clean checkout failed
  before preview. Prepend the root engine build (npm --prefix .. run build).

Site browser 34/34; tsc + fmt clean.

* docs: document the browser demo's bounded-fetch limitation

Codex keeps flagging "file X outside the harness dirs isn't fetched" on
fetchRepo — all one root cause: the demo does a BOUNDED, SELECTIVE fetch (to
respect GitHub's 60-req/hr anonymous limit), not the CLI's whole-repo read. So
document it as the intentional, safe-by-invariant design it is, rather than chase
an infinite tail:

- fetchRepo.ts MODULE HEADER (where Codex + contributors read): the
  NEVER-GRADE-PARTIAL-DATA invariant (too-large/error bail-outs; advisory data
  best-effort), what IS fetched to match the CLI, the documented non-goals, and
  the "unless it's a wrong grade not already covered, it's the accepted cost"
  framing.
- research/browser-demo-fetch-limits.md: the canonical record + the edge-case
  ledger (every divergence Codex found on PR #100 → closed or a documented
  non-goal, incl. extensionless relative scripts). Indexed in the research spec
  (dogfood green) so the next session finds it.
- HANDOFF pointer so the next session doesn't re-litigate the tail.

No behavior change — documentation only.

* fix(demo): collect bundled-resource refs only from SKILL.md (regression)

The bundled-resource collector (BUNDLE_RES / MD_LINK_REL) ran over ALL fetched
files, but the engine only grades those refs from a SKILL.md body
(skillResourceIssues). So a normal CLAUDE.md with ordinary doc links like
`[guide](docs/x.md)` had those docs treated as REQUIRED second-pass files —
enough of them → a false `too-large`, or one failed fetch → a false `error`,
refusing to grade a repo the CLI grades fine. Restrict BUNDLE_RES/MD_LINK_REL to
SKILL.md files (hook-script refs stay all-files, since a hook command lives in a
manifest/settings/config). Test: a CLAUDE.md's md-links are dropped, a SKILL.md's
kept, a hook script kept.

Also document (research/browser-demo-fetch-limits.md) the narrow oversized-bundled
-resource edge (a > 256 KiB SKILL.md resource skipped by the size cap → a possible
false-missing) as an accepted, deferred approximation.

Site browser 35/35; tsc + fmt clean.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-22 04:07:14 +05:00