skills/zernie__vigiles
1
0
Fork 0
mirror of https://github.com/zernie/vigiles.git synced 2026-09-14 20:53:57 +08:00
Code Issues Packages Projects Releases Wiki Activity
205 Commits 153 Branches 139 Tags
v27.1.2
Commit Graph

5 Commits

Author SHA1 Message Date
zernie 195c1e14b7 feat: launch blockers, a hook-state test API, and the site's missing proof (#210) 
Ten launch blockers, all sharing one cause: a first reader gets a false claim or an embarrassing failure in their first ten minutes.

Fixes: ruff enabled-state was never read on a real repo; compile was unreachable without node_modules/vigiles (every Python/Rust/Go repo); a hooks-only plugin graded F/0 while its own report listed inlineHooks: 2; skill-resource findings printed a body-relative line number; the README quoted a caveman number its own FINDINGS.md had invalidated; STABILITY.md claimed v21 against 26.0.1; the Action's floating v1 tag was never created; a react's notice() reached nobody.

Features: experimental_hookState, so a stateful compiled hook can be tested by backdating the fact it throttles on; the site now leads with the proof (compiled hooks, the disaster battery, 2/7) and its OG card is rendered rather than hand-drawn, with a generator that fails loud if the page fetches anything off file:.

Also: this repository's first compiled hook, which is what exposed the notice-delivery defect; a floor under the harness CI job, which was green on zero files; and a smoke test of the PUBLISHED package, since every gate until now ran dist/.
 2026-09-08 00:52:20 +05:00
zernie 4f0a01b2f9 CLI clarity, an execution-tier finding, and the Action's boundary documented (#163) 
Five changes from a measured review of the CLI surface and of a consumer's CI.

1. BEHAVIOUR — a positional target that matches nothing now FAILS. Bare discovery
   finding nothing still exits 0: an empty repo is a legitimate place to stand, but a
   path or glob you NAMED that matched nothing is stale, wrong, or never reached.
   Collapsing the two cost a real repo three days of green CI verifying zero files.
   Jest and Vitest fail on no-tests-found and make you opt in with --passWithNoTests;
   pytest exits 5. --min=0 is the opt-out here, so no flag is added.

2. HELP — the four checkers (audit/lint/test/eval) all read as "check my stuff". They
   are now grouped by READ vs RUN, and each line states its own consequence positively.
   The success criterion was checkable and is met: "NOT a CI step — use vigiles lint in
   CI" is deleted, because nothing needs defending. A help line that says what a command
   ISN'T is a description that failed. Per-verb flag detail moves into the per-verb
   help; audit's four wrapped lines naming nine flags were most of the felt crowding.
   Verb count is unchanged at 8 — the smallest of every tool measured (vitest 8 verbs
   and 164 flags, cargo 48, docker 57, git 166), and none of them thinned their surface
   by removing verbs.

3. DOCBLOCK — src/cli.ts listed four of the eight commands, having quietly stopped
   growing. Replaced by a pointer to the single source; a new test asserts every verb
   appears in the RENDERED help exactly once, because a constant is what drifted.

4. AUDIT — new Tested finding: "N surfaces counted as covered by PLACEMENT only — no
   run on record ever exercised one". Colocated evidence is a claim about the
   filesystem and the provenance line already admits it; nobody reads a provenance
   line. Measured in a consumer: 26 colocated and 0 executed, while its CI skipped
   every one of those harnesses and reported success. The corpus-wide threshold is a
   stated limit in the code, not a silence.

5. DOCS + SKILL — the Action reference now explains why there is no test input (the
   Action never runs npm ci; harnesses import the library), and the test-harness skill
   gained a six-line CI pointer: an Action exists, init wires it, inputs are documented
   there. It REFERENCES rather than explains, because copied explanations rot — three
   measured proofs, one of them inside this product.

Gates, in order: vitest, lint, prettier, api:check, build — via the check script,
18 of 18.

Co-authored-by: Claude <noreply@anthropic.com>
 2026-08-19 14:02:16 +05:00
zernie 81be2fd07e feat: capability-diff PR comment + local observe ledger (#55) 
* docs(vault): harness-quality tech camps + telemetry/default-install decision

Two-camps tech map (runtime/observability vs pre-deploy simulation vs
app-frameworks), the field direction, and the live fork: local-first
deterministic measurement over hosted runtime telemetry. Cross-linked
from the vault index and funded-adjacency.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(vault): add pseudocode DX + fundability/exit thesis to tech-direction

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(research): tech direction — four-instrument loop + local observability ledger

The architecture decision of record for the next build phase: typed spec as
declared ground truth; verify/gate/measure/observe as facets of one loop; the
deterministic-vs-behavioral precision principle; the local agent-readable
runs.jsonl flight-recorder ledger; promote-prose; per-surface map. Monetization
kept out (vault only). Indexed + recompiled.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: add Direction section to root CLAUDE.md positioning

Elevate the next-phase direction to the north-star doc (read first every
session): the four-instrument loop (verify/gate/measure/observe) with the typed
spec as declared ground truth, the deterministic-vs-behavioral precision
principle, the local agent-readable runs.jsonl flight recorder, promote-prose,
and the per-surface map. Tech only; monetization stays in the vault.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(observe): local agent-readable flight-recorder ledger (runs.jsonl)

The connective layer of the four-instrument loop: a versioned, append-only,
tolerant-to-torn-lines JSONL ledger every instrument (verify/gate/measure/observe)
appends to, read off one schema by audit, by the agent debugging its own harness,
and later by an aggregation surface. Harness-agnostic; lives at the composition
root (not core, not an adapter). Best-effort append never breaks a session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: refresh HANDOFF for direction + observe-ledger session

* feat(observe): emit compiled-hook + agent-rail decisions to the ledger

First emit points into .vigiles/runs.jsonl: the compiled-hook gate (emitGate —
enforce deny/ask + observe would-be) and the subagent tool-contract rail
(agentHookCommand deny). The legacy hook-observations.jsonl still written for
back-compat; the unified ledger is additive. E2E asserts the observe-mode run
populates runs.jsonl with a decision-shaped hook record.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(observe): render the flight recorder in vigiles audit

formatLedgerSummary (pure, tested) — total + counts by kind + recent
high-signal denials (blocked gates, out-of-contract tool calls) — and audit
prints it (terminal, non-json) off the local ledger. Empty/skipped until
something is recorded. JSON AuditReport unchanged (a versioned schema bump is
the deliberate follow-up).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: gitignore the observability ledger (.vigiles/runs.jsonl)

Runtime flight-recorder artifact — local record, never committed (sibling of
active-skill.json + eval-cache/).

* feat(observe): emit skill-fire + trigger-rate metrics to the ledger

skill-start records a SkillObservation{fired:true} (name = the skill dir); a
completed measureTriggerRate records recall (+ precision when measured) as eval
records. Extends ledger coverage across the behavioral surfaces alongside the
deterministic gate/rail emits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(skills): add model-invocable debug-my-harness skill

Reads the .vigiles/runs.jsonl flight recorder to diagnose why the harness
misbehaved — skill fires/collisions, hook decisions (blocked vs wrongly-allowed),
subagent contract violations, trigger-rate drift — and recommends a fix, handing
off to strengthen/edit-spec/test-harness. The agent-readable payoff of the
observe layer. Ships a dogfood trigger eval (recall + precision); loads clean
(skills-dogfood). Skill enumerations + keyFiles synced (3→4 model-invocable).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(observe): emit capability-diff to the ledger

A computed capability-diff (--capability-diff) now records a capability-diff
record (added/removed tools + widened verdict). Completes the ledger's emit
coverage — all five record kinds (hook/agent/skill/eval/capability-diff) now
have live emitters.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: refresh HANDOFF — observe layer built (ledger + 5 emits + audit render + skill)

* feat(observe): carry the flight-recorder summary in the AuditReport JSON + HTML

summarizeObservations (pure, shares denial logic with the terminal formatter)
→ an additive/optional observations field on AuditReport (no schema bump, per the
adoptability/adoptable precedent). The CLI reads the ledger once, feeding both the
JSON (the product boundary — audit --json + the HTML) and the terminal render. New
report app: schema mirror + a presentational Observations component + a Flight
recorder section. Covered by observe + audit-report tests; end-to-end --json + HTML
verified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: refresh HANDOFF — AuditReport JSON/HTML ledger integration done

* docs(readme): verify-first front door (catch the silent breakage)

Retune the tagline + hook to lead with the verify moat (references are real,
not just well-formed) over the prior test-first framing; add a coding-scoped
subhead naming CLAUDE.md/skills/hooks + Claude Code & Codex. audit stays the
proof-first hero.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(readme): lock verify-first tagline in the editor-guide (prevent drift)

* docs: encode research-depth discipline (vault) + link directionality (doc-tiers)

Vault operating rule 6: competitive/strategy research saved in full detail,
depth-tagged (deep/medium/thin), coverage-mapped, cross-linked, indexed — shallow
never masquerades as deep. Root doc-tiers: link directionality invariant — links
point outward only (vault may cite research; research/public must never cite the vault).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(readme): demote the 7-catalog breadth from the headline

Lead the reference-verification pitch with the harness-native cross-ref (a
referenced tool/hook/file/skill that doesn't resolve). The 7-linter-catalog
breadth becomes a supporting mention, not the headline number.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor: delete hook-spec spike and demote scaffold-test to skill-internal

Trim the launch surface of two dead-weight items flagged by a strategy review:

- Delete the src/core/hook-spec.ts + hook-spec.test.ts spike — imported
  nowhere and superseded by the shipped compiled hooks (src/core/hook-program.ts).
  Drops its two keyFiles entries and recompiles CLAUDE.md.

- Demote the scaffold-test VERB to skill-internal: remove it from VERBS +
  its dispatch/handler in the CLI (the engine src/scaffold-test.ts + its test
  stay, used by the test-harness skill). Prune the now-orphaned helpers/imports,
  the scan-cli e2e block for the removed verb, and the scaffold-demo example,
  and update the docs (cli.md, spec-format.md, STABILITY.md, CLAUDE.md).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: refresh HANDOFF with observe layer + verify-first pivot state

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* style: format harness-observability-direction.md

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(action): capability-diff PR comment (blast-radius diff vs PR base)

Fold the agent capability-surface diff into the sticky PR comment: on a
pull_request the Action materializes the PR base with `git archive` (no
worktree/smudge), runs `audit --capability-diff`, and appends a short
section only when the surface changed. `fail-on-widen` gates a widening.
Dogfooded in this repo's own CI (fetch-depth: 0 + capability-diff: true).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* vault: salvage deep agent-eval-testing competitor teardown (partial workflow)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: refresh HANDOFF — capability-diff shipped + teardown salvaged

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(vault): post-mortem the teardown throttle+interrupt; log the 2x2 re-run

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* vault: fold in the 4 re-run teardown clusters (all 5 now deep, 41 companies)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: HANDOFF — teardown re-run complete (5 clusters, 41 companies)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* vault: add raw structured teardown appendix (41 companies, durable source)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* vault: add the Opus cross-corpus strategic synthesis (conformance-kernel thesis)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* vault: fold in Fable contrarian pass + Opus reconciliation (registry-first bet)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* vault: record the distro thesis (leading direction) + index it

* docs: fix doc-tiers — repo is PUBLIC, strategy is vault-only (stop leaks)

The doc-tiering rules wrongly treated research/ + CLAUDE.md as a 'private
record' holding the moat/positioning narrative. The repo is open-source, so
those tiers are public. Corrected: research/CLAUDE.md hold the TECHNICAL record
(how the tool works, public-safe); all strategy/moat/monetization/pivot theses
live ONLY in the git-crypt vault. Adds the competitor-playbook test.

* docs: de-strategize + refresh HANDOFF (it's public — strategy → vault)

* docs: move 6 strategic docs to the git-crypt vault (repo is public)

measurement-authority, divergent-bets, distribution-strategy, typed-spec-moat,
strategic-synthesis-2026-06, eval-startups-positioning move from research/ (public)
to startup/ (encrypted). Inbound references neutralized (no vault inward-links),
index/keyFiles entries removed, both specs recompiled, vault index updated.

* fix: encrypt the 6 migrated vault docs (git mv bypassed git-crypt)

* docs: record the git-mv-into-vault-doesn't-encrypt gotcha + fix rule 3 (all public)

* docs: pivot-decision (vault) + roadmap reshape + finish moved-doc ref cleanup

Adds startup/pivot-decision-2026-07.md (decision + validation gate). Adds two
pivot-agnostic technical items to research/roadmap.md 'Now' (step-enforcement A/B,
observe→promote) + de-strategizes its top direction banner. Neutralizes the
relative-path refs to the 6 vault-moved docs the migration missed (README +
plugin-structural fully clean; roadmap partial — 37 scattered refs flagged as
follow-up).

* docs: HANDOFF — migration done (targeted-6) + flag roadmap ref-sweep follow-up

* vault: save the pivot exploration log (all options considered + why)

* docs: finish roadmap de-strategization (neutralize moved-doc refs + gut GTM section)

* fix(cli): write capability-diff observation to the audited root's ledger

appendObservation defaulted to process.cwd(), so `audit ./after
--capability-diff=./before` from a parent dir recorded the blast-radius event
into the caller's .vigiles/runs.jsonl instead of ./after's — where the audit
reads it back via readObservations(root). Pass root. (Codex review #55.)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-07-02 23:18:27 +05:00
zernie 15c0607a5b feat!: eval lock (CI staleness gate) + full Codex parity for hooks & inject (#53) 
* feat!: eval lock (CI staleness gate) + per-adapter harness versionKey

Real-model evals run on your subscription, so they only run locally — never
in CI. The eval LOCK closes the resulting gap: a committed integrity stamp
that lets CI verify the committed numbers still match the current inputs
WITHOUT a model call (distinct from the gitignored, local-speed eval cache).

- `vigiles eval --update` (local): records each NAMED eval's report to a
  committed `.vigiles/eval-locks/<name>.lock.json` + prints the per-number delta.
- `vigiles eval --check` (CI): recompute the input hash, fail "stale" on a
  mismatch — binary-free, deterministic. Green no-op until the first lock is
  committed (smooth adoption).
- The clean split: the lock stores only the model's observed behavior; the
  script's own assertions re-run live against the replayed report, so a
  threshold-only edit is a valid replay while an input change is stale.
- Honest scope: the harness version is provenance, NOT a hash input (CI's
  pinned claude ≠ a dev's local claude would false-trip), and there is no
  nightly run — model/harness drift is a local `--update` concern.

Also extracts the harness version reduction to a per-adapter port method
(`HarnessRuntime.versionKey`): Claude Code keys major.minor (~quarterly),
Codex opts out with "" (its minor is patch-cadence, ~weekly). This fixes the
local cache churning weekly for Codex and is where the cadence decision belongs.

Whole-package coherence: the GHA gains `command: eval-check`, `vigiles init`
scaffolds the eval-check CI job, and this repo dogfoods it. Wired at both eval
seams (runEvalWith + measureTriggerRateWith), 100%-covered, docs updated
(internal research + public guides).

BREAKING CHANGE: `HarnessRuntime` (exported on `vigiles/adapter`) gains a
required `versionKey(raw: string): string` method. Third-party adapters must
add it; return "" to opt out of version partitioning. First-party adapters
(claude-code, codex) are updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9

* feat: eval-lock agent awareness (skill + nudge hook) + cohesive-delivery rule

Close the loop on the eval lock so the AGENT knows it exists and maintains it —
without the invasive option of editing the user's CLAUDE.md.

- test-harness skill documents the lock lifecycle (--update/--check, commit the
  lock, re-run after editing a skill).
- A PostToolUse nudge hook (hooks/eval-lock-nudge.sh → hook-runtime
  eval-lock-nudge): after editing a SKILL.md / *.eval.* with committed locks
  present, inject a NON-BLOCKING reminder to re-run `vigiles eval --update`.
  Self-gated (silent until a lock is committed), never blocks (the gate is CI's
  `eval --check`). Wired into the vigiles plugin; runHook + unit + .sh tests.
- NO init edit to the user's CLAUDE.md — verified that's not how plugins deliver
  context (no plugin-level instructions file exists; the channel is a hook/skill).

Docs + dedup:
- research/agent-context-delivery.md captures the VERIFIED CC/Codex
  context-delivery mechanics ONCE (SessionStart additionalContext is the plugin
  channel; CLAUDE.md is the user's file; skill descriptions always-in-context;
  Codex AGENTS.md + hooks but inject deferred) so we stop re-researching it.
- Per-adapter guides (harness-testing-claude-code/-codex.md) explain how the
  reminder reaches the agent per harness (CC injects; Codex inject is a documented
  follow-on), with the agnostic guide linking DOWN to them (no duplication).

Cohesive-view rule:
- New CLAUDE.md `cohesive-feature-delivery` rule — a definition-of-done CHECKLIST
  (CLI × GHA × init × agent-awareness × both adapters × per-adapter deduped docs ×
  tests) so a feature's INTEGRATION across surfaces is checked as one thing,
  generalizing cohesive-cli-surface + harness-parity + great-agent-flow.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9

* test: CLI e2e for the eval-lock invalidation flow (update → check → stale)

Closes the gap flagged in review: the joined CLI flow wasn't tested as one
thing. A real *.eval.cjs fixture (model-free — drives runEvalWith from dist with
a fake runner) run through the built CLI proves the whole loop: `eval --update`
writes a committed lock, `eval --check` with the same inputs replays green (no
model), and an input change makes `eval --check` go red ("stale"). Binary-free,
no subscription — exactly how it runs in CI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9

* docs: scannable eval-lock docs + prose-clarity & easy-adoption rules

Make the eval-lock docs readable at a glance and frame upkeep as agent-driven,
not a manual chore.

Docs (scannable rewrite — ASCII flow diagram, comparison tables, ✅/❌/⚠️
signposts, short sentences, no walls of text):
- docs/harness-testing.md — the lock how-to now leads with "you mostly won't
  touch this by hand; the test-harness skill records it + a hook reminds you,"
  then a diagram + tables.
- docs/harness-testing-claude-code.md / -codex.md — the per-harness nudge
  delivery as tables; the Codex inject caveat is one clear ⚠️ row.
- docs/eval-architecture.md — the "what's stale vs a free replay" split as
  ✅/❌ bullets.
- skills/test-harness/SKILL.md — Step 5 now tells the agent to run
  `eval --update` + commit the lock as part of its job (so it IS done for you).
- README — the Quick-start reassurance leads with "Adoption is smooth: one
  command, then your agent does the rest," names skills AND hooks, no chores.

Two new CLAUDE.md rules:
- prose-clarity — public prose must be clear + scannable (lead with the point,
  one idea per sentence, define/drop jargon, tables/bullets/ASCII/Mermaid over
  walls, emoji as signposts, one bold per point). Prefer ASCII/Mermaid; never
  commit generated raster images into docs.
- lead-with-easy-adoption — docs (README first screen especially) must make it
  clear up front that adoption is low-friction because skills + hooks do the
  work; frame every feature's upkeep as agent/skill/hook-driven, not homework.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9

* docs: prose-clarity sweep across 13 user-facing guides

Apply the new prose-clarity rule to the prose-heavy public guides: lead with the
point, one idea per sentence, tables/bullets/ASCII over walls of text, ✅/❌/⚠️/ℹ️
signposts, one bold phrase per point, jargon glossed or dropped. Net −578 lines —
shorter and far more scannable, with meaning unchanged.

Swept: for-plugin-authors, what-vigiles-catches, measuring-skills, skills,
harnesses, comparison, faq, sandboxing, safety, agent-setup, agent-workflows,
verifying-instruction-files, compiled-hooks. The onboarding pair leads with the
low-friction "one command, the agent does the rest" framing.

Reference docs (cli, testing-api, spec-format, …) and the per-rule docs were
deliberately left alone — they're already table-shaped, and churning them is
high-risk / low-value.

Verified: no heading text changed (anchors intact), no `research/` links added to
public docs, the validation-rules matrix is complete (28 rule links → 28), safety
claims retained, command refs resolve (self-command-refs), no orphan docs, fmt
clean. Produced by parallel rewrite agents under strict preserve-anchors/links/
code constraints, then verified + spot-read.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9

* feat!: encode hook inject support into HookProtocol.injectableEvents

Codex inject was documented as "deferred" but the official Codex hooks
docs confirm `additionalContext` is honored on SessionStart /
UserPromptSubmit / PreToolUse / PostToolUse / SubagentStart — the same
shape as Claude Code. The gap existed because inject support was a prose
deferral, not a tested contract: nothing in the adapter conformance kit
asserted a shellHooks harness could actually deliver an inject hook.

Encode it as a required port field so the gap can't recur silently:

- HookProtocol gains `injectableEvents: readonly string[]` — the events
  that honor `additionalContext` injection, per-adapter.
- claudeCodeHookProtocol: [SessionStart, UserPromptSubmit, PostToolUse];
  codexHookProtocol: + [PreToolUse, SubagentStart].
- Conformance rejects a shellHooks adapter with an empty list; the
  adapter-contract suite asserts every shellHooks adapter declares the
  events vigiles' shipped hooks use (PostToolUse, SessionStart) — the
  test that would have caught the original miss.
- `compile` warns on an inject hook only when its event isn't injectable
  on the target harness (react output stays CC-confirmed-only).
- Correct every "Codex inject deferred" reference across docs, research,
  and CLAUDE.md; document injectableEvents in the adapter API + authoring
  guides.

BREAKING CHANGE: HookProtocol (exported on vigiles/adapter) gains a
required `injectableEvents` field — a third-party adapter must declare it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: refresh HANDOFF for the eval-lock branch + inject-port encoding

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat: fan out hook install + the CI workflow to every declared harness

A repo can enable BOTH harnesses (`harness: ["claude-code","codex"]`),
and instruction files already mirror to both — but `compile` installed a
hook into only ONE harness (`installHooks` ignored `config.harness` and
auto-detected a single adapter), and the scaffolded CI workflow hardcoded
the Claude Code binary. So an inject/gate hook that works on both could
land in `.claude/settings.json` while silently missing from
`.codex/config.toml` — the "works on CC, absent on Codex" failure the
harness-parity rule exists to prevent.

- `resolveHarnessAdapters` (adapter-registry.ts) returns the FULL declared
  set: `--harness=` flag → that one; else config `harness` list → all of
  them; else auto-detect → the detected one (de-duped by name).
- `installHooks` loops over it, merging the SAME compiled hook into every
  enabled harness's native config (JSON / TOML), with per-harness warnings.
- The `init` CI workflow is harness-aware: the deterministic harness job
  installs `@anthropic-ai/claude-code` and/or `@openai/codex` per declared
  harness (`harnessTestBinaries`).

Skills compile byte-identical across harnesses and subagents are CC-only
by design, so hooks + the CI binary were the real fan-out gaps.

Tests: resolveHarnessAdapters units (fan-out / flag-singular / detect /
alias-dedup), a both-harness hook-install e2e (the same hook in both
configs), and harness-aware workflow e2es (codex-only, both). Documented
as §4 in research/multi-harness-compile.md + the compiled-hooks guide.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat: wire vigiles's nudge hooks into .codex/config.toml during init

Closes the last Codex-parity gap in agent-awareness: Claude Code gets the
eval-lock + refs nudge hooks from the global marketplace plugin, but Codex
has no global plugin store, so it got skills only — the proactive nudges
never reached a Codex user (the "works on CC, silently absent on Codex"
failure).

Codex config is repo-committed (the idiomatic place), so `init` now wires
the two PostToolUse nudges into `.codex/config.toml`:

- `codexPluginHooks` (pure) declares them as DIRECT `npx vigiles
  hook-runtime …` commands — no plugin root, no vendored bash script. The
  runtime entrypoints read the event JSON on stdin and emit the
  `additionalContext` shape Codex honors on PostToolUse (confirmed +
  encoded in HookProtocol.injectableEvents).
- `applyCodexPluginHooks` (pure, unit-tested) merges them idempotently:
  re-runs replace in place, and the user's own Codex hooks + every other
  config key are preserved. `wireCodexHooks` in cli.ts does only the IO.
- Safety: only an INTENTIONAL exit 2 blocks an edit (the refs nudge under
  `unmarked-refs: error`); an npx-resolution failure exits non-2, so a
  missing dep never blocks.

Loud, documented deferral (no-silent-skips): the SessionStart lint summary
(plain stdout, unconfirmed on Codex) and the compile-on-edit / pre-edit
guards (filename-gated bash, no harness-neutral entrypoint yet) stay manual
on Codex.

Also: the remaining pre-release Codex item — running the eval tier against
the real `codex` binary once — is added to the launch-readiness P0 list in
the roadmap.

Tests: codexPluginHooks shape + applyCodexPluginHooks (fresh / idempotent /
preserves user config); the planPluginInstall codex notes/vendors updated.
Verified end-to-end that `init --harness=codex` writes the config and a
pre-existing `.codex/config.toml` (model + a user hook) is preserved.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: update codex install e2e for the new vendors:true (repo hooks)

The codex install plan now writes nudge hooks into .codex/config.toml, so
`vendors` flipped to true. Updated the unit assertion last commit but missed
this e2e (run by both the `e2e` and `test` CI jobs). The skills-are-global
checks below are unchanged and still stand.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: cover the non-Actions lock message branch + address Codex review

The `test` job's coverage gate failed: `emitLockMessage`'s console.warn/
console.log branches (the non-GITHUB_ACTIONS path) are unreachable in CI,
where GITHUB_ACTIONS is globally set. Added a wiring test that DELETES the
var to exercise both branches regardless of environment.

Also addresses two Codex review findings on the eval-lock work:

- P1 (soundness): `evalArmsInputs` omitted `spec.stubs` from the hash, though
  tool stubs are written onto PATH and are model-facing input — so a changed
  canned CLI output after `--update` left the same `inputsHash` and `--check`
  replayed stale results. Now folds a canonical (name-sorted) view of the
  stubs into the hash; a new wiring test proves a stub change → stale.

- P2 (collision): two eval names normalizing to the same slug (`foo/bar` vs
  `foo bar` → `foo-bar.lock.json`) could let `--check` replay the WRONG eval's
  report. `readLock` now rejects a lock whose stored `name` differs from the
  requested one, degrading a collision to a safe miss (→ stale/re-run), never
  wrong data. Unit-tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: two more eval-lock soundness gaps from Codex review

- An unnamed eval in `--check` mode called `produce()` (ran the model),
  breaking the no-model-in-CI contract: in CI there's no subscription, so it
  would hit the model or fail for missing auth. Now `check` mode throws a
  clear, actionable error for an unnamed eval (never reaching the model);
  `off`/`update` keep producing (the eval just isn't gated locally).

- `spec.ephemeralEnv` toggles the trial's run environment (scrubbed env +
  throwaway HOME vs the inherited process env) — a model-facing input that
  was omitted from the lock hash, so a report recorded under one mode could
  be replayed under the other with the same inputsHash. Now folded into the
  hashed inputs.

Both proven by new wiring tests (unnamed-check throws without calling the
runner; an ephemeralEnv toggle → stale). Coverage holds at 100% lines/
statements/functions (verified with GITHUB_ACTIONS=true).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: fold the harness identity into the trigger-rate lock hash

Codex review (P2): a trigger-rate eval recorded with one harness driver and
later switched via `measureTriggerRate(spec, { evalDriver: codexEvalDriver })`
hashed identically (prompts/model/plugin unchanged), so `--check` could
replay a Claude-recorded report for a Codex run — defeating the gate for the
Codex eval path this PR adds.

`EvalDriver` gains an optional `harness` identity (defaults to "claude-code",
so existing single-harness locks are unaffected); `claudeEvalDriver` and
`codexEvalDriver` declare theirs, and `measureTriggerRate`/`...With` thread it
into the hashed trigger inputs. A new wiring test proves a harness switch →
stale. API surface regenerated (additive `harness?` on EvalDriver).

This closes the input-hash bug class on the trigger-rate seam (the runEval
seam's stubs + ephemeralEnv were fixed in the prior two commits).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: refresh HANDOFF — PR #53 merge loop + the 5 review fixes

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: three more Codex-review findings (matchers + plugin-root path hash)

- Codex nudge hooks used Claude tool names (`Edit|Write`) in their matcher,
  but Codex's edit tool is `apply_patch` — so the eval-lock + refs nudges
  wired into `.codex/config.toml` would never fire. Matcher → `^apply_patch$`.

- The compiled-hook FAN-OUT to Codex regex-wraps the raw matcher without
  translating tool names, so a CC-authored `tools("Edit","Write")` gate emits
  `^(Edit|Write)$` into Codex config and silently never fires though compile
  reports success. vigiles has no cross-dialect tool-name map yet, so install
  now WARNS LOUDLY when a tool-matched hook is fanned to a non-CC harness
  (no silent non-firing success); a real mapping is the follow-up.

- For eval arms using `plugin`, `resolveHarness` expands `${PLUGIN_ROOT}` to
  the checkout's ABSOLUTE path before the settings are hashed, so a lock
  recorded at `/home/dev/...` was falsely STALE when `--check` recomputed it
  at `/home/runner/...` in CI. `stripPluginRoot` normalizes the prefix back
  to a token → location-independent hash. New wiring test: the same plugin at
  two different abs paths replays (not stale).

Coverage holds at 100% lines/statements/functions (verified GITHUB_ACTIONS=true).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: guard stripPluginRoot against a hookless plugin's undefined settings

A plugin that ships only instructions/skills (no hooks) resolves to
settings:undefined; JSON.stringify(undefined) is undefined, so the prior
.split() threw before the eval could run. Guard the non-string case (pass it
through). Regression I introduced in the plugin-root normalization last commit;
new test covers a hookless-plugin arm (update + check, no throw).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: add a lock-hash COMPLETENESS guard for both seams

The Codex review kept finding the same class of bug: a model-facing input
omitted from the lock hash, so `--check` replays stale results (stubs,
ephemeralEnv, driver-harness, plugin-root all slipped through). Our tests
proved the MECHANISM (record→replay, task-change→stale) but never asserted
the input set was COMPLETE.

This adds a structural guard: for each input that steers the run, mutate
ONLY it and assert the lock inputsHash CHANGES — one parametrized test per
seam (runEval's evalArmsInputs + trigger-rate's triggerInputs). A new input
added to EvalSpec/TriggerRateSpec without hashing it now fails a row HERE in
CI, instead of a reviewer finding it later. Covers task/model/tools/fixture/
arm.files/arm.settings/arm.model/interceptTools/stubs/ephemeralEnv and
prompts/irrelevantPrompts/model/harness.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-07-01 01:55:49 +05:00
zernie d8c639f8da refactor(cli)!: consolidate the verb surface (13→8) + docs/README overhaul (#46) 
* docs: sharpen README lint/guard copy

- Replace internal "7 catalogs" jargon with "7 linters (ESLint, Ruff,
  Clippy, and four more)"; Cedar stays in the subdoc
- Drop the plugin-audit paragraph from the Lint section (secondary use
  case, still reachable from More)
- Rewrite the Guard intro to lead with the concrete "fail open" failure
  and name what vigiles emits (exit code, JSON, AST-backed matcher)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: surface the $0/subscription cost story in the first paragraph

The "no metered tokens, runs on your Claude subscription" wedge was
buried in the last line of the Eval section. Pull a crisp version into
the opening paragraph (above the fold); the Eval section keeps the
detailed promptfoo/DeepEval contrast as proof.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: surface agent-writes-the-harness flow, reorder More, drop internal links

- Quick start now leads with "then your agent writes the harness for
  you": the three model-invocable skills (test-harness/strengthen/
  edit-spec) and the plain-English ask each one answers
- Reorder More so CLI sits above the leaderboard (whose link points into
  the CLI docs); trim the now-redundant Skills bullet to a pointer
- Stop linking internal research/ docs from the README (the moat link and
  the Research index) per public-vs-internal-docs

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(compile): auto-refresh an existing harness.gen.ts

compile now keeps a whole-harness registry in sync as a side effect, so
the user never hand-runs generate-harness. Gated on the file already
existing (compile maintains a registry you opted into, never imposes
one) and cheap (parses specs, no linter spawn — unlike generate-types/
generate-schema, which spawn every linter and stay on the config-change
guard). A duplicate agent name fails the compile.

Docs: note the auto-refresh in the compile section and mark the
generate-* artifacts as dev-toolchain output rarely run by hand.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(src): add nested CLAUDE.md with a high-bar-for-new-commands rule

Directory-scoped guidance loaded when working in src/: adding a new CLI
verb or hook-runtime subcommand must clear a high bar (genuinely new
mental action, no overlap with an existing command, siblings collapse to
one verb + subcommand, runtime entrypoints stay hidden). Complements the
root cohesive-cli-surface rule, placed next to the code that defines the
command surface (cli.ts, cli-commands.ts). Spec-managed (require-spec +
integrity); dogfoods vigiles compile.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: drop the "(generated)" annotation from the API reference link

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(build): stop src/CLAUDE.md.spec.ts shadowing the root spec + add docs rule

The nested src/CLAUDE.md.spec.ts (added last commit) compiles under
rootDir:src to dist/CLAUDE.md.spec.js — the exact path loadSpec resolves
the root CLAUDE.md.spec.ts to, so recompiling the root spec loaded the
src spec and clobbered CLAUDE.md. Exclude src/*.md.spec.ts from tsc so
both load via the tsx fallback (as the root always has).

Also adds the no-internal-links-in-public-docs guidance rule (public
docs never link research/*.md; keep the fact in research, don't link it),
reconciles the old "link to research" line in public-vs-internal-docs,
and files the deterministic lint-rule version as a P1 roadmap item.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: separate public/internal indexes + surface eval safety

- docs/README.md no longer links into internal research/ (public docs
  don't point at the private record)
- research/README.md is explicitly labelled the internal record
- README Eval section gains a "Safe to repeat" note (sandbox + ephemeral
  dir + egress + interceptTools, past-the-sandbox case flagged) linking
  docs/safety.md

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: stop public docs linking internal research/ + add safety FAQ

Strip every research/*.md link from docs/*.md (the no-internal-links-in-
public-docs rule) — ~60 links across 16 guides plus the docs/ index;
prose kept readable, no orphaned research doc (all still cited from
CLAUDE.md keyFiles / research/README.md, orphan-docs green).

Add a FAQ to docs/safety.md answering the side-effects-beyond-the-sandbox
question honestly: network via egress allowlist, a real DB/redis/browser
via a disposable container you stand up (turnkey provisioning is a
committed next step), irreversible externals via interceptTools, and the
no-Docker deterministic/record-replay tiers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(cli)!: consolidate verbs — measure/explain→scan flags, refs→lint, generate-*→generate <kind>

De-sprawl the human verb surface from 13 to 8 (the high-bar-for-new-commands
rule, dogfooded). All breaking:

- `measure <dir>` → `scan <dir> --trigger` (the model-gated behavioral
  column folded into scan; trigger-rate + collision matrix unchanged)
- `explain <dir> [name]` → `scan <dir> --explain [name]` (the deterministic
  WHY, a sibling of --fix-plan on the same ScanReport)
- `refs <file>` → folded into `lint` (the unmarked-refs check is already a
  lint rule; the PostToolUse hook keeps its shared collectRefIssues entry)
- `generate-types|schema|harness` → `generate <kind>` (one verb + subcommand;
  the three dev-toolchain generators grouped, not N hyphenated siblings)

scan reuses handleMeasure/handleExplain directly behind --trigger/--explain.
Docs, help text, the regen guard, tests, and the self-command-refs dogfood
all updated together (cohesive-cli-surface); 1588 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: split the GitHub Action reference out of cli.md into its own doc

cli.md was 718 lines (kitchen sink). Extract the ~108-line GitHub Action
section into docs/github-action.md (quick start, inputs, output channels,
versioning) and leave a short pointer; retitle the cli.md intro. Update
the README More bullet (CLI · GitHub Action) and the docs index. cli.md
→ 619 lines. orphan-docs + self-command-refs green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(lint): add prefer-compiled-hooks rule — one nudge toward compiled hooks

A single repo-level recommendation (one finding regardless of hook count):
when a plugin ships hand-written hook commands that aren't compiled
vigiles/hook artifacts, nudge toward compiling them — the message links
docs/compiled-hooks.md and points at guardrail-check for verifying the
ones you have.

A discovery nudge, not a defect: the hand-written shell lane stays
first-class, so it fires ONCE (never per-hook), defaults to warn (info-
level ℹ; informational, doesn't fail CI), and is opt-out. Detector
(manualHookCount: a command that isn't a hook-runtime invocation) lives
in scan.ts and is shared by scan + lint (one-detector-no-drift). Doc +
matrix rows added (rules-docs-in-sync). Honest caveat in the doc: it's
form-based, so it stays a recommendation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(research): record the eval-startups critique mapped onto positioning

New research/eval-startups-positioning.md captures Thomas Liao's "why eval
startups fail" critique and its implications: vigiles dodges the trap
(harness-eval not model-eval, tooling not services, eval already a feature),
but it's a yellow flag on the public ecosystem-benchmark as a MOAT
(Goodhart-gameable → keep it a thin acquisition flywheel) and it
independently validates the attestation/safety direction (capability-diff,
guardrail proof) as the only defensible eval-shaped niche.

Linked from research/README.md, measurement-authority.md (whose benchmark
thesis it challenges), and strategic-synthesis-2026-06.md (whose Tier-2
conformance/attestation play it validates). Internal-only per
no-internal-links-in-public-docs; orphan-docs green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: purge internal positioning from public docs (jargon, name-drops, benchmark)

Public docs (README + docs/) now state user-facing value, not vigiles's
internal strategic vocabulary:

- strip "moat" from 14 places → user-facing wording (engine / cost
  advantage / differentiator / cross-reference)
- remove all bare research/*.md name-drops from docs/ (the backtick form
  the link-strip + deterministic check miss); scope-descriptions reworded
  to "configured directories" rather than naming research/
- README: drop the "ranks the hyped ecosystem: what works vs hype"
  benchmark banner from the Eval row + kicker (depth stays in
  measuring-skills.md) — per eval-startups, the benchmark is a flywheel,
  not a headline
- docs/cli.md: "moat #2 (research/…)" → "the capability-diff check"

Codified in CLAUDE.md: public-vs-internal-docs now has a strategic-
vocabulary clause (moat/measurement-authority/flywheel stay internal;
public docs name the benefit), so this doesn't regress. orphan-docs +
self-command-refs green; the public↔private split stays docs/ vs research/.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(readme): apply multi-persona review fixes

From a multi-persona README review (power user / staff eng / plugin author
/ non-TS user / eng manager):

- non-TS users: a visible "No TypeScript needed — lint + inline comments
  work on any CLAUDE.md" line (was invisible; contradicted progressive adoption)
- plugin authors: surface "vigiles scan ranks a marketplace, no key" above
  the fold (was buried in More)
- eng-manager cost: hero now says deterministic layers run free in CI;
  merged kicker+affordability nails "real-model on a dev's own subscription,
  never a CI token"
- staff-eng skepticism: the 2/7→7/7 claim now reads "measured, not asserted";
  the typed-pipeline claim is concretized to the real pipe()/needs() mechanism
- fixed a broken blockquote and trimmed the safe-to-repeat para to hold length

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(readme): capture the README design direction as an editor-facing comment

A top-of-file comment block records the framing the README must hold: lead
with benefits, present compiled specs as the easy/auto-enforced next step
(skills write them, hooks compile them; JS→TS evolution, not a wall), keep
all four instruments first-class incl. Eval, stay scannable + under the
length cap, and no internal vocabulary/research links. Guides future edits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(readme): benefits-forward rewrite + specs-are-easy framing

Per the README direction: lead with what the user GETS, and present the
typed spec as the frictionless auto-enforced next step, not a TS wall.

- hero: stronger, concrete failure modes (stale ref / guard fails open /
  skill never fires), keeps Agent=Model frame, cost above the fold;
  dropped the redundant centered pitch (merge)
- table cells condensed to punchy benefit one-liners
- "You don't hand-write specs — your agent does. TypeScript for your
  harness: opt-in, gradual, auto-compiled, never a chore" (skills write
  them; JS→TS evolution)
- de-dupe cost (drop "not metered tokens" from the Test cell)
- "Not for you if …" honest-scope line
- docs/verifying-instruction-files.md: back the claim — "you rarely
  hand-write one; edit-spec/strengthen write it, a hook recompiles on save"

Eval stays a first-class instrument. fmt + self-command-refs + orphan-docs green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(readme): don't lead callouts with "typed spec" — benefit first

Casual readers shouldn't be greeted by "a typed spec is a program" /
"TypeScript for your harness" in the prominent callouts. Reframe both to
lead with the outcome and demote the spec to a soft, opt-in aside:

- specs-are-easy block: "You don't hand-write any of this — your agent
  does … you never touch it by hand" (drop the TypeScript-for-your-harness
  headline)
- Lint blockquote: "Bad states stop compiling … turn it up only when you
  want it" (drop "Markdown is prose; a typed spec is a program")
- the edit-spec bullet says "edits the source", not "the typed spec"

"No TypeScript?" reassurance and the optional .spec.ts mention stay.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: refresh HANDOFF for the CLI-consolidation + docs/README overhaul session

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: prettier-format HANDOFF

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ci): update the stale generate-types ref the verb rename missed

The CI "Check generated types compile" step ran `node dist/cli.js
generate-types` — renamed to `generate types` in the verb consolidation,
so the job failed with "Unknown command". self-command-refs didn't catch
it because it didn't scan .github/; add .github/ to its scan scope so a
stale `cli.js <cmd>` ref in a workflow fails the dogfood from now on.

Also saves the prefer-compiled-hooks-vs-require-hook-spec decision to
research/hook-modes-and-testing.md (the nudge, not a requirement; the
write-hook authoring skill is the real authoring lever, unbuilt).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(lint): eslint-ignore src/*.md.spec.ts (excluded from tsconfig)

The dist-collision fix excluded src/*.md.spec.ts from tsconfig, so
eslint's type-aware project service can't resolve src/CLAUDE.md.spec.ts
and errors "not found by the project service". It's a tsx-loaded build
input (like the root CLAUDE.md.spec.ts, which isn't linted either), so
ignore it in eslint to match. Latent since the exclude landed; surfaced
now that the earlier generate-types step no longer fails first.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-25 01:20:05 +07:00