mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
zernie
9b3bf050d3
* research: capture skill-authoring pains + documentation-vs-procedure note
Three parallel research sweeps on the real pains of authoring agent skills
(triggering, silent script/path drift, testing gaps, distribution). Key
finding: the most-documented skill pain — referenced scripts/commands/paths
drift and the skill silently lies to the agent — maps directly onto vigiles's
cross-referencing engine, and existing skill linters are schema-only, so that
gap is unclaimed. Adds a strategic note on the documentation (CLAUDE.md) vs
procedure (SKILL.md) split and an asymmetric require-spec policy. Wired into
Key Files and recompiled.
* wip: parse vigiles:file / vigiles:cmd refs in inline + frontmatter modes
Extends the markdown-mode parsers to also extract file() and cmd() style
references so markdown mode can verify paths and npm scripts, not just linter
rules:
- inline: <!-- vigiles:file <path> --> and <!-- vigiles:cmd "<command>" -->
- frontmatter: vigiles.files and vigiles.commands string lists
- hasInlineRules/hasFrontmatterRules now count file/cmd refs as spec-equivalent
Parser-only and additive (returns new files/commands arrays); audit wiring in
cli.ts and docs are intentionally not yet done — paused pending a scope
decision on whether to lead with skill verification. Compiles clean (tsc).
* research: skill-as-pipeline design — harness-driven control-flow graph with deterministic gates
Capture the converged model from the skill-design discussion: a skill as a
control-flow graph the harness interprets (Railway out, monads in), deterministic
gates as the verification layer, the enforcement-strength spectrum (detect →
gate-progression → interpreter-driven), the linear→graph axis, and the resulting
markdown↔spec boundary (linear → markdown checklist, branching → structured graph).
Cross-linked from skill-authoring-pains.md.
* research: add prior-art map (occupied/free) + borrowable patterns + build order to skill-as-pipeline
Three landscape sweeps (agent harnesses / LLM forcing+verification / adjacent
domains) converge: the prose-step + per-step deterministic gate + author-time
reference verification union is unoccupied. Behavior trees are the closest
execution-skeleton analog (SUCCESS/FAILURE/RUNNING, Sequence/Selector short-circuit,
Condition=gate vs Action). Records the phased build order: author-time skill-gate
verification first (reuses the cross-referencing engine + the in-flight file/cmd
parser), then a runtime gate-progression hook, then structured graph + hard driving.
* feat: gated skill pipeline — typed inputs + per-step gates + result gate
Extend skill() with:
- inputs (typed) → argument-hint frontmatter + a ## Arguments section
- steps with a deterministic gate (cmd/file) + optional retry → ## Steps
checklist, each rendering a vigiles:gate marker for the future runtime
- result: a terminal postcondition gate → ## Result + vigiles:result marker
compileSkill verifies every gate's reference (cmd in package.json, file on
disk) at compile time, reusing the cross-referencing engine. This occupies the
empty quadrant from the prior-art research: author-time verification of the
deterministic gates that close each prose step.
Adds src/skill-pipeline.test.ts and examples/ship-pr as a worked example.
* dogfood: author generate-rule as a gated skill spec
Convert the hand-written generate-rule SKILL.md into a typed skill() spec
using the new model: typed input -> argument-hint, five linear steps, a
verified gate (npm run build, retry:2) on the compile-and-verify step, and a
result gate. Compiled with vigiles' own CLI; the gate reference is verified
against package.json at compile time. Closes require-skill-spec for this skill.
* chore: add compiled ship-pr example SKILL.md (sibling of its spec)
* feat: skill runtime — execute the gate ladder (vigiles run-skill)
Parse the vigiles:gate / vigiles:result markers a compiled SKILL.md carries and
*execute* them: run each step gate in order, short-circuit on the first failure
(Railway error track), then the result gate. Exposed as `vigiles run-skill
<SKILL.md>` — exit 0 when every gate passes, exit 2 (blocked) otherwise.
This turns the markers from documentation into enforcement: a skill is not done
until its result gate passes. v0 runs the gates only; driving the model through
the prose steps one at a time needs a live harness (later phase). retry:N is
parsed/surfaced but run once — it is meaningful only with the model in the loop.
Also: wire skill-pipeline + skill-runtime tests into npm test (they were not
enumerated), and extract annotateAuditForGitHub to keep main() under the
enforced cognitive-complexity limit.
* chore: regenerate types for new skill-runtime sources; format generate-rule spec
* feat: Stop-hook skill enforcement (skill-start / skill-hook / skill-done)
Track the active skill in .vigiles/active-skill.json; the Stop hook
(`vigiles skill-hook`) runs that skill's result gate and blocks completion
until it passes — exit 2 + a fix message fed back to the model, exit 0 + clear
the marker on success. This makes the result gate enforce automatically in a
live session, not just via manual run-skill.
evaluateStopHook is a pure, harness-agnostic decision function (fully tested
with tmp skills). Auto-firing skill-start on skill activation is the Claude Code
integration step. Group the skill subcommands under handleSkillCommand to keep
main() under the enforced cognitive-complexity limit.
* feat: portable project-role gates (cross-repo fix)
A skill gate can now be a project role — project("test"|"build"|"lint") —
instead of a hard-coded command. Role gates skip author-time ref verification
(they carry no repo-specific reference) and resolve at run time to the HOST
project's real command, detected from its ecosystem (npm scripts, pyproject/
pytest, Cargo, go.mod). A role that resolves to nothing fails the gate rather
than silently passing.
Fixes the cross-repo gap the earlier dogfood surfaced: cmd("npm run build")
was verified against the authoring repo but a skill runs in others. Re-authors
generate-rule's gate/result as project("build"); it compiles to role: markers
and run-skill resolves them to this repo's npm run build.
* refactor: table-driven detectProjectCommand under cognitive-complexity limit
* feat: generator skill driver (durable-imperative form with branching/loops)
The declarative skill({ steps }) form compiles to a static SKILL.md and can't
express branching/looping. The generator form can: a skill is a generator that
yields effects (act/checkpoint/finish); driveSkill() runs it, feeding each act's
model answer back in so real if/for control flow works, running deterministic
gates between yields and short-circuiting on failure (Railway).
The model is an injected seam (runStep) — production calls the LLM, tests script
it — so the driver mechanics are fully tested without a live model: branching on
the model's answer, real loops, gate short-circuit, terminal result gate.
This is the form we designed (iterators inserting markers/effects for the
harness) that the declarative prototype had not yet implemented.
* chore: regenerate types for new skill-driver files
* test: harden skill-driver tests — trace content, failing result gate, file/role gates, degenerate program
* test(e2e): deterministic E2E driving real Claude Code against a mocked LLM
A dependency-free mock Anthropic Messages endpoint (mock-anthropic.mjs: scripted
turns, SSE, tool_use, count_tokens) + a bash runner (run.sh) that spawns the real
claude CLI with ANTHROPIC_BASE_URL pointed at the mock. Three scenarios, all
green against claude 2.1.x:
1. simple text relay
2. full tool-use loop (claude runs the tool, sends tool_result, loop closes)
3. Stop-hook enforcement — passing result gate → claude stops (1 turn);
failing gate → our skill-hook blocks completion, claude forced to continue
(hits --max-turns). Proves the gate enforces in a LIVE session.
claude inherits auth from the env (managed OAuth fd here / ANTHROPIC_API_KEY in
CI). Run via npm run test:e2e (skips cleanly if claude is absent). The mock is
self-contained and extractable into a standalone mini-package.
* chore: regenerate types after adding test:e2e script
* feat: skill spec — knowledge body composes with steps; script-runner gate verification
Two fixes surfaced by dogfooding the official docx skill:
- body + steps now render together (knowledge ## Reference section alongside the
gated ## Steps), instead of being mutually exclusive — so a knowledge-heavy
skill keeps its reference body AND gains a verified procedure.
- validateCommandRef now verifies script-runner gates (python/node/bash/ruby
<file>): cmd("python scripts/x.py") checks the script exists, not just npm
scripts. Module forms (python -m pkg) are skipped (no path). Catches the
missing-bundled-script rot for non-npm skills at compile time.
* refactor: extract collectSkillRefs/renderSkillFrontmatter/renderSkillSections to keep compileSkill under the cognitive-complexity limit
* feat: vigiles/skill-test — deterministic skill tests over the generator driver
A thin wrapper (runSkill + scriptModel) over driveSkill for use inside ordinary
node:test/Vitest — no custom DSL. Script the model (the non-deterministic part),
assert the deterministic spine: gate sequence, branch taken, loop iterations,
whether the result gate blocked. Makes 'write deterministic tests for your
skills' a first-class feature, reusing the driver we already have.
* fix(skill-test): single scriptModel signature, no unsafe return (lint clean)
* feat: max-length on inline code blocks — force extraction to files
A skill's inline fenced code block longer than maxInlineCodeLines (default 20)
is a compile error suggesting extraction to a file + file() reference. Keeps big
scripts out of the SKILL.md body (token budget + progressive disclosure); set 0
to disable.
* feat: generator -> SKILL.md compiler (ts compiler API)
Parse a generator skill's source and render its structure to markdown:
yield act() -> prose step, yield gate/checkpoint/result() -> vigiles:gate /
vigiles:result markers, if/else -> '### If <cond>' / '### Otherwise', for/while
-> '### Repeat (...)'. Gate references (cmd/file/project literals) are collected
and verified against the project, so the cross-referencing moat works on
generators too. This is what makes 'generator = the one spec' compile to a
hash-headed SKILL.md (agent-readable; auto-protected by the edit hook) while the
harness drives the real generator at runtime. typescript moved to dependencies.
* test: port real community skills as generators + prove coverage
Structural ports of the deep/praised community skills the flat model can't hold:
pr-review-loop (the corpus stress test — bounded 7-round loop, quality-weighted
exit, per-finding for-each, nested bounded CI retry), TDD red-green-refactor, and
subagent-driven (per-task nested bounded review loops). Driven deterministically
via runSkill + scriptModel (now supports per-key answer sequences), asserting the
control flow (round counts, ceiling, retries). The real source also compiles to
nested markdown with verified refs. Generator compiler now handles namespaced
callees (ns.act(...)).
* docs: comprehensive docs/skills.md (authoring tiers, gates, runtime, testing)
Document the full skill system: markdown / declarative / generator authoring,
gate verification (cmd/file/project/script-runner + maxInlineCodeLines), the
runtime (run-skill gate ladder, Stop-hook enforcement, edit-protection hook),
and deterministic skill testing (vigiles/skill-test). Referenced from the design
doc (orphan-docs clean); illustrative refs opted out via vigiles:ignore-file.
* research: scope — vigiles skills (authored, reusable, verified) vs dynamic workflows
Fix a scope blur after Anthropic's dynamic-workflows launch. Anthropic treats
skills and workflows as different primitives: skills are authored/reusable/
distributed units (our niche: verify refs + enforce result, valuable across many
invocations); dynamic workflows are ephemeral, runtime-generated orchestration.
Our gated-step model is NOT subsumed for skills (corrects an earlier overstatement).
Deterministic verification of dynamic workflows is an adjacent, optional frontier
reachable only by reframing the unit step -> action/acceptance gate. Records the
0/12 false-done benchmark and the honest value-regime (reuse / weak agents /
side-effects, not single strong-model runs).
* feat: generator skills compile via the CLI (genSkill)
Author a generator skill as: export default genSkill({ name, description },
function* () { ... }). The CLI detects genSkill in the source and compiles it
from source (compileGeneratorSkill: parse the meta object + generator, render
frontmatter + body + integrity hash, verify gate refs) — generators can't be
executed to markdown. Closes the loop: generator = the one typed spec ->
hash-headed SKILL.md (agent-readable, edit-hook-protected) while the harness
drives the real generator at runtime. runSkill now accepts a GeneratorSkill too.
* refactor: extract compile branches (generator/claude/skill) to helpers, keep compile() under the complexity limit
* feat: action gates — deterministic checks bound to a tool action (dynamic-workflow reframe)
The dynamic-workflow frontier from research/dynamic-workflows-and-scope.md: bind
a gate to an action TYPE (Write/Edit/Bash) instead of a plan step, so it fires
regardless of the runtime-decided order. evaluateAction runs the matching gates
(filtered by tool + optional 'when' input substring; cmd may template {file}
with the action's path) and blocks on the first failure. Delivered as a
PostToolUse hook: 'vigiles action-hook' reads the tool event on stdin + gates
from .vigiles/action-gates.json, exit 2 blocks with the reason. Same gate
primitive (reuses runGate) re-anchored step -> action. Prototype of the adjacent
frontier; plan-agnostic so it works in dynamic workflows.
* chore: sync package-lock with typescript runtime dependency
* bench: gated-skill vs vanilla harness (trap tasks + Stop-hook A/B runner)
Three validated trap fixtures (naive solution leaves the suite red): hidden
edge-cases (parse-list), missed call-site on a rename (rename-caller), and a
feature-add that regresses an existing test (cart-discount). run.sh runs the
real claude CLI on each task in two arms — vanilla vs a Stop-hook result gate —
and measures end-state `npm test` after the agent stops. Also refreshes the
generated .d.ts to include the skills modules added earlier.
* bench: drop stray generate-types artifacts from task fixtures
* bench: process-conformance harness — PreToolUse RED-before-GREEN gate (TDD)
Second benchmark targeting the claim a result gate cannot make: a deterministic
constraint that forces *procedure* order. Reuses action-gate.ts as a PreToolUse
hook that blocks any src/ write (Edit/Write/Bash) until a test is currently
failing, so the agent is test-first by construction. Both arms share identical
SKILL.md prose and a tool-agnostic observer that records write order; the only
difference is the gate. Metric: did the test file appear before src.
* bench: context-load sweep — does agent degradation reveal the result gate's value
Benchmarks #1/#2 were flat/negative because a fresh short agent self-verifies.
This sweeps prepended context (the project's own docs as realistic filler) and
measures end-state pass-rate vs context load, vanilla vs result-gated. Tests the
hypothesis that the gate's value appears only once the agent is degraded.
* bench: enlarge context corpus for near-cap load levels (doc set x2 ~180k tokens)
* bench: feed context-load prompt via stdin (128KB single-arg limit broke 80k+ runs)
* research: benchmark writeup — runtime gates are a no-op/net-negative for capable agents
Empirical result from bench/: result gate flat (48/48 both arms), process gate
net-negative (Goodhart — agent games the RED proxy, 6/8 vs 8/8 real tests at
2.7x turns), context load flat to 160k. Redirects center of gravity from
enforcing the route to verifying the map (the cross-referencing core).
* feat: markdown-mode verifies file/cmd references, not just linter rules
Wire the half-built vigiles:file / vigiles:cmd path (inline comments and
frontmatter files:/commands: lists) into audit: it was parsed but the refs were
discarded. Export validateFileRef/validateCommandRef, verify them relative to
the markdown file's dir, count errors, exit 2 on stale. Schema now allows
files/commands keys so the YAML LSP stops flagging them. Tests: parse-level
(inline + frontmatter) and audit E2E (valid passes, stale → exit 2).
* research: symbol-verification design & requirements (variant A, ast-grep, code-block cross-compat)
* feat: cross-language symbol index (ast-grep) — kernel for harness-pinned refs
definedSymbols extracts functions/constants/classes/methods (incl. enclosing
scope and bare-assignment constants) from TS/JS/CSS (core) and Python/Rust/Ruby
(bundled grammars), one generic name-field traversal. SymbolIndex builds a
project-wide name -> locations map; resolveSymbol returns unique/ambiguous/
missing for bare and scoped (Class#method) references — never a guess. This is
the resolver the write-time pinning hook will use.
* fix: find rbenv/asdf-installed linters by augmenting PATH with version bins
rubocop (and other gem/version-manager tools) install into a per-version bin
that is absent from PATH in non-login shells and CI, so 'rubocop not found on
PATH' fired even when installed. linters.ts now prepends the concrete
rbenv/asdf version bin dirs on load. Fixes the 2 rubocop CLI tests.
* feat: harness-pinned symbol references via PostToolUse hook
The corpus says authors write bare refs (parseConfig, User#full_name), ~never a
file#symbol form. So the harness resolves them: vigiles pin-hook (PostToolUse on
SKILL/CLAUDE/AGENTS.md) resolves each code-shaped inline reference against the
live project symbol index and pins the unique ones to a .vigiles sidecar,
surfacing typos/ambiguity to the agent at write time (non-blocking). audit
re-checks the pins — a renamed/removed/moved symbol breaks the pin (exit 2).
Opportunistic: only code-shaped spans are resolved, fenced blocks untouched (R1),
prose left alone. Adds vigiles pin (manual) + pin-hook (hook). Tests: 4.
* refactor: live symbol-ref verification, drop the sidecar (no drift)
A sidecar pin is a snapshot that drifts from the markdown (line shifts, out-of-
band edits, prose changes) — a new failure mode. The reliable design stores
nothing: there are two sources of truth (the instruction file's references, the
code's symbols) and a check is their live join at audit time. refs.ts replaces
pin.ts: audit re-extracts code-shaped refs from the CURRENT markdown and
resolves against the CURRENT index — a rename surfaces immediately with zero
state. Inferred refs are warnings (exit 1), not errors. The hook (refs-hook)
becomes pure write-time feedback, writing nothing. Commands: refs / refs-hook.
* docs: reconcile symbol-verification design with shipped live approach (no sidecar/markers)
* feat: file-qualified symbol refs (variant A) + enforcement hook
Revert to variant A as chosen: a reference names the file inline —
`src/config.ts#parseConfig` — and vigiles parses THAT file to prove the symbol
exists. No project-wide index, no bare-name resolution, no ambiguity (the file
disambiguates). Because the mark is deliberately authored, a broken ref is an
ERROR (exit 2), not a warning.
The refs-hook (PostToolUse) enforces the mark: it blocks (exit 2) any
code-shaped inline reference that is not file-qualified, telling the agent to
write `path.ext#symbol` or opt out with <!-- vigiles:ignore -->. The harness
makes the agent mark its references at write time, with context; vigiles then
verifies the agent's claim against the named file. audit re-checks marked refs.
Drops the SymbolIndex; symbols.ts keeps the per-file extractor + fileDefinesSymbol.
* docs: reconcile symbol-verification with variant A + enforcement hook
* feat: symbol() spec builder + co-located .d.ts/.rbi fallback
- symbol(file, name) spec builder: compile-verifies the named file defines the
symbol (cross-language ast-grep) and renders to the same file#symbol inline
form that markdown audit/refs-hook verify, so spec and markdown modes agree.
- fileDefinesSymbol falls back to a co-located declaration file (.d.ts for JS/TS,
.rbi for Ruby) so a metaprogrammed / ambient symbol still resolves without
running Sorbet or the TS compiler. Completes the symbol-verification feature.
* docs: mark symbol-verification items 3-4 done (spec builder, decl fallback)
* feat: explicit vigiles:symbol mark (consistent prefix, self-contained)
The bare path#symbol shape was inconsistent with the rest of vigiles' markers
and detached comments bind ambiguously in long lines. Switch to an explicit,
self-contained inline mark: `vigiles:symbol path.ext#symbol`. Literal prefix =
zero detection heuristic; file+symbol in one token = unambiguous binding even
with several refs per line. The refs-hook forces bare code refs into this form;
symbol() renders to it; audit/compile verify it.
* fix: inline rule-parser no longer flags skill/opt-out markers as unknown
The inline parser only knew enforce/file/cmd and errored on vigiles:result /
vigiles:gate (skill-runtime gates) and vigiles:ignore-file, failing audit on
files that legitimately carry them. Allowlist the non-rule markers; a genuinely
unknown marker still surfaces.
* docs: document symbol verification (README, design doc, CLAUDE.md keyFiles)
README: symbol() in Verified References + a cross-language Symbol references
section (vigiles:symbol mark, refs-hook enforcement); refs command in CLI list;
hero summary mentions code symbols. symbol-verification.md updated to the
vigiles:symbol mark form. CLAUDE.md.spec.ts adds src/symbols.ts + src/refs.ts to
keyFiles (recompiled).
* bench: symbol-ref benchmark harness (refs-hook value + Goodhart check)
Task: document the fixture's functions in a SKILL.md. vanilla (no hook) vs gated
(refs-hook forces vigiles:symbol marks). Measures marks produced, ignores
(Goodhart opt-out), broken marks, and the payoff: after renaming a documented
function, does audit catch the break (gated) or miss it (vanilla bare refs).
remeasure-refs.sh recomputes metrics from saved workdirs.
* fix: detect function-call refs foo(args) for the refs-hook (callee)
The hook missed the dominant way docs cite a function — `foo(args)` — because
isCodeShaped rejected the parens, so it never challenged that form. Strip a
trailing (...) and check the callee; the mark suggestion uses the callee name.
* bench: write up benchmark #4 (symbol-ref enforcement hook)
Result: gated CATCH_RENAME 2/6 vs vanilla 0/6 — the refs-hook is not a no-op,
but heavily gamed (avg 2.8 vigiles:ignore/run; 3 of 6 runs ignore-game). Draws
the clean line: verification of a declared reference is non-gameable (the win);
the hook that forces declaration is gamed like any gate. Also fix a pipefail bug
in the catch measurement (audit exits 2, masking grep under pipefail).
* feat: deterministic Claude Code harness testing (scriptable mock model)
mock-model.ts: a scriptable, dependency-free Anthropic Messages SSE mock — point
real claude at it via ANTHROPIC_BASE_URL (+ dummy key, no real auth/cost).
harness-test.ts: runHarnessTest({ files, settings, model: scriptModel([...]) })
runs the real claude CLI with your real hooks/settings against a scripted model,
so harness behavior is reproducible. Stop-hook enforcement tests pass
deterministically; tool-event/Edit-Write execution via the mock is environment-
sensitive (headless edit-tool gating + nested-CLI throttling) — those belong in
the real-model eval tier. The scripted 'steps' find their home here, in testing.
* feat: harness eval API + README testing section
runEval (src/eval.ts): drives the real claude CLI across arms x trials and
aggregates a metric — the empirical half of harness testing (generalizes bench/).
Adds the README 'Test your Claude Code harness' section (evals headline +
deterministic Stop-hook tests) and the testing modules to CLAUDE.md keyFiles.
* feat: runnable example eval (refs-hook A/B via runEval) + sh non-zero-exit fix
bench/evals/refs-hook.eval.mjs reproduces benchmark #4 as a library runEval call
(the bash runner's API form) — vanilla vs refs-hook, with the rename-caught
payoff metric. Fix ctx.sh to return captured stdout on a non-zero exit so a
measure can read 'audit' output (which exits 2). Verified: vanilla caught=0,
gated marks/caught > 0.
* docs: consolidate this PR's research + conclusions
Two new synthesis docs capturing the session's reasoning that lived only in
discussion:
- reference-verification-limits.md: the proxy-vs-judgment gap, prose
undecidability (active mark vs passive symbol-table sweep), the doc-format
landscape (explicit-link = marking; identity-based = the real fix, needs
non-plaintext), and the delegate/ignore/own rule (Sphinx etc.).
- harness-testing.md: the two-tier testing/eval design + a coverage assessment
against real plugins (protect-mcp, obra/superpowers, block-no-verify, skills).
Cross-linked into the benchmarks/symbol-verification web; added to CLAUDE.md
keyFiles (recompiled).
* chore: format bench fixture (fmt:check clean)
* fix: no-unsafe-assignment in frontmatter parseStringList (Array.isArray widens to any[])
The CI lint step (eslint src/) failed on one error: raw[i] is 'any' after
Array.isArray narrows unknown→any[]. Annotate the element as unknown so typeof
narrows it. (npm test doesn't run lint locally, so this slipped through.)
---------
Co-authored-by: Claude <noreply@anthropic.com>