mirror of https://github.com/zernie/vigiles.git synced 2026-09-14 20:53:57 +08:00

Files

T

zernie 195c1e14b7 feat: launch blockers, a hook-state test API, and the site's missing proof (#210 )

Ten launch blockers, all sharing one cause: a first reader gets a false claim or an embarrassing failure in their first ten minutes.

Fixes: ruff enabled-state was never read on a real repo; compile was unreachable without node_modules/vigiles (every Python/Rust/Go repo); a hooks-only plugin graded F/0 while its own report listed inlineHooks: 2; skill-resource findings printed a body-relative line number; the README quoted a caveman number its own FINDINGS.md had invalidated; STABILITY.md claimed v21 against 26.0.1; the Action's floating v1 tag was never created; a react's notice() reached nobody.

Features: experimental_hookState, so a stateful compiled hook can be tested by backdating the fact it throttles on; the site now leads with the proof (compiled hooks, the disaster battery, 2/7) and its OG card is rendered rather than hand-drawn, with a generator that fails loud if the page fetches anything off file:.

Also: this repository's first compiled hook, which is what exposed the notice-delivery defect; a floor under the harness CI job, which was green on zero files; and a smoke test of the PUBLISHED package, since every gate until now ran dist/.

2026-09-08 00:52:20 +05:00

8.8 KiB

Raw Permalink Blame History

FAQ

Quick answers to the questions people ask before adopting vigiles. The README has the pitch; this fills in the "but wait…".

Isn't this just a markdown linter?

No. A markdown linter checks style — heading levels, line length. vigiles checks whether your instruction file is true: every file path, script, code symbol, and linter rule it names must actually exist and be enabled, resolved against your real repo and your real linter config.

A renamed file or a disabled rule is caught before your agent ever acts on the stale claim. And that's just the Lint layer — vigiles also tests your hooks/skills and measures whether a skill helps. See verifying instruction files.

Why a typed spec — and do I have to write TypeScript?

No — your agent writes the spec, not you. init adopts your existing CLAUDE.md into one faithfully: every heading becomes a verbatim section, no rule invented. The edit-spec / strengthen skills author and maintain it from plain-English asks, recompiling on save. You rarely touch it by hand.

The spec is the default because markdown has no structure to check against. A typed .spec.ts gives you:

Autocomplete — rule names and paths resolve as you type.
Edit-time red squiggles — a dead path is caught before lint, not after.
Reuse — compose and share rules across files and repos.

Not ready for a spec? Lint plain markdown with zero new files — inline  comments, no TypeScript — and vigiles eject turns a spec back into markdown anytime. The deeper compiler-grade guarantees are gradual and opt-in, like TypeScript's strict — here's why.

Why are the strongest guarantees opt-in, not the default?

Because "opt-in" here means gradual, not permissive — and the deepest tier needs a typed program to work on. Three things to separate:

The default already catches real breakage. On your plain markdown — no spec, no TypeScript — vigiles fails CI on a typo'd tool, a dead hook, a broken MCP server, or two skills the model can't tell apart. A clean repo stays green. You get protection at rung zero; opt-in never means vigiles ignores what's actually broken.
Compiler-grade proofs need a program to check. Guarantees like "a config that leaks won't compile" or "a multi-agent pipeline whose handoffs don't line up won't compile" are properties of a typed .spec.ts — the same way tsc can type-check a .ts file but has nothing to check in inert prose. A markdown CLAUDE.md gives a compiler nothing to verify. So this tier is "opt-in" mainly in the sense that you first have to have written the typed spec. It can't be retrofitted onto a file that isn't a program.
Gradual is how adoption actually works — the same bet TypeScript made with strict. You climb a ladder (markdown → inline  comments → frontmatter → typed spec), and each rung pays off without forcing the next. Demanding the deep end at the front door is how a tool ends up with no users. And the guarantee loses no strength by being opt-in: once the spec exists, an unsafe config genuinely won't tsc.

In short: structural correctness is free and on by default; mathematical-strength proofs are opt-in because they require you to bring a typed program — and forcing that at adoption time is the surest way to get zero adopters.

Does `init` overwrite or touch my files?

No. init is non-destructive. If you already have a CLAUDE.md / AGENTS.md, it adopts it into a spec — every heading becomes a verbatim section, no rule invented — but leaves your file exactly as-is. You only switch a file to spec-managed when you run vigiles compile, with a diff to review.

vigiles eject hands any managed file back as plain markdown you own. It's never a one-way door.

Does it work with Claude Code and Codex?

Yes. vigiles targets Claude Code today and OpenAI Codex via vigiles/codex; the CLI auto-detects which one your repo uses (.claude/ vs AGENTS.md / .codex/). The Lint layer is harness-agnostic; the Test layer drives the real claude / codex CLI. You can even teach it your own harness.

Separately, vigiles reads plugins packaged to Agent Plugins. That is a packaging format, not another agent — a plugin shipped that way still runs inside Claude Code, Codex or VS Code, so it does not add a harness. What it does add: if your plugin uses that layout, its skills and its root mcp.json are audited with no flag and no config. vigiles ships as a conformant plugin itself (how to do the same).

Do I need an API key? What does it cost?

Most of vigiles needs no model and no key. Lint and the deterministic Test tiers run in milliseconds on every commit, free.

The only thing that needs a model is a real-model eval. That runs on your own Claude Pro/Max subscription via the claude CLI — $0 of metered API tokens. Tools like promptfoo / DeepEval hit a metered API and bill per token on every run. See measuring skills.

What does `vigiles audit` actually run — and why did it "find nothing"?

audit has two layers, and by default you get only the first:

The deterministic read (always on). A plain audit executes nothing. It reads your files and reports structural facts — a broken hook path, an undeclared MCP server, a tool-contract typo, two near-identical descriptions, a subagent holding all three lethal-trifecta legs. Safe on any repo, identical on every OS, no key.
The executing checks (only with your yes). Two questions can't be answered by reading: does each skill actually fire? (recall + precision) and do two skills fight over the same prompt? (the selection-collision matrix). Answering them runs the real model on your subscription, so audit asks once at a terminal and remembers your choice in .vigilesrc.json (audit.measure). Headless — CI, an agent, --json — it stays a read and prints a one-line nudge. It never hangs and never executes on its own.

So if audit graded a skill-heavy repo a clean A and "found little," it almost certainly ran only the deterministic read. The behavioral findings — the skills that never fire, the ones that collide — live behind that one consent. Say yes at a terminal, or measure in a script with measureTriggerRate, to see them.

audit is a local report, like Lighthouse — not a CI gate. For CI, use vigiles lint, which is deterministic and gates real breakage. The full author workflow is in shipping plugins.

Does it work on a non-JavaScript repo (Python, Rust, Go…)?

✅ Yes for Lint — npx vigiles lint verifies your CLAUDE.md against your repo and linters (Ruff, Clippy, Pylint, RuboCop, golangci-lint, detekt, … alongside ESLint — 11 in all, covering Python, Rust, Go, Kotlin, Java, Ruby, and CSS) with no install.

✅ Yes for the typed spec too — npx vigiles compile needs no package.json and no npm install. The CLI resolves a spec's vigiles/spec import from its own install, so init → compile → lint works end to end on a repo that has no Node toolchain. Prefer no TypeScript at all? Inline-comment mode is equally supported.

Is it safe to run `init`? What if I change my mind?

Safe. init never overwrites instruction files (see above) — an adopted file is left byte-for-byte alone until you run compile yourself and review the diff.

To back out: vigiles eject <file> returns the file to plain markdown and removes the spec. Uninstalling the dev dependency leaves your repo as plain markdown plus CI you can delete. Nothing is locked in.

How is the Eval layer different from promptfoo / DeepEval?

Two things set it apart.

Cost: promptfoo / DeepEval bill per token against a metered API on every run. vigiles answers most questions with no model at all and runs the rest on your subscription.

What's under test: vigiles loads your harness exactly as it ships — the real agent + your real CLAUDE.md + hooks. It A/Bs it on real tasks and reports cost + the target metric + a correctness check. A "saved tokens" win that breaks the code isn't counted as a win. See measuring skills.

8.8 KiB Raw Permalink Blame History

FAQ