Files
zernie__vigiles/docs
zernie c4e75ae538 fix: read a space-separated tool list, so a working fence stops auditing as ineffective (#221)
* fix: read a space-separated tool list, so a working fence stops auditing as ineffective

Claude Code documents three equivalent spellings of `allowed-tools:` /
`disallowed-tools:` / a subagent's `tools:` — a comma-separated string, a
SPACE-separated string, and a YAML list. `splitList` in core/frontmatter-read.ts
split on `,` alone, so the space-separated form arrived as ONE token matching no
built-in name.

Reported with a reproduction by @vlad-ryzhkov in #217: on a 22-skill harness,
changing only the separator moved Safety 81 -> 80 -> 81, reversibly. Reproduced
here before changing anything — three of the seven documented shapes came back as
a single token: `WebFetch WebSearch`, `Bash(git add *) Bash(git commit *)`, and
`"Read Write Glob"` (a YAML-quoted scalar whose VALUE is a space-separated list,
which is what most skills in the wild write).

TWO SURFACES, AND THE SECOND IS WORSE THAN THE REPORTED ONE. #217 asks about a
skill's fence, where `allowed-tools:` only pre-approves, so a mis-split MISLEADS:
a fence Claude Code really enforces was reported as closing no lethal-trifecta
leg, naming as still-supplied the very tools the author had just denied. But a
SUBAGENT's `tools:` is the allowlist the PreToolUse rail denies against, and it
reads the same `splitList`. Measured on `tools: Read Grep Glob`, pre-fix:

  Read DENY · Grep DENY · Bash DENY

— the joined token matched nothing, so the agent was denied EVERY tool, including
the three it was explicitly granted. #217 raised this path as an open question
("I have not worked out whether anything downstream depends on that"); it did.

THE NAIVE FIX IS WORSE THAN THE BUG. `split(/[,\s]+/)` shreds `Bash(git push *)`
into `Bash(git`, `push`, `*)`, and `bashGrantIsUnbounded()` answers "unbounded"
when it cannot recognise a grant — so a BOUNDED grant would begin reading as an
unbounded one. That trades a false-clean verdict for a false-exposed one, which
is the worse side of the trade core/lethal-trifecta.ts already reasons about, for
a reason the author cannot see anywhere in their file.

So this is a tokenizer: whitespace separates only at paren depth 0. Quotes are
stripped per token AFTER splitting, never treated as a delimiter — the outer
quotes of `"Read Write Glob"` are YAML syntax, and treating them as token
boundaries would keep exactly this bug for every skill that quotes its list. An
unclosed `(` keeps the remainder as one token rather than fragmenting it into
garbage tool names.

The tokenizer is the one @vlad-ryzhkov proposed, with his six-row table plus
irregular spacing and an unbalanced paren as tests. Every new test is
mutation-proven with the patch verified landed:

  separator back to `,` only (pre-fix)          -> the 4 new tests fail
  whitespace splits at ANY depth (the naive fix) -> the parens test fails

One home, so the surfaces cannot drift: `frontmatterList` is the single reader
behind scan, adopt, and the rail.

Closes #217

Co-Authored-By: vlad-ryzhkov <211849729+vlad-ryzhkov@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: a parenthesis inside a quoted string is a character, not a bracket

Follow-up within #217, found by the Codex review bot on the first cut of the
tokenizer. The depth counter read every `(` as structure, so a grant carrying a
literal unmatched paren in its own command left depth at 1 after the grant
closed and swallowed everything after it:

  tools: Bash(printf '( %s' foo) Read
    ->  ["Bash(printf '( %s' foo) Read"]     one token

On a subagent that is the same failure this PR exists to fix: the rail denies
`Read`, a tool the author granted. Reproduced before changing anything.

Quote state now gates the DEPTH COUNTER ONLY, never the splitting. That
asymmetry is the fix, not an accident: if being inside quotes also suppressed
whitespace splitting, a quoted list would collapse back to one token and
reintroduce #217 itself for every skill that writes `allowed-tools: "Read Write
Glob"`. Backslash escapes are honoured outside single quotes, per shell rules.

MUTATION-PROVEN BOTH WAYS, each with the patch verified landed:

  depth counted inside quotes (the pre-fix state)  -> the new test fails
  quotes suppress splitting too (the over-fix)     -> the new test fails

🔴 AND THE SECOND MUTATION EXPOSED A DEFECT IN THE TEST FIRST. Its original
counterweight asserted the quoted list on a VALID YAML block and stayed GREEN
under the widening mutation — because js-yaml strips the quotes before
`splitList` ever runs, so no quote character was present and the mutation could
not bite. Measured, then moved onto a MALFORMED block, where the regex salvage
path hands the quotes through: that is the only path on which the property is
observable at all. A green mutation is a finding about the test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: vlad-ryzhkov <211849729+vlad-ryzhkov@users.noreply.github.com>
2026-09-09 21:37:34 +05:00
..

Documentation — index

How-to and reference docs for using vigiles. New here? Start with the README for the pitch and a 5-minute quick start, or try the live demo at vigiles.sh — grade any repo in your browser.

The docs are grouped by what you're trying to do:

  • Guides — step-by-step, task-first ("help me do X").
  • Reference — exact flags, symbols, rules.
  • Explanation — the reasoning and trade-offs.

Guides — "help me do X"

Verify your instruction files (the Lint layer)

  • verifying-instruction-files.md — the master guide: the markdown → typed-spec ladder, the three rule types (enforce / guidance / guard), verified references, and the before/after tables. Holds the full validation-rules matrix.
  • markdown-mode.md — the no-spec on-ramp: verify rules in plain markdown with inline <!-- vigiles:enforce --> comments, no TypeScript.
  • skills-monorepo.md — adopt vigiles in a CI-tested skill library or a plain .claude/ repo (no plugin.json).

Test & measure your harness (the Test + Eval layers)

  • harness-testing.md — task-first how-to: pick what you want to test (hook / safety-hook battery / wiring / skill firing / behaviour) and the tier that answers it, with a copy-paste first test and CI.
    • harness-testing-claude-code.md — Claude Code specifics: scriptModel, ${CLAUDE_PLUGIN_ROOT} / pluginDir / the Skill tool, the bubblewrap sandbox.
    • harness-testing-codex.md — Codex specifics: runHarnessTest({ adapter: codexAdapter }) against real codex exec, the Responses mock, what maps and what doesn't.
  • measuring-skills.md — A/B a skill, plugin, model, or rule change on real coding tasks: the metric triple (bill / target / blast-radius), the worked example, and why it's affordable on your subscription.
  • migrating-from-promptfoo.md — move existing skill evals onto the subscription: the concept + assertion mapping, a worked side-by-side, and the honest gaps.

Author & ship

  • skills.md — authoring a SKILL.md across the three on-ramps; the prose-vs-gates split.
  • compiled-hooks.md — author a hook as a pure typed function against the closed vigiles/hook vocabulary and compile it, making whole classes of hook bugs unrepresentable (false confidence, matcher bypass, capability creep).
  • railway-subagents.md — the typed Result subagent contract: declare a typed outcome, compose flat workers, and assert the outcome deterministically (no model judge).
  • for-plugin-authors.md — the plugin-author journey end to end: scan a draft, fix what it flags, make your skills fire, rank against a marketplace, gate it in CI.
  • github-action.md — run vigiles in CI: the composite Action, every input, the sticky PR comment, versioning.

Harnesses, adapters & agents

  • harnesses.md — which harness vigiles targets and how you pick one (by import), plus the capability matrix.
  • authoring-an-adapter.md — teach vigiles a new harness: the five ports, a worked skeleton, validating with the conformance kit.
  • non-js-harnesses.md — running vigiles on a harness whose project is Kotlin, Go or anything else non-JS: what needs Node and what does not.
  • agent-setup.md — agent setup & workflows in one guide: what init does, per-agent recipes (Claude Code / Codex / multi-agent / Cursor), non-interactive setup + fallback hooks, and CI.

Reference — "the exact flag, symbol, or rule"

  • cli.md — the full CLI: every verb and flag, the Claude Code plugin, lint vs audit.
  • commands-and-how-they-relate.md — the mental model: how audit / lint / test / eval / init fit together, and why measuring "do my skills fire?" uses init, not audit.
  • testing-matrix.md — every use case of the harness-testing API mapped to the tier that tests it, and what each tier costs.
  • testing-api.md — the full harness-testing API: every predicate, assertion, check, matcher, and option (measureTriggerRate / runEval / significance).
  • spec-format.md — the typed .spec.ts format (target, sections, rules, verified references) — the source of truth.
  • linter-support.md — the 11 linters + generate-types / generate-schema.
  • adapter-api.md — the adapter API reference: every port field, the conformance functions, the registry API.
  • Validation rules: the full matrix lives in the linting guide; each rule has a doc under rules/.
  • Library entry points (grouped by concern so a future harness can sit beside the current one):
    • vigiles/linting — the compiler + reference verification (compileClaude, compileSkill, …).
    • vigiles/spec — the spec builders (claude, enforce, guidance, file, cmd, symbol, …) and the module-augmentation target for generated types.
    • vigiles (the package root) — the free testing surface: the harness/hook runners, the check vocabulary and the runner-agnostic assertions.
    • vigiles/eval — everything that can spend money, every symbol prefixed paid_: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver.
    • defineEval is on the free root, not here, and that is the point: a *.eval.mjs file DESCRIBES its eval and vigiles eval runs it, so importing one cannot spend. See eval files describe their eval.
    • vigiles/claude-code, vigiles/codex — the per-harness adapters.
    • vigiles/adapter — the adapter-authoring kit.
  • API reference (generated) → — every exported symbol across all entry points, generated from the source. The hand-written guides here are the human-facing layer; this is the exhaustive symbol-level reference.

Explanation — "why it's built this way"

  • what-vigiles-catches.md — the taxonomy of problems vigiles handles: the prevented / caught / measured model, biggest-problem-first.
  • comparison.md — before/after tables, the determinism breakdown, the flow diagram, and what vigiles composes with rather than replaces.
  • sandboxing.md — what the sandbox isolates vs records (honestly): IO / rm -rf, the three network modes, tiers and limits.
  • emit-channel.md — experimental_emitTool: a skill returns a typed result by calling a tool instead of ending its turn, which is how an INLINE skill can carry an output contract at all. Includes the measurements, and when to prefer the stable output: instead.
  • experimental.md — what an experimental_ name promises (nothing), what is experimental today, and what has to be true before the prefix comes off.
  • faq.md — the front-door FAQ across all four layers.