mirror of https://github.com/zernie/vigiles.git synced 2026-09-14 20:53:57 +08:00

Files

T

zernie 7afa00f00a fix(refs): 17 false "file is missing" accusations, none of them true (#157 )

* fix(scan): the reference extractor accused files that were present

Three false-accusation defects found by a 32-repo OSS dogfood sweep, all the
same shape — vigiles telling a maintainer a file they have is missing. For a
tool whose pitch is deterministic precision, a false positive costs more than a
miss: it is the failure mode that gets the tool uninstalled.

Measured on the corpus (91 audit units): 74 reference findings before, 57
after. All 17 removed are verified false; zero true positives lost, zero new
findings anywhere. Eight units change grade, every one upward — rsmdt/the-startup
F(54)→C(70), ayghri/i-have-adhd B(85)→A(100), fcakyon fable-advisor B(82)→A(90),
jjanczur/tyran D(64)→C(72), and Truthfulness 47→100 / 25→100 on two Microsoft
plugins.

One extractor was being run over five grammars. It now reads three, each through
that grammar's parser, and deliberately stops reading the other two:

  markdown  markdownRefs() in core/markdown.ts — a link's DESTINATION, and code
            spans that are NOT inside link text. The old line-scanner reported a
            link's LABEL as a missing resource while its destination resolved.
  shell     commandWords() in core/bash-effects.ts — the words a shell would
            resolve, with inline program text (`node -e`, `python -c`) removed
            and a nested `sh -c` parsed rather than discarded. The old regex cut
            a character run out of a JavaScript payload and reported it as the
            hook's script name.
  source    core/source-refs.ts — the one place a path-token pattern is built,
            with both boundaries un-omittable. `js` could claim the head of
            `json`; `(?:agents)/` could claim the tail of `claude-agents/`. Both
            were live.

  NOT READ: frontmatter (already stripped) and full-line comments in every
  language, not only shell — both remaining corpus false positives here came out
  of JSDoc.

Subtracted: four hand-written extension alternations (two of them duplicated
across the disk detector and its browser twin), a hand-rolled markdown link
regex, a hand-rolled inline-code regex, a whitespace split standing in for shell
words, and `stripShellComments`.

No new CLI verb, no new flag, no public API change (`api:check`: no drift).

Tests: 10 mutations, each proven applied by grep before running and each killed
by its own assertion, across the whole relevant suite. Every new case carries
both halves — fires on the planted defect, silent on the clean shape beside it.

* fix(scan): drop the redundant start-of-content guard in isPluginRooted

`startsAtSeparator` already answers idx <= 0 with true, and the next line
reaches it for idx === 0 (content[-1] is undefined, so never "/"). The guard
was a second copy of a boundary rule this module deliberately does not own —
the same duplication the module comment warns about — and it was the one
statement the 100% gate could not reach.

Also regenerates .vigiles/generated.d.ts for the new source files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd

---------

Co-authored-by: Claude <noreply@anthropic.com>

2026-08-18 20:13:34 +05:00

6.1 KiB

Raw Permalink Blame History

skill-resource-resolves

Flag a SKILL.md body that references a bundled file which doesn't exist on disk under the skill directory. A SKILL.md routinely points the agent at local files shipped beside it — scripts/run.sh, references/api.md, an inline `scripts/setup.py`. When the referenced file is missing, the agent reads the instruction, gets nothing, and silently continues — a documented top skill pain. This is the cross-reference moat applied to the SKILL.md body. Same detector vigiles audit uses (skillResourceIssues in src/core/skill-resources.ts).

What it flags

A body reference that is unambiguously a local bundled resource but doesn't resolve on disk:

✗ pdf-extract: bundled resource "scripts/extract.py" (line 14) is referenced but
  missing — the agent reads the instruction and gets nothing.

It matches two shapes:

a markdown link [setup](./scripts/run.sh) whose target is a relative path with a file extension, and
an inline-code path `scripts/foo.sh` that is prefixed by a standard bundle dir (scripts/, references/, assets/), has an extension, and sits in a line that directs the agent to use it (see below).

High-precision (FP-safe)

By the same don't-cry-wolf discipline as the loader's danglingRefs, it flags only what is clearly a local resource and skips everything ambiguous:

URLs (http://…, mailto:), absolute paths (/etc/x, C:\…),
${VAR} / $VAR tokens (plugin-root / runtime paths, uncheckable),
../ escapes out of the skill dir (a sibling skill or the repo),
extension-less mentions (a bare word or a directory name is undecidable prose),
and a generic inline `config.json` / `src/foo.ts` API mention (not a bundle-dir path).

An inline-code path gets one extra gate, because it is a weaker signal than a link. A skill that teaches how to build skills is full of bundle paths used as examples of what a skill could contain, not files it ships — "a `scripts/rotate_pdf.py` would be helpful to store", "Examples: `references/finance.md`", "- `references/patterns.md` — Common patterns". So a reference — an inline path or a markdown link — is treated as real only when the line directs the agent to use the file (read / run / see / load / …) and carries no illustrative cue (example, e.g., such as, would be, template, →). A markdown link is a higher-confidence shape than a bare backtick span, but an example link — "For example, see [a report](assets/example.pdf)" — is still a prose illustration, not a real dead ref, so the same prose gate applies to both.

A heading counts as a directive. The verb gate reads prose, and a heading is not prose — it is the section's label — so the most common way a skill points at its own script went unchecked:

## 🏗 START WITH THE MECHANICAL LEG — `scripts/structure.mjs`

Measured on a real skill whose structure.mjs sits at the skill root, not under scripts/: that line yielded nothing, while rewriting it to "Run the mechanical leg" — same file, same missing target — reported it correctly. The tool's answer depended on the author's choice of verb. An ATX heading naming a bundle path is now treated like an explicit "run …". The illustrative-cue veto still applies (## Examples: … stays skipped), and the wider net stops there: a mid-paragraph mention with no directive is still not checked, because flagging every backticked bundle path is what cries wolf on teaching skills.

A link's TEXT is not a reference. Candidates come from a markdown parse (markdown-it), not from regexes over each line, so the destination and the label can be told apart:

See [`references/api.md` § Lookups](../add-dataverse/references/api.md#lookups)

The destination is what the agent follows, and it resolves. The backtick span is its human-readable label. A line-oriented scan could not see that the span sat inside a link, and reported the label as a missing bundled resource — an accusation against a correct link (measured on microsoft/power-platform-skills and rohitg00/pro-workflow, 2026-08-17). A code span nested in a link's or an image's text is therefore not a candidate at all. For the same reason, link syntax that lives inside a code span is not a link: `- [ ] [Phase N](phase-N.md)` documents a checklist format, and phase-N.md is a metavariable that will never exist.

The detector prefers missing a real ref over a false positive — a noisy resource check would teach users to ignore it. A skill whose body is inherently full of illustrative bundle-path examples (a skill-authoring tutorial) can opt out entirely with a  marker.

Configuration

{ "rules": { "skill-resource-resolves": "warn" } }

Severity

Value	Behavior
`"error"`	`vigiles lint` exits non-zero (2) on a missing bundled resource
`"warn"` (default)	Prints a warning, exits 0 (don't-cry-wolf rollout)
`false`	Skip the check

Default is warn during rollout; raise to "error" to gate CI once you trust it on your own skills.

Scope

Skills (skills/*/SKILL.md) — every harness has skills, so this rule is not capability-gated. References resolve against the skill's own directory (resources ship beside the SKILL.md).

Why

A SKILL.md is freeform markdown, so a renamed or un-shipped scripts/ file rots silently — the model just doesn't get the resource it was told to use. vigiles already verifies file/script refs inside typed specs and intra-plugin hook refs in the loader; this extends that guarantee to the SKILL.md body.

6.1 KiB Raw Permalink Blame History