mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
cf40a0f96c
* docs: clarify pillar decoupling and surface install in harness section
State the two-pillar decoupling plainly (neither pillar gates the other)
instead of framing it by which file each pillar doesn't need. Add a
concrete install command to the "Test your harness" setup block so it
anchors to `npx skills add zernie/vigiles` / `vigiles/testing` instead of
a vague "Install vigiles" prompt.
* chore: gitignore locally-installed dev skills
Ignore .agents/, .claude/skills/, and skills-lock.json so locally
installed skills (e.g. obra/superpowers) used for internal development
stay out of the repo and don't show as uncommitted changes.
* chore: vendor obra/superpowers skills for internal dev use
Add the superpowers skills under .agents/skills/ (with .claude/skills/
symlinks) and skills-lock.json for internal development. Verified via
`npm pack --dry-run` that none of these paths ship in the npm package or
the Claude Code plugin (.claude-plugin/plugin.json ships only ./skills/),
so they're part of the repo but not part of what vigiles ships.
Reverts the prior .gitignore exclusion of these paths.
* chore: enable superpowers via plugin marketplace instead of vendoring files
Replace the vendored obra/superpowers skills (.agents/, .claude/skills/,
skills-lock.json) with a project-scoped plugin declaration in
.claude/settings.json (extraKnownMarketplaces + enabledPlugins). Claude
Code fetches the plugin into its ephemeral cache on startup, so the repo
carries the enablement config, not 60+ third-party files.
* docs: add production-grade GHA + CLI as a tool requirement
Add the "prod-grade-gha-cli" guidance rule to the spec (compiled into
CLAUDE.md): the CLI and GitHub Action are first-class, production-grade
surfaces — CLI is the single source of truth with every Action input
mapped to a real flag, the Action is a composite over the published CLI
that works as uses: zernie/vigiles@v1, declares outputs via
$GITHUB_OUTPUT, ships a full documented workflow, and is dogfooded in CI.
* feat!: production-grade GitHub Action + CLI flag parity
Make the GitHub Action and CLI first-class, production-grade surfaces
(satisfies the new prod-grade-gha-cli rule).
CLI:
- Add --max-rules=<n> and --catalog-only flags (src/cli-flags.ts,
applyConfigFlags) so every Action input maps to a real CLI flag instead
of a config-file-only knob. Pure + unit-tested.
GitHub Action:
- Rewrite action.yml as a composite action over the published `npx
vigiles` CLI. The old node20 entry pointed at dist/action.js, which is
gitignored and never committed, so `uses: zernie/vigiles@<ref>` could
never actually run — only npm did. The composite reuses the same tested,
published artifact.
- Declare the `valid` output and set it via $GITHUB_OUTPUT (drops the
deprecated ::set-output). Add `version` (incl. `local` for dogfood) and
`working-directory` inputs.
- Remove the now-dead src/action.ts entry.
CI / release:
- Dogfood the Action in ci.yml via `uses: ./` with version: local and
assert the `valid` output.
- release.yml maintains a floating `v1` major tag so `uses:
zernie/vigiles@v1` tracks the latest release.
Docs/tests:
- Full Action reference in docs/cli.md (copy-paste workflow, permissions,
inputs, output, @v1-vs-@main versioning) + README pointer.
- Structural test (src/action-yml.test.ts) locks in the composite/outputs/
no-set-output contract.
BREAKING CHANGE: the GitHub Action is now a composite action and no longer
ships dist/action.js. Inputs/outputs are unchanged in name where they
overlap (command, paths, max-rules, catalog-only) but the action now adds
`version`/`working-directory` inputs and a `valid` output. Pin to
`zernie/vigiles@v1` going forward.
* feat: PR comments + job summary output, JS/TS harness scripts
GitHub Action output:
- Post an automatic STICKY PR comment on pull_request events (found by a
hidden marker and updated in place, never duplicated) + always write a
job summary ($GITHUB_STEP_SUMMARY), alongside the existing inline
annotations. New `comment` / `github-token` inputs; best-effort so a
fork PR without write access degrades to a warning. Dogfooded in ci.yml
(check job granted pull-requests: write).
Dual-language harness/eval scripts:
- `vigiles test` / `vigiles eval` now discover and run scripts authored in
JavaScript (.mjs/.cjs/.js) OR TypeScript (.ts/.mts/.cts). A TS script
runs through tsx when installed, else Node's native type stripping
(>=22.6), with an actionable error otherwise (src/adapters/claude-code/
run-scripts.ts: interpreterArgs/detectNodeCaps/scriptGlob).
Requirements + docs:
- Extend the prod-grade-gha-cli rule (three output channels) and add the
dual-language-tests rule to the spec (compiled into CLAUDE.md).
- Document the comment behavior + pull-requests: write in docs/cli.md;
rewrite the JS-vs-TS section of docs/testing-matrix.md.
Our own unit suite remains 100% TypeScript (every src/**/*.test.ts);
examples/harness/*.mjs stay JS on purpose as zero-dep CLI demos.
* docs: slim README to a billboard landing page
Cut the README from ~207 to ~118 lines, moving it out of the "long but
neither pitch nor index" middle into a focused landing page (Knip/esbuild
shape), per a survey of dev-tool READMEs (Ruff/Biome/Vitest/ast-grep/…)
and standard-readme/Diátaxis guidance.
- Pitch once, up top: tagline + one "why" paragraph + the single
two-pillar table (was restated 4x across header/nav/table/TOC).
- Keep the two concrete "first taste" artifacts: the stale-CLAUDE.md
snippet and the runHook example.
- Collapse the CLI, Skills, and Related Tools sections + the duplicated
promptfoo rebuttal into one-line links (the detail already lives in
docs/). Delete the <details> table of contents.
- Drop the README section anchor from docs/verifying-instruction-files.md
(now points at the README top).
Verified: `vigiles audit` passes with zero orphan docs (every doc the
README no longer links is still indexed by docs/README.md or CLAUDE.md).
* docs: document how the test tiers run in CI
Capture the CI tier mapping in docs/testing-matrix.md (it previously
lived only in discussion): which tier runs in which ci.yml job, that CI
runs every tier except evals, that evals are manual ($/non-deterministic),
and the cheapest-first best practice + the per-tier test:unit/integration/
e2e scripts that enable the split.
* docs: recast both README pillars as Without/With with real output
Replace the prose-heavy pillar pitches with tight "Without vigiles / With
vigiles" comparisons that show ACTUAL vigiles output (captured from real
runs, not invented):
- Pillar ①: `vigiles audit` flagging a rule that "exists but is disabled
in eslint config" (the valid-≠-true differentiator), a passing rule, and
a typo with a did-you-mean suggestion.
- Pillar ②: the runHook assertion + a real `vigiles test` pass summary,
framed around a silently-broken --no-verify guard.
Shorter and more concrete; shows the payoff instead of describing it.
* feat: refs-hook nudges the agent to mark references on save
Close the authoring-time half of pillar 1: an automatic, in-loop nudge so
the agent expresses references as verifiable vigiles marks instead of
unverifiable prose. Ships in the plugin — end-users author nothing.
- New `unmarked-refs` rule (default "warn"). The PostToolUse refs-hook
(hooks/refs-nudge.sh → `vigiles refs-hook`) fires on every CLAUDE.md /
AGENTS.md / SKILL.md edit and, by severity: "warn" → injects a
non-blocking nudge into the agent's context (mark these refs); "error" →
blocks the edit (exit 2); false → off. Wired into .claude-plugin/
plugin.json so it's on by default after `vigiles init`.
- Refactor the detection into core: collectRefIssues (shared by the
`vigiles refs` CLI and the hook) + the pure refsHookAction severity map,
unit-tested.
- Honest scope: catches code-shaped backticked refs; plaintext prose is
undecidable (the floor). Hooks are Claude Code / Codex only; other
harnesses fall back to instructions + the `vigiles audit` CI backstop.
- Docs: the from-file-save flow in verifying-instruction-files.md, a new
rules/unmarked-refs.md, and the rule added to the cli.md / docs index.
* test: dogfood the refs-hook through the runHook tier
skills-dogfood.test.ts only proves the hook script exists; this drives the
actual built `vigiles refs-hook` via the repo's own runHook unit tier and
asserts it FIRES correctly — nudges (non-blocking, additionalContext) by
default, blocks (exit 2) under unmarked-refs:"error", stays silent when
off, and no-ops on a clean or non-instruction file. Closes the "test your
harness, don't trust it on vibes" gap for the new hook.
* fix: scope refs-hook nudge to linter-rule names (kill false positives)
Dogfooding the hook on the repo's own SKILL.md files exposed heavy
over-firing: skills/test-harness/SKILL.md got 23 nudges, all on API names
(`runHook`, `runEval`, …) that are documentation prose, not stale
references. The `isCodeShaped` heuristic flagged every backticked
camelCase/Pascal/SCREAMING identifier — far too noisy to ship at
warn-default.
Tighten to high-signal only: flag a slash-scoped name with NO file
extension (`eslint/no-console`, `@typescript-eslint/no-explicit-any`,
`boundaries/dependencies`) and nudge toward `enforce()` / inline
`<!-- vigiles:enforce -->`. Bare identifiers and file paths are no longer
flagged (paths re-introduce the noise — the same SKILL.md files cite
`docs/harness-testing.md`, `package.json`, …; symbol marking stays opt-in
via an explicit `vigiles:symbol`).
Re-dogfooded: test-harness 23→0, strengthen 2→1 (keeps the one real rule
`@typescript-eslint/no-unused-vars`), migrate-to-spec 0. Tests + docs
updated to the narrowed scope.
* test: real-claude dogfood proves the refs-hook fires in a session
Install-and-verify: with the claude CLI present, drive the deterministic
harness tier (real claude + scripted mock model, no API key) so the mock
Writes a CLAUDE.md naming `eslint/no-console`, the PostToolUse refs-hook
fires, and requestContains asserts the non-blocking nudge reached the
model's context. Closes the last gap — previously verified only through
the runHook unit tier. examples/harness/refs-nudge.harness.mjs runs in CI
via `npm run test:harness` (claude is installed there) and skips cleanly
without the binary.
* feat: vigiles init sets up both pillars, interactive-or-not, via the v1 Action
Make onboarding smooth and complete (the smooth-adoption rule):
- BOTH pillars by default. Pillar 1 (spec + types + compile) as before, PLUS
Pillar 2 — scaffold a starter vigiles.harness.mjs the user adapts to a
real hook. `--pillars=verify|test|both` scopes it.
- Interactive when a human's at a TTY (prompts: pillars / CI / plugin),
NON-INTERACTIVE for agents / CI / piped input or `--yes` — so "set up
vigiles" from a Claude Code / Codex prompt Just Works without hanging.
Pure plan logic in src/setup-plan.ts (parseSetupArgs/shouldPrompt/
resolvePlan), unit-tested; IO stays in cli.ts.
- CI wiring now creates a real `.github/workflows/vigiles.yml` using the
production `zernie/vigiles@v1` Action (verify job + a harness job when
Pillar 2 is on), instead of appending a raw `npx vigiles audit` line to
an existing workflow.
- New flags: --yes/-y, --pillars=, --no-plugin. Docs updated (cli.md init
reference, agent-setup.md agent flow, README quick start).
* test: CI-test the init command's new behaviors end-to-end
Spawn the real CLI in a temp project and assert the new init behaviors,
offline via --no-plugin (no network skills-add): default sets up both
pillars + a zernie/vigiles@v1 workflow (asserts the spec, vigiles.harness.mjs,
and the workflow YAML contents); --pillars=test scaffolds only the harness
(no spec/workflow); --pillars=verify only the spec. Also add --no-plugin to
the existing bare-init spawns so they stay offline and deterministic.
* docs: make the "agent installs + tests skills" flow first-class
Viewed from an agent handed "install vigiles and test my skills", the
README install path was clear but the test-skills path was muddy — the
only pillar-2 example was a hook, measureTriggerRate (the "does the skill
fire?" primitive) was never named, and the model-invocable test-harness
skill was buried as 1-of-8.
- README: name measureTriggerRate (recall + precision) in the tier line,
and surface that `init` installs the test-harness skill so "test my
skills" fires it.
- Add the `great-agent-flow` guidance rule: the agent-told-a-short-prompt
experience must be discoverable from the README front door and
frictionless (one non-interactive install, the skill fires the test,
the skill-testing path is high-signal not buried).
* fix: no blanket claude-gate; skips are loud, never a silent green
Two fixes for `vigiles test` / `vigiles eval`, behind a stated principle
(the new no-silent-skips rule):
1. Drop the blanket `claude`-gate in handleRunScripts. Unit-tier runHook
tests need no model, so they now ALWAYS run — previously the whole
command bailed if `claude` was absent, so a scaffolded vigiles.harness.mjs
+ `npx vigiles test` looked broken on a machine without claude.
2. Make skips LOUD. The runner classifies each script pass/skip/fail
(exit 0 / SKIP_EXIT_CODE 77 / else), tallies them separately ("1 passed,
1 skipped"), shows `⊘ SKIPPED`, and a skip never fails the run (anyFailed).
Add a `skip(reason)` helper (vigiles/testing) and convert the four
deterministic examples from `console.log(...); process.exit(0)` (a silent
fake ✓) to `skip()`. Up front, `vigiles test` notes when claude is absent.
Verified end-to-end with claude off PATH: the unit example runs+passes, the
deterministic example reports ⊘ SKIPPED, summary "1 passed, 1 skipped",
exit 0. Adds the no-silent-skips guidance rule + skip-aware runner tests.
* feat: vigiles test --no-skip (fail on skip) + README agent prompt
Answer "shouldn't a skip fail loudly?" — yes, where you assert the
capability is present. A skip passes by default (tiers need different
capabilities not present everywhere; failing-by-default would make the
command red in any partial env — this repo's harness job legitimately
skips the egress tier), but `vigiles test --no-skip` makes any skip a
failure for the CI job that asserts everything must run: a green-with-skips
is itself untested surface. The skip stays loud + tallied either way.
Extends the no-silent-skips rule; integration-tested.
Also (per the agent-flow ask): add a paste-into-Claude-Code/Codex prompt to
the README quick start — install + verify + write/run a harness test with
good defaults, asking about CI gating / real-model eval / --strict.
* fix: cover skip() under the 100% gate; flush skip/--no-skip docs
The no-silent-skips commit broke CI's coverage gate (step 13): skip() in
harness-assert.ts (a coverage-included file) calls process.exit, which is
unreachable in-process → dropped below 100%. Mark it with the repo's
`v8 ignore` convention (same as the spawn boundary); it's exercised via a
child process in cli.test.ts / run-scripts.test.ts. Verified locally with
`npm run coverage`: 100% stmts/funcs/lines, branches 92.43% (≥90 floor).
Docs flush (before merge): document the loud-skip / unit-runs-without-claude
/ `--no-skip` behavior in docs/testing-matrix.md and the test-harness skill's
"Run it" step, so the agent's authoring guide matches the runner.
---------
Co-authored-by: Claude <noreply@anthropic.com>
45 lines
1.3 KiB
TypeScript
45 lines
1.3 KiB
TypeScript
/**
|
|
* Command-line config overrides.
|
|
*
|
|
* The GitHub Action is a thin composite over this CLI, so every Action
|
|
* input must map to a real CLI flag (no config-file-only knobs the Action
|
|
* can't reach — see the `prod-grade-gha-cli` rule). This module turns the
|
|
* shared flags into `VigilesConfig` overrides on top of the loaded config.
|
|
*
|
|
* Pure and exported for unit testing.
|
|
*/
|
|
|
|
import type { VigilesConfig } from "./core/types.js";
|
|
|
|
const MAX_RULES = "--max-rules=";
|
|
|
|
/**
|
|
* Apply shared command-line overrides to a loaded config.
|
|
*
|
|
* --max-rules=<n> → config.maxRules (positive integer; ignored otherwise)
|
|
* --catalog-only → config.catalogOnly = true
|
|
*
|
|
* Unknown flags are left untouched (each command parses its own besides
|
|
* these). Returns a new object; the input is not mutated.
|
|
*/
|
|
export function applyConfigFlags(
|
|
config: VigilesConfig,
|
|
args: readonly string[],
|
|
): VigilesConfig {
|
|
const next: VigilesConfig = { ...config };
|
|
|
|
const maxRulesFlag = args.find((a) => a.startsWith(MAX_RULES));
|
|
if (maxRulesFlag) {
|
|
const n = Number(maxRulesFlag.slice(MAX_RULES.length));
|
|
if (Number.isInteger(n) && n > 0) {
|
|
next.maxRules = n;
|
|
}
|
|
}
|
|
|
|
if (args.includes("--catalog-only")) {
|
|
next.catalogOnly = true;
|
|
}
|
|
|
|
return next;
|
|
}
|