Files
zernie__vigiles/.github/workflows/ci.yml
zernie b7458a6526 fix(ci): remove the root↔site cross-reads the path filter assumes away, and check the assumption (#223)
* fix(ci): delete the root test that read site/ instead of patching around it

#219 was a site-only PR that deleted a file a ROOT test read off disk. The
`changes` filter set root=false, the root jobs were SKIPPED, the PR merged GREEN
and main broke on an ENOENT — a skipped job and a passing job are the same tick
in the checks list, so nothing announced it.

That was patched twice before anyone asked whether the test should exist. It
should not. What it asserted:

    expect(hero).toMatch(/\bLANGUAGES\b[^;]*from ["']@\/lib\/linters["']/s);
    expect(heroCode).not.toMatch(/Any language/);

Both are properties of the SITE — an import line and a piece of copy — checked by
regexing the SOURCE TEXT of two site files from a test about the engine's linter
registry. Three defects, only the third visible: it asserts the shape of the text
rather than the value, so a rename or a reformat breaks it; a substring search
cannot tell an assertion from a comment discussing one, and it had already fired
on exactly that (its own comment says so); and it made root depend on site, which
is the dependency that produced the green-merge-red-main above.

Deleted. The property moves to where it belongs and gets stronger on the way:
site/src/lib/linters.browser.test.ts now RECOMPUTES the derivation from the
engine and compares values —

    expect(LINTER_NAMES).toEqual(BUILTIN_LINTERS.map((l) => LINTER_LABELS[l] ?? l));
    expect(LANGUAGES).toEqual(LANGUAGE_ORDER.filter((lang) => …));

so a hand-typed array is no longer forbidden, it is made HARMLESS: it must equal
what the engine implies and goes red the moment BUILTIN_LINTERS moves under it.
That test runs in the `site` job, which is gated on site changes — the job that
would have run for #219.

With the read gone, the filter's own invariant is true again, and now it is
CHECKED rather than asserted in a comment: src/ci-path-filter.test.ts fails if
any root test reads site/ off disk. Mutation-proven — reinstating one read fails
exactly that test.

⚠️ Two of my own errors on the way here, both mine and both worth the record.
The first fix I wrote for this declared the cross-read in ci.yml instead of
removing it — treating the symptom a third time. The second was in the guard
itself: `grep -rl` exits 1 when it finds nothing and execFileSync throws on a
non-zero exit, so the check went RED precisely when the property HELD. It now
reads the exit code instead of letting it decide.

Gates: npm run check 18/18. npm run coverage 3959 passed, 0 failed; the 99.94%
statement gate is src/mock-model.ts:147,339,374, untouched here and green on CI
(25 tests skip locally and never reach those lines).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(site): assert the derivation's properties, not a copy of its formula

The two tests added in 8f790e5 compared each exported value to a retyped
copy of its own definition:

  export const LINTER_NAMES = BUILTIN_LINTERS.map((l) => LINTER_LABELS[l] ?? l);
  expect(LINTER_NAMES).toEqual(BUILTIN_LINTERS.map((l) => LINTER_LABELS[l] ?? l));

That passes by construction, and anyone editing the formula edits the copy
with it — the same defect the header of src/ci-path-filter.test.ts names:
a test that agrees with its own copy of the rule proves nothing about the
rule that runs.

Replaced with properties that survive a refactor of the mapping and still
fail on a wrong value:

  - every shipped linter is named exactly once, under its label
    (containment + count => the exact set; order deliberately not asserted)
  - LANGUAGES is EXACTLY the covered language set, not merely a subset of
    the display order — which is all the pre-existing test could see

Net new coverage over what was already there: the CONTENTS of LINTER_NAMES
(only its length was checked) and the reverse inclusion for LANGUAGES.

site: 51 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(ci): close the second site cross-read, and the guard that could not see it

Codex flagged that the guard added in 8f790e5 does not enforce its advertised
invariant: `src/comparison-snapshot.test.ts` reads
`site/src/comparison/validate-overlap.json` off disk and the pattern missed it.
Verified — the file exists, the read is real, and the grep found nothing.

TWO independent reasons it evaded, either enough on its own:

  1. `grep` is LINE-based and prettier had split the call, so `resolve(` and
     the literal are never on one line:

         const SNAPSHOT = resolve(
           __dirname,
           "..",
           "site/src/comparison/validate-overlap.json",
         );

  2. the pattern encoded ONE SPELLING (`"../../site/`) — the path assembled
     from separate segments did not match it.

Both are the same defect: a guard written from the spelling its author had
just deleted rather than from the shape it forbids.

The data moves instead of the dependency being declared. The snapshot is
WRITTEN by a root tool (tools/measure-validate-overlap.mjs) and ASSERTED by a
root test, so it now lives at the root; the site imports it through a new
`@measured/validate-overlap.json` alias. The arrow points site -> root, which
a site-only diff cannot break.

The guard is Node, not grep: it matches the whole CALL across line breaks, so
neither reason above can recur. It keys on a `site/` segment inside a string
literal passed to resolve/join/read*, NOT on the mere mention — measured, not
assumed: forbidding the mention fires on 3 legitimate fixtures today (the
classifier is FED "site/src/App.tsx", it does not read it), and a guard with
false positives gets switched off, which is how the first hole survived.

| | |
| --- | --- |
| root reads under site/ | 1 -> 0 |
| forms the guard sees | one spelling, single-line -> any call, any layout |
| new fixture | the exact multi-line call the old guard missed |

Mutation-proven: restoring the site/ path in comparison-snapshot.test.ts (patch
landed, grep count 1) fails exactly `finds no disk read of site/ anywhere under
src/`, and reverting restores green.

Two of my own errors on the way, both caught by a run rather than by reading:
dropping the `execFileSync` import broke `decide()` six tests above, and the
first `SITE_LITERAL` anchored `^` to the start of the argument STRING instead
of the position after the quote, so it never matched a literal that was not
first in the call — its own fixture caught that one.

Gates: npm run check 18/18 · coverage 3960 passed 0 failed · site build ok ·
site test:browser 51 passed. The 99.94% statement gate fails locally on
src/mock-model.ts (untouched; 25 tests skip locally), green on CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(site): cover the hero chip's CONSUMER, not just the value it renders

Codex: moving the language assertion into the site suite validated only the
exported array, so a hand-typed chip — or `Hero` dropping the import entirely —
now passes everything. Verified: NO site test renders `Hero`, and the root test
this PR deleted did cover the consumer (it checked the import line and the
absence of the old "Any language" copy). The consumer half was genuinely
dropped; this restores it.

Restored on the RENDERED OUTPUT rather than the source text, which is what the
deleted test got wrong: it regexed `Hero.tsx` for an import line, so a rename or
a reformat broke it and a substring could not tell an assertion from a comment
discussing one — on top of making a root job read a site file (#219).

`Hero` is NOT rendered whole, and that is measured, not squeamishness: it mounts
`DemoAudit`, which on mount sweeps idb and issues one `fetchStars` request per
featured chip. A test that reaches GitHub from a browser runner is a flake, and
a flaky test is a disabled test. The chip is its own `HeroChip` component
instead — the same move `Guard.tsx` already makes by exporting `BATTERY_ROWS`,
for the same reason.

Three assertions, each covering a defect the others miss:

  - every language the engine covers is rendered (with a non-empty guard, since
    a vacuous loop proves nothing)
  - the chip contains the derivation VERBATIM — containment alone passes a chip
    naming one extra language, or the right ones in a stale order
  - "Any language" never returns

Mutation-proven: replacing `LANGUAGES.join(" · ")` with the old hand-typed
"Any language · 11 linter catalogs" (patch landed, grep count 1) fails all
three; reverting restores 54 passed.

Gates: npm run check 18/18 · site build ok · site test:browser 54 passed
(was 51).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(ci): derive the guard's scan roots from vitest.config, not from memory

Codex, third finding in this same file: the scan walked `src/` only, while the
root `unit` project also runs `scripts/**/*.test.ts` and
`eslint-rules/**/*.test.ts`. A site read in either was invisible to the guard
AND skipped by a site-only diff — exactly the hole this PR closes, in the part
of the suite the guard could not see. Verified against vitest.config.mjs.

Three findings in one file is the signal to stop patching and fix the root: all
three were a SPELLING I remembered (one path form, then one directory list)
where the rule was sitting in a config file, readable. So the roots are now
EXTRACTED from the unit project's own `include` globs — the same move
`patternFor` already makes for the classifier's patterns in ci.yml, and for the
same reason: a copy of a rule cannot prove anything about the rule that runs,
and it rots on the next glob someone adds.

A restructured config now fails LOUD (a named error saying the guard can no
longer see what it must cover) rather than walking nothing, and every derived
root is checked to exist — a scan that quietly resolves to empty reports a
clean repo forever.

⚠️ Deliberately NOT asserting `roots.length > 1`: narrowing the unit project
back to `src/` alone is a legitimate change, and a guard that goes red on a
legitimate change is a guard someone deletes.

Mutation-proven twice, patch landed both times:
  - a `resolve(__dirname, "..", "site/…")` + readFileSync planted in `scripts/`
    fails the guard, which names that exact file (the old scan saw nothing)
  - renaming the config's `name: "unit"` fails with the named error rather than
    silently passing

Gates: npm run check 18/18 · coverage 3960 passed, 0 failed. The 99.94%
statement gate fails locally on src/mock-model.ts (untouched; 25 tests skip
locally), green on CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(ci): decide whether the call resolves under site/, instead of matching spellings

Codex, fourth finding, fourth SPELLING: `resolve(__dirname, "..", "site",
"src/foo")` — `site` as its own argument — produced no offender, because the
matcher required one literal to contain `site/`.

Four rounds, four spellings, one cause. Widening the pattern a fourth time is
patching a symptom of a check that looks at text which RESEMBLES a site path
rather than deciding where the call lands. So the literal segments of the call
are now JOINED IN ORDER and the joined path is judged:

  resolve(__dirname, "..", "site", "src/foo")  ->  ../site/src/foo   ✗
  resolve(__dirname, "../../site/src/x.ts")    ->  ../../site/src/x  ✗
  join(ROOT, "website", "index.html")          ->  website/index.html ✓ legal

However an author breaks the path up, the join puts it back together, so the
class is closed rather than one more member of it. The segment test is a PATH
SEGMENT (`(?:^|/)site(?:/|$)`), not a substring, so `website/` stays legal.

The single fixture became a TABLE of every spelling review has caught plus the
shapes that must stay legal — a guard that goes red on `["site/src/App.tsx"]`
(bare data the classifier is FED, not a read) is a guard someone deletes.

HONEST LIMIT, stated in the file: an argument that is not a literal
(`resolve(dir, name)`) is not decidable statically and is out of scope. It is a
narrow gap — a test names the fixture it reads.

Mutation-proven, patch landed: a file in `scripts/` using the newly-caught
`"site", "src/…"` form fails the guard and is named in the output; removing it
returns green.

Gates: npm run check 18/18 · coverage 3967 passed, 0 failed (was 3960; +7 table
rows). The 99.94% statement gate fails locally on src/mock-model.ts (untouched;
25 tests skip locally), green on CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(ci): cover every file the skipped jobs run, and imports as well as reads

Two more from Codex, both real, both about this guard's own completeness rather
than a new defect in the repo:

  1. it walked only what the vitest `unit` project lists, and only `.ts`. The
     root jobs also run `test/runners` under vitest (`.mjs`) and under Jest
     (`.cjs`) — a site read in either was invisible AND skipped by a site-only
     diff.
  2. it inspected CALLS only, so a static or dynamic import
     (`import snapshot from "../site/…json"`) depended on a site file without
     ever being an offender.

For (1) the scan now takes EVERY test glob out of vitest.config.mjs and jest's
testMatch, and uses BOTH halves of each: the leading directory says where to
walk, the extension says what counts as a file there. Hard-coding either is what
put the runner suites outside the scan — the third time in this file that a
value was remembered while the config that owns it sat unread.

For (2) module specifiers go through the SAME verdict function as call
arguments; only the channel differs.

Also fixed while widening: the glob regex spanned newlines, so `[^"]*` swallowed
a 14-line comment and produced one absurd "glob" whose directory did not exist.
The existing sanity assertion caught it — a directory that must exist and an
extension that must be non-empty, per glob, so a mis-parse fails loud instead of
walking nothing.

Mutation-proven, patch landed each time:
  - `test/runners/__mutantA.vitest.mjs` reading a site file -> named as an
    offender (the previous scan took no `.mjs` anywhere)
  - `scripts/__mutantB.test.ts` importing a site JSON -> named as an offender
    (no matching call anywhere in it)
  - removing both -> green

Gates: npm run check 18/18 · coverage 3967 passed, 0 failed. The 99.94%
statement gate fails locally on src/mock-model.ts (untouched; 25 tests skip
locally), green on CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* refactor(ci): parse the guard's inputs instead of pattern-matching their text

Five rounds of review, five spellings this guard did not match — a call split
across lines, `"site"` as its own argument, a static import, files whose
extension the walk never took, and one where `[^"]*` ran through newlines and
swallowed a 14-line comment as a "glob". Each fix was a wider pattern, i.e. the
sixth waiting to be found.

They are one defect, and this repo already names it:
`parse-structured-input-with-a-real-parser` — the rule that routed five markdown
detectors onto one markdown-it oracle for exactly this reason. The guard was
regexing TypeScript. `typescript` is already a runtime dependency and
`src/core/compile-generator.ts` already parses with it.

TWO inputs, two corrections:

  - THE CONFIGS ARE NOW READ, NOT PARSED. They are modules; importing them
    yields the include globs as real arrays, so there is nothing to
    pattern-match. `unknown` is narrowed once at that boundary (parse, don't
    validate) and typed values go inward.
  - THE SOURCES ARE NOW PARSED. A call is a CallExpression however it is
    formatted; an import is its own node kind, found by asking rather than by
    adding an alternation.

What the AST buys that no pattern could, now pinned as two table rows:

  a site path inside a COMMENT                  -> not a dependency
  a site path inside a STRING that is not a call -> not a dependency

Which is why the self-exclusion is GONE. The text version had to skip this file,
because its own fixture table looked like a violation to it.

Also dropped: the computed-vs-literal import specifier is deliberate — a literal
`.mjs` import is a tsc error here (TS7016, no declaration file), and shipping a
.d.ts for a config would be ceremony around a value we only want to read.

Mutation-proven, patch landed each time: a read planted in `scripts/` and a
static import planted in `test/runners/*.mjs` are each named as offenders;
removing both returns green.

Gates: npm run check 18/18 · coverage 3972 passed, 0 failed. The 99.94%
statement gate fails locally on src/mock-model.ts (untouched; 25 tests skip
locally), green on CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(ci): discover the vitest config's filename instead of spelling it

`vitest.config.mjs` is plain JS while `site/vitest.config.ts` is TypeScript.
Measured: that is an inconsistency, not a constraint — renaming the root config
to `.ts` typechecks and vitest picks it up unchanged. Settling it touches eight
prose references including the compiled CLAUDE.md and a skill, so it is a
separate change, NOT this one.

What this change does is remove the landmine that separate change would hit: the
guard no longer knows the extension. It finds `vitest.config.*` at the repo root
and throws by name if there is none.

Mutation-proven three ways: green on `.mjs`, green after renaming the config to
`.ts`, and a loud named failure with no config present.

Gates: npm run check 18/18 · coverage 3972 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(site): render the real Hero, not just the chip component

Codex: the consumer test rendered `HeroChip` directly, so deleting
`<HeroChip />` from `Hero` left every assertion green while the first screen
showed no languages at all. That is the exact gap the test was added to close,
reproduced one level down — the test proved the component works, not that the
page uses it.

`Hero` itself is rendered now, with `DemoAudit` stubbed. Stubbing the one heavy
child is what makes it deterministic: on mount `DemoAudit` sweeps idb and issues
one `fetchStars` request per featured chip, and a browser test that reaches
GitHub is a flake — a flaky test is a disabled test.

Mutation-proven, patch landed: replacing `<HeroChip />` with a comment fails all
three assertions; restoring it returns 54 passed. The earlier version stayed
green through that same mutation, which is the finding.

Gates: npm run check 18/18 · site build ok · site test:browser 54 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-09-10 04:40:39 +05:00

539 lines
27 KiB
YAML
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
name: CI
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
# ── WHICH JOBS THIS COMMIT ACTUALLY NEEDS ────────────────────────────────────
# Until 2026-08-19 this workflow had no path filter at all, so every PR paid for
# all five jobs whatever it touched. Measured on PR #167 (16 files: src/, docs/,
# CLAUDE.md — none under site/): the `site` job built the engine, downloaded a
# browser and ran the demo suite against code the diff could not reach. Twice, at
# ~15 minutes each, because the browser install wedged.
#
# A workflow-level `paths:` cannot express this — it decides whether to run the
# workflow AT ALL and says nothing about individual jobs. Hence this job: one API
# call, no checkout, ~10s, and an `if:` on the jobs whose inputs are disjoint from
# what changed.
#
# 🔴 THE RULE INSIDE: any doubt runs everything. An empty list, a failed call, a
# push whose `before` is all zeros (a new branch), a manual dispatch — all fall
# through to true. A SKIPPED job and a PASSED job look identical in the checks
# list, so the cost of a wrong skip is a green tick over work nobody did, while
# the cost of a wrong run is a few minutes.
changes:
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
root: ${{ steps.f.outputs.root }}
site: ${{ steps.f.outputs.site }}
steps:
- id: f
env:
GH_TOKEN: ${{ github.token }}
run: |
set -u
files=""
case "${{ github.event_name }}" in
pull_request)
files=$(gh api --paginate "repos/${{ github.repository }}/pulls/${{ github.event.number }}/files" \
--jq '.[].filename' 2>/dev/null || true) ;;
push)
before="${{ github.event.before }}"
if [ -n "$before" ] && [ "$before" != "0000000000000000000000000000000000000000" ]; then
files=$(gh api "repos/${{ github.repository }}/compare/$before...${{ github.sha }}" \
--jq '.files[].filename' 2>/dev/null || true)
fi ;;
esac
if [ -z "$files" ]; then
echo "root=true" >> "$GITHUB_OUTPUT"
echo "site=true" >> "$GITHUB_OUTPUT"
echo "no usable file list for ${{ github.event_name }} — running everything"
exit 0
fi
echo "$files" | sed 's/^/ /'
# `root` — everything except the site. False ONLY when every changed file
# is under site/, because nothing outside site/ reads anything inside it.
#
# 🔴 THAT SENTENCE WAS FALSE ONCE, AND IT COST A RED MAIN.
# src/core/linter-contract.test.ts read two files under site/ off disk;
# #219 was a site-only PR that deleted one, so these jobs were skipped,
# the PR merged GREEN and main broke on an ENOENT — a skipped job and a
# passing job are the same tick in the checks list, so nothing said so.
# The read is gone (that test moved to the site suite, where it belongs),
# and src/ci-path-filter.test.ts now FAILS if any root test reads site/
# again. The invariant above is checked, not asserted.
if echo "$files" | grep -qvE '^site/'; then root=true; else root=false; fi
# `site` — the demo. It imports the built engine through three aliases
# (@engine/scan-files, @engine/audit-report, @engine/spec), so any src/
# change can reach it and must run it. What CANNOT reach it: prose and
# agent config. Verified 2026-08-19 by grepping the three bundles the site
# actually loads — the lint rule table (`DEFAULT_RULES`, `frontmatter-valid`)
# appears in none of them, so a rules change is invisible to the demo.
# Its check pages are generated from site/src/checks/checks.ts, NOT from
# docs/, so docs cannot reach it through the page generator either.
if echo "$files" | grep -qvE '^(docs/|\.claude/|[^/]*\.md$)'; then site=true; else site=false; fi
echo "root=$root" >> "$GITHUB_OUTPUT"
echo "site=$site" >> "$GITHUB_OUTPUT"
[ "$root" = true ] || echo "site-only change — the root jobs are skipped"
[ "$site" = true ] || echo "prose/agent-config only — the site job is skipped"
test:
needs: changes
if: needs.changes.outputs.root == 'true'
runs-on: ubuntu-latest
# Bounded 2026-08-19. Every job in this repository was unbounded, so a hung one
# ran to GitHub's six-hour ceiling — the one failure mode a green checkmark
# cannot show you, because it does not fail, it bills. Sizes are ~3-4x the
# MEASURED duration on this repo's own runs, quoted per job so a later edit can
# see where the number came from instead of guessing again.
# measured 5m49s.
timeout-minutes: 20
env:
# JVM/Go linter pins for the gated real-binary catalog tests
# (src/core/linters.test.ts — describe.skipIf(!hasBinary(...))). Installing
# these makes those tests RUN in CI instead of silently skipping forever
# (a green-with-skips is a hidden gap — no-silent-skips). detekt is pinned
# to the latest STABLE (2.0 is still alpha).
DETEKT_VERSION: "1.23.8"
KTLINT_VERSION: "1.8.0"
CHECKSTYLE_VERSION: "13.8.0" # 13.x requires JDK >= 17
GOLANGCI_LINT_VERSION: "2.12.2"
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
cache: "npm"
- uses: actions/setup-python@v5
with:
python-version: "3.x"
- uses: ruby/setup-ruby@v1
with:
ruby-version: "3.3"
# One JDK serves detekt + ktlint + checkstyle. Temurin 21 is preinstalled
# on ubuntu-24.04 runners, so this is env setup (~seconds), not a download.
- uses: actions/setup-java@v4
with:
distribution: "temurin"
java-version: "21"
- name: Cache pip packages
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: pip-${{ runner.os }}-ruff-pylint
- name: Cache gem packages
uses: actions/cache@v4
with:
path: ~/.local/share/gem
key: gem-${{ runner.os }}-rubocop
- name: Cache JVM/Go linter tools
uses: actions/cache@v4
with:
path: ~/.vigiles-linter-tools
key: linter-tools-${{ runner.os }}-detekt${{ env.DETEKT_VERSION }}-ktlint${{ env.KTLINT_VERSION }}-checkstyle${{ env.CHECKSTYLE_VERSION }}-golangci${{ env.GOLANGCI_LINT_VERSION }}
- run: npm ci
- name: Install linter CLIs for tests
run: |
pip install ruff pylint
gem install rubocop
rustup component add clippy
- name: Install JVM/Go linter CLIs (detekt, ktlint, checkstyle, golangci-lint)
# Lets the gated real-binary catalog tests in src/core/linters.test.ts
# actually RUN instead of skipping. Each tool must land on PATH under the
# exact name its LINTERS adapter's `cliTool` expects (src/core/linters.ts):
# detekt, ktlint, checkstyle, golangci-lint. checkstyle ships only a fat jar,
# so it gets a 2-line `java -jar` shim; detekt's zip launcher is symlinked.
run: |
TOOLS="$HOME/.vigiles-linter-tools"
BIN="$TOOLS/bin"
mkdir -p "$BIN"
if [ ! -x "$BIN/detekt" ]; then
curl -sSfL -o /tmp/detekt-cli.zip \
"https://github.com/detekt/detekt/releases/download/v${DETEKT_VERSION}/detekt-cli-${DETEKT_VERSION}.zip"
unzip -q -o /tmp/detekt-cli.zip -d "$TOOLS"
launcher=$(find "$TOOLS/detekt-cli-${DETEKT_VERSION}/bin" -maxdepth 1 -type f ! -name '*.bat' | head -n1)
[ -n "$launcher" ] || { echo "detekt launcher not found in zip"; exit 1; }
chmod +x "$launcher" && ln -sf "$launcher" "$BIN/detekt"
curl -sSfL -o "$BIN/ktlint" \
"https://github.com/pinterest/ktlint/releases/download/${KTLINT_VERSION}/ktlint"
chmod +x "$BIN/ktlint"
curl -sSfL -o "$TOOLS/checkstyle-all.jar" \
"https://github.com/checkstyle/checkstyle/releases/download/checkstyle-${CHECKSTYLE_VERSION}/checkstyle-${CHECKSTYLE_VERSION}-all.jar"
printf '#!/usr/bin/env bash\nexec java -jar "%s" "$@"\n' "$TOOLS/checkstyle-all.jar" > "$BIN/checkstyle"
chmod +x "$BIN/checkstyle"
curl -sSfL "https://raw.githubusercontent.com/golangci/golangci-lint/v${GOLANGCI_LINT_VERSION}/install.sh" \
| sh -s -- -b "$BIN" "v${GOLANGCI_LINT_VERSION}"
fi
echo "$BIN" >> "$GITHUB_PATH"
- name: Verify linter tools resolve (no silent skips)
# hasBinary() gates on `which <name>`; fail FAST here rather than letting
# the gated catalog tests silently skip (no-silent-skips). If a download
# shape changes, this step goes red instead of the tests quietly passing.
run: |
for t in detekt ktlint checkstyle golangci-lint; do
command -v "$t" >/dev/null || { echo "MISSING on PATH: $t"; exit 1; }
done
detekt --version
ktlint --version
checkstyle --version || checkstyle -V
golangci-lint --version
- name: Install claude + codex CLIs + bubblewrap (deterministic harness + sandbox tests)
# claude and codex (no API key needed — both reached via a local mock-model
# tier) let the binary-gated tests actually RUN instead of skipping: claude
# for the Anthropic-Messages mock, codex for the OpenAI-Responses mock
# (src/adapters/codex/mock-model.test.ts drives real codex against it).
# bubblewrap lets the sandbox confinement test execute and prove egress is
# blocked, instead of skipping.
run: |
# Pin claude-code to the dialect-validated version — SINGLE SOURCE OF
# TRUTH: VALIDATED_CC_VERSION in src/dialect-drift.ts. Keeps the
# dialect-drift alarm deterministic (fires only on a deliberate bump,
# not every unpinned CC release on an unrelated PR) and the real-claude
# harness/eval tests reproducible.
CC=$(grep -oP -m1 'VALIDATED_CC_VERSION = "\K[0-9.]+' src/dialect-drift.ts)
[ -n "$CC" ] || { echo "could not read VALIDATED_CC_VERSION"; exit 1; }
echo "Pinning @anthropic-ai/claude-code@$CC"
# Codex is pinned for the SAME reason, and the reason stopped being
# hypothetical on 2026-09-03: `latest` was 0.153.1, whose TARBALL
# 404s while its registry metadata still resolves, so `npm i -g
# @openai/codex` failed for every job on every branch — a red CI
# nobody's diff caused. Reproduced: metadata `npm view` returns
# 0.153.1, GET .../codex-0.153.1.tgz is 404; 0.153.0 and older are
# 200. Leaving this unpinned beside a deliberately pinned
# claude-code let one upstream publish take the build down.
npm i -g "@anthropic-ai/claude-code@$CC" "@openai/codex@0.153.0"
sudo apt-get update && sudo apt-get install -y bubblewrap
- name: Allow unprivileged user namespaces (Ubuntu 24.04 AppArmor restriction)
# ubuntu-latest (24.04) ships `kernel.apparmor_restrict_unprivileged_userns=1`,
# which blocks unconfined bubblewrap from creating the namespaces it needs —
# so `bwrap --unshare-all` fails (EPERM) even though the binary is installed.
# Disable that restriction so the sandbox confinement tests actually RUN and
# PASS here (genuine verification), instead of `sandboxAvailable()` correctly
# detecting it can't confine and skipping them. Best-effort: `|| true` so a
# future runner without this knob doesn't fail the job (the probe then skips).
run: |
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 || true
# Sanity-check that bwrap can now actually namespace:
bwrap --unshare-all --ro-bind / / true && echo "bwrap confinement: OK" || echo "bwrap confinement: STILL BLOCKED (tests will skip)"
- name: Build
run: npm run build
# This is the whole job now, and it used to be about half of it. Measured
# on run 31631013102 (ubuntu-latest), `test` was CI's SOLE long pole at
# 554 s while check/site/e2e/harness each finished inside 2 min on their
# own runners and idled. Two of the four items in that 554 s were waste:
#
# 100 s setup (python/ruby/java/go linters, claude+codex+bubblewrap, ci, build)
# 208 s this step
# 182 s `npx vitest run && npx jest` — a SECOND full pass over the very
# suite this step had just run. `vitest run` runs all four vitest
# projects, `runners` (the matcher constraint) among them, so the
# coverage run above already covered it. Only jest was unique, and
# jest takes 1.4 s. Deleted; jest moved to `check`.
# 63 s 3× typecheck · lint · prettier — none of which need a single one
# of the language toolchains installed above. Moved to `check`,
# which finishes in ~90 s and then sits idle.
#
# Expect ~4.9 min here instead of 9.2, on the same one runner.
- name: Run tests + 100% coverage gate (harness-testing pillar)
run: npm run coverage
# ── Sharding: measured, and NOT worth it (2026-08-12) ───────────────────
# The obvious next move is `vitest --shard=i/3` across three runners. It
# was built and measured on this checkout before being dropped, so nobody
# has to re-derive it:
#
# WHY IT WAS DROPPED — it buys 10–20 s. Local wall (4 cores): unsharded
# 177 s; three shards 123 s / 76 s / 45 s. The long pole stays at 123 s
# because vitest shards by SHA-1 of the spec PATH — deterministic but
# duration-blind — so the split was 1.5× the ideal 81 s. Add the ~100 s
# toolchain setup, which no shard can skip (the gated linter-binary tests
# live in the unit project and must not silently skip), and a ~40 s serial
# merge job, and the lane lands at ~4.7 min against the ~4.9 min above.
# Cost: 9.2 → ~13 runner-minutes, and the required status check renames
# from `test` to `test (shard 1/3)` + a new job, which can silently block
# every PR on a branch-protection rule.
#
# HOW IT WOULD HAVE TO WORK, if the suite ever grows enough to revisit.
# The gate is the hard part: a shard only loads the files its own slice of
# tests imports, so per-shard coverage is partial BY CONSTRUCTION, and a
# gate left inside a shard has to be relaxed to pass — a relaxed gate that
# still prints green is the exact failure class this repo exists to catch.
# The shape that works: shards run
# vitest run --shard=i/3 --reporter=blob --coverage --coverage.reporter=none
# --coverage.thresholds.{lines,functions,statements,branches}=0
# and upload `.vitest-reports/*` (a DOTDIR — needs upload-artifact's
# `include-hidden-files: true` or the artifact is silently empty); a
# dependent job downloads them and runs `vitest --merge-reports --coverage`,
# where the thresholds in vitest.config.mjs apply once, over the union.
#
# THE THREE PROOFS IT STAYS HONEST, all run locally on this checkout:
# · FAITHFUL — merged 3 shards vs the unsharded run: same 19 files, 3509
# coverage hit-bits (statements, per-path branches, functions) compared
# element-by-element, 0 mismatches, identical summary. Compare hit BITS,
# not counts: counts legitimately differ, since each side sums over a
# different set of tests.
# · STILL BITES — with src/share-link.test.ts removed, all three shards
# exit 0 while the MERGED report shows share-link.ts at 0% and fails
# with three threshold errors. Untested files survive the split because
# each shard records the whole include-list, zero-filling what it never
# loaded (`generateCoverage` gets allTestsRun=true even under --shard).
# · FAILS CLOSED — merging a single blob reports 66.53% against the same
# 1775-statement denominator and errors, so a lost or empty artifact
# reddens the gate rather than shrinking it. That property, not a
# blob-count assertion, is what makes the design safe.
# Also measured: the merge job needs no `npm run build` (the v8 remap reads
# the TS sources, not dist/), and `--coverage.reporter=none` on a shard
# loses nothing from the blob — it only suppresses a partial table that
# reads like a coverage figure.
check:
runs-on: ubuntu-latest
# measured 2m36s.
timeout-minutes: 10
# pull-requests: write lets the dogfooded action post its sticky PR comment.
permissions:
contents: read
pull-requests: write
steps:
# fetch-depth: 0 so the dogfooded action's capability-diff can reach the
# PR base commit (it materializes it as a detached worktree).
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-node@v4
with:
node-version: "20"
- run: npm ci
- name: Build
run: npm run build
# Dogfood the shipped GitHub Action FIRST — it's the primary lint gate
# users actually rely on, so run the composite over the repo's own build
# (`version: local`) ahead of the redundant direct CLI call below. The
# action emits per-line `::error file=,line=::` annotations (rendered inline
# on the PR diff) and posts a sticky PR comment. See the prod-grade-gha-cli
# rule. Running it first means a lint failure surfaces THROUGH the action
# (comment + annotations), not a bare CLI step before it. `capability-diff`
# folds the blast-radius diff vs the PR base into the SAME sticky comment
# (informational — no fail-on-widen — so a legit widening doesn't red CI).
- name: Dogfood the vigiles GitHub Action (primary lint gate)
id: vigiles-action
uses: ./
with:
command: lint
version: local
capability-diff: "true"
- name: Assert the action reported success
run: |
got="${{ steps.vigiles-action.outputs.valid }}"
if [ "$got" != "true" ]; then
echo "expected the vigiles action output valid=true, got '$got'"
exit 1
fi
echo "vigiles action output: valid=$got"
# Dogfood the eval staleness gate via the composite Action. It now VERIFIES
# a real committed lock instead of no-opping: `paths` names the one eval
# that has one, which is what the gate's own error message prescribes for
# an eval without a lock ("exclude it from the --check run").
#
# 🔴 It was a NO-OP FOR ITS WHOLE LIFE, and nobody could tell. With no lock
# committed anywhere, `eval --check` returned early on `anyLocksCommitted`
# and printed "nothing to verify" — exit 0. The first commit of a lock got
# past that early return and the step failed twice over: once on a quota
# consent that should never gate a verify (fixed in run-scripts.ts), then
# on the 22 evals that are unnamed and unlocked. Both were invisible while
# the step never ran. Widen `paths` as evals earn locks; a bare `--check`
# here fails until every discovered eval has a name AND a committed lock.
#
# 🔴 Kept OUT of `npm run check` on purpose, and nearly lost by accident when
# the consolidation below was written: it sat between two command-steps, so a
# span-based edit swallowed it. Like the lint step above it, it exercises the
# ACTION WRAPPER, which has no shell equivalent.
- name: Dogfood the eval staleness gate (eval-check)
uses: ./
with:
command: eval-check
paths: examples/harness/skill-trigger-rate.eval.mjs
version: local
comment: "false"
# ONE list, and it lives in `scripts/check.mjs` — not restated here.
#
# On 2026-08-16 three PRs went to `main` red because the author ran a
# five-command list from memory while this job ran thirteen. The failing one
# (`tsc -p test/types/tsconfig.json`) sits in a separate tsconfig the root
# type-check cannot see, and three `@ts-expect-error` assertions had quietly
# stopped failing. A longer list to remember would not have fixed that; a
# single list that CI INVOKES does — the pre-commit / just / mise pattern.
#
# `npm run check` runs `build`, then the read-only checks in parallel, then
# the file-WRITING ones serially (a writer beside a reader is a race), and
# names whatever failed — which is what the per-step names here used to buy.
# `node scripts/check.mjs --list` prints the commands without running them.
#
# The two `uses: ./` steps above are NOT in it on purpose: they dogfood this
# repo's own composite Action, which has no shell equivalent.
- name: Check suite (identical to `npm run check` locally)
run: npm run check
harness:
# Dogfood the harness-testing pillar on the in-repo build: the deterministic
# tier runs the real claude CLI against a scripted mock model, so it needs
# no API key and costs nothing. The eval tier (real model) is run manually
# via `npm run test:eval`.
needs: changes
if: needs.changes.outputs.root == 'true'
runs-on: ubuntu-latest
# measured 45s.
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
cache: "npm"
- run: npm ci
- name: Install claude CLI (pinned to the dialect-validated version; no API key needed)
run: |
CC=$(grep -oP -m1 'VALIDATED_CC_VERSION = "\K[0-9.]+' src/dialect-drift.ts)
[ -n "$CC" ] || { echo "could not read VALIDATED_CC_VERSION"; exit 1; }
npm i -g "@anthropic-ai/claude-code@$CC"
- name: Build
run: npm run build
- name: Run deterministic harness tests
# --min is a FLOOR, and it is the whole point of this line: without it a
# broken glob (a rename, a moved directory, a changed extension) makes this
# job print "No **/*.harness.* files found." and exit 0 — green, having run
# nothing. Measured 2026-09-07: `vigiles test` on zero files exits 0.
# Adding harness files keeps passing; dropping below the floor fails.
run: node dist/cli.js test --min=14
# The in-browser "grade a repo" demo (site/) runs the SAME compiled engine the
# CLI runs (`@engine/*` → the built dist/), so its browser-mode vitest suite is a
# real gate: parity (pako === node:zlib on real Chromium), the DemoAudit
# interaction states, and the idb-keyval grade-cache (real IndexedDB). No API key,
# deterministic. This closes the gap where demo/engine-parity regressions were
# only caught locally. The Playwright e2e (`test:e2e`) runs here too — it serves
# the REAL built bundle and drives it, incl. the 390px mobile-overflow gate that
# catches layout bleed a component-level test can't (no viewport in @vitest/browser).
site:
needs: changes
if: needs.changes.outputs.site == 'true'
runs-on: ubuntu-latest
# measured 3m35s.
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
cache: "npm"
- run: npm ci
# The site imports the built root dist/ via the @engine/* aliases (the
# literally-same compiled engine), so it must exist before the browser tests.
- name: Build engine (root dist/)
run: npm run build
# 🔴 THIS STEP ATE THE WHOLE JOB TWICE (2026-08-19, runs 32300064395 and
# 32303343213), and neither failure was in the diff under review. Two separate
# causes, fixed separately:
#
# 1. `--with-deps` shells out to `apt-get update`, which hung on
# azure.archive.ubuntu.com (`Ign:` on every line, then silence for 12
# minutes until the job bond killed it). `ubuntu-latest` already ships the
# system libraries Chromium needs — `--with-deps` exists for bare
# containers. Dropping it removes the apt call, and with it the only part
# of this step that touches a distro mirror. If a library really is
# missing, the browser fails to LAUNCH in the next step, in seconds, with
# the missing .so named — a loud failure instead of a silent 15-minute hang.
# 2. Chromium (~150 MB) was re-downloaded on every run with no cache, so the
# step's floor was a full download even on a good day.
#
# The step bond is the third leg: a hang now costs 5 minutes, not the job. The
# job bond cannot distinguish "this step wedged" from "the suite is slow".
- name: Cache the Playwright browser
id: pw-cache
uses: actions/cache@v4
with:
path: ~/.cache/ms-playwright
# Keyed on the resolved Playwright version: a bump must miss the cache,
# because the browser build is pinned to it.
key: pw-${{ runner.os }}-${{ hashFiles('site/package-lock.json', 'package-lock.json') }}
- name: Install Chromium for Playwright (vitest browser mode)
if: steps.pw-cache.outputs.cache-hit != 'true'
timeout-minutes: 5
run: npx playwright install chromium
working-directory: site
- name: Run site browser tests (parity + interaction + grade-cache)
run: npm run test:browser
working-directory: site
# Playwright e2e over the real built bundle (serves dist/, network mocked
# per-test) — the in-frame report/empty states AND the 390px mobile-overflow
# gate. Chromium is already installed above; the webServer builds engine+site.
- name: Run site e2e (Playwright — incl. mobile-overflow gate)
run: npm run test:e2e
working-directory: site
# Dogfood the per-level CI action for the e2e tier (allowlisted REAL egress).
# The egress connector needs /dev/net/tun + privilege to open the tap and
# configure the netns — the non-root host runner can't, which is why slirp4netns
# "fails" there (only `lo`). So this job runs as ROOT in a privileged container,
# where slirp4netns attaches exactly as it does on a root dev box and the egress
# tests run FOR REAL. Tests still self-skip via egressRoutes() if egress can't
# route, so the job never fails red. See research/egress-sandbox-tooling.md.
e2e:
needs: changes
if: needs.changes.outputs.root == 'true'
runs-on: ubuntu-latest
# measured 1m16s.
timeout-minutes: 10
container:
image: node:20-bookworm
options: --privileged --device /dev/net/tun
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/harness-tier
with:
tier: e2e