mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
b7458a6526
* fix(ci): delete the root test that read site/ instead of patching around it #219 was a site-only PR that deleted a file a ROOT test read off disk. The `changes` filter set root=false, the root jobs were SKIPPED, the PR merged GREEN and main broke on an ENOENT — a skipped job and a passing job are the same tick in the checks list, so nothing announced it. That was patched twice before anyone asked whether the test should exist. It should not. What it asserted: expect(hero).toMatch(/\bLANGUAGES\b[^;]*from ["']@\/lib\/linters["']/s); expect(heroCode).not.toMatch(/Any language/); Both are properties of the SITE — an import line and a piece of copy — checked by regexing the SOURCE TEXT of two site files from a test about the engine's linter registry. Three defects, only the third visible: it asserts the shape of the text rather than the value, so a rename or a reformat breaks it; a substring search cannot tell an assertion from a comment discussing one, and it had already fired on exactly that (its own comment says so); and it made root depend on site, which is the dependency that produced the green-merge-red-main above. Deleted. The property moves to where it belongs and gets stronger on the way: site/src/lib/linters.browser.test.ts now RECOMPUTES the derivation from the engine and compares values — expect(LINTER_NAMES).toEqual(BUILTIN_LINTERS.map((l) => LINTER_LABELS[l] ?? l)); expect(LANGUAGES).toEqual(LANGUAGE_ORDER.filter((lang) => …)); so a hand-typed array is no longer forbidden, it is made HARMLESS: it must equal what the engine implies and goes red the moment BUILTIN_LINTERS moves under it. That test runs in the `site` job, which is gated on site changes — the job that would have run for #219. With the read gone, the filter's own invariant is true again, and now it is CHECKED rather than asserted in a comment: src/ci-path-filter.test.ts fails if any root test reads site/ off disk. Mutation-proven — reinstating one read fails exactly that test. ⚠️ Two of my own errors on the way here, both mine and both worth the record. The first fix I wrote for this declared the cross-read in ci.yml instead of removing it — treating the symptom a third time. The second was in the guard itself: `grep -rl` exits 1 when it finds nothing and execFileSync throws on a non-zero exit, so the check went RED precisely when the property HELD. It now reads the exit code instead of letting it decide. Gates: npm run check 18/18. npm run coverage 3959 passed, 0 failed; the 99.94% statement gate is src/mock-model.ts:147,339,374, untouched here and green on CI (25 tests skip locally and never reach those lines). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(site): assert the derivation's properties, not a copy of its formula The two tests added in8f790e5compared each exported value to a retyped copy of its own definition: export const LINTER_NAMES = BUILTIN_LINTERS.map((l) => LINTER_LABELS[l] ?? l); expect(LINTER_NAMES).toEqual(BUILTIN_LINTERS.map((l) => LINTER_LABELS[l] ?? l)); That passes by construction, and anyone editing the formula edits the copy with it — the same defect the header of src/ci-path-filter.test.ts names: a test that agrees with its own copy of the rule proves nothing about the rule that runs. Replaced with properties that survive a refactor of the mapping and still fail on a wrong value: - every shipped linter is named exactly once, under its label (containment + count => the exact set; order deliberately not asserted) - LANGUAGES is EXACTLY the covered language set, not merely a subset of the display order — which is all the pre-existing test could see Net new coverage over what was already there: the CONTENTS of LINTER_NAMES (only its length was checked) and the reverse inclusion for LANGUAGES. site: 51 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(ci): close the second site cross-read, and the guard that could not see it Codex flagged that the guard added in8f790e5does not enforce its advertised invariant: `src/comparison-snapshot.test.ts` reads `site/src/comparison/validate-overlap.json` off disk and the pattern missed it. Verified — the file exists, the read is real, and the grep found nothing. TWO independent reasons it evaded, either enough on its own: 1. `grep` is LINE-based and prettier had split the call, so `resolve(` and the literal are never on one line: const SNAPSHOT = resolve( __dirname, "..", "site/src/comparison/validate-overlap.json", ); 2. the pattern encoded ONE SPELLING (`"../../site/`) — the path assembled from separate segments did not match it. Both are the same defect: a guard written from the spelling its author had just deleted rather than from the shape it forbids. The data moves instead of the dependency being declared. The snapshot is WRITTEN by a root tool (tools/measure-validate-overlap.mjs) and ASSERTED by a root test, so it now lives at the root; the site imports it through a new `@measured/validate-overlap.json` alias. The arrow points site -> root, which a site-only diff cannot break. The guard is Node, not grep: it matches the whole CALL across line breaks, so neither reason above can recur. It keys on a `site/` segment inside a string literal passed to resolve/join/read*, NOT on the mere mention — measured, not assumed: forbidding the mention fires on 3 legitimate fixtures today (the classifier is FED "site/src/App.tsx", it does not read it), and a guard with false positives gets switched off, which is how the first hole survived. | | | | --- | --- | | root reads under site/ | 1 -> 0 | | forms the guard sees | one spelling, single-line -> any call, any layout | | new fixture | the exact multi-line call the old guard missed | Mutation-proven: restoring the site/ path in comparison-snapshot.test.ts (patch landed, grep count 1) fails exactly `finds no disk read of site/ anywhere under src/`, and reverting restores green. Two of my own errors on the way, both caught by a run rather than by reading: dropping the `execFileSync` import broke `decide()` six tests above, and the first `SITE_LITERAL` anchored `^` to the start of the argument STRING instead of the position after the quote, so it never matched a literal that was not first in the call — its own fixture caught that one. Gates: npm run check 18/18 · coverage 3960 passed 0 failed · site build ok · site test:browser 51 passed. The 99.94% statement gate fails locally on src/mock-model.ts (untouched; 25 tests skip locally), green on CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(site): cover the hero chip's CONSUMER, not just the value it renders Codex: moving the language assertion into the site suite validated only the exported array, so a hand-typed chip — or `Hero` dropping the import entirely — now passes everything. Verified: NO site test renders `Hero`, and the root test this PR deleted did cover the consumer (it checked the import line and the absence of the old "Any language" copy). The consumer half was genuinely dropped; this restores it. Restored on the RENDERED OUTPUT rather than the source text, which is what the deleted test got wrong: it regexed `Hero.tsx` for an import line, so a rename or a reformat broke it and a substring could not tell an assertion from a comment discussing one — on top of making a root job read a site file (#219). `Hero` is NOT rendered whole, and that is measured, not squeamishness: it mounts `DemoAudit`, which on mount sweeps idb and issues one `fetchStars` request per featured chip. A test that reaches GitHub from a browser runner is a flake, and a flaky test is a disabled test. The chip is its own `HeroChip` component instead — the same move `Guard.tsx` already makes by exporting `BATTERY_ROWS`, for the same reason. Three assertions, each covering a defect the others miss: - every language the engine covers is rendered (with a non-empty guard, since a vacuous loop proves nothing) - the chip contains the derivation VERBATIM — containment alone passes a chip naming one extra language, or the right ones in a stale order - "Any language" never returns Mutation-proven: replacing `LANGUAGES.join(" · ")` with the old hand-typed "Any language · 11 linter catalogs" (patch landed, grep count 1) fails all three; reverting restores 54 passed. Gates: npm run check 18/18 · site build ok · site test:browser 54 passed (was 51). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(ci): derive the guard's scan roots from vitest.config, not from memory Codex, third finding in this same file: the scan walked `src/` only, while the root `unit` project also runs `scripts/**/*.test.ts` and `eslint-rules/**/*.test.ts`. A site read in either was invisible to the guard AND skipped by a site-only diff — exactly the hole this PR closes, in the part of the suite the guard could not see. Verified against vitest.config.mjs. Three findings in one file is the signal to stop patching and fix the root: all three were a SPELLING I remembered (one path form, then one directory list) where the rule was sitting in a config file, readable. So the roots are now EXTRACTED from the unit project's own `include` globs — the same move `patternFor` already makes for the classifier's patterns in ci.yml, and for the same reason: a copy of a rule cannot prove anything about the rule that runs, and it rots on the next glob someone adds. A restructured config now fails LOUD (a named error saying the guard can no longer see what it must cover) rather than walking nothing, and every derived root is checked to exist — a scan that quietly resolves to empty reports a clean repo forever. ⚠️ Deliberately NOT asserting `roots.length > 1`: narrowing the unit project back to `src/` alone is a legitimate change, and a guard that goes red on a legitimate change is a guard someone deletes. Mutation-proven twice, patch landed both times: - a `resolve(__dirname, "..", "site/…")` + readFileSync planted in `scripts/` fails the guard, which names that exact file (the old scan saw nothing) - renaming the config's `name: "unit"` fails with the named error rather than silently passing Gates: npm run check 18/18 · coverage 3960 passed, 0 failed. The 99.94% statement gate fails locally on src/mock-model.ts (untouched; 25 tests skip locally), green on CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(ci): decide whether the call resolves under site/, instead of matching spellings Codex, fourth finding, fourth SPELLING: `resolve(__dirname, "..", "site", "src/foo")` — `site` as its own argument — produced no offender, because the matcher required one literal to contain `site/`. Four rounds, four spellings, one cause. Widening the pattern a fourth time is patching a symptom of a check that looks at text which RESEMBLES a site path rather than deciding where the call lands. So the literal segments of the call are now JOINED IN ORDER and the joined path is judged: resolve(__dirname, "..", "site", "src/foo") -> ../site/src/foo ✗ resolve(__dirname, "../../site/src/x.ts") -> ../../site/src/x ✗ join(ROOT, "website", "index.html") -> website/index.html ✓ legal However an author breaks the path up, the join puts it back together, so the class is closed rather than one more member of it. The segment test is a PATH SEGMENT (`(?:^|/)site(?:/|$)`), not a substring, so `website/` stays legal. The single fixture became a TABLE of every spelling review has caught plus the shapes that must stay legal — a guard that goes red on `["site/src/App.tsx"]` (bare data the classifier is FED, not a read) is a guard someone deletes. HONEST LIMIT, stated in the file: an argument that is not a literal (`resolve(dir, name)`) is not decidable statically and is out of scope. It is a narrow gap — a test names the fixture it reads. Mutation-proven, patch landed: a file in `scripts/` using the newly-caught `"site", "src/…"` form fails the guard and is named in the output; removing it returns green. Gates: npm run check 18/18 · coverage 3967 passed, 0 failed (was 3960; +7 table rows). The 99.94% statement gate fails locally on src/mock-model.ts (untouched; 25 tests skip locally), green on CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(ci): cover every file the skipped jobs run, and imports as well as reads Two more from Codex, both real, both about this guard's own completeness rather than a new defect in the repo: 1. it walked only what the vitest `unit` project lists, and only `.ts`. The root jobs also run `test/runners` under vitest (`.mjs`) and under Jest (`.cjs`) — a site read in either was invisible AND skipped by a site-only diff. 2. it inspected CALLS only, so a static or dynamic import (`import snapshot from "../site/…json"`) depended on a site file without ever being an offender. For (1) the scan now takes EVERY test glob out of vitest.config.mjs and jest's testMatch, and uses BOTH halves of each: the leading directory says where to walk, the extension says what counts as a file there. Hard-coding either is what put the runner suites outside the scan — the third time in this file that a value was remembered while the config that owns it sat unread. For (2) module specifiers go through the SAME verdict function as call arguments; only the channel differs. Also fixed while widening: the glob regex spanned newlines, so `[^"]*` swallowed a 14-line comment and produced one absurd "glob" whose directory did not exist. The existing sanity assertion caught it — a directory that must exist and an extension that must be non-empty, per glob, so a mis-parse fails loud instead of walking nothing. Mutation-proven, patch landed each time: - `test/runners/__mutantA.vitest.mjs` reading a site file -> named as an offender (the previous scan took no `.mjs` anywhere) - `scripts/__mutantB.test.ts` importing a site JSON -> named as an offender (no matching call anywhere in it) - removing both -> green Gates: npm run check 18/18 · coverage 3967 passed, 0 failed. The 99.94% statement gate fails locally on src/mock-model.ts (untouched; 25 tests skip locally), green on CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(ci): parse the guard's inputs instead of pattern-matching their text Five rounds of review, five spellings this guard did not match — a call split across lines, `"site"` as its own argument, a static import, files whose extension the walk never took, and one where `[^"]*` ran through newlines and swallowed a 14-line comment as a "glob". Each fix was a wider pattern, i.e. the sixth waiting to be found. They are one defect, and this repo already names it: `parse-structured-input-with-a-real-parser` — the rule that routed five markdown detectors onto one markdown-it oracle for exactly this reason. The guard was regexing TypeScript. `typescript` is already a runtime dependency and `src/core/compile-generator.ts` already parses with it. TWO inputs, two corrections: - THE CONFIGS ARE NOW READ, NOT PARSED. They are modules; importing them yields the include globs as real arrays, so there is nothing to pattern-match. `unknown` is narrowed once at that boundary (parse, don't validate) and typed values go inward. - THE SOURCES ARE NOW PARSED. A call is a CallExpression however it is formatted; an import is its own node kind, found by asking rather than by adding an alternation. What the AST buys that no pattern could, now pinned as two table rows: a site path inside a COMMENT -> not a dependency a site path inside a STRING that is not a call -> not a dependency Which is why the self-exclusion is GONE. The text version had to skip this file, because its own fixture table looked like a violation to it. Also dropped: the computed-vs-literal import specifier is deliberate — a literal `.mjs` import is a tsc error here (TS7016, no declaration file), and shipping a .d.ts for a config would be ceremony around a value we only want to read. Mutation-proven, patch landed each time: a read planted in `scripts/` and a static import planted in `test/runners/*.mjs` are each named as offenders; removing both returns green. Gates: npm run check 18/18 · coverage 3972 passed, 0 failed. The 99.94% statement gate fails locally on src/mock-model.ts (untouched; 25 tests skip locally), green on CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(ci): discover the vitest config's filename instead of spelling it `vitest.config.mjs` is plain JS while `site/vitest.config.ts` is TypeScript. Measured: that is an inconsistency, not a constraint — renaming the root config to `.ts` typechecks and vitest picks it up unchanged. Settling it touches eight prose references including the compiled CLAUDE.md and a skill, so it is a separate change, NOT this one. What this change does is remove the landmine that separate change would hit: the guard no longer knows the extension. It finds `vitest.config.*` at the repo root and throws by name if there is none. Mutation-proven three ways: green on `.mjs`, green after renaming the config to `.ts`, and a loud named failure with no config present. Gates: npm run check 18/18 · coverage 3972 passed, 0 failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(site): render the real Hero, not just the chip component Codex: the consumer test rendered `HeroChip` directly, so deleting `<HeroChip />` from `Hero` left every assertion green while the first screen showed no languages at all. That is the exact gap the test was added to close, reproduced one level down — the test proved the component works, not that the page uses it. `Hero` itself is rendered now, with `DemoAudit` stubbed. Stubbing the one heavy child is what makes it deterministic: on mount `DemoAudit` sweeps idb and issues one `fetchStars` request per featured chip, and a browser test that reaches GitHub is a flake — a flaky test is a disabled test. Mutation-proven, patch landed: replacing `<HeroChip />` with a comment fails all three assertions; restoring it returns 54 passed. The earlier version stayed green through that same mutation, which is the finding. Gates: npm run check 18/18 · site build ok · site test:browser 54 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
539 lines
27 KiB
YAML
539 lines
27 KiB
YAML
name: CI
|
||
|
||
on:
|
||
push:
|
||
branches: [main]
|
||
pull_request:
|
||
branches: [main]
|
||
|
||
jobs:
|
||
# ── WHICH JOBS THIS COMMIT ACTUALLY NEEDS ────────────────────────────────────
|
||
# Until 2026-08-19 this workflow had no path filter at all, so every PR paid for
|
||
# all five jobs whatever it touched. Measured on PR #167 (16 files: src/, docs/,
|
||
# CLAUDE.md — none under site/): the `site` job built the engine, downloaded a
|
||
# browser and ran the demo suite against code the diff could not reach. Twice, at
|
||
# ~15 minutes each, because the browser install wedged.
|
||
#
|
||
# A workflow-level `paths:` cannot express this — it decides whether to run the
|
||
# workflow AT ALL and says nothing about individual jobs. Hence this job: one API
|
||
# call, no checkout, ~10s, and an `if:` on the jobs whose inputs are disjoint from
|
||
# what changed.
|
||
#
|
||
# 🔴 THE RULE INSIDE: any doubt runs everything. An empty list, a failed call, a
|
||
# push whose `before` is all zeros (a new branch), a manual dispatch — all fall
|
||
# through to true. A SKIPPED job and a PASSED job look identical in the checks
|
||
# list, so the cost of a wrong skip is a green tick over work nobody did, while
|
||
# the cost of a wrong run is a few minutes.
|
||
changes:
|
||
runs-on: ubuntu-latest
|
||
timeout-minutes: 5
|
||
outputs:
|
||
root: ${{ steps.f.outputs.root }}
|
||
site: ${{ steps.f.outputs.site }}
|
||
steps:
|
||
- id: f
|
||
env:
|
||
GH_TOKEN: ${{ github.token }}
|
||
run: |
|
||
set -u
|
||
files=""
|
||
case "${{ github.event_name }}" in
|
||
pull_request)
|
||
files=$(gh api --paginate "repos/${{ github.repository }}/pulls/${{ github.event.number }}/files" \
|
||
--jq '.[].filename' 2>/dev/null || true) ;;
|
||
push)
|
||
before="${{ github.event.before }}"
|
||
if [ -n "$before" ] && [ "$before" != "0000000000000000000000000000000000000000" ]; then
|
||
files=$(gh api "repos/${{ github.repository }}/compare/$before...${{ github.sha }}" \
|
||
--jq '.files[].filename' 2>/dev/null || true)
|
||
fi ;;
|
||
esac
|
||
if [ -z "$files" ]; then
|
||
echo "root=true" >> "$GITHUB_OUTPUT"
|
||
echo "site=true" >> "$GITHUB_OUTPUT"
|
||
echo "no usable file list for ${{ github.event_name }} — running everything"
|
||
exit 0
|
||
fi
|
||
echo "$files" | sed 's/^/ /'
|
||
|
||
# `root` — everything except the site. False ONLY when every changed file
|
||
# is under site/, because nothing outside site/ reads anything inside it.
|
||
#
|
||
# 🔴 THAT SENTENCE WAS FALSE ONCE, AND IT COST A RED MAIN.
|
||
# src/core/linter-contract.test.ts read two files under site/ off disk;
|
||
# #219 was a site-only PR that deleted one, so these jobs were skipped,
|
||
# the PR merged GREEN and main broke on an ENOENT — a skipped job and a
|
||
# passing job are the same tick in the checks list, so nothing said so.
|
||
# The read is gone (that test moved to the site suite, where it belongs),
|
||
# and src/ci-path-filter.test.ts now FAILS if any root test reads site/
|
||
# again. The invariant above is checked, not asserted.
|
||
if echo "$files" | grep -qvE '^site/'; then root=true; else root=false; fi
|
||
|
||
# `site` — the demo. It imports the built engine through three aliases
|
||
# (@engine/scan-files, @engine/audit-report, @engine/spec), so any src/
|
||
# change can reach it and must run it. What CANNOT reach it: prose and
|
||
# agent config. Verified 2026-08-19 by grepping the three bundles the site
|
||
# actually loads — the lint rule table (`DEFAULT_RULES`, `frontmatter-valid`)
|
||
# appears in none of them, so a rules change is invisible to the demo.
|
||
# Its check pages are generated from site/src/checks/checks.ts, NOT from
|
||
# docs/, so docs cannot reach it through the page generator either.
|
||
if echo "$files" | grep -qvE '^(docs/|\.claude/|[^/]*\.md$)'; then site=true; else site=false; fi
|
||
|
||
echo "root=$root" >> "$GITHUB_OUTPUT"
|
||
echo "site=$site" >> "$GITHUB_OUTPUT"
|
||
[ "$root" = true ] || echo "site-only change — the root jobs are skipped"
|
||
[ "$site" = true ] || echo "prose/agent-config only — the site job is skipped"
|
||
|
||
test:
|
||
needs: changes
|
||
if: needs.changes.outputs.root == 'true'
|
||
runs-on: ubuntu-latest
|
||
# Bounded 2026-08-19. Every job in this repository was unbounded, so a hung one
|
||
# ran to GitHub's six-hour ceiling — the one failure mode a green checkmark
|
||
# cannot show you, because it does not fail, it bills. Sizes are ~3-4x the
|
||
# MEASURED duration on this repo's own runs, quoted per job so a later edit can
|
||
# see where the number came from instead of guessing again.
|
||
# measured 5m49s.
|
||
timeout-minutes: 20
|
||
env:
|
||
# JVM/Go linter pins for the gated real-binary catalog tests
|
||
# (src/core/linters.test.ts — describe.skipIf(!hasBinary(...))). Installing
|
||
# these makes those tests RUN in CI instead of silently skipping forever
|
||
# (a green-with-skips is a hidden gap — no-silent-skips). detekt is pinned
|
||
# to the latest STABLE (2.0 is still alpha).
|
||
DETEKT_VERSION: "1.23.8"
|
||
KTLINT_VERSION: "1.8.0"
|
||
CHECKSTYLE_VERSION: "13.8.0" # 13.x requires JDK >= 17
|
||
GOLANGCI_LINT_VERSION: "2.12.2"
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
|
||
- uses: actions/setup-node@v4
|
||
with:
|
||
node-version: "20"
|
||
cache: "npm"
|
||
|
||
- uses: actions/setup-python@v5
|
||
with:
|
||
python-version: "3.x"
|
||
|
||
- uses: ruby/setup-ruby@v1
|
||
with:
|
||
ruby-version: "3.3"
|
||
|
||
# One JDK serves detekt + ktlint + checkstyle. Temurin 21 is preinstalled
|
||
# on ubuntu-24.04 runners, so this is env setup (~seconds), not a download.
|
||
- uses: actions/setup-java@v4
|
||
with:
|
||
distribution: "temurin"
|
||
java-version: "21"
|
||
|
||
- name: Cache pip packages
|
||
uses: actions/cache@v4
|
||
with:
|
||
path: ~/.cache/pip
|
||
key: pip-${{ runner.os }}-ruff-pylint
|
||
|
||
- name: Cache gem packages
|
||
uses: actions/cache@v4
|
||
with:
|
||
path: ~/.local/share/gem
|
||
key: gem-${{ runner.os }}-rubocop
|
||
|
||
- name: Cache JVM/Go linter tools
|
||
uses: actions/cache@v4
|
||
with:
|
||
path: ~/.vigiles-linter-tools
|
||
key: linter-tools-${{ runner.os }}-detekt${{ env.DETEKT_VERSION }}-ktlint${{ env.KTLINT_VERSION }}-checkstyle${{ env.CHECKSTYLE_VERSION }}-golangci${{ env.GOLANGCI_LINT_VERSION }}
|
||
|
||
- run: npm ci
|
||
|
||
- name: Install linter CLIs for tests
|
||
run: |
|
||
pip install ruff pylint
|
||
gem install rubocop
|
||
rustup component add clippy
|
||
|
||
- name: Install JVM/Go linter CLIs (detekt, ktlint, checkstyle, golangci-lint)
|
||
# Lets the gated real-binary catalog tests in src/core/linters.test.ts
|
||
# actually RUN instead of skipping. Each tool must land on PATH under the
|
||
# exact name its LINTERS adapter's `cliTool` expects (src/core/linters.ts):
|
||
# detekt, ktlint, checkstyle, golangci-lint. checkstyle ships only a fat jar,
|
||
# so it gets a 2-line `java -jar` shim; detekt's zip launcher is symlinked.
|
||
run: |
|
||
TOOLS="$HOME/.vigiles-linter-tools"
|
||
BIN="$TOOLS/bin"
|
||
mkdir -p "$BIN"
|
||
if [ ! -x "$BIN/detekt" ]; then
|
||
curl -sSfL -o /tmp/detekt-cli.zip \
|
||
"https://github.com/detekt/detekt/releases/download/v${DETEKT_VERSION}/detekt-cli-${DETEKT_VERSION}.zip"
|
||
unzip -q -o /tmp/detekt-cli.zip -d "$TOOLS"
|
||
launcher=$(find "$TOOLS/detekt-cli-${DETEKT_VERSION}/bin" -maxdepth 1 -type f ! -name '*.bat' | head -n1)
|
||
[ -n "$launcher" ] || { echo "detekt launcher not found in zip"; exit 1; }
|
||
chmod +x "$launcher" && ln -sf "$launcher" "$BIN/detekt"
|
||
|
||
curl -sSfL -o "$BIN/ktlint" \
|
||
"https://github.com/pinterest/ktlint/releases/download/${KTLINT_VERSION}/ktlint"
|
||
chmod +x "$BIN/ktlint"
|
||
|
||
curl -sSfL -o "$TOOLS/checkstyle-all.jar" \
|
||
"https://github.com/checkstyle/checkstyle/releases/download/checkstyle-${CHECKSTYLE_VERSION}/checkstyle-${CHECKSTYLE_VERSION}-all.jar"
|
||
printf '#!/usr/bin/env bash\nexec java -jar "%s" "$@"\n' "$TOOLS/checkstyle-all.jar" > "$BIN/checkstyle"
|
||
chmod +x "$BIN/checkstyle"
|
||
|
||
curl -sSfL "https://raw.githubusercontent.com/golangci/golangci-lint/v${GOLANGCI_LINT_VERSION}/install.sh" \
|
||
| sh -s -- -b "$BIN" "v${GOLANGCI_LINT_VERSION}"
|
||
fi
|
||
echo "$BIN" >> "$GITHUB_PATH"
|
||
|
||
- name: Verify linter tools resolve (no silent skips)
|
||
# hasBinary() gates on `which <name>`; fail FAST here rather than letting
|
||
# the gated catalog tests silently skip (no-silent-skips). If a download
|
||
# shape changes, this step goes red instead of the tests quietly passing.
|
||
run: |
|
||
for t in detekt ktlint checkstyle golangci-lint; do
|
||
command -v "$t" >/dev/null || { echo "MISSING on PATH: $t"; exit 1; }
|
||
done
|
||
detekt --version
|
||
ktlint --version
|
||
checkstyle --version || checkstyle -V
|
||
golangci-lint --version
|
||
|
||
- name: Install claude + codex CLIs + bubblewrap (deterministic harness + sandbox tests)
|
||
# claude and codex (no API key needed — both reached via a local mock-model
|
||
# tier) let the binary-gated tests actually RUN instead of skipping: claude
|
||
# for the Anthropic-Messages mock, codex for the OpenAI-Responses mock
|
||
# (src/adapters/codex/mock-model.test.ts drives real codex against it).
|
||
# bubblewrap lets the sandbox confinement test execute and prove egress is
|
||
# blocked, instead of skipping.
|
||
run: |
|
||
# Pin claude-code to the dialect-validated version — SINGLE SOURCE OF
|
||
# TRUTH: VALIDATED_CC_VERSION in src/dialect-drift.ts. Keeps the
|
||
# dialect-drift alarm deterministic (fires only on a deliberate bump,
|
||
# not every unpinned CC release on an unrelated PR) and the real-claude
|
||
# harness/eval tests reproducible.
|
||
CC=$(grep -oP -m1 'VALIDATED_CC_VERSION = "\K[0-9.]+' src/dialect-drift.ts)
|
||
[ -n "$CC" ] || { echo "could not read VALIDATED_CC_VERSION"; exit 1; }
|
||
echo "Pinning @anthropic-ai/claude-code@$CC"
|
||
# Codex is pinned for the SAME reason, and the reason stopped being
|
||
# hypothetical on 2026-09-03: `latest` was 0.153.1, whose TARBALL
|
||
# 404s while its registry metadata still resolves, so `npm i -g
|
||
# @openai/codex` failed for every job on every branch — a red CI
|
||
# nobody's diff caused. Reproduced: metadata `npm view` returns
|
||
# 0.153.1, GET .../codex-0.153.1.tgz is 404; 0.153.0 and older are
|
||
# 200. Leaving this unpinned beside a deliberately pinned
|
||
# claude-code let one upstream publish take the build down.
|
||
npm i -g "@anthropic-ai/claude-code@$CC" "@openai/codex@0.153.0"
|
||
sudo apt-get update && sudo apt-get install -y bubblewrap
|
||
|
||
- name: Allow unprivileged user namespaces (Ubuntu 24.04 AppArmor restriction)
|
||
# ubuntu-latest (24.04) ships `kernel.apparmor_restrict_unprivileged_userns=1`,
|
||
# which blocks unconfined bubblewrap from creating the namespaces it needs —
|
||
# so `bwrap --unshare-all` fails (EPERM) even though the binary is installed.
|
||
# Disable that restriction so the sandbox confinement tests actually RUN and
|
||
# PASS here (genuine verification), instead of `sandboxAvailable()` correctly
|
||
# detecting it can't confine and skipping them. Best-effort: `|| true` so a
|
||
# future runner without this knob doesn't fail the job (the probe then skips).
|
||
run: |
|
||
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 || true
|
||
# Sanity-check that bwrap can now actually namespace:
|
||
bwrap --unshare-all --ro-bind / / true && echo "bwrap confinement: OK" || echo "bwrap confinement: STILL BLOCKED (tests will skip)"
|
||
|
||
- name: Build
|
||
run: npm run build
|
||
|
||
# This is the whole job now, and it used to be about half of it. Measured
|
||
# on run 31631013102 (ubuntu-latest), `test` was CI's SOLE long pole at
|
||
# 554 s while check/site/e2e/harness each finished inside 2 min on their
|
||
# own runners and idled. Two of the four items in that 554 s were waste:
|
||
#
|
||
# 100 s setup (python/ruby/java/go linters, claude+codex+bubblewrap, ci, build)
|
||
# 208 s this step
|
||
# 182 s `npx vitest run && npx jest` — a SECOND full pass over the very
|
||
# suite this step had just run. `vitest run` runs all four vitest
|
||
# projects, `runners` (the matcher constraint) among them, so the
|
||
# coverage run above already covered it. Only jest was unique, and
|
||
# jest takes 1.4 s. Deleted; jest moved to `check`.
|
||
# 63 s 3× typecheck · lint · prettier — none of which need a single one
|
||
# of the language toolchains installed above. Moved to `check`,
|
||
# which finishes in ~90 s and then sits idle.
|
||
#
|
||
# Expect ~4.9 min here instead of 9.2, on the same one runner.
|
||
- name: Run tests + 100% coverage gate (harness-testing pillar)
|
||
run: npm run coverage
|
||
|
||
# ── Sharding: measured, and NOT worth it (2026-08-12) ───────────────────
|
||
# The obvious next move is `vitest --shard=i/3` across three runners. It
|
||
# was built and measured on this checkout before being dropped, so nobody
|
||
# has to re-derive it:
|
||
#
|
||
# WHY IT WAS DROPPED — it buys 10–20 s. Local wall (4 cores): unsharded
|
||
# 177 s; three shards 123 s / 76 s / 45 s. The long pole stays at 123 s
|
||
# because vitest shards by SHA-1 of the spec PATH — deterministic but
|
||
# duration-blind — so the split was 1.5× the ideal 81 s. Add the ~100 s
|
||
# toolchain setup, which no shard can skip (the gated linter-binary tests
|
||
# live in the unit project and must not silently skip), and a ~40 s serial
|
||
# merge job, and the lane lands at ~4.7 min against the ~4.9 min above.
|
||
# Cost: 9.2 → ~13 runner-minutes, and the required status check renames
|
||
# from `test` to `test (shard 1/3)` + a new job, which can silently block
|
||
# every PR on a branch-protection rule.
|
||
#
|
||
# HOW IT WOULD HAVE TO WORK, if the suite ever grows enough to revisit.
|
||
# The gate is the hard part: a shard only loads the files its own slice of
|
||
# tests imports, so per-shard coverage is partial BY CONSTRUCTION, and a
|
||
# gate left inside a shard has to be relaxed to pass — a relaxed gate that
|
||
# still prints green is the exact failure class this repo exists to catch.
|
||
# The shape that works: shards run
|
||
# vitest run --shard=i/3 --reporter=blob --coverage --coverage.reporter=none
|
||
# --coverage.thresholds.{lines,functions,statements,branches}=0
|
||
# and upload `.vitest-reports/*` (a DOTDIR — needs upload-artifact's
|
||
# `include-hidden-files: true` or the artifact is silently empty); a
|
||
# dependent job downloads them and runs `vitest --merge-reports --coverage`,
|
||
# where the thresholds in vitest.config.mjs apply once, over the union.
|
||
#
|
||
# THE THREE PROOFS IT STAYS HONEST, all run locally on this checkout:
|
||
# · FAITHFUL — merged 3 shards vs the unsharded run: same 19 files, 3509
|
||
# coverage hit-bits (statements, per-path branches, functions) compared
|
||
# element-by-element, 0 mismatches, identical summary. Compare hit BITS,
|
||
# not counts: counts legitimately differ, since each side sums over a
|
||
# different set of tests.
|
||
# · STILL BITES — with src/share-link.test.ts removed, all three shards
|
||
# exit 0 while the MERGED report shows share-link.ts at 0% and fails
|
||
# with three threshold errors. Untested files survive the split because
|
||
# each shard records the whole include-list, zero-filling what it never
|
||
# loaded (`generateCoverage` gets allTestsRun=true even under --shard).
|
||
# · FAILS CLOSED — merging a single blob reports 66.53% against the same
|
||
# 1775-statement denominator and errors, so a lost or empty artifact
|
||
# reddens the gate rather than shrinking it. That property, not a
|
||
# blob-count assertion, is what makes the design safe.
|
||
# Also measured: the merge job needs no `npm run build` (the v8 remap reads
|
||
# the TS sources, not dist/), and `--coverage.reporter=none` on a shard
|
||
# loses nothing from the blob — it only suppresses a partial table that
|
||
# reads like a coverage figure.
|
||
|
||
check:
|
||
runs-on: ubuntu-latest
|
||
# measured 2m36s.
|
||
timeout-minutes: 10
|
||
# pull-requests: write lets the dogfooded action post its sticky PR comment.
|
||
permissions:
|
||
contents: read
|
||
pull-requests: write
|
||
steps:
|
||
# fetch-depth: 0 so the dogfooded action's capability-diff can reach the
|
||
# PR base commit (it materializes it as a detached worktree).
|
||
- uses: actions/checkout@v4
|
||
with:
|
||
fetch-depth: 0
|
||
|
||
- uses: actions/setup-node@v4
|
||
with:
|
||
node-version: "20"
|
||
|
||
- run: npm ci
|
||
|
||
- name: Build
|
||
run: npm run build
|
||
|
||
# Dogfood the shipped GitHub Action FIRST — it's the primary lint gate
|
||
# users actually rely on, so run the composite over the repo's own build
|
||
# (`version: local`) ahead of the redundant direct CLI call below. The
|
||
# action emits per-line `::error file=,line=::` annotations (rendered inline
|
||
# on the PR diff) and posts a sticky PR comment. See the prod-grade-gha-cli
|
||
# rule. Running it first means a lint failure surfaces THROUGH the action
|
||
# (comment + annotations), not a bare CLI step before it. `capability-diff`
|
||
# folds the blast-radius diff vs the PR base into the SAME sticky comment
|
||
# (informational — no fail-on-widen — so a legit widening doesn't red CI).
|
||
- name: Dogfood the vigiles GitHub Action (primary lint gate)
|
||
id: vigiles-action
|
||
uses: ./
|
||
with:
|
||
command: lint
|
||
version: local
|
||
capability-diff: "true"
|
||
|
||
- name: Assert the action reported success
|
||
run: |
|
||
got="${{ steps.vigiles-action.outputs.valid }}"
|
||
if [ "$got" != "true" ]; then
|
||
echo "expected the vigiles action output valid=true, got '$got'"
|
||
exit 1
|
||
fi
|
||
echo "vigiles action output: valid=$got"
|
||
|
||
# Dogfood the eval staleness gate via the composite Action. It now VERIFIES
|
||
# a real committed lock instead of no-opping: `paths` names the one eval
|
||
# that has one, which is what the gate's own error message prescribes for
|
||
# an eval without a lock ("exclude it from the --check run").
|
||
#
|
||
# 🔴 It was a NO-OP FOR ITS WHOLE LIFE, and nobody could tell. With no lock
|
||
# committed anywhere, `eval --check` returned early on `anyLocksCommitted`
|
||
# and printed "nothing to verify" — exit 0. The first commit of a lock got
|
||
# past that early return and the step failed twice over: once on a quota
|
||
# consent that should never gate a verify (fixed in run-scripts.ts), then
|
||
# on the 22 evals that are unnamed and unlocked. Both were invisible while
|
||
# the step never ran. Widen `paths` as evals earn locks; a bare `--check`
|
||
# here fails until every discovered eval has a name AND a committed lock.
|
||
#
|
||
# 🔴 Kept OUT of `npm run check` on purpose, and nearly lost by accident when
|
||
# the consolidation below was written: it sat between two command-steps, so a
|
||
# span-based edit swallowed it. Like the lint step above it, it exercises the
|
||
# ACTION WRAPPER, which has no shell equivalent.
|
||
- name: Dogfood the eval staleness gate (eval-check)
|
||
uses: ./
|
||
with:
|
||
command: eval-check
|
||
paths: examples/harness/skill-trigger-rate.eval.mjs
|
||
version: local
|
||
comment: "false"
|
||
|
||
# ONE list, and it lives in `scripts/check.mjs` — not restated here.
|
||
#
|
||
# On 2026-08-16 three PRs went to `main` red because the author ran a
|
||
# five-command list from memory while this job ran thirteen. The failing one
|
||
# (`tsc -p test/types/tsconfig.json`) sits in a separate tsconfig the root
|
||
# type-check cannot see, and three `@ts-expect-error` assertions had quietly
|
||
# stopped failing. A longer list to remember would not have fixed that; a
|
||
# single list that CI INVOKES does — the pre-commit / just / mise pattern.
|
||
#
|
||
# `npm run check` runs `build`, then the read-only checks in parallel, then
|
||
# the file-WRITING ones serially (a writer beside a reader is a race), and
|
||
# names whatever failed — which is what the per-step names here used to buy.
|
||
# `node scripts/check.mjs --list` prints the commands without running them.
|
||
#
|
||
# The two `uses: ./` steps above are NOT in it on purpose: they dogfood this
|
||
# repo's own composite Action, which has no shell equivalent.
|
||
- name: Check suite (identical to `npm run check` locally)
|
||
run: npm run check
|
||
|
||
harness:
|
||
# Dogfood the harness-testing pillar on the in-repo build: the deterministic
|
||
# tier runs the real claude CLI against a scripted mock model, so it needs
|
||
# no API key and costs nothing. The eval tier (real model) is run manually
|
||
# via `npm run test:eval`.
|
||
needs: changes
|
||
if: needs.changes.outputs.root == 'true'
|
||
runs-on: ubuntu-latest
|
||
# measured 45s.
|
||
timeout-minutes: 10
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
|
||
- uses: actions/setup-node@v4
|
||
with:
|
||
node-version: "20"
|
||
cache: "npm"
|
||
|
||
- run: npm ci
|
||
|
||
- name: Install claude CLI (pinned to the dialect-validated version; no API key needed)
|
||
run: |
|
||
CC=$(grep -oP -m1 'VALIDATED_CC_VERSION = "\K[0-9.]+' src/dialect-drift.ts)
|
||
[ -n "$CC" ] || { echo "could not read VALIDATED_CC_VERSION"; exit 1; }
|
||
npm i -g "@anthropic-ai/claude-code@$CC"
|
||
|
||
- name: Build
|
||
run: npm run build
|
||
|
||
- name: Run deterministic harness tests
|
||
# --min is a FLOOR, and it is the whole point of this line: without it a
|
||
# broken glob (a rename, a moved directory, a changed extension) makes this
|
||
# job print "No **/*.harness.* files found." and exit 0 — green, having run
|
||
# nothing. Measured 2026-09-07: `vigiles test` on zero files exits 0.
|
||
# Adding harness files keeps passing; dropping below the floor fails.
|
||
run: node dist/cli.js test --min=14
|
||
|
||
# The in-browser "grade a repo" demo (site/) runs the SAME compiled engine the
|
||
# CLI runs (`@engine/*` → the built dist/), so its browser-mode vitest suite is a
|
||
# real gate: parity (pako === node:zlib on real Chromium), the DemoAudit
|
||
# interaction states, and the idb-keyval grade-cache (real IndexedDB). No API key,
|
||
# deterministic. This closes the gap where demo/engine-parity regressions were
|
||
# only caught locally. The Playwright e2e (`test:e2e`) runs here too — it serves
|
||
# the REAL built bundle and drives it, incl. the 390px mobile-overflow gate that
|
||
# catches layout bleed a component-level test can't (no viewport in @vitest/browser).
|
||
site:
|
||
needs: changes
|
||
if: needs.changes.outputs.site == 'true'
|
||
runs-on: ubuntu-latest
|
||
# measured 3m35s.
|
||
timeout-minutes: 15
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
|
||
- uses: actions/setup-node@v4
|
||
with:
|
||
node-version: "20"
|
||
cache: "npm"
|
||
|
||
- run: npm ci
|
||
|
||
# The site imports the built root dist/ via the @engine/* aliases (the
|
||
# literally-same compiled engine), so it must exist before the browser tests.
|
||
- name: Build engine (root dist/)
|
||
run: npm run build
|
||
|
||
# 🔴 THIS STEP ATE THE WHOLE JOB TWICE (2026-08-19, runs 32300064395 and
|
||
# 32303343213), and neither failure was in the diff under review. Two separate
|
||
# causes, fixed separately:
|
||
#
|
||
# 1. `--with-deps` shells out to `apt-get update`, which hung on
|
||
# azure.archive.ubuntu.com (`Ign:` on every line, then silence for 12
|
||
# minutes until the job bond killed it). `ubuntu-latest` already ships the
|
||
# system libraries Chromium needs — `--with-deps` exists for bare
|
||
# containers. Dropping it removes the apt call, and with it the only part
|
||
# of this step that touches a distro mirror. If a library really is
|
||
# missing, the browser fails to LAUNCH in the next step, in seconds, with
|
||
# the missing .so named — a loud failure instead of a silent 15-minute hang.
|
||
# 2. Chromium (~150 MB) was re-downloaded on every run with no cache, so the
|
||
# step's floor was a full download even on a good day.
|
||
#
|
||
# The step bond is the third leg: a hang now costs 5 minutes, not the job. The
|
||
# job bond cannot distinguish "this step wedged" from "the suite is slow".
|
||
- name: Cache the Playwright browser
|
||
id: pw-cache
|
||
uses: actions/cache@v4
|
||
with:
|
||
path: ~/.cache/ms-playwright
|
||
# Keyed on the resolved Playwright version: a bump must miss the cache,
|
||
# because the browser build is pinned to it.
|
||
key: pw-${{ runner.os }}-${{ hashFiles('site/package-lock.json', 'package-lock.json') }}
|
||
|
||
- name: Install Chromium for Playwright (vitest browser mode)
|
||
if: steps.pw-cache.outputs.cache-hit != 'true'
|
||
timeout-minutes: 5
|
||
run: npx playwright install chromium
|
||
working-directory: site
|
||
|
||
- name: Run site browser tests (parity + interaction + grade-cache)
|
||
run: npm run test:browser
|
||
working-directory: site
|
||
|
||
# Playwright e2e over the real built bundle (serves dist/, network mocked
|
||
# per-test) — the in-frame report/empty states AND the 390px mobile-overflow
|
||
# gate. Chromium is already installed above; the webServer builds engine+site.
|
||
- name: Run site e2e (Playwright — incl. mobile-overflow gate)
|
||
run: npm run test:e2e
|
||
working-directory: site
|
||
|
||
# Dogfood the per-level CI action for the e2e tier (allowlisted REAL egress).
|
||
# The egress connector needs /dev/net/tun + privilege to open the tap and
|
||
# configure the netns — the non-root host runner can't, which is why slirp4netns
|
||
# "fails" there (only `lo`). So this job runs as ROOT in a privileged container,
|
||
# where slirp4netns attaches exactly as it does on a root dev box and the egress
|
||
# tests run FOR REAL. Tests still self-skip via egressRoutes() if egress can't
|
||
# route, so the job never fails red. See research/egress-sandbox-tooling.md.
|
||
e2e:
|
||
needs: changes
|
||
if: needs.changes.outputs.root == 'true'
|
||
runs-on: ubuntu-latest
|
||
# measured 1m16s.
|
||
timeout-minutes: 10
|
||
container:
|
||
image: node:20-bookworm
|
||
options: --privileged --device /dev/net/tun
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
- uses: ./.github/actions/harness-tier
|
||
with:
|
||
tier: e2e
|