mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
zernie
e13d219ad0
The root config was plain JS while site/vitest.config.ts was TypeScript — an inconsistency, not a constraint. The file was already ESM with a typed defineConfig, so nothing inside it changed. The landmine was removed first (#223): the path guard DISCOVERS the filename via readdirSync().find(f => f.startsWith("vitest.config.")) instead of spelling the extension, so this rename touches its comment, not its code. Nine references updated; one was a live relative link in docs/testing-matrix.md that would have broken. CLAUDE.md was regenerated from its spec, not hand-edited. Mutation-proven three ways: green on .mjs, green on .ts, and a VALID config not named vitest.config.* fails by the guard's own message ("no vitest.config.* at the repo root") on exactly one assertion. Gates: npm run check 18/18 · npm run coverage 3976 passed, 0 failed — byte-identical to the pre-rename run. Merged without an automated review: the Codex reviewer reported its code-review usage limit was exhausted, so the signal was unavailable rather than pending. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
586 lines
30 KiB
YAML
586 lines
30 KiB
YAML
name: CI
|
||
|
||
on:
|
||
push:
|
||
branches: [main]
|
||
pull_request:
|
||
branches: [main]
|
||
|
||
jobs:
|
||
# ── WHICH JOBS THIS COMMIT ACTUALLY NEEDS ────────────────────────────────────
|
||
# Until 2026-08-19 this workflow had no path filter at all, so every PR paid for
|
||
# all five jobs whatever it touched. Measured on PR #167 (16 files: src/, docs/,
|
||
# CLAUDE.md — none under site/): the `site` job built the engine, downloaded a
|
||
# browser and ran the demo suite against code the diff could not reach. Twice, at
|
||
# ~15 minutes each, because the browser install wedged.
|
||
#
|
||
# A workflow-level `paths:` cannot express this — it decides whether to run the
|
||
# workflow AT ALL and says nothing about individual jobs. Hence this job, plus an
|
||
# `if:` on the jobs whose inputs are disjoint from what changed.
|
||
#
|
||
# 🔴 THE RULE INSIDE: any doubt runs everything. A SKIPPED job and a PASSED job
|
||
# look identical in the checks list, so the cost of a wrong skip is a green tick
|
||
# over work nobody did, while the cost of a wrong run is a few minutes. The action
|
||
# errs the same way by construction — "all files are considered as added if there
|
||
# is no common ancestor with base branch or no previous commit", i.e. a new branch
|
||
# or a shallow history runs everything.
|
||
#
|
||
# ── WHY AN ACTION AND NOT OUR OWN `grep` (2026-09-09) ────────────────────────
|
||
# This was a hand-written shell classifier: one `gh api` call for the file list,
|
||
# then two `grep -qvE`, then a fallback for every way that list can come back
|
||
# unusable (empty, failed call, a push whose `before` is all zeros). Every one of
|
||
# those branches is a thing `dorny/paths-filter` already does, and does more
|
||
# carefully — it is the standard action for exactly this, and the edge cases were
|
||
# the half we wrote by hand.
|
||
#
|
||
# THE COST, stated because it is real: on a `push` the action diffs with git and
|
||
# therefore needs a checkout, while the old classifier needed none and finished in
|
||
# ~3s. On a `pull_request` it reads the REST API and still needs nothing. Push
|
||
# here only ever means a merge into main, so the checkout is paid once per merge
|
||
# and the job still rounds to the same single billable minute.
|
||
#
|
||
# ── WHAT A PUSH COMPARES, because it reads like a regression and is not ─────
|
||
# `base` is deliberately UNSET, and that does NOT mean "only the last commit".
|
||
# Traced through the action's main.ts: with no `base` it defaults to the
|
||
# repository's `default_branch` (main), while `head` comes from `github.ref`
|
||
# (also main), so `isBaseSameAsHead` holds and `baseSha` becomes the push
|
||
# event's own `before` — i.e. `git diff <before>..<head>`, the WHOLE pushed
|
||
# range, exactly what the hand-written classifier did. The "changes will be
|
||
# detected from last commit" branch is reachable only when `before` is absent
|
||
# from the payload, which a push event never is.
|
||
#
|
||
# Confirmed in the first push run after the switch (run 34420710819):
|
||
# Changes will be detected between 236615e97... and main
|
||
# git diff --no-renames --name-status -z 236615e97...refs/remotes/origin/main
|
||
# where 236615e97 is that push's `before`. The same log shows it `git fetch
|
||
# --depth=1` that SHA on demand, which is why the shallow checkout above is
|
||
# enough.
|
||
#
|
||
# 🔴 `predicate-quantifier: some-with-excludes` IS LOAD-BEARING, NOT DECORATION.
|
||
# Under the default `some`, patterns are OR-ed (`patterns.some(...)` in the
|
||
# action's own filter.ts), so `['**', '!site/**']` would match a file under site/
|
||
# via `**` and set root=true for a site-only diff — the exact hole this filter
|
||
# exists to prevent, silently. `some-with-excludes` makes a negated pattern a
|
||
# final exclusion: "included by >=1 pattern AND excluded by 0". Pinned by
|
||
# src/ci-path-filter.test.ts, which fails if the setting is missing.
|
||
changes:
|
||
runs-on: ubuntu-latest
|
||
timeout-minutes: 5
|
||
# BOTH lines are required, and the second is the one that is easy to get
|
||
# wrong. A job-level `permissions:` block REPLACES the default for this job
|
||
# rather than adding to it — everything unlisted becomes `none` — so naming
|
||
# only `pull-requests: read` would strip `contents: read` and break the
|
||
# `actions/checkout@v4` below, which the `push` path needs to diff with git.
|
||
#
|
||
# `pull-requests: read` is what the action's README requires for the
|
||
# `pull_request` path: with a token it calls `pulls.listFiles` (REST) instead
|
||
# of diffing, which is exactly why that path needs no checkout. Today this
|
||
# workflow declares no top-level `permissions:`, so the job inherits the
|
||
# repository default and the REST call happens to be allowed. That is luck,
|
||
# not a contract: the moment anyone adds a restrictive top-level block — the
|
||
# standard hardening step — the call 403s. The old classifier degraded to
|
||
# `gh api ... || true` and ran everything; the action raises, so the failure
|
||
# is loud rather than silent, but it blocks every PR until someone finds it.
|
||
# Declaring the two permissions the job actually uses costs nothing and
|
||
# removes the dependency on a default we do not control.
|
||
permissions:
|
||
contents: read
|
||
pull-requests: read
|
||
outputs:
|
||
root: ${{ steps.f.outputs.root }}
|
||
site: ${{ steps.f.outputs.site }}
|
||
steps:
|
||
# `push` ONLY — see the cost note above. Default depth is fine: the action
|
||
# fetches what it needs on top of it (`initial-fetch-depth`, default 100).
|
||
- uses: actions/checkout@v4
|
||
if: github.event_name == 'push'
|
||
|
||
- uses: dorny/paths-filter@v4.0.3
|
||
id: f
|
||
with:
|
||
predicate-quantifier: some-with-excludes
|
||
filters: |
|
||
# `root` — everything except the site. False ONLY when every changed
|
||
# file is under site/, because nothing outside site/ depends on
|
||
# anything inside it.
|
||
#
|
||
# 🔴 THAT SENTENCE WAS FALSE ONCE, AND IT COST A RED MAIN. Two root
|
||
# tests read files under site/ off disk; #219 was a site-only PR that
|
||
# deleted one, so these jobs were skipped, the PR merged GREEN and main
|
||
# broke on an ENOENT. Both reads are gone (#223 — one test moved to the
|
||
# site suite, one snapshot moved to tools/measured/ so the site imports
|
||
# it instead), and src/ci-path-filter.test.ts now FAILS if anything the
|
||
# root suite runs depends on site/ again. The invariant is checked, not
|
||
# asserted.
|
||
root:
|
||
- '**'
|
||
- '!site/**'
|
||
|
||
# `site` — the demo. It imports the built engine through three aliases
|
||
# (@engine/scan-files, @engine/audit-report, @engine/spec), so any src/
|
||
# change can reach it and must run it. What CANNOT reach it: prose and
|
||
# agent config. Verified 2026-08-19 by grepping the three bundles the
|
||
# site actually loads — the lint rule table (`DEFAULT_RULES`,
|
||
# `frontmatter-valid`) appears in none of them, so a rules change is
|
||
# invisible to the demo. Its check pages are generated from
|
||
# site/src/checks/checks.ts, NOT from docs/, so docs cannot reach it
|
||
# through the page generator either.
|
||
site:
|
||
- '**'
|
||
- '!docs/**'
|
||
- '!.claude/**'
|
||
- '!*.md'
|
||
|
||
test:
|
||
needs: changes
|
||
if: needs.changes.outputs.root == 'true'
|
||
runs-on: ubuntu-latest
|
||
# Bounded 2026-08-19. Every job in this repository was unbounded, so a hung one
|
||
# ran to GitHub's six-hour ceiling — the one failure mode a green checkmark
|
||
# cannot show you, because it does not fail, it bills. Sizes are ~3-4x the
|
||
# MEASURED duration on this repo's own runs, quoted per job so a later edit can
|
||
# see where the number came from instead of guessing again.
|
||
# measured 5m49s.
|
||
timeout-minutes: 20
|
||
env:
|
||
# JVM/Go linter pins for the gated real-binary catalog tests
|
||
# (src/core/linters.test.ts — describe.skipIf(!hasBinary(...))). Installing
|
||
# these makes those tests RUN in CI instead of silently skipping forever
|
||
# (a green-with-skips is a hidden gap — no-silent-skips). detekt is pinned
|
||
# to the latest STABLE (2.0 is still alpha).
|
||
DETEKT_VERSION: "1.23.8"
|
||
KTLINT_VERSION: "1.8.0"
|
||
CHECKSTYLE_VERSION: "13.8.0" # 13.x requires JDK >= 17
|
||
GOLANGCI_LINT_VERSION: "2.12.2"
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
|
||
- uses: actions/setup-node@v4
|
||
with:
|
||
node-version: "20"
|
||
cache: "npm"
|
||
|
||
- uses: actions/setup-python@v5
|
||
with:
|
||
python-version: "3.x"
|
||
|
||
- uses: ruby/setup-ruby@v1
|
||
with:
|
||
ruby-version: "3.3"
|
||
|
||
# One JDK serves detekt + ktlint + checkstyle. Temurin 21 is preinstalled
|
||
# on ubuntu-24.04 runners, so this is env setup (~seconds), not a download.
|
||
- uses: actions/setup-java@v4
|
||
with:
|
||
distribution: "temurin"
|
||
java-version: "21"
|
||
|
||
- name: Cache pip packages
|
||
uses: actions/cache@v4
|
||
with:
|
||
path: ~/.cache/pip
|
||
key: pip-${{ runner.os }}-ruff-pylint
|
||
|
||
- name: Cache gem packages
|
||
uses: actions/cache@v4
|
||
with:
|
||
path: ~/.local/share/gem
|
||
key: gem-${{ runner.os }}-rubocop
|
||
|
||
- name: Cache JVM/Go linter tools
|
||
uses: actions/cache@v4
|
||
with:
|
||
path: ~/.vigiles-linter-tools
|
||
key: linter-tools-${{ runner.os }}-detekt${{ env.DETEKT_VERSION }}-ktlint${{ env.KTLINT_VERSION }}-checkstyle${{ env.CHECKSTYLE_VERSION }}-golangci${{ env.GOLANGCI_LINT_VERSION }}
|
||
|
||
- run: npm ci
|
||
|
||
- name: Install linter CLIs for tests
|
||
run: |
|
||
pip install ruff pylint
|
||
gem install rubocop
|
||
rustup component add clippy
|
||
|
||
- name: Install JVM/Go linter CLIs (detekt, ktlint, checkstyle, golangci-lint)
|
||
# Lets the gated real-binary catalog tests in src/core/linters.test.ts
|
||
# actually RUN instead of skipping. Each tool must land on PATH under the
|
||
# exact name its LINTERS adapter's `cliTool` expects (src/core/linters.ts):
|
||
# detekt, ktlint, checkstyle, golangci-lint. checkstyle ships only a fat jar,
|
||
# so it gets a 2-line `java -jar` shim; detekt's zip launcher is symlinked.
|
||
run: |
|
||
TOOLS="$HOME/.vigiles-linter-tools"
|
||
BIN="$TOOLS/bin"
|
||
mkdir -p "$BIN"
|
||
if [ ! -x "$BIN/detekt" ]; then
|
||
curl -sSfL -o /tmp/detekt-cli.zip \
|
||
"https://github.com/detekt/detekt/releases/download/v${DETEKT_VERSION}/detekt-cli-${DETEKT_VERSION}.zip"
|
||
unzip -q -o /tmp/detekt-cli.zip -d "$TOOLS"
|
||
launcher=$(find "$TOOLS/detekt-cli-${DETEKT_VERSION}/bin" -maxdepth 1 -type f ! -name '*.bat' | head -n1)
|
||
[ -n "$launcher" ] || { echo "detekt launcher not found in zip"; exit 1; }
|
||
chmod +x "$launcher" && ln -sf "$launcher" "$BIN/detekt"
|
||
|
||
curl -sSfL -o "$BIN/ktlint" \
|
||
"https://github.com/pinterest/ktlint/releases/download/${KTLINT_VERSION}/ktlint"
|
||
chmod +x "$BIN/ktlint"
|
||
|
||
curl -sSfL -o "$TOOLS/checkstyle-all.jar" \
|
||
"https://github.com/checkstyle/checkstyle/releases/download/checkstyle-${CHECKSTYLE_VERSION}/checkstyle-${CHECKSTYLE_VERSION}-all.jar"
|
||
printf '#!/usr/bin/env bash\nexec java -jar "%s" "$@"\n' "$TOOLS/checkstyle-all.jar" > "$BIN/checkstyle"
|
||
chmod +x "$BIN/checkstyle"
|
||
|
||
curl -sSfL "https://raw.githubusercontent.com/golangci/golangci-lint/v${GOLANGCI_LINT_VERSION}/install.sh" \
|
||
| sh -s -- -b "$BIN" "v${GOLANGCI_LINT_VERSION}"
|
||
fi
|
||
echo "$BIN" >> "$GITHUB_PATH"
|
||
|
||
- name: Verify linter tools resolve (no silent skips)
|
||
# hasBinary() gates on `which <name>`; fail FAST here rather than letting
|
||
# the gated catalog tests silently skip (no-silent-skips). If a download
|
||
# shape changes, this step goes red instead of the tests quietly passing.
|
||
run: |
|
||
for t in detekt ktlint checkstyle golangci-lint; do
|
||
command -v "$t" >/dev/null || { echo "MISSING on PATH: $t"; exit 1; }
|
||
done
|
||
detekt --version
|
||
ktlint --version
|
||
checkstyle --version || checkstyle -V
|
||
golangci-lint --version
|
||
|
||
- name: Install claude + codex CLIs + bubblewrap (deterministic harness + sandbox tests)
|
||
# claude and codex (no API key needed — both reached via a local mock-model
|
||
# tier) let the binary-gated tests actually RUN instead of skipping: claude
|
||
# for the Anthropic-Messages mock, codex for the OpenAI-Responses mock
|
||
# (src/adapters/codex/mock-model.test.ts drives real codex against it).
|
||
# bubblewrap lets the sandbox confinement test execute and prove egress is
|
||
# blocked, instead of skipping.
|
||
run: |
|
||
# Pin claude-code to the dialect-validated version — SINGLE SOURCE OF
|
||
# TRUTH: VALIDATED_CC_VERSION in src/dialect-drift.ts. Keeps the
|
||
# dialect-drift alarm deterministic (fires only on a deliberate bump,
|
||
# not every unpinned CC release on an unrelated PR) and the real-claude
|
||
# harness/eval tests reproducible.
|
||
CC=$(grep -oP -m1 'VALIDATED_CC_VERSION = "\K[0-9.]+' src/dialect-drift.ts)
|
||
[ -n "$CC" ] || { echo "could not read VALIDATED_CC_VERSION"; exit 1; }
|
||
echo "Pinning @anthropic-ai/claude-code@$CC"
|
||
# Codex is pinned for the SAME reason, and the reason stopped being
|
||
# hypothetical on 2026-09-03: `latest` was 0.153.1, whose TARBALL
|
||
# 404s while its registry metadata still resolves, so `npm i -g
|
||
# @openai/codex` failed for every job on every branch — a red CI
|
||
# nobody's diff caused. Reproduced: metadata `npm view` returns
|
||
# 0.153.1, GET .../codex-0.153.1.tgz is 404; 0.153.0 and older are
|
||
# 200. Leaving this unpinned beside a deliberately pinned
|
||
# claude-code let one upstream publish take the build down.
|
||
npm i -g "@anthropic-ai/claude-code@$CC" "@openai/codex@0.153.0"
|
||
sudo apt-get update && sudo apt-get install -y bubblewrap
|
||
|
||
- name: Allow unprivileged user namespaces (Ubuntu 24.04 AppArmor restriction)
|
||
# ubuntu-latest (24.04) ships `kernel.apparmor_restrict_unprivileged_userns=1`,
|
||
# which blocks unconfined bubblewrap from creating the namespaces it needs —
|
||
# so `bwrap --unshare-all` fails (EPERM) even though the binary is installed.
|
||
# Disable that restriction so the sandbox confinement tests actually RUN and
|
||
# PASS here (genuine verification), instead of `sandboxAvailable()` correctly
|
||
# detecting it can't confine and skipping them. Best-effort: `|| true` so a
|
||
# future runner without this knob doesn't fail the job (the probe then skips).
|
||
run: |
|
||
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 || true
|
||
# Sanity-check that bwrap can now actually namespace:
|
||
bwrap --unshare-all --ro-bind / / true && echo "bwrap confinement: OK" || echo "bwrap confinement: STILL BLOCKED (tests will skip)"
|
||
|
||
- name: Build
|
||
run: npm run build
|
||
|
||
# This is the whole job now, and it used to be about half of it. Measured
|
||
# on run 31631013102 (ubuntu-latest), `test` was CI's SOLE long pole at
|
||
# 554 s while check/site/e2e/harness each finished inside 2 min on their
|
||
# own runners and idled. Two of the four items in that 554 s were waste:
|
||
#
|
||
# 100 s setup (python/ruby/java/go linters, claude+codex+bubblewrap, ci, build)
|
||
# 208 s this step
|
||
# 182 s `npx vitest run && npx jest` — a SECOND full pass over the very
|
||
# suite this step had just run. `vitest run` runs all four vitest
|
||
# projects, `runners` (the matcher constraint) among them, so the
|
||
# coverage run above already covered it. Only jest was unique, and
|
||
# jest takes 1.4 s. Deleted; jest moved to `check`.
|
||
# 63 s 3× typecheck · lint · prettier — none of which need a single one
|
||
# of the language toolchains installed above. Moved to `check`,
|
||
# which finishes in ~90 s and then sits idle.
|
||
#
|
||
# Expect ~4.9 min here instead of 9.2, on the same one runner.
|
||
- name: Run tests + 100% coverage gate (harness-testing pillar)
|
||
run: npm run coverage
|
||
|
||
# ── Sharding: measured, and NOT worth it (2026-08-12) ───────────────────
|
||
# The obvious next move is `vitest --shard=i/3` across three runners. It
|
||
# was built and measured on this checkout before being dropped, so nobody
|
||
# has to re-derive it:
|
||
#
|
||
# WHY IT WAS DROPPED — it buys 10–20 s. Local wall (4 cores): unsharded
|
||
# 177 s; three shards 123 s / 76 s / 45 s. The long pole stays at 123 s
|
||
# because vitest shards by SHA-1 of the spec PATH — deterministic but
|
||
# duration-blind — so the split was 1.5× the ideal 81 s. Add the ~100 s
|
||
# toolchain setup, which no shard can skip (the gated linter-binary tests
|
||
# live in the unit project and must not silently skip), and a ~40 s serial
|
||
# merge job, and the lane lands at ~4.7 min against the ~4.9 min above.
|
||
# Cost: 9.2 → ~13 runner-minutes, and the required status check renames
|
||
# from `test` to `test (shard 1/3)` + a new job, which can silently block
|
||
# every PR on a branch-protection rule.
|
||
#
|
||
# HOW IT WOULD HAVE TO WORK, if the suite ever grows enough to revisit.
|
||
# The gate is the hard part: a shard only loads the files its own slice of
|
||
# tests imports, so per-shard coverage is partial BY CONSTRUCTION, and a
|
||
# gate left inside a shard has to be relaxed to pass — a relaxed gate that
|
||
# still prints green is the exact failure class this repo exists to catch.
|
||
# The shape that works: shards run
|
||
# vitest run --shard=i/3 --reporter=blob --coverage --coverage.reporter=none
|
||
# --coverage.thresholds.{lines,functions,statements,branches}=0
|
||
# and upload `.vitest-reports/*` (a DOTDIR — needs upload-artifact's
|
||
# `include-hidden-files: true` or the artifact is silently empty); a
|
||
# dependent job downloads them and runs `vitest --merge-reports --coverage`,
|
||
# where the thresholds in vitest.config.ts apply once, over the union.
|
||
#
|
||
# THE THREE PROOFS IT STAYS HONEST, all run locally on this checkout:
|
||
# · FAITHFUL — merged 3 shards vs the unsharded run: same 19 files, 3509
|
||
# coverage hit-bits (statements, per-path branches, functions) compared
|
||
# element-by-element, 0 mismatches, identical summary. Compare hit BITS,
|
||
# not counts: counts legitimately differ, since each side sums over a
|
||
# different set of tests.
|
||
# · STILL BITES — with src/share-link.test.ts removed, all three shards
|
||
# exit 0 while the MERGED report shows share-link.ts at 0% and fails
|
||
# with three threshold errors. Untested files survive the split because
|
||
# each shard records the whole include-list, zero-filling what it never
|
||
# loaded (`generateCoverage` gets allTestsRun=true even under --shard).
|
||
# · FAILS CLOSED — merging a single blob reports 66.53% against the same
|
||
# 1775-statement denominator and errors, so a lost or empty artifact
|
||
# reddens the gate rather than shrinking it. That property, not a
|
||
# blob-count assertion, is what makes the design safe.
|
||
# Also measured: the merge job needs no `npm run build` (the v8 remap reads
|
||
# the TS sources, not dist/), and `--coverage.reporter=none` on a shard
|
||
# loses nothing from the blob — it only suppresses a partial table that
|
||
# reads like a coverage figure.
|
||
|
||
check:
|
||
runs-on: ubuntu-latest
|
||
# measured 2m36s.
|
||
timeout-minutes: 10
|
||
# pull-requests: write lets the dogfooded action post its sticky PR comment.
|
||
permissions:
|
||
contents: read
|
||
pull-requests: write
|
||
steps:
|
||
# fetch-depth: 0 so the dogfooded action's capability-diff can reach the
|
||
# PR base commit (it materializes it as a detached worktree).
|
||
- uses: actions/checkout@v4
|
||
with:
|
||
fetch-depth: 0
|
||
|
||
- uses: actions/setup-node@v4
|
||
with:
|
||
node-version: "20"
|
||
|
||
- run: npm ci
|
||
|
||
- name: Build
|
||
run: npm run build
|
||
|
||
# Dogfood the shipped GitHub Action FIRST — it's the primary lint gate
|
||
# users actually rely on, so run the composite over the repo's own build
|
||
# (`version: local`) ahead of the redundant direct CLI call below. The
|
||
# action emits per-line `::error file=,line=::` annotations (rendered inline
|
||
# on the PR diff) and posts a sticky PR comment. See the prod-grade-gha-cli
|
||
# rule. Running it first means a lint failure surfaces THROUGH the action
|
||
# (comment + annotations), not a bare CLI step before it. `capability-diff`
|
||
# folds the blast-radius diff vs the PR base into the SAME sticky comment
|
||
# (informational — no fail-on-widen — so a legit widening doesn't red CI).
|
||
- name: Dogfood the vigiles GitHub Action (primary lint gate)
|
||
id: vigiles-action
|
||
uses: ./
|
||
with:
|
||
command: lint
|
||
version: local
|
||
capability-diff: "true"
|
||
|
||
- name: Assert the action reported success
|
||
run: |
|
||
got="${{ steps.vigiles-action.outputs.valid }}"
|
||
if [ "$got" != "true" ]; then
|
||
echo "expected the vigiles action output valid=true, got '$got'"
|
||
exit 1
|
||
fi
|
||
echo "vigiles action output: valid=$got"
|
||
|
||
# Dogfood the eval staleness gate via the composite Action. It now VERIFIES
|
||
# a real committed lock instead of no-opping: `paths` names the one eval
|
||
# that has one, which is what the gate's own error message prescribes for
|
||
# an eval without a lock ("exclude it from the --check run").
|
||
#
|
||
# 🔴 It was a NO-OP FOR ITS WHOLE LIFE, and nobody could tell. With no lock
|
||
# committed anywhere, `eval --check` returned early on `anyLocksCommitted`
|
||
# and printed "nothing to verify" — exit 0. The first commit of a lock got
|
||
# past that early return and the step failed twice over: once on a quota
|
||
# consent that should never gate a verify (fixed in run-scripts.ts), then
|
||
# on the 22 evals that are unnamed and unlocked. Both were invisible while
|
||
# the step never ran. Widen `paths` as evals earn locks; a bare `--check`
|
||
# here fails until every discovered eval has a name AND a committed lock.
|
||
#
|
||
# 🔴 Kept OUT of `npm run check` on purpose, and nearly lost by accident when
|
||
# the consolidation below was written: it sat between two command-steps, so a
|
||
# span-based edit swallowed it. Like the lint step above it, it exercises the
|
||
# ACTION WRAPPER, which has no shell equivalent.
|
||
- name: Dogfood the eval staleness gate (eval-check)
|
||
uses: ./
|
||
with:
|
||
command: eval-check
|
||
paths: examples/harness/skill-trigger-rate.eval.mjs
|
||
version: local
|
||
comment: "false"
|
||
|
||
# ONE list, and it lives in `scripts/check.mjs` — not restated here.
|
||
#
|
||
# On 2026-08-16 three PRs went to `main` red because the author ran a
|
||
# five-command list from memory while this job ran thirteen. The failing one
|
||
# (`tsc -p test/types/tsconfig.json`) sits in a separate tsconfig the root
|
||
# type-check cannot see, and three `@ts-expect-error` assertions had quietly
|
||
# stopped failing. A longer list to remember would not have fixed that; a
|
||
# single list that CI INVOKES does — the pre-commit / just / mise pattern.
|
||
#
|
||
# `npm run check` runs `build`, then the read-only checks in parallel, then
|
||
# the file-WRITING ones serially (a writer beside a reader is a race), and
|
||
# names whatever failed — which is what the per-step names here used to buy.
|
||
# `node scripts/check.mjs --list` prints the commands without running them.
|
||
#
|
||
# The two `uses: ./` steps above are NOT in it on purpose: they dogfood this
|
||
# repo's own composite Action, which has no shell equivalent.
|
||
- name: Check suite (identical to `npm run check` locally)
|
||
run: npm run check
|
||
|
||
harness:
|
||
# Dogfood the harness-testing pillar on the in-repo build: the deterministic
|
||
# tier runs the real claude CLI against a scripted mock model, so it needs
|
||
# no API key and costs nothing. The eval tier (real model) is run manually
|
||
# via `npm run test:eval`.
|
||
needs: changes
|
||
if: needs.changes.outputs.root == 'true'
|
||
runs-on: ubuntu-latest
|
||
# measured 45s.
|
||
timeout-minutes: 10
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
|
||
- uses: actions/setup-node@v4
|
||
with:
|
||
node-version: "20"
|
||
cache: "npm"
|
||
|
||
- run: npm ci
|
||
|
||
- name: Install claude CLI (pinned to the dialect-validated version; no API key needed)
|
||
run: |
|
||
CC=$(grep -oP -m1 'VALIDATED_CC_VERSION = "\K[0-9.]+' src/dialect-drift.ts)
|
||
[ -n "$CC" ] || { echo "could not read VALIDATED_CC_VERSION"; exit 1; }
|
||
npm i -g "@anthropic-ai/claude-code@$CC"
|
||
|
||
- name: Build
|
||
run: npm run build
|
||
|
||
- name: Run deterministic harness tests
|
||
# --min is a FLOOR, and it is the whole point of this line: without it a
|
||
# broken glob (a rename, a moved directory, a changed extension) makes this
|
||
# job print "No **/*.harness.* files found." and exit 0 — green, having run
|
||
# nothing. Measured 2026-09-07: `vigiles test` on zero files exits 0.
|
||
# Adding harness files keeps passing; dropping below the floor fails.
|
||
run: node dist/cli.js test --min=14
|
||
|
||
# The in-browser "grade a repo" demo (site/) runs the SAME compiled engine the
|
||
# CLI runs (`@engine/*` → the built dist/), so its browser-mode vitest suite is a
|
||
# real gate: parity (pako === node:zlib on real Chromium), the DemoAudit
|
||
# interaction states, and the idb-keyval grade-cache (real IndexedDB). No API key,
|
||
# deterministic. This closes the gap where demo/engine-parity regressions were
|
||
# only caught locally. The Playwright e2e (`test:e2e`) runs here too — it serves
|
||
# the REAL built bundle and drives it, incl. the 390px mobile-overflow gate that
|
||
# catches layout bleed a component-level test can't (no viewport in @vitest/browser).
|
||
site:
|
||
needs: changes
|
||
if: needs.changes.outputs.site == 'true'
|
||
runs-on: ubuntu-latest
|
||
# measured 3m35s.
|
||
timeout-minutes: 15
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
|
||
- uses: actions/setup-node@v4
|
||
with:
|
||
node-version: "20"
|
||
cache: "npm"
|
||
|
||
- run: npm ci
|
||
|
||
# The site imports the built root dist/ via the @engine/* aliases (the
|
||
# literally-same compiled engine), so it must exist before the browser tests.
|
||
- name: Build engine (root dist/)
|
||
run: npm run build
|
||
|
||
# 🔴 THIS STEP ATE THE WHOLE JOB TWICE (2026-08-19, runs 32300064395 and
|
||
# 32303343213), and neither failure was in the diff under review. Two separate
|
||
# causes, fixed separately:
|
||
#
|
||
# 1. `--with-deps` shells out to `apt-get update`, which hung on
|
||
# azure.archive.ubuntu.com (`Ign:` on every line, then silence for 12
|
||
# minutes until the job bond killed it). `ubuntu-latest` already ships the
|
||
# system libraries Chromium needs — `--with-deps` exists for bare
|
||
# containers. Dropping it removes the apt call, and with it the only part
|
||
# of this step that touches a distro mirror. If a library really is
|
||
# missing, the browser fails to LAUNCH in the next step, in seconds, with
|
||
# the missing .so named — a loud failure instead of a silent 15-minute hang.
|
||
# 2. Chromium (~150 MB) was re-downloaded on every run with no cache, so the
|
||
# step's floor was a full download even on a good day.
|
||
#
|
||
# The step bond is the third leg: a hang now costs 5 minutes, not the job. The
|
||
# job bond cannot distinguish "this step wedged" from "the suite is slow".
|
||
- name: Cache the Playwright browser
|
||
id: pw-cache
|
||
uses: actions/cache@v4
|
||
with:
|
||
path: ~/.cache/ms-playwright
|
||
# Keyed on the resolved Playwright version: a bump must miss the cache,
|
||
# because the browser build is pinned to it.
|
||
key: pw-${{ runner.os }}-${{ hashFiles('site/package-lock.json', 'package-lock.json') }}
|
||
|
||
- name: Install Chromium for Playwright (vitest browser mode)
|
||
if: steps.pw-cache.outputs.cache-hit != 'true'
|
||
timeout-minutes: 5
|
||
run: npx playwright install chromium
|
||
working-directory: site
|
||
|
||
- name: Run site browser tests (parity + interaction + grade-cache)
|
||
run: npm run test:browser
|
||
working-directory: site
|
||
|
||
# Playwright e2e over the real built bundle (serves dist/, network mocked
|
||
# per-test) — the in-frame report/empty states AND the 390px mobile-overflow
|
||
# gate. Chromium is already installed above; the webServer builds engine+site.
|
||
- name: Run site e2e (Playwright — incl. mobile-overflow gate)
|
||
run: npm run test:e2e
|
||
working-directory: site
|
||
|
||
# Dogfood the per-level CI action for the e2e tier (allowlisted REAL egress).
|
||
# The egress connector needs /dev/net/tun + privilege to open the tap and
|
||
# configure the netns — the non-root host runner can't, which is why slirp4netns
|
||
# "fails" there (only `lo`). So this job runs as ROOT in a privileged container,
|
||
# where slirp4netns attaches exactly as it does on a root dev box and the egress
|
||
# tests run FOR REAL. Tests still self-skip via egressRoutes() if egress can't
|
||
# route, so the job never fails red. See research/egress-sandbox-tooling.md.
|
||
e2e:
|
||
needs: changes
|
||
if: needs.changes.outputs.root == 'true'
|
||
runs-on: ubuntu-latest
|
||
# measured 1m16s.
|
||
timeout-minutes: 10
|
||
container:
|
||
image: node:20-bookworm
|
||
options: --privileged --device /dev/net/tun
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
- uses: ./.github/actions/harness-tier
|
||
with:
|
||
tier: e2e
|