The previous suite was 37/37 green against a workflow that could not have
run at all: the guard step had no `packages: write` and no GHCR login, so
every `imagetools create` would have 401'd. Asserting that a step EXISTS
proves nothing about whether it can succeed. Each change below was checked
by reintroducing the defect and confirming the test goes red.
Registry-auth precondition (the miss that let the above ship)
- Both redeploy jobs must declare `packages: write` AND a `docker/login-action`
step for ghcr.io, ordered BEFORE the guard. Dropping either reds the suite.
The intersection, executed rather than restated
- The old test only checked that the string `images=` appears in the compute
step; swapping its jq for the full matrix — so a FAILED build moves
`:latest` — kept it green. The real `changed` shell now RUNS, against the
real ALL_SERVICES matrix read out of the workflow, and the emitted
$GITHUB_OUTPUT is asserted: a failed build is in neither set, and the
`skip_build` slot is in `services` but NOT in `images` (handing it to the
guard fails "manifest unknown" and blocks the redeploy for the whole fleet).
Same treatment for the starter lane.
Fixtures joined to real registry output
- `extractRevisionLabel` was pinned to hand-written payloads that were never
compared with reality. Since `readLatestRevision` maps every failure to
null, and null ADVANCES, a parser that silently never matches yields a
permanently-blind guard with a fully green suite. Both fixtures are now
verbatim `docker buildx imagetools inspect --format '{{json .Image}}'`
output (buildx v0.35.0), unformatted so key order survives: a platform-keyed
multi-arch image carrying the label, and our own `:latest`, which turns out
to be a bare config object with NO labels at all. The label key used in the
injection test is read from the workflow's own `labels:` input, so parser
and producer cannot drift apart silently.
Closed vacuities
- The `:latest` ban read one step's `tags` and was vacuously green on an empty
list. It now covers every tagged step plus hand-rolled `docker push`/`docker
tag`/`imagetools create`, and fails on an empty `tags`.
- The `cancel-in-progress` ban read only top-level config; a job-level
`concurrency` on `build` reproduced the exact harm and stayed green. All
jobs are checked now.
- `already-current` is keyed on the digest, matching the script.
- The unreachable `(null, "ahead")` row is labelled as the defensive
input-space case it is, and a new test pins that the real flow never calls
compare() with an unknown revision.
New coverage: classifyProbeFailure (incl. a REAL execFileSync timeout, and
the two precedence traps — gh's rate-limited 403 is throttling, not auth; a
registry's 404-with-denied is auth, not absent), readFlag, escapeAnnotationData
(a forged `\n::error::` stays inert), isDirectInvocation through a symlink,
the `::error` annotation on fleet failure, and the digest-mismatch advance.
Also: GITHUB_SHA / GITHUB_REPOSITORY are no longer shadowed in the workflow.
The runner exports both and the script reads process.env, so the old
assertions pinned a redundancy rather than a capability — removed together,
as the comment there required. Workflow-reading scaffolding duplicated with
redeploy-guard.test.ts is extracted and the YAML parse memoized.
Build run 30162773601 (merge of #6160) forced a full-fleet rebuild; 5 of
28 slots were killed by their `timeout-minutes` budget, the other 23
built and WERE redeployed to staging, and the run emitted no signal at
all: `notify` was skipped, so no Slack alert and no PR comment, and the
run rolled up to conclusion `cancelled`.
No existing guard could catch it. Measured on purpose-built probe run
30166429073 (matrix leg killed by `timeout-minutes`, sibling leg green):
killed leg `job.status` ........ cancelled
matrix rollup `needs.*.result` . cancelled
`if: cancelled()` .............. SKIPPED (evaluated FALSE)
`if: failure()` ................ SKIPPED (evaluated FALSE)
pre-fix `notify` condition ..... SKIPPED <- the bug
post-fix `notify` condition .... RAN <- the fix
run conclusion ................. cancelled
So `failure() || cancelled()` would NOT have fixed this. The signal has
to come from the per-slot build results.
- stop laundering `cancelled` into `skipped` in the per-slot writer
- expose `any_cancelled` / `cancelled_services` from the aggregator job
- add `notify-cancelled-builds`: exits non-zero so the run concludes
`failure` rather than `cancelled` (a slot killed by its timeout budget
is a failure, and `cancelled` is what suppressed everything), and
Slacks the affected service names
- add the `any_cancelled` clause to `notify` so the merge author gets the
PR comment, with wording that distinguishes incomplete from failed
`!cancelled()` is retained on both jobs as the intentional-vs-flake
discriminator: a human cancelling the whole RUN makes `cancelled()` true
and stays silent, while a leg-level timeout leaves it false and alerts.
Extends redeploy-guard.test.ts, which evaluates the LIVE `if:` strings
from the workflow, with the exact production scenario. It pins the
pre-fix guard string as a literal so the test proves the difference the
fix makes, not merely the current behaviour.
The notify-all-builds-failed and notify jobs keyed off a 'failure' rollup /
bare failure(), so a build where every real service failed but one leg was
cancelled (contention) rolled up to 'cancelled' and sent no alert — the same
blind spot as the redeploy guard. Fire on any_success == 'false' (guarded by a
status function so a user-cancelled run stays silent). Extends the guard test.
The redeploy-staging and redeploy-staging-starters jobs guarded on
needs.build.result != 'cancelled', so a single cancelled matrix leg (e.g. the
Git-LFS shell build under runner contention) skipped the whole fleet's staging
redeploy even when the other 27 services built fine. Relax both guards to
redeploy the already-computed successful-service list. Adds a guard-evaluation
test that reads the live workflow if: strings and models GitHub's matrix rollup.
Jordan Ritter