## The race, with evidence
Three PRs merged within 34 seconds on 2026-07-26. `Showcase: Build &
Push` has **no concurrency group** (deliberately), so all three ran
simultaneously and raced to push the same `:latest` tags.
| run | commit | PR | start → end |
|---|---|---|---|
| `30190815370` | `7b28934387` | #6162 | 06:18:05 → 06:29:44 |
| `30190823203` | `59f275eedc` | #6161 | 06:18:21 → 06:29:44 |
| `30190831480` | `db75a04837` | #6158 | 06:18:39 → **06:29:13** ←
newest, finished FIRST |
The newest commit finished first, so the older builds overwrote its
`:latest`. Per-service job completion times — older beating newer on
every shared slot:
| service | newer (`db75a04837`) | older (`59f275eedc`) | older won by |
|---|---|---|---|
| `shell-dashboard` | 06:25:33 | 06:25:34 | +1s |
| `showcase-harness` | 06:27:51 | 06:27:55 | +4s |
| `shell` | 06:27:17 | 06:27:28 | +11s |
| `shell-dojo` | 06:25:10 | 06:25:26 | +16s |
**All three runs reported `success`.** Staging served pre-#6158 code
while CI, the redeploy gate and deploy verification all looked clean.
Same failure class as #6171: a success that doesn't mean what it says.
## What I verified in YAML vs took on trust
Verified by reading the files / querying the API:
- **Tagging** — `showcase_build.yml` pushed `:latest` **and** `:${{
github.sha }}` in one `depot/build-push-action` step, in both the
`build` and `build-starters` matrices. **A per-commit sha tag already
existed**; confirmed in GHCR (`showcase-shell-dashboard` has digests
tagged `db75a04837…`, `59f275eedc…`, `d28384a2eb…`).
- **Nothing serialized the pushes.** No concurrency group; confirmed the
header comment states this is intentional.
- **Deploy consumes `:latest`** — `verify-railway-image-refs.ts` is the
SSOT assertion: staging is `ghcr.io/copilotkit/<repo>:latest` (mutable),
**prod is `ghcr.io/copilotkit/<repo>@sha256:<digest>` (already immutably
pinned)**. So this race is **staging-only**; prod was never exposed.
- **`Showcase: Verify Deploy` structurally cannot catch it.** It is a
health probe; it asserts no digest or commit provenance anywhere. Its
#6171 per-commit concurrency key is about *which run verifies*, not
*what image is running*. A stale-but-healthy service passes.
- **The racing runs build DISJOINT service sets** (see below) — I pulled
the actual job lists.
Taken on trust: nothing material. The issue description matched the API
on every point I checked.
## Why NOT a concurrency group
`detect-changes` builds a **per-push, path-filtered** matrix, so
concurrent runs build overlapping but **non-identical** service sets:
- `7b28934387` → ag2, agno, built-in-agent, claude-sdk-python,
claude-sdk-typescript, crewai-crews, langgraph-fastapi,
langgraph-python, langroid, llamaindex, mastra, pydantic-ai, shell-docs,
spring-ai, strands (**15**)
- `db75a04837` → crewai-crews, llamaindex, shell, shell-dashboard,
shell-docs, shell-dojo, showcase-harness (**7**)
`cancel-in-progress: true` would have cancelled the `7b28934387` run and
the ~10 services **only it builds would never have shipped** — trading a
stale-image bug for a never-shipped bug. Per [GitHub's
docs](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#concurrency),
`cancel-in-progress: false` is no better: *"Any previously pending job
or workflow in the concurrency group will be canceled"* — with three
rapid merges the **middle** commit's build is dropped outright. GitHub
also does not guarantee FIFO ordering of queued runs.
Concurrent runs here are **not redundant**, so they must not be
cancelled.
**Re the #6171 interaction:** this design does **not** use
`cancel-in-progress`, so no build is ever superseded-and-cancelled and
the cancelled-slot notifier is never tripped by this change. That
interaction stays theoretical — deliberately.
## The design
Make the one shared mutable resource monotonic instead of serializing
the fleet.
1. The build step pushes **only** the immutable `:<sha>` tag, plus an
`org.opencontainers.image.revision` label.
2. A guard resolves the commit behind the current `:latest`, asks GitHub
`compare/<theirs>...<ours>`, and advances `:latest` (registry-side
retag, no pull) **unless ours is `behind`** — i.e. `:latest` already
holds a descendant and moving it would roll staging back.
**Fails open by design.** No `:latest` yet, unlabelled legacy image,
unreachable API, diverged history → advance. A stuck `:latest` is the
very failure being fixed, so it declines only on *positive proof* of
regression.
**Placement:** in `redeploy-staging` / `redeploy-staging-starters`,
immediately before the Railway pull that consumes `:latest` — not per
build slot. Those jobs already have Node (build slots do **not**, so
per-slot would mean an unpinned `npx tsx` fetch on ~50 parallel
runners), and deciding right before the pull makes the window as narrow
as possible. The image list is the **same matrix ∩ build-success
intersection** that decides what gets redeployed, so a failed build can
never move a tag.
Also adds `showcase_build.yml` to `showcase_validate.yml`'s trigger
paths — the new test asserts against that file's live text, and without
the path a PR re-adding `:latest` would never run the test that catches
it.
## Tradeoffs / what stays open
- **Residual sub-second TOCTOU.** GHCR has no compare-and-swap on tags,
so two runs reading `:latest` simultaneously could still both advance.
This narrows the window from the whole build (~10 min) to inspect→retag.
Fully closing it means retiring the mutable staging tag and pinning
staging to digests the way prod already is — a change to the Railway
image-ref SSOT contract, not a workflow change. **Recommended
follow-up.**
- **The guard is inert for one build per image.** Today's `:latest`
images carry no labels (verified: `showcase-shell-dashboard:latest` has
no `Labels` at all), so the first post-merge build fails open and
advances unconditionally — same as today. Protection starts from the
second build of each image.
- Failure to retag exits non-zero, redding the redeploy job and stopping
the deploy. That is intended: redeploying against a tag that did not
move is exactly the silent false green being fixed.
## Proof
**Red/green on the live YAML.** `advance-latest-tag.test.ts` parses the
real `showcase_build.yml` (extending #6171's `redeploy-guard.test.ts`
pattern). Reverting the workflow to its pre-fix state: **14 failed / 23
passed**. With the fix: **37 passed**. Full `showcase/scripts` suite:
**2382 passed, 73 files**.
**The load-bearing predicate, verified live against the real incident
commits:**
```
compare/db75a04837...59f275eedc => behind (older run arriving late → DECLINE)
compare/59f275eedc...db75a04837 => ahead (newer run → advance)
compare/db75a04837...db75a04837 => identical
```
**Label reading, verified against a real multi-platform registry image**
— `docker buildx imagetools inspect ghcr.io/astral-sh/uv:latest` piped
through `extractRevisionLabel()` returns
`3010295ae7ff572de459987ad70db315a62ecd61`, matching `jq` exactly. The
platform-keyed shape is handled.
**Shell/jq transforms** exercised directly, including the empty-CSV edge
case (empty → empty, step skipped by its `if:`).
**Lint:** `actionlint` finding counts byte-identical to the pre-change
baseline (no new findings; the 11 pre-existing are unrelated). `zizmor
--min-severity low` with the repo config: **no findings**.
**Typecheck:** both new files are in `showcase/scripts/tsconfig.json`'s
include set and produce **zero** errors. Worth stating plainly: `nx
run-many -t check-types` **does not reach `showcase/scripts`** — the
project isn't in the nx graph and has no `check-types` target (there are
9 pre-existing type errors in sibling files, which is how I confirmed
it). So the typecheck above is mine, not CI's. The *tests* are gated:
`showcase_validate.yml` runs bare `pnpm exec vitest run` in
`showcase/scripts`, which auto-discovers the new file.
### What I could NOT prove
**I did not construct a real concurrent race on scratch branches.**
Doing it faithfully needs two builds pushing the same GHCR repo with
controlled finish ordering, which means merging to `main` — the only
branch the build workflow triggers on. No run IDs for a live race
demonstration; I am not implying one.
Unproven until this runs on main: that `docker buildx imagetools create`
retags cleanly under the runner's GHCR credentials, and that `npx tsx`
behaves in the redeploy jobs (it is already the established invocation
there — `redeploy-env.ts` — so this is low risk, not zero).
## Normal single-merge builds are unaffected
No concurrency group is added, so nothing queues or cancels. A lone
merge finds `:latest` at its own parent → `ahead` → advances, exactly as
before. Cost is one `imagetools inspect` + one `gh api` + one
registry-side retag per built service, in a job that already exists — no
extra job, no extra checkout, no change to build parallelism.
---
Branched from `db75a04837`; #6156/#6159 landed after, so this will need
main merged in before it goes green.
Probable conflict with the concurrent `git lfs pull` work in
`showcase_validate.yml` — my edit there is only the top-level `on:
paths:` list, so it should merge cleanly, but flagging it.
The previous suite was 37/37 green against a workflow that could not have
run at all: the guard step had no `packages: write` and no GHCR login, so
every `imagetools create` would have 401'd. Asserting that a step EXISTS
proves nothing about whether it can succeed. Each change below was checked
by reintroducing the defect and confirming the test goes red.
Registry-auth precondition (the miss that let the above ship)
- Both redeploy jobs must declare `packages: write` AND a `docker/login-action`
step for ghcr.io, ordered BEFORE the guard. Dropping either reds the suite.
The intersection, executed rather than restated
- The old test only checked that the string `images=` appears in the compute
step; swapping its jq for the full matrix — so a FAILED build moves
`:latest` — kept it green. The real `changed` shell now RUNS, against the
real ALL_SERVICES matrix read out of the workflow, and the emitted
$GITHUB_OUTPUT is asserted: a failed build is in neither set, and the
`skip_build` slot is in `services` but NOT in `images` (handing it to the
guard fails "manifest unknown" and blocks the redeploy for the whole fleet).
Same treatment for the starter lane.
Fixtures joined to real registry output
- `extractRevisionLabel` was pinned to hand-written payloads that were never
compared with reality. Since `readLatestRevision` maps every failure to
null, and null ADVANCES, a parser that silently never matches yields a
permanently-blind guard with a fully green suite. Both fixtures are now
verbatim `docker buildx imagetools inspect --format '{{json .Image}}'`
output (buildx v0.35.0), unformatted so key order survives: a platform-keyed
multi-arch image carrying the label, and our own `:latest`, which turns out
to be a bare config object with NO labels at all. The label key used in the
injection test is read from the workflow's own `labels:` input, so parser
and producer cannot drift apart silently.
Closed vacuities
- The `:latest` ban read one step's `tags` and was vacuously green on an empty
list. It now covers every tagged step plus hand-rolled `docker push`/`docker
tag`/`imagetools create`, and fails on an empty `tags`.
- The `cancel-in-progress` ban read only top-level config; a job-level
`concurrency` on `build` reproduced the exact harm and stayed green. All
jobs are checked now.
- `already-current` is keyed on the digest, matching the script.
- The unreachable `(null, "ahead")` row is labelled as the defensive
input-space case it is, and a new test pins that the real flow never calls
compare() with an unknown revision.
New coverage: classifyProbeFailure (incl. a REAL execFileSync timeout, and
the two precedence traps — gh's rate-limited 403 is throttling, not auth; a
registry's 404-with-denied is auth, not absent), readFlag, escapeAnnotationData
(a forged `\n::error::` stays inert), isDirectInvocation through a symlink,
the `::error` annotation on fleet failure, and the digest-mismatch advance.
Also: GITHUB_SHA / GITHUB_REPOSITORY are no longer shadowed in the workflow.
The runner exports both and the script reads process.env, so the old
assertions pinned a redundancy rather than a capability — removed together,
as the comment there required. Workflow-reading scaffolding duplicated with
redeploy-guard.test.ts is extracted and the YAML parse memoized.
Reverts only the cosmetic third item from the previous commit. Dropping the
redundant GITHUB_SHA / GITHUB_REPOSITORY shadowing is functionally correct --
the runner exports both and advance-latest-tag.ts reads them off process.env
-- but showcase/scripts/advance-latest-tag.test.ts asserts those exact keys
are declared on the guard step, so removing them red "Validate Showcase".
RED (run 30238078822, "Validate Showcase" > Run build pipeline tests):
Failed Tests 2
FAIL advance-latest-tag.test.ts > showcase_build.yml - redeploy-staging job
> runs the guard with the image list, sha, repo and a token
FAIL advance-latest-tag.test.ts > showcase_build.yml - redeploy-staging-starters
job > runs the guard with the image list, sha, repo and a token
AssertionError: expected undefined to be '${{ github.sha }}'
- Expected: "${{ github.sha }}"
+ Received: undefined
advance-latest-tag.test.ts:424
GREEN (same six contract assertions per job, replicated against the YAML --
the test is pure yaml-parse, no runtime deps):
redeploy-staging / redeploy-staging-starters
PASS IMAGES PASS GITHUB_SHA PASS GITHUB_REPOSITORY
PASS GH_TOKEN truthy PASS guard `if` PASS guard BEFORE redeploy
The test file is owned by a concurrent agent on this PR, so the assertions
cannot be relaxed here. If the redundancy is worth removing, the env keys and
the assertions must go together in one change.
Everything else from the previous commit is untouched and was already
validated by that CI run -- only these two assertions failed, so the GHCR
login, `packages: write`, the skip_build/images filter, the timeout bump and
the tsx pin all passed.
Lint unchanged: actionlint 10 findings before and after (0 new); zizmor
CI-equivalent reports "No findings" before and after.
Two Critical defects in the new "Advance :latest" guard, both of which would
have broken staging redeploys on this PR's own merge commit.
1. Registry auth missing. `redeploy-staging` and `redeploy-staging-starters`
run `docker buildx imagetools create` (an authenticated GHCR WRITE) but
declared only `permissions: contents: read` with no `docker/login-action`
step. Every retag would 401, the guard would exit 1, the redeploy step
would be skipped -- and since the build step no longer pushes `:latest`
itself, `:latest` would freeze permanently and staging would never
redeploy again. The same missing auth also 401s `imagetools inspect` on
these private packages, which the guard treats as "unknown revision" and
advances anyway: it would have failed OPEN and never actually guarded.
Fixed by adding `packages: write` plus a GHCR login mirroring the `build`
job's existing "Login to GHCR" step. Restores both the read and write path.
2. skip_build slots entered the retag list. The `images` CSV projection did
not exclude `skip_build` slots. `webhooks` is built out-of-band, never
pushes a `:<sha>` tag, yet reports job.status: success -- so it reached the
guard, `imagetools create` failed with "manifest unknown", and the whole
fleet's redeploy was blocked. Fires on `service=all` dispatch and via the
`workflow_config` path filter.
Fixed by carrying `skip_build` through the jq projection and filtering it
out of `images` ONLY. It stays in `services`: bouncing webhooks so Railway
re-pulls its out-of-band `:latest` is the documented intent (see the
`webhooks` entry in showcase/scripts/railway-envs.ts).
Also, same file:
- timeout-minutes 5 -> 20 on both jobs. The guard adds up to 3 serial
round-trips per slot (~84 at full fleet width). An overrun concludes
`cancelled`, which is invisible to both `if: failure()` and
`if: cancelled()` downstream -- the alerting-suppression class this repo
has been fixing.
- `npx tsx` -> `npx --yes tsx@4.21.0` (the pnpm-lock-resolved version),
matching the no-drift convention showcase_validate.yml enforces via
`pnpm exec tsx`. `pnpm exec` is unavailable here: no workspace install.
- Dropped the redundant GITHUB_SHA / GITHUB_REPOSITORY shadowing; the runner
exports both and advance-latest-tag.ts reads them off process.env.
Deliberately NOT added: a `concurrency` group. Its absence is load-bearing --
detect-changes builds a path-filtered per-push matrix, so concurrent runs
build disjoint service sets and cancelling the older run would leave services
never shipped rather than merely stale.
RED/GREEN
---------
Workflow auth cannot be fully proven without merging, so the proof is scoped
to what is actually demonstrable locally. Both probes lift the code under test
VERBATIM out of the workflow at runtime rather than paraphrasing it.
Probe A -- the real jq projection, fed the real ALL_SERVICES array with a
`service=all` / `workflow_config` dispatch (every slot selected, every slot
status:success, which is exactly what the skip_build slot reports):
RED images : ...,ghcr.io/copilotkit/showcase-eval-webhook (28 entries)
services: ...,webhooks
-> webhooks IS in the retag set. imagetools create -> manifest unknown.
GREEN images : ends ...,ghcr.io/copilotkit/showcase-pocketbase (27 entries)
services: ...,webhooks (unchanged, still 28)
-> webhooks OUT of images, STILL IN services. Exactly the split wanted.
Probe B -- structural parse of the workflow, over every job that runs
advance-latest-tag.ts:
RED redeploy-staging permissions={'contents':'read'} login=NO
redeploy-staging-starters permissions={'contents':'read'} login=NO
VERDICT: 2 retag job(s) lacking GHCR auth
GREEN redeploy-staging permissions={'contents':'read',
'packages':'write'} login=YES
redeploy-staging-starters permissions={'contents':'read',
'packages':'write'} login=YES
VERDICT: 0 retag job(s) lacking GHCR auth
Lint (delta vs the pre-fix file, not raw counts)
------------------------------------------------
actionlint : 10 findings before, 10 after; 0 new, 0 removed (all pre-existing
depot runner-label + SC2086 noise).
zizmor : CI-equivalent (--min-severity low, default persona, repo config)
reports "No findings" both before and after. Auditor persona at
all severities is also identical before/after (1 concurrency-limits,
1 template-injection, 3 undocumented-permissions, 6
anonymous-definition, 9 secrets-outside-env) -- the two new
`packages: write` entries carry trailing justification comments so
they add no undocumented-permissions findings.
Three merges landed on main within 34 seconds on 2026-07-26. The build
workflow has no concurrency group by design, so all three ran at once and
raced to push the same `:latest` tags. The NEWEST commit's build finished
FIRST, so the two older builds overwrote it:
run 30190815370 7b28934387 (#6162) 06:18:05 -> 06:29:44
run 30190823203 59f275eedc (#6161) 06:18:21 -> 06:29:44
run 30190831480 db75a04837 (#6158) 06:18:39 -> 06:29:13 <- NEWEST
Per-service, the older commit beat the newer one on every shared slot:
shell-dashboard (+1s), showcase-harness (+4s), shell (+11s), shell-dojo
(+16s). All three runs reported `success`. Staging served pre-#6158 code
while CI, the redeploy gate and deploy verification all looked clean.
A concurrency group is the wrong fix. `detect-changes` builds a per-push,
path-filtered matrix, so concurrent runs build overlapping but NON-IDENTICAL
service sets -- 7b28934387 was the only run building ag2, agno, langroid,
spring-ai, strands and 10 others. `cancel-in-progress: true` would have
dropped those entirely, trading a stale-image bug for a never-shipped bug.
`cancel-in-progress: false` is no better: GitHub keeps at most one pending
run per group and cancels any previously-pending one.
Instead make the mutable pointer monotonic. The build step now pushes only
the immutable `:<sha>` tag plus an `org.opencontainers.image.revision`
label. A new guard then advances `:latest` unless the tag already points at
a DESCENDANT of the commit being built, which is exactly the regression
case. Every ambiguous state (no tag, unlabelled legacy image, unreachable
API, diverged history) advances -- a stuck `:latest` is the failure mode
being fixed, so the guard declines only on positive proof of regression.
The guard runs in redeploy-staging / redeploy-staging-starters, immediately
before the Railway pull that consumes `:latest`, over the same
matrix-intersect-build-success set that decides what gets redeployed. That
placement keeps the read->retag window minimal and avoids an unpinned
`npx tsx` fetch on ~50 parallel build slots that have no Node.
Also adds showcase_build.yml to showcase_validate.yml's trigger paths: the
new test asserts against that file's live text, and without the path a PR
re-adding `:latest` to the push step would never run the test that catches it.
Prod is unaffected -- verify-railway-image-refs.ts already pins prod to
`@sha256:<digest>`; only staging consumes the mutable tag.
Root cause of the cancellations, from the Depot logs of the killed slots:
they were not a transient flake. Every one died at EXACTLY its
`timeout-minutes` value (shell/shell-docs/shell-dashboard/shell-dojo at
10m04s of a 10-minute budget; showcase-aimock at 5 minutes), and the log
ends with `failed to solve: Canceled: context canceled` followed by
Depot's `Step canceled by GitHub`. GitHub reports a `timeout-minutes`
kill as job conclusion `cancelled`, which is what fed the whole silent
chain.
The builds were making real but slow progress the entire time — repeated
docker.io base-image pull stalls, and the final `generate-registry` layer
alone took 54s — and shell was still 2 steps from done when the budget
killed it. The `context: "."` slots build the whole monorepo root and are
the heaviest in the fleet, yet they carried the SMALLEST budgets (10 min
vs 15 for each small per-integration build). Measured, same slots:
warm / uncontended ......... 2-5 min
contended, killed at ....... 10 min (runs 30162773601, 30162770765)
showcase-pocketbase ........ 8.8 min of a 10 min budget (near-miss)
So a retry is the wrong lever twice over: a cancelled job cannot run
further steps, and the build was not erroring. Give the four shell slots
20, showcase-aimock 12, and showcase-pocketbase 15. `webhooks` stays at 5
(it is skip_build, measured at 0.6 min).
This is mitigation, not the root fix — the underlying cause is Depot
builder contention from three concurrent full-fleet rebuilds (~84
simultaneous amd64 builds). Noted inline as a follow-up, since the
durable lever is a non-cancelling concurrency queue and this workflow
deliberately has no concurrency group.
Build run 30162773601 (merge of #6160) forced a full-fleet rebuild; 5 of
28 slots were killed by their `timeout-minutes` budget, the other 23
built and WERE redeployed to staging, and the run emitted no signal at
all: `notify` was skipped, so no Slack alert and no PR comment, and the
run rolled up to conclusion `cancelled`.
No existing guard could catch it. Measured on purpose-built probe run
30166429073 (matrix leg killed by `timeout-minutes`, sibling leg green):
killed leg `job.status` ........ cancelled
matrix rollup `needs.*.result` . cancelled
`if: cancelled()` .............. SKIPPED (evaluated FALSE)
`if: failure()` ................ SKIPPED (evaluated FALSE)
pre-fix `notify` condition ..... SKIPPED <- the bug
post-fix `notify` condition .... RAN <- the fix
run conclusion ................. cancelled
So `failure() || cancelled()` would NOT have fixed this. The signal has
to come from the per-slot build results.
- stop laundering `cancelled` into `skipped` in the per-slot writer
- expose `any_cancelled` / `cancelled_services` from the aggregator job
- add `notify-cancelled-builds`: exits non-zero so the run concludes
`failure` rather than `cancelled` (a slot killed by its timeout budget
is a failure, and `cancelled` is what suppressed everything), and
Slacks the affected service names
- add the `any_cancelled` clause to `notify` so the merge author gets the
PR comment, with wording that distinguishes incomplete from failed
`!cancelled()` is retained on both jobs as the intentional-vs-flake
discriminator: a human cancelling the whole RUN makes `cancelled()` true
and stays silent, while a leg-level timeout leaves it false and alerts.
Extends redeploy-guard.test.ts, which evaluates the LIVE `if:` strings
from the workflow, with the exact production scenario. It pins the
pre-fix guard string as a literal so the test proves the difference the
fix makes, not merely the current behaviour.
Demo assets stay uniformly LFS-tracked. The build is the single place this
gets resolved, so every workflow that checks out and then serves or bakes
integration `public/` assets now fetches LFS objects.
`showcase_build.yml` (the workflow that builds and pushes the deployed
images) has hardcoded `lfs: true` since 7bde1eef3a, so the deploy path was
already correct. Two paths were not:
- `showcase_build_check.yml` read `lfs: ${{ matrix.service.lfs }}`, which was
`false` for every integration, so the pre-merge build check baked pointer
stubs into the images it built. Build-only, so nothing deployed, but it
meant PR CI could never catch a broken demo asset.
- `test_e2e-showcase-on-demand.yml` passed no `lfs` input at all. That job
runs `next dev` on the integration package and drives it with Playwright,
serving `public/` straight from the working tree, so `multimodal.spec.ts`
would click "Try with sample image", receive a ~130-byte pointer stub as
`image/png`, and fail on the magic-byte guard — with nothing wrong in the
code under test. Nine integrations ship that spec.
Both now use a uniform `lfs: true`.
Also deletes the per-slot `"lfs"` field from both `ALL_SERVICES` definitions.
After the above there were no consumers left: `showcase_build.yml` ignored it
in favour of a hardcoded `true`, and `showcase_build_check.yml` was its only
reader. Leaving a dead flag that reads as authoritative is how this was
mis-set in the first place, and deleting it means a new integration cannot be
added with LFS fetching silently off — there is no flag to forget.
No asset or .gitattributes changes: LFS is used as intended and a committed
pointer remains the correct on-disk state for an LFS-tracked file.
The notify-all-builds-failed and notify jobs keyed off a 'failure' rollup /
bare failure(), so a build where every real service failed but one leg was
cancelled (contention) rolled up to 'cancelled' and sent no alert — the same
blind spot as the redeploy guard. Fire on any_success == 'false' (guarded by a
status function so a user-cancelled run stays silent). Extends the guard test.
The redeploy-staging and redeploy-staging-starters jobs guarded on
needs.build.result != 'cancelled', so a single cancelled matrix leg (e.g. the
Git-LFS shell build under runner contention) skipped the whole fleet's staging
redeploy even when the other 27 services built fine. Relax both guards to
redeploy the already-computed successful-service list. Adds a guard-evaluation
test that reads the live workflow if: strings and models GitHub's matrix rollup.
## The incident
PR #6061 (`fix(starters): add python-multipart to agno starter`) merged,
its
`starter-agno` image built to GHCR via `showcase_build.yml`'s
`build-starters`
job — and then was **never deployed to Railway**. `starter-agno` stayed
crashed
until someone manually redeployed. That is not a one-off: it is
structural. Any
change touching **only** starter files
(`examples/integrations/<slug>/**`) hits
the same hole.
## Root cause (job-graph)
`showcase_build.yml` has two independent lanes:
| Lane | Detect | Build | Aggregate | Redeploy |
| --- | --- | --- | --- | --- |
| **Showcase fleet** | `detect-changes` | `build` (matrix) |
`aggregate-build-results` | `redeploy-staging` |
| **Starters** | `detect-starter-changes` | `build-starters` (matrix) |
— | **(none)** |
`redeploy-staging` is scoped **entirely to the showcase fleet**:
```
redeploy-staging:
needs: [detect-changes, build, aggregate-build-results]
if: >-
!cancelled()
&& needs.detect-changes.outputs.has_changes == 'true'
&& needs.build.result != 'skipped'
&& needs.build.result != 'cancelled'
&& needs.aggregate-build-results.outputs.any_success == 'true'
```
(`.github/workflows/showcase_build.yml:780-799`)
On a **starter-only** push:
- `detect-changes` filters are all `showcase/**` paths → none match
`examples/integrations/**` → `has_changes=false` → `build` skips →
`aggregate-build-results` skips → `redeploy-staging` skips (fails its
`has_changes=='true'` and `build.result != 'skipped'` clauses).
- `detect-starter-changes` matches → `build-starters` builds
`starter-<slug>:latest` to GHCR — **and stops.** There is no aggregate
and
**no redeploy job for the starter lane at all.**
Net: the starter image is built and pushed, but nothing ever calls
Railway's
`serviceInstanceRedeploy`, so the running container keeps the stale
image.
The `starter-build-result-*` per-slot artifacts already emitted by
`build-starters` were **write-only** — nothing consumed them.
## The fix
A new `redeploy-staging-starters` job that mirrors `redeploy-staging`
for the
starter lane, reusing existing mechanisms (no new script, no new
aggregator):
```
redeploy-staging-starters:
needs: [detect-starter-changes, build-starters]
if: >-
!cancelled()
&& needs.detect-starter-changes.outputs.has_changes == 'true'
&& needs.build-starters.result != 'skipped'
&& needs.build-starters.result != 'cancelled'
```
Steps:
1. Download the already-emitted `starter-build-result-*` artifacts
(`pattern` download → succeeds with zero matches if the build crashed
before writing any).
2. Compute `matrix ∩ build-success`: read each per-slot
`{"service":"<raw slug>","status":...}`, keep `status:success` slugs,
map
each **raw slug → `starter-<slug>` SSOT key** via the starter matrix
`.image` field. (The raw slug must NOT be passed to `redeploy-env.ts` —
e.g. `"agno"` collides with the **showcase** `agno` dispatch_name.)
3. If the CSV is non-empty, `npx tsx showcase/scripts/redeploy-env.ts
staging
--services <csv>` — the exact same invocation `redeploy-staging` uses.
`redeploy-env.ts` already resolves each `starter-<slug>` as an SSOT key
(verified: all 12 starter `.image` values exist as keys in
`railway-envs.ts`).
**Deploy-on-failure is impossible:** the "don't deploy on build failure"
guard
is the per-slot success intersection, not the job `if:`.
`build-starters`
`result == 'failure'` still enters the job (fail-fast is false, so some
slots
may have succeeded), but only `status:success` slots are redeployed; an
all-failed or crashed build yields an empty CSV → the redeploy step is
skipped.
This is the same net guarantee `redeploy-staging` gets from its
`any_success`
guard, computed inline to avoid standing up a second aggregator job.
**No `redeploy-summary` artifact is written** by this job on purpose:
that
name is owned by `redeploy-staging` and downloaded by
`showcase_deploy.yml` by
exact name — a second same-named upload would collide on a combined
push.
Starter *staging verification* is intentionally out of scope for this
deploy-gap fix (starters are already smoke-covered by
`test_smoke-starter.yml`
and the harness `starter_smoke` axis). `redeploy-staging-starters` was
also
added to the `notify` job's `needs` so a starter redeploy failure
alerts.
## Before / after truth table
| Scenario | `build` | `redeploy-staging` | `build-starters` |
`redeploy-staging-starters` |
| --- | --- | --- | --- | --- |
| **(a) main-fleet-only change** | runs | **redeploys fleet** | skipped
| skipped |
| **(b) starter-only change** | skipped | skipped | runs | **redeploys
starter (NEW)** |
| **(c) both changed** | runs | **redeploys fleet** | runs | **redeploys
starter (NEW)** |
| **(d) starter build failure** | (n/a) | (n/a) | failure | runs, but
CSV empty → **no redeploy** |
- **(a)** unchanged — the showcase lane is untouched.
- **(b)** is the fix: the starter now auto-deploys instead of sitting on
GHCR.
- **(c)** unchanged for the fleet; the starter additionally deploys. No
artifact
collision because `redeploy-staging-starters` uploads no
`redeploy-summary`.
- **(d)** partial failure redeploys only the slots that succeeded; a
wholesale
failure redeploys nothing.
## Fail-loud hardening (CR follow-up)
A CR flagged that the new `redeploy-staging-starters` job could itself
**silently under-deploy** — re-opening the very hole it exists to close.
Two
guards added to the `Compute successfully-built starter services` step,
mirroring
the sibling `redeploy-staging` job's empty-intersection guard:
1. **Empty deploy-set → fail loud.** When `build-starters.result ==
'success'`
(all slots built) but the `matrix ∩ success` CSV is **empty**, the job
now
`exit 1`s with an actionable `::error::` instead of silently skipping
the
redeploy at green CI (a slug↔`.image` contract skew, or a success set
that
maps to no matrix entry). Gated on `'success'` so a partial/total build
**failure** keeps the legitimate no-deploy path and is not
double-reported —
that failure is already surfaced by `build-starters` itself.
2. **Missing/unreadable result artifact → fail loud.** Dropped
`2>/dev/null` on
the `result.json` read so a read error surfaces and trips `pipefail`,
and
added a parsed-record-count vs matrix-slot-count assertion (every slot
writes
a `result.json` via `if: always()`, so on a full-success build the
counts must
match). A missing/expired `starter-build-result-*` artifact — which
would
otherwise silently drop a built starter from the redeploy set — now
fails the
job. Also gated on `'success'` so a crashed slot's legitimately-absent
artifact isn't double-reported on a build failure.
Updated truth table with the new fail-loud row:
| Scenario | `build-starters.result` | deploy CSV |
`redeploy-staging-starters` |
| --- | --- | --- | --- |
| success + non-empty CSV | success | non-empty | **redeploys** |
| success + **empty** CSV | success | empty | **exit 1 (NEW fail-loud)**
|
| build failure (partial) | failure | subset | redeploys successful
subset, no spurious exit |
| build failure (all/crash) | failure | empty | no deploy, failure
surfaced, no spurious exit |
| skipped (no starter changes) | skipped | (n/a) | job `if:` excludes it
— never runs |
Both guards were locally red→green exercised: pre-fix the empty-CSV and
missing-artifact cases went **green with nothing/partial deployed**;
post-fix
they `exit 1`. actionlint still 9/9 (no new findings).
## actionlint
Clean. Baseline (`origin/main`) = 9 findings; this branch = 9 findings,
all at
pre-existing lines (custom `depot-*` runner label + pre-existing SC2086
infos).
**Zero new findings** from the added job. The `matrix ∩ success` jq
mapping was
locally exercised across the four scenarios above
(success/partial/all-fail/
crash-before-artifacts) and produced the expected CSVs.
## Prod-promote path
**Does NOT share the gap.** `showcase_promote.yml` is
`workflow_dispatch`-only
("Humans trigger. No automatic prod promotes.") and already lists all 12
`starter-*` services in its choices with `resolve-targets` handling
them. There
is no push-driven prod path to fix.
## Residual verification (honest note)
Workflows can't be safely dry-run end-to-end (the redeploy path hits
live
Railway). Static validation is complete (actionlint clean, jq logic
exercised,
all 12 starter SSOT keys confirmed, `redeploy-env.ts` reused unchanged),
but
the true end-to-end confirmation is the **next starter-only change
auto-deploying to staging**. That first real starter-only merge after
this
lands should be watched to confirm `redeploy-staging-starters` fires and
the
Railway service picks up the new image.
---
Draft — do not merge until reviewed.
The new redeploy-staging-starters job pinned actions/checkout to
9c091bb (tag v7.0.0) but commented it # v7. zizmor's ref-version-mismatch
flagged the discrepancy: the v7 moving tag points to 3d3c42e, not 9c091bb.
Repin to 3d3c42e # v7 — the canonical checkout pin already used across
every other job on main — so the comment matches the SHA's tag.
The redeploy-staging-starters job could go green while deploying
nothing (all starters built, empty deploy CSV) or silently drop a
built-but-missing starter from the redeploy set. Both re-open the exact
gap this job exists to close.
- Finding 1: when build-starters.result == 'success' but the matrix ∩
success CSV is empty, fail the job (exit 1) instead of silently
skipping the redeploy. Mirrors redeploy-staging's empty-intersection
guard. Gated on 'success' so a partial/total build FAILURE keeps the
legitimate no-deploy path and isn't double-reported.
- Finding 2: drop 2>/dev/null on the result.json read (surface read
errors via pipefail) and assert parsed-record count == matrix slot
count when all slots built, so a missing/expired starter-build-result-*
artifact fails loud instead of silently under-deploying.
A push touching only starter files (examples/integrations/<slug>/**) built a
fresh starter-<slug>:latest image to GHCR via build-starters but never
redeployed it to Railway: the starter lane (detect-starter-changes ->
build-starters) ended at the GHCR push, and redeploy-staging only covers the
showcase build lane. Starter fixes sat undeployed until a manual redeploy
(the #6061 agno incident).
Add a redeploy-staging-starters job that mirrors redeploy-staging for the
starter lane: it reads the already-emitted per-slot starter-build-result-*
artifacts, intersects the build matrix with the build-success set (mapping raw
slug -> starter-<slug> SSOT key), and redeploys only the successfully-built
starters to staging. No deploy on build failure (empty success set -> redeploy
step skipped). Reuses redeploy-env.ts unchanged.
## Two silent-failure gaps in the showcase build/deploy/notify pipeline
These are **pre-existing** silent-failure holes surfaced in code review
(not
caused by any recent PR). This PR fixes the two load-bearing ones.
### 1. Green-but-zero-redeploy (silent "we thought we shipped but
didn't")
The `redeploy-staging` job computes the redeploy set as the intersection
of the
build matrix and the build-success set. This job **only runs when
`aggregate-build-results.outputs.any_success == 'true'`** (job-level
`if:`
guard). So if that intersection comes back **EMPTY**, it does NOT mean
"nothing
to deploy" — it means at least one slot built successfully yet none of
those
successes maps back to a matrix `dispatch_name`. That's a
`dispatch_name`↔
`service` contract skew (the aggregator's `service` values and the
matrix's
`dispatch_name` values drifted apart).
The old code emitted `services=` (empty) and exited 0 → the build went
**GREEN
while redeploying NOTHING**, silently.
**Fix:** on an empty intersection in this any_success-guaranteed step,
fail loud
(`::error::` + `exit 1`) with a diagnostic naming both sides of the
skew.
The legitimate "nothing changed / nothing succeeded" no-op paths are
guarded at
the **job level** (`has_changes=='true' && any_success=='true'`), so the
fixed
step never runs there — no false-red.
### 2. Starter build failures had no alert surface (invisible failures)
The `notify` job's `needs` (and its `if: failure()`) omitted
`detect-starter-changes` and `build-starters`, and `build-starters`
wrote no
per-slot build-result artifact. So a **failed starter image build
produced NO
Slack alert and NO PR comment** — it shipped silently.
**Fix:**
- Added `detect-starter-changes` + `build-starters` to `notify.needs` so
`if: failure()` sees a starter build failure → Slack alert + PR comment.
- Gave `build-starters` a per-slot build-result artifact **mirroring the
main
`build` matrix** (same `{service,status}` shape, `cancelled→skipped`
normalization, `if: always()`, `if-no-files-found: error`), using a
**distinct `starter-build-result-*` prefix** so it never matches the
aggregator's `build-result-*` download pattern (starters must not
pollute the
showcase redeploy set keyed by `dispatch_name`).
### Red / Green
**Finding #1** — extracted the step's shell/jq logic and drove it with
synthetic
inputs:
RED (pre-fix), any_success=true + empty intersection:
```
No services in matrix ∩ success-set — skipping redeploy.
Computed services CSV (matrix ∩ build-success):
EXIT=0 # $GITHUB_OUTPUT: services= -> silent pass, redeploys NOTHING
```
GREEN (post-fix), same inputs:
```
::error::Build succeeded (any_success=true) but matrix ∩ success-set is EMPTY — dispatch_name/service contract skew; nothing would be redeployed.
Successful build service values: ["shell-RENAMED","mastra-RENAMED"]
Scheduled matrix dispatch_name values: ["shell","mastra"]
EXIT=1 # fails loud
```
No-regression: non-empty intersection → `EXIT=0 ; services=shell`. The
nothing-changed/nothing-succeeded paths are skipped at the job level
(never
reach the step) → no false-red.
**Finding #2** — modeled `if: failure()` (fires iff any `needs` job
result is
`failure`):
```
BEFORE (starters NOT in needs), starter=failure -> notify fires = False (INVISIBLE, the bug)
AFTER (starters IN needs), starter=failure -> notify fires = True (FIXED)
AFTER no-regression, starters=skipped, all green -> notify fires = False (quiet)
```
### Validation
- `python3 yaml.safe_load` parses OK.
- `actionlint`: only pre-existing findings remain (matrix jq SC2086 +
the known
`depot-ubuntu-24.04-4` runner-label warning); no new errors in edited
regions.
- `yamllint`: only pre-existing line-length/document-start/truthy
warnings.
### Scope
Touches **only** `.github/workflows/showcase_build.yml`, and only these
two
concerns. Does NOT touch the `shell_dashboard` paths-filter region (PR
#5955's
domain), nor the other backlog debt (false-root-cause comment,
double-alert,
check-lockfile guard). Self-contained; not stacked on #5955.
Two pre-existing silent-failure gaps in the showcase build/deploy/notify
pipeline (surfaced in code review):
1. Green-but-zero-redeploy: the redeploy-staging job computes the redeploy
set as (build matrix ∩ build-success). This job only runs when
any_success=='true', so an EMPTY intersection means builds succeeded but
none maps to a matrix dispatch_name — a dispatch_name/service contract
skew. The old code emitted an empty services= and exited 0, going GREEN
while redeploying nothing. Now it fails loud with a diagnostic naming both
sides of the skew. The legitimate nothing-changed/nothing-succeeded no-ops
stay guarded at the job level, so they are unaffected.
2. Starter-failure-invisible: the notify job's needs omitted build-starters,
so a failed starter image build produced no Slack alert and no PR comment.
Added detect-starter-changes + build-starters to notify.needs, and gave
build-starters a per-slot build-result artifact mirroring the main build
matrix (distinct starter-build-result-* prefix so it never pollutes the
showcase aggregator's build-result-* set).
PR #5952 (9a8cf615) added explicit `.js` extensions to the relative imports
inside the harness's shared cell-model fold
(showcase/harness/src/shared/cell-model/{cell-model,live-status,staleness}.ts)
— REQUIRED for the harness's pure-Node-ESM runtime and correct as-is.
But the dashboard re-exports that fold via shims
(showcase/shell-dashboard/src/lib/{cell-model,live-status,staleness,format-ts}.ts
`export * from "../../../harness/src/shared/cell-model/*"`), pulling the fold
into the dashboard's `next build`. `export *` does not rewrite the fold's
INTERNAL `.js` edges, and the dashboard's empty next.config.ts had no
extensionAlias, so webpack resolved `./live-status.js` literally, found only
the `.ts` source, and failed:
Module not found: Can't resolve './live-status.js'
Module not found: Can't resolve './staleness.js'
Module not found: Can't resolve './format-ts.js'
> Build failed because of webpack errors
Two-part fix (one coherent subject):
1. Resolution: add `webpack.resolve.extensionAlias` to
showcase/shell-dashboard/next.config.ts so `.js`/`.mjs` specifiers resolve
to `.ts`/`.tsx`/`.mts` sources — the bundler complement to TS NodeNext's
`.js`-import convention. Covers the `next build` (webpack) path CI uses.
The harness fold's `.js` imports are left untouched (they are correct).
2. CI gap: the dashboard build did not run on #5952 because the build matrix
is path-filtered and #5952 only touched `showcase/harness/**`, which
selects `showcase_harness` but not `shell_dashboard`. Add
`showcase/harness/src/shared/**` to the `shell_dashboard` paths-filter so
any change to the shared fold the dashboard compiles in also selects the
dashboard build — a fold change can never again ship an unbuilt dashboard.
Local red-green proof:
- RED (main, before fix): `next build` in showcase/shell-dashboard emitted the
4 fold-resolve errors above.
- GREEN (after extensionAlias): same build → 0 fold-resolve errors; the fold
resolves. Remaining `@/data/*.json` errors are the prebuild-generated files
(generate-registry/probe-docs) skipped in the local repro, produced in CI's
Docker build — unrelated to this fix.
Wire the strands-typescript showcase integration for staging deployment,
mirroring how the Python strands integration is deployed.
- manifest: flip deployed: true so the shell lists it in the integration menu
- railway-envs.ts: add showcase-strands-typescript SSOT entry (staging-only
for now: prod instance not yet provisioned, so it omits the prod env and is
gateIgnore'd until promoted dual-env); regenerate railway-envs.generated.json
- showcase_build.yml + showcase_build_check.yml: add the strands-typescript
build matrix entry, change-detection filter, and dispatch option (railway_id
is the new Railway service id)
- golden fixture + image-ref-gate inventory tests updated for the new service
Railway staging service showcase-strands-typescript provisioned
(showcase-strands-typescript-staging.up.railway.app, health /api/health,
OpenAI-via-aimock env). Prod is added later via the promote pipeline.
GitHub Actions expression string literals don't interpret `\n`, so
`toJSON(format('...\n...'))` emits literal `\\n` and Slack renders the
two characters backslash-n instead of a line break. Inject real newlines
via a `fromJSON('"\n"')` placeholder, matching the starter-smoke fix.
Fixes the "all builds failed" and "Showcase Build Failed" alerts in
showcase_build.yml and the multi-line "showcase_validate failed" alert
in showcase_validate.yml. showcase_promote.yml already used the
fromJSON placeholder; the single-line validate alert has no newlines
and was left untouched.
PocketBase had no CI build path: `ghcr.io/copilotkit/showcase-pocketbase`
was a stale April `:latest`, and there was no way to ship pb_migrations /
pb_hooks changes without an ad-hoc manual build. Add a `pocketbase` slot to
showcase_build.yml's build matrix, mirroring the harness/aimock entries:
- dispatch_name `showcase-pocketbase`, context `showcase/pocketbase`, its
own Dockerfile, health `/api/health`, railway_id from the SSOT.
- a paths-filter key gated to `showcase/pocketbase/**` so the slot only
rebuilds when PB's own files change (the image is self-contained — no
shared-module copy), not on every showcase push.
- the workflow_dispatch service choice so PB is human-targetable.
Flip the SSOT entry (railway-envs.ts) to `ciBuilt: true` with
`dispatchName: "showcase-pocketbase"` so it is built+pushed (`:sha` +
`:latest`) and joins the default staging-redeploy scope; the build's
redeploy step only touches the matrix-intersect-success set, so PB still
only redeploys when its own files change. Regenerate
railway-envs.generated.json and the showcase_promote.yml service dropdown,
and update the SSOT/redeploy tests that pinned PB as out-of-band
(CI_BUILT_SERVICES 25 -> 26; webhooks stays the only non-CI-built service).
showcase_build.yml's existing detect-changes job shared the workflow_dispatch
`service` input with the new detect-starter-changes job, so dispatching
`service=starter-<slug>` tripped detect-changes's fail-loud `exit 1` ("did
not match any entry in ALL_SERVICES") and reddened the run even though
build-starters published fine. Scope the showcase fail-loud to non-`starter-*`
inputs via a case statement (mirroring how the starter job scopes its own
fail-loud) so a `starter-*` dispatch resolves to an empty showcase matrix and
SKIPS; typo'd showcase service names still fail loud.
Restore the 6h `schedule` cron in test_smoke-starter.yml. The PR had removed
it, leaving NO post-merge floating-dependency breakage detector for starters
— and its harness-probe replacement depends on S5 Railway services that don't
exist yet. Keep the cron until S5 starter-service probing is confirmed live.
showcase_build.yml: add detect-starter-changes + build-starters jobs building
each of the 12 starters from examples/integrations/<slug>/Dockerfile via Depot
(--platform linux/amd64) and pushing ghcr.io/copilotkit/starter-<slug>:latest +
:<sha>. The starter- prefix is disjoint from showcase-* so harness discovery
stays clean.
test_smoke-starter.yml: drop the 6h schedule cron (live signal now comes from
the harness probing deployed Railway services + the harness alert path); keep
the examples/integrations/** PR build-sanity gate + offline aimock smoke run.
Workflows slot (S4) of the starter-row-group spec (model B). Railway
provisioning (S5) is intentionally out of scope, gated on cost approval.
Collapse the promote workflow to an input-agnostic concurrency group so promotes can't race the
same Railway service; add #oss-alerts failure notifications to the build and validate workflows
(build via extended needs, validate via a new workflow-level notify job).
Closing hardening pass on the showcase deploy-gate's verify-matrix
resolver. The 7-agent review confirmed the gate is correct; this
commit fixes the residual rough edges.
- showcase_deploy.yml: correct the false §3 ok-non-empty comment.
The empty-intersection case can coexist with redeploy_red=false
(every redeploy succeeded, just none probe-eligible) — that's a
correctly-green run, not a red one.
- showcase_deploy.yml: tighten the summary.json shape guard to catch
PARTIAL drift (TOTAL>0 && WITH_STATUS<TOTAL). The previous all-or-
nothing TOTAL>0 && WITH_STATUS==0 check silently dropped drifted
rows on a mixed summary. Validated locally on mixed/normal/empty/
total-drift jq samples.
- resolve-verify-matrix.ts: add asSupportedEventName narrowing helper
+ use it in the CLI. Replaces the unchecked `as` cast — type system
and runtime now tell one story. Resolver's internal eventName
guard becomes defense-in-depth for direct (test) callers.
- resolve-verify-matrix.ts: make the workflow_run boundary total —
summaryPresent MUST be exactly "true"/"false". Any other value
(including "" from a step-id-rename wiring break) throws now
instead of silently emitting has_services=false.
- resolve-verify-matrix.ts: drop the try/catch around
fileURLToPath(import.meta.url) in `invokedDirectly`. The catch
used to swallow ESM-interop failures and silently no-op the CLI
(exit 0, no GITHUB_OUTPUT write → verify skipped = false-green).
- resolve-verify-matrix.ts: reword parseSsotServices JSDoc to
distinguish schema-drift from truncation (the two are different
failure modes, not one conflated story).
- showcase_build.yml: comment addendum on the redeploy-summary
upload — swapping the guard to `if: always()` would red the
legitimate services=='' path (no summary written), trading the
already-closed false-green for a false-red on every non-buildable
push.
- resolve-verify-matrix.cli.test.ts: switch to spawnSync so stderr
is captured on both zero and non-zero exit (execFileSync only
exposes stderr on throw). Hard-code two stable probe-eligible
names ("aimock", "harness") for the sorted-CSV test rather than
picking probe[0]/probe[1] off the live SSOT — the prior test was
tautological (already-sorted in, sorted out) and would silently
pass if the resolver did nothing.
- resolve-verify-matrix.cli.test.ts: add CLI coverage for the
dropped-token ::warning:: path (FIX 3 — the entire drift-detection
contract had zero CLI coverage), the unexpected-EVENT_NAME error
(FIX 5), and the workflow_run-summary_present total boundary
(FIX 7, both "" and "True" inputs).
- resolve-verify-matrix.test.ts: add unit coverage for the new
workflow_run summaryPresent boundary (empty + "True" + the
workflow_dispatch ignores-summaryPresent regression).
Red-green: 6 tests RED before code changes (FIX 3 warning, FIX 5
unknown EVENT_NAME, FIX 7 unit + CLI ×2 for "" and "True"); 79
tests GREEN after.
Validation: 4 vitest files / 79 tests passing; 87/87 ruby specs
passing; actionlint findings unchanged vs integration baseline
(8 → 8, identical diff); yaml.safe_load OK on both workflows.
A 7-agent review of the verify-matrix resolver and its surrounding workflow plumbing found three
boundary surfaces that could silently produce a GREEN deploy on a broken release, plus an
untested CLI contract that CI compares against the literal strings 'true' / 'false'.
FIX 1 — Validate the SSOT shape in loadSsotServices(). The prior `JSON.parse(...) as
{services: SsotService[]}` was an unchecked cast: a truncated/drifted SSOT (emitter crashed
mid-write, or schema renamed) parses fine but silently shrinks/empties the probe-eligible set
→ some redeployed services go unverified, or verify is skipped on a real redeploy. Extract a
pure exported parseSsotServices(raw, path) that requires the shape we depend on (non-empty
services array; each entry has a non-empty string name, an optional string|null dispatchName,
and a probe object with a boolean staging). Throw `::error::SSOT <path> malformed: <detail>`
on any violation. Also re-check existsSync(SSOT_JSON) after the regenerate-if-missing
execFileSync — a regen that exits 0 without writing must not proceed to a useless JSON.parse
crash. Drop the defensive `probe?.staging` once shape is guaranteed.
FIX 2 — Validate summary.json shape in the redeploy-gate bash. The bullseye false-green
surface: if redeploy-env.ts's schema ever drifts (e.g. `status` → `state`, `ok` → `success`),
every `jq select(.status==...)` yields empty → redeploy_red=false AND ok_services="" →
resolver skips verify → GREEN CI on a real unverified redeploy. Add a TOTAL vs WITH_STATUS
shape guard right after loading the summary: if TOTAL > 0 && WITH_STATUS == 0, emit
::error::summary.json has $TOTAL entries but none with status ok|error (schema drift?) and
exit 1. The legitimate empty-array path (TOTAL=0) is preserved.
FIX 3 — Fail loud on unknown eventName in resolveVerifyMatrix. The prior code fell through to
the workflow_run intersection branch for ANY unrecognized eventName (typo, unexpected
trigger), silently emitting has_services=false → indistinguishable from a legit "summary
absent" skip. Add an explicit guard so only workflow_run / workflow_dispatch are accepted;
anything else throws ::error::resolve-verify-matrix: unexpected eventName '<value>'. Tighten
the eventName parameter type to the literal union.
FIX 4 — Trim ok tokens + warn on dropped tokens in okCsvToCanonicalNames. Split, then
.map(t => t.trim()).filter(Boolean) so "a, b" (spaces) matches. Collect tokens that match NO
SSOT service (by name or dispatchName) and have the CLI wrapper emit ::warning::ok_services
tokens dropped (no SSOT match): <list> on stderr when non-empty — surfaces SSOT/build drift.
The pure function stays IO-free; logging lives in the wrapper.
FIX 5 — CLI wrapper integration test. New resolve-verify-matrix.cli.test.ts spawns
`npx tsx showcase/scripts/resolve-verify-matrix.ts` with a temp $GITHUB_OUTPUT file across
four scenarios and asserts the temp file contents EXACTLY (the workflow YAML compares
has_services against the literal strings 'true'/'false', so the byte-for-byte format is part
of the contract). Uses the real railway-envs.generated.json so the loader exercise is real.
FIX 6 — Cleanup. Remove the dead `env: DISPATCH_SERVICE: ...` block on the redeploy-gate
step (the next step redeclares it — leftover from the extraction). Soften the §3
decision-table all-errors bullet to match resolve-verify-matrix.ts's careful wording, and
append that when the success-set is empty (or the intersection collapses to empty), verify
is skipped and the gate reds independently. Append to showcase_build.yml's "Upload redeploy
summary" path-(A) comment that `if-no-files-found: error` still reds path (A) even if a
future change adds `if: always()`.
Tests: red→green for FIX 1/3/4/5 verified locally. Resolve-verify-matrix vitest count:
12 → 28. Full requested suite (resolve-verify-matrix + cli + aggregate-build-results +
lint-rule-no-public-env): 72 passed. showcase/bin ruby specs: 87 runs / 0 failures / 0
errors / 0 skips. actionlint baseline preserved (8 findings, identical to integration tip).
Extract the inline bash+jq decision logic from showcase_deploy.yml's
resolve-matrix job into showcase/scripts/resolve-verify-matrix.ts, a
pure function with a vitest suite. The bash had produced two confirmed
bugs across prior CR rounds, so making it testable is the lasting fix.
Issue A (the bug this PR fixes): when summary_present=true but
ok_services is empty (every service errored on redeploy), the old bash
skipped the intersection and fell through to the full probe-eligible
fleet, gratuitously probing every service against stale :latest. The
resolver now returns has_services=false in that case — enforce-redeploy
-gate independently reds the workflow on redeploy_red=true, so this
case is already loud; there is nothing left to verify.
Parity preserved for unchanged cases:
- workflow_dispatch + 'all'/empty → full probe-eligible set
- workflow_dispatch + specific svc → that one (unknown → error exit)
- workflow_run + summary_present=false → has_services=false
- workflow_run + present + ok non-empty → intersection with probe-
eligible (SSOT key OR dispatchName aliases both resolve)
Also clarified the Upload-redeploy-summary comment in showcase_build.yml
to document both red paths (hard crash → redeploy step exits non-zero;
exit-0-but-no-file → if-no-files-found:error reds the step) so no
false-green path is possible.
Tests: 12-case vitest suite covers each decision-table row plus the
Issue A fix (written red-first; failed against a naive full-fleet
fallback, passed once the early return was added). CLI parity verified
against the real generated SSOT for the three representative env-var
combinations (workflow_run + present + ok=[a,c]; workflow_run + present
+ ok empty; workflow_dispatch + 'all').
Three correctness holes uncovered by confirmation review of the earlier
deploy-gate fix:
(1) showcase_deploy.yml — Build verify matrix step: when workflow_run fires
with summary_present=false (legitimate "build redeployed nothing", e.g.
docs/script-only push under showcase/**), the gate correctly no-oped but
the matrix fell through the empty-OK_FROM_REDEPLOY branch and resolved to
the FULL probe-eligible set. Verify then ran against the whole staging
fleet for a push that deployed nothing — gratuitous, and false-reds the
deploy workflow if any unrelated staging service happens to be unhealthy
at probe time. Thread github.event_name + summary_present into the step
via env and add an explicit (workflow_run && summary_present==false)
guard that sets services_csv="" / has_services=false. workflow_dispatch
fall-through (full fleet / chosen service) preserved. workflow_run +
summary present + all-errors path unchanged: enforce-redeploy-gate still
trips RED on redeploy_red=true.
(2) showcase_build.yml — Upload redeploy summary step: was gated on
services != '' && hashFiles('.redeploy/summary.json') != ''. If
redeploy-env.ts crashes before writing summary.json (the script is
documented "always exits 0", but a crash/OOM/unhandled-rejection can
skip the write), services != '' but hashFiles == '' silently skipped
the upload. The deploy side then saw "artifact absent", treated it as
"nothing redeployed", skipped the gate, and produced a FALSE GREEN
despite a real redeploy failure. Drop the hashFiles clause so the upload
is mandatory whenever a redeploy was attempted; if-no-files-found:error
(already set) then fails the step → fails the redeploy-staging job →
fails the build workflow → showcase_deploy.yml's resolve-matrix.if
(workflow_run.conclusion == 'success') blocks the deploy run from
starting at all. Loud failure on the build side. The legitimate
services == '' (matrix ∩ success-set empty) path is preserved by the
services != '' guard.
(3) showcase_deploy.yml — check-redeploy-summary github-script: was a
single per_page:100 list call. While the current run uploads ~28
artifacts (well within 100), a future expansion past 100 could push
redeploy-summary off the first page and produce a false "absent" → gate
skipped → false-green. Switch to github.paginate.iterator with the
endpoint's name="redeploy-summary" filter for an exact-match,
pagination-safe lookup. No try/catch is added: github-script propagates
unhandled rejections by default, so a 5xx/permission error fails the
step → resolve-matrix.result == 'failure' → enforce-redeploy-gate trips
RED. Silent default-to-false on API error would open the gate on a
broken pipeline, which is what we explicitly do NOT want.
Validation: actionlint shows 8 findings on both files, identical to the
integration baseline (zero new findings). python3 yaml.safe_load OK on
both. Regression suites green: showcase/scripts vitest 44/44
(aggregate-build-results + lint-rule-no-public-env);
showcase/bin/spec/all_tests.rb 87 runs / 251 assertions / 0 failures.
Seven-agent CR surfaced correctness defects in the build/deploy/promote
pipeline and in the no-public-env-shell-read oxlint rule. This commit
closes the false-green paths and broadens lint coverage.
Workflow fixes:
- showcase_deploy.yml: drop `continue-on-error: true` on the redeploy-summary
artifact download. The dispatch path is already guarded by the `if:
workflow_run` clause, so the bash "no summary" branch handles legitimate
manual dispatches. A genuine workflow_run download failure must now fail
loud instead of silently widening verify to the full service set against
stale `:latest`.
- showcase_build.yml: redeploy-staging now intersects the build matrix with
the aggregator success set (`needs.aggregate-build-results.outputs.results`,
status == "success") before producing the redeploy CSV. Failed/skipped
slots no longer get redeployed (which would just re-pull stale `:latest`
and look healthy).
- showcase_build.yml: `notify-all-builds-failed` now additionally requires
`needs.build.result == 'failure'` so it doesn't Slack-spam when the build
job was SKIPPED (verify-image-refs upstream failure).
- showcase_build.yml: `notify` now lists [build, aggregate-build-results,
redeploy-staging] in `needs:` so aggregator/redeploy failures still emit
a Slack signal. `if: failure()` still skips when none of the needs failed.
- showcase_build.yml: `set -euo pipefail` on the Prepare build args step
so a transient $GITHUB_OUTPUT write failure can't ship images without
COMMIT_SHA/BRANCH baked in.
- showcase_deploy.yml: `enforce-redeploy-gate` now also trips on a
resolve-matrix failure (`needs.resolve-matrix.result == 'failure'`) so
an upstream crash that leaves `redeploy_red` empty can't bypass the gate.
- Doc-comment accuracy: drop stale `(PR #5093)` reference; correct the
env-IDs source-of-truth comment; document the optional `skip_build` field
in ALL_SERVICES; clarify that health_path is informational and verify
uses per-service drivers; add the missing `resolve-targets` step 0 to the
promote workflow's "Order:" header.
Aggregator fix (RED-GREEN):
- aggregate-build-results.ts: throw on zero slot dirs. The job is gated
upstream on has_changes == 'true', so zero slot dirs is a broken artifact
download, not a legitimate empty build set. Silently emitting
any_success=false + results=[] is indistinguishable from "all builds
failed" and lets the deploy workflow fall back to probing the full
service set against stale `:latest`. Refuse the ambiguity.
- aggregate-build-results.test.ts: existing empty-INPUT_DIR test was
updated to assert the throw (was: return []).
Oxlint rule (RED-GREEN):
- no-public-env-shell-read.mjs: handle destructuring reads
(const { NEXT_PUBLIC_X } = process.env and aliased form), template-literal
computed keys (process.env[\`NEXT_PUBLIC_X\`]), and explicitly skip
assignment-LHS / `delete` targets (writes are not reads). Optional
chaining already worked through the existing MemberExpression path.
Aliasing (`const e = process.env; e.X`) is intentionally documented as
out of scope (needs scope tracking). Description sharpened to say the
rule guards a specific banned-key set, not all NEXT_PUBLIC_* reads.
- .oxlintrc.json: tighten the off-override glob from
`showcase/**/*runtime-config*` to
`showcase/**/lib/runtime-config*.{ts,tsx}` so it only silences the
intended implementation files, not arbitrary paths containing that
substring.
- lint-rule-no-public-env.test.ts: rewritten as table-driven coverage of
every BANNED_KEYS entry (dotted + bracket-string forms), every ALLOWED
key (asserting non-firing), all new variants from the rule expansion,
the assignment/delete non-fire cases, and override scoping
(runtime-config exempt; packages exempt; shell-tree non-runtime-config
flagged).
Validation:
- actionlint on all three workflows: 8 pre-existing findings (depot label,
pre-existing SC2086 infos in untouched steps); my edits add zero.
- python3 yaml.safe_load: all three workflows OK.
- vitest aggregate-build-results.test.ts: 6/6 pass (incl. new throw test).
- vitest lint-rule-no-public-env.test.ts: 34/34 pass.
- vitest full showcase/scripts suite: 1654/1654 pass across 46 files.
- ruby showcase/bin/spec/all_tests.rb: 87 runs, 0 failures.
- Intersection jq proof (matrix a,b,c × success a,c) → "a,c"; all-failed
→ ""; skipped status excluded.
Implements plan-B B11. URL and analytics NEXT_PUBLIC_* values now reach
each shell at runtime via Option B (env-driven runtime-config), so the
GHA showcase_build.yml workflow no longer threads them through as Docker
build-args and the shell-dashboard/shell-docs Dockerfiles no longer
declare the matching ARG/ENV pairs.
- showcase_build.yml: shell-dashboard and shell-docs matrix entries lose
build_args_pb_url / build_args_shell_url / build_args_ops_url /
build_args_base_url / build_args_analytics; the 'Prepare build args'
step drops the corresponding env: keys and if-branches plus the five
analytics NEXT_PUBLIC_* secrets. COMMIT_SHA and BRANCH stay — they
identify the artifact.
- showcase/shell-dashboard/Dockerfile: remove ARG/ENV for
NEXT_PUBLIC_SHELL_URL, NEXT_PUBLIC_POCKETBASE_URL, OPS_BASE_URL plus
the explanatory comments. Update the runner-stage comment to point at
runtime-config.ts as the new source of truth.
- showcase/shell-docs/Dockerfile: remove ARG/ENV for
NEXT_PUBLIC_BASE_URL, NEXT_PUBLIC_SHELL_URL, NEXT_PUBLIC_POSTHOG_KEY,
NEXT_PUBLIC_REB2B_KEY, NEXT_PUBLIC_SCARF_PIXEL_ID, NEXT_PUBLIC_REO_KEY,
NEXT_PUBLIC_GOOGLE_ANALYTICS_TRACKING_ID. COMMIT_SHA / BRANCH retained.
shell/Dockerfile and shell-dojo/Dockerfile already only declare commit-sha
and branch ARGs — no changes needed there (per plan-B B11.4).
Adds 'webhooks' as a workflow_dispatch choice in both showcase_build.yml
and showcase_deploy.yml so humans can redeploy/verify the webhooks service
on demand. webhooks' GHCR image (showcase-eval-webhook) is built by a
separate release workflow in the showcase-eval-webhook repo, so:
- paths-filter uses a sentinel that cannot match any in-tree path,
keeping push-driven runs from ever including webhooks.
- The build matrix entry carries skip_build: true; the Build and push
step skips the Depot build for that slot. The per-slot result still
publishes (job.status == success) so the redeploy path proceeds and
redeploy-env.ts picks the existing :latest from GHCR.
The SSOT entry in railway-envs.ts gains dispatchName: 'webhooks' so the
forward/reverse round-trip tests cover it. New tests pin that the SSOT
dispatchName is mirrored in both workflow files' dispatch choice lists
AND ALL_SERVICES JSON.
[BLITZ:L3-wf] E-6a/E-6b/E-6c/E-6d wiring.
redeploy-staging now also gates on aggregate-build-results.outputs.any_success
== 'true'. When every slot fails, the previous behavior silently kicked a
redeploy that just re-pulled the stale :latest and reported healthy. With this
gate, the redeploy is suppressed and a sibling notify-all-builds-failed job
marks the workflow red and posts to #oss-alerts so the all-broken state cannot
hide behind a green run.
The new notify-all-builds-failed job is distinct from the existing notify: job
(which fires on any build-slot failure); both can fire and that overlap is
intentional, per the Slack alert SOP.
[BLITZ:L3-wf] E-5c wiring.
renovate[bot]