PR #6163 converted showcase/integrations/*/public/demo-files/* to Git LFS.
showcase_validate.yml's python-unit-tests job checks out without LFS, so the
working tree holds 129-byte pointer text instead of the real assets. The new
ms-agent-python multimodal test is the first Python test to read those bytes:
pypdf reads the pointer, extracts nothing, and the test fails with
"real pypdf text extraction produced nothing".
Fetch only the demo-file assets (40 objects, ~320 KB) rather than setting
lfs: true, which would pull all ~248 tracked objects (~475 MB, including
13-26 MB README gifs) into a 2-4 minute job under a 10 minute cap. A job that
hits timeout-minutes reports 'cancelled' -- neither success nor failure -- and
silently suppresses alerting, so the timeout margin is worth protecting.
The repo is public, so LFS downloads resolve anonymously and the pull works
with persist-credentials: false. A post-pull %PDF- header check fails the step
immediately if the pull ever no-ops, instead of surfacing minutes later as a
misleading pytest assertion.
Root cause of the cancellations, from the Depot logs of the killed slots:
they were not a transient flake. Every one died at EXACTLY its
`timeout-minutes` value (shell/shell-docs/shell-dashboard/shell-dojo at
10m04s of a 10-minute budget; showcase-aimock at 5 minutes), and the log
ends with `failed to solve: Canceled: context canceled` followed by
Depot's `Step canceled by GitHub`. GitHub reports a `timeout-minutes`
kill as job conclusion `cancelled`, which is what fed the whole silent
chain.
The builds were making real but slow progress the entire time — repeated
docker.io base-image pull stalls, and the final `generate-registry` layer
alone took 54s — and shell was still 2 steps from done when the budget
killed it. The `context: "."` slots build the whole monorepo root and are
the heaviest in the fleet, yet they carried the SMALLEST budgets (10 min
vs 15 for each small per-integration build). Measured, same slots:
warm / uncontended ......... 2-5 min
contended, killed at ....... 10 min (runs 30162773601, 30162770765)
showcase-pocketbase ........ 8.8 min of a 10 min budget (near-miss)
So a retry is the wrong lever twice over: a cancelled job cannot run
further steps, and the build was not erroring. Give the four shell slots
20, showcase-aimock 12, and showcase-pocketbase 15. `webhooks` stays at 5
(it is skip_build, measured at 0.6 min).
This is mitigation, not the root fix — the underlying cause is Depot
builder contention from three concurrent full-fleet rebuilds (~84
simultaneous amd64 builds). Noted inline as a follow-up, since the
durable lever is a non-cancelling concurrency queue and this workflow
deliberately has no concurrency group.
Two independent ways verification was silently lost on 2026-07-25.
1. `resolve-matrix` required `github.event.workflow_run.conclusion ==
'success'`. But `redeploy-staging` gates on the artifact-derived
`any_success`, not on the matrix rollup, so a build with some slots
cancelled (rollup `cancelled`) or failed (rollup `failure`) still
pushes real images and still redeploys the slots that built. Run
30162773601 redeployed 23 services to staging and this workflow never
started. An unverified real deploy is worse than a verified partial
one, so the trigger is now "the build reached a terminal conclusion"
and WHAT to verify stays decided by evidence: no redeploy-summary
artifact still means has_services=false and verify is skipped, and the
redeploy gate still narrows to the per-service success set, so a
cancelled slot can never be probed against a stale `:latest`.
2. The `showcase-verify-deploy` concurrency group was global, so any
later-finishing build preempted an earlier run's verification even
though they verify DIFFERENT commits. Verify run 30163309977 for the
one genuinely successful build of the day (#6168, 7282ecddf) started
at 15:17:13 and was cancelled 9 seconds later by run 30163312882 —
which was triggered by a different, partially-cancelled build and then
skipped every job anyway. Key the group per triggering commit.
Build run 30162773601 (merge of #6160) forced a full-fleet rebuild; 5 of
28 slots were killed by their `timeout-minutes` budget, the other 23
built and WERE redeployed to staging, and the run emitted no signal at
all: `notify` was skipped, so no Slack alert and no PR comment, and the
run rolled up to conclusion `cancelled`.
No existing guard could catch it. Measured on purpose-built probe run
30166429073 (matrix leg killed by `timeout-minutes`, sibling leg green):
killed leg `job.status` ........ cancelled
matrix rollup `needs.*.result` . cancelled
`if: cancelled()` .............. SKIPPED (evaluated FALSE)
`if: failure()` ................ SKIPPED (evaluated FALSE)
pre-fix `notify` condition ..... SKIPPED <- the bug
post-fix `notify` condition .... RAN <- the fix
run conclusion ................. cancelled
So `failure() || cancelled()` would NOT have fixed this. The signal has
to come from the per-slot build results.
- stop laundering `cancelled` into `skipped` in the per-slot writer
- expose `any_cancelled` / `cancelled_services` from the aggregator job
- add `notify-cancelled-builds`: exits non-zero so the run concludes
`failure` rather than `cancelled` (a slot killed by its timeout budget
is a failure, and `cancelled` is what suppressed everything), and
Slacks the affected service names
- add the `any_cancelled` clause to `notify` so the merge author gets the
PR comment, with wording that distinguishes incomplete from failed
`!cancelled()` is retained on both jobs as the intentional-vs-flake
discriminator: a human cancelling the whole RUN makes `cancelled()` true
and stays silent, while a leg-level timeout leaves it false and alerts.
Extends redeploy-guard.test.ts, which evaluates the LIVE `if:` strings
from the workflow, with the exact production scenario. It pins the
pre-fix guard string as a literal so the test proves the difference the
fix makes, not merely the current behaviour.
## The gap
**No CI job ran the showcase unit suites.** Not "ran them partially" —
no job invoked them at all. Verified per suite against `91f0321397` (the
tip of `main` this branched from), by reading every workflow's actual
`run:` lines rather than trusting job names:
| Suite | Tests | Ran in CI before? | Where |
|---|---|---|---|
| `showcase/harness` | 3646 (3625 pass / 3 fail / 18 skip) | **NO** |
nowhere |
| `showcase/shell-dashboard` | 1331 (1330 pass / 1 skip) | **NO** |
nowhere |
| `showcase/shell` | ~57 (8 files) | **NO** | nowhere |
| `showcase/scripts` | — | yes | `showcase_validate.yml` → "Run build
pipeline tests" |
Why each candidate workflow misses them:
- **`test_unit.yml`** — the one whose name implies it covers this —
excludes them *twice over*: `paths-ignore` lists `showcase/**`, **and**
its nx selection is hard-scoped to `--projects='packages/**'`.
- **`showcase_validate.yml`** runs `pnpm exec vitest run` only in
`showcase/scripts`. It has two `working-directory: showcase/harness`
steps, but they are a CVDIAG perf bench (`vitest bench
src/cvdiag/emit-perf.bench.ts`) and an ESM boot-smoke — neither runs the
unit suite.
- **`static_quality.yml`** does run `nx run-many -t check-types`. This
resolves the reported name/invocation divergence: the *job* is called
`check-types` and the harness's script is called `typecheck`, so no
`check-types` target exists on the harness and **the harness is not
typechecked either**.
Net effect: ~5000 showcase unit tests gated nothing. A PR could carry
real defects in `showcase/harness/src/**` and still show an all-green
check list, because no job structurally could have caught them.
## What this wires
New workflow **`test / unit-showcase`**
(`.github/workflows/test_unit-showcase.yml`), two parallel jobs on PR +
push-to-main filtered to `showcase/**`:
- **`harness unit suite`** — `nx run
@copilotkit/showcase-harness:test:ci`, then `nx run
@copilotkit/showcase-harness:test:quarantine-ratchet`. Driven through nx
per the root `CLAUDE.md` task convention.
- **`shell-dashboard unit suite`** — `vitest run` directly; the
dashboard is deliberately outside the pnpm workspace (see the note in
`pnpm-workspace.yaml`) so it is not an nx project and has no target to
run.
New nx targets are declared in **`showcase/harness/package.json`'s
`nx.targets` block**, not in `nx.json` — they are harness-specific, and
a `nx.json` `targetDefaults` entry would silently apply to any future
project that happens to share the script name. Both are `cache: false`
on purpose: the nx `test` named-input covers `src/**` and `*.test.*` but
**not** `vitest.ci.config.ts` or `vitest.quarantine.json`, so a cached
result could survive an edit to the quarantine list.
### Why a separate workflow rather than steps in `showcase_validate.yml`
That file is a single ~900-line job already budgeted at 25 minutes and
is the busiest merge-path workflow in the repo. Splitting the unit
suites out gives them their own name in the check list, their own
concurrency group, and — since the two suites need different package
managers — lets them run in parallel instead of lengthening the critical
job. Naming follows the existing convention (`test_unit.yml`,
`test_unit-python-sdk.yml`, `test_unit-spring-ai.yml`).
**No `continue-on-error` and no `|| true` anywhere in the file.** A gate
that cannot fail is not a gate.
## Hazards, and how each is handled
### 1. The hanging spike test
*Correction to the reported location:* it is
`showcase/shell-dashboard/tests/runtime-env-switch.spike.test.ts`, not
under `showcase/harness/`. It reaches the dashboard suite via that
package's `include: ["src/**/*.test.{ts,tsx}",
"tests/**/*.spike.test.ts"]`.
**Reproduced locally.** vitest sat at **0.0% CPU with zero output** and
never even spawned the `next build` its `beforeAll` calls — no worker
fork, nothing. Killed at ~2 minutes; consistent with the reported
~25-minute stall.
**Handled by excluding it from the unit gate**, not by a hard timeout.
Justification: it is an integration spike by construction — one `next
build` plus two `next start` boots on fixed ports 3801/3802 — and this
package's own config comment already says it is *"too heavy for the
per-file unit suite"* even though the `include` glob pulls it in anyway.
A hard timeout would convert a 25-minute stall into a red gate with
nothing actionable in it; this suite has to stay a fast, trustworthy
unit signal. The spike needs its own job with a real server budget,
which is out of scope for wiring up the unit gate.
Note the `--exclude` values restate the package's existing config
excludes (`tests/visual/**`, `node_modules/**`) because vitest's CLI
`--exclude` **replaces** the config value rather than extending it —
dropping them would silently re-enable them.
### 2. Gitignored generated artifacts
`showcase/.gitignore` ignores `shell/src/data/*.json` and
`shell-dashboard/src/data/*.json`, but the suites consume them:
- `harness/src/probes/frontend-matrix.test.ts` **statically imports**
`shell/src/data/frontend-catalog.json` — a missing file is a module
*load error*, not a test failure.
- `harness/src/fleet/control-plane/d0-gone-predicate.test.ts` reads
`shell/src/data/registry.json` at runtime.
- `shell-dashboard/src/lib/docs-status.ts` **statically imports**
`@/data/docs-status.json`.
Generated as an explicit job step (`generate-registry.ts`, plus
`probe-docs.ts` for the dashboard), mirroring how
`showcase_validate.yml` drives the same generators. The dashboard job
invokes vitest directly rather than `npm test` specifically so the
`pretest` hook does not re-run `probe-docs.ts` and double its ~50
outbound HEAD requests.
Also load-bearing: `npm ci --ignore-scripts` in the dashboard job. That
package's `postinstall` is `cd ../scripts && npm install`, which would
lay an npm-resolved `showcase/scripts/node_modules` over the
pnpm-managed one.
### 3. The 3 pre-existing failures — quarantined behind a ratchet, none
deleted or weakened
Confirmed byte-identical to `origin/main` and pre-existing. A job that
is red on arrival gets ignored or disabled, so the three files are
excluded via `showcase/harness/vitest.quarantine.json` +
`vitest.ci.config.ts` — **and `scripts/quarantine-ratchet.ts` re-runs
each one and requires it to STILL FAIL.** The moment somebody fixes a
quarantined test, the ratchet goes red and names the entry to delete. An
entry therefore cannot outlive the failure it excuses. The ratchet also
fails on an entry pointing at a missing file, on an entry whose filter
matches more than one file, and on a malformed or under-documented
manifest (every entry must carry `file` / `since` / `reason` /
`unquarantineWhen`).
Local `pnpm test` still uses the unfiltered config, so a developer sees
the quarantined failures.
| File | Disposition | Why |
|---|---|---|
| `src/probes/frontend-matrix.test.ts` | quarantine | Asserts hard-coded
per-frontend counts (`react=660`, `angular=636`) against the
**generated** `frontend-catalog.json`, which has grown to 1302 runnable
cells (`react=664`). Stale literals, not a product defect — the two
assertions that express the test's stated intent ("without loss or
duplication": `length === metadata.runnable`, ids unique) both pass.
Exit criterion: derive the counts from the catalog; re-hard-coding fresh
numbers just resets the clock. |
| `src/probes/helpers/d5-mapping-drift.test.ts` | quarantine |
**Stranded by a completed refactor.** It regex-scrapes
`CATALOG_TO_D5_KEY` out of `shell-dashboard/src/lib/live-status.ts` to
prove the dashboard copy hasn't drifted from the harness's
`REGISTRY_TO_D5`. That file is now a one-line `export *` barrel over the
canonical `harness/src/shared/cell-model/live-status.ts` — there is no
second copy to drift and nothing for the regex to find, so the test
throws its own "could not locate" guard. Exit criterion: cell-model
owner deletes it (single-source is a strictly stronger guarantee) or
rewrites it to assert the barrel re-export. |
| `src/probes/helpers/d5-representatives.test.ts` | quarantine |
**Genuine product gap, deliberately not fixed here.**
`d5-browser-use-smoke` registers `browser-use-smoke` in `D5_REGISTRY`
but `D5_REPRESENTATIVES` has no entry, so
`getD5Representative('browser-use-smoke')` returns `undefined` and the
D5 driver has no representative fixture. The test is correct and
reporting a real hole; the fix belongs to the D5 owner, and this PR
touches no product source. |
### 4. New finding: a load-sensitive flaky test
Not one of the 3, and worth fixing separately:
`src/probes/loader/probe-invoker.test.ts` → *"times out at the invoker
level when enumerate() ignores abortSignal"* asserts a **wall-clock**
`elapsed < 150` against a 100ms invoker timeout racing a 200ms
enumerate. 50ms of headroom does not survive CPU contention — it passed
**5/5 in isolation** but failed twice at ~217ms while the machine was
running other vitest suites, and a 4-vCPU runner executing 173 test
files in parallel is exactly that condition.
Mitigated with `retry: 2` **in the CI config only** (base config
unchanged, so local runs still surface non-determinism). This is not
suppression: a genuinely broken test fails all three attempts and the
gate stays red, and vitest prints every retried test. The real fix is
for the probe-loader owner to drive that assertion off fake timers or
widen the bound, after which the `retry` should come out. Flagging the
tradeoff explicitly rather than burying it.
## Deliberate-breakage proof
A job that has never failed is not yet a gate. Scratch commit
`e100df27a7` broke two product-source files, then `4f092eccb1` reverted
it. The final branch diff is **578 insertions, 0 deletions, and zero
product source touched.**
| Run | SHA | `harness unit suite` | `shell-dashboard unit suite` |
Quarantine ratchet |
|---|---|---|---|---|
| baseline
[`30140087665`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140087665)
| `452a26115e` | **success** (3m33s) | **success** (1m35s) | success |
| **breakage**
[`30140218159`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140218159)
| `e100df27a7` | **FAILURE** (3m31s) | **FAILURE** (1m28s) | success |
The breakage — both reproduced locally before pushing:
- `showcase/harness/src/shared/cell-model/staleness.ts`:
`D4_STALE_AFTER_MS` 1h → 10h (a frozen-green D4 row would credit D4 for
ten hours instead of one)
- `showcase/shell-dashboard/src/lib/sort-order.ts`: `sortOrder` `mastra:
8` → `88`
What the RED run actually caught:
```
harness unit suite — Test Files 3 failed | 168 passed | 2 skipped (173)
Tests 4 failed | 3612 passed | 18 skipped (3634)
FAIL src/http/matrix.test.ts > stale green ladder: raw state='green' but /api/matrix folds to gray (U8)
FAIL src/shared/cell-model/cell-model.contribution.test.ts > family fold: one stale-green sibling forces STALE_DEGRADED
AssertionError: expected 'GREEN_FRESH' to be 'STALE_DEGRADED'
FAIL src/shared/cell-model/cell-model.equivalence.test.ts > golden-master identity > fixture: pos-d4-stale-green
FAIL src/shared/cell-model/cell-model.equivalence.test.ts > golden-master identity > fixture: stale-column
shell-dashboard unit suite — Test Files 3 failed | 64 passed (67)
Tests 4 failed | 1326 passed | 1 skipped (1331)
FAIL src/components/__tests__/depth-utils.test.ts > does not credit D4 when its green chat row is stale
AssertionError: expected 4 to be 3
FAIL src/lib/__tests__/cell-model.test.ts > comm-error staleness window > scopes the window PER ROW FAMILY (G3e)
AssertionError: expected 36060000 to be less than 21600000
```
Three things this establishes beyond "the job can go red":
1. **It catches exactly the class of defect that went undetected on
#6156** — a change to `showcase/harness/src/shared/cell-model/**` landed
4 failures in the harness gate, including two golden-master equivalence
fixtures.
2. **The dashboard job also caught the harness defect**, through the
`live-status.ts` re-export barrel. The shared cell-model now has two
independent gates over it.
3. **`retry: 2` is not suppression.** Every `FAIL` line above appears
three times in the log — the test was retried the full budget and failed
all three attempts. A genuinely broken test cannot pass through it. And
the **quarantine ratchet stayed green on the broken run**, confirming
the quarantine is scoped to its three files and does not swallow
unrelated regressions.
## Runtime
Measured on CI (Depot 4-vCPU), baseline run
[`30140087665`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140087665)
— **both jobs green**:
| Job | CI wall-clock (whole job) | Suite only (local) | Files |
|---|---|---|---|
| `harness unit suite` | **3m33s** | 63.5s | 173 (171 pass / 2 skip,
3616 tests) |
| `shell-dashboard unit suite` | **1m35s** | 7.2s | 67 (1330 pass / 1
skip) |
They run in parallel, so the added critical-path cost is ~3m33s —
comfortably under `showcase_validate.yml`'s existing 25-minute budget
and nowhere near irritating-enough-to-disable. In both jobs the budget
is dominated by install and (for the harness) nx `^build`, not by the
tests.
**No sharding or `nx affected` needed.** `nx affected` would in fact be
actively wrong here: the entire point is that a change anywhere under
`showcase/**` should run the suite covering it, and the historical
failure mode was scope-narrowing, not slowness. If install time later
dominates, the lever is pnpm/npm store caching (already configured) —
not cutting test scope.
## Not covered
`showcase/shell`'s 8 test files (~57 tests) are **also ungated** and I
left them out to keep this change to one concern. They need a third
install path (`showcase/shell` is likewise outside the pnpm workspace)
and its `vitest.config.ts` already has a `globalSetup` that generates
its own `registry.json`. Cheap follow-up.
## Coordination
Concurrent work is adding a vacuity/drift text gate under
`showcase/scripts` and compile-time completeness exports. This PR owns
only the **suite invocation**. The one plausible cherry-pick conflict is
`showcase/harness/package.json` (I added two `scripts` entries and an
`nx.targets` block); `.github/workflows/test_unit-showcase.yml` is a new
file and `nx.json` is deliberately untouched.
No CI job ran the showcase unit suites. Verified per suite on 91f0321397:
showcase/harness 3646 tests NOT RUN
showcase/shell-dashboard 1331 tests NOT RUN
showcase/shell 57 tests NOT RUN
showcase/scripts - run by showcase_validate.yml
`test_unit.yml` excludes them twice over: `paths-ignore` lists `showcase/**`
AND its nx selection is scoped to `--projects='packages/**'`.
`showcase_validate.yml` runs vitest only in `showcase/scripts`; its two
`working-directory: showcase/harness` steps are a CVDIAG perf bench and an ESM
boot-smoke. `static_quality.yml` runs `nx run-many -t check-types` and the
harness's script is named `typecheck`, so it is not typechecked there either.
So ~5000 showcase unit tests gated nothing — a PR could carry real defects in
`showcase/harness/src/**` and still show an all-green check list.
This adds `test / unit-showcase` with two parallel jobs (harness via nx,
dashboard via npm since it is deliberately outside the pnpm workspace). Both
hazards that make a bare `vitest run` useless here are handled as job steps:
- generated artifacts: `showcase/.gitignore` ignores `shell/src/data/*.json`
and `shell-dashboard/src/data/*.json`, but tests statically import
`frontend-catalog.json` and `docs-status.json` — a missing file is a module
load error, not a test failure. Generated before the suite runs.
- `runtime-env-switch.spike.test.ts` HANGS: vitest sits at 0.0% CPU with zero
output and never spawns the `next build` its beforeAll calls. It is an
integration spike (one `next build` plus two `next start` boots on fixed
ports) that this package's own config comment already calls "too heavy for
the per-file unit suite". Excluded from the unit gate; it needs its own job
with a real server budget.
A separate workflow rather than steps in showcase_validate.yml: that file is a
single ~900-line job already budgeted at 25 minutes, and the two suites need
different package managers, so they parallelize instead of lengthening the
critical job. Naming matches test_unit.yml / test_unit-python-sdk.yml.
No `continue-on-error` and no `|| true` anywhere in the file.
Demo assets stay uniformly LFS-tracked. The build is the single place this
gets resolved, so every workflow that checks out and then serves or bakes
integration `public/` assets now fetches LFS objects.
`showcase_build.yml` (the workflow that builds and pushes the deployed
images) has hardcoded `lfs: true` since 7bde1eef3a, so the deploy path was
already correct. Two paths were not:
- `showcase_build_check.yml` read `lfs: ${{ matrix.service.lfs }}`, which was
`false` for every integration, so the pre-merge build check baked pointer
stubs into the images it built. Build-only, so nothing deployed, but it
meant PR CI could never catch a broken demo asset.
- `test_e2e-showcase-on-demand.yml` passed no `lfs` input at all. That job
runs `next dev` on the integration package and drives it with Playwright,
serving `public/` straight from the working tree, so `multimodal.spec.ts`
would click "Try with sample image", receive a ~130-byte pointer stub as
`image/png`, and fail on the magic-byte guard — with nothing wrong in the
code under test. Nine integrations ship that spec.
Both now use a uniform `lfs: true`.
Also deletes the per-slot `"lfs"` field from both `ALL_SERVICES` definitions.
After the above there were no consumers left: `showcase_build.yml` ignored it
in favour of a hardcoded `true`, and `showcase_build_check.yml` was its only
reader. Leaving a dead flag that reads as authoritative is how this was
mis-set in the first place, and deleting it means a new integration cannot be
added with LFS fetching silently off — there is no flag to forget.
No asset or .gitattributes changes: LFS is used as intended and a committed
pointer remains the correct on-disk state for an LFS-tracked file.
Two release-tooling defects turned a pair of canary publishes into a broken
combination for consumers (a canary runtime resolving the last STABLE
channels-intelligence, which still called the removed `channel.addAdapter`).
1. Canary versions were prereleases of an ALREADY-PUBLISHED version. A stable
release leaves the working tree on the version it just published, and
computePrereleaseVersion appended `-canary.<id>` to exactly that, so the
canary sorted BELOW its own release (`0.2.1-canary.x < 0.2.1`): the `canary`
dist-tag pointed behind `latest`, and no dependent range could ever resolve
it. Base the canary on the next unreleased version instead (patch bump,
reusing computeNextStableVersion's prerelease rule).
2. A canary published one scope at a time, but the scopes are only independent
on the version axis. `@copilotkit/runtime` carries
`"@copilotkit/channels-intelligence": "workspace:*"`, and `pnpm pack`
resolves that against the working tree — so a `monorepo` canary pinned the
channels family to its last stable release even when the commit changed both
sides of the contract. Add a prerelease-only `all` selector that bumps and
publishes every scope from one commit under one shared canary id, and warn
loudly when a single-scope canary leaves a cross-scope pin behind.
`all` is a selector, never a scope: stable releases stay single-scope (their
tag, release branch, and npm/Slack links all derive from one scope name), which
publish-release.yml enforces in both jobs and the dropdown guard enforces per
workflow.
The notify-all-builds-failed and notify jobs keyed off a 'failure' rollup /
bare failure(), so a build where every real service failed but one leg was
cancelled (contention) rolled up to 'cancelled' and sent no alert — the same
blind spot as the redeploy guard. Fire on any_success == 'false' (guarded by a
status function so a user-cancelled run stays silent). Extends the guard test.
## Showcase deploy-mechanism consolidation
Consolidates the showcase Railway deploy path onto a single
**CI-explicit** mechanism, so we can safely retire Railway's registry
auto-watch (the source of the surprise "Service aimock upgraded to
latest" emails). Design proposal: [Notion — Showcase Deploy-Mechanism
Consolidation](https://app.notion.com/p/3a33aa38185281e4b64cc5bebde92d91).
### What & why
The "aimock upgraded to latest" email was never a per-service config
choice — it was a **CI bug** letting Railway's watcher win a race: a
Renovate PR that only touches `showcase_build.yml` forces a full-fleet
rebuild; the LFS `shell` leg gets cancelled under runner contention; and
the `redeploy-staging` guard (`needs.build.result != 'cancelled'`) then
skipped the CI redeploy for the **whole fleet**, orphaning aimock's
fresh digest for Railway's watcher to pick up. The `autoUpdates` setting
itself had also silently drifted (24 services `minor` / 17 none) —
tracked in no SSOT, gated by nothing.
### The four changes (one commit each)
1. **`fix(showcase)` — the P0 guard bug.** Relax the `redeploy-staging`
**and** `redeploy-staging-starters` guards so a cancelled sibling leg no
longer skips the fleet's staging redeploy; they now redeploy the
already-computed successful-service list. A guard-evaluation test reads
the live workflow `if:` strings and models GitHub's matrix rollup.
2. **`feat(showcase)` — autoUpdates SSOT (per-env, staging-first).** Add
a **per-env** `autoUpdates` policy to every service in `railway-envs.ts`
— **staging: `disabled`** (enforced), **prod: `unmanaged`** (left
exactly as-is until a later migration). Regenerate
`railway-envs.generated.json`. CI-explicit redeploy becomes the single
deploy path on staging.
3. **`feat(showcase)` — drift gate.** New CI gate fails when a live
Railway service's `autoUpdates` diverges from the SSOT. Reads
`Environment.config` (autoUpdates isn't on the typed `ServiceSource`
output), **enforces managed (`disabled`) envs and skips `unmanaged`
ones** (so prod is untouched), **fails closed per-env** on zero-checked,
and skips cleanly on fork PRs with no Railway token.
4. **`feat(showcase)` — scheduled reconcile.** CI-owned self-heal (every
15m) comparing each staging service's deployed digest against GHCR
`:latest`, re-running the staging redeploy for lagging services and
alerting Slack. Invariant: **green ⟺ every in-scope service confirmed
current**; any unconfirmed service (lag, digest error, dropped redeploy,
empty scope, thrown redeploy) alerts and exits non-zero.
### Verification
- Every behavior change carries red-green tests; **230 tests pass**,
`tsc` clean, `oxfmt`/`oxlint` clean, generated JSON in sync, workflows
parse.
- Reviewed via a full CR loop (Tier 3, 5 rounds to convergence); the
reconcile's fail-loud invariant was hardened across rounds (silent-green
holes, stale-digest ordering, expansion false-positives, test hygiene).
### Rollout (staging-first)
- **Staging is flipped live as part of this change** — `autoUpdates`
disabled on all staging services (snapshot-first, verified only
`autoUpdates` changed). The drift gate now enforces staging.
- **Prod is untouched** — its `autoUpdates` stay exactly as-is and the
gate marks prod `unmanaged` (skipped). Migrating prod is a deliberate
follow-up (flip prod live + change prod SSOT `unmanaged`→`disabled`
together) once we're comfortable with staging on the new mechanism. No
transition window where anything is unguarded.
### Follow-ups (from CR, non-blocking)
- Dedup the reconcile alert's `unconfirmed` list by service key
(cosmetic double-listing; exit code already correct).
- Harden the sibling `notify-all-builds-failed`/`notify` jobs against
the same all-legs-cancelled rollup (pre-existing, in a job this PR
doesn't touch).
- Minor: `postSlackAlert` try/catch belt; a few added test assertions;
comment/doc accuracy.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Adds a CI-owned reconcile (every 15m) that compares each staging service's
deployed digest against GHCR :latest and re-runs the staging redeploy for
lagging services, alerting Slack. The run is green only when every in-scope
service is confirmed current; any unconfirmed service (lag, digest error,
dropped redeploy, empty scope, or a thrown redeploy) alerts and exits non-zero.
Exposes per-service redeploy records from redeploy-env for accurate per-service
remediation confirmation.
New CI gate fails when a live Railway service's autoUpdates diverges from the
SSOT (every service must be disabled). Reads Environment.config (autoUpdates is
not on the typed ServiceSource output), fails closed per-env when it verifies
zero services, and skips cleanly on fork PRs that lack a Railway token.
The redeploy-staging and redeploy-staging-starters jobs guarded on
needs.build.result != 'cancelled', so a single cancelled matrix leg (e.g. the
Git-LFS shell build under runner contention) skipped the whole fleet's staging
redeploy even when the other 27 services built fine. Relax both guards to
redeploy the already-computed successful-service list. Adds a guard-evaluation
test that reads the live workflow if: strings and models GitHub's matrix rollup.
Add an emoji reaction to the original promote-notify init message
reflecting the net run outcome, so operators can see success/failure at
a glance without opening the thread reply:
success -> white_check_mark (checkmark)
partial -> warning
total -> x
The live workflow calls reactions.add on the init post (guarded on a
successful init post, warn-only on failure to mirror the thread reply).
The dry-run harness emits the reaction it would add, using a
byte-identical case mapping enforced by a new anti-drift bats guard.
Adds bats coverage asserting the emitted reaction name per fixture.
## The incident
PR #6061 (`fix(starters): add python-multipart to agno starter`) merged,
its
`starter-agno` image built to GHCR via `showcase_build.yml`'s
`build-starters`
job — and then was **never deployed to Railway**. `starter-agno` stayed
crashed
until someone manually redeployed. That is not a one-off: it is
structural. Any
change touching **only** starter files
(`examples/integrations/<slug>/**`) hits
the same hole.
## Root cause (job-graph)
`showcase_build.yml` has two independent lanes:
| Lane | Detect | Build | Aggregate | Redeploy |
| --- | --- | --- | --- | --- |
| **Showcase fleet** | `detect-changes` | `build` (matrix) |
`aggregate-build-results` | `redeploy-staging` |
| **Starters** | `detect-starter-changes` | `build-starters` (matrix) |
— | **(none)** |
`redeploy-staging` is scoped **entirely to the showcase fleet**:
```
redeploy-staging:
needs: [detect-changes, build, aggregate-build-results]
if: >-
!cancelled()
&& needs.detect-changes.outputs.has_changes == 'true'
&& needs.build.result != 'skipped'
&& needs.build.result != 'cancelled'
&& needs.aggregate-build-results.outputs.any_success == 'true'
```
(`.github/workflows/showcase_build.yml:780-799`)
On a **starter-only** push:
- `detect-changes` filters are all `showcase/**` paths → none match
`examples/integrations/**` → `has_changes=false` → `build` skips →
`aggregate-build-results` skips → `redeploy-staging` skips (fails its
`has_changes=='true'` and `build.result != 'skipped'` clauses).
- `detect-starter-changes` matches → `build-starters` builds
`starter-<slug>:latest` to GHCR — **and stops.** There is no aggregate
and
**no redeploy job for the starter lane at all.**
Net: the starter image is built and pushed, but nothing ever calls
Railway's
`serviceInstanceRedeploy`, so the running container keeps the stale
image.
The `starter-build-result-*` per-slot artifacts already emitted by
`build-starters` were **write-only** — nothing consumed them.
## The fix
A new `redeploy-staging-starters` job that mirrors `redeploy-staging`
for the
starter lane, reusing existing mechanisms (no new script, no new
aggregator):
```
redeploy-staging-starters:
needs: [detect-starter-changes, build-starters]
if: >-
!cancelled()
&& needs.detect-starter-changes.outputs.has_changes == 'true'
&& needs.build-starters.result != 'skipped'
&& needs.build-starters.result != 'cancelled'
```
Steps:
1. Download the already-emitted `starter-build-result-*` artifacts
(`pattern` download → succeeds with zero matches if the build crashed
before writing any).
2. Compute `matrix ∩ build-success`: read each per-slot
`{"service":"<raw slug>","status":...}`, keep `status:success` slugs,
map
each **raw slug → `starter-<slug>` SSOT key** via the starter matrix
`.image` field. (The raw slug must NOT be passed to `redeploy-env.ts` —
e.g. `"agno"` collides with the **showcase** `agno` dispatch_name.)
3. If the CSV is non-empty, `npx tsx showcase/scripts/redeploy-env.ts
staging
--services <csv>` — the exact same invocation `redeploy-staging` uses.
`redeploy-env.ts` already resolves each `starter-<slug>` as an SSOT key
(verified: all 12 starter `.image` values exist as keys in
`railway-envs.ts`).
**Deploy-on-failure is impossible:** the "don't deploy on build failure"
guard
is the per-slot success intersection, not the job `if:`.
`build-starters`
`result == 'failure'` still enters the job (fail-fast is false, so some
slots
may have succeeded), but only `status:success` slots are redeployed; an
all-failed or crashed build yields an empty CSV → the redeploy step is
skipped.
This is the same net guarantee `redeploy-staging` gets from its
`any_success`
guard, computed inline to avoid standing up a second aggregator job.
**No `redeploy-summary` artifact is written** by this job on purpose:
that
name is owned by `redeploy-staging` and downloaded by
`showcase_deploy.yml` by
exact name — a second same-named upload would collide on a combined
push.
Starter *staging verification* is intentionally out of scope for this
deploy-gap fix (starters are already smoke-covered by
`test_smoke-starter.yml`
and the harness `starter_smoke` axis). `redeploy-staging-starters` was
also
added to the `notify` job's `needs` so a starter redeploy failure
alerts.
## Before / after truth table
| Scenario | `build` | `redeploy-staging` | `build-starters` |
`redeploy-staging-starters` |
| --- | --- | --- | --- | --- |
| **(a) main-fleet-only change** | runs | **redeploys fleet** | skipped
| skipped |
| **(b) starter-only change** | skipped | skipped | runs | **redeploys
starter (NEW)** |
| **(c) both changed** | runs | **redeploys fleet** | runs | **redeploys
starter (NEW)** |
| **(d) starter build failure** | (n/a) | (n/a) | failure | runs, but
CSV empty → **no redeploy** |
- **(a)** unchanged — the showcase lane is untouched.
- **(b)** is the fix: the starter now auto-deploys instead of sitting on
GHCR.
- **(c)** unchanged for the fleet; the starter additionally deploys. No
artifact
collision because `redeploy-staging-starters` uploads no
`redeploy-summary`.
- **(d)** partial failure redeploys only the slots that succeeded; a
wholesale
failure redeploys nothing.
## Fail-loud hardening (CR follow-up)
A CR flagged that the new `redeploy-staging-starters` job could itself
**silently under-deploy** — re-opening the very hole it exists to close.
Two
guards added to the `Compute successfully-built starter services` step,
mirroring
the sibling `redeploy-staging` job's empty-intersection guard:
1. **Empty deploy-set → fail loud.** When `build-starters.result ==
'success'`
(all slots built) but the `matrix ∩ success` CSV is **empty**, the job
now
`exit 1`s with an actionable `::error::` instead of silently skipping
the
redeploy at green CI (a slug↔`.image` contract skew, or a success set
that
maps to no matrix entry). Gated on `'success'` so a partial/total build
**failure** keeps the legitimate no-deploy path and is not
double-reported —
that failure is already surfaced by `build-starters` itself.
2. **Missing/unreadable result artifact → fail loud.** Dropped
`2>/dev/null` on
the `result.json` read so a read error surfaces and trips `pipefail`,
and
added a parsed-record-count vs matrix-slot-count assertion (every slot
writes
a `result.json` via `if: always()`, so on a full-success build the
counts must
match). A missing/expired `starter-build-result-*` artifact — which
would
otherwise silently drop a built starter from the redeploy set — now
fails the
job. Also gated on `'success'` so a crashed slot's legitimately-absent
artifact isn't double-reported on a build failure.
Updated truth table with the new fail-loud row:
| Scenario | `build-starters.result` | deploy CSV |
`redeploy-staging-starters` |
| --- | --- | --- | --- |
| success + non-empty CSV | success | non-empty | **redeploys** |
| success + **empty** CSV | success | empty | **exit 1 (NEW fail-loud)**
|
| build failure (partial) | failure | subset | redeploys successful
subset, no spurious exit |
| build failure (all/crash) | failure | empty | no deploy, failure
surfaced, no spurious exit |
| skipped (no starter changes) | skipped | (n/a) | job `if:` excludes it
— never runs |
Both guards were locally red→green exercised: pre-fix the empty-CSV and
missing-artifact cases went **green with nothing/partial deployed**;
post-fix
they `exit 1`. actionlint still 9/9 (no new findings).
## actionlint
Clean. Baseline (`origin/main`) = 9 findings; this branch = 9 findings,
all at
pre-existing lines (custom `depot-*` runner label + pre-existing SC2086
infos).
**Zero new findings** from the added job. The `matrix ∩ success` jq
mapping was
locally exercised across the four scenarios above
(success/partial/all-fail/
crash-before-artifacts) and produced the expected CSVs.
## Prod-promote path
**Does NOT share the gap.** `showcase_promote.yml` is
`workflow_dispatch`-only
("Humans trigger. No automatic prod promotes.") and already lists all 12
`starter-*` services in its choices with `resolve-targets` handling
them. There
is no push-driven prod path to fix.
## Residual verification (honest note)
Workflows can't be safely dry-run end-to-end (the redeploy path hits
live
Railway). Static validation is complete (actionlint clean, jq logic
exercised,
all 12 starter SSOT keys confirmed, `redeploy-env.ts` reused unchanged),
but
the true end-to-end confirmation is the **next starter-only change
auto-deploying to staging**. That first real starter-only merge after
this
lands should be watched to confirm `redeploy-staging-starters` fires and
the
Railway service picks up the new image.
---
Draft — do not merge until reviewed.
The new redeploy-staging-starters job pinned actions/checkout to
9c091bb (tag v7.0.0) but commented it # v7. zizmor's ref-version-mismatch
flagged the discrepancy: the v7 moving tag points to 3d3c42e, not 9c091bb.
Repin to 3d3c42e # v7 — the canonical checkout pin already used across
every other job on main — so the comment matches the SHA's tag.
The redeploy-staging-starters job could go green while deploying
nothing (all starters built, empty deploy CSV) or silently drop a
built-but-missing starter from the redeploy set. Both re-open the exact
gap this job exists to close.
- Finding 1: when build-starters.result == 'success' but the matrix ∩
success CSV is empty, fail the job (exit 1) instead of silently
skipping the redeploy. Mirrors redeploy-staging's empty-intersection
guard. Gated on 'success' so a partial/total build FAILURE keeps the
legitimate no-deploy path and isn't double-reported.
- Finding 2: drop 2>/dev/null on the result.json read (surface read
errors via pipefail) and assert parsed-record count == matrix slot
count when all slots built, so a missing/expired starter-build-result-*
artifact fails loud instead of silently under-deploying.
A push touching only starter files (examples/integrations/<slug>/**) built a
fresh starter-<slug>:latest image to GHCR via build-starters but never
redeployed it to Railway: the starter lane (detect-starter-changes ->
build-starters) ended at the GHCR push, and redeploy-staging only covers the
showcase build lane. Starter fixes sat undeployed until a manual redeploy
(the #6061 agno incident).
Add a redeploy-staging-starters job that mirrors redeploy-staging for the
starter lane: it reads the already-emitted per-slot starter-build-result-*
artifacts, intersects the build matrix with the build-success set (mapping raw
slug -> starter-<slug> SSOT key), and redeploys only the successfully-built
starters to staging. No deploy on build failure (empty success set -> redeploy
step skipped). Reuses redeploy-env.ts unchanged.
## What
Adds `wolfmanfx` (Murat Sari) and `rainerhahnekamp` (Rainer Hahnekamp)
as code owners of the Angular package paths in `.github/CODEOWNERS`:
- `packages/angular/`
- `packages/v2/angular/`
Existing core-dev owners are retained on both lines, so this is purely
additive.
## Notes
- **Access is granted separately.** This CODEOWNERS entry only assigns
review ownership; it does not grant repo access. The two contributors
still need to be added as repo collaborators via GitHub settings for the
entry to take effect (GitHub permissions are repo-level, not
folder-scoped).
- **Branch protection is currently disabled on `main`**, so this assigns
reviewers but does not gate merges until protection requiring code-owner
review is enabled.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The repo uses a flat package layout; there is no packages/v2/ directory,
so that CODEOWNERS entry never matches and fails path validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The notify job's intent gate treated ANY workflow_dispatch as stable-release
intent (npm_intended=true), including prerelease/canary dispatches. canary.yml
orchestrates publish-release.yml with mode=prerelease on a short-lived canary/*
ref, so a canary whose build fails (e.g. a stale pnpm-lock.yaml on the source
branch tripping pnpm install --frozen-lockfile) flipped the notify job to
failure() and fired the self-watchdog's best-effort Slack post — a red
'a release alert may have been swallowed' page for a release that was never
going to publish.
Narrow the npm_intended dispatch arm with 'inputs.mode != prerelease' so a
canary yields npm_intended=false and never self-pages, while a stable dispatch
(default mode, and the main-only retry escape hatch) and a merged
release/publish/* PR still page correctly on genuine failure. This mirrors the
existing 'mode != prerelease' guards on the publish/tag steps and the builder's
should_post=false suppression for canaries.
Restore the 12 Python integration tools/ dirs to symlinks into
shared/python/tools. They had eroded to real, drifting copies via an
accidental stage_shared() leak (commit 534cd1efa7) — the structural root
cause of showcase divergence bugs. Symlinking re-establishes the single
source of truth; content is identical to shared (only render_a2ui naming
and the shared roll_dice/sanitize additions are adopted).
Add showcase/AGENTS.md documenting the 4 iron rules and the single-source
symlink mechanism, plus a validate-shared-symlinks CI guard (shrink-only
baseline) that fails on any NEW erosion.
## Two silent-failure gaps in the showcase build/deploy/notify pipeline
These are **pre-existing** silent-failure holes surfaced in code review
(not
caused by any recent PR). This PR fixes the two load-bearing ones.
### 1. Green-but-zero-redeploy (silent "we thought we shipped but
didn't")
The `redeploy-staging` job computes the redeploy set as the intersection
of the
build matrix and the build-success set. This job **only runs when
`aggregate-build-results.outputs.any_success == 'true'`** (job-level
`if:`
guard). So if that intersection comes back **EMPTY**, it does NOT mean
"nothing
to deploy" — it means at least one slot built successfully yet none of
those
successes maps back to a matrix `dispatch_name`. That's a
`dispatch_name`↔
`service` contract skew (the aggregator's `service` values and the
matrix's
`dispatch_name` values drifted apart).
The old code emitted `services=` (empty) and exited 0 → the build went
**GREEN
while redeploying NOTHING**, silently.
**Fix:** on an empty intersection in this any_success-guaranteed step,
fail loud
(`::error::` + `exit 1`) with a diagnostic naming both sides of the
skew.
The legitimate "nothing changed / nothing succeeded" no-op paths are
guarded at
the **job level** (`has_changes=='true' && any_success=='true'`), so the
fixed
step never runs there — no false-red.
### 2. Starter build failures had no alert surface (invisible failures)
The `notify` job's `needs` (and its `if: failure()`) omitted
`detect-starter-changes` and `build-starters`, and `build-starters`
wrote no
per-slot build-result artifact. So a **failed starter image build
produced NO
Slack alert and NO PR comment** — it shipped silently.
**Fix:**
- Added `detect-starter-changes` + `build-starters` to `notify.needs` so
`if: failure()` sees a starter build failure → Slack alert + PR comment.
- Gave `build-starters` a per-slot build-result artifact **mirroring the
main
`build` matrix** (same `{service,status}` shape, `cancelled→skipped`
normalization, `if: always()`, `if-no-files-found: error`), using a
**distinct `starter-build-result-*` prefix** so it never matches the
aggregator's `build-result-*` download pattern (starters must not
pollute the
showcase redeploy set keyed by `dispatch_name`).
### Red / Green
**Finding #1** — extracted the step's shell/jq logic and drove it with
synthetic
inputs:
RED (pre-fix), any_success=true + empty intersection:
```
No services in matrix ∩ success-set — skipping redeploy.
Computed services CSV (matrix ∩ build-success):
EXIT=0 # $GITHUB_OUTPUT: services= -> silent pass, redeploys NOTHING
```
GREEN (post-fix), same inputs:
```
::error::Build succeeded (any_success=true) but matrix ∩ success-set is EMPTY — dispatch_name/service contract skew; nothing would be redeployed.
Successful build service values: ["shell-RENAMED","mastra-RENAMED"]
Scheduled matrix dispatch_name values: ["shell","mastra"]
EXIT=1 # fails loud
```
No-regression: non-empty intersection → `EXIT=0 ; services=shell`. The
nothing-changed/nothing-succeeded paths are skipped at the job level
(never
reach the step) → no false-red.
**Finding #2** — modeled `if: failure()` (fires iff any `needs` job
result is
`failure`):
```
BEFORE (starters NOT in needs), starter=failure -> notify fires = False (INVISIBLE, the bug)
AFTER (starters IN needs), starter=failure -> notify fires = True (FIXED)
AFTER no-regression, starters=skipped, all green -> notify fires = False (quiet)
```
### Validation
- `python3 yaml.safe_load` parses OK.
- `actionlint`: only pre-existing findings remain (matrix jq SC2086 +
the known
`depot-ubuntu-24.04-4` runner-label warning); no new errors in edited
regions.
- `yamllint`: only pre-existing line-length/document-start/truthy
warnings.
### Scope
Touches **only** `.github/workflows/showcase_build.yml`, and only these
two
concerns. Does NOT touch the `shell_dashboard` paths-filter region (PR
#5955's
domain), nor the other backlog debt (false-root-cause comment,
double-alert,
check-lockfile guard). Self-contained; not stacked on #5955.
Two pre-existing silent-failure gaps in the showcase build/deploy/notify
pipeline (surfaced in code review):
1. Green-but-zero-redeploy: the redeploy-staging job computes the redeploy
set as (build matrix ∩ build-success). This job only runs when
any_success=='true', so an EMPTY intersection means builds succeeded but
none maps to a matrix dispatch_name — a dispatch_name/service contract
skew. The old code emitted an empty services= and exited 0, going GREEN
while redeploying nothing. Now it fails loud with a diagnostic naming both
sides of the skew. The legitimate nothing-changed/nothing-succeeded no-ops
stay guarded at the job level, so they are unaffected.
2. Starter-failure-invisible: the notify job's needs omitted build-starters,
so a failed starter image build produced no Slack alert and no PR comment.
Added detect-starter-changes + build-starters to notify.needs, and gave
build-starters a per-slot build-result artifact mirroring the main build
matrix (distinct starter-build-result-* prefix so it never pollutes the
showcase aggregator's build-result-* set).
renovate[bot]