## What broke
The regression was introduced by commit fec70d086 (feat(angular):
checkpoint 2 - core and package), merged through PR #6109 as b07482da5.
That commit established Angular 20 as the package’s compiler and support
floor, but the demo remained on Angular 21 after 8b13fbcb7 (build:
update ng).
It also introduced the packed smoke runner without canonicalizing macOS
temporary paths, allowing /var/... and /private/var/... to resolve
inconsistently.
## Why I made this change
I moved the demo back to Angular 20 so it exercises the lowest supported
Angular version, aligned the Angular 20 dependencies and support
contract on
20.3.27, and canonicalized the packed consumer directory before starting
the SSR server.
This keeps the demo, package metadata, tests, and lockfile consistent
while making the packed smoke test reliable across symlinked temporary
directories.
## Changes
- Align the Angular demo with the Angular 20 support floor.
- Update Angular 20 dependencies and support-policy tests to 20.3.27.
- Resolve the packed consumer directory to its real path before
launching SSR.
- Remove multimodal from features (already in not_supported_features);
the duplicate made generate-registry reject the manifest, blocking D6
from starting.
- Pin agent-framework-{ag-ui,openai,core} to exact published latest
(1.0.1 / 1.12.0 / 1.13.0) so validate-pins classifies them as exact
framework deps; drop unused langchain-openai/langchain-core.
- Ratchet validate-pins baseline down 31 -> 27 (the 4 non-exact framework
fails are now exact); ms-agent-python's only remaining FAIL is the
shared-frontend openai ^5.9.0, identical to every other integration.
## The race, with evidence
Three PRs merged within 34 seconds on 2026-07-26. `Showcase: Build &
Push` has **no concurrency group** (deliberately), so all three ran
simultaneously and raced to push the same `:latest` tags.
| run | commit | PR | start → end |
|---|---|---|---|
| `30190815370` | `7b28934387` | #6162 | 06:18:05 → 06:29:44 |
| `30190823203` | `59f275eedc` | #6161 | 06:18:21 → 06:29:44 |
| `30190831480` | `db75a04837` | #6158 | 06:18:39 → **06:29:13** ←
newest, finished FIRST |
The newest commit finished first, so the older builds overwrote its
`:latest`. Per-service job completion times — older beating newer on
every shared slot:
| service | newer (`db75a04837`) | older (`59f275eedc`) | older won by |
|---|---|---|---|
| `shell-dashboard` | 06:25:33 | 06:25:34 | +1s |
| `showcase-harness` | 06:27:51 | 06:27:55 | +4s |
| `shell` | 06:27:17 | 06:27:28 | +11s |
| `shell-dojo` | 06:25:10 | 06:25:26 | +16s |
**All three runs reported `success`.** Staging served pre-#6158 code
while CI, the redeploy gate and deploy verification all looked clean.
Same failure class as #6171: a success that doesn't mean what it says.
## What I verified in YAML vs took on trust
Verified by reading the files / querying the API:
- **Tagging** — `showcase_build.yml` pushed `:latest` **and** `:${{
github.sha }}` in one `depot/build-push-action` step, in both the
`build` and `build-starters` matrices. **A per-commit sha tag already
existed**; confirmed in GHCR (`showcase-shell-dashboard` has digests
tagged `db75a04837…`, `59f275eedc…`, `d28384a2eb…`).
- **Nothing serialized the pushes.** No concurrency group; confirmed the
header comment states this is intentional.
- **Deploy consumes `:latest`** — `verify-railway-image-refs.ts` is the
SSOT assertion: staging is `ghcr.io/copilotkit/<repo>:latest` (mutable),
**prod is `ghcr.io/copilotkit/<repo>@sha256:<digest>` (already immutably
pinned)**. So this race is **staging-only**; prod was never exposed.
- **`Showcase: Verify Deploy` structurally cannot catch it.** It is a
health probe; it asserts no digest or commit provenance anywhere. Its
#6171 per-commit concurrency key is about *which run verifies*, not
*what image is running*. A stale-but-healthy service passes.
- **The racing runs build DISJOINT service sets** (see below) — I pulled
the actual job lists.
Taken on trust: nothing material. The issue description matched the API
on every point I checked.
## Why NOT a concurrency group
`detect-changes` builds a **per-push, path-filtered** matrix, so
concurrent runs build overlapping but **non-identical** service sets:
- `7b28934387` → ag2, agno, built-in-agent, claude-sdk-python,
claude-sdk-typescript, crewai-crews, langgraph-fastapi,
langgraph-python, langroid, llamaindex, mastra, pydantic-ai, shell-docs,
spring-ai, strands (**15**)
- `db75a04837` → crewai-crews, llamaindex, shell, shell-dashboard,
shell-docs, shell-dojo, showcase-harness (**7**)
`cancel-in-progress: true` would have cancelled the `7b28934387` run and
the ~10 services **only it builds would never have shipped** — trading a
stale-image bug for a never-shipped bug. Per [GitHub's
docs](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#concurrency),
`cancel-in-progress: false` is no better: *"Any previously pending job
or workflow in the concurrency group will be canceled"* — with three
rapid merges the **middle** commit's build is dropped outright. GitHub
also does not guarantee FIFO ordering of queued runs.
Concurrent runs here are **not redundant**, so they must not be
cancelled.
**Re the #6171 interaction:** this design does **not** use
`cancel-in-progress`, so no build is ever superseded-and-cancelled and
the cancelled-slot notifier is never tripped by this change. That
interaction stays theoretical — deliberately.
## The design
Make the one shared mutable resource monotonic instead of serializing
the fleet.
1. The build step pushes **only** the immutable `:<sha>` tag, plus an
`org.opencontainers.image.revision` label.
2. A guard resolves the commit behind the current `:latest`, asks GitHub
`compare/<theirs>...<ours>`, and advances `:latest` (registry-side
retag, no pull) **unless ours is `behind`** — i.e. `:latest` already
holds a descendant and moving it would roll staging back.
**Fails open by design.** No `:latest` yet, unlabelled legacy image,
unreachable API, diverged history → advance. A stuck `:latest` is the
very failure being fixed, so it declines only on *positive proof* of
regression.
**Placement:** in `redeploy-staging` / `redeploy-staging-starters`,
immediately before the Railway pull that consumes `:latest` — not per
build slot. Those jobs already have Node (build slots do **not**, so
per-slot would mean an unpinned `npx tsx` fetch on ~50 parallel
runners), and deciding right before the pull makes the window as narrow
as possible. The image list is the **same matrix ∩ build-success
intersection** that decides what gets redeployed, so a failed build can
never move a tag.
Also adds `showcase_build.yml` to `showcase_validate.yml`'s trigger
paths — the new test asserts against that file's live text, and without
the path a PR re-adding `:latest` would never run the test that catches
it.
## Tradeoffs / what stays open
- **Residual sub-second TOCTOU.** GHCR has no compare-and-swap on tags,
so two runs reading `:latest` simultaneously could still both advance.
This narrows the window from the whole build (~10 min) to inspect→retag.
Fully closing it means retiring the mutable staging tag and pinning
staging to digests the way prod already is — a change to the Railway
image-ref SSOT contract, not a workflow change. **Recommended
follow-up.**
- **The guard is inert for one build per image.** Today's `:latest`
images carry no labels (verified: `showcase-shell-dashboard:latest` has
no `Labels` at all), so the first post-merge build fails open and
advances unconditionally — same as today. Protection starts from the
second build of each image.
- Failure to retag exits non-zero, redding the redeploy job and stopping
the deploy. That is intended: redeploying against a tag that did not
move is exactly the silent false green being fixed.
## Proof
**Red/green on the live YAML.** `advance-latest-tag.test.ts` parses the
real `showcase_build.yml` (extending #6171's `redeploy-guard.test.ts`
pattern). Reverting the workflow to its pre-fix state: **14 failed / 23
passed**. With the fix: **37 passed**. Full `showcase/scripts` suite:
**2382 passed, 73 files**.
**The load-bearing predicate, verified live against the real incident
commits:**
```
compare/db75a04837...59f275eedc => behind (older run arriving late → DECLINE)
compare/59f275eedc...db75a04837 => ahead (newer run → advance)
compare/db75a04837...db75a04837 => identical
```
**Label reading, verified against a real multi-platform registry image**
— `docker buildx imagetools inspect ghcr.io/astral-sh/uv:latest` piped
through `extractRevisionLabel()` returns
`3010295ae7ff572de459987ad70db315a62ecd61`, matching `jq` exactly. The
platform-keyed shape is handled.
**Shell/jq transforms** exercised directly, including the empty-CSV edge
case (empty → empty, step skipped by its `if:`).
**Lint:** `actionlint` finding counts byte-identical to the pre-change
baseline (no new findings; the 11 pre-existing are unrelated). `zizmor
--min-severity low` with the repo config: **no findings**.
**Typecheck:** both new files are in `showcase/scripts/tsconfig.json`'s
include set and produce **zero** errors. Worth stating plainly: `nx
run-many -t check-types` **does not reach `showcase/scripts`** — the
project isn't in the nx graph and has no `check-types` target (there are
9 pre-existing type errors in sibling files, which is how I confirmed
it). So the typecheck above is mine, not CI's. The *tests* are gated:
`showcase_validate.yml` runs bare `pnpm exec vitest run` in
`showcase/scripts`, which auto-discovers the new file.
### What I could NOT prove
**I did not construct a real concurrent race on scratch branches.**
Doing it faithfully needs two builds pushing the same GHCR repo with
controlled finish ordering, which means merging to `main` — the only
branch the build workflow triggers on. No run IDs for a live race
demonstration; I am not implying one.
Unproven until this runs on main: that `docker buildx imagetools create`
retags cleanly under the runner's GHCR credentials, and that `npx tsx`
behaves in the redeploy jobs (it is already the established invocation
there — `redeploy-env.ts` — so this is low risk, not zero).
## Normal single-merge builds are unaffected
No concurrency group is added, so nothing queues or cancels. A lone
merge finds `:latest` at its own parent → `ahead` → advances, exactly as
before. Cost is one `imagetools inspect` + one `gh api` + one
registry-side retag per built service, in a job that already exists — no
extra job, no extra checkout, no change to build parallelism.
---
Branched from `db75a04837`; #6156/#6159 landed after, so this will need
main merged in before it goes green.
Probable conflict with the concurrent `git lfs pull` work in
`showcase_validate.yml` — my edit there is only the top-level `on:
paths:` list, so it should merge cleanly, but flagging it.
Six defects reported on staging. All six reproduce; five share one
cause.
## The shared cause
`built-in-agent/src/app/api/copilotkit/route.ts` maps **~20 demos to the
same prompt-less `createBuiltInAgent()`**. The reference wires each to
its own graph **and its own system prompt** (28 graphs in
`langgraph-python/langgraph.json`).
aimock replays a scripted tool-call sequence keyed on `userMessage` +
`context`, so D6 is green whether or not the model could have reasoned
its way there. This is `showcase/GOTCHAS.md` #8 verbatim — including its
stated "structural tell": *grep the integration's `route.ts` for demos
left in the generic fallthrough that LGP wires to a dedicated graph.*
## Per-defect
| Demo | Symptom | Root cause |
|---|---|---|
| `gen-ui-tool-based` | charts plot zeros | No prompt. The assistant
says so itself: *"I used placeholder values since no sales figures were
provided."* Nondeterministic — one run gave a real 0..220 axis, the next
`0 1 2 3 4` with no bars, which is how it passed review. |
| `gen-ui-agent` | wall of text, steps frozen at "step 1 of 4" |
`set_steps` + its `/steps` STATE_DELTA already worked; nothing told the
model to *walk* pending→in_progress→completed. |
| `subagents` | left panel always empty | `grep -rn delegations src/ \|
grep -v demos/subagents` → **no output**. Frontend reads
`agent.state.delegations`; the converter handles `/steps` and `/notes`
and has no `delegations` case. Chat works because the tools do run. |
| `declarative-json-render` | raw JSON for pills 2 & 3 | Intercepted the
SSE: **the wire is already one closing brace short** — nothing is
dropped in transport, and the client parser correctly requires a
balanced object. Pill 1 only works because the model copies the prompt's
worked example verbatim (`\$1.24M`, `+18% vs Q2`). Model-level JSON
enforcement had been removed with the note *"the system prompt already
enforces JSON-only output"* — it does not. |
| `a2ui-recovery` | 5 identical cards | Reuses declarative-gen-ui's
prompt with no "call once" constraint, while the pills literally ask it
to *"self-correct a malformed first attempt"*. The retry loop is
**inside** the tool and returns on first valid pass, so every supervisor
retry is pure duplication. |
| `declarative-gen-ui` | D4 though it works live | **The inverse case.**
Badges: `UI ✓ BE ✓ 1P ✗ D6 —` (*"gated — blocked by a lower rung"*). Not
a live bug — a fixture bug. See below. |
## The D4 cap (`declarative-gen-ui`, `a2ui-recovery`)
Both demos' secondary design-call fixtures gated on
`match.responseFormat: \"json_object\"`. That matcher **can never match
this backend**:
- The agent talks to the OpenAI *Responses* API, where JSON mode is
`text.format`.
- aimock has **no `text.format` handling at all** —
`responsesToCompletionRequest` forwards only a top-level
`response_format` (absent in 1.19.1, forwarded in 1.37.4), a key the
Responses API doesn't accept and this client doesn't send.
- So `effective.response_format?.type` is always `undefined` and
`router.ts` skips the fixture → the design call never matched → surface
never painted → D5 red, D6 never ran.
Re-keyed onto `match.toolName`, which aimock *does* normalize out of a
Responses request (`responsesToolsToCompletionsTools`): the outer call
declares `generate_a2ui`, the in-tool design call declares nothing.
Tool-less secondary fixtures moved last, because several pills' brief is
a substring of the pill text (`\"Build my Q2 revenue summary …\"`
contains `\"Q2 revenue summary\"`) and would otherwise win the outer
request.
## Verification
**Reproduced live** (staging, real LLM): the zero-axis chart plus its
self-incriminating message, the unbalanced JSON on the wire, and the
reference rendering the *same pill* correctly for contrast.
**Two mutation-verified test suites** — both fail with the fix reverted:
- `aimock-a2ui-routing.test.ts` drives aimock's **real `matchFixture`**.
On the old fixtures 14/16 cases fail with `no fixture matched the
secondary design call for brief \"…\"` — the D5 red, reproduced as a
unit test.
- `tanstack-factory.test.ts` covers the `/delegations` and `/steps`
deltas (3 delegation tests fail with the branch disabled; the 2 controls
still pass).
**No regressions:** `tsc --noEmit` is 61 errors before *and* after (all
pre-existing — `gpt-5.4` missing from the adapter's model union, zod
v3/v4 skew); oxlint clean on changed files; the repo-wide fixture
validator passes (858 assertions).
`text.format` was verified by reading the request mapping in the
**pinned** `@tanstack/ai-openai@0.15.6` → `@tanstack/openai-base@0.9.2`:
`modelOptions` is spread straight into `responses.create()`,
`validateTextProviderOptions` only inspects
`metadata`/`conversation`/`previous_response_id`, and the adapter sets
`text.format` itself only when an `outputSchema` is passed (none here,
so nothing is clobbered).
## What is NOT verified
**The four prompt/`text.format` changes have not had a real-LLM
click-through** — that needs an OpenAI key this environment doesn't
have, and D6/aimock cannot verify them by construction (that's the whole
point of gotcha #8). Reasoning is documented inline at each site. This
area has burned the repo before: a previous `response_format` attempt
made the call return an empty string "verified against real OpenAI",
which is why `text.format` — the Responses API's own param — is used
instead. **Worth a live click-through on the staging deploy before this
is considered closed.**
## Deliberate non-goals
- **`a2ui-recovery` only demonstrates recovery under aimock.** The
heal/exhaust branches need a designer LLM that emits invalid surfaces on
demand; a real one succeeds on attempt 1. The single-call constraint
fixes the 5-card bug and makes it honest; making recovery visible live
needs deliberate fault injection — a product decision, recorded in
`PARITY_NOTES.md`.
- **The reference pins `openai:gpt-4o-mini`** (`gen_ui_agent.py:92`).
Flagged, not touched.
## Docs
`GOTCHAS.md` gains built-in-agent as a second worked instance of #8
(including that the masking runs *both* ways, and that `1P ✗ D6 —` means
gated, not failing), plus the `text.format` and dead-`responseFormat`
traps. `PARITY_NOTES.md` records that the named-agent registry is
**not** prompt-neutral, and that any new `state.<slot>` needs a
converter branch.
Every affected cell was D6-green (or D4-gated) while the deployed demo was
visibly broken. The common cause is GOTCHAS #8: aimock replays a scripted
tool-call sequence keyed on userMessage + context, so the fixture answers a
question the model was never asked.
Prompt gaps — ~20 demos shared one prompt-less `createBuiltInAgent()` where the
reference wires each to its own graph AND its own system prompt. Adds
`createBuiltInAgent({ systemPrompt })` + `demo-prompts.ts`, ported from the
reference graphs:
- gen-ui-tool-based plotted zeros; the assistant said "I used placeholder values
since no sales figures were provided". Nondeterministic — some runs invented
real values, which is how it passed review.
- gen-ui-agent published its plan once then narrated, freezing the progress card
on step 1 of 4.
- a2ui-recovery painted five identical cards: nothing constrained the supervisor
to one `generate_a2ui` call, and the pills literally ask it to "self-correct".
The retry loop lives inside the tool, so supervisor retries are duplication.
subagents' delegation panel was permanently empty — the frontend reads
`agent.state.delegations` and no code emitted that slot. The converter now emits
a `/delegations` delta per sub-agent result (whole-array `add`, since initial
state is `{}` and strict fast-json-patch rejects unresolvable paths while
@ag-ui/client swallows the throw), and ports the reference's
`_MAX_CRITIQUE_ITERATIONS = 1` cap.
declarative-json-render dumped raw JSON for any prompt the model couldn't crib
from the worked example. Captured the SSE: the wire is already one closing brace
short, so nothing is dropped in transport — model-level enforcement had been
removed on the grounds that "the system prompt already enforces JSON-only
output". It does not. Restores it via the Responses API's `text.format`, the
param the removed `response_format` maps to, verified against the pinned
@tanstack/ai-openai@0.15.6 -> @tanstack/openai-base@0.9.2 request mapping.
declarative-gen-ui and a2ui-recovery were the inverse: capped at D4 (UI ✓ BE ✓
1P ✗ D6 gated) while working live. Their secondary design-call fixtures gated on
`match.responseFormat`, which aimock can never satisfy here — it has no
`text.format` handling and only forwards a top-level `response_format` the
Responses API doesn't accept. Re-keyed onto `match.toolName` (the outer call
declares `generate_a2ui`, the in-tool design call declares nothing) with the
tool-less fixtures moved last, since several pills' brief is a substring of the
pill text.
Tests (both mutation-verified — they fail with the fix reverted):
- aimock-a2ui-routing.test.ts drives aimock's real `matchFixture`; 14/16 cases
fail on the old fixtures with "no fixture matched the secondary design call".
- tanstack-factory.test.ts covers the `/delegations` and `/steps` deltas.
Typecheck unchanged at 61 pre-existing errors; oxlint clean on changed files.
NOT verified live: the four prompt/`text.format` changes need a real-LLM
click-through, which needs a key this environment doesn't have. Reasoning and
the exact request mapping are documented inline.
PR #6130 added `src/middleware.ts`, which sets the `x-pathname` header that
`src/app/demos/layout.tsx`'s `generateMetadata()` reads. That header is what
makes the layout actually call `loadDemoIndex()` — a request-time
`readFileSync(process.cwd()/manifest.yaml)`. The Dockerfile's runner stage
never copied `manifest.yaml`, so every `/demos/*` route now 500s with "An
error occurred in the Server Components render" (ENOENT), while the
statically-prerendered home page keeps returning 200.
On the dashboard that reads as "service is up, every cell pinned at D3": D4
fails on `page.type` waiting for `textarea`, D5 times out, D6 fails on
`waitForSelector('[role="textbox"]')` — the chat input never mounts because
the page is an error boundary.
langgraph-python hit this exact bug and fixed it with the same one-line COPY;
ms-agent-dotnet is the only integration shipping `middleware.ts` without it.
Adds a ratchet test over that invariant (mutation-verified: fails with the
COPY removed).
## What
Brings the **ms-agent-dotnet** (Microsoft Agent Framework .NET) showcase
integration from D5 to **D6**, using **langgraph-python** as the
north-star reference.
### 1. Frontend parity with langgraph-python
Restores near-identical frontends where ms-agent-dotnet had drifted,
while **preserving the load-bearing .NET adaptations** (per the showcase
iron rules — differences belong in fixtures/minimal backend, not the
shared frontend):
- Root shell: `globals.css` (Tailwind `@theme` block + brand green),
manifest-driven index `page.tsx`, `layout.tsx`, new `middleware.ts`
(`x-pathname`), `tsconfig` include.
- `declarative-gen-ui` subtree restored (fixes divergent pill testids
the shared probe asserts).
- Doc-snippet `@region` markers, import-style normalization, `subagents`
revert, stale-file cleanup, `auth` inspector flag.
- **Kept** (load-bearing, not reverted): `parse-json-result` 3-layer
unwrap, multimodal legacy-shim, tool-based `hitl` (MAF has no
`interrupt()`), `agent-config` `properties=`.
### 2. shared-state-streaming → per-token (removed from
`not_supported_features`)
`write_document`'s `document` arg now streams into `state.document`
per-token via a `createSharedStateStreamingAgent` route shim (mirrors
the proven `createGenUiAgent` bridge, with a partial-JSON string
decoder), since the .NET AG-UI host has no `predict_state_config`.
### 3. a2ui-recovery cell (new)
First MS-Agent-Framework implementation of the A2UI
validate→retry→`a2ui_recovery_exhausted` recovery loop. Because the MAF
AG-UI adapter can't emit the custom `ACTIVITY_SNAPSHOT{status:"failed"}`
the exhausted card needs, it's a **raw-SSE `MapPost` endpoint**
(`RecoveryAgent.cs`) — the same adapter-bypass pattern already shipped
for `/multimodal`. Adds the demo frontend, API route, deterministic
aimock fixture (heal seq0-invalid→seq1-valid; exhaust always-invalid),
and a unique per-slug `PROMPTS` entry in the shared probe.
### 4. threadid-frontend-tool-roundtrip demo (parity)
Added for demo-set parity (reuses the `frontend_tools` passthrough; not
a D6-scored feature, mirroring the reference).
`gen-ui-interrupt` / `interrupt-headless` remain honestly quarantined
(upstream `@copilotkit/react-core` `useInterrupt` resume-path bug — not
a backend gap).
## Verification
- Code was authored in parallel worktree-isolated slots, each
cross-verified against the reference + the shared probe contracts; the
a2ui-recovery fixture was cross-checked against
`RecoveryAgent.ValidateComponents`.
- Local D6 harness: the image builds and the stack + probes run, but
**full local green was blocked by Windows-only harness friction**
(`core.symlinks=false` breaks `stage_shared`'s `[ -L ]` materialization;
`--direct` doesn't context-scope the `x-aimock-context` header so
context-keyed a2ui fixtures miss). These are environmental, not code
issues. **Relying on CI's Linux harness (real symlinks + fleet worker)
for authoritative D6.**
## Follow-up (not in this PR)
- `stage_shared()` should also materialize Windows symlink-as-file
entries (detect a regular file whose content is a relative path), so
forced local rebuilds work on `core.symlinks=false` checkouts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The auto-format bot reflowed both captured registry payloads. Harmless this
time — oxfmt only rewraps arrays, it does not reorder keys — but the whole
value of these two files is that they are verbatim `imagetools inspect`
output. A formatter in the loop means the next re-capture churns, and the
"this is exactly what the registry emits" claim stops being checkable.
Exempt the fixture directory in .oxfmtrc.json, restore the captured bytes,
and say so in the provenance comment so nobody tidies them by hand.
The previous suite was 37/37 green against a workflow that could not have
run at all: the guard step had no `packages: write` and no GHCR login, so
every `imagetools create` would have 401'd. Asserting that a step EXISTS
proves nothing about whether it can succeed. Each change below was checked
by reintroducing the defect and confirming the test goes red.
Registry-auth precondition (the miss that let the above ship)
- Both redeploy jobs must declare `packages: write` AND a `docker/login-action`
step for ghcr.io, ordered BEFORE the guard. Dropping either reds the suite.
The intersection, executed rather than restated
- The old test only checked that the string `images=` appears in the compute
step; swapping its jq for the full matrix — so a FAILED build moves
`:latest` — kept it green. The real `changed` shell now RUNS, against the
real ALL_SERVICES matrix read out of the workflow, and the emitted
$GITHUB_OUTPUT is asserted: a failed build is in neither set, and the
`skip_build` slot is in `services` but NOT in `images` (handing it to the
guard fails "manifest unknown" and blocks the redeploy for the whole fleet).
Same treatment for the starter lane.
Fixtures joined to real registry output
- `extractRevisionLabel` was pinned to hand-written payloads that were never
compared with reality. Since `readLatestRevision` maps every failure to
null, and null ADVANCES, a parser that silently never matches yields a
permanently-blind guard with a fully green suite. Both fixtures are now
verbatim `docker buildx imagetools inspect --format '{{json .Image}}'`
output (buildx v0.35.0), unformatted so key order survives: a platform-keyed
multi-arch image carrying the label, and our own `:latest`, which turns out
to be a bare config object with NO labels at all. The label key used in the
injection test is read from the workflow's own `labels:` input, so parser
and producer cannot drift apart silently.
Closed vacuities
- The `:latest` ban read one step's `tags` and was vacuously green on an empty
list. It now covers every tagged step plus hand-rolled `docker push`/`docker
tag`/`imagetools create`, and fails on an empty `tags`.
- The `cancel-in-progress` ban read only top-level config; a job-level
`concurrency` on `build` reproduced the exact harm and stayed green. All
jobs are checked now.
- `already-current` is keyed on the digest, matching the script.
- The unreachable `(null, "ahead")` row is labelled as the defensive
input-space case it is, and a new test pins that the real flow never calls
compare() with an unknown revision.
New coverage: classifyProbeFailure (incl. a REAL execFileSync timeout, and
the two precedence traps — gh's rate-limited 403 is throttling, not auth; a
registry's 404-with-denied is auth, not absent), readFlag, escapeAnnotationData
(a forged `\n::error::` stays inert), isDirectInvocation through a symlink,
the `::error` annotation on fleet failure, and the digest-mismatch advance.
Also: GITHUB_SHA / GITHUB_REPOSITORY are no longer shadowed in the workflow.
The runner exports both and the script reads process.env, so the old
assertions pinned a redundancy rather than a capability — removed together,
as the comment there required. Workflow-reading scaffolding duplicated with
redeploy-guard.test.ts is extracted and the YAML parse memoized.
Targeted repairs around the decision logic, which is unchanged: the
compare direction, the `["behind"]`-only decline, and the fail-open
invariant are all preserved.
Digest/identity
- `already-current` keyed on the revision LABEL treated a rebuild of the
same commit (CVE refresh, cache-miss, retry) as a no-op, stranding
`:latest` on the older digest while digest-pinned prod moved. It now
compares manifest DIGESTS, via a new `GuardIo.readDigest`, and only
when the label claims we are already current — so the common path pays
nothing.
- Retag passes `--prefer-index=false`. buildx carbon-copies a single
index source (digest preserved) but WRAPS a bare manifest in a new
index (digest changes) because `--prefer-index` defaults true. Our
images are indexes only because provenance is enabled elsewhere; this
pins the property regardless.
Observability — fail-open stays, silence does not
- `readLatestRevision` / `compare` no longer swallow the error. Failures
are classified (absent / unauthorized / throttled / timeout /
unavailable / unknown) and everything but a genuinely absent tag emits
a `::warning` saying the guard has stopped guarding. Throttling is the
likeliest failure under the concurrency this exists to handle and was
previously indistinguishable from a clean first build.
- Annotations are properly escaped. Multi-line registry stderr was
folding the real reason out of the annotation meant to report it;
encoding newlines also makes an injected `::` inert.
- Subprocess timeout (60s, `ADVANCE_LATEST_TAG_TIMEOUT_MS`). A hung
registry previously burned the job budget and concluded `cancelled`.
- The fleet result now names which images advanced, were unchanged, or
declined — not just which failed.
Correctness / hygiene
- `extractRevisionLabel` collects every occurrence and answers only when
they agree, so a multi-config payload no longer returns whichever label
key ordering happened to surface last. Dropped the dead cycle guard.
- Flag parser no longer binds a following flag as a value (`--sha --repo
x` yielded the literal `"--repo"` as the sha); `--flag=value` accepted;
an empty `--images` is the documented no-op instead of exit 2.
- Entry guard canonicalises both sides, so a symlinked invocation runs
instead of exiting 0 having done nothing.
- Docs: corrected the never-rethrown claim, the unreachable `identical`
example, the self-contradicting `npx tsx` rationale, and the
preconditions (a retag is a registry WRITE and needs `packages: write`
plus a login, which the redeploy jobs do not currently have).
The `<1.0.0` upper bound changes the text of an existing `[FAIL]` line
from `ag2 is not an exact pin (>=0.9.0)` to `(>=0.9.0,<1.0.0)`, which
shifts the ratchet's SHA-256 over the sorted FAIL set.
The FAIL *set* is otherwise identical -- count stays 31, nothing healed
and nothing regressed. Verified by reproducing the committed baseline
hash b47ca987 on a pristine origin/main tree, then diffing the FAIL
lines against the fixed tree; exactly one line differs:
-[FAIL] ag2: ag2 is not an exact pin (>=0.9.0)
+[FAIL] ag2: ag2 is not an exact pin (>=0.9.0,<1.0.0)
`ag2` matches FRAMEWORK_PATTERNS, which demands an exact `==` pin, so
the dep was already in the baseline's drift set before this change and
remains in it after. Hash-only update; the count is untouched, so the
"never raise the count without sign-off" invariant is not engaged.
Three merges landed on main within 34 seconds on 2026-07-26. The build
workflow has no concurrency group by design, so all three ran at once and
raced to push the same `:latest` tags. The NEWEST commit's build finished
FIRST, so the two older builds overwrote it:
run 30190815370 7b28934387 (#6162) 06:18:05 -> 06:29:44
run 30190823203 59f275eedc (#6161) 06:18:21 -> 06:29:44
run 30190831480 db75a04837 (#6158) 06:18:39 -> 06:29:13 <- NEWEST
Per-service, the older commit beat the newer one on every shared slot:
shell-dashboard (+1s), showcase-harness (+4s), shell (+11s), shell-dojo
(+16s). All three runs reported `success`. Staging served pre-#6158 code
while CI, the redeploy gate and deploy verification all looked clean.
A concurrency group is the wrong fix. `detect-changes` builds a per-push,
path-filtered matrix, so concurrent runs build overlapping but NON-IDENTICAL
service sets -- 7b28934387 was the only run building ag2, agno, langroid,
spring-ai, strands and 10 others. `cancel-in-progress: true` would have
dropped those entirely, trading a stale-image bug for a never-shipped bug.
`cancel-in-progress: false` is no better: GitHub keeps at most one pending
run per group and cancels any previously-pending one.
Instead make the mutable pointer monotonic. The build step now pushes only
the immutable `:<sha>` tag plus an `org.opencontainers.image.revision`
label. A new guard then advances `:latest` unless the tag already points at
a DESCENDANT of the commit being built, which is exactly the regression
case. Every ambiguous state (no tag, unlabelled legacy image, unreachable
API, diverged history) advances -- a stuck `:latest` is the failure mode
being fixed, so the guard declines only on positive proof of regression.
The guard runs in redeploy-staging / redeploy-staging-starters, immediately
before the Railway pull that consumes `:latest`, over the same
matrix-intersect-build-success set that decides what gets redeployed. That
placement keeps the read->retag window minimal and avoids an unpinned
`npx tsx` fetch on ~50 parallel build slots that have no Node.
Also adds showcase_build.yml to showcase_validate.yml's trigger paths: the
new test asserts against that file's live text, and without the path a PR
re-adding `:latest` to the push step would never run the test that catches it.
Prod is unaffected -- verify-railway-image-refs.ts already pins prod to
`@sha256:<digest>`; only staging consumes the mutable tag.
## What this is
`multimodal` (Attachments) has never worked on **`llamaindex`** or
**`crewai-crews`**, but both manifests
listed it under `features`, so the fleet probed it and reported **red**.
A red chip says "this regressed".
The truth is "this was never built". This PR marks both cells
**unsupported** instead. It does **not**
implement the feature, and it does **not** suppress the cell — the cell
still exists, the demo stays wired,
and the chip renders the 🚫 unsupported glyph.
## Mechanism used (existing, not invented)
The repo already has exactly one way to declare a feature unsupported
for an integration:
**`not_supported_features` in the integration's `manifest.yaml`**. The
full derivation chain:
| Step | Location |
|---|---|
| Declaration | `showcase/integrations/<slug>/manifest.yaml` ->
`not_supported_features:` |
| Schema | `showcase/shared/manifest.schema.json` — *"feature IDs that
this integration's framework cannot architecturally support … excluded
from parity computation"* |
| Status fold |
`showcase/harness/src/shared/catalog/catalog-flatten.ts:239` —
`determineCellStatus()` checks `not_supported_features` **first**,
returns `status: "unsupported"` |
| Input mapping |
`showcase/harness/src/shared/cell-model/catalog-input.ts:53` —
`isSupported: cell.status !== "unsupported"` |
| Model | `showcase/harness/src/shared/cell-model/cell-model.ts:847` —
`if (!isSupported) return UNSUPPORTED;` (the frozen singleton at `:551`:
`supported: false`, `chipColor: "gray"`, `isRegression: false`) |
| `/api/matrix` | `showcase/harness/src/http/matrix.ts:206` ->
`matrix-compute.ts:53` — projects that same model, so the API value
**is** the rendered chip by construction |
| Render |
`showcase/shell-dashboard/src/components/unified-cell.tsx:308` — `if
(!model.supported)` renders `data-testid="unified-cell-unsupported"`
with 🚫 and `title="Not supported by this framework"` |
The mechanical guard at `catalog-flatten.ts:169` rejects a feature that
appears in **both** `features` and
`not_supported_features`, so each entry was **moved**, not duplicated.
Note this mechanism is strictly stronger than a probe-side skip:
`buildCellModel` returns `UNSUPPORTED`
regardless of what the live PocketBase row says. Verified against the
existing not-supported cells on these
same two integrations, which carry **green** PB rows and still render 🚫:
```
llamaindex/gen-ui-interrupt matrix: chip=gray supported=False | PB rows: d5=green d6=green e2e=green
llamaindex/shared-state-streaming matrix: chip=gray supported=False | PB rows: d5=green d6=green e2e=green
crewai-crews/mcp-apps matrix: chip=gray supported=False | PB rows: d5=green d6=green e2e=green
```
So the cell can never read green *or* red once declared here — which is
the property we want.
## Per-integration reason (recorded inline in each manifest)
**`llamaindex` — upstream gap.** The pinned
`llama-index-protocols-ag-ui==0.2.2`
(`llama_index/protocols/ag_ui/utils.py:82-85`) passes an AG-UI
`UserMessage`'s `content` straight into
`ChatMessage(...)`. A text-only turn passes a plain string (fine — every
other llamaindex cell is green);
an attachment turn passes a **list** of AG-UI content-part models, which
pydantic routes into
`ChatMessage.blocks`, a union discriminated on `block_type` — a field
AG-UI's `TextInputContent` /
`ImageInputContent` / `BinaryInputContent` do not carry. Live backend
error:
```
pydantic_core._pydantic_core.ValidationError: 3 validation errors for ChatMessage
blocks.0
Unable to extract tag using discriminator 'block_type' [type=union_tag_not_found,
input_value=TextInputContent(type='te... image I just attached'), input_type=TextInputContent]
```
Fixing it needs a content-part ->
`TextBlock`/`ImageBlock`/`DocumentBlock` conversion, upstream or on our
side of `get_ag_ui_workflow_router`. This PR also corrects that module's
docstring, which asserted the
router *"normalizes them via the OpenAI `input_file` path"* — it does
not.
**`crewai-crews` — never implemented, ours.** `src/agent_server.py`
registers a dedicated AG-UI endpoint for
every other demo but **no `/multimodal` route** (the block ends at the
catch-all
`add_crewai_crew_fastapi_endpoint(app, LatestAiDevelopment(), "/")`),
and there is no
`src/agents/multimodal_agent.py`. So
`src/app/api/copilotkit-multimodal/route.ts` aliases the generic shared
crew, which has no vision handling and dies on a content-part message:
```
[CopilotKit] Error (agent_run_error_event): Error: thread=… run=…: CrewAI flow failed; see server logs
```
That route file's own header comment already concedes the gap ("A
dedicated per-demo crew with vision-tuned
agent prompts is tracked as follow-up work").
## Proof
Method: the **real** `GET /api/matrix` handler (`registerMatrixRoute`)
driven over the **real** production
PocketBase `status` collection (all 3082 rows, fetched verbatim from
`showcase-pocketbase-production.up.railway.app`) and the **real**
on-disk manifests (default `loadCells` =
`buildCatalogCells`, the single flattening authority). Same rows, same
fixed clock
(`now = max(observed_at) + 60s = 1784932566438`) for both runs — the
only variable is the manifest diff.
### BEFORE — real `GET /api/matrix`, real prod PocketBase rows,
manifests at `origin/main` (38613623f4)
```
llamaindex/multimodal
{"chipColor": "red", "supported": true, "achievedDepth": 4, "ceilingDepth": 6, "isRegression": true, "surfaceState": "red", "isStaleCell": false}
crewai-crews/multimodal
{"chipColor": "red", "supported": true, "achievedDepth": 4, "ceilingDepth": 6, "isRegression": true, "surfaceState": "red", "isStaleCell": false}
```
### AFTER — same route, same rows, same clock, manifests with this PR
```
llamaindex/multimodal
{"chipColor": "gray", "supported": false, "achievedDepth": 0, "ceilingDepth": 0, "isRegression": false, "surfaceState": "gray", "isStaleCell": false}
crewai-crews/multimodal
{"chipColor": "gray", "supported": false, "achievedDepth": 0, "ceilingDepth": 0, "isRegression": false, "surfaceState": "gray", "isStaleCell": false}
```
### Full multimodal column, AFTER (control)
```
ag2 chip=green supported=True depth=6/6 [unchanged]
agno chip=red supported=True depth=4/6 [unchanged]
built-in-agent chip=red supported=True depth=4/6 [unchanged]
claude-sdk-python chip=green supported=True depth=6/6 [unchanged]
claude-sdk-typescript chip=green supported=True depth=6/6 [unchanged]
crewai-crews chip=gray supported=False depth=0/0 [CHANGED]
google-adk chip=green supported=True depth=6/6 [unchanged]
langgraph-fastapi chip=green supported=True depth=6/6 [unchanged]
langgraph-python chip=green supported=True depth=6/6 [unchanged]
langgraph-typescript chip=green supported=True depth=6/6 [unchanged]
langroid chip=green supported=True depth=6/6 [unchanged]
llamaindex chip=gray supported=False depth=0/0 [CHANGED]
mastra chip=red supported=True depth=4/6 [unchanged]
ms-agent-dotnet chip=green supported=True depth=6/6 [unchanged]
ms-agent-harness-dotnet chip=green supported=True depth=6/6 [unchanged]
ms-agent-python chip=red supported=True depth=4/6 [unchanged]
pydantic-ai chip=green supported=True depth=6/6 [unchanged]
spring-ai chip=green supported=True depth=6/6 [unchanged]
strands chip=green supported=True depth=6/6 [unchanged]
strands-typescript chip=green supported=True depth=6/6 [unchanged]
```
### CONTROL — an already-supported multimodal cell is untouched
`langgraph-python/multimodal` (the reference integration) is `chip=green
supported=true depth=6/6` **before
and after**, byte-identical. So is every other integration's multimodal
cell, including the four that are
red for unrelated reasons (`agno`, `mastra`, `ms-agent-python`,
`built-in-agent`) — those stay **red**, they
were not swept up.
### Complete set of cells whose state changed — exactly 2 of 1000
Diffing every field of every cell in the `/api/matrix` body, before vs
after (identical 1000-cell keyset):
```
crewai-crews/multimodal red -> unsupported
llamaindex/multimodal red -> unsupported
```
Nothing else. Aggregate confirms no green was manufactured:
```
BEFORE AFTER
chipColor green=632 gray=317 red=51 green=632 gray=319 red=49
supported=false 94 96
```
`green` is **unchanged at 632** — this PR turned two reds into
unsupported and created zero greens.
Second, independent derivation (the dashboard's own generated
`catalog.json`, via
`showcase/scripts/generate-registry.ts`, which runs full AJV validation
first) agrees, and also changes
exactly 2 cells:
```
crewai-crews/multimodal (integrated): status wired -> unsupported, max_depth 4 -> 0
llamaindex/multimodal (integrated): status wired -> unsupported, max_depth 4 -> 0
metadata BEFORE: total_cells 980, wired 688, stub 0, unshipped 198, unsupported 94, docs_only 20
metadata AFTER : total_cells 980, wired 686, stub 0, unshipped 198, unsupported 96, docs_only 20
```
`parity_tier` is unchanged on both columns (already `partial`), and no
other integration's cells moved.
### Live dashboard render
`shell-dashboard` run locally against the prod PocketBase, Playwright
over the real DOM. On the
`feature-row-multimodal` ("Attachments") row, exactly two of twenty
columns carry
`data-testid="unified-cell-unsupported"`:
```
CrewAI (Crews) unsupportedGlyph=TRUE text="🚫"
LlamaIndex unsupportedGlyph=TRUE text="🚫"
LangGraph (Python) unsupportedGlyph=false text="Demo ↗ Code </> D6 UI ✓ BE ✓ 1P ✓ D6 ✓"
Agno unsupportedGlyph=false text="Demo ↗ Code </> D4 UI ✓ BE ✓ 1P ✗ D6 —"
… 16 more, all unsupportedGlyph=false
```
Visually: a grey outlined 🚫 badge in those two columns — plainly not a
green `D6` pill, and not a red `✗`.
Control row `feature-row-agentic-chat` has `unsupportedGlyph=false` in
all twenty columns.
## Quality gates
- `oxfmt --check` on all three changed files — clean
- `oxlint` — 0 warnings, 0 errors
- `generate-registry.ts --validate-only` (AJV against
`manifest.schema.json`) — passes; confirms no
`features` / `not_supported_features` overlap
- `shell-dashboard`: production build ✓, **68 test files / 1333 tests
passed**, 1 skipped
- `harness`: `tsc --noEmit` clean; **171 test files / 3625 tests
passed**. 3 pre-existing failures
(`d5-mapping-drift`, `frontend-matrix`, `d5-representatives` — the last
complains about
`browser-use-smoke`) fail **identically on clean `origin/main`**,
verified by stashing this diff and
re-running. Unrelated to this change.
- Diff is 3 files, no lockfile churn, no generated artifacts, no stray
worktree files.
## Deliberately not done
- Not implementing the feature for either integration (the upstream
conversion for llamaindex and the
vision crew for crewai-crews remain open work).
- Not touching `agno` / `mastra` / `ms-agent-python` / `built-in-agent`,
whose multimodal cells are red for
four unrelated reasons and stay red here.
- Not loosening the D5 assertion, not dropping `skipSend`, not deleting
the cell, not skipping the probe,
and not special-casing the probe to pass — every one of those would
green a broken cell.
## Follow-up commit: CI-caught wired-count bound
The first push failed **Showcase: Validate** -> "Run build pipeline
tests" (the `showcase/scripts` vitest,
which I had not run locally — my mistake; the harness and dashboard
suites both passed):
```
FAIL __tests__/generate-catalog.test.ts > parity tier: crewai-crews wired cells render
at_parity or partial against the elected reference
AssertionError: expected 29 to be greater than or equal to 30
❯ __tests__/generate-catalog.test.ts:257:32
```
Real and caused by this PR: reclassifying `crewai-crews/multimodal` from
`wired` to `unsupported` drops that
integration's wired-cell count 30 -> 29.
That `30` is a **snapshot floor, not an invariant** — the assertion's
own comment states the partial parity
tier requires only `intersection >= 3` with the reference's wired set.
29 clears that by a wide margin, and
the two tier assertions immediately below it (`parity_tier` in
`["at_parity","partial"]`, uniform across the
column) still pass untouched. So the floor was updated to 29 with a
comment recording exactly which cell
moved and why, rather than being deleted or loosened to a no-op.
Nothing else in that suite moved: `metadata.total_cells` is still 980
and the
`wired + stub + unshipped + unsupported == total_cells` sum invariant
still holds (686 + 0 + 198 + 96 = 980).
Re-run locally after the fix: **72 test files / 2323 tests passed, 0
failed.**
`oxfmt --check` and `oxlint` clean on the changed test file.
Build run 30162773601 (merge of #6160) forced a full-fleet rebuild; 5 of
28 slots were killed by their `timeout-minutes` budget, the other 23
built and WERE redeployed to staging, and the run emitted no signal at
all: `notify` was skipped, so no Slack alert and no PR comment, and the
run rolled up to conclusion `cancelled`.
No existing guard could catch it. Measured on purpose-built probe run
30166429073 (matrix leg killed by `timeout-minutes`, sibling leg green):
killed leg `job.status` ........ cancelled
matrix rollup `needs.*.result` . cancelled
`if: cancelled()` .............. SKIPPED (evaluated FALSE)
`if: failure()` ................ SKIPPED (evaluated FALSE)
pre-fix `notify` condition ..... SKIPPED <- the bug
post-fix `notify` condition .... RAN <- the fix
run conclusion ................. cancelled
So `failure() || cancelled()` would NOT have fixed this. The signal has
to come from the per-slot build results.
- stop laundering `cancelled` into `skipped` in the per-slot writer
- expose `any_cancelled` / `cancelled_services` from the aggregator job
- add `notify-cancelled-builds`: exits non-zero so the run concludes
`failure` rather than `cancelled` (a slot killed by its timeout budget
is a failure, and `cancelled` is what suppressed everything), and
Slacks the affected service names
- add the `any_cancelled` clause to `notify` so the merge author gets the
PR comment, with wording that distinguishes incomplete from failed
`!cancelled()` is retained on both jobs as the intentional-vs-flake
discriminator: a human cancelling the whole RUN makes `cancelled()` true
and stays silent, while a leg-level timeout leaves it false and alerts.
Extends redeploy-guard.test.ts, which evaluates the LIVE `if:` strings
from the workflow, with the exact production scenario. It pins the
pre-fix guard string as a literal so the test proves the difference the
fix makes, not merely the current behaviour.
`job.status` for a matrix slot is success|failure|cancelled, but the
per-slot writer laundered cancelled into `skipped` before publishing its
result, so a slot killed by `timeout-minutes` became indistinguishable
from one that legitimately never built.
That erased the only signal that could tell a partially-cancelled fleet
build from a clean one. GitHub's status functions cannot recover it:
`cancelled()` is documented as "returns true if the workflow was
canceled" (workflow-scoped, and FALSE for a leg-only cancel), and a
cancelled ancestor is not a FAILED ancestor so `failure()` is false too.
Add `cancelled` to the BuildOutcome contract, add `cancelledSet()`, and
have the aggregator publish `any_cancelled` + `cancelled_services`
alongside `any_success`. `successSet` still excludes cancelled slots, so
the redeploy intersection is unchanged — a slot that pushed no image
still cannot enter the redeploy CSV.
`showcase/integrations/mastra/shared-tools` was a real committed directory
where a symlink into the single source of truth belongs, violating the
single-source symlink mechanism documented in showcase/AGENTS.md.
Root cause: commit 534cd1efa7 ("fix(showcase): D5 integration fixes across
12 frameworks") deleted the symlink and committed 14 real files in its place
— the classic clobber the guard in showcase/scripts/validate-shared-symlinks.ts
was written to catch. The symlink was originally added by 93d5815cdb and
pointed at `../../shared/typescript/tools`.
Divergence inventory: NONE. All 14 files were byte-identical to
showcase/shared/typescript/tools (identical git blob hashes and sha256s), so
nothing mastra-specific, stale, or additive was carried in the copy. Restoring
the symlink is therefore a pure structural fix with zero content change — the
restored link blob is the same object (ddc634b6) the pre-erosion symlink had.
Also removes the healed `mastra/shared-tools` key from
validate-shared-symlinks.baseline.json, per the shrink-only ratchet (the
validator reports stale entries specifically so this is mechanical).
langgraph-typescript/shared-tools and claude-sdk-typescript/shared-tools are
eroded the same way by the same commit and are also byte-identical to the
shared source; they stay baselined here and are left for follow-up so each
conversion carries its own behavior proof.
`generate-catalog.test.ts` pinned crewai-crews' wired-cell count at >= 30. Moving
`multimodal` from `features` to `not_supported_features` reclassifies that cell
from `wired` to `unsupported`, so the real count is 29 and CI's
`showcase/scripts` vitest failed.
The bound is a snapshot floor, not an invariant: the test's own comment says the
partial parity tier only requires intersection >= 3 with the reference's wired
set. 29 still clears that by a wide margin, so the floor tracks the manifest.
Updated to 29 with a note recording which cell moved and why, so the next reader
does not read the decrement as a regression.
Ran Shemtov