The verify-image-refs gate (verify-railway-image-refs.ts) failed because the
live Railway service `showcase-crewai-conversational-flows` had no entry in the
SERVICES map, tripping the Railway->SSOT drift check (1 untracked service).
Add the service as a STAGING-ONLY entry: the live Railway service currently has
a serviceInstance in staging only (no prod instance is provisioned), so the
env-map schema declares only the env that exists. All values (serviceId,
staging instanceId, domain, healthcheckPath) are read verbatim from the live
Railway API, not guessed. ciBuilt:false because the integration is wired only
into the PR-check build (showcase_build_check.yml), not showcase_build.yml's
ALL_SERVICES matrix, so it stays out of CI_BUILT_SERVICES.
Regenerate railway-envs.generated.json and update the count/coverage
assertions in the affected tests (SSOT now has 42 services; findMissingServices
is intentionally asymmetric for the staging-only entry: 41 prod, 42 staging).
## The race, with evidence
Three PRs merged within 34 seconds on 2026-07-26. `Showcase: Build &
Push` has **no concurrency group** (deliberately), so all three ran
simultaneously and raced to push the same `:latest` tags.
| run | commit | PR | start → end |
|---|---|---|---|
| `30190815370` | `7b28934387` | #6162 | 06:18:05 → 06:29:44 |
| `30190823203` | `59f275eedc` | #6161 | 06:18:21 → 06:29:44 |
| `30190831480` | `db75a04837` | #6158 | 06:18:39 → **06:29:13** ←
newest, finished FIRST |
The newest commit finished first, so the older builds overwrote its
`:latest`. Per-service job completion times — older beating newer on
every shared slot:
| service | newer (`db75a04837`) | older (`59f275eedc`) | older won by |
|---|---|---|---|
| `shell-dashboard` | 06:25:33 | 06:25:34 | +1s |
| `showcase-harness` | 06:27:51 | 06:27:55 | +4s |
| `shell` | 06:27:17 | 06:27:28 | +11s |
| `shell-dojo` | 06:25:10 | 06:25:26 | +16s |
**All three runs reported `success`.** Staging served pre-#6158 code
while CI, the redeploy gate and deploy verification all looked clean.
Same failure class as #6171: a success that doesn't mean what it says.
## What I verified in YAML vs took on trust
Verified by reading the files / querying the API:
- **Tagging** — `showcase_build.yml` pushed `:latest` **and** `:${{
github.sha }}` in one `depot/build-push-action` step, in both the
`build` and `build-starters` matrices. **A per-commit sha tag already
existed**; confirmed in GHCR (`showcase-shell-dashboard` has digests
tagged `db75a04837…`, `59f275eedc…`, `d28384a2eb…`).
- **Nothing serialized the pushes.** No concurrency group; confirmed the
header comment states this is intentional.
- **Deploy consumes `:latest`** — `verify-railway-image-refs.ts` is the
SSOT assertion: staging is `ghcr.io/copilotkit/<repo>:latest` (mutable),
**prod is `ghcr.io/copilotkit/<repo>@sha256:<digest>` (already immutably
pinned)**. So this race is **staging-only**; prod was never exposed.
- **`Showcase: Verify Deploy` structurally cannot catch it.** It is a
health probe; it asserts no digest or commit provenance anywhere. Its
#6171 per-commit concurrency key is about *which run verifies*, not
*what image is running*. A stale-but-healthy service passes.
- **The racing runs build DISJOINT service sets** (see below) — I pulled
the actual job lists.
Taken on trust: nothing material. The issue description matched the API
on every point I checked.
## Why NOT a concurrency group
`detect-changes` builds a **per-push, path-filtered** matrix, so
concurrent runs build overlapping but **non-identical** service sets:
- `7b28934387` → ag2, agno, built-in-agent, claude-sdk-python,
claude-sdk-typescript, crewai-crews, langgraph-fastapi,
langgraph-python, langroid, llamaindex, mastra, pydantic-ai, shell-docs,
spring-ai, strands (**15**)
- `db75a04837` → crewai-crews, llamaindex, shell, shell-dashboard,
shell-docs, shell-dojo, showcase-harness (**7**)
`cancel-in-progress: true` would have cancelled the `7b28934387` run and
the ~10 services **only it builds would never have shipped** — trading a
stale-image bug for a never-shipped bug. Per [GitHub's
docs](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#concurrency),
`cancel-in-progress: false` is no better: *"Any previously pending job
or workflow in the concurrency group will be canceled"* — with three
rapid merges the **middle** commit's build is dropped outright. GitHub
also does not guarantee FIFO ordering of queued runs.
Concurrent runs here are **not redundant**, so they must not be
cancelled.
**Re the #6171 interaction:** this design does **not** use
`cancel-in-progress`, so no build is ever superseded-and-cancelled and
the cancelled-slot notifier is never tripped by this change. That
interaction stays theoretical — deliberately.
## The design
Make the one shared mutable resource monotonic instead of serializing
the fleet.
1. The build step pushes **only** the immutable `:<sha>` tag, plus an
`org.opencontainers.image.revision` label.
2. A guard resolves the commit behind the current `:latest`, asks GitHub
`compare/<theirs>...<ours>`, and advances `:latest` (registry-side
retag, no pull) **unless ours is `behind`** — i.e. `:latest` already
holds a descendant and moving it would roll staging back.
**Fails open by design.** No `:latest` yet, unlabelled legacy image,
unreachable API, diverged history → advance. A stuck `:latest` is the
very failure being fixed, so it declines only on *positive proof* of
regression.
**Placement:** in `redeploy-staging` / `redeploy-staging-starters`,
immediately before the Railway pull that consumes `:latest` — not per
build slot. Those jobs already have Node (build slots do **not**, so
per-slot would mean an unpinned `npx tsx` fetch on ~50 parallel
runners), and deciding right before the pull makes the window as narrow
as possible. The image list is the **same matrix ∩ build-success
intersection** that decides what gets redeployed, so a failed build can
never move a tag.
Also adds `showcase_build.yml` to `showcase_validate.yml`'s trigger
paths — the new test asserts against that file's live text, and without
the path a PR re-adding `:latest` would never run the test that catches
it.
## Tradeoffs / what stays open
- **Residual sub-second TOCTOU.** GHCR has no compare-and-swap on tags,
so two runs reading `:latest` simultaneously could still both advance.
This narrows the window from the whole build (~10 min) to inspect→retag.
Fully closing it means retiring the mutable staging tag and pinning
staging to digests the way prod already is — a change to the Railway
image-ref SSOT contract, not a workflow change. **Recommended
follow-up.**
- **The guard is inert for one build per image.** Today's `:latest`
images carry no labels (verified: `showcase-shell-dashboard:latest` has
no `Labels` at all), so the first post-merge build fails open and
advances unconditionally — same as today. Protection starts from the
second build of each image.
- Failure to retag exits non-zero, redding the redeploy job and stopping
the deploy. That is intended: redeploying against a tag that did not
move is exactly the silent false green being fixed.
## Proof
**Red/green on the live YAML.** `advance-latest-tag.test.ts` parses the
real `showcase_build.yml` (extending #6171's `redeploy-guard.test.ts`
pattern). Reverting the workflow to its pre-fix state: **14 failed / 23
passed**. With the fix: **37 passed**. Full `showcase/scripts` suite:
**2382 passed, 73 files**.
**The load-bearing predicate, verified live against the real incident
commits:**
```
compare/db75a04837...59f275eedc => behind (older run arriving late → DECLINE)
compare/59f275eedc...db75a04837 => ahead (newer run → advance)
compare/db75a04837...db75a04837 => identical
```
**Label reading, verified against a real multi-platform registry image**
— `docker buildx imagetools inspect ghcr.io/astral-sh/uv:latest` piped
through `extractRevisionLabel()` returns
`3010295ae7ff572de459987ad70db315a62ecd61`, matching `jq` exactly. The
platform-keyed shape is handled.
**Shell/jq transforms** exercised directly, including the empty-CSV edge
case (empty → empty, step skipped by its `if:`).
**Lint:** `actionlint` finding counts byte-identical to the pre-change
baseline (no new findings; the 11 pre-existing are unrelated). `zizmor
--min-severity low` with the repo config: **no findings**.
**Typecheck:** both new files are in `showcase/scripts/tsconfig.json`'s
include set and produce **zero** errors. Worth stating plainly: `nx
run-many -t check-types` **does not reach `showcase/scripts`** — the
project isn't in the nx graph and has no `check-types` target (there are
9 pre-existing type errors in sibling files, which is how I confirmed
it). So the typecheck above is mine, not CI's. The *tests* are gated:
`showcase_validate.yml` runs bare `pnpm exec vitest run` in
`showcase/scripts`, which auto-discovers the new file.
### What I could NOT prove
**I did not construct a real concurrent race on scratch branches.**
Doing it faithfully needs two builds pushing the same GHCR repo with
controlled finish ordering, which means merging to `main` — the only
branch the build workflow triggers on. No run IDs for a live race
demonstration; I am not implying one.
Unproven until this runs on main: that `docker buildx imagetools create`
retags cleanly under the runner's GHCR credentials, and that `npx tsx`
behaves in the redeploy jobs (it is already the established invocation
there — `redeploy-env.ts` — so this is low risk, not zero).
## Normal single-merge builds are unaffected
No concurrency group is added, so nothing queues or cancels. A lone
merge finds `:latest` at its own parent → `ahead` → advances, exactly as
before. Cost is one `imagetools inspect` + one `gh api` + one
registry-side retag per built service, in a job that already exists — no
extra job, no extra checkout, no change to build parallelism.
---
Branched from `db75a04837`; #6156/#6159 landed after, so this will need
main merged in before it goes green.
Probable conflict with the concurrent `git lfs pull` work in
`showcase_validate.yml` — my edit there is only the top-level `on:
paths:` list, so it should merge cleanly, but flagging it.
Six defects reported on staging. All six reproduce; five share one
cause.
## The shared cause
`built-in-agent/src/app/api/copilotkit/route.ts` maps **~20 demos to the
same prompt-less `createBuiltInAgent()`**. The reference wires each to
its own graph **and its own system prompt** (28 graphs in
`langgraph-python/langgraph.json`).
aimock replays a scripted tool-call sequence keyed on `userMessage` +
`context`, so D6 is green whether or not the model could have reasoned
its way there. This is `showcase/GOTCHAS.md` #8 verbatim — including its
stated "structural tell": *grep the integration's `route.ts` for demos
left in the generic fallthrough that LGP wires to a dedicated graph.*
## Per-defect
| Demo | Symptom | Root cause |
|---|---|---|
| `gen-ui-tool-based` | charts plot zeros | No prompt. The assistant
says so itself: *"I used placeholder values since no sales figures were
provided."* Nondeterministic — one run gave a real 0..220 axis, the next
`0 1 2 3 4` with no bars, which is how it passed review. |
| `gen-ui-agent` | wall of text, steps frozen at "step 1 of 4" |
`set_steps` + its `/steps` STATE_DELTA already worked; nothing told the
model to *walk* pending→in_progress→completed. |
| `subagents` | left panel always empty | `grep -rn delegations src/ \|
grep -v demos/subagents` → **no output**. Frontend reads
`agent.state.delegations`; the converter handles `/steps` and `/notes`
and has no `delegations` case. Chat works because the tools do run. |
| `declarative-json-render` | raw JSON for pills 2 & 3 | Intercepted the
SSE: **the wire is already one closing brace short** — nothing is
dropped in transport, and the client parser correctly requires a
balanced object. Pill 1 only works because the model copies the prompt's
worked example verbatim (`\$1.24M`, `+18% vs Q2`). Model-level JSON
enforcement had been removed with the note *"the system prompt already
enforces JSON-only output"* — it does not. |
| `a2ui-recovery` | 5 identical cards | Reuses declarative-gen-ui's
prompt with no "call once" constraint, while the pills literally ask it
to *"self-correct a malformed first attempt"*. The retry loop is
**inside** the tool and returns on first valid pass, so every supervisor
retry is pure duplication. |
| `declarative-gen-ui` | D4 though it works live | **The inverse case.**
Badges: `UI ✓ BE ✓ 1P ✗ D6 —` (*"gated — blocked by a lower rung"*). Not
a live bug — a fixture bug. See below. |
## The D4 cap (`declarative-gen-ui`, `a2ui-recovery`)
Both demos' secondary design-call fixtures gated on
`match.responseFormat: \"json_object\"`. That matcher **can never match
this backend**:
- The agent talks to the OpenAI *Responses* API, where JSON mode is
`text.format`.
- aimock has **no `text.format` handling at all** —
`responsesToCompletionRequest` forwards only a top-level
`response_format` (absent in 1.19.1, forwarded in 1.37.4), a key the
Responses API doesn't accept and this client doesn't send.
- So `effective.response_format?.type` is always `undefined` and
`router.ts` skips the fixture → the design call never matched → surface
never painted → D5 red, D6 never ran.
Re-keyed onto `match.toolName`, which aimock *does* normalize out of a
Responses request (`responsesToolsToCompletionsTools`): the outer call
declares `generate_a2ui`, the in-tool design call declares nothing.
Tool-less secondary fixtures moved last, because several pills' brief is
a substring of the pill text (`\"Build my Q2 revenue summary …\"`
contains `\"Q2 revenue summary\"`) and would otherwise win the outer
request.
## Verification
**Reproduced live** (staging, real LLM): the zero-axis chart plus its
self-incriminating message, the unbalanced JSON on the wire, and the
reference rendering the *same pill* correctly for contrast.
**Two mutation-verified test suites** — both fail with the fix reverted:
- `aimock-a2ui-routing.test.ts` drives aimock's **real `matchFixture`**.
On the old fixtures 14/16 cases fail with `no fixture matched the
secondary design call for brief \"…\"` — the D5 red, reproduced as a
unit test.
- `tanstack-factory.test.ts` covers the `/delegations` and `/steps`
deltas (3 delegation tests fail with the branch disabled; the 2 controls
still pass).
**No regressions:** `tsc --noEmit` is 61 errors before *and* after (all
pre-existing — `gpt-5.4` missing from the adapter's model union, zod
v3/v4 skew); oxlint clean on changed files; the repo-wide fixture
validator passes (858 assertions).
`text.format` was verified by reading the request mapping in the
**pinned** `@tanstack/ai-openai@0.15.6` → `@tanstack/openai-base@0.9.2`:
`modelOptions` is spread straight into `responses.create()`,
`validateTextProviderOptions` only inspects
`metadata`/`conversation`/`previous_response_id`, and the adapter sets
`text.format` itself only when an `outputSchema` is passed (none here,
so nothing is clobbered).
## What is NOT verified
**The four prompt/`text.format` changes have not had a real-LLM
click-through** — that needs an OpenAI key this environment doesn't
have, and D6/aimock cannot verify them by construction (that's the whole
point of gotcha #8). Reasoning is documented inline at each site. This
area has burned the repo before: a previous `response_format` attempt
made the call return an empty string "verified against real OpenAI",
which is why `text.format` — the Responses API's own param — is used
instead. **Worth a live click-through on the staging deploy before this
is considered closed.**
## Deliberate non-goals
- **`a2ui-recovery` only demonstrates recovery under aimock.** The
heal/exhaust branches need a designer LLM that emits invalid surfaces on
demand; a real one succeeds on attempt 1. The single-call constraint
fixes the 5-card bug and makes it honest; making recovery visible live
needs deliberate fault injection — a product decision, recorded in
`PARITY_NOTES.md`.
- **The reference pins `openai:gpt-4o-mini`** (`gen_ui_agent.py:92`).
Flagged, not touched.
## Docs
`GOTCHAS.md` gains built-in-agent as a second worked instance of #8
(including that the masking runs *both* ways, and that `1P ✗ D6 —` means
gated, not failing), plus the `text.format` and dead-`responseFormat`
traps. `PARITY_NOTES.md` records that the named-agent registry is
**not** prompt-neutral, and that any new `state.<slot>` needs a
converter branch.
Every affected cell was D6-green (or D4-gated) while the deployed demo was
visibly broken. The common cause is GOTCHAS #8: aimock replays a scripted
tool-call sequence keyed on userMessage + context, so the fixture answers a
question the model was never asked.
Prompt gaps — ~20 demos shared one prompt-less `createBuiltInAgent()` where the
reference wires each to its own graph AND its own system prompt. Adds
`createBuiltInAgent({ systemPrompt })` + `demo-prompts.ts`, ported from the
reference graphs:
- gen-ui-tool-based plotted zeros; the assistant said "I used placeholder values
since no sales figures were provided". Nondeterministic — some runs invented
real values, which is how it passed review.
- gen-ui-agent published its plan once then narrated, freezing the progress card
on step 1 of 4.
- a2ui-recovery painted five identical cards: nothing constrained the supervisor
to one `generate_a2ui` call, and the pills literally ask it to "self-correct".
The retry loop lives inside the tool, so supervisor retries are duplication.
subagents' delegation panel was permanently empty — the frontend reads
`agent.state.delegations` and no code emitted that slot. The converter now emits
a `/delegations` delta per sub-agent result (whole-array `add`, since initial
state is `{}` and strict fast-json-patch rejects unresolvable paths while
@ag-ui/client swallows the throw), and ports the reference's
`_MAX_CRITIQUE_ITERATIONS = 1` cap.
declarative-json-render dumped raw JSON for any prompt the model couldn't crib
from the worked example. Captured the SSE: the wire is already one closing brace
short, so nothing is dropped in transport — model-level enforcement had been
removed on the grounds that "the system prompt already enforces JSON-only
output". It does not. Restores it via the Responses API's `text.format`, the
param the removed `response_format` maps to, verified against the pinned
@tanstack/ai-openai@0.15.6 -> @tanstack/openai-base@0.9.2 request mapping.
declarative-gen-ui and a2ui-recovery were the inverse: capped at D4 (UI ✓ BE ✓
1P ✗ D6 gated) while working live. Their secondary design-call fixtures gated on
`match.responseFormat`, which aimock can never satisfy here — it has no
`text.format` handling and only forwards a top-level `response_format` the
Responses API doesn't accept. Re-keyed onto `match.toolName` (the outer call
declares `generate_a2ui`, the in-tool design call declares nothing) with the
tool-less fixtures moved last, since several pills' brief is a substring of the
pill text.
Tests (both mutation-verified — they fail with the fix reverted):
- aimock-a2ui-routing.test.ts drives aimock's real `matchFixture`; 14/16 cases
fail on the old fixtures with "no fixture matched the secondary design call".
- tanstack-factory.test.ts covers the `/delegations` and `/steps` deltas.
Typecheck unchanged at 61 pre-existing errors; oxlint clean on changed files.
NOT verified live: the four prompt/`text.format` changes need a real-LLM
click-through, which needs a key this environment doesn't have. Reasoning and
the exact request mapping are documented inline.
PR #6130 added `src/middleware.ts`, which sets the `x-pathname` header that
`src/app/demos/layout.tsx`'s `generateMetadata()` reads. That header is what
makes the layout actually call `loadDemoIndex()` — a request-time
`readFileSync(process.cwd()/manifest.yaml)`. The Dockerfile's runner stage
never copied `manifest.yaml`, so every `/demos/*` route now 500s with "An
error occurred in the Server Components render" (ENOENT), while the
statically-prerendered home page keeps returning 200.
On the dashboard that reads as "service is up, every cell pinned at D3": D4
fails on `page.type` waiting for `textarea`, D5 times out, D6 fails on
`waitForSelector('[role="textbox"]')` — the chat input never mounts because
the page is an error boundary.
langgraph-python hit this exact bug and fixed it with the same one-line COPY;
ms-agent-dotnet is the only integration shipping `middleware.ts` without it.
Adds a ratchet test over that invariant (mutation-verified: fails with the
COPY removed).
The auto-format bot reflowed both captured registry payloads. Harmless this
time — oxfmt only rewraps arrays, it does not reorder keys — but the whole
value of these two files is that they are verbatim `imagetools inspect`
output. A formatter in the loop means the next re-capture churns, and the
"this is exactly what the registry emits" claim stops being checkable.
Exempt the fixture directory in .oxfmtrc.json, restore the captured bytes,
and say so in the provenance comment so nobody tidies them by hand.
The previous suite was 37/37 green against a workflow that could not have
run at all: the guard step had no `packages: write` and no GHCR login, so
every `imagetools create` would have 401'd. Asserting that a step EXISTS
proves nothing about whether it can succeed. Each change below was checked
by reintroducing the defect and confirming the test goes red.
Registry-auth precondition (the miss that let the above ship)
- Both redeploy jobs must declare `packages: write` AND a `docker/login-action`
step for ghcr.io, ordered BEFORE the guard. Dropping either reds the suite.
The intersection, executed rather than restated
- The old test only checked that the string `images=` appears in the compute
step; swapping its jq for the full matrix — so a FAILED build moves
`:latest` — kept it green. The real `changed` shell now RUNS, against the
real ALL_SERVICES matrix read out of the workflow, and the emitted
$GITHUB_OUTPUT is asserted: a failed build is in neither set, and the
`skip_build` slot is in `services` but NOT in `images` (handing it to the
guard fails "manifest unknown" and blocks the redeploy for the whole fleet).
Same treatment for the starter lane.
Fixtures joined to real registry output
- `extractRevisionLabel` was pinned to hand-written payloads that were never
compared with reality. Since `readLatestRevision` maps every failure to
null, and null ADVANCES, a parser that silently never matches yields a
permanently-blind guard with a fully green suite. Both fixtures are now
verbatim `docker buildx imagetools inspect --format '{{json .Image}}'`
output (buildx v0.35.0), unformatted so key order survives: a platform-keyed
multi-arch image carrying the label, and our own `:latest`, which turns out
to be a bare config object with NO labels at all. The label key used in the
injection test is read from the workflow's own `labels:` input, so parser
and producer cannot drift apart silently.
Closed vacuities
- The `:latest` ban read one step's `tags` and was vacuously green on an empty
list. It now covers every tagged step plus hand-rolled `docker push`/`docker
tag`/`imagetools create`, and fails on an empty `tags`.
- The `cancel-in-progress` ban read only top-level config; a job-level
`concurrency` on `build` reproduced the exact harm and stayed green. All
jobs are checked now.
- `already-current` is keyed on the digest, matching the script.
- The unreachable `(null, "ahead")` row is labelled as the defensive
input-space case it is, and a new test pins that the real flow never calls
compare() with an unknown revision.
New coverage: classifyProbeFailure (incl. a REAL execFileSync timeout, and
the two precedence traps — gh's rate-limited 403 is throttling, not auth; a
registry's 404-with-denied is auth, not absent), readFlag, escapeAnnotationData
(a forged `\n::error::` stays inert), isDirectInvocation through a symlink,
the `::error` annotation on fleet failure, and the digest-mismatch advance.
Also: GITHUB_SHA / GITHUB_REPOSITORY are no longer shadowed in the workflow.
The runner exports both and the script reads process.env, so the old
assertions pinned a redundancy rather than a capability — removed together,
as the comment there required. Workflow-reading scaffolding duplicated with
redeploy-guard.test.ts is extracted and the YAML parse memoized.
`job.status` for a matrix slot is success|failure|cancelled, but the
per-slot writer laundered cancelled into `skipped` before publishing its
result, so a slot killed by `timeout-minutes` became indistinguishable
from one that legitimately never built.
That erased the only signal that could tell a partially-cancelled fleet
build from a clean one. GitHub's status functions cannot recover it:
`cancelled()` is documented as "returns true if the workflow was
canceled" (workflow-scoped, and FALSE for a leg-only cancel), and a
cancelled ancestor is not a FAILED ancestor so `failure()` is false too.
Add `cancelled` to the BuildOutcome contract, add `cancelledSet()`, and
have the aggregator publish `any_cancelled` + `cancelled_services`
alongside `any_success`. `successSet` still excludes cancelled slots, so
the redeploy intersection is unchanged — a slot that pushed no image
still cannot enter the redeploy CSV.
`generate-catalog.test.ts` pinned crewai-crews' wired-cell count at >= 30. Moving
`multimodal` from `features` to `not_supported_features` reclassifies that cell
from `wired` to `unsupported`, so the real count is 29 and CI's
`showcase/scripts` vitest failed.
The bound is a snapshot floor, not an invariant: the test's own comment says the
partial parity tier only requires intersection >= 3 with the reference's wired
set. 29 still clears that by a wide margin, so the floor tracks the manifest.
Updated to 29 with a note recording which cell moved and why, so the next reader
does not read the decrement as a regression.
copilotkit-qa-bot[bot]