PR #6130 added `src/middleware.ts`, which sets the `x-pathname` header that
`src/app/demos/layout.tsx`'s `generateMetadata()` reads. That header is what
makes the layout actually call `loadDemoIndex()` — a request-time
`readFileSync(process.cwd()/manifest.yaml)`. The Dockerfile's runner stage
never copied `manifest.yaml`, so every `/demos/*` route now 500s with "An
error occurred in the Server Components render" (ENOENT), while the
statically-prerendered home page keeps returning 200.
On the dashboard that reads as "service is up, every cell pinned at D3": D4
fails on `page.type` waiting for `textarea`, D5 times out, D6 fails on
`waitForSelector('[role="textbox"]')` — the chat input never mounts because
the page is an error boundary.
langgraph-python hit this exact bug and fixed it with the same one-line COPY;
ms-agent-dotnet is the only integration shipping `middleware.ts` without it.
Adds a ratchet test over that invariant (mutation-verified: fails with the
COPY removed).
Two gaps from the previous commit, found reviewing it.
The five platform pages previously carried a `your-intelligence-url`
placeholder, which was neutral about where Intelligence runs. Replacing it with
the managed hosts read as "this is the endpoint" to a self-hosted reader, and
only the quickstart said otherwise; every env block now notes that self-hosted
deployments substitute their own two hosts.
index.mdx and mcp.mdx are code-only pages with no env block, so after requiring
a second variable they referenced COPILOTKIT_INTELLIGENCE_WS_URL without ever
telling the reader where it comes from. Both now point at the dashboard and the
quickstart's env block.
Refs OSS-621
The langgraph-typescript agent runs a custom server.mjs whose hardcoded
graphSpec — NOT langgraph.json — is the runtime graph registry. gen_ui_tool_based
was missing from it, so the langgraph server returned 404 for that graph and the
Tool-Based Generative UI cell fell through. Add gen_ui_tool_based and drop the
three redundant graph ids (reasoning-default-render, tool-rendering-{default,
custom}-catchall) that route.ts now consolidates onto shared graphs, keeping
server.mjs in sync with langgraph.json.
Bring the langgraph-typescript Showcase integration to parity with the
reference langgraph-python: byte-identical demo frontends, the same demo set
and manifest overview, framework-native TypeScript backend wiring, and
canonical mirrored aimock D6 fixtures. Scope is limited to
integrations/langgraph-typescript and its aimock/d6 fixtures.
Frontend: realign ~25 drifted demo files to the north star; add missing
src/components/ui primitives, READMEs, declarative-gen-ui/sales-context, and
the threadid-frontend-tool-roundtrip page; remove LGT-only drift; replace the
hand-rolled landing page with the manifest-driven one. Only the sanctioned
"LangGraph (TypeScript)" identity strings differ.
Manifest: rebuild to match langgraph-python's feature order, demo order,
names, tags, descriptions, and routes; drop two bogus NSF entries that were
never real features (reasoning-default-render, agentic-chat-reasoning); restore
shared-state-streaming and tool-rendering-reasoning-chain as supported cells.
not_supported_features now matches the north star (gen-ui-interrupt /
interrupt-headless — a shared upstream react-core resume-path limitation).
Backend: add the dedicated gen-ui-tool-based data-viz graph; consolidate the
tool-rendering variants and reasoning cells onto their shared graphs (matching
langgraph-python) and drop redundant duplicate graph ids; register
threadid-frontend-tool-roundtrip; move the MCP Apps runtime to the
[[...slug]] catch-all route.
Fixtures: re-mirror all 43 aimock/d6 fixtures from the canonical langgraph-python
set with match.context re-keyed, replacing previously drifted per-integration
fixtures.
## What
Brings the **ms-agent-dotnet** (Microsoft Agent Framework .NET) showcase
integration from D5 to **D6**, using **langgraph-python** as the
north-star reference.
### 1. Frontend parity with langgraph-python
Restores near-identical frontends where ms-agent-dotnet had drifted,
while **preserving the load-bearing .NET adaptations** (per the showcase
iron rules — differences belong in fixtures/minimal backend, not the
shared frontend):
- Root shell: `globals.css` (Tailwind `@theme` block + brand green),
manifest-driven index `page.tsx`, `layout.tsx`, new `middleware.ts`
(`x-pathname`), `tsconfig` include.
- `declarative-gen-ui` subtree restored (fixes divergent pill testids
the shared probe asserts).
- Doc-snippet `@region` markers, import-style normalization, `subagents`
revert, stale-file cleanup, `auth` inspector flag.
- **Kept** (load-bearing, not reverted): `parse-json-result` 3-layer
unwrap, multimodal legacy-shim, tool-based `hitl` (MAF has no
`interrupt()`), `agent-config` `properties=`.
### 2. shared-state-streaming → per-token (removed from
`not_supported_features`)
`write_document`'s `document` arg now streams into `state.document`
per-token via a `createSharedStateStreamingAgent` route shim (mirrors
the proven `createGenUiAgent` bridge, with a partial-JSON string
decoder), since the .NET AG-UI host has no `predict_state_config`.
### 3. a2ui-recovery cell (new)
First MS-Agent-Framework implementation of the A2UI
validate→retry→`a2ui_recovery_exhausted` recovery loop. Because the MAF
AG-UI adapter can't emit the custom `ACTIVITY_SNAPSHOT{status:"failed"}`
the exhausted card needs, it's a **raw-SSE `MapPost` endpoint**
(`RecoveryAgent.cs`) — the same adapter-bypass pattern already shipped
for `/multimodal`. Adds the demo frontend, API route, deterministic
aimock fixture (heal seq0-invalid→seq1-valid; exhaust always-invalid),
and a unique per-slug `PROMPTS` entry in the shared probe.
### 4. threadid-frontend-tool-roundtrip demo (parity)
Added for demo-set parity (reuses the `frontend_tools` passthrough; not
a D6-scored feature, mirroring the reference).
`gen-ui-interrupt` / `interrupt-headless` remain honestly quarantined
(upstream `@copilotkit/react-core` `useInterrupt` resume-path bug — not
a backend gap).
## Verification
- Code was authored in parallel worktree-isolated slots, each
cross-verified against the reference + the shared probe contracts; the
a2ui-recovery fixture was cross-checked against
`RecoveryAgent.ValidateComponents`.
- Local D6 harness: the image builds and the stack + probes run, but
**full local green was blocked by Windows-only harness friction**
(`core.symlinks=false` breaks `stage_shared`'s `[ -L ]` materialization;
`--direct` doesn't context-scope the `x-aimock-context` header so
context-keyed a2ui fixtures miss). These are environmental, not code
issues. **Relying on CI's Linux harness (real symlinks + fleet worker)
for authoritative D6.**
## Follow-up (not in this PR)
- `stage_shared()` should also materialize Windows symlink-as-file
entries (detect a regular file whose content is a relative path), so
forced local rebuilds work on `core.symlinks=false` checkouts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Aligns the **langgraph-fastapi** showcase integration to the north-star
**langgraph-python** (OSS-582 — "Align LangGraph (FastAPI) Showcase
demos and code"). Moves the integration onto the product-centric demo
set: backend agent graphs, runtime wiring, frontend chrome, the
demo-browser overview (`manifest.yaml`), and aimock fixtures.
Linear: OSS-582.
## What changed
**Backend agents (`src/agents/src/`)**
- Ported the missing dedicated graphs: `agentic_chat`, `gen_ui_agent`,
`gen_ui_tool_based`, `shared_state_streaming`.
- v2 `create_agent` ports + config alignment for `agent_config_agent`,
`headless_complete`, `reasoning_agent`, `tool_rendering_agent`,
`tool_rendering_reasoning_chain_agent`, `a2ui_dynamic`.
**Runtime wiring**
- `route.ts`: wired the canonical demos to their dedicated graphs and
removed them from the generic `sample_agent` fallthrough loop (this is
what made several demos behave correctly against a real LLM — see
below); added `recursion_limit`.
- `langgraph.json`: registered the 3 new graphs.
- Renamed the dedicated API routes
`copilotkit-byoc-{hashbrown,json-render}` →
`copilotkit-declarative-{hashbrown,json-render}`.
**Overview / content**
- `manifest.yaml`: demos + features aligned to LGP (names, descriptions,
tags, order, canonical ids; deprecated aliases migrated; phantom
`hitl-in-chat-booking` removed; `shared-state-streaming` un-quarantined
now that it works). Integration identity (name/slug/logo) preserved. The
demo-browser overview is now card-for-card identical to
langgraph-python.
- Restored two missing `@region` markers (factory-automation snippet
extraction).
**Frontend chrome**
- `globals.css`, `layout.tsx`, `middleware.ts` (was missing),
`declarative-gen-ui/*` + `sales-context.ts`, `beautiful-chat`;
`Dockerfile` now copies `manifest.yaml` (fixed an RSC crash on every
`/demos/*`).
**aimock fixtures (`aimock/d4|d6/langgraph-fastapi/`)**
- D6 fixture fixes for the previously-red cells (stale `turnIndex`
gates, cross-file substring shadowing, missing `chunkSize`, prompt
narrowing).
**Showcase tooling / docs**
- Fixed harness services racing on a shared image tag
(`docker-compose.local.yml`).
- `GOTCHAS.md` #8: documents that aimock D6 can be green while a demo is
broken against a real LLM (fixtures replay scripted tool calls
regardless of which graph ran), plus how to catch it.
- `PARITY_NOTES.md`: sanctioned divergences (a2ui-recovery per-slug
prompt isolation; declarative-json-render scoped-test divergence).
## Verification
- **D6 sweep: 38 green / 1 red.** The single red is **a2ui-recovery**,
which reproduces **identically on the north star** (shared Python
`ag_ui_langgraph` recovery loop; mastra's TS impl is green). It is not a
fastapi defect and is tracked as a separate PR against the north star.
Documented in `PARITY_NOTES.md`.
- `validate-manifests` (manifest → registry, all 20 integrations) —
green.
- `validate-routes --all` (runtime-route wiring) — green.
- Build + TypeScript typecheck (`next build` via the Docker image build)
— green.
- `@region` audit — all region pairs balanced and LGP-consistent (0
orphans/typos/mismatches).
- Live manual QA against real OpenAI (`:3102` vs `:3100`) confirmed the
key demos (reasoning-custom, shared-state-streaming, gen-ui-tool-based,
agentic-chat).
## Notable finding
Several demos were D6-green but broke against a real model because they
fell back to the generic `sample_agent` instead of their dedicated graph
— the aimock fixture masked the wiring bug by replaying scripted tool
calls. This PR fixes the wiring and documents the gap (GOTCHAS #8).
D6-green is necessary but not sufficient for graph/tool-dependent demos;
a real-LLM click-through is required.
## Deferred / follow-ups
- **a2ui-recovery** — shared north-star defect in the Python
`ag_ui_langgraph` recovery loop; separate PR against the north star.
- **d5-byoc probe** — always sends the hashbrown pill even on the
json-render page (fleet-wide harness limitation); tracked as a
follow-up. The gating `byoc` grid cell is green.
## Scope note
Only `langgraph-fastapi` integration code, its aimock fixtures, and
showcase tooling/docs changed. **langgraph-python (the north star) was
not touched.**
The published Channels docs configured the Intelligence client with
`wsUrl: apiUrl.replace(/^http/, "ws")` in all eight pages. Because the API and
realtime planes are separate hosts, that yields a URL serving no socket, and
the failure is a silent 30s hang rather than an error — so this was the most
harmful copy of the bug: it is the surface developers actually read.
Every page now reads COPILOTKIT_INTELLIGENCE_WS_URL from the environment
alongside the API URL, and the six env blocks that previously showed only a
`your-intelligence-url` placeholder document both hosts with the values that
work against the managed service. The quickstart gains a short paragraph on why
the second host is required and what going wrong looks like.
Refs OSS-621
Independent re-verification against packages/react-native source found the
page overstated polyfill auto-install: only /headless (and the root barrel,
via its /headless re-export) side-effect-imports the polyfill barrel.
src/components/index.ts imports no polyfills, so a consumer importing only
from @copilotkit/react-native/components does not get them auto-installed;
it relies on the provider imported alongside it. Corrects the 'all three
surfaces auto-install' and 'every entry point installs these automatically'
statements accordingly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI 'Showcase: Validate' failed: aimock-fixtures.test.ts asserts every d6/*.json
yields >=1 fixture, but gen-ui-open.json had been emptied to a _meta stub (0
fixtures) by an earlier open-gen-ui shadowing fix that consolidated the pills
into gen-ui-tool-based.json.
Restore gen-ui-open.json as a byte-for-byte context-swap of the canonical
langgraph-python file (context -> langgraph-fastapi): 4 fixtures for the '3D
axis' and 'Inline expression evaluator' pills, LGP's clean shape
(hasToolResult:false leg-1, distinct toolCallId terminals). No shadowing loop:
each file's leg-1/terminal pair is anchored by its own distinct toolCallId, so
the done-signal always resolves.
Verified: aimock-fixtures.test.ts 839/839 green locally; D6 gen-ui-open,
gen-ui-open-advanced, gen-ui-custom all green (two real runs each).
Same class as the hitl-in-app fix: turnIndex:0 emit legs get re-served on the
post-tool-result turn under aimock's default (relaxed) turn-index selection,
re-firing the frontend tool instead of narrating.
frontend-tools-async (query_notes): add hasToolResult:false to the 4 emit legs
(3 pills + the 'project planning' catch-all) so they match the emit turn and are
excluded once the tool result returns. The toolCallId narration legs round-trip
fine here (verified on the wire) and are unchanged.
Verified under relaxed aimock: pill -> single result card + narration, no
re-emit, no error (was: double card + 'toLowerCase' error).
subagents.json left as-is: its turnIndex 1/2 legs are intentional server-side
step sequencing, not the client-re-POST pattern.
open-gen-ui / open-gen-ui-advanced NOT touched here: their real fixtures live in
gen-ui-open.json / gen-ui-open-advanced.json (already hasToolResult-gated, no
turnIndex). The stale duplicate generateSandboxedUi entries in
gen-ui-tool-based.json are a separate cleanup, tracked outside this PR.
The factory automation extracts curated snippets via @region markers.
readonly_state_agent_context.py and shared_state_read_write.py were byte-aligned
to langgraph-python except their @region markers had been stripped, so the
factory would extract nothing for those two regions. Restore them to match LGP:
- agent-context-setup around create_agent in readonly_state_agent_context.py
- shared-state-setup around create_agent in shared_state_read_write.py
Comment-only; no behavior change. Audit confirms all 49 fastapi region pairs are
now balanced and LGP-consistent (0 orphans/typos/mismatches).
fe-parity.ts was a scratch parity checker for this alignment work; it is no
longer needed in the worktree. Drop it and de-reference it from
langgraph-fastapi/PARITY_NOTES.md (reword the intro, convert the
machine-readable fe-parity-allow allowlist into a plain human-readable
'sanctioned file-level divergences' list). No other references existed. D6
behavior remains the parity judge.
The byoc D6 featureType covers both declarative demos; the grid cell is green
(hashbrown route+pill). A scoped declarative-json-render --d6 run reds for two
non-fastapi reasons: (1) the shared d5-byoc probe always sends the hashbrown
pill even on the json-render page (harness limitation — build context lacks
demos[]; tracked as a separate follow-up), and (2) fastapi deliberately keeps
hashbrown (@hashbrownai/react) and json-render (@json-render/react) as separate
library integrations with their own pills/fixtures, vs LGP's unified
render_dashboard contract. Sanctioned; do not collapse fastapi's json-render to
the unified contract (content downgrade) — the real fix is probe-side.
aimock replays scripted tool calls keyed on userMessage+context regardless of
which graph ran or its tool set, so a demo mis-wired to the sample_agent
fallback passes D6 green while breaking against a real model (OSS-582:
gen-ui-tool-based looped on query_data; shared-state-streaming wrote only to
chat; agentic-chat ran with the wrong tool set). Document that D6-green is
necessary but not sufficient, the real-LLM live click-through procedure, and
the structural tell (route.ts fallthrough loop vs LGP's dedicated wiring).
Live QA surfaced demos that behaved wrong on fastapi because they fell back to
the generic sample_agent (or were unregistered) instead of the dedicated graph
LGP uses. aimock D6 masked these (fixtures script the tool calls), so they were
green in the grid but broken against a real model. Audit of fastapi's
agentNames fallthrough vs LGP's neutralAssistantCells found exactly these:
- gen-ui-tool-based: port gen_ui_tool_based graph (tools=[], frontend supplies
render_bar/pie_chart via useComponent). Was sample_agent, whose query_data
tool + prompt made the model loop on data queries instead of rendering.
- shared-state-streaming: port shared_state_streaming graph (StateStreaming
middleware + write_document tool + document state). Was sample_agent, which
never emits state.document, so it only wrote to chat. Remove from
not_supported_features (now works, D6 green).
- agentic-chat: port agentic_chat graph (tools=[]). Was sample_agent (7+ tools).
- threadid-frontend-tool-roundtrip: wire to frontend_tools (was unregistered).
- reasoning-custom: align reasoning_agent config to LGP (gpt-5.4 / effort
medium / summary detailed; were gpt-5-mini / low / auto).
Register the 3 new graphs in langgraph.json; remove the 3 names from the
sample_agent fallthrough loop in route.ts. D6 green x2 for reasoning-display,
gen-ui-custom, shared-state-streaming, agentic-chat; agent-config +
tool-rendering-reasoning-chain re-verified (no regression).
Soften the stale 'D6 green'/'under investigation' wording. A live trace
confirmed the a2ui-recovery surface-missing D6 red reproduces identically on
the langgraph-python north star (shared Python ag_ui_langgraph recovery loop;
mastra's TS impl is green), so fastapi is already behaviorally aligned and
there is nothing to fix under integrations/langgraph-fastapi/. Fixing the
shared loop is a separate PR against the north star / the package.
The demo-browser overview (page.tsx, identical to LGP) groups cards by
demo.tags[0] and orders within a group by manifest.features[] index, so the
card arrangement is fully driven by manifest.yaml. fastapi's manifest was an
older curation: a different naming scheme (~26 demos), deprecated feature/demo
ids (agentic-chat-reasoning, reasoning-default-render, byoc-hashbrown,
byoc-json-render, hitl, phantom hitl-in-chat-booking), divergent tags/order,
and a missing shared-state-streaming/shared-state-read card.
Align manifest.yaml demos+features to LGP (names, descriptions, tags, order,
canonical ids), preserving fastapi identity (name/slug/logo/description) and
the sanctioned a2ui-recovery prompt divergence. Overview is now card-for-card
identical to langgraph-python.
Canonicalizing the ids pulled two cells into the D6 tested set that were
mis-wired to the old names; wire them to LGP's shape:
- reasoning-custom/reasoning-default -> reasoning_agent in copilotkit/route.ts
(were wired under agentic-chat-reasoning/reasoning-default-render).
- rename api routes copilotkit-byoc-{hashbrown,json-render} ->
copilotkit-declarative-{hashbrown,json-render}; align endpoints + the
declarative-hashbrown-demo agent id to what the demo pages request.
Full D6 sweep: 37 green / 1 red; reasoning-display and byoc now green (two
real runs each); the lone red is a2ui-recovery (known shared north-star
defect, identical on LGP).
The three turn-0 pill fixtures (refund/downgrade/escalate) were keyed on
turnIndex:0. aimock's default (relaxed) turn-index selection re-serves a
turnIndex-only fixture on the post-approval turn, so approving a
useFrontendTool(request_user_approval) call re-emitted the tool call
instead of narrating the outcome — the turn never settled (stuck
text-unstable).
Re-key each emit fixture on hasToolResult:false instead of turnIndex:0.
It matches the emit turn (no tool result yet) and is excluded on the
post-approval turn (hasToolResult:true), where the toolCallId narration
fixture wins. No turnIndex dependency, so it is correct under both strict
and relaxed aimock, and multi-pill-safe.
Not a bridge/runtime bug: @ag-ui/mastra loops back correctly on real
OpenAI and under strict aimock. Fault was fixture selection only.
Two independent fastapi divergences from the green north-star kept this red:
1. Missing chunkSize:9999 on the 8 tool-call fixtures. Under aimock's global
8-byte chunking the large tool-call args failed to JSON-parse in one piece,
so the AAPL->MSFT reasoning chain stopped at leg 1 (turns 1 & 2). Byte-align
to LGP (and the green mastra/langgraph-typescript siblings carry the same
pattern).
2. Fixture-pool shadowing broke turn 3. aimock pools all d4+d6 fixtures for a
context and matches by userMessage substring, first-match-wins in load order
(d4 before d6). fastapi had broad keys LGP doesn't:
- d4/chat.json: 'weather' -> 'weather in San Francisco', 'flights from SFO
to JFK' -> 'Find flights from SFO to JFK.' (period-terminated so it stops
being a substring of turn 3's 'Find flights from SFO to JFK and show me
the weather there.').
- tool-rendering-{custom,default}-catchall.json: bare 'Find flights' ->
'Find flights from SFO to JFK.' (now matches LGP's exact key).
Also byte-align the backend agent to LGP: scriptable get_stock_price signature,
the detailed chaining system prompt, model gpt-5.4, reasoning summary detailed.
D6 tool-rendering-reasoning-chain green (two real ~21s runs). Shared-fixture
regression all green: agentic-chat, tool-rendering, tool-rendering-custom-catchall,
tool-rendering-default-catchall, headless-complete (Tokyo-weather consumer).
The auto-format bot reflowed both captured registry payloads. Harmless this
time — oxfmt only rewraps arrays, it does not reorder keys — but the whole
value of these two files is that they are verbatim `imagetools inspect`
output. A formatter in the loop means the next re-capture churns, and the
"this is exactly what the registry emits" claim stops being checkable.
Exempt the fixture directory in .oxfmtrc.json, restore the captured bytes,
and say so in the provenance comment so nobody tidies them by hand.
The previous suite was 37/37 green against a workflow that could not have
run at all: the guard step had no `packages: write` and no GHCR login, so
every `imagetools create` would have 401'd. Asserting that a step EXISTS
proves nothing about whether it can succeed. Each change below was checked
by reintroducing the defect and confirming the test goes red.
Registry-auth precondition (the miss that let the above ship)
- Both redeploy jobs must declare `packages: write` AND a `docker/login-action`
step for ghcr.io, ordered BEFORE the guard. Dropping either reds the suite.
The intersection, executed rather than restated
- The old test only checked that the string `images=` appears in the compute
step; swapping its jq for the full matrix — so a FAILED build moves
`:latest` — kept it green. The real `changed` shell now RUNS, against the
real ALL_SERVICES matrix read out of the workflow, and the emitted
$GITHUB_OUTPUT is asserted: a failed build is in neither set, and the
`skip_build` slot is in `services` but NOT in `images` (handing it to the
guard fails "manifest unknown" and blocks the redeploy for the whole fleet).
Same treatment for the starter lane.
Fixtures joined to real registry output
- `extractRevisionLabel` was pinned to hand-written payloads that were never
compared with reality. Since `readLatestRevision` maps every failure to
null, and null ADVANCES, a parser that silently never matches yields a
permanently-blind guard with a fully green suite. Both fixtures are now
verbatim `docker buildx imagetools inspect --format '{{json .Image}}'`
output (buildx v0.35.0), unformatted so key order survives: a platform-keyed
multi-arch image carrying the label, and our own `:latest`, which turns out
to be a bare config object with NO labels at all. The label key used in the
injection test is read from the workflow's own `labels:` input, so parser
and producer cannot drift apart silently.
Closed vacuities
- The `:latest` ban read one step's `tags` and was vacuously green on an empty
list. It now covers every tagged step plus hand-rolled `docker push`/`docker
tag`/`imagetools create`, and fails on an empty `tags`.
- The `cancel-in-progress` ban read only top-level config; a job-level
`concurrency` on `build` reproduced the exact harm and stayed green. All
jobs are checked now.
- `already-current` is keyed on the digest, matching the script.
- The unreachable `(null, "ahead")` row is labelled as the defensive
input-space case it is, and a new test pins that the real flow never calls
compare() with an unknown revision.
New coverage: classifyProbeFailure (incl. a REAL execFileSync timeout, and
the two precedence traps — gh's rate-limited 403 is throttling, not auth; a
registry's 404-with-denied is auth, not absent), readFlag, escapeAnnotationData
(a forged `\n::error::` stays inert), isDirectInvocation through a symlink,
the `::error` annotation on fleet failure, and the digest-mismatch advance.
Also: GITHUB_SHA / GITHUB_REPOSITORY are no longer shadowed in the workflow.
The runner exports both and the script reads process.env, so the old
assertions pinned a redundancy rather than a capability — removed together,
as the comment there required. Workflow-reading scaffolding duplicated with
redeploy-guard.test.ts is extracted and the YAML parse memoized.
Targeted repairs around the decision logic, which is unchanged: the
compare direction, the `["behind"]`-only decline, and the fail-open
invariant are all preserved.
Digest/identity
- `already-current` keyed on the revision LABEL treated a rebuild of the
same commit (CVE refresh, cache-miss, retry) as a no-op, stranding
`:latest` on the older digest while digest-pinned prod moved. It now
compares manifest DIGESTS, via a new `GuardIo.readDigest`, and only
when the label claims we are already current — so the common path pays
nothing.
- Retag passes `--prefer-index=false`. buildx carbon-copies a single
index source (digest preserved) but WRAPS a bare manifest in a new
index (digest changes) because `--prefer-index` defaults true. Our
images are indexes only because provenance is enabled elsewhere; this
pins the property regardless.
Observability — fail-open stays, silence does not
- `readLatestRevision` / `compare` no longer swallow the error. Failures
are classified (absent / unauthorized / throttled / timeout /
unavailable / unknown) and everything but a genuinely absent tag emits
a `::warning` saying the guard has stopped guarding. Throttling is the
likeliest failure under the concurrency this exists to handle and was
previously indistinguishable from a clean first build.
- Annotations are properly escaped. Multi-line registry stderr was
folding the real reason out of the annotation meant to report it;
encoding newlines also makes an injected `::` inert.
- Subprocess timeout (60s, `ADVANCE_LATEST_TAG_TIMEOUT_MS`). A hung
registry previously burned the job budget and concluded `cancelled`.
- The fleet result now names which images advanced, were unchanged, or
declined — not just which failed.
Correctness / hygiene
- `extractRevisionLabel` collects every occurrence and answers only when
they agree, so a multi-config payload no longer returns whichever label
key ordering happened to surface last. Dropped the dead cycle guard.
- Flag parser no longer binds a following flag as a value (`--sha --repo
x` yielded the literal `"--repo"` as the sha); `--flag=value` accepted;
an empty `--images` is the documented no-op instead of exit 2.
- Entry guard canonicalises both sides, so a symlinked invocation runs
instead of exiting 0 having done nothing.
- Docs: corrected the never-rethrown claim, the unreachable `identical`
example, the self-contradicting `npx tsx` rationale, and the
preconditions (a retag is a registry WRITE and needs `packages: write`
plus a login, which the redeploy jobs do not currently have).
The `<1.0.0` upper bound changes the text of an existing `[FAIL]` line
from `ag2 is not an exact pin (>=0.9.0)` to `(>=0.9.0,<1.0.0)`, which
shifts the ratchet's SHA-256 over the sorted FAIL set.
The FAIL *set* is otherwise identical -- count stays 31, nothing healed
and nothing regressed. Verified by reproducing the committed baseline
hash b47ca987 on a pristine origin/main tree, then diffing the FAIL
lines against the fixed tree; exactly one line differs:
-[FAIL] ag2: ag2 is not an exact pin (>=0.9.0)
+[FAIL] ag2: ag2 is not an exact pin (>=0.9.0,<1.0.0)
`ag2` matches FRAMEWORK_PATTERNS, which demands an exact `==` pin, so
the dep was already in the baseline's drift set before this change and
remains in it after. Hash-only update; the count is untouched, so the
"never raise the count without sign-off" invariant is not engaged.
ag2 1.0.0 (published 2026-07-27T00:58:37Z) renames its top-level module
from `autogen` to `ag2` and ships no `autogen/` package at all. The ag2
integration floats on `ag2[openai,ag-ui]>=0.9.0`, so every fresh CI
resolution now picks up 1.0.0 and collection fails at:
src/agents/_multimodal_normalize.py:75
from autogen.ag_ui import AGUIStream, RunAgentInput
E ModuleNotFoundError: No module named 'autogen'
This breaks `Python unit tests (3.10)` and `(3.12)` in showcase_validate
on main and on every open PR. It is time-based rather than commit-based:
main only looks green because it has not re-run since the release.
Pin rather than migrate. 1.0.0 is not a rename -- it deletes the stable
`autogen/ag_ui/adapter.py` and promotes the former `autogen.beta.ag_ui`
stream into its place, dropping `AGUIStream(event_interceptors=...)` and
replacing `dispatch(context=...)` with `dispatch(variables=...)`. Since
`_multimodal_normalize.py` overrides `dispatch` and forwards `context=`,
rewriting the import path alone would swap an import error for a runtime
TypeError. A real migration needs to port ~20 `autogen` import sites and
re-verify every ag2 demo cell, which is not an urgent-unblock change.
Port ms-agent-harness-dotnet's ApiKeyResolver into ms-agent-dotnet so the 15
agent factories resolve the OpenAI credential as OPENAI_API_KEY (env) ->
config[OPENAI_API_KEY] -> GitHubToken, and the endpoint via OPENAI_BASE_URL ->
default, instead of hardcoding configuration["GitHubToken"] per agent.
Previously ms-agent-dotnet's main chat clients authenticated only with the
GitHub token. On a fixture-miss fall-through, aimock proxies to real
api.openai.com, which rejects the GitHub token (invalid_api_key, surfaced as
502). ms-agent-harness-dotnet already works because it resolves OPENAI_API_KEY
first; this brings ms-agent-dotnet to parity so its fall-through returns 200.
Interim showcase-side mitigation while the aimock cross-provider guard
(PNI-108, CopilotKit/aimock#340) is deferred.
- Copy agent/ApiKeyResolver.cs verbatim from ms-agent-harness-dotnet (code
copy, no new dependency)
- Rewire 15 factories + A2uiSecondaryToolCaller to ResolveApiKey/ResolveEndpoint;
drop the dead per-file DefaultOpenAiEndpoint const
- Add tests/ApiKeyResolverTests.cs (precedence, mock-endpoint fallback,
non-mock fail-fast)
Verified: dotnet build 0/0, dotnet test 76/76, whitespace format clean, and a
live OpenAI call through the resolver returned 200.
`d6:ms-agent-python/multimodal` has been red in staging and prod since
2026-05-30. Turn 1 (image) passes; turn 2 (PDF) fails. This fixes it —
**without touching the fixture**, because the fixture was never the
problem.
## The verbatim turn-2 error
Backend (`showcase-ms-agent-python`), and reproduced locally:
```
[/multimodal] Streaming failed
openai.InternalServerError: Error code: 503 - {'error': {'message': 'Strict mode: no fixture matched',
'type': 'invalid_request_error', 'param': None, 'code': 'no_fixture_match'}}
The above exception was the direct cause of the following exception:
agent_framework.exceptions.ChatClientException: ("<class
'agent_framework_openai._chat_completion_client.OpenAIChatCompletionClient'> service failed to
complete the prompt: Error code: 503 - {'error': {'message': 'Strict mode: no fixture matched', …
```
Surfaced in the browser as `An internal error has occurred while
streaming events.`, with the probe reporting `failure_turn: 2`,
`turns_completed: 1`.
## Request-shape diagnosis
This reads like a fixture gap and is not one. I pulled the **actual
outbound request** off the local aimock's `GET /__aimock/journal` during
a failing run. Turn 2, verbatim (bodies elided):
```
[0] role=system "You are a helpful assistant. The user may attach images or documents…"
[1] role=user "can you tell me what is in this demo image I just attached"
[2] role=user [image_url <data:image/png;base64,iVBORw0K…>]
[3] role=user [image_url <data:image/png;base64,iVBORw0K…>]
[4] role=assistant "The attached image is the CopilotKit logo — a clean, geometric mark…"
[5] role=user "can you tell me what is in this demo pdf I just attached"
[6] role=user "[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to your React…"
[7] role=user "[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to your React…"
```
One logical user turn arrived as **three separate user messages**, and
the *last* one carries only the flattened document — the question is
nowhere in it. That is why aimock's strict mode refused it:
`userMessage` is a substring match against the last user turn, and the
last user turn was a PDF dump.
**Root cause:** `agent_framework_openai` emits **one OpenAI message per
`Content`**. `_chat_completion_client._prepare_message_for_openai`
builds a fresh `args` dict on every iteration of its content loop, so a
user `Message` carrying `[prompt_text, flattened_doc_text]` serialises
to two consecutive user messages — prompt-only, then document-only.
`_PdfFlattenChatMiddleware` was appending the flattened `[Attached
document]` text as a *second* text `Content` beside the prompt, which is
exactly the shape that gets split.
Two corroborating details that make the mechanism airtight:
- **Why turn 1 (image) passes.** aimock already skips *text-less*
trailing user messages (`getLastUserText` in `router.ts`, whose comment
documents this exact MS Agent Framework behavior). The image turn's
split-off trailing message has no text at all, so aimock falls back to
the prompt message and matches. The PDF turn's trailing message *does*
have text — the document — so there is nothing to skip past.
- **Why `langgraph-python` is green** doing the identical `[Attached
document]` flattening: LangChain keeps multiple text parts *inside one
message* rather than splitting them into separate messages.
This is a product bug, not a mock artefact. Against a real LLM it would
not 503 — the model would just answer the wrong thing, because the
question is buried behind a document dump instead of being the current
turn.
## The fix
`showcase/integrations/ms-agent-python/src/agents/multimodal_agent.py`
1. **Merge** the flattened document *into* the message's existing prompt
text content instead of appending it as a second content. The turn stays
a single text content and serialises to a single user message:
`"<prompt>\n[Attached document]\n<body>"`.
2. The merge **copies** the prompt `Content` rather than mutating it.
This is load-bearing: the middleware restores the original `contents`
list after `call_next`, and that restore only undoes the *list* swap —
an in-place mutation would leak the raw PDF body into the AG-UI
`MESSAGES_SNAPSHOT` and render a wall of PDF text in the user's chat
bubble. There is a test for this.
3. **Attachment-only turns** (a PDF with no question) still work: with
no text content to merge into, the flattened document stands alone as
the message body.
4. **Dedupe identical flattened blocks.** The page's
`LegacyConverterShim` appends a legacy `binary` mirror alongside every
modern attachment part, so the same PDF reached the middleware twice and
its body was being sent to the model twice (visible as the duplicated
`[6]`/`[7]` above). Now emitted once.
Post-fix outbound turn 2, same journal endpoint:
```
[5] role=user "can you tell me what is in this demo pdf I just attached\n[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to your React application with CopilotKit…"
matched fixture userMessage: "can you tell me what is in this demo pdf I just attached"
```
One user message, prompt intact, document intact, emitted once.
## The fixture is untouched
```
$ git diff --stat origin/main -- showcase/aimock/
(empty)
```
The existing `userMessage` match key was always correct; the corrected
request shape is what satisfies it. Relaxing or re-recording the fixture
to match the broken request was an explicit non-goal — it would have
made the cell actively certify a model that never sees the user's
question.
## Same-pattern audit
- `_PdfFlattenChatMiddleware` is the **only** `ChatMiddleware` in
`ms-agent-python`, and the only place in the integration that constructs
`Content` or reassigns `message.contents` (`grep` for `ChatMiddleware` /
`Content.from_text` / `.contents =` across `src/` returns hits in this
one file only). No second instance of the pattern to fix.
- `ms-agent-python` is the only MS-Agent-Framework Python integration
doing PDF flattening — `ms-agent-dotnet` has a multimodal e2e spec but
no Python agent. The other `[Attached document]` implementations
(`langgraph-python`, `langgraph-fastapi`, `agno`, `claude-sdk-python`,
`langroid`, `pydantic-ai`, `langgraph-typescript`, `built-in-agent`) run
on frameworks that do not split a message's contents into separate wire
messages, so they are not exposed to this. The upstream
one-message-per-`Content` behavior is pinned by a dedicated test, so if
it ever changes we find out by that test failing rather than by a silent
regression.
- The file is a regular per-integration file, not a `shared/` symlink
(`git ls-files -s` → `100644`). No shared code touched;
`validate-shared-symlinks.ts` confirms no new erosion.
## Red / green / control
All three on the real probe surface, from a clean worktree at
`origin/main` `38613623f4`.
### RED — before the change
```
$ bin/showcase test ms-agent-python:multimodal --d6 --direct --verbose --cycle --isolate
[conversation-runner] turn 1/2 — assistant settled { bubbleIndex: 0, textLength: 100, hasAssertions: true }
[conversation-runner] turn 1/2 — assertions passed
[conversation-runner] turn 2/2 — sending message { inputLength: 29, timeoutMs: 60000 }
[conversation-runner] turn 2/2 — FAILED {
errorCategory: 'assertion-failed',
turnsCompleted: 1,
elapsedMs: 1577,
bodyTextLength: 421,
hasTextarea: true,
hasErrorBoundary: false
}
[warn] CVDIAG component=harness-d6 boundary=fixture-match … status=miss … error=chat errored: copilot-error-banner visible — An internal error has occurred while streaming events.
[info] probe.e2e-full.service-complete {"slug":"ms-agent-python","passed":0,"failed":1,"skipped":0,"incapable":0,"total":1,"state":"red","durationMs":9384}
✗ d6:ms-agent-python red (9.5s)
multimodal: chat errored: copilot-error-banner visible — An internal error has occurred while streaming events.
0 passed, 1 failed (9.5s)
⚠ Tests failed for ms-agent-python:multimodal (exit 1)
```
Evidence the outbound request lacked the prompt — aimock journal from
that run, 8 entries, `200,503,503,503,200,503,503,503` (2 attempts × 3
retries on turn 2):
```
[5] role=user STRING "can you tell me what is in this demo pdf I just attached"
[6] role=user STRING "[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to…"
[7] role=user STRING "[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to…"
status: 503
```
### GREEN — after the change, fixture unchanged
```
$ bin/showcase test ms-agent-python:multimodal --d6 --direct --verbose --rebuild --keep --isolate
[conversation-runner] turn 1/2 — assistant settled { bubbleIndex: 0, textLength: 100, hasAssertions: true }
[conversation-runner] turn 1/2 — assertions passed
[conversation-runner] turn 2/2 — assistant settled { bubbleIndex: 1, textLength: 233, hasAssertions: true }
[conversation-runner] turn 2/2 — assertions passed
[conversation-runner] conversation completed successfully { turnsCompleted: 2, totalDurationMs: 8279 }
[info] probe.e2e-full.feature-complete {"slug":"ms-agent-python","featureType":"multimodal","pass":true,"durationMs":8788}
[info] probe.e2e-full.service-complete {"slug":"ms-agent-python","passed":1,"failed":0,"skipped":0,"incapable":0,"total":1,"state":"green","durationMs":10187}
✓ d6:ms-agent-python green (10.5s)
1 passed (10.5s)
✓ Tests passed for ms-agent-python:multimodal
```
Both turns pass. aimock journal for that run: **2 entries, statuses
`200,200`** (down from 8 entries with six 503s — no retries needed).
**The fixture was not modified**; `git diff origin/main --
showcase/aimock/` is empty and the diff is two files, both under
`showcase/integrations/ms-agent-python/`.
### CONTROL — an already-green integration, same command, same stack
```
$ bin/showcase test langgraph-python:multimodal --d6 --direct --isolate
[conversation-runner] turn 2/2 — assistant settled { bubbleIndex: 1, textLength: 233, hasAssertions: true }
[conversation-runner] turn 2/2 — assertions passed
[conversation-runner] conversation completed successfully { turnsCompleted: 2, totalDurationMs: 8395 }
✓ d6:langgraph-python green (9.1s)
1 passed (9.1s)
✓ Tests passed for langgraph-python:multimodal
```
Local harness, shared probe, shared frontend and fixtures are all sound
— the red was specific to this integration.
## Covering test
`showcase/integrations/ms-agent-python/tests/python/test_multimodal_pdf_prompt.py`
— 7 tests. Not fakes: each one drives the real
`_PdfFlattenChatMiddleware` and then the real
`OpenAIChatCompletionClient._prepare_message_for_openai`, and asserts
against the actual OpenAI wire payload. The PDF is the bundled
`public/demo-files/sample.pdf` through real `pypdf`, and the prompt
asserted on is **read out of the real aimock fixture** rather than
hardcoded, so the test fails if either side drifts.
Test-level red→green (stash the source change, keep the tests):
```
# pre-fix
FAILED test_multimodal_pdf_prompt.py::test_pdf_turn_last_user_message_contains_the_prompt
FAILED test_multimodal_pdf_prompt.py::test_pdf_turn_serialises_to_a_single_user_message
FAILED test_multimodal_pdf_prompt.py::test_duplicate_pdf_parts_are_flattened_once
3 failed, 4 passed in 2.37s
```
with the primary failure reading:
```
AssertionError: expected the PDF turn to serialise to 1 user message, got 2:
['can you tell me what is in this demo pdf I just attached',
'[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to']
```
```
# post-fix — full integration suite (6 pre-existing CVDIAG + 7 new), CI's exact invocation
$ PYTHONPATH=".:src" python -m pytest tests/python/ -q
13 passed in 2.40s
```
Coverage: prompt survives to the final user turn; the turn stays one
user message; the upstream one-message-per-`Content` split is pinned;
original `contents` restored and the prompt `Content` not mutated;
duplicate mirror parts flattened once; attachment-only turn still
flattens; image turn left byte-identical.
## Pre-push
`validate-parity.ts` 20/20 pass · `validate-shared-symlinks.ts` no new
erosion · `aimock-fixtures.test.ts` 842 pass · full `tests/python/`
suite 13 pass · lefthook `lint-fix` + `commitlint` clean · Python lines
≤88 cols matching the file's existing style · no lockfile churn, two
files in the diff.
## Scope
One cell, one middleware, one integration. The other five red
`multimodal` cells from the same sweep have five different root causes
and are not addressed here.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01PYdjeveT8Xof9TyHWMLoJr
Three merges landed on main within 34 seconds on 2026-07-26. The build
workflow has no concurrency group by design, so all three ran at once and
raced to push the same `:latest` tags. The NEWEST commit's build finished
FIRST, so the two older builds overwrote it:
run 30190815370 7b28934387 (#6162) 06:18:05 -> 06:29:44
run 30190823203 59f275eedc (#6161) 06:18:21 -> 06:29:44
run 30190831480 db75a04837 (#6158) 06:18:39 -> 06:29:13 <- NEWEST
Per-service, the older commit beat the newer one on every shared slot:
shell-dashboard (+1s), showcase-harness (+4s), shell (+11s), shell-dojo
(+16s). All three runs reported `success`. Staging served pre-#6158 code
while CI, the redeploy gate and deploy verification all looked clean.
A concurrency group is the wrong fix. `detect-changes` builds a per-push,
path-filtered matrix, so concurrent runs build overlapping but NON-IDENTICAL
service sets -- 7b28934387 was the only run building ag2, agno, langroid,
spring-ai, strands and 10 others. `cancel-in-progress: true` would have
dropped those entirely, trading a stale-image bug for a never-shipped bug.
`cancel-in-progress: false` is no better: GitHub keeps at most one pending
run per group and cancels any previously-pending one.
Instead make the mutable pointer monotonic. The build step now pushes only
the immutable `:<sha>` tag plus an `org.opencontainers.image.revision`
label. A new guard then advances `:latest` unless the tag already points at
a DESCENDANT of the commit being built, which is exactly the regression
case. Every ambiguous state (no tag, unlabelled legacy image, unreachable
API, diverged history) advances -- a stuck `:latest` is the failure mode
being fixed, so the guard declines only on positive proof of regression.
The guard runs in redeploy-staging / redeploy-staging-starters, immediately
before the Railway pull that consumes `:latest`, over the same
matrix-intersect-build-success set that decides what gets redeployed. That
placement keeps the read->retag window minimal and avoids an unpinned
`npx tsx` fetch on ~50 parallel build slots that have no Node.
Also adds showcase_build.yml to showcase_validate.yml's trigger paths: the
new test asserts against that file's live text, and without the path a PR
re-adding `:latest` to the push step would never run the test that catches it.
Prod is unaffected -- verify-railway-image-refs.ts already pins prod to
`@sha256:<digest>`; only staging consumes the mutable tag.
Alem Tuzlak