Two built-in-agent demos were broken against a real model while their D5/D6
rows stayed green, because aimock exercises neither failure. Both root causes
were confirmed against the live OpenAI API.
gen-ui-agent stopped mid-plan on every run. `@tanstack/ai`'s `chat()` applies
`maxIterations(5)` when no `agentLoopStrategy` is passed, and nothing errors
when the budget runs out — the run just ends. GEN_UI_AGENT_PROMPT scripts 7
`set_steps` calls (1 initial + in_progress/completed per step x 3) plus a
closing message, so the walk died two calls short with the last step pinned at
`pending` and no narration. Reproduced with the real model and the real prompt:
default budget -> 5 calls, "completed, completed, pending", no message
maxIterations(25) -> 7 calls, all completed, 333-char summary
Every demo factory now passes the shared DEMO_AGENT_LOOP_STRATEGY (25 —
several times the longest scripted walk, still bounded). The two non-streaming
tool-free `chat()` calls keep the default: they have no loop to exhaust.
declarative-json-render rendered nothing at all: RUN_STARTED -> RUN_FINISHED,
no events, no console error, no banner. `text.format: { type: "json_object" }`
has a server-side precondition that the word "json" appear in the request
`input`, but the adapter sends `systemPrompts` as `instructions` and only
`messages` as `input` — so with the JSON directive living solely in
SYSTEM_PROMPT the API rejected every run with
400 Response input messages must contain the word 'json' in some form to
use 'text.format' of type 'json_object'. (param: input)
The directive now rides in `messages` (as `user`, since TanStackChatMessage
admits no `system` role and the runtime hoists system messages into
systemPrompts — the half that is not input). json_object enforcement is kept;
verified live that the run then streams a complete, JSON.parse-able spec.
That 400 was invisible because the hand-rolled converters forward a whitelist
of chunk types and dropped RUN_ERROR — every one except reasoning-factory. The
`type: "tanstack"` factories were fine (the runtime's converter rethrows), so
the four `type: "custom"` converters now call the shared throwOnRunError.
Also: the gen-ui-agent progress card announced "All N steps complete" whenever
the RUN ended, ignoring the step data, so a truncated run read as a UI glitch
instead of an agent that stopped early. The wording is now derived from the
steps (`describeProgress`, extracted pure so it is testable without a DOM) and
a stalled run says so.
Both gotchas recorded in showcase/GOTCHAS.md.
## What does this PR do?
Fixes a fleet-wide false red on `mcp-apps` D5/D6: red on all 18
integrations that support the feature since `first_failure_at`
2026-07-28 23:03Z, while the demos rendered correctly by hand.
**Root cause.** The `completeOnMount` gate added in d70d48a561 named the
`mcp-app-iframe` testid, which only Angular's `copilot-mcp-apps-widget`
declares. `react-core` and `vue` build the sandbox iframe imperatively
(`document.createElement("iframe")`) with no testid, so the settle gate
could never be satisfied — every React/Vue integration timed the turn
out at 30s with `reason=surface-missing` and never reached
`assertIframePresent`, whose `iframe[sandbox]` fallback would have
passed. `crewai-crews` and `langroid` stayed green only because they
skip the feature (`errorClass: "skipped-incapable"`).
Verified against live staging (`showcase-built-in-agent-staging`, after
clicking the "Sketch a system diagram" pill):
```
oldGate_testIdOnly: 0 <- what the probe waited for
newGate_cascade: 1 <- the surface that was there all along
allIframes: [{ sandbox: "allow-scripts allow-same-origin allow-forms", testid: null, hasSrcdoc: true }]
```
**Fixed on both sides of the contract:**
1. **Product** — `react-core` and `vue` `MCPAppsActivityRenderer` now
set `data-testid="mcp-app-iframe"` and `title="Interactive MCP
application"` on the host-created iframe, matching Angular. Pinned by a
new test in each package, so dropping the attribute fails in the package
that owns it rather than silently reddening the fleet a day later.
2. **Harness** — `completeOnMount` accepts CSS `selectors` alongside
`testIds`, so the probe settles on the cascade its own module doc and
assertion already declare:
```ts
completeOnMount: { selectors: ['[data-testid="mcp-app-iframe"],
iframe[sandbox]'] }
```
`querySelectorAll` unions comma-separated branches, so one entry
expresses "any conforming form of this surface" while the
conjunctive-across-entries and `minNewMounts` delta semantics stay
exactly as before; `testIds: ["x"]` is now sugar for `selectors:
['[data-testid="x"]']`, so every other probe is untouched. This half
matters on its own: the integrations pin
`@copilotkit/react-core@1.61.2`, so a testid-only fix would leave 18
cells red until a release plus a fleet redeploy.
Also: a `completeOnMount` spec naming no surface now throws instead of
burning the whole turn budget and reporting a misleading
`surface-missing`. The trap is recorded in `showcase/GOTCHAS.md` next to
the sibling `copilot-assistant-message` testid gotcha.
## Verification
- harness `conversation-runner.test.ts` + `d5-mcp-apps.test.ts` —
101/101, including new red-green pairs: a `selectors` cascade greens, a
never-mounting cascade still reds `surface-missing`, a leftover-only
surface still reds, an empty spec fails loud
- `@copilotkit/vue` `MCPAppsActivityRenderer.test.ts` — 6/6
- `@copilotkit/react-core` `MCPAppsActivityRenderer.e2e.test.tsx` —
17/17
- `oxfmt` clean, `check-types` clean on both packages
- Pre-existing and unrelated (fail identically on an untouched worktree
on Windows): harness `typecheck` wants the gitignored generated
`showcase/shell/src/data/frontend-catalog.json`; 6 `src/probes` tests
assert POSIX path separators
## Related PRs and Issues
- Regressed by #6212-era commit d70d48a561 ("test(showcase): add
deterministic Angular parity audit")
## Checklist
- [x] I have read the Contribution Guide
- [x] If the PR changes or adds functionality, I have updated the
relevant documentation (`showcase/GOTCHAS.md`)
- [x] "Allow edits by maintainers" is checked
🤖 Generated with [Claude Code](https://claude.com/claude-code)
mcp-apps is aimock-green but live-degraded: gpt-4o-mini emits malformed Excalidraw
create_view elements JSON (stray trailing }) -> MCP JSON.parse fails -> black
canvas live. gpt-5.4 fixes it live but reds aimock D6 because the TS MCP-Apps
middleware doesn't act on aimock's REPLAYED responses-path create_view tool call
(never mounts the iframe; fixture is byte-identical to LGP, LGP's Python runtime
mounts it — so TS-runtime-specific, not a fixture issue). Kept gpt-4o-mini to hold
D6 green; the real fix is a TS MCP-Apps-middleware/@copilotkit runtime change +
then gpt-5.4. Flagged as follow-up.
The `completeOnMount` gate added in d70d48a561 named the `mcp-app-iframe`
testid, which only Angular's `copilot-mcp-apps-widget` declared. react-core
and vue build the sandbox iframe imperatively with no testid, so every
React/Vue integration timed the turn out at 30s with
`reason=surface-missing` and never reached `assertIframePresent` — whose
`iframe[sandbox]` fallback would have passed. D5 + D6 `mcp-apps` went red on
all 18 integrations that support the feature (first_failure_at 2026-07-28
23:03Z) while the demos rendered correctly by hand.
Fixed on both sides of the contract:
- react-core and vue now set `data-testid="mcp-app-iframe"` and a `title` on
the host-created iframe, matching Angular. Pinned by a test in each package.
- `completeOnMount` accepts CSS `selectors` alongside `testIds`, so the probe
settles on the same cascade its module doc and assertion already use
(`[data-testid="mcp-app-iframe"], iframe[sandbox]`). A comma-joined entry is
one conjunctive surface whose branches `querySelectorAll` unions, so the
delta/`minNewMounts` semantics are unchanged and `testIds` is now sugar for
the equivalent selector. This half greens the fleet on the next sweep
without waiting for a package release, since the integrations pin
@copilotkit/react-core 1.61.2.
A spec naming no surface now throws instead of burning the turn budget and
reporting a misleading `surface-missing`.
Verified against live staging: after clicking the pill, the old gate matched
0 elements and the cascade matched 1 (the sandboxed iframe was there all
along). Also recorded in showcase/GOTCHAS.md.
Record the durable findings from OSS-583:
- langgraph-typescript per-framework notes: langchain-js streaming-reassembly bugs
(reasoning index-collision -> disableStreaming; tool-call+content reassembly ->
normalizeAssistantMessage), a2ui-recovery inner-render header forwarding via
ALS/wrapToolCall (mirrors mastra), and multimodal's LFS-asset build requirement.
- 'What Was Green But Still Wrong' #8: single-pass sweeps hide flaky cells and
aimock-green can mask a real backend bug -> re-run for stability + verify the
right graph runs.
- Known fleet-wide follow-ups: the frontend-tools-async stale turnIndex fixture
reds the north star too (needs the same deletion in langgraph-python); image
builds must git lfs pull so multimodal ships real assets not LFS pointers.
Every affected cell was D6-green (or D4-gated) while the deployed demo was
visibly broken. The common cause is GOTCHAS #8: aimock replays a scripted
tool-call sequence keyed on userMessage + context, so the fixture answers a
question the model was never asked.
Prompt gaps — ~20 demos shared one prompt-less `createBuiltInAgent()` where the
reference wires each to its own graph AND its own system prompt. Adds
`createBuiltInAgent({ systemPrompt })` + `demo-prompts.ts`, ported from the
reference graphs:
- gen-ui-tool-based plotted zeros; the assistant said "I used placeholder values
since no sales figures were provided". Nondeterministic — some runs invented
real values, which is how it passed review.
- gen-ui-agent published its plan once then narrated, freezing the progress card
on step 1 of 4.
- a2ui-recovery painted five identical cards: nothing constrained the supervisor
to one `generate_a2ui` call, and the pills literally ask it to "self-correct".
The retry loop lives inside the tool, so supervisor retries are duplication.
subagents' delegation panel was permanently empty — the frontend reads
`agent.state.delegations` and no code emitted that slot. The converter now emits
a `/delegations` delta per sub-agent result (whole-array `add`, since initial
state is `{}` and strict fast-json-patch rejects unresolvable paths while
@ag-ui/client swallows the throw), and ports the reference's
`_MAX_CRITIQUE_ITERATIONS = 1` cap.
declarative-json-render dumped raw JSON for any prompt the model couldn't crib
from the worked example. Captured the SSE: the wire is already one closing brace
short, so nothing is dropped in transport — model-level enforcement had been
removed on the grounds that "the system prompt already enforces JSON-only
output". It does not. Restores it via the Responses API's `text.format`, the
param the removed `response_format` maps to, verified against the pinned
@tanstack/ai-openai@0.15.6 -> @tanstack/openai-base@0.9.2 request mapping.
declarative-gen-ui and a2ui-recovery were the inverse: capped at D4 (UI ✓ BE ✓
1P ✗ D6 gated) while working live. Their secondary design-call fixtures gated on
`match.responseFormat`, which aimock can never satisfy here — it has no
`text.format` handling and only forwards a top-level `response_format` the
Responses API doesn't accept. Re-keyed onto `match.toolName` (the outer call
declares `generate_a2ui`, the in-tool design call declares nothing) with the
tool-less fixtures moved last, since several pills' brief is a substring of the
pill text.
Tests (both mutation-verified — they fail with the fix reverted):
- aimock-a2ui-routing.test.ts drives aimock's real `matchFixture`; 14/16 cases
fail on the old fixtures with "no fixture matched the secondary design call".
- tanstack-factory.test.ts covers the `/delegations` and `/steps` deltas.
Typecheck unchanged at 61 pre-existing errors; oxlint clean on changed files.
NOT verified live: the four prompt/`text.format` changes need a real-LLM
click-through, which needs a key this environment doesn't have. Reasoning and
the exact request mapping are documented inline.
aimock replays scripted tool calls keyed on userMessage+context regardless of
which graph ran or its tool set, so a demo mis-wired to the sample_agent
fallback passes D6 green while breaking against a real model (OSS-582:
gen-ui-tool-based looped on query_data; shared-state-streaming wrote only to
chat; agentic-chat ran with the wrong tool set). Document that D6-green is
necessary but not sufficient, the real-LLM live click-through procedure, and
the structural tell (route.ts fallthrough loop vs LGP's dedicated wiring).
Re-tier the showcase docs tree to be an agent entry point: README.md
opens with a 'when X, see Y' fanout table that routes to the right
procedural doc; each procedural doc gets a one-line tagline answering
'what does this answer'.
Consolidation:
- DELETE showcase/RUNBOOK.md — operational content merged into DEBUGGING.md
(Integration Patterns, Docker Compose Environment, Production Debugging,
Anti-Patterns, Aimock Fixture Deployment, Dev Iteration Speed). The
--isolate mechanics + CLI rules were already duplicated in DEBUGGING.md.
- DELETE showcase/QA-COVERAGE.md — per-demo coverage matrix + starter hero
matrix + probe depth + infra locations + gaps folded into TESTING.md as
the 'Per-Demo Coverage Matrix' section.
Taglines added (no behavioral change to content): TESTING.md, DEBUGGING.md,
GOTCHAS.md, INTEGRATION-CHECKLIST.md, STYLING-GUIDE.md, FRONTEND-STRATEGY.md,
RAILWAY.md, bin/README.md, aimock/README.md, aimock/RAILWAY.md,
harness/README.md, harness/docs/rotation-drill.md.
Cross-link fixups: FRONTEND-STRATEGY.md (was QA-COVERAGE.md →
TESTING.md#per-demo-coverage-matrix), TESTING.md (removed dangling RUNBOOK
companion reference), README.md (rewritten as fanout entry + retained
from-scratch setup + dashboard SOPs below the fanout).
PARITY_NOTES.md × 12 left alone (per-slug context, not redundant).
(cherry picked from commit 75c9d9755c9118c8abc1fa52deda2012b768cab1)
(cherry picked from commit b64189bae0fe2c9e3a5e3ca440013deb4121f23b)
New content:
- TESTING.md: add 10-step cell red→green SOP + bin/showcase test invocation
table (control-plane vs --direct, per-demo scoping matrix); retain
existing CI gating matrix below.
- GOTCHAS.md: add operational gotchas — aimock caches fixtures at container
startup (warm-slot reuse needs docker restart) + --isolate slot collisions
with foreign Docker projects.
- README.md: cross-link to TESTING.md SOP from CLI section; flesh out
--isolate / --direct in test options table; update use cases.
- RUNBOOK.md: update Verifying a Slug's D6 State to use auto-named --isolate;
note A21+A21b per-slug rebuild scoping; rewrite Fixture Matching to teach
picking the backend-id-invariant discriminator (turnIndex post-A12/A13/A20);
modernize Debugging Sequence to --isolate flow.
- DEBUGGING.md: lead with TESTING.md SOP cross-link; update Phase 1 to
--isolate canonical; soften turnIndex-only log-line description; note
aimock startup caching in Phase 5; switch Strategy 5 gold-standard check
to --isolate.
Pruned/updated stale claims (post-A11/A12/A13/A18/A20/A21/A21b):
- RUNBOOK.md "Do not use turnIndex in new fixtures" — turnIndex is now
the canonical backend-id-invariant alternative when toolCallId is fragile
(Anthropic / TanStack Responses API ID rewrites). Replaced with discriminator
selection guidance.
- RUNBOOK.md anti-pattern "NEVER use turnIndex" — replaced with NEVER
anchor on toolCallId strict equality against ID-rewriting backends, and
NEVER use --direct for value-tests.
- RUNBOOK.md bin/showcase test <slug> --d5 (no --isolate) as canonical SOP
— replaced with --isolate canonical, no manual name required.
- README.md --d5 option description claiming "subagents/tool-rendering/agentic-chat"
fixed slate — replaced with "defaults to agentic-chat representative; :demo
qualifier honored post-A18".
- DEBUGGING.md Phase 1 "showcase up aimock <slug> && showcase test <slug> --d5"
as primary — kept as legacy alternative; --isolate is now lead.
- DEBUGGING.md Phase 5 "fixtures baked into Docker image" — clarified that
aimock additionally caches fixtures in memory at startup (volume-mounted
isolated stack still requires docker restart for warm-slot edits).
- DEBUGGING.md Strategy 5 "showcase test langgraph-python --d5" — replaced
with :demo + --isolate so the gold-standard check exercises the same cell.
(cherry picked from commit 0e548455043396972f7fb5b96f8c0ea8abdf1d98)
(cherry picked from commit 592c02d392350d02cc5e17544e663a6605b8da65)
- SUPERSEDED annotations on unreachable recorded calculator entries (load-order shadowing is
the only guarantee; model gate is not a safety net)
- GOTCHAS sequenceIndex rewritten to per-X-Test-Id semantics with co-increment/eviction caveats
- hasToolResult paragraph corrected (omission = no gate, thread-global predicate)
- statelessness claim reconciled with sequence counters
Add fixture analysis/split/merge scripts for the D6 restructure,
update aimock fixture collision detection tests for the new directory
layout, update GOTCHAS.md and QA-COVERAGE.md with D6 notes, and
add --d6 flag support to the showcase test CLI.
What we learned getting 18 integrations to D5 green — and what was
still wrong even when green. Cross-framework patterns (V1/V2, testids,
runtime hoisting, agent naming, multimodal shims, Pydantic models),
per-framework edge cases for all 15 frameworks, aimock fixture gotchas,
and the 6 things that were "green but wrong."
github-actions[bot]