Lands the Bucket A framework fix and the fixture-correctness changes
needed to flip the remaining D4/D2 cells in the langgraph-python column
to D5. Full local D5 sweep is green (1 passed, 0 failed).
Framework: every Python `ToolMessage` constructed via `Command(update=...)`
now sets `name=` and `id=str(uuid.uuid4())`. Without these, @ag-ui/langgraph
synthesises TOOL_CALL_START events with `toolCallName: null` and
`parentMessageId: null`, which @ag-ui/client@0.0.53's Zod schema rejects;
the rejection is silently swallowed by `withAbortErrorHandling -> EMPTY`,
completing the SSE observable mid-stream so post-tool state never reaches
the consumer. Fix is applied across shared_state_streaming, shared_state_read_write,
gen_ui_agent, beautiful_chat, subagents (2 sites). Single-flag change in
a2ui_dynamic flips the secondary `_design_a2ui_surface` LLM call to
`streaming=True` so aimock's record/replay (SSE-only) sees it.
Fixtures (d5-all.json):
- toolCallId follow-ups for set_steps (3), display_flight, generate_a2ui (4),
schedule_meeting (2), generateSandboxedUi (7), and revenue chart so
multi-turn probes don't recurse into recursion-limit loops
- four hand-crafted secondary `_design_a2ui_surface` fixtures so A2UI
dynamic renders without a real LLM
- mcp-apps fixture rewritten to emit `create_view` tool call with a
minimal Excalidraw element payload; runtime middleware fetches the UI
resource and the iframe mounts
- AAPL and revenue `hasToolResult: true` follow-ups tightened to
`toolCallId` so they don't match cross-turn after prior turns' tool
results
- voice fast-path content-only fixture
- beautiful-chat-schedule-meeting first-turn fixture gains content so
the conversation runner sees an assistant message before the picker
click assertion
Probes: bumped per-card waitForSelector in d5-gen-ui-headless-complete from
15s to 60s — recharts ResponsiveContainer can be slow under 4 sequential
turns.
Shell-dojo: hide CLI Start Command from the dojo navigation via
`HIDDEN_DOJO_FEATURE_IDS`. Registry/manifests untouched so
harness/parity/dashboard still see it.
Lands the Bucket A framework fix and the fixture-correctness changes
needed to flip the remaining D4/D2 cells in the langgraph-python column
to D5. Full local D5 sweep is green (1 passed, 0 failed).
Framework: every Python `ToolMessage` constructed via `Command(update=...)`
now sets `name=` and `id=str(uuid.uuid4())`. Without these, @ag-ui/langgraph
synthesises TOOL_CALL_START events with `toolCallName: null` and
`parentMessageId: null`, which @ag-ui/client@0.0.53's Zod schema rejects;
the rejection is silently swallowed by `withAbortErrorHandling -> EMPTY`,
completing the SSE observable mid-stream so post-tool state never reaches
the consumer. Fix is applied across shared_state_streaming, shared_state_read_write,
gen_ui_agent, beautiful_chat, subagents (2 sites). Single-flag change in
a2ui_dynamic flips the secondary `_design_a2ui_surface` LLM call to
`streaming=True` so aimock's record/replay (SSE-only) sees it.
Fixtures (d5-all.json):
- toolCallId follow-ups for set_steps (3), display_flight, generate_a2ui (4),
schedule_meeting (2), generateSandboxedUi (7), and revenue chart so
multi-turn probes don't recurse into recursion-limit loops
- four hand-crafted secondary `_design_a2ui_surface` fixtures so A2UI
dynamic renders without a real LLM
- mcp-apps fixture rewritten to emit `create_view` tool call with a
minimal Excalidraw element payload; runtime middleware fetches the UI
resource and the iframe mounts
- AAPL and revenue `hasToolResult: true` follow-ups tightened to
`toolCallId` so they don't match cross-turn after prior turns' tool
results
- voice fast-path content-only fixture
- beautiful-chat-schedule-meeting first-turn fixture gains content so
the conversation runner sees an assistant message before the picker
click assertion
Probes: bumped per-card waitForSelector in d5-gen-ui-headless-complete from
15s to 60s — recharts ResponsiveContainer can be slow under 4 sequential
turns.
Shell-dojo: hide CLI Start Command from the dojo navigation via
`HIDDEN_DOJO_FEATURE_IDS`. Registry/manifests untouched so
harness/parity/dashboard still see it.
agent-config: drop the "introduce yourself per your config" baseline — it
substring-matched every prefixed probe prompt (tone:, expertise:,
responseLength:), forcing byte-identical responses and masking the
per-knob behavior the cell is meant to demonstrate.
shared-state-streaming: add toolCallId-keyed follow-up fixtures
(poem/email/quantum) above each first-turn entry so the matcher returns
content-only after the tool result lands. Without these the unchanged
last user message kept re-matching the first-turn fixture, the agent
re-fired write_document, and langgraph hit its 100-step recursion
limit. The frontend state-binding regression that prevents the streamed
document from rendering is a separate framework-level issue.
Synced to showcase/aimock/d5-all.json so replay-mode picks up both
fixes.
Two follow-up fixes layered on the previous internal-tool rename:
(1) `a2ui_fixed.py` — fixed-schema demo infinite loop on deploy. The
`display_flight` tool returns the raw `a2ui.render(...)` JSON descriptor
as its tool result. gpt-4o-mini reads that opaque blob, can't tell the
flight was rendered, and re-calls `display_flight` indefinitely (visible
on the deployed showcase as 6+ duplicate flight cards stacked under
repeated assistant text). Local was just lucky.
Hardened the docstring + system prompt to spell out: the JSON return
value is the surface descriptor, the card is already rendered, do NOT
call again, reply with one short confirmation and stop.
(2) Rename `render_a2ui` → `_design_a2ui_surface` in shared and
langgraph-python parity copies of `tools/generate_a2ui.py` (+
`tools/__init__.py` re-export `RENDER_A2UI_TOOL_SCHEMA` →
`DESIGN_A2UI_SURFACE_TOOL_SCHEMA`), and in `showcase/shared/typescript/
tools/generate-a2ui.ts`. These shared helpers were the source-of-truth
for the secondary-LLM tool name across integrations; renaming here keeps
parity with the langgraph-python agents already renamed in
`beautiful_chat.py` / `a2ui_dynamic.py`. Other framework integrations
keep their own `render_a2ui` for now (separate parity sweep).
(3) `showcase/aimock/feature-parity.json` — added a sibling fixture
matching `toolName: "_design_a2ui_surface"` for the beautiful-chat Sales
Dashboard pill so the langgraph-python e2e suite still hits a
deterministic mock on Railway. The original `render_a2ui` fixture is
kept above it so other integrations whose secondary LLM still requests
`render_a2ui` continue to match.
(4) Comment update in `beautiful-chat.spec.ts` to name the new internal
tool.
In replay mode (no real OpenAI), the existing frontend-tools fixtures
match by 'userMessage' substring only. Each pill (Sunset / Forest /
Cosmic) had a single fixture that returned both content + a
change_background toolCall. After the frontend tool ran, the agent's
follow-up LLM call still has the same last user message — substring
matches the same fixture again — agent re-fires change_background,
loops to langgraph's recursion limit, probe's settle window never
quiets, run reds out at the assistant-settle timeout.
Add a toolCallId-keyed follow-up per pill that returns content-only,
plus a static tool_call_id on the first-turn toolCall for the follow-
up to pin to. The follow-up combines userMessage + toolCallId so it
can't cross-match the next pill's first-turn request (which still has
a tool result in history from the prior pill, but a different last
user message).
Probe result: d5:langgraph-python/frontend-tools flips green again in
replay mode — was passing in PR #4724 only because the real-LLM run
that PR was tested against happened to settle without re-firing.
Investigation of the remaining bucket-C D4 cells uncovered that the
'recorded' fixtures shipped in PR #4725 are at best redundant with
hand-curated canonical fixtures already in d5-all.json, and at worst
override them with worse data — the recordings collapsed beautiful-chat's
canonical $349/$289 search_flights payload to the LLM-of-the-day's
$319/$289 (probe asserts $349 verbatim) and reasoning-custom's
canonical fixture (which carries a 'reasoning' field) with a content-only
recording (refused chain-of-thought disclosure).
Net D5 effect of removing the recordings:
- d5:langgraph-python/gen-ui-custom (catalog gen-ui-tool-based)
→ green (canonical d5-all.json #39/#40)
- d5:langgraph-python/reasoning-display (catalog reasoning-custom)
→ green (canonical d5-all.json #91 with 'reasoning' field that
cleanly satisfies the probe's reasoning-role testid contract)
Both flips confirmed live against PocketBase via --d5 --live.
The aimock record/replay infrastructure introduced alongside these
fixtures (showcase/docker-compose.{record,replay}.yml,
showcase/scripts/record-d5-fixtures.mjs) stays — it is reusable for
future demos where canonical fixtures don't yet cover the prompts —
just without the misleading initial recordings.
Bucket-C cells that remain at D4 after this change require fixes
outside the fixture/recording scope; see the PR description for
per-cell findings (release-blocked package testids, frontend-tool
dispatch defect, suggestion-bar misrender, resume-from-interrupt UI,
built-in reasoning-message testid).
Adds a record/replay loop on top of the existing local Docker stack so
each langgraph-python D5 demo can capture real-LLM responses once and
replay them deterministically thereafter. One fewer source of D5 flake
per demo: no more 'tests pass with real OpenAI / fail with aimock'
because the prompts have no fixture coverage.
What lands here:
- showcase/docker-compose.record.yml — overlays aimock with --record
+ --provider-openai/anthropic and a writable mount for the recording
dir. Drops the baseline d5-all.json/feature-parity.json/smoke.json
loads so prompts that already match a stale fixture can still proxy
through to the real provider.
- showcase/docker-compose.replay.yml — same writable mount, no
--record, no provider URLs; layers the per-demo fixtures alongside
the baseline ones for normal probe runs.
- showcase/scripts/record-d5-fixtures.mjs — orchestrator. For each
demo (catalog feature ID), drops any prior consolidated <slug>.json,
restarts aimock to clear in-memory recorded fixtures, snapshots
recorded/, runs the d5 probe through pnpm exec tsx, then merges
every per-call file written under recorded/ into a single
showcase/aimock/d5-recorded/<slug>.json (one fixture per LLM turn,
in chronological order).
- showcase/aimock/d5-recorded/<slug>.json × 6 — initial recordings
for the still-red bucket-C cells: beautiful-chat (8), gen-ui-interrupt
(2), gen-ui-tool-based (1), headless-complete (1), reasoning-custom
(1), tool-rendering-default-catchall (2). 15 fixtures total.
- showcase/aimock/d5-recorded/.gitignore — keeps the per-call
recorded/ scratch dir out of the tree (orchestrator deletes per-call
files after consolidation, this just guards against re-runs).
D5 impact: replaying the recordings flips
d5:langgraph-python/gen-ui-custom (catalog gen-ui-tool-based) from red
to green deterministically. The other five demos still fail their UI
or assertion-side checks, but their LLM-side responses are now fixed,
so the remaining work is probe/UI fixes against a stable baseline
rather than flake hunting.
Aimock recorder requires a one-line patch (turnIndex + hasToolResult
on each recorded fixture's match) for multi-turn flows to record
correctly. Upstream fix proposed for @copilotkit/aimock; until it
ships, the orchestrator probes for the patch and aborts loudly with
the missing-fields message rather than silently producing single-turn
fixtures. See the script's header comment for the exact patch payload.
After PR #4718 + the timeout bump in #4722 unblocked the abort-cascade,
post-merge cycles surfaced the real per-probe failures that were
previously hidden. Each fix here is targeted at one real failure mode
identified from PB error class + message:
* chat-css: HALCYON probe selector was `[class~="bg-muted"]`, but v2
emits PREFIXED utilities (`cpk:bg-muted`). Whole-token match on the
prefixed name (`[class~="cpk:bg-muted"]`) fixes both the static
USER_BUBBLE_INNER_SELECTOR and the page.evaluate selector — and is
unambiguous against `cpk:bg-muted-foreground` on nested children.
* gen-ui-agent: probe assumed `agent-step` DOM rows accumulated across
pills (delta ≥ 2 vs. pre-pill baseline). The backend's `set_steps`
reducer is `last-write-wins` (per `gen_ui_agent.py`), so each pill
REPLACES `state.steps` and the DOM swaps. delta was always 0 after
pill 1, even when the new steps rendered correctly. Drop the
baseline-capture preFill, assert ≥ 2 visible rows + per-pill
fingerprint dedup. Tests rewritten to match the swap model.
* gen-ui-headless-complete: chip click used `button >> text="<title>"`
(Playwright chained selector). The button has a stable
`aria-label="Suggestion: ${title}"` attribute set inline; switch to
`button[aria-label="Suggestion: ${title}"]` so we match the moment
the DOM exists rather than waiting for inner text-node hydration.
* shared-state-streaming: fixture sent `{"content": "..."}` but the
langgraph-python `write_document(document: str)` tool expects
`document`. Mismatch meant `state.document` never updated; probe saw
no content delta. Rename the argument key to `document` in both the
per-feature fixture and `d5-all.json` (the runtime fixture). The
middleware's `tool_argument="document"` config requires this name.
* reasoning-display: fixture only emitted `content` (text response).
Probe asserted the v2 `<ReasoningBlock>` (rendered behind
`[data-testid="reasoning-block"]`), which mounts only when a
reasoning-role message lands in the transcript. Add a `reasoning`
field so aimock emits REASONING_MESSAGE_* events.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-feature fixtures (gen-ui-agent.json, shared-state-streaming.json) were updated
to JSON-stringified arguments + explicit tool-call IDs to match the OpenAI wire
shape, but d5-all.json — the runtime fixture mounted by aimock per its Dockerfile
— still had raw-object arguments without IDs. The wire-shape parity claim was
true for the per-feature files but contradicted by the live fixture; aligning
d5-all.json closes that drift so what we tested matches what runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Aimock fixtures match by userMessage substring, which collides across
integrations whose pills share the same prompt template. The validator
correctly catches that the d5 probe fixtures my PR added would dangle
on integrations whose tool surfaces don't include the emitted tool.
Fix by adding `match.toolName` constraints to the affected fixtures so
aimock only fires them when the agent emits that specific tool — and
the validator skips them for demos that don't declare the tool.
Also delete the obsolete `Visualize pitch, yaw, and roll` fixture
(F3 changed the d5-gen-ui-open-advanced probe to use the
`Inline expression evaluator` pill instead).
Locally `pnpm exec tsx validate-fixture-tool-surface.ts` returns
0 drift after this change.
Two small hygiene fixes:
- frontend-tools cosmic-gradient pill matched bare token 'navy' which
could collide with any prompt mentioning navy. Tighten to
'navy → magenta cosmic gradient' — verbatim substring of the
pill prompt in frontend-tools/suggestions.ts. Mirror into d5-all.json.
- d5-all.json header comment claimed the first 7 entries are the
open-gen-ui pills, but ordering shifts as features land (headless-
simple/complete pills currently sit ahead of open-gen-ui after
recent merges). Rewrite the header to describe the ordering
convention generically: high-priority verbatim-prompt fixtures
appear first, first-match-wins; per-fixture _comments and the
per-feature source files are the source of truth.
The render_a2ui matcher emitted the same Card+Metric payload for every
pill, so the d5-gen-ui-declarative probe went red on the second pill —
its per-pill expectedTestIds map demands declarative-pie-chart for the
pie-chart pill, declarative-bar-chart for the bar-chart pill, and
declarative-status-badge for the status-report pill, none of which
were rendered.
Branch the render_a2ui response by combining userMessage substring with
toolName so each pill emits the catalog component its probe expects:
- KPI dashboard → Card + 3 Metric children
- pie chart → PieChart with regional sales data
- bar chart → BarChart with quarterly revenue data
- status report → Card + 3 StatusBadge children
Keep the bare toolName-only matcher at the bottom as a deterministic
fallback so unforeseen pills still render something instead of erroring.
Mirror all five fixture entries into the bundled d5-all.json.
Two pill mismatches were silently routing the headless-complete Stock and
Highlight pills to the showcase-assistant catch-all in feature-parity.json:
- The Stock fixture matched 'AAPL trading' but the SuggestionBar pill
configured by use-headless-suggestions.ts sends 'What's the price of
AAPL right now?' — substring 'AAPL trading' is not in that prompt.
- The Highlight fixture matched 'Highlight \'meeting at 3pm\'' but neither
the empty-state pill ('Highlight: ship the demo on Friday') nor the
SuggestionBar pill ('Highlight this note for me: ...ship the demo on
Friday...') contains that substring.
Switch both matchers to short distinctive substrings ('AAPL' and
'ship the demo on Friday') that appear verbatim in BOTH the empty-state
and SuggestionBar prompts. The tool-rendering AAPL fixture (matcher
'What\'s the current price of AAPL?') stays at higher priority via
array order — first-match-wins keeps it pinned to the tool-rendering
pill, so substring 'AAPL' here only catches headless-complete pills.
Update narration response text to reference the correct highlighted
phrase.
The HELLO_LEADING phrase was the showcase-assistant catch-all
boilerplate ('I can help you with weather lookups...') that other
tests in this PR explicitly guard AGAINST. The dedicated d5-all.json
fixture for 'Say hello in one short sentence' now returns a distinct
non-boilerplate greeting; the spec asserts that distinct phrase, so a
fixture-priority misroute fails loudly instead of passing by accident.
Resolved d5-all.json conflict by appending B6's headless-simple/complete fixtures (Say hello, joke, fun fact, Highlight, chart) before the tool-rendering and open-gen-ui entries already on integration.
Adds per-pill aimock fixtures backing the Phase-2B genuine D5 probes:
- agent-config: 6 fixtures (3 knob pairs) so concise vs detailed
responses differ deterministically; satisfies the new probe's
text-diff + length-diff assertions.
- frontend-tools: 3 per-pill fixtures (sunset/forest/cosmic) emitting
change_background tool calls with family-specific gradient hexes.
- frontend-tools-async: query_notes tool call so the NotesCard mounts.
- gen-ui-agent: 3 per-pill set_steps tool calls with distinct step
content so the per-pill content-fingerprint assertion catches
fixture-drift.
- gen-ui-declarative: 4 per-pill generate_a2ui calls + render_a2ui
fixture for the secondary LLM call that paints the catalog.
- gen-ui-a2ui-fixed: SFO/JFK display_flight tool call.
- gen-ui-interrupt: 2 per-pill schedule_meeting tool calls with
distinct time-slot payloads.
- gen-ui-open: generateSandboxedUi tool call with a non-trivial
HTML payload so the iframe[srcdoc]-mount assertion has ≥ 100 chars
to observe.
- shared-state-streaming: 3 per-pill write_document tool calls with
substantive content payloads (≥ 100 chars each).
- readonly-state-context: pill-prompt fixture; the probe's network-
payload assertion checks the request body, not the response.
d5-all.json is updated with the new entries prepended so first-match
precedence routes specific pill prompts to their per-pill fixtures
ahead of the generic adjacent matches.
Add 14 fixtures at the top of d5-all.json so they win precedence over
the existing 'weather in Tokyo' / 'd5 beautiful-chat probe: search
flights' substring matches:
- Chain tools: 3 toolCalls in one assistant turn (get_weather Tokyo +
search_flights SFO->Tokyo + roll_d20=11) plus follow-up text.
- Weather in SF: get_weather(San Francisco) + text (deterministic).
- Find flights: dedicated search_flights(SFO,JFK) fixture, no longer
leaks the a2ui beautiful-chat shape.
- Stock price: get_stock_price(AAPL) returning $338.37 / -2.96%.
- Roll a 20-sided die: 5 sequential roll_d20 fixtures with
turnIndex 0..4 (stateless across runs), values [7,14,3,19,20].
Adds deterministic fixtures to d5-all.json (loaded before feature-parity.json
so it wins match precedence) for the Phase 1A genuine-pass cells:
- hitl-in-app refund #12345 and escalate #12347 pills: each emits a
request_user_approval tool call on the first turn, then branches via
sequenceIndex on the second turn (0 = approve response, 1 = reject
response). Test ordering (serial mode in the spec) drives which branch
is claimed first per pill pair.
- frontend-tools-async project-planning, auth, and reading pills: each
emits a query_notes tool call so the async client-side handler runs
against NOTES_DB and renders project-planning / auth / reading-tagged
matches. The reading pill's 2nd-turn narration locks the canonical
leading phrase per spec test #4.
Replaces nothing — purely additive. The dedicated pill-prompt entries
beat the broad 'plan' and showcase-assistant catch-all entries that were
silently winning before.
- Drop the 8 stale tests that asserted travel-planner shapes
(Current Itinerary / supervisor-indicator / .bg-gray-50). They
predated the supervisor + 3-subagent rewrite and could not catch
the 3 production bugs.
- New suite (5 tests, 0 skipped):
1) page loads with composer + 3 verbatim suggestion pills + 3
subagent role indicators (testid-based).
2-4) one test per pill (Write a blog post / Explain a topic /
Summarize a topic). Each clicks the pill, waits for all
three role-scoped subagent cards to reach data-status=
"complete", then asserts each card's subagent-result is
non-empty AND does not contain the showcase-assistant
boilerplate fragments. Test 4 also serves as a regression
gate on the delegations reducer fix — without it, that pill
returns HTTP 400 and the cards never reach complete.
5) clicks any pill, waits for terminal state, then asserts the
critic-card count is exactly 1 and stays at 1 with status
complete across a 5s dwell — catches any return of the
supervisor -> critic loop.
- Aimock fixtures: 3 verbatim pill chains (cold exposure training,
LLM tool calling, reusable rockets) added to both d5-all.json and
the harness source d5/mcp-subagents.json. Each chain drives the
full supervisor flow (turnIndex 0..3) plus three nested sub-agent
fixtures so every Researcher/Writer/Critic card surfaces real
prose instead of showcase boilerplate.
The 5 of 7 open-gen-ui pill prompts (3D axis, neural network, quicksort,
calculator, ping-the-host) currently match the broad userMessage: 'hi'
catch-all in feature-parity.json (substring match on 'hi' inside 'Say hi
to the host', 'this', etc.) and resolve to the showcase-assistant
boilerplate greeting instead of emitting the generateSandboxedUi tool
call. That kills the iframe path on 5/7 of the pills.
Add 7 high-priority verbatim-prompt fixtures in d5-all.json (and the
source-of-truth showcase/harness/fixtures/d5/gen-ui-open.json) that emit
deterministic generateSandboxedUi tool calls with inline HTML + CSS in
the tool result. d5-all.json loads BEFORE feature-parity.json so the
first-match-wins ordering naturally beats the catch-all. Pill fixtures
appear at the top of the file so they also win against intra-bundle
specifics. Inline HTML is intentionally minimal — the assertion bar is
iframe presence + non-empty srcdoc, not iframe DOM introspection
(cross-origin blocked under sandbox=allow-scripts only).
Adds high-priority verbatim-prompt fixtures for the headless-simple and
headless-complete pill prompts so the deterministic leading phrases stay
locked under aimock:
- headless-simple: 'Say hello in one short sentence' / 'Tell me a one-line
joke' / 'Give me a fun fact' lock the greeting, joke, and fun fact.
- headless-complete: 'Highlight: ship the demo on Friday' is a NEW
high-priority entry that emits a highlight_note tool call with text
'ship the demo on Friday' plus a deterministic narration. Without it
the showcase-assistant catch-all in feature-parity.json wins and the
pill never paints a Highlight card.
- headless-complete: 'chart of revenue over the last six months' emits a
get_revenue_chart tool call plus a deterministic narration leading
phrase 'Here is the chart of revenue over the last six months'.
Weather Tokyo and AAPL fixtures already lock the leading phrases the
headless-complete spec asserts on; they remain unchanged.
Add four production-code testids on demo-layout.tsx:
- identity-name on the user-name display
- identity-timezone on the timezone display
- identity-avatar on the first-letter avatar circle
- activity-<slug> on each activity <label>, kebab-case of activity name
These let Playwright lock the identity card against the published
context defaults (Atai, America/Los_Angeles, A) and assert the
default-checked activity rows ("Viewed the pricing page", "Watched
the product demo video") without depending on assistant text.
Add two verbatim-prompt aimock fixtures to showcase/aimock/d5-all.json
for the demo's suggestion pills:
- "What do you know about me from my context?" leading phrase
"I see you're Atai, and you're in the America/Los_Angeles timezone.
Recently, you viewed the pricing page and watched the product demo
video."
- "Based on my recent activity, what should I try next?" leading phrase
"Since you recently viewed the pricing page and watched the product
demo video, ..."
Pinned to the verbatim message bodies so they don't contend with the
showcase-assistant catch-all fixture.
The aggregated multi-turn probe from #4672 hit a CopilotKit v2 quirk on
/demos/beautiful-chat: only the FIRST useComponent tool call in a
conversation paints its component. Subsequent tool calls emit (the
agent's followup content arrives) but the component never mounts.
Reproduced cleanly without any frontend tool involvement —
pie-chart turn 1 paints 5 svg circles in seconds, bar-chart turn 2
emits "Bar chart rendered above..." but paints zero recharts elements.
The runner can't sidestep this from inside one conversation without a
page.reload() between turns, which the structural Page type doesn't
expose. Splitting into per-pill scripts means each probe gets its own
browser launch — fresh page state, fresh conversation, no useComponent
ordering pollution. CATALOG_TO_D5_KEY maps `beautiful-chat` to all
listed literals; isD5Green requires every key green for the cell to
advance to D5, and per-pill failure isolation surfaces in PB row names.
Coverage in this PR (5 pills):
- beautiful-chat-toggle-theme (frontend tool, html.dark flip)
- beautiful-chat-pie-chart (controlled gen-UI useComponent)
- beautiful-chat-bar-chart (controlled gen-UI useComponent)
- beautiful-chat-search-flights (A2UI fixed-schema FlightCards)
- beautiful-chat-schedule-meeting (HITL with slot-click resolution)
All 5 verified locally end-to-end (5/5 pass against the local stack).
Out of scope, intentionally (track in follow-up):
- Excalidraw — depends on mcp.excalidraw.com reachability
- Calculator — sandboxed iframe; dup of d5-gen-ui-open
- Sales Dashboard — generate_a2ui → render_a2ui chain renders
Metric labels but Row-bound charts don't paint
recharts containers under aimock fixtures (live
pill against same fixture chain shows the
inverse symptom). Suggests aimock's
non-progressive arg streaming differs from a
live LLM in a way the A2UI binder is sensitive
to. Needs separate aimock/binder investigation.
- Task Manager — manage_todos dispatches and agent emits closing
content, but StateStreamingMiddleware's
state.todos propagation doesn't populate the
App pane TodoList through aimock — same suspected
root cause as Sales Dashboard.
Architecture details:
- _beautiful-chat-shared.ts factors DOM helpers + per-pill
assertions, mirroring _hitl-shared.ts's pattern for an extended
Page type with click() + a runtime guard
- Each fixture uses unique D5-prefixed userMessage substrings; the
multi-stage Schedule Meeting flow uses hasToolResult false→true
for round disambiguation (no toolCallId leakage since each probe
runs in its own fresh page session)
Promotes /demos/headless-complete to its own D5 feature type so the
dashboard cell can reach D5 instead of riding on the headless-simple
probe (which was navigating to /demos/headless-simple regardless of
which catalog feature triggered it).
- New gen-ui-headless-complete D5 feature type + script that clicks
each suggestion chip via preFill and asserts the right surface
renders: WeatherCard (get_weather), StockCard (get_stock_price),
HighlightNote (frontend useComponent), Excalidraw best-effort, and
the canonical "Asia is the largest continent" text reply.
- Existing gen-ui-headless script now drives both turns by chip
click (Profile card + Largest continent) instead of typing.
- Fixtures pin narration legs with both userMessage AND toolCallId
and order them before the bare userMessage toolCall fixture —
aimock's toolCallId matcher reads the LAST tool message in the
request, but in a multi-turn probe that "last tool" stays on a
previous turn's id until a new tool runs, which would otherwise
hijack a later turn's prompt with a stale narration.
- headless-complete UserBubble + AssistantBubble now carry
data-message-role so the harness conversation runner can detect
message arrivals (mirrors the headless-simple convention).
- Mappings updated in lockstep:
- REGISTRY_TO_D5: headless-complete -> ["gen-ui-headless-complete"]
- CATALOG_TO_D5_KEY (dashboard): same.
Builds on the initial Search Flights-only probe by extending to a
multi-turn ConversationTurn[] that asserts surface-specific render
fingerprints for 7 of the 9 pills in /demos/beautiful-chat.
Pills covered (turn order is load-bearing — see script jsdoc):
1. Toggle Theme — html.dark class flip (toggleTheme frontend tool)
2. Pie Chart — >= 3 svg circles (pieChart useComponent)
3. Bar Chart — recharts container + >= 2 bar rectangles
4. Search Flights — A2UI FlightCard literal fingerprints (#4668 path)
5. Schedule Meeting — MeetingTimePicker mounts; assertion CLICKS a slot
to resolve the HITL pause before subsequent turns
6. Sales Dashboard — A2UI dynamic generate_a2ui → secondary render_a2ui;
"Total Revenue" + recharts container (90s budget)
7. Task Manager — enableAppMode + manage_todos; "To Do" column +
canonical todo title visible. MUST be last
(flips layout, breaks chat input on narrow widths)
Pills intentionally skipped (track in follow-up):
- Excalidraw Diagram — depends on mcp.excalidraw.com reachability,
turning D5 reliability into a 3rd-party uptime bet
- Calculator App — sandboxed iframe makes assertions cross-frame-fragile;
generateSandboxedUi already covered by d5-gen-ui-open
on a different demo route (would duplicate coverage)
Probe uses the runner's structural ConversationPage type for most assertions
(via page.evaluate helpers in the d5-chat-css globalThis-cast pattern, which
keeps the harness's Node-only tsconfig clean of DOM lib types). Schedule
Meeting narrows to an extended Page type with click() — same runtime-guarded
cast used in d5-hitl-text-input — so it can dispatch the HITL slot click.
Fixture file expanded from 2 to 15 entries covering all 7 turns plus their
multi-stage chains (Sales Dashboard primary + secondary LLM + post-tool;
Task Manager enableAppMode → manage_todos → narration). Re-bundled into
showcase/aimock/d5-all.json (additions only — no other fixtures changed).
Conversation runner reports failure_turn so per-pill failure isolation is
preserved on the dashboard's drilldown without needing 7 separate D5 literals.
Beautiful Chat was capped at D4 in the dashboard because it had no
dedicated D5 probe and was deliberately excluded from CATALOG_TO_D5_KEY
(commit 974494ecb stripped the freeloading "agentic-chat" alias). PR
#4668 fixed the A2UI surface rendering and added e2e tests, but those
land at the D3 tier — D5 is a separate probe with its own driver.
Changes:
- New d5-beautiful-chat probe asserts the A2UI fixed-schema FlightCard
surface renders with literal United/Delta/$349/$289 fingerprints from
the search_flights tool. 60s budget on first card, 5s on siblings.
- New harness/fixtures/d5/beautiful-chat.json with two-stage fixture
(hasToolResult false→true) mirroring the gen-ui-headless pattern.
Fixture spliced into the bundled aimock/d5-all.json.
- New "beautiful-chat" D5FeatureType literal in the registry's union +
runtime mirror.
- d5-feature-mapping.ts: replace "beautiful-chat": ["agentic-chat"]
alias with ["beautiful-chat"] so the probe targets its own dedicated
PB key instead of freeloading agentic-chat's green status.
- live-status.ts CATALOG_TO_D5_KEY: re-add "beautiful-chat":
["beautiful-chat"] so computeMaxPossible lifts the D4 cap to D5.
## Summary
Adds hand-rolled persistent suggestion chips to the `headless-simple`
and `headless-complete` demos in the langgraph-python north-star,
propagates the same surface to the other 17 showcase integrations, and
adds a deterministic aimock fixture so a new chip-click e2e test
("Largest continent") rounds-trips against a stable `Asia is the largest
continent…` response across all 18 demos.
## What changed
**Phase 0 — north-star (commit `7cbc5ea8`)**
- `showcase/aimock/feature-parity.json` — new fixture: `What is the
largest continent?` → `Asia is the largest continent — about 30% of
Earth's land area, home to over 4.6 billion people.`
- `langgraph-python/src/app/demos/headless-{simple,complete}/page.tsx` —
refactor `send` / `handleSubmit` to accept `(override?: string)` so chip
clicks dispatch synchronously without a `setInput` round-trip; render a
persistent `<div data-testid="headless-suggestions">` chip row above the
composer with 5 canonical entries; remove the dead
`useConfigureSuggestions` call from headless-complete (it was
registering suggestions nothing rendered).
- `langgraph-python/tests/e2e/headless-{simple,complete}.spec.ts` —
append one new test in each spec asserting chip click → user message →
`Asia` reply.
**Phase 1 — parity propagation across 17 integrations (commit
`4882c61f`)**
- Spec files `headless-simple.spec.ts` and `headless-complete.spec.ts`
are now byte-identical to the north-star in every integration (10 tests
each = 5 simple + 5 complete; verified via `cmp` for all 34 spec files).
- The 5-entry `suggestions` const is byte-identical between every
integration's simple and complete demos.
- All 17 integrations now expose the same selector surface (canonical
headings, empty-state text, `data-testid="headless-complete-messages"`,
dynamic placeholder, `rounded-br-sm` user bubble, no CopilotChat-default
testids).
**Glue preserved per integration** (verified by post-blitz code review):
- `built-in-agent`: `<CopilotKitProvider runtimeUrl="/api/copilotkit"
useSingleEndpoint>` + `agentId: "default"`
- `google-adk` / `llamaindex`: `agentId: "headless_simple"` /
`"headless_complete"` (Python-style underscores)
- `claude-sdk-typescript`: headless-complete
`runtimeUrl="/api/copilotkit-headless-complete"`
- `spring-ai`: 70-line `deduplicateMessages` adapter workaround +
`useMemo` import preserved verbatim
- All `@region[...]` markers preserved in place
**Adapter-specific decisions worth flagging in review:**
- `google-adk` headless-complete: rewrote `message-list.tsx` from
`msg-user`/`msg-assistant`/`agent-thinking` testid scheme to the
canonical `headless-complete-messages` wrapper; rewrote `input-bar.tsx`
placeholder to canonical dynamic; added the missing subtitle and
empty-state hint
- `ms-agent-dotnet`: extracted inline composer to a new `input-bar.tsx`
to match north-star structure
- `llamaindex`, `ms-agent-python`: added the canonical empty-state hint
(was missing entirely)
- `agno`, `built-in-agent`, `crewai-crews`, `mastra`, `ms-agent-dotnet`,
`pydantic-ai`: replaced per-integration empty-state hint with the
canonical Excalidraw line — chosen for parity over per-integration
accuracy (some demos don't actually wire an Excalidraw tool; alignment
was the explicit goal)
## Verification
- `validate-parity.ts`: 18/18 packages pass, 0 MUST failures
- `aimock-fixtures` test suite: 18/18 pass
- aimock fixture probed directly: `What is the largest continent?`
returns the canonical Asia response
- Each propagation slot reported `tsc --noEmit` clean (0 new errors) +
`playwright --list` shows all 10 expected tests
- Code review (`pr-review-toolkit:code-reviewer`) on the full diff: 0
Critical / Important / Minor findings, 1 stylistic nit (north-star
`input-bar.tsx` `onSubmit` type contravariant-loose, harmless)
## What was NOT done
Live per-integration Playwright runs against rebuilt Docker images. The
17 containers would each need a no-cache rebuild (~5-15 min each = hours
total) and the canonical local-test path is `showcase test <slug>` per
the existing CLI / CI pipeline. Static + structural verification covers
the propagation pattern.
## Test plan
- [ ] Run `showcase test <slug>` (or equivalent CI job) for at least one
drift-heavy integration: `google-adk` (testid scheme rewrite),
`built-in-agent` (provider glue), `spring-ai` (dedup workaround),
`llamaindex` (added testid + empty-state)
- [ ] Run the existing per-integration Playwright suites for at least
the north-star (`langgraph-python`) to confirm the new chip test passes
against a real backend + aimock
- [ ] Confirm aimock fixture validation still passes after deploy
Search Flights and Sales Dashboard pills both produce visible surfaces
on the langgraph-python beautiful-chat demo. Three independent bugs were
masking each other:
- Flight TypedDict required `id` + `statusIcon`, which the aimock fixture
doesn't supply. langchain rejected the call with `flights.0.id: Field
required` and the agent surfaced the error string as the tool result.
Made the type permissive (only the fields `_build_flight_components`
reads need to be there).
- search_flights now expands flights into literal-children FlightCard
components server-side instead of relying on the structural-children
template form (the binder doesn't reliably expand it for our custom
catalog — sibling demos avoid the form for the same reason).
- Sales Dashboard pill went into a tool-call loop because the
userMessage+toolName fixtures matched both the initial call and the
post-tool turn. Hoisted the toolCallId fixture above them so the
follow-up turn returns content and breaks the loop.
Custom Row/Column reintroduced with `gap` support — the basic catalog's
versions ignore it, leaving cards squished. Children are array-of-strings
only (matches what the agent and fixture emit).
Two new e2e tests cover both pills end-to-end. 3s wait in beforeEach so
the v2 chat provider hydrates before the click dispatches. Full spec:
7/7 green.
PR #4579 added a `reasoning` field to the "show your reasoning step by
step" fixture so aimock would emit response.reasoning_summary_* deltas
for the OpenAI Responses API path. Side effect: aimock's Chat
Completions handler also emits non-standard `reasoning_content` deltas
(DeepSeek/Qwen-style) ahead of the role/content chunks. Many
integrations' OpenAI client adapters don't expect those deltas and
either hang or fail to parse the stream — manifesting as "assistant
did not respond within 30000ms" across most reasoning cells in
production.
Restore the original content-only fixture. The langgraph-python /
langgraph-fastapi agent fixes from #4579 still work against real
OpenAI (gpt-5-mini + Responses API streams real reasoning summaries),
but the aimock-driven path no longer exercises the role-reasoning
render — keyword-only assertion in the d5 probe handles that.
## Summary
The `agentic-chat-reasoning` and `reasoning-default-render` cells in
`langgraph-python` and `langgraph-fastapi` never rendered any reasoning
content. Root cause: both agents were configured with `gpt-4o-mini` +
`use_responses_api=False`, so the underlying model produced no reasoning
content blocks and the Chat Completions API has no reasoning summary
surface in the first place. The frontend's `reasoningMessage` slot
stayed empty even though the cells are billed as reasoning demos.
This PR:
- Switches both agents (and their `tool_rendering_reasoning_chain`
siblings) to `gpt-5-mini` through the Responses API with
`reasoning={"effort":"medium","summary":"detailed"}`, mirroring the
`langgraph-typescript` and `pydantic-ai` agents that already worked.
Model is overridable via `OPENAI_REASONING_MODEL`.
- Updates the aimock `d5-all.json` fixture (and the matching harness
`reasoning-display.json`) to set the `reasoning` field on the `show your
reasoning step by step` match. Aimock now emits
`response.reasoning_summary_text.delta` events so the demo renders
deterministically without a real LLM call.
- Adds a `Show reasoning` `useConfigureSuggestions` pill on both
reasoning pages in both integrations so the demo is one click to
exercise.
- Tightens the `d5-reasoning-display` probe to also assert that a
reasoning-role message rendered (`[data-testid="reasoning-block"]` or
`[data-message-role="reasoning"]`), not just that the word "reasoning"
appears in the transcript.
- Un-skips the three streaming reasoning-block tests in
`agentic-chat-reasoning.spec.ts`, adds a suggestion-pill test, and
extends `reasoning-default-render.spec.ts` to cover the default
reasoning slot.
- Updates the `langgraph-python` QA doc to describe the new model +
Responses API setup and the pill flow.
Verified locally end-to-end: clicking the pill at
`/demos/agentic-chat-reasoning` renders the amber `ReasoningBlock` with
the fixture's reasoning text above the final answer bubble.
## Out of scope
Other integrations were audited and intentionally left alone:
- `langgraph-typescript`, `pydantic-ai` already use a reasoning model +
Responses API and work today.
- `agno`, `claude-sdk-python`, `ms-agent-python` use deliberate
workarounds (XML-tag reasoning + custom AGUI handler, Claude
extended-thinking deltas, `think` tool respectively) because their AG-UI
bridges either don't translate Responses-API reasoning items, run a
multi-call CoT loop incompatible with fixture replay, or don't emit
reasoning events at all.
- `llamaindex` uses `gpt-4.1` and surfaces reasoning inline as assistant
text. Its bridge (`llama-index-protocols-ag-ui`) does not translate
Responses-API reasoning items into AG-UI events; fixing that needs an
upstream patch and is out of scope here.
## Notes
Committed with `--no-verify` (explicit user request) — this worktree has
no `node_modules`, so the lefthook `test-and-check-packages` step
couldn't run locally. Changes are entirely under `showcase/` and CI runs
the same checks.
## Test plan
- [ ] CI fixture-validation passes on `showcase/aimock/d5-all.json`
- [ ] `showcase test langgraph-python --d5 --verbose` —
`reasoning-display` probe green (asserts `reasoning-block` selector +
keyword)
- [ ] `showcase test langgraph-fastapi --d5 --verbose` — same
- [ ] `nx run @copilotkit/showcase-langgraph-python:test:e2e -- --grep
reasoning` — un-skipped specs pass against the deployed Railway image
- [ ] Manual: visit `/demos/agentic-chat-reasoning` on a deployed
langgraph-python, click `Show reasoning`, confirm amber `REASONING —
Agent reasoning` block renders with italic step text above the final
answer bubble
- [ ] Manual: same on `/demos/reasoning-default-render`, confirm
CopilotKit's default `CopilotChatReasoningMessage` card renders
The agentic-chat-reasoning and reasoning-default-render cells in
langgraph-python and langgraph-fastapi were configured with
gpt-4o-mini + use_responses_api=False, which never produces AG-UI
REASONING_MESSAGE_* events: gpt-4o-mini is not a reasoning model and
the Chat Completions API does not surface reasoning summary items at
all. The frontend's reasoningMessage slot was rendering nothing,
even though the cells were billed as "reasoning" demos.
- Switch both reasoning agents to gpt-5-mini (override via
OPENAI_REASONING_MODEL) routed through the Responses API with
reasoning={"effort":"medium","summary":"detailed"} so the model's
chain of thought streams as content blocks that @ag-ui/langgraph
translates into REASONING_MESSAGE_* events.
- Update the aimock d5-all.json and harness reasoning-display.json
fixtures to include a "reasoning" field so aimock emits
response.reasoning_summary_text.delta SSE events deterministically
in CI without hitting a real LLM.
- Add a "Show reasoning" useConfigureSuggestions pill on both
reasoning demo pages so the user can trigger the fixture-matched
prompt with one click.
- Tighten the d5-reasoning-display probe: it now also asserts a
reasoning-role message rendered via [data-testid="reasoning-block"]
or [data-message-role="reasoning"], so a plain text response
containing the word "reasoning" no longer falsely passes.
- Un-skip the three streaming reasoning-block tests in
langgraph-python's agentic-chat-reasoning.spec.ts and add a
suggestion-pill test; expand the reasoning-default-render spec to
cover the default reasoning slot.
- Update the langgraph-python QA doc to describe the new model +
Responses API setup and the suggestion-pill flow.
The 6 beautiful-chat demos (spring-ai, strands, langroid, agno,
claude-sdk-typescript, claude-sdk-python) ship three identical
suggestion chips: "Plan a 3-day Tokyo trip", "Explain RAG like I'm
12", and "Draft a launch email". Against the deployed aimock-backed
showcase, all three were broken:
- Tokyo trip: hijacked by the broad `userMessage: "hi"` fixture,
because the substring "hi" appears inside "arc**hi**tecture" in
the prompt. Returned a generic "Hi there! I'm your showcase
assistant..." greeting with nothing about Tokyo.
- RAG explain: no fixture matched, aimock returned an error.
- Launch email: same — no fixture, error.
Add three on-topic fixtures with the full suggestion sentence as
`userMessage` (effectively-exact substring match). Place them
before the broad "hi" fixture in the file so first-match-wins
routes each suggestion to the right response.
Add a `beautiful-chat.spec.ts` regression suite to all 6
integrations: send each suggestion, assert the right keywords
appear in the assistant reply ("Day 1/2/3" for Tokyo,
"open-book/RAG" for RAG, "Subject:/co-pilot" for email), AND
assert the hijacked greeting is absent. If the broad "hi" fixture
re-broadens or the new fixtures are reordered/removed, these
tests fail loudly.
Add D5 voice test that exercises sample-audio transcription via aimock.
Infrastructure: voice in D5 feature type registry + mapping, skipFill
support in conversation runner (9 new tests), inputValue forwarding
in e2e-deep Page wrappers, aimock transcription fixture, tool-free
weather fallback fixture for agents without tools. Verified locally:
D5 suite passes green on langgraph-python (60.4s).
agent-config: drop the AgentConfigLangGraphAgent subclass and use plain
LangGraphAgent. The subclass repacked CopilotKit provider properties
into forwardedProps.config.configurable.properties so the Python graph
could read them via RunnableConfig.configurable.properties — but
@ag-ui/langgraph@0.0.31 builds the LangGraph SDK request as
{ ..., config, context: { ...input.context, ...config.configurable } }
which merges configurable INTO context. LangGraph 0.6.0+ then rejects
with HTTP 400 'Cannot specify both configurable and context' on every
chat round-trip. Net effect: chat sent the user message, runtime 400'd,
no assistant response ever rendered. Removing the subclass unbreaks
the round-trip; the Python agent falls back to its DEFAULT_* constants
so the demo's frontend toggles no longer steer the system prompt
(known regression, tracked separately pending @ag-ui/langgraph fix
that decouples context from configurable).
byoc:
- D5 probe now sends the 'Sales dashboard' pill prompt (matches the
fixtures added in main:f0a89b843 in feature-parity.json) instead of
the previous generic 'render a byoc hashbrown' prompt that had no
matching JSON-shaped fixture. Removed the now-obsolete byoc.json D5
fixture file and regenerated the d5-all.json bundle (52 -> 50
fixtures).
- Added data-testid='copilot-assistant-message' + data-message-role=
'assistant' to the byoc-hashbrown and byoc-json-render renderer
wrapper divs. The CopilotChat default assistantMessage slot includes
these markers; overriding the slot with a custom JSON-rendering
component dropped them, so the e2e-deep conversation runner's
settle-detection cascade (which counts these selectors) never saw
the response and timed out at 30s. Re-attaching the markers is a
purely additive change that doesn't affect the renderers'
behavior.
- D5 byoc assertion now waits for [data-testid='metric-card'] AND a
chart (bar-chart or pie-chart) to render — a structural check on
the BYOC contract output, not a transcript-keyword check that the
custom renderer would never produce.
E2E status: 31/31 passing locally against
./bin/showcase up langgraph-python aimock with this branch's bundle.
Tyler Slaton