Three follow-ups on top of PR #4837 that I had on the same branch but
didn't make it into the squash merge.
1. **packages/runtime: stamp `audio/webm` on empty-type Blobs in the
transcription handler.** Browser MediaRecorder writes the audio as
webm/opus, but the Blob's `type` field is often empty by the time it
hits the server. `isValidAudioType` lets empty / octet-stream through
for compatibility, but OpenAI Whisper then rejects the upload with
`502 Invalid file format. Supported formats: ['flac', 'm4a', 'mp3',
'mp4', 'mpeg', 'mpga', 'oga', 'ogg', 'wav', 'webm']` because it
can't pick a decoder. Reconstructing the File with an explicit
`audio/webm` type (and a `.webm` filename fallback) makes Whisper
accept the bytes that were already valid. Monorepo-wide — applies to
every integration using `/api/copilotkit-voice/transcribe`.
2. **showcase/aimock/feature-parity.json: port 12 subagents fixtures
from d5-all.json** so the three pills (cold-exposure blog, LLM
tool-calling explanation, reusable-rockets summary) work in
production. d5-all.json already has the full research → writing →
critique chain with substantive content; feature-parity only had the
single LP remote-work pill. Production aimock loads both files but
any case where feature-parity wins first-match needs the same
content. Verbatim port — no fabricated text. Net result: no more
`[sub-agent error] the writing agent...` on the demo's pills.
3. **showcase/aimock both files: scope shared-state-read-write Greet +
Plan-a-weekend fixtures with a true all-defaults systemMessage
gate.** The PR #4837 gate (`systemMessage: "tone: casual"`) only
caught tone changes — name / language / interests changes still hit
the canned fixture. Replaced with a two-element array gate (aimock
supports all-present substring matching, verified in
`/app/dist/router.js`):
- `preferences:\n- Preferred tone: casual\n` — breaks if name is
set (Name line inserts between signature and tone) or tone changes.
- `- Preferred language: English\nTailor every response` — breaks
if language changes or interests are added (Interests line
inserts between language and Tailor).
With `--provider-gemini` already wired in both local docker-compose
and Railway prod, any state change now proxies to real Gemini and
returns a personalised reply.
4. **showcase/aimock/feature-parity.json: re-remove bare 'plan' /
'steps' / 'mars' / 'dashboard' / 'report' substring catch-alls + the
bare 'alice' / 'Alice' fixtures.** These were removed in commit
`ddc2e179` on the PR #4837 branch but didn't survive the squash
merge, so they're back in main and still hijacking hitl-in-app
downgrade-#12346 ('plan'), shared-state-rw weekend pill ('plan'),
subagents 'rockets' pills, hitl-in-chat Schedule-1:1 with Alice
('alice'). Replace the alice pair with a single scoped
`Hi, my name is Alice` fixture for the showcase-assistant
introduction flow.
Local verification:
- `bin/showcase test google-adk --d5` → 38/38 green, 165s.
- Paired curl on shared-state-read-write:
- Default state → canned fixture ("Hi — I'm your shared-state co-pilot…")
- `name=alem` → real Gemini ("Hi there! …")
- `interests=[Cooking, Travel]` weekend pill → real Gemini ("Hey
there! Since you're into cooking and travel, how about a weekend
plan that combines both?")
Production deploys this PR will pick up the aimock fixture changes
(prod loads feature-parity.json from GitHub raw at boot — no image
rebuild needed for that file) plus the runtime change once the
packages/runtime build is republished.
Production-fix bundle for beautiful-chat + 3 other google-adk demos. All
issues either reported on Railway prod or unmasked by the catch-all
render_a2ui fallback I added in #4836. Local D5 stays 38/38 green.
- Missing copilotkit logo: `<img src="/copilotkit-logo-mark.svg">` returned
404 because the SVG never shipped in the integration's public/. Copy
copilotkit-logo-mark.svg + copilotkit-logo.svg over from langgraph-python.
- Multimodal sample.png / sample.pdf were LFS pointers in prod (Railway
build runs without `git lfs pull`), so the magic-byte check rejected them
on first click. Add an integration-scoped .gitattributes that exempts
these two paths from LFS (mirrors what every working sibling integration
already does) and re-stage the files as real binaries (10KB / 2.5KB).
- Sales Dashboard pill returned a generic "Step 1... Step 2..." narration
instead of rendering the A2UI surface. Root cause: the render_a2ui
catch-all fallback I added to aimock/d5-all.json in #4836 fired before
feature-parity.json's specific sales-dashboard fixture (load order
d5-all → smoke → feature-parity). Removing the catch-all from both
d5-all.json and the per-demo source so the specific fixture wins.
- Calculator App pill returned text only with a white iframe because
beautiful-chat's agent instruction never mentioned generateSandboxedUi —
Gemini saw the tool listed via AGUIToolset but had no nudge to use it.
Added a one-line "Interactive / sandboxed widgets" entry to the
instruction; verified Gemini now emits the tool call.
- hitl-in-app refund (#12345) and escalate (#12347) pills broke on the
second click because the 2nd-turn fixtures keyed on `sequenceIndex` 0/1
(a global thread-position counter that drifts when other pills land
tool messages in the same thread). Convert both to `toolCallId +
hasToolResult: true` and drop the reject branch — Gemini reasons
correctly from the tool's `approved: false` return without a fixture
override. Same pattern that fixed tool-rendering-reasoning-chain
previously.
- hitl-in-app downgrade (#12346) pill produced an unrelated
"Research / Outline / Draft / Review / Finalize" plan because the
prompt contains the substring "plan" and feature-parity.json has a
generic catch-all match on `userMessage: "plan"`. Add a specific
downgrade fixture in d5-all.json (loaded before feature-parity.json)
with hasToolResult: false / true branches.
- hitl-in-chat "Schedule a 1:1 with Alice" returned the wrong
"Nice to meet you, Alice in Tokyo" response when clicked AFTER another
pill in the same thread. The 2nd-turn fixture only matched on
`toolCallId` (no hasToolResult), so the bare "alice" / "Alice" greeting
fixtures further down won. Add `hasToolResult: true` to the 2nd-turn
fixture so it scopes correctly regardless of thread state.
- Voice manual recordings always returned "What is the weather in Tokyo?"
regardless of audio content. aimock had a catch-all transcription fixture
(`match: { endpoint: "transcription" }`) that returned the canned
Tokyo string for any audio input. The D5 voice probe uses the sample
audio button which bypasses /transcribe entirely (it injects text
directly into the composer), so removing the transcription fixture
drops aimock into --proxy-only fall-through to real OpenAI Whisper for
mic recordings while D5 stays green. Verified.
- readonly-state-agent-context and shared-state-read-write fixtures
returned hardcoded "Atai" / generic preferences responses even when
the user changed the state values in the UI inputs. Gate the
Who-am-I / Suggest-next-steps / Greet / Plan-a-weekend fixtures on
systemMessage substring matching the canonical default state values
("Atai" name for readonly-state-context; "tone: casual" for
shared-state-read-write). When the user changes state, the agent's
before-model callback rebuilds the system prompt with the new values,
the fixture's systemMessage substring no longer matches, and aimock
--proxy-only falls through to the real model so the response reflects
the actual state. Confirmed with paired curl tests (default state =
fixture match; alem name = real-LLM response).
Local verification: bin/showcase test google-adk --d5 finishes 38/38
green (137s). Calculator pill confirmed via direct ADK invocation
against real Gemini (GOOGLE_GEMINI_BASE_URL=) emits TOOL_CALL_START
toolCallName=generateSandboxedUi. Sales-dashboard pill confirmed end-to-end
returning the full Column / DashboardCards / PieChart / BarChart payload.
readonly-state-context confirmed with name=Atai matching fixture vs
name=alem falling through to real-LLM response that uses the actual
context.
Five distinct root causes were keeping google-adk from full D5 parity
with langgraph-python. Fixing them takes the integration to 38/38 D5
green under aimock locally (verified end-to-end with --live writing
to PocketBase).
- readonly-state-context: page.tsx asked for agent slug
readonly_state_agent_context (underscore) but the registry mounts
it kebab-case as readonly-state-agent-context. useAgent threw,
the demo layout never mounted, and the ctx-name input never rendered.
Align with the registry (and with langgraph-python).
- multimodal: the secondary failure was a backend Pydantic
ValidationError on HttpOptions.api_endpoint. google-genai 1.75
renamed the field to base_url; all three direct genai client
constructors (main.py, beautiful_chat_agent.py, subagents_agent.py)
needed the rename so the secondary A2UI / sub-agent LLM calls stop
crashing. (The pre-existing LFS-pointer issue on the bundled
sample.png/pdf was a worktree hydration problem, not a tracked
code change — git lfs pull handles it.)
- gen-ui-declarative (two bugs stacked):
1. The same api_endpoint -> base_url rename above. The secondary
generate_a2ui planner LLM was failing every request.
2. The D5 fixture emitted components in {id, type, props: {...}}
shape, but sanitize_a2ui_components requires component, so
every entry was dropped and the renderer received an empty
surface. Rewrite both the per-demo fixture and the d5-all.json
aggregate to the flat {id, component, ...props} shape that
langgraph-python's _design_a2ui_surface fixture already uses,
wrapping multi-child layouts in a basic-catalog Column (the
custom Card schema has a single child slot). Add
_design_a2ui_surface variants so LGP gets per-pill payloads too.
- shared-state-streaming: ADK's write_document took content and
the PredictStateMapping read tool_argument="content", but the
shared D5 fixture (and the LGP function signature) names the
argument document. Rename both sides so the fixture's tool_call
args plumb into the function and into PredictStateMapping's
state-key emission. STATE_DELTA now propagates and DocumentView
streams live.
- tool-rendering-reasoning-chain: in thinking mode
(include_thoughts=True), Gemini emits a turn as two separate
non-partial chunks — a text-only chunk with finish_reason=None
and a function-call-only chunk with finish_reason=FUNCTION_CALL.
stop_on_terminal_text fired on the first (text-only) chunk and
set end_invocation=True before the function-call chunk arrived,
which broke AAPL->MSFT chaining. Gate termination on
finish_reason=STOP; FUNCTION_CALL and None both mean "more
chunks inbound — defer". Applies to every agent that uses the
shared callback, so chain-aware behavior is uniform.
Local verification: bin/showcase test google-adk --d5 --live
finishes green for all 38 cells (~140s), dashboard reflects the
results from PocketBase. Manual real-Gemini click-through of the
five fixed demos also passes end-to-end via GOOGLE_GEMINI_BASE_URL=
(empty) recreate.
31 fixtures had userMessage match + toolCalls response but no
hasToolResult constraint. They re-matched on follow-up turns where
a tool result was present, returning another tool call — infinite loop.
`<CopilotKit agent="beautiful-chat">` routes the chat to agent id
"beautiful-chat", but ExampleCanvas called `useAgent()` with no args and
fell back to DEFAULT_AGENT_ID ("default"). The frontend's agent registry
tracks state per id, so `manage_todos` state-deltas from the chat run
landed on "beautiful-chat" and never reached the canvas's "default"
subscription — the Task Manager pill auto-flipped the panel to App mode
but the To Do column stayed empty. Drop the unused "default" alias from
the runtime route and pin the canvas to `useAgent({ agentId:
"beautiful-chat" })` so both halves share one ProxiedCopilotRuntimeAgent
instance. Adds a Playwright regression test asserting the 3 verbatim
todo titles render after the pill click, plus 3 aimock fixtures for the
multi-turn flow (enableAppMode -> manage_todos -> confirmation).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The tool-rendering-reasoning-chain demo previously promised chained tool
calls in its pill titles but the agent and fixtures only delivered single
tools — clicking "Weather + flights to Tokyo" produced just a WeatherCard,
"Compare two stocks" only fetched AAPL, "Find flights from SFO to JFK"
showed flights but no destination weather. Three changes close the gap.
Agent: replace the soft "call 2+ tools when relevant" system prompt with
concrete per-pill chain examples mirroring the pattern already used by the
langgraph-typescript `tool-rendering` agent (weather→flights, ticker→peer,
roll→contrast die, flights→destination weather).
Pills: drop the redundant Tokyo pill (it was the SFO/JFK chain in reverse)
and reword each remaining pill message to PRE-DISCLOSE the chain so the
model commits to the follow-up call:
- "Compare AAPL and MSFT stocks for me."
- "Roll a 20-sided die for me and compare it to a smaller one."
- "Find flights from SFO to JFK and show me the weather there."
Fixtures: 9 fixtures (3 per pill: final-content → second-leg → first-leg,
ordered by toolCallId specificity for first-match-wins). Each fixture is
scoped by a langgraph-python-UNIQUE userMessage tail ("Compare AAPL and
MSFT stocks", "compare it to a smaller one", "show me the weather there").
Those substrings appear nowhere else across the 14+ integrations sharing
showcase-aimock on Railway, so the new fixtures cannot cross-contaminate
the other reasoning-chain demos that still ship the older prompt set.
A toolName-based gate was considered and rejected because most fleet
agents register `roll_dice` and aimock's `toolName` matcher is a tool-LIST
gate, not a tool-CALL gate — it would NOT have isolated this demo.
Probe: collapse the two-turn flow (Tokyo + SFO/JFK) into one chained turn
(SFO→JFK + JFK weather) that asserts BOTH per-tool renderers
(FlightListCard + WeatherCard) mount in a single response. Same coverage
at half the wall-clock and exercises the actual chained-tool path.
Four independent showcase production bugs Alem reported, plus the
D5 multimodal harness regression they unblocked.
Shared-state-read-write: "Greet me" ("Say hi and introduce yourself.")
and "Plan a weekend" ("Suggest a weekend plan based on my interests.")
were matching the bare `hi` and `plan` catch-alls in feature-parity.json
and returning the generic showcase-assistant blurb / 5-step content plan
instead of shared-state-aware responses. Added pill-specific fixtures in
shared-state.json (mirrored into d5-all.json) so the longer userMessage
substrings win first-match-wins ahead of feature-parity.
Auth sign-out: signing out unmounted CopilotKit entirely and bounced
the user back to the SignInCard, so the demo never showcased the
runtime returning 401 — its whole point. The QA contract in
qa/auth.md spelled out the intended UX. Restored it: CopilotKit stays
mounted after the first sign-in, the AuthBanner flips to an amber
"Signed out — the agent will reject your messages" state with a
re-Sign-in button, and CopilotKit's `onError` callback drives a
`data-testid="auth-demo-error"` surface that displays the runtime's
401 the moment the user sends an unauthenticated message. Updated the
e2e spec to match (the old "SignInCard re-mounts after sign-out" test
pinned the regression).
Gen-ui-agent: the aimock fixture short-circuited the 7-step
progression spelled out in `gen_ui_agent.py`'s SYSTEM_PROMPT to a
single set_steps call with all three steps already `completed`, so
the InlineAgentStateCard rendered the final 3/3 state instantly with
no sequential pending → in_progress → completed animation.
Regenerated as a 7-leg toolCallId chain per pill (8 fixtures × 3
pills): seed leg keyed on userMessage with NO `hasToolResult` gate
(matching PR #4770's pattern — `hasToolResult: false` would block the
seed from firing on the second pill in a multi-pill session), then
six toolCallId-keyed transitions, then a final narration. Fixture
order: toolCallId legs FIRST so the most specific match wins.
Multimodal D5: the sample-attachment buttons auto-send via
`agent.addMessage + copilotkit.runAgent` (restored in PR #4761), but
the D5 harness still typed `input` + pressed Enter via the runner
after `preFill`, sending a second user message that competed with the
in-flight image upload — the v1 LangGraph runtime SSE stream got
tangled (browser DevTools showed `statusCode: pending` indefinitely)
and the assistant message never rendered. Added `skipSend?: boolean`
to ConversationTurn (distinct from `skipFill`, which still presses
Enter once the textarea has content) and switched d5-multimodal.ts to
`skipSend: true` with `responseTimeoutMs: 60_000` so the runner waits
on the assistant response without poking the chat further. Bumped the
PDF auto-prompt fixture in feature-parity.json to include the word
"document" so the existing `buildModalityAssertion("document")` check
still lands.
D5 result: 37 → 39 of 40 features passing. Only
`tool-rendering-reasoning-chain` remains and is a separate
agent/runtime bug (Tokyo Responses-API `reasoning` message survives
into the next turn's conversation history, runtime returns
`RUN_ERROR: "message role is not supported"`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The tool-rendering, frontend-tools-async, and hitl-in-app fixtures all gated
their first-leg (tool-emitting) vs. follow-up (narration) responses on
`hasToolResult: false/true` and/or `turnIndex`. Those constraints count the
*entire* thread, so once a user clicked a tool-using pill the thread already
contained tool messages and assistant turns and subsequent pill clicks fell
through to the wrong branch — d20 dropped from 5 rolls to 3, Chain tools
emitted no cards, query_notes returned narration without the Notes DB card,
and the second HITL pill never raised an approval dialog.
Re-key every follow-up fixture on the prior step's `toolCallId` (the matcher
checks `messages[last].tool_call_id`), drop the global `hasToolResult` gates
from the tool-emitting fixtures, and reorder so the toolCallId-specific
fixtures come first under first-match-wins. The d20 chain becomes a linear
toolCallId graph (`call_tr_d20_seq_001` → `_002` → … → `_005`), Chain tools
gets disambiguators for each of its three parallel tool_call_ids, and
Weather/AAPL/query_notes/HITL approve+reject branches all gate on the
specific request_user_approval / get_weather / query_notes / get_stock_price
id that landed last. userMessage matchers are unchanged.
Adds Playwright multi-pill regression tests to the four affected demos that
click every pill sequentially in one thread and assert the full card counts:
- tool-rendering-default-catchall: Find flights → 5 d20 rolls (with 20 last)
- tool-rendering-custom-catchall: 1 flights + 5 d20 + 3 chain = 9 cards
- frontend-tools-async: 3 NOTES DB cards with the right keyword per pill
- hitl-in-app: refund approve then escalate, each with its own dialog
The three interactive Open-Generative-UI (Advanced) pills — Calculator,
Ping the host, and Inline expression evaluator — shipped HTML + CSS
only in d5-all.json. The agent system prompt instructs the LLM to also
emit jsFunctions, but the fixtures didn't, so every in-iframe button
became a silent no-op: the iframe rendered, clicks dispatched no events,
and the sandbox-function bridges to evaluateExpression / notifyHost
were never exercised.
Adds the missing jsFunctions to all three fixtures, using single-quoted
JS so no escaping is needed inside the JSON-stringified tool arguments.
Each handler respects the sandbox="allow-scripts"-only iframe constraints
(no <form>, plain addEventListener) and calls back into the host via
Websandbox.connection.remote.<name>, with the expected return-shape
contract (res.ok + res.value / res.receivedAt).
The "Draw a flowchart" suggestion pill in the mcp-apps demo sent
"Use Excalidraw to draw a simple flowchart with three steps." which
had no matching create_view fixture in d5-all.json. aimock walked
through to feature-parity.json's `{userMessage: "steps"}` substring
fixture and returned a generic "Here is my plan..." content blurb
with no MCP tool call, so the runtime never invoked create_view, the
MCP middleware never fetched the UI resource, and the sandboxed
iframe never mounted.
Add a fixture pair in d5-all.json (and its harness mirror) keyed on
"draw a simple flowchart": turn 1 emits create_view with a three-
step Start -> Process -> End flowchart, turn 2 emits the narration
after the tool result. The distinctive substring beats the generic
feature-parity catch-alls under first-match-wins.
Adds a regression test that loads the same fixture files in the same
order as docker-compose.local.yml and asserts via aimock's matchFixture
that each mcp-apps pill routes to its create_view fixture on turn 1 and
its narration fixture on turn 2.
The langgraph-python multimodal-attachments demo had a stack of bugs
that compounded each other. Fixing them required touching the local
docker-compose, the aimock fixtures, the LangChain middleware, the
client-side AG-UI shim, and the sample-attachment buttons. This
commit lands the full set together because they only make sense as
a unit — verified end-to-end against `showcase up langgraph-python`
in a headed browser. New e2e suite pins each regression.
Supersedes #4584 (the original fix from May 1 that never landed —
this is a fresh port onto the post-refactor file layout where
page.tsx is split into legacy-converter-shim.tsx, multimodal-chat.tsx,
file-to-data-attachment.ts).
What was broken and what changed:
1. Random uploads crashed with `Failed to fetch`. aimock returned
HTTP 404 on no-match, the LangGraph SDK surfaced `NotFoundError`,
the AG-UI stream surfaced a `RUN_ERROR`, the demo crashed.
Added `--proxy-only` + `--provider-openai https://api.openai.com`
to the local aimock command so unmatched user prompts fall through
to real OpenAI (mirrors the Railway aimock setup).
2. Bundled-sample fixtures keyed on user-visible canned prompts.
The auto-prompts are deliberately long, specific, and natural-
reading ("can you tell me what is in this demo image/pdf I just
attached") so they (a) render cleanly as the user message bubble,
and (b) can't collide with arbitrary user prompts — random
uploads phrase questions differently and fall through to the
proxy.
3. Sample buttons now auto-send via `useAgent`. The previous
DataTransfer-based path queued the attachment via the chat's
hidden file input, then required clicking send while the
attachment was still uploading — `CopilotChat.onSubmitInput`
rejects submits during upload AND clears the input regardless,
so the canned prompt was eaten. Rewrite to call
`agent.addMessage(...)` + `copilotkit.runAgent({ agent })`
directly with the base64'd content part, sidestepping the
upload race entirely.
4. PDF flattened text bled into the rendered user message.
`_PdfFlattenMiddleware` ran in `before_model` and returned
`{"messages": rewritten}`, which persisted to agent state. The
chat UI then rendered the `[Attached document]\n<pdf body>` text
part inline with the user prompt. Switched to `wrap_model_call`
so the PDF→text rewrite is scoped to the outgoing model request
only and never pollutes state.
5. Attachments doubled (and PDFs rendered as broken `<img>`). The
`@ag-ui/langgraph` round-trip translates outgoing `binary` parts
to LangChain `image_url` and incoming `image_url` back to `image`
AG-UI parts — regardless of mimeType, so PDFs came back as
`type: "image"` with `mimeType: "application/pdf"` and were
forced into `ImageAttachment`, where the load failed and the
chat showed two "Failed to load image" boxes. Plus the user's
original modern part survived alongside the round-tripped one,
doubling visible chips.
Added a `dedupeUserMessageMedia` subscriber on both
`onMessagesSnapshotEvent` and `onRunFinalized` to:
- dedupe media parts by `source.value` so the local + round-
tripped copy collapse to one chip
- re-key part `type` from `mimeType` so PDFs route to
`DocumentAttachment` (icon + filename) and images to
`ImageAttachment`.
Also flipped the `onRunInitialized` shim from REPLACE to APPEND
— keep the modern part for the UI AND emit a legacy `binary`
sibling for the converter.
6. Regression suite (`tests/e2e/multimodal.spec.ts`). Replaces the
pre-rewrite suite with five focused tests:
- page loads with all expected affordances
- sample image: auto-sends, EXACTLY ONE `<img>`, assistant
references the logo
- sample PDF: auto-sends, EXACTLY ONE `DocumentAttachment` chip
("PDF" label), NO `<img>`, no `[Attached document]` text bleed
- image then PDF in the same session: each message keeps its own
single chip, no cross-contamination
- PDF then image in the same session: symmetric
All 5 pass against the live local stack (15.4s).
D5 probes were red for two demos because the aimock fixtures didn't
exist or were dispatched through the wrong Responses-API path.
- `harness/fixtures/d5/mcp-apps.json` (new) — the MCP Apps probe sent
"Open Excalidraw and sketch a system diagram" with no matching
fixture, so aimock 404'd and the agent threw `NotFoundError`.
Added a two-leg fixture: leg 1 emits a `create_view` MCP tool call
with Excalidraw element JSON; leg 2 is the `hasToolResult: true`
content reply. Mirrored verbatim into `aimock/d5-all.json` so the
Docker-baked aimock has the same matches.
- `harness/fixtures/d5/tool-rendering-reasoning-chain.json` — added
non-empty `content` fields to the first-leg fixtures (Tokyo weather
+ SFO/JFK flights). aimock's Responses-API dispatch routes
tool-call-only responses (no `content`) through
`buildToolCallResponse`, which silently drops the `reasoning`
payload; with a non-empty `content`, dispatch shifts to
`buildContentWithToolCallsResponse`, which emits the
reasoning_summary events the v2 `<ReasoningBlock>` needs to mount.
Mirrored into `d5-all.json`.
- `harness/fixtures/d5/reasoning-display.json` — added a second
matcher for the `reasoning-default` e2e pill prompt ("sky appears
blue"), keyed alongside the existing D5 probe matcher ("show your
reasoning step by step"). Both responses include `reasoning` fields
so the built-in `CopilotChatReasoningMessage` "Thinking…/Thought
for…" header lands.
- `aimock/feature-parity.json` — added a fixture for the agentic-chat
multi-turn name-recall test ("What name did I just give") replying
"You said your name is Alice." The matcher is intentionally narrow
to avoid colliding with the generic showcase-assistant catch-all.
Net D5 effect: `mcp-apps` and `tool-rendering-reasoning-chain` turn 1
go from red → green. The remaining red on `tool-rendering-reasoning-
chain` turn 2 is the deeper aimock+deepagents+Responses-API multi-
turn state-management issue (production with real OpenAI works); out
of scope per the original audit's Phase F.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
LGP probe hung on Turn 2 (Find flights from SFO to JFK) after Turn 1
(weather Tokyo) succeeded. Root cause: the SFO/JFK first-leg fixture
matched on `hasToolResult: false`, but Turn 1's tool result remains in
the conversation by Turn 2, so the matcher skipped it and fell through
to the second-leg fixture (`hasToolResult: true`), returning narration
without the search_flights tool call. The probe then timed out waiting
for the FlightListCard testid that never mounted.
Fix: switch the second-leg fixture to `toolCallId` matching (the last
message must be a tool result with the specific call_id) and drop
`hasToolResult` from the first-leg fixture. The second-leg fixture
must come BEFORE the first-leg in file order because the matcher is
first-match-wins; with `toolCallId`, it cannot accidentally swallow
first-leg requests (whose last message is the user prompt, not a
tool result).
Also adds a `content` field to the Tokyo first-leg fixtures
(`get_weather` and `get-weather` Mastra-hyphen variants) so they route
through `buildContentWithToolCallsStreamEvents` instead of
`buildToolCallStreamEvents` — the latter codepath silently drops the
`reasoning` field, which is why the reasoning-block testid never
mounted on Turn 1 either before this change.
Tokyo (Turn 1) keeps the simpler `hasToolResult` pattern because it has
no prior turns; only the SFO/JFK fixture pair needs the toolCallId
shape to survive multi-turn use.
Three fixes that follow up on PR #4743 to bring LGP closer to fully-green
on the dashboard:
1. tool-rendering-reasoning-chain probe: was failing with
`expected [data-testid="reasoning-block"] to mount within 30000ms`.
Root cause: the demo's `<ReasoningBlock>` slot only mounts when a
reasoning-role message lands in the transcript, which requires
aimock to emit REASONING_MESSAGE_* events, which in turn requires
the fixture's first-leg response to carry a `reasoning` field. The
weather/Tokyo and SFO/JFK first-leg fixtures were missing it.
Mirrors the convention documented in reasoning-display.json:2.
Patched both source (harness/fixtures/d5/) and bundle (aimock/d5-all.json).
2. gen-ui-interrupt source fixture: the source fixture file was missing
the resume-leg toolCallId entries that already existed in the bundle.
Cosmetic mirror so re-bundling stays consistent. Same chip prompts +
same toolCallIds as interrupt-headless.json (both probes share the
same agent and aimock fixture set; the difference is the FRONTEND
rendering — useInterrupt inline vs useHeadlessInterrupt separate-pane).
3. Dashboard gold-standard filter: 4 deprecated/legacy features
(agentic-chat-reasoning, hitl, hitl-in-chat-booking,
reasoning-default-render) used to render as X-marked rows in the
LGP gold-standard dashboard view because LGP intentionally does
NOT implement them — they were consolidated into the modern shape
(reasoning-custom + reasoning-default; hitl-in-chat with
useHumanInTheLoop). Other 17 integrations still serve those legacy
demos, so we don't yank the features from feature-registry.json
entirely. Instead: marked them `deprecated: true` and updated
generate-registry.ts to skip emitting cells when a deprecated
feature is unshipped for an integration. LGP cells: 43 → 39 (the
4 deprecated rows disappear). Other integrations: unchanged
(audit trail preserved). Catalog total: 774 → 770.
Tests:
- 1588/1588 harness vitest passing
- 19/19 generate-catalog + generate-registry tests passing
(counts updated for the 4 dropped LGP cells + new deprecated-
feature filter test)
- validate-fixture-tool-surface clean (282 fixtures × 627 demos)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- 3 new D5 probes (multi-turn, agentic-chat-style) for
`/demos/{interrupt-headless, shared-state-read,
tool-rendering-reasoning-chain}` — closes the demo↔probe coverage gap so
every langgraph-python demo now has its own dashboard cell.
- Driver retry-once in `e2e-deep.ts` — transient `goto-error` /
`conversation-error` failures (≥2s on attempt 1) get one retry before
recording red, cutting ~10× the dashboard flap rate. Persistent
assertion-style fails skip retry.
- Drops the legacy dual-claim where `d5-shared-state.ts` owned both
`shared-state-read` AND `shared-state-write` for a single bidirectional
probe — `shared-state-read` now belongs to the standalone recipe-editor
probe.
## Scope rationale
LGP is the north-star integration; this PR codifies its current demo
surface in the test infra. **Other integrations may flip red on the new
probes — that's expected and welcome.** Cross-integration parity follows
in a separate wave; we're using LGP as the template.
## Files
- `showcase/harness/src/probes/scripts/d5-interrupt-headless.ts` (new) —
`useHeadlessInterrupt` flow: chip → interrupt popup → slot pick →
resume.
-
`showcase/harness/src/probes/scripts/d5-tool-rendering-reasoning-chain.ts`
(new) — combines reasoning-block slot + per-tool renderer (WeatherCard /
FlightListCard).
- `showcase/harness/src/probes/scripts/d5-shared-state-read.ts` (new) —
recipe-editor with neutral default agent, asserts `recipe-card` form
mounts AND agent references recipe context across turns.
- `showcase/harness/src/probes/drivers/e2e-deep.ts` — retry-once loop
around `runFeature`.
- `d5-registry.ts` / `d5-feature-mapping.ts` / `live-status.ts`
(dashboard) / LGP `manifest.yaml` / `feature-registry.json` /
`constraints.yaml` — wire the new featureTypes and features end-to-end.
- `d5-shared-state.ts` (+test) — drops dual-claim.
- 2 pre-existing test fixes folded in: `d5-gen-ui-interrupt.test.ts`
(mock updated to current evaluate-poll resume signal) and
`conversation-runner.test.ts` (preFill ordering assertion now matches
actual deferred-cascade contract).
- aimock `d5-all.json` — +2 shared-state-read fixtures.
interrupt-headless and tool-rendering-reasoning-chain reuse existing
fixtures whose substrings already match their chip prompts.
## Test plan
- [x] `pnpm vitest run` in `showcase/harness/` → **1588 / 1588 passing**
(was 1585 / 1588 with 3 pre-existing fails before this PR; 2 are fixed
here, 1 was an obsolete-mock issue).
- [x] `npx tsx showcase/scripts/validate-fixture-tool-surface.ts` →
clean (282 fixtures × 627 demos, no drift).
- [x] `npx tsx showcase/scripts/generate-registry.ts` → clean (18
integrations, 38 wired LGP features, 756 catalog cells).
- [x] `pnpm typecheck` in `showcase/harness/` → clean.
- [x] `d5-mapping-drift.test.ts` → green (CATALOG_TO_D5_KEY mirrors
REGISTRY_TO_D5).
- [ ] After merge: watch the e2e-deep dashboard rotation — the 3 new LGP
cells should write rows on first tick (no red zombies — these are
first-time emissions).
## Known follow-up (NOT in this PR)
`auth.spec.ts` test #5 ("signing back in re-mounts a fresh chat
surface") fails on Railway. Symptom: after sign-out → sign-in cycle, the
second `Hello again` send doesn't produce an assistant response within
30s. Looks like a `react-core/v2` ref-handling regression on
`<CopilotKit>` unmount/remount — deserves its own focused investigation
rather than expanding this PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Closes the demo↔probe coverage gap for /demos/{interrupt-headless,
shared-state-read, tool-rendering-reasoning-chain} so every demo
under langgraph-python (the north-star integration) now has a D5
probe writing to its own PocketBase cell — not relying on cross-
demo umbrella records.
New probes (multi-turn, mirroring the agentic-chat structure):
- d5-interrupt-headless: exercises useHeadlessInterrupt — chip
prompt → backend interrupt(...) → app-surface popup → slot pick
→ resume → assistant confirmation. Distinct from gen-ui-interrupt
(which uses inline useInterrupt).
- d5-tool-rendering-reasoning-chain: combines reasoning-block slot
+ per-tool renderer (WeatherCard, FlightListCard) on the same
chat surface. Catches a regression in either side.
- d5-shared-state-read: recipe-editor demo (neutral default agent,
no tools) — verifies recipe-card form mounts AND agent reads
shared state across turns. Drops the dual-claim that
d5-shared-state.ts had on `shared-state-read` (now write-only).
Driver retry-once (e2e-deep.ts):
Probes that fail with a transient class (`goto-error` /
`conversation-error`) AND took ≥2s on the first attempt now retry
once before recording red. Persistent assertion-style failures
(sub-2s) and intentional aborts/feature-timeouts skip retry —
retrying a deterministic mismatch just burns clock and obscures
the signal. Cuts ~10× the dashboard flap rate.
Plumbing:
- D5FeatureType enum: +interrupt-headless, +tool-rendering-reasoning-chain.
- REGISTRY_TO_D5 (harness) + CATALOG_TO_D5_KEY (dashboard) mirror
the new mappings; d5-mapping-drift test enforces this.
- LGP manifest features + demos entries + constraints allowlist.
- feature-registry.json: +shared-state-read.
- aimock d5-all.json: +2 shared-state-read fixtures (interrupt-
headless + tool-rendering-reasoning-chain reuse existing fixtures
that already match their chip prompts).
Tests: 1588/1588 harness vitest green. validate-fixture-tool-surface
clean (282 fixtures × 627 demos, no drift). Two pre-existing test
fixes folded in — d5-gen-ui-interrupt assertion mock updated to
match the current evaluate-poll resume signal; conversation-runner
preFill ordering test now asserts the actual deferred-cascade
contract instead of a stricter pre-preFill ban that the runner
never enforced.
Known follow-up (not in this PR): auth.spec.ts test #5 ("signing
back in re-mounts a fresh chat surface") fails on Railway — second
sign-in's "Hello again" never produces an assistant response. Looks
like a react-core/v2 ref-handling regression on <CopilotKit>
unmount/remount; deserves its own focused investigation.
Other integrations may flip red on the new probes — that's
expected. We're treating LGP as the template; cross-integration
parity follows in a separate wave.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands the Bucket A framework fix and the fixture-correctness changes
needed to flip the remaining D4/D2 cells in the langgraph-python column
to D5. Full local D5 sweep is green (1 passed, 0 failed).
Framework: every Python `ToolMessage` constructed via `Command(update=...)`
now sets `name=` and `id=str(uuid.uuid4())`. Without these, @ag-ui/langgraph
synthesises TOOL_CALL_START events with `toolCallName: null` and
`parentMessageId: null`, which @ag-ui/client@0.0.53's Zod schema rejects;
the rejection is silently swallowed by `withAbortErrorHandling -> EMPTY`,
completing the SSE observable mid-stream so post-tool state never reaches
the consumer. Fix is applied across shared_state_streaming, shared_state_read_write,
gen_ui_agent, beautiful_chat, subagents (2 sites). Single-flag change in
a2ui_dynamic flips the secondary `_design_a2ui_surface` LLM call to
`streaming=True` so aimock's record/replay (SSE-only) sees it.
Fixtures (d5-all.json):
- toolCallId follow-ups for set_steps (3), display_flight, generate_a2ui (4),
schedule_meeting (2), generateSandboxedUi (7), and revenue chart so
multi-turn probes don't recurse into recursion-limit loops
- four hand-crafted secondary `_design_a2ui_surface` fixtures so A2UI
dynamic renders without a real LLM
- mcp-apps fixture rewritten to emit `create_view` tool call with a
minimal Excalidraw element payload; runtime middleware fetches the UI
resource and the iframe mounts
- AAPL and revenue `hasToolResult: true` follow-ups tightened to
`toolCallId` so they don't match cross-turn after prior turns' tool
results
- voice fast-path content-only fixture
- beautiful-chat-schedule-meeting first-turn fixture gains content so
the conversation runner sees an assistant message before the picker
click assertion
Probes: bumped per-card waitForSelector in d5-gen-ui-headless-complete from
15s to 60s — recharts ResponsiveContainer can be slow under 4 sequential
turns.
Shell-dojo: hide CLI Start Command from the dojo navigation via
`HIDDEN_DOJO_FEATURE_IDS`. Registry/manifests untouched so
harness/parity/dashboard still see it.
Lands the Bucket A framework fix and the fixture-correctness changes
needed to flip the remaining D4/D2 cells in the langgraph-python column
to D5. Full local D5 sweep is green (1 passed, 0 failed).
Framework: every Python `ToolMessage` constructed via `Command(update=...)`
now sets `name=` and `id=str(uuid.uuid4())`. Without these, @ag-ui/langgraph
synthesises TOOL_CALL_START events with `toolCallName: null` and
`parentMessageId: null`, which @ag-ui/client@0.0.53's Zod schema rejects;
the rejection is silently swallowed by `withAbortErrorHandling -> EMPTY`,
completing the SSE observable mid-stream so post-tool state never reaches
the consumer. Fix is applied across shared_state_streaming, shared_state_read_write,
gen_ui_agent, beautiful_chat, subagents (2 sites). Single-flag change in
a2ui_dynamic flips the secondary `_design_a2ui_surface` LLM call to
`streaming=True` so aimock's record/replay (SSE-only) sees it.
Fixtures (d5-all.json):
- toolCallId follow-ups for set_steps (3), display_flight, generate_a2ui (4),
schedule_meeting (2), generateSandboxedUi (7), and revenue chart so
multi-turn probes don't recurse into recursion-limit loops
- four hand-crafted secondary `_design_a2ui_surface` fixtures so A2UI
dynamic renders without a real LLM
- mcp-apps fixture rewritten to emit `create_view` tool call with a
minimal Excalidraw element payload; runtime middleware fetches the UI
resource and the iframe mounts
- AAPL and revenue `hasToolResult: true` follow-ups tightened to
`toolCallId` so they don't match cross-turn after prior turns' tool
results
- voice fast-path content-only fixture
- beautiful-chat-schedule-meeting first-turn fixture gains content so
the conversation runner sees an assistant message before the picker
click assertion
Probes: bumped per-card waitForSelector in d5-gen-ui-headless-complete from
15s to 60s — recharts ResponsiveContainer can be slow under 4 sequential
turns.
Shell-dojo: hide CLI Start Command from the dojo navigation via
`HIDDEN_DOJO_FEATURE_IDS`. Registry/manifests untouched so
harness/parity/dashboard still see it.
agent-config: drop the "introduce yourself per your config" baseline — it
substring-matched every prefixed probe prompt (tone:, expertise:,
responseLength:), forcing byte-identical responses and masking the
per-knob behavior the cell is meant to demonstrate.
shared-state-streaming: add toolCallId-keyed follow-up fixtures
(poem/email/quantum) above each first-turn entry so the matcher returns
content-only after the tool result lands. Without these the unchanged
last user message kept re-matching the first-turn fixture, the agent
re-fired write_document, and langgraph hit its 100-step recursion
limit. The frontend state-binding regression that prevents the streamed
document from rendering is a separate framework-level issue.
Synced to showcase/aimock/d5-all.json so replay-mode picks up both
fixes.
Two follow-up fixes layered on the previous internal-tool rename:
(1) `a2ui_fixed.py` — fixed-schema demo infinite loop on deploy. The
`display_flight` tool returns the raw `a2ui.render(...)` JSON descriptor
as its tool result. gpt-4o-mini reads that opaque blob, can't tell the
flight was rendered, and re-calls `display_flight` indefinitely (visible
on the deployed showcase as 6+ duplicate flight cards stacked under
repeated assistant text). Local was just lucky.
Hardened the docstring + system prompt to spell out: the JSON return
value is the surface descriptor, the card is already rendered, do NOT
call again, reply with one short confirmation and stop.
(2) Rename `render_a2ui` → `_design_a2ui_surface` in shared and
langgraph-python parity copies of `tools/generate_a2ui.py` (+
`tools/__init__.py` re-export `RENDER_A2UI_TOOL_SCHEMA` →
`DESIGN_A2UI_SURFACE_TOOL_SCHEMA`), and in `showcase/shared/typescript/
tools/generate-a2ui.ts`. These shared helpers were the source-of-truth
for the secondary-LLM tool name across integrations; renaming here keeps
parity with the langgraph-python agents already renamed in
`beautiful_chat.py` / `a2ui_dynamic.py`. Other framework integrations
keep their own `render_a2ui` for now (separate parity sweep).
(3) `showcase/aimock/feature-parity.json` — added a sibling fixture
matching `toolName: "_design_a2ui_surface"` for the beautiful-chat Sales
Dashboard pill so the langgraph-python e2e suite still hits a
deterministic mock on Railway. The original `render_a2ui` fixture is
kept above it so other integrations whose secondary LLM still requests
`render_a2ui` continue to match.
(4) Comment update in `beautiful-chat.spec.ts` to name the new internal
tool.
In replay mode (no real OpenAI), the existing frontend-tools fixtures
match by 'userMessage' substring only. Each pill (Sunset / Forest /
Cosmic) had a single fixture that returned both content + a
change_background toolCall. After the frontend tool ran, the agent's
follow-up LLM call still has the same last user message — substring
matches the same fixture again — agent re-fires change_background,
loops to langgraph's recursion limit, probe's settle window never
quiets, run reds out at the assistant-settle timeout.
Add a toolCallId-keyed follow-up per pill that returns content-only,
plus a static tool_call_id on the first-turn toolCall for the follow-
up to pin to. The follow-up combines userMessage + toolCallId so it
can't cross-match the next pill's first-turn request (which still has
a tool result in history from the prior pill, but a different last
user message).
Probe result: d5:langgraph-python/frontend-tools flips green again in
replay mode — was passing in PR #4724 only because the real-LLM run
that PR was tested against happened to settle without re-firing.
Investigation of the remaining bucket-C D4 cells uncovered that the
'recorded' fixtures shipped in PR #4725 are at best redundant with
hand-curated canonical fixtures already in d5-all.json, and at worst
override them with worse data — the recordings collapsed beautiful-chat's
canonical $349/$289 search_flights payload to the LLM-of-the-day's
$319/$289 (probe asserts $349 verbatim) and reasoning-custom's
canonical fixture (which carries a 'reasoning' field) with a content-only
recording (refused chain-of-thought disclosure).
Net D5 effect of removing the recordings:
- d5:langgraph-python/gen-ui-custom (catalog gen-ui-tool-based)
→ green (canonical d5-all.json #39/#40)
- d5:langgraph-python/reasoning-display (catalog reasoning-custom)
→ green (canonical d5-all.json #91 with 'reasoning' field that
cleanly satisfies the probe's reasoning-role testid contract)
Both flips confirmed live against PocketBase via --d5 --live.
The aimock record/replay infrastructure introduced alongside these
fixtures (showcase/docker-compose.{record,replay}.yml,
showcase/scripts/record-d5-fixtures.mjs) stays — it is reusable for
future demos where canonical fixtures don't yet cover the prompts —
just without the misleading initial recordings.
Bucket-C cells that remain at D4 after this change require fixes
outside the fixture/recording scope; see the PR description for
per-cell findings (release-blocked package testids, frontend-tool
dispatch defect, suggestion-bar misrender, resume-from-interrupt UI,
built-in reasoning-message testid).
Adds a record/replay loop on top of the existing local Docker stack so
each langgraph-python D5 demo can capture real-LLM responses once and
replay them deterministically thereafter. One fewer source of D5 flake
per demo: no more 'tests pass with real OpenAI / fail with aimock'
because the prompts have no fixture coverage.
What lands here:
- showcase/docker-compose.record.yml — overlays aimock with --record
+ --provider-openai/anthropic and a writable mount for the recording
dir. Drops the baseline d5-all.json/feature-parity.json/smoke.json
loads so prompts that already match a stale fixture can still proxy
through to the real provider.
- showcase/docker-compose.replay.yml — same writable mount, no
--record, no provider URLs; layers the per-demo fixtures alongside
the baseline ones for normal probe runs.
- showcase/scripts/record-d5-fixtures.mjs — orchestrator. For each
demo (catalog feature ID), drops any prior consolidated <slug>.json,
restarts aimock to clear in-memory recorded fixtures, snapshots
recorded/, runs the d5 probe through pnpm exec tsx, then merges
every per-call file written under recorded/ into a single
showcase/aimock/d5-recorded/<slug>.json (one fixture per LLM turn,
in chronological order).
- showcase/aimock/d5-recorded/<slug>.json × 6 — initial recordings
for the still-red bucket-C cells: beautiful-chat (8), gen-ui-interrupt
(2), gen-ui-tool-based (1), headless-complete (1), reasoning-custom
(1), tool-rendering-default-catchall (2). 15 fixtures total.
- showcase/aimock/d5-recorded/.gitignore — keeps the per-call
recorded/ scratch dir out of the tree (orchestrator deletes per-call
files after consolidation, this just guards against re-runs).
D5 impact: replaying the recordings flips
d5:langgraph-python/gen-ui-custom (catalog gen-ui-tool-based) from red
to green deterministically. The other five demos still fail their UI
or assertion-side checks, but their LLM-side responses are now fixed,
so the remaining work is probe/UI fixes against a stable baseline
rather than flake hunting.
Aimock recorder requires a one-line patch (turnIndex + hasToolResult
on each recorded fixture's match) for multi-turn flows to record
correctly. Upstream fix proposed for @copilotkit/aimock; until it
ships, the orchestrator probes for the patch and aborts loudly with
the missing-fields message rather than silently producing single-turn
fixtures. See the script's header comment for the exact patch payload.
After PR #4718 + the timeout bump in #4722 unblocked the abort-cascade,
post-merge cycles surfaced the real per-probe failures that were
previously hidden. Each fix here is targeted at one real failure mode
identified from PB error class + message:
* chat-css: HALCYON probe selector was `[class~="bg-muted"]`, but v2
emits PREFIXED utilities (`cpk:bg-muted`). Whole-token match on the
prefixed name (`[class~="cpk:bg-muted"]`) fixes both the static
USER_BUBBLE_INNER_SELECTOR and the page.evaluate selector — and is
unambiguous against `cpk:bg-muted-foreground` on nested children.
* gen-ui-agent: probe assumed `agent-step` DOM rows accumulated across
pills (delta ≥ 2 vs. pre-pill baseline). The backend's `set_steps`
reducer is `last-write-wins` (per `gen_ui_agent.py`), so each pill
REPLACES `state.steps` and the DOM swaps. delta was always 0 after
pill 1, even when the new steps rendered correctly. Drop the
baseline-capture preFill, assert ≥ 2 visible rows + per-pill
fingerprint dedup. Tests rewritten to match the swap model.
* gen-ui-headless-complete: chip click used `button >> text="<title>"`
(Playwright chained selector). The button has a stable
`aria-label="Suggestion: ${title}"` attribute set inline; switch to
`button[aria-label="Suggestion: ${title}"]` so we match the moment
the DOM exists rather than waiting for inner text-node hydration.
* shared-state-streaming: fixture sent `{"content": "..."}` but the
langgraph-python `write_document(document: str)` tool expects
`document`. Mismatch meant `state.document` never updated; probe saw
no content delta. Rename the argument key to `document` in both the
per-feature fixture and `d5-all.json` (the runtime fixture). The
middleware's `tool_argument="document"` config requires this name.
* reasoning-display: fixture only emitted `content` (text response).
Probe asserted the v2 `<ReasoningBlock>` (rendered behind
`[data-testid="reasoning-block"]`), which mounts only when a
reasoning-role message lands in the transcript. Add a `reasoning`
field so aimock emits REASONING_MESSAGE_* events.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-feature fixtures (gen-ui-agent.json, shared-state-streaming.json) were updated
to JSON-stringified arguments + explicit tool-call IDs to match the OpenAI wire
shape, but d5-all.json — the runtime fixture mounted by aimock per its Dockerfile
— still had raw-object arguments without IDs. The wire-shape parity claim was
true for the per-feature files but contradicted by the live fixture; aligning
d5-all.json closes that drift so what we tested matches what runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Aimock fixtures match by userMessage substring, which collides across
integrations whose pills share the same prompt template. The validator
correctly catches that the d5 probe fixtures my PR added would dangle
on integrations whose tool surfaces don't include the emitted tool.
Fix by adding `match.toolName` constraints to the affected fixtures so
aimock only fires them when the agent emits that specific tool — and
the validator skips them for demos that don't declare the tool.
Also delete the obsolete `Visualize pitch, yaw, and roll` fixture
(F3 changed the d5-gen-ui-open-advanced probe to use the
`Inline expression evaluator` pill instead).
Locally `pnpm exec tsx validate-fixture-tool-surface.ts` returns
0 drift after this change.
Two small hygiene fixes:
- frontend-tools cosmic-gradient pill matched bare token 'navy' which
could collide with any prompt mentioning navy. Tighten to
'navy → magenta cosmic gradient' — verbatim substring of the
pill prompt in frontend-tools/suggestions.ts. Mirror into d5-all.json.
- d5-all.json header comment claimed the first 7 entries are the
open-gen-ui pills, but ordering shifts as features land (headless-
simple/complete pills currently sit ahead of open-gen-ui after
recent merges). Rewrite the header to describe the ordering
convention generically: high-priority verbatim-prompt fixtures
appear first, first-match-wins; per-fixture _comments and the
per-feature source files are the source of truth.
The render_a2ui matcher emitted the same Card+Metric payload for every
pill, so the d5-gen-ui-declarative probe went red on the second pill —
its per-pill expectedTestIds map demands declarative-pie-chart for the
pie-chart pill, declarative-bar-chart for the bar-chart pill, and
declarative-status-badge for the status-report pill, none of which
were rendered.
Branch the render_a2ui response by combining userMessage substring with
toolName so each pill emits the catalog component its probe expects:
- KPI dashboard → Card + 3 Metric children
- pie chart → PieChart with regional sales data
- bar chart → BarChart with quarterly revenue data
- status report → Card + 3 StatusBadge children
Keep the bare toolName-only matcher at the bottom as a deterministic
fallback so unforeseen pills still render something instead of erroring.
Mirror all five fixture entries into the bundled d5-all.json.
Two pill mismatches were silently routing the headless-complete Stock and
Highlight pills to the showcase-assistant catch-all in feature-parity.json:
- The Stock fixture matched 'AAPL trading' but the SuggestionBar pill
configured by use-headless-suggestions.ts sends 'What's the price of
AAPL right now?' — substring 'AAPL trading' is not in that prompt.
- The Highlight fixture matched 'Highlight \'meeting at 3pm\'' but neither
the empty-state pill ('Highlight: ship the demo on Friday') nor the
SuggestionBar pill ('Highlight this note for me: ...ship the demo on
Friday...') contains that substring.
Switch both matchers to short distinctive substrings ('AAPL' and
'ship the demo on Friday') that appear verbatim in BOTH the empty-state
and SuggestionBar prompts. The tool-rendering AAPL fixture (matcher
'What\'s the current price of AAPL?') stays at higher priority via
array order — first-match-wins keeps it pinned to the tool-rendering
pill, so substring 'AAPL' here only catches headless-complete pills.
Update narration response text to reference the correct highlighted
phrase.
The HELLO_LEADING phrase was the showcase-assistant catch-all
boilerplate ('I can help you with weather lookups...') that other
tests in this PR explicitly guard AGAINST. The dedicated d5-all.json
fixture for 'Say hello in one short sentence' now returns a distinct
non-boilerplate greeting; the spec asserts that distinct phrase, so a
fixture-priority misroute fails loudly instead of passing by accident.
Resolved d5-all.json conflict by appending B6's headless-simple/complete fixtures (Say hello, joke, fun fact, Highlight, chart) before the tool-rendering and open-gen-ui entries already on integration.
Adds per-pill aimock fixtures backing the Phase-2B genuine D5 probes:
- agent-config: 6 fixtures (3 knob pairs) so concise vs detailed
responses differ deterministically; satisfies the new probe's
text-diff + length-diff assertions.
- frontend-tools: 3 per-pill fixtures (sunset/forest/cosmic) emitting
change_background tool calls with family-specific gradient hexes.
- frontend-tools-async: query_notes tool call so the NotesCard mounts.
- gen-ui-agent: 3 per-pill set_steps tool calls with distinct step
content so the per-pill content-fingerprint assertion catches
fixture-drift.
- gen-ui-declarative: 4 per-pill generate_a2ui calls + render_a2ui
fixture for the secondary LLM call that paints the catalog.
- gen-ui-a2ui-fixed: SFO/JFK display_flight tool call.
- gen-ui-interrupt: 2 per-pill schedule_meeting tool calls with
distinct time-slot payloads.
- gen-ui-open: generateSandboxedUi tool call with a non-trivial
HTML payload so the iframe[srcdoc]-mount assertion has ≥ 100 chars
to observe.
- shared-state-streaming: 3 per-pill write_document tool calls with
substantive content payloads (≥ 100 chars each).
- readonly-state-context: pill-prompt fixture; the probe's network-
payload assertion checks the request body, not the response.
d5-all.json is updated with the new entries prepended so first-match
precedence routes specific pill prompts to their per-pill fixtures
ahead of the generic adjacent matches.
Add 14 fixtures at the top of d5-all.json so they win precedence over
the existing 'weather in Tokyo' / 'd5 beautiful-chat probe: search
flights' substring matches:
- Chain tools: 3 toolCalls in one assistant turn (get_weather Tokyo +
search_flights SFO->Tokyo + roll_d20=11) plus follow-up text.
- Weather in SF: get_weather(San Francisco) + text (deterministic).
- Find flights: dedicated search_flights(SFO,JFK) fixture, no longer
leaks the a2ui beautiful-chat shape.
- Stock price: get_stock_price(AAPL) returning $338.37 / -2.96%.
- Roll a 20-sided die: 5 sequential roll_d20 fixtures with
turnIndex 0..4 (stateless across runs), values [7,14,3,19,20].
Adds deterministic fixtures to d5-all.json (loaded before feature-parity.json
so it wins match precedence) for the Phase 1A genuine-pass cells:
- hitl-in-app refund #12345 and escalate #12347 pills: each emits a
request_user_approval tool call on the first turn, then branches via
sequenceIndex on the second turn (0 = approve response, 1 = reject
response). Test ordering (serial mode in the spec) drives which branch
is claimed first per pill pair.
- frontend-tools-async project-planning, auth, and reading pills: each
emits a query_notes tool call so the async client-side handler runs
against NOTES_DB and renders project-planning / auth / reading-tagged
matches. The reading pill's 2nd-turn narration locks the canonical
leading phrase per spec test #4.
Replaces nothing — purely additive. The dedicated pill-prompt entries
beat the broad 'plan' and showcase-assistant catch-all entries that were
silently winning before.
- Drop the 8 stale tests that asserted travel-planner shapes
(Current Itinerary / supervisor-indicator / .bg-gray-50). They
predated the supervisor + 3-subagent rewrite and could not catch
the 3 production bugs.
- New suite (5 tests, 0 skipped):
1) page loads with composer + 3 verbatim suggestion pills + 3
subagent role indicators (testid-based).
2-4) one test per pill (Write a blog post / Explain a topic /
Summarize a topic). Each clicks the pill, waits for all
three role-scoped subagent cards to reach data-status=
"complete", then asserts each card's subagent-result is
non-empty AND does not contain the showcase-assistant
boilerplate fragments. Test 4 also serves as a regression
gate on the delegations reducer fix — without it, that pill
returns HTTP 400 and the cards never reach complete.
5) clicks any pill, waits for terminal state, then asserts the
critic-card count is exactly 1 and stays at 1 with status
complete across a 5s dwell — catches any return of the
supervisor -> critic loop.
- Aimock fixtures: 3 verbatim pill chains (cold exposure training,
LLM tool calling, reusable rockets) added to both d5-all.json and
the harness source d5/mcp-subagents.json. Each chain drives the
full supervisor flow (turnIndex 0..3) plus three nested sub-agent
fixtures so every Researcher/Writer/Critic card surfaces real
prose instead of showcase boilerplate.
The 5 of 7 open-gen-ui pill prompts (3D axis, neural network, quicksort,
calculator, ping-the-host) currently match the broad userMessage: 'hi'
catch-all in feature-parity.json (substring match on 'hi' inside 'Say hi
to the host', 'this', etc.) and resolve to the showcase-assistant
boilerplate greeting instead of emitting the generateSandboxedUi tool
call. That kills the iframe path on 5/7 of the pills.
Add 7 high-priority verbatim-prompt fixtures in d5-all.json (and the
source-of-truth showcase/harness/fixtures/d5/gen-ui-open.json) that emit
deterministic generateSandboxedUi tool calls with inline HTML + CSS in
the tool result. d5-all.json loads BEFORE feature-parity.json so the
first-match-wins ordering naturally beats the catch-all. Pill fixtures
appear at the top of the file so they also win against intra-bundle
specifics. Inline HTML is intentionally minimal — the assertion bar is
iframe presence + non-empty srcdoc, not iframe DOM introspection
(cross-origin blocked under sandbox=allow-scripts only).
Alem Tuzlak