CI 'Showcase: Validate' failed: aimock-fixtures.test.ts asserts every d6/*.json
yields >=1 fixture, but gen-ui-open.json had been emptied to a _meta stub (0
fixtures) by an earlier open-gen-ui shadowing fix that consolidated the pills
into gen-ui-tool-based.json.
Restore gen-ui-open.json as a byte-for-byte context-swap of the canonical
langgraph-python file (context -> langgraph-fastapi): 4 fixtures for the '3D
axis' and 'Inline expression evaluator' pills, LGP's clean shape
(hasToolResult:false leg-1, distinct toolCallId terminals). No shadowing loop:
each file's leg-1/terminal pair is anchored by its own distinct toolCallId, so
the done-signal always resolves.
Verified: aimock-fixtures.test.ts 839/839 green locally; D6 gen-ui-open,
gen-ui-open-advanced, gen-ui-custom all green (two real runs each).
Two independent fastapi divergences from the green north-star kept this red:
1. Missing chunkSize:9999 on the 8 tool-call fixtures. Under aimock's global
8-byte chunking the large tool-call args failed to JSON-parse in one piece,
so the AAPL->MSFT reasoning chain stopped at leg 1 (turns 1 & 2). Byte-align
to LGP (and the green mastra/langgraph-typescript siblings carry the same
pattern).
2. Fixture-pool shadowing broke turn 3. aimock pools all d4+d6 fixtures for a
context and matches by userMessage substring, first-match-wins in load order
(d4 before d6). fastapi had broad keys LGP doesn't:
- d4/chat.json: 'weather' -> 'weather in San Francisco', 'flights from SFO
to JFK' -> 'Find flights from SFO to JFK.' (period-terminated so it stops
being a substring of turn 3's 'Find flights from SFO to JFK and show me
the weather there.').
- tool-rendering-{custom,default}-catchall.json: bare 'Find flights' ->
'Find flights from SFO to JFK.' (now matches LGP's exact key).
Also byte-align the backend agent to LGP: scriptable get_stock_price signature,
the detailed chaining system prompt, model gpt-5.4, reasoning summary detailed.
D6 tool-rendering-reasoning-chain green (two real ~21s runs). Shared-fixture
regression all green: agentic-chat, tool-rendering, tool-rendering-custom-catchall,
tool-rendering-default-catchall, headless-complete (Tokyo-weather consumer).
Two compounding defects kept this cell red:
1. Backend missing get_revenue_chart. headless_complete.py never registered
the revenue-chart tool, so the chart-turn never emitted a tool call and the
card never mounted. Port LGP's get_revenue_chart tool + its system-prompt
routing rule (tool return shape byte-identical to LGP).
2. Stale turnIndex fixtures caused turn-2 and turn-4 loops to the recursion
limit. Align to LGP's canonical shape:
- tool-rendering.json (shared): AAPL first-leg turnIndex:0 -> hasToolResult:false
so it stops re-firing at turnIndex>=2 in the multi-pill thread.
- headless-complete.json: rewrite to LGP's structure (narration/toolCallId
fixtures first, tool-call legs after with userMessage+context only, no
stale turnIndex); drop the divergent fastapi-only highlight fixtures,
subsumed by the broader substring fixture.
D6 langgraph-fastapi:gen-ui-headless-complete now green (two real ~29s runs);
tool-rendering re-verified green (shared-fixture no regression).
gen-ui-agent had no dedicated backend graph: langgraph.json lacked a
gen_ui_agent entry and route.ts routed the name through the neutral-assistant
loop to sample_agent, which has no steps state or set_steps tool, so the
progress card never mounted (agent hit the default recursion limit of 25).
- Port LGP's gen_ui_agent.py (byte-identical) and register it in langgraph.json.
- route.ts: bind gen-ui-agent to createAgent("gen_ui_agent") and bake
assistantConfig.recursion_limit (default 100) into every LangGraphAgent —
the graph's Python with_config isn't visible to the server runs API, so the
multi-step set_steps walk overran 25. Mirrors langgraph-python.
- aimock: regenerate d6/gen-ui-agent.json from LGP (adds chunkSize:9999 on the
24 tool-call fixtures) and narrow the over-broad d4 chat.json "summarize" key
to "Summarize the sales pipeline" so it stops substring-shadowing the
competitor set_steps chain (and other summarize prompts). Matches LGP.
D6 langgraph-fastapi:gen-ui-agent now green (two real ~14s runs);
agent-config re-verified green (no regression).
open-gen-ui and open-gen-ui-advanced (both probes load gen-ui-open.json) hung
with done-signal-missing (~12 agent runs, 1.3MB tool-call payload each turn).
Cause: two files (gen-ui-open.json + gen-ui-tool-based.json) both defined leg-1
generateSandboxedUi tool-call fixtures for the SAME userMessages ('3D axis…',
'Inline expression evaluator') under the same context but different toolCallIds.
aimock pools every d6/<slug>/*.json into one substring-match set, and the
turnIndex:0 fixture in gen-ui-tool-based.json got served on follow-up turns too
(turnIndex-relaxed), so the toolCallId-anchored terminal never won → no
finish_reason:stop → loop.
Fix (fastapi fixtures only, aligned to GOTCHAS 'canonical fixtures have no
turnIndex' + toolCallId-terminal-before-userMessage-leg1):
- gen-ui-tool-based.json: strip spurious turnIndex:0 from leg-1 tool-call
fixtures; add chunkSize:9999 so the large args JSON-parse in one chunk. Now
the single source for all open-gen-ui pills.
- gen-ui-open.json: empty its fixtures (keep a documenting _note) to remove the
duplicate leg-1/terminal for the 2 probed pills.
- gen-ui-open-advanced.json: delete orphan (no probe declares it; both open
probes use gen-ui-open.json).
D6 green + stable (two ~4s runs each); regression-checked gen-ui-custom
(gen-ui-tool-based route) still green. NOTE: LGP + likely all integrations have
the same open-gen-ui loop (fleet-wide fixture-shadow bug) — no sibling was green.
fixture[8] (userMessage:"project planning", turnIndex:0) returned a
query_notes toolCall. aimock matches userMessage by substring, so
"project planning" also matched the test prompt "Find my notes about
project planning"; as a turnIndex:0 'scripted turn' it beat the correct
terminal fixture[2] (toolCallId-keyed, content-only) on the follow-up turn,
so the agent kept getting another tool call and looped ~30x, never emitting
the done signal (D6 done-signal-missing).
Remove turnIndex:0 from fixture[8] so it's a plain fallback that loses to
the terminal fixture by registration order (matching the langgraph-typescript
shape). D6 now green: two real ~6s runs (was a 120s timeout).
Note: langgraph-python has the identical bug (still red) — same one-line fix
applies there; tracked separately.
Flip the 4 TS a2ui builders (shared/typescript + mastra,
claude-sdk-typescript, langgraph-typescript) from the legacy flat
operation shape to v0.9 nested (createSurface / updateComponents /
updateDataModel), matching the Python builder and what A2UI consumers
process. Flat ops were never processed as valid nested operations, so
the surface schema and components were never applied.
Also align the empty-data guard to Python's `if data:` semantics (empty
object -> no updateDataModel), add a v0.9 parity guard test to all 4
test files, and add 12 gen-ui-a2ui-fixed aimock fixtures.
Commit 7c3edca changed sample-attachment-buttons.tsx across all integrations
to auto-send via agent.addMessage with autoPrompt strings:
- "can you tell me what is in this demo image I just attached"
- "can you tell me what is in this demo pdf I just attached"
But the d5 harness fixture and all 19 d6 per-integration multimodal.json
fixtures still matched on the old strings:
- "describe the sample image"
- "summarize the sample document"
Aimock received requests with the new prompts, found no match, returned
a STRICT 404, and the agent emitted a streaming error back to the UI
(exact symptom: "An internal error has occurred while streaming events").
Also update agentic-chat.json across all 20 integrations (those files had
duplicate fallback entries for the old prompts) and fix split-fixtures.ts
to route the new strings to the "multimodal" feature bucket.
Local RED: ms-agent-python and crewai-crews both fail with fixture-miss
status=miss before this change.
Local GREEN: langgraph-typescript passes after this change (both turns
settle with "image" / "document" keywords confirmed in transcript).
Remaining failures after this fix are pre-existing Python backend issues
(ChatClientException on binary content parts in ms-agent-python; CrewAI
flow failure on binary content in crewai-crews) — unrelated to fixture
keys and tracked separately in the pydantic-ai multimodal work.
Port the google-adk a2ui-recovery demo to langgraph (python, fastapi,
typescript) and aws-strands (python, typescript). Each ships a dedicated
recovery agent, route, demo page/chat/suggestions, manifest entry, aimock
d6 fixtures, e2e spec, and QA doc.
Backend-owned recovery on langgraph via get_a2ui_tools / getA2UITools
(injectA2UITool=false); auto-inject recovery on the strands adapter path.
Heal stages an invalid-then-valid render via aimock sequenceIndex (the
toolkit validate->retry loop rejects the whole surface, so a single-pass
parse_and_fix heal is ADK-specific and does not apply here). Recovery
prompts are unique per framework and the fixtures carry no context match
field, so they fire for real browser (dojo) traffic, not just the harness.
Also harden the strands declarative-gen-ui composition guide to name the
exact catalog component (Metric, not MetricTile) and update the
generate-catalog + aimock-fixtures test expectations.
ag2/claude-sdk-typescript/google-adk/langgraph-fastapi/langgraph-typescript/mastra/ms-agent-dotnet: AAPL is turn-1 so turnIndex:0 fallback unreachable; toolName gate fails on wildcard-renderer integrations that don't register get_stock_price. Aligned to LGP-gold pattern (userMessage+context discriminator, no toolName, no turnIndex:0 fallback). Preserved legitimate multi-pill matchers (SF/flights/d20/chain) on the 4 multi-pill integrations.
(cherry picked from commit c10821b5d904b31bee2ab2a39db3b1565aecba26)
Closed PR #5465 introduced cross-fixture leakage by stripping the
toolName discriminator on shared {userMessage, context} keys: with
aimock's alphabetical first-match-wins ordering,
tool-rendering-custom-catchall.json sorts before
tool-rendering-default-catchall.json, so default-catchall page requests
were served the custom file's content. The probe was structurally blind
because it asserted only DOM testids (copilot-tool-render +
data-tool-name=get_weather) — both fixtures emit get_weather, so the
testid signal passed regardless of which fixture won.
Real LGP-gold pattern is disjoint userMessages between default and
custom catchall fixtures (NOT shared keys discriminated by toolName).
This PR ports the LGP-gold pattern to the other 17 integrations and
adds page-text content assertions to both d5 catchall probes so this
class of regression can't recur silently:
- Probe prompts disjoint between default-catchall and custom-catchall
- Fixture userMessages updated to match the new disjoint prompts
- Page-text content assertions in both d5 catchall probes
(default negatively asserts the custom-catchall leak phrase;
custom positively asserts it)
Local gold-standard red-green proof captured:
- Step A: live staging Playwright baseline (RED-OF-RECORD)
- Step B: local control-plane on origin/main reproduces structural
fragility (probes pass at testid level, custom fixture wins on
default's userMessage path)
- Step C: local control-plane on this branch — both catchall probes
GREEN with the new content-asserting assertions
tool-rendering-default-catchall: pass=true (5443ms)
tool-rendering-custom-catchall: pass=true (8956ms)
cross-tool signature passed
LGP (langgraph-python) was already disjoint; it remains untouched.
The catchall userMessage rename (#5453) only renamed STALE strings to 'forecast for Tokyo'. Integrations whose catchall fixtures used PILL-ALIGNED userMessages ('What's the weather in San Francisco?', 'Find flights from SFO to JFK.', 'Chain a few tools in this single turn', etc.) had no 'forecast for Tokyo' matcher to begin with — so the D5 catchall probe still fell through to live LLM and the cells stayed red on staging.
This PR ADDS the canonical 'forecast for Tokyo' (default+custom catchall) and 'What's the current price of AAPL?' (custom catchall only) emit+narrate fixture pairs to each integration that was missing them. Existing pill-aligned fixtures are preserved (pure prepend at the start of the fixtures array; first-match-wins means new fixtures match the D5 probe inputs without colliding with existing pill prompts).
Integrations touched:
- langgraph-typescript: both default + custom
- langgraph-fastapi: both default + custom
- google-adk: custom only (default already green)
- ms-agent-dotnet: custom only (default already green)
After merge, fleet-cp's e2e-deep probe will re-run within ≤6h staleness window and flip cells GREEN.
- SUPERSEDED annotations on unreachable recorded calculator entries (load-order shadowing is
the only guarantee; model gate is not a safety net)
- GOTCHAS sequenceIndex rewritten to per-X-Test-Id semantics with co-increment/eviction caveats
- hasToolResult paragraph corrected (omission = no gate, thread-global predicate)
- statelessness claim reconciled with sequence counters
- replace Websandbox host-bridge evaluateExpression (never registered in beautiful-chat) with
in-sandbox allowlist-gated Function() eval
- reorder specific calculator pair before generic (first-match-wins)
- allow exponential notation (eE) so chained evaluation of large results works
- drop thread-global hasToolResult gate that broke the pill after other tool-producing pills,
reorder toolCallId follow-ups before leg-1
- mint distinct tool_call_ids per repeat click via sequenceIndex variants + non-sequenced
fallback (fixes second-widget collapse)
- restore google-adk trailing newline
- document ordering invariants, gate tradeoffs, and sequence-counter scoping caveats in
fixture comments
Verified via local Playwright red-green (render, 7+8=15, chained exponent, interleaved pills,
repeat clicks).
The declarative-gen-ui demo across the three langgraph integrations now
relies on the middleware to inject and execute generate_a2ui — the agents
collapse to create_agent + CopilotKitMiddleware with no hand-rolled tool.
Adds render_a2ui fixtures for the new tool path and pins the integrations
to the A2UI alpha SDKs (copilotkit 0.1.94a1, @copilotkit/sdk-js 1.59.3-alpha.1).
The gen-ui-headless-complete probe
(showcase/harness/src/probes/scripts/d5-gen-ui-headless-complete.ts)
references fixtureFile "gen-ui-headless-complete.json", but that file
existed in no D6 slug, so the probe's first leg 503'd under strict.
Add gen-ui-headless-complete.json for all 18 D6 slugs, modeled on the
green langgraph-python headless-complete.json pattern: the four gen-UI
pills (weather/stock/highlight/revenue) with narration (toolCallId)
fixtures FIRST and toolcall (userMessage+context) fixtures AFTER, no
turnIndex gate, so every one of the probe's four sequential turns in one
chat thread matches regardless of prior assistant/tool history. Interrupt
fixtures are intentionally omitted (interrupt-headless is a separate
cluster item).
These 8 fixtures per slug share match keys with the pre-existing
headless-complete.json for the same context (the demos share pills and
are disambiguated at runtime by probe path), which raises the
aimock-fixtures collision-detection exact-duplicate count by 46. Bump
KNOWN_DUPLICATE_CEILING 230 -> 276 to match, consistent with how prior
per-integration feature fixtures bumped the baseline; the substring-shadow
ceiling is unchanged (no new shadows).
Validated: all 732 aimock-fixtures schema/collision tests pass, and a
local aimock --strict run returns 200 for each of the four pills across
turns (langgraph-python + pydantic-ai spearheads, plus spring-ai).
D6 drives each feature's pills as sequential turns in one chat thread.
aimock matches turnIndex against the count of assistant messages in
history, so a "turnIndex": 0 gate on a per-pill tool-call leg only
matches the FIRST pill — pills 2+ (turnIndex 1/2/3) match no fixture,
503 under strict, and the harness reports "timeout: assistant did not
respond".
Remove the turnIndex:0 gate from the multi-turn tool-call legs of
frontend-tools (sunset/forest/cosmic) and tool-rendering-reasoning-chain
(the three chained pills: Compare AAPL/MSFT, compare-to-smaller dice,
weather-there flights) so they match on their distinct userMessage
substrings + context, mirroring the already-green langgraph-python
fixtures. Each pill's substring is unambiguous and its toolCallId-keyed
follow-up fixture is ordered before the de-gated first leg (first-match
wins), so follow-up narration still wins and there is no cross-matching.
The reasoning-chain "Find flights from SFO to JFK." pill keeps its
turnIndex:0 gate to match the green langgraph-python reference exactly.
Proven locally with aimock --strict: multi-turn requests for these pills
returned 503 with the gate (negative control) and now return 200 across
all turns with the correct fixture/narration.
The aimock matcher gates turnIndex against the request's assistant-message
count (assistantCount !== turnIndex → skip). The agno, spring-ai, and
langgraph-fastapi agentic-chat fixtures baked turnIndex:0 on all three
goldfish conversation turns. Turn 2+ carries >=1 assistant message, so
turnIndex:0 could never match those turns — aimock returned no-match and the
request fell through to proxy (503), failing the d6 cell.
Mirror the canonical clean pattern used by the other 15 frameworks
(langgraph-python et al.): omit turnIndex on the multi-turn conversation
turns and disambiguate purely by userMessage. Single-turn fixtures in the
same files keep turnIndex:0 (unchanged). Verified locally against the aimock
matcher: turn-2 goes 404→200 for all three frameworks, turn-1 unregressed.
Adds per-integration D6 fixtures for langgraph-python, langgraph-typescript,
and langgraph-fastapi. Each fixture is keyed by match.context for cross-
integration isolation and uses hasToolResult:false on toolCall responses
to prevent re-match loops.
Move monolithic d5-all.json + feature-parity.json + smoke.json into
per-integration directories under d4/<slug>/, d6/<slug>/, and shared/.
Every fixture file is now context-scoped to enable server-side aimock
routing via match.context. Migrated 12 HITL fixtures from main's
d5-all.json additions into shared/_migrated-from-d5-all-hitl.json
for follow-up distribution into per-integration files.
Mark