AG2's ConversableAgent runs every user message through
``autogen.code_utils.content_str``, which only accepts content-part
types in {"text", "input_text", "image_url", "input_image", "function",
"tool_call", "tool_calls"}. CopilotChat / the AG-UI runtime emits image
and document attachments as the modern shape
{"type": "image" | "document", "source": {...}}
and the demo page's legacy-converter-shim.tsx ALSO appends a legacy
{"type": "binary", mimeType, data | url}
mirror alongside it (to keep the @ag-ui/langgraph converter happy on
LangChain-based integrations — it rides through on the ag2 path too).
Both shapes trip autogen's allowed-types gate with
ValueError("Wrong content format: unknown type image within the
content")
…BEFORE the request reaches the vision model — observed live in the
D6 multimodal probe (commit d8a0a25db, which originally quarantined
the feature as NSF).
Fix
---
Add ``agents/_multimodal_normalize.py``: a ``NormalizingAGUIStream``
subclass of ``AGUIStream`` that overrides ``dispatch()`` to normalize
AG-UI image/document/binary content parts to OpenAI Chat Completions
``image_url`` parts AFTER ``RunAgentInput`` Pydantic parsing and BEFORE
``AgentService`` serialises the messages for autogen.
This is the only correct interception point:
- Too early (ASGI body rewrite before Pydantic): ``RunAgentInput``
rejects ``image_url`` because it is not an AG-UI standard type —
the discriminated union only accepts image/document/binary/text.
- Too late (inside ConversableAgent): requires patching autogen
internals.
The override works by calling ``normalize_messages_for_autogen()`` on
the dict-serialised messages (same form as ``run_stream`` produces via
``model_dump()``) and re-injecting them via a ``_PatchedRunAgentInput``
wrapper that overrides only ``.messages``, delegating all other
attribute access to the original ``RunAgentInput``.
Conversions:
- {"type": "image", "source": {"type": "data", value, mime_type}} →
{"type": "image_url", "image_url": {"url": "data:<mime>;base64,<value>"}}
- {"type": "image", "source": {"type": "url", value}} →
{"type": "image_url", "image_url": {"url": value}}
- {"type": "document", "source": ...} → image_url with the document's
mime preserved (data:application/pdf;base64,...). The vision model
still can't natively read PDFs, but the request reaches the model
instead of being rejected upstream, which is the failure mode this
fix targets.
- {"type": "binary", mimeType, data | url} → image_url (the
legacy-shim parts ride through cleanly).
- {"type": "text", ...} and already-normalised image_url parts pass
through unchanged (identity-preserved on no-op turns).
Failure path: any normalization error is logged at WARNING and the
original messages are forwarded unchanged — autogen's own ValueError
fires verbatim with its error surface intact.
Manifest + fixture
------------------
- showcase/integrations/ag2/manifest.yaml: remove multimodal from
not_supported_features (with its now-stale comment) and add it back
to the features list next to voice.
- showcase/aimock/d6/ag2/multimodal.json: add the D6 fixture pair
using the actual autoPrompt strings from sample-attachment-buttons.tsx
("can you tell me what is in this demo image I just attached" /
"can you tell me what is in this demo pdf I just attached").
TDD evidence (red-green)
------------------------
showcase/integrations/ag2/tests/python/test_multimodal_normalize.py
contains 14 unit tests, pinned at three layers:
1. RED/GREEN against autogen's actual content gate:
* test_autogen_rejects_raw_agui_image_part — confirms
content_str([{type: image, source: ...}]) raises the verbatim
ValueError the D6 probe surfaced. This is the regression pin: if
autogen ever relaxes the gate, this test fails and we know to
revisit the normalizer.
* test_normalized_content_is_accepted_by_autogen — after
normalize_messages_for_autogen(...), content_str accepts every
part and renders "<image>" for the image_url part.
2. Shape coverage: modern image data/url, modern document, legacy
binary data/url, mimeType camelCase alias, plain-text passthrough,
plain-string content, assistant/tool messages untouched,
unrecognised source → text placeholder, idempotency.
3. NormalizingAGUIStream class surface tripwire.
Control-plane D6 RED→GREEN:
RED (no normalizer, pre-fix container): d6:ag2/multimodal → red
(HTTP 500 agent_run_error_event from content_str ValueError)
GREEN (NormalizingAGUIStream applied): d6:ag2/multimodal → green
## Summary
- Productizes the Claude SDK Python and TypeScript showcase demos with
LangGraph-parity frontends.
- Wires the Claude demo backends through the official Claude Agent
SDK/AG-UI adapter paths using `claude-sonnet-4.6`.
- Keeps Claude integration docs hidden for this PR and excludes
generated/authored docs artifacts from scope.
## Why
The goal is to bring the productized LangGraph demo surface to Claude
Agents SDKs without publishing integration docs in this pass. This keeps
the PR focused on local showcase demos, runtime behavior, fixtures, and
validation support.
## How
- Ported the demo frontend surfaces and local shell-dojo support for
Claude SDK Python/TypeScript.
- Added official Claude SDK adapter/backend wiring plus real-Claude
local compose support.
- Updated Claude aimock fixtures and validation ratchets for the
expanded demo set.
- Set both Claude manifests to `docs_mode: hidden` and removed docs
setup/snippet artifacts from the PR scope.
Commit 7c3edca changed sample-attachment-buttons.tsx across all integrations
to auto-send via agent.addMessage with autoPrompt strings:
- "can you tell me what is in this demo image I just attached"
- "can you tell me what is in this demo pdf I just attached"
But the d5 harness fixture and all 19 d6 per-integration multimodal.json
fixtures still matched on the old strings:
- "describe the sample image"
- "summarize the sample document"
Aimock received requests with the new prompts, found no match, returned
a STRICT 404, and the agent emitted a streaming error back to the UI
(exact symptom: "An internal error has occurred while streaming events").
Also update agentic-chat.json across all 20 integrations (those files had
duplicate fallback entries for the old prompts) and fix split-fixtures.ts
to route the new strings to the "multimodal" feature bucket.
Local RED: ms-agent-python and crewai-crews both fail with fixture-miss
status=miss before this change.
Local GREEN: langgraph-typescript passes after this change (both turns
settle with "image" / "document" keywords confirmed in transcript).
Remaining failures after this fix are pre-existing Python backend issues
(ChatClientException on binary content parts in ms-agent-python; CrewAI
flow failure on binary content in crewai-crews) — unrelated to fixture
keys and tracked separately in the pydantic-ai multimodal work.
The bare-substring 'weather' fixture in aimock/d4/llamaindex/chat.json emitted a
get_weather tool call with no toolName gate. The tool-free voice agent's prompt
"What is the weather in Tokyo?" (substring "weather") leaked into this fixture,
emitting a get_weather call the voice agent could never resolve, so the voice D6
cell hung (done-signal-missing, body stuck on get_weather/Running).
Add toolName:"get_weather" so the fixture only fires when the requesting agent
actually registers get_weather (mirrors the gate in d6 tool-rendering.json). The
tool-free voice request now falls through to voice.json's exact content match.
Local red->green proof (showcase test llamaindex:voice --d6 --direct):
- RED: done-signal-missing; body "What is the weather in Tokyo? get_weather Running"
- GREEN: assistant settled "The weather in Tokyo is currently 22C with partly
cloudy skies and light easterly winds."; 1 passed (3.0s)
Direct aimock probes confirm the gate: tool-free -> content; with get_weather tool
-> tool call still fires. Regression: tool-rendering D6 still green; headless-complete
weather turn still passes (uses its own gen-ui-headless-complete.json fixture).
(cherry picked from commit cf6ff7c08153367239437d6c4fff425d546eb245)
The bare 'summarize' userMessage in d4/llamaindex/chat.json substring-matched
the D6 gen-ui-agent pill 'Research our top competitor and summarize their
strengths and weaknesses.', returning the sales-pipeline text fixture instead
of the gen-ui-agent set_steps tool call. The competitor pill then produced
no/duplicate steps, failing d6:llamaindex. Narrow the match to the verbatim D4
toolbar probe 'Summarize the sales pipeline' (langgraph-python parity), which
no demo pill contains as a substring. D4 llamaindex stays green (the bare entry
was unused by any D4 cell).
(cherry picked from commit c23801c8b32d292cacf6fb2e7e2a68270eebaa84)
The _note asserted injectA2UITool:false (unchanged) and that flipping to true
would blank-render, and that generate_a2ui returns an a2ui_operations container
for the middleware to forward. Both are now false: this PR set injectA2UITool:true,
generate_a2ui returns raw planner args, and the surface mounts from a streamed
render_a2ui tool-call (START/ARGS/END) the agent re-emits, which the middleware
watches under injectA2UITool:true. Prose-only; no match keys or payloads changed.
(cherry picked from commit 5679b001580615f2e7d988d8c7063994076ace29)
Port the google-adk a2ui-recovery demo to langgraph (python, fastapi,
typescript) and aws-strands (python, typescript). Each ships a dedicated
recovery agent, route, demo page/chat/suggestions, manifest entry, aimock
d6 fixtures, e2e spec, and QA doc.
Backend-owned recovery on langgraph via get_a2ui_tools / getA2UITools
(injectA2UITool=false); auto-inject recovery on the strands adapter path.
Heal stages an invalid-then-valid render via aimock sequenceIndex (the
toolkit validate->retry loop rejects the whole surface, so a single-pass
parse_and_fix heal is ADK-specific and does not apply here). Recovery
prompts are unique per framework and the fixtures carry no context match
field, so they fire for real browser (dojo) traffic, not just the harness.
Also harden the strands declarative-gen-ui composition guide to name the
exact catalog component (Metric, not MetricTile) and update the
generate-catalog + aimock-fixtures test expectations.
The secondary-LLM prompt was far thinner than the canonical generation guidelines,
so it emitted trees that (correctly) failed the renderer's paint gate → surface-missing.
Port the canonical generation rules into the prompt, add output validation, add catalog
parity (DataTable + info-row), ground the planner with sales-context, and record
multi-turn aimock fixtures. Includes CR fixes: two-arg z.record for the DataTable rows
schema (zod@4 API), index-based DataTable row key, and Metric trendValue rendering for
neutral trend.
## Summary
`showcase/aimock/RAILWAY.md` described the aimock fixture infrastructure
as it worked during an earlier migration phase, not as it works today.
The stale claims actively misled debugging. This PR corrects them to
match the real, verified infra.
## Before → After
| Claim | Before (stale) | After (correct) |
| --- | --- | --- |
| Fixture delivery (§4/§5) | Three fixtures fetched at boot from
GitHub-raw URLs: `d5-all.json`, `smoke.json`, `feature-parity.json` |
Fixtures are **baked into the image** at build time under
`/fixtures/{shared,d4,d6}`; no remote fetch |
| `d5-all.json` | Authoritative D5 bundle, re-bundled from
`harness/fixtures/d5/*` | **Gone** — a one-time migration source split
into the per-slug `showcase/aimock/d6/<slug>/` tree (the source of
truth) |
| startCommand `--fixtures` (§5) | 3 `raw.githubusercontent.com` URLs |
`--fixtures /fixtures/shared --fixtures /fixtures/d4 --fixtures
/fixtures/d6` (the baked-in dirs). Added a warning that a single
`--fixtures /fixtures` parent loads nothing |
| Runtime image (§3) | Bare `ghcr.io/copilotkit/aimock:<version>` pulled
directly | `showcase-aimock` image, `FROM
ghcr.io/copilotkit/aimock:latest` + baked fixtures, built by
`showcase_build.yml` |
| Dockerfile (§8) | "Dead code / legacy wrapper, safe to remove" |
**LIVE** — `showcase_build.yml` builds it (`dockerfile:
showcase/aimock/Dockerfile`) and it bakes `shared/`, `d4/`, `d6/` into
`/fixtures/`. Do not remove |
| Env vars (§6), references (§9) | Referenced public GitHub-raw fixture
URLs and ~5 min raw-edge cache propagation | Local baked dirs;
propagation is via image rebuild + Railway deploy |
Also documents the working Railway mutation auth path by mechanism only
(§2): account-scoped `RAILWAY_TOKEN` from the DevOps `showcase`
1Password item via GraphQL `Authorization: Bearer` — the CLI session
token is not authorized for showcase-project mutations. No token value
is in the doc.
## Verification
- `showcase/aimock/d5-all.json` does not exist; the tree is `shared/`,
`d4/`, `d6/`.
- `showcase/aimock/Dockerfile` is `FROM
ghcr.io/copilotkit/aimock:latest` + `COPY shared|d4|d6 ->
/fixtures/...`.
- `.github/workflows/showcase_build.yml` matrix entry `showcase-aimock`
builds `dockerfile: showcase/aimock/Dockerfile`, context
`showcase/aimock`.
Docs-only change. Do not merge / do not enable auto-merge.
PR #5661 mis-templated this fixture from ms-agent-dotnet: the inner
secondary-LLM blocks used toolName "_design_a2ui_surface" (never matches
pydantic-ai, whose inner tool is render_a2ui) and only the sales-dashboard
pill had any block, so the other 3 declarative pills matched nothing and
the cell 503'd with no_fixture_match on turn 1.
Re-author 1:1 from the canonical langgraph-python fixture (identical inner
tool render_a2ui + per-pill surfaces), adapted for pydantic-ai: context
"pydantic-ai" on the outer generate_a2ui + narration entries, inner
render_a2ui entries matched by toolName alone (the agent's inner OpenAI()
client does not forward x-aimock-context). All 4 pills (sales-dashboard,
team-performance, at-risk, top-account) now get both an outer
(generate_a2ui) and inner (render_a2ui) block whose component payloads meet
each pill's probe assertion. Dead KPI/pie/bar/status blocks removed.
Real-probe proof: with this fixture the inner render_a2ui call matches and
the outer narration renders ("Here's your Q2 sales dashboard.") — the cell
advances from "Strict mode: no fixture matched / 503" to all LLM calls
matched. Remaining surface-missing failure is a non-fixture frontend/agent
A2UI delivery gap (see PR description), out of scope for this fixture fix.
The showcase-aimock RAILWAY.md described fixtures as fetched from GitHub-raw
URLs at boot and called the Dockerfile dead code. Both are false and actively
misled debugging: fixtures are baked into the image at build time under
/fixtures/{shared,d4,d6}, the Dockerfile is the live image builder driven by
showcase_build.yml, and the d5-all.json bundle no longer exists (split into
the per-slug d6/ tree). Corrects sections 3, 4, 5, 6, 7, 8, 9 to match the
real infra, fixes the startCommand to load the three baked-in subdirectories,
and documents the account-scoped RAILWAY_TOKEN mutation path by mechanism.
## Summary
- The pydantic-ai `generate_a2ui` declarative D6 turn was missing an
aimock fixture, producing HTTP 503 `no_fixture_match` on staging
(pydantic-ai 503 vs ms-agent-dotnet 200 for the same turn) — a source of
dashboard flapping.
- Adds the canonical mirror fixtures (outer `generate_a2ui` + matching
inner `_design_a2ui_surface`) to
`showcase/aimock/d6/pydantic-ai/gen-ui-declarative.json`. These are
deterministic canonical mirrors matching the langgraph-python convention
— **not** a non-deterministic real-LLM recording — preserving the
mandatory LGP 1:1 parity.
## Red-green proof
- **RED:** exact failing request (`POST /v1/responses`, gpt-4.1, "Show
me my sales dashboard for this quarter.", tools=[`generate_a2ui`],
header `x-aimock-context: pydantic-ai`, strict) against the pre-fix
fixture set → **HTTP 503 `no_fixture_match`** (reproduces staging
exactly; confirmed live on staging too).
- **GREEN:** same request against the new set → **HTTP 200** SSE
emitting the `generate_a2ui` tool call; the inner `_design_a2ui_surface`
turn also returns 200 with the dashboard surface.
- Independently re-verified. `validate-on-load` clean (no fixture
shadowing); existing pydantic-ai D6 turns (KPI/pie/bar/status) still
match identically — no regression.
## Notes
- No credentials in the committed fixture — the OpenAI key was never
even resolved (canonical mirror, not a recording). Credential scan of
the diff + full blob: zero matches.
## Test plan
- [ ] CI green
- [ ] After deploy, confirm the `generate_a2ui` declarative D6 cell
flips red→green on staging
Brings google-adk to 39/39 D6 (reproduced across two independent full-matrix
runs, zero regressions). Four changes:
- entrypoint.sh: remove ADK_DISABLE_PROGRESSIVE_SSE_STREAMING=1. That flag's
non-progressive aggregation path ended ADK's agentic loop after the first
tool round (no post-tool LLM re-invoke), which broke every demo needing a
second turn: the subagents chain (research -> writing -> critique),
tool-rendering-reasoning-chain (AAPL -> MSFT), shared-state-read-write's
confirmation, and the custom-catchall narration. The partial-event abort it
guarded against is already handled in-callback by stop_on_terminal_text.
- manifest.yaml: un-skip-list tool-rendering-reasoning-chain (now passes with
the loop restored).
- headless_complete_agent.py: add AGUIToolset() so the frontend highlight_note
tool is injected and routed to the browser. Removing the flag unmasked this
pre-existing gap — turn 3 dispatched highlight_note server-side and the
backend registry rejected it. langgraph-python auto-injects frontend tools;
ADK needs AGUIToolset() in the agent's tools list.
- aimock/d6/google-adk/gen-ui-interrupt.json: order each pill's narration leg
(toolCallId) before its emit leg and drop the thread-global hasToolResult
gate, so the alice pill no longer 503s after the sales pill leaves a tool
result in the thread.
Takes d6:strands and d6:strands-typescript from 32/35 to 34/35.
- shared-state-read: the turn-2 fixture leg wrongly pinned turnIndex:0, so the
aimock matcher skipped it on turn 2 -> 404 -> turn-2 sse-missing. Drop
turnIndex to mirror the langgraph-python gold-standard fixture.
- multimodal: sample.png/pdf/wav shipped as git-LFS pointers, so deploy/test
environments without 'git lfs pull' served the ~130-byte pointer text as the
upload -> the run never started (runsFinished=0). Ship them as regular
binaries via a per-integration .gitattributes lfs-unset + real bytes,
mirroring langgraph-python's convention.
Remaining red (gen-ui-declarative) is a Strands A2UI-dynamic run-completion bug
(reproduces on real-LLM staging too): the surface paints but generate_a2ui
never completes, so the run hangs 'Running'. Tracked separately.
Switch declarative-gen-ui to runtime auto-injection (injectA2UITool: true, plain
agent), matching the langgraph-python / AWS Strands gold standard. The ag-ui-adk
0.7.0 adapter auto-injects generate_a2ui via plan_a2ui_injection. Verified under
the canonical D6 driver: all four declarative pills pass (sales-dashboard,
team-performance, at-risk, top-account), plus a2ui-fixed-schema and
beautiful-chat.
Remove the stale known-failing mark from the a2ui-recovery heal e2e and the
OSS-374 inner-render-disambiguation notes (fixture _meta, QA doc, agent
docstring, route comment). Verified against the aimock journal: ag-ui-adk 0.7.0
forwards the run conversation into the inner render_a2ui call, so each pill
matches its own fixture by userMessage and both heal and exhaust pass. The
recovery demo stays backend-owned (the only path with the recovery loop) and
ADK-only (OSS-375 tracks langgraph-python parity).
Replace the "Tracked in Linear" placeholders with the filed issue IDs:
OSS-374 (showcase-aimock inner-subagent disambiguation) and OSS-375
(recovery-demo langgraph-python parity).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `heal` e2e test fails against the live showcase aimock because the harness
cannot disambiguate the two pills' inner render_a2ui sub-agent calls (the
backend-owned middleware issues the inner call with a generic render prompt +
shared suggestion context, so the "last user turn" aimock keys on is not the
pill prompt). Both pills match the same inner fixture (the EXHAUST one, by
first-match order), so the heal pill exhausts instead of healing.
This is an aimock harness limitation, not a middleware/demo bug: the middleware
heals free-form args correctly in the OSS-158 toolkit gate and against real
Gemini. Keep the test running (not skipped) as a deliberate demonstration for
the Showcase team; it does not red CI (these specs aren't run for google-adk in
CI). Update the spec header, the heal-test note, the fixture _note, the qa doc,
and the recovery_agent docstring to reflect this accurately (and drop the stale
sequenceIndex description). Tracked in Linear: aimock inner-subagent
disambiguation + recovery-demo LP parity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The heal fixture relied on per-attempt aimock response switching (sequenceIndex
0=invalid -> 1=valid), but sequenceIndex doesn't advance across the recovery
loop's intra-turn render_a2ui calls in the per-integration Playwright context
(no X-Test-Id), so aimock served invalid on every attempt and the demo never
healed (0 metrics).
Switch the heal pill to a single deterministic response: the inner render_a2ui
returns FREE-FORM args (components/data as JSON strings) that the middleware
heals via parse_and_fix into a valid surface in one pass — no per-attempt
switching. Verified in-sandbox: free-form args -> a2ui_operations with 2 Metric
components. The exhaust pill is unchanged (structurally invalid every attempt ->
a2ui_recovery_exhausted hard-fail, already verified e2e).
Updates the agent docstring, qa checklist, and spec test name/comment to reflect
heal = parse_and_fix healing (not invalid->valid retry). aimock-fixtures schema
test: 818 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements the Declarative Generative UI (A2UI dynamic schema) demo for both
the strands (Python) and strands-typescript integrations, bringing them to 1:1
parity with the canonical langgraph-python demo.
Backend:
- Add a dedicated dynamic-A2UI agent (a2ui_dynamic.py / buildA2uiDynamicAgent)
that wires no generate_a2ui tool. The runtime route sets
injectA2UITool: true + defaultCatalogId "declarative-gen-ui-catalog"; the
Strands adapter auto-injects generate_a2ui and drives a secondary render_a2ui
planner. StrandsAgentConfig.a2ui supplies the catalog id and a
composition_guide (sales dataset + composition rules) so the planner is
self-contained.
- Mount the agent on the /declarative-gen-ui sub-path and point the route's
HttpAgent at it, mirroring the a2ui-fixed-schema pattern.
- Bump ag_ui_strands 0.1.9 -> 0.2.1 (Python): the A2UI auto-injection config
(default_catalog_id, guidelines, render planner) landed in 0.2.0. Pulls
ag-ui-a2ui-toolkit transitively. strands-typescript already on 0.2.2.
Frontend:
- Port the langgraph-python declarative-gen-ui frontend (suggestions,
sales-context, definitions incl DataTable, renderers, chat) so the demo
matches the shared D5/D6 probe's sales-analyst pills and catalog components.
Fixtures:
- Author gen-ui-declarative D6 fixtures for both slugs from the langgraph-python
template (3 legs per pill: outer generate_a2ui, inner render_a2ui, narration).
Verified on real OpenAI (gpt-4o): both integrations paint a real A2UI surface
(4 Metric tiles + Revenue-by-Region PieChart + Monthly Revenue BarChart +
narration), matching the probe's expected testids and counts.
Note: the gen-ui-declarative D6 cell currently cannot go green because the
shared harness hits "ReferenceError: __name is not defined" in the probe's
page.evaluate (turn 1 preFill), before any agent call. That harness-tooling bug
is owned separately; the fixtures and wiring here are correct and the real-LLM
paint confirms the expected surface.
Strands mints UUID tool_call_ids for tool results (confirmed via the aimock
journal), so the langgraph-python toolCallId-keyed follow-up fixtures never
matched and the agent re-emitted the tool, looping (text-unstable) across
frontend-tools, gen-ui-agent, gen-ui-open, gen-ui-open-advanced,
gen-ui-headless-complete, reasoning-chain and the weather pills. Re-key the
affected multi-leg fixtures to the id- and thread-history-invariant
sequenceIndex pattern (as built-in-agent does), for both integrations.
Also:
- Add write_document poem/email/quantum fixtures for shared-state-streaming
(was a stale single fixture that 404d).
- Strip content from reasoning-chain tool legs (content+toolCalls in one
fixture is undefined behavior); reasoning rides toolCalls alone.
- Narrow the over-broad d4 summarize catch-all to "Summarize the sales
pipeline" so it stops shadowing gen-ui-agent competitor pill; matches
langgraph-python.
- Commit the real multimodal sample.png/sample.pdf (were git-LFS pointers
the harness could not resolve).
Add the missing "Show me my sales dashboard for this quarter." pill to
the pydantic-ai gen-ui-declarative D6 fixture set: an outer turn
(generate_a2ui, no args) plus the matching _design_a2ui_surface inner
turn carrying the dashboard component payload (KPI metrics row + revenue
pie + monthly-revenue bar), mirrored from the langgraph-python canonical
and the ms-agent-dotnet equivalent.
Closes the staging pydantic-ai D-chat 503 (no_fixture_match): the backend
hits aimock with tools=[generate_a2ui] for this userMessage under
x-aimock-context: pydantic-ai, but only ms-agent-dotnet had the fixture.
Deterministic canonical mirror (no real-LLM recording); contains only the
request-match shape and the A2UI response — no credentials.
The ag2 declarative-gen-ui route pointed its HttpAgent at the root
catch-all mount (agents/agent.py) instead of the dedicated
/declarative-gen-ui mount (a2ui_dynamic.py), and generate_a2ui declared
a required context arg that the model emits as {}. pydantic rejected
every call with "context Field required" and AG2 retried without bound —
a 630-iteration hot loop per pill that flooded logs and starved the
frontend.
Fix: route to the dedicated mount with injectA2UITool:false (the
dedicated agent owns generate_a2ui and emits a2ui_operations itself);
make generate_a2ui a no-arg tool matching the D6 fixtures and the
langgraph-python gold standard, with a constant inner system prompt
(per-pill distinctness comes from the captured user message). Regenerated
the gen-ui-declarative fixture and ported the LP definitions/renderers
catalog (all 7 driver testids) for parity. Eliminates the validation
loop: runsFinished=1, zero validation errors.
Bring the TypeScript AWS Strands integration to parity with the Python
strands sibling now that the @ag-ui/aws-strands TS adapter is confirmed to
support the same feature surface (per its examples/server):
- Restore A2UI: the declarative-gen-ui + a2ui-fixed-schema demos, their
routes, qa, specs, the @copilotkit/a2ui-renderer dep, beautiful-chat's
A2UI catalog, and the manifest entries (generative_ui / features / demos /
a2ui_pattern). manifest now matches strands-python feature-for-feature.
- Header forwarding: attach `x-aimock-context: strands-typescript` as a
static defaultHeader on the OpenAI client (model-factory + sub-agent
client) — the TS analog of the Python integration's _header_forwarding
shim — so aimock matches this integration's fixtures.
- aimock fixtures: add d6/strands-typescript + d4/strands-typescript
(ported from the Python sibling, context retargeted).
- playwright.config: X-AIMock-Context → strands-typescript.
Note: the raw tests/e2e Playwright suite is flaky and not a CI merge gate
(demo e2e / `/eval` D5 are comment-triggered, not required) — it fails the
same specs for strands-python too. The auto-gates (build, validate-
constraints, oxlint/oxfmt, unit) are green.
Mirrors LGP's `generate_a2ui` outer-emit fixture into 8 non-LGP slugs to close
the production 503 gap on the "Show me my sales dashboard for this quarter."
userMessage. Per /tmp/staging-journal-diff.md, aimock-staging returns 503 on
shape-B traffic (model=gpt-4.1, stream=true, tools=[generate_a2ui],
UA=AsyncOpenAI/Python) for all 8 slugs because no fixture matched.
Slugs patched (one entry each in d6/<slug>/gen-ui-declarative.json):
- llamaindex
- built-in-agent
- ag2
- langroid
- claude-sdk-typescript
- claude-sdk-python
- ms-agent-dotnet
- ms-agent-python
Each entry matches `userMessage` + `context: "<slug>"` and emits a
`generate_a2ui` toolcall with no args, mirroring LGP's sales-dashboard outer
entry. Per-slug unique toolCallId.
Red-green proof (local aimock, ghcr.io/copilotkit/aimock:latest):
RED (baseline main, 8 slugs): HTTP=404 no_fixture_match
GREEN (this branch, 8 slugs): HTTP=200 tool=generate_a2ui id=call_d6_decl_dash_outer_<slug>_001
LGP regression (baseline+fix): HTTP=200 (unchanged)
aimock fixture validation: 737/737 tests pass.
PR scope is intentionally narrow per CLAUDE.md "Scope PRs to flagged
findings": this closes ONE userMessage gap. Shape-A 503s (no tools key) and
other userMessage gaps remain as separate follow-ups.
Staging strict-mode replay confirmed ag2 was the only integration missing a
fixture for `Open Excalidraw and sketch a system diagram...` + `create_view`
(the D5 mcp-apps probe's turn-1). All 11 other contexts (langgraph-python,
google-adk, ms-agent-{dotnet,python}, strands, llamaindex, built-in-agent,
claude-sdk-{typescript,python}, langroid, mastra) returned 200 against
aimock-staging with the same payload; ag2 returned 503
`{"code":"no_fixture_match"}`.
Add the missing turn-1 entry to ag2/mcp-apps.json, mirroring the LGP
gold-standard at langgraph-python/tool-rendering-reasoning-chain.json (same
fixture id `call_d5_mcp_apps_create_view_001`, same arguments payload, same
chunkSize). Match keys: userMessage + toolName + context.
Local GREEN proof: iso9 aimock (port 6010) with new fixture mounted returns
200 with the canonical tool_call payload for the exact staging-replay request
body. LGP regression check (same probe with X-AIMock-Context: langgraph-python)
stays green.
Scope intentionally narrow per "scope PRs to the originally-flagged findings":
this PR closes only the one genuine fixture gap identified by the staging
verification audit. The 9 other 503s in that audit (sales-dashboard across 9
slugs + google-adk Excalidraw) are NOT fixture gaps — they are backend
tool-array forwarding issues and are tracked separately.
Re-tier the showcase docs tree to be an agent entry point: README.md
opens with a 'when X, see Y' fanout table that routes to the right
procedural doc; each procedural doc gets a one-line tagline answering
'what does this answer'.
Consolidation:
- DELETE showcase/RUNBOOK.md — operational content merged into DEBUGGING.md
(Integration Patterns, Docker Compose Environment, Production Debugging,
Anti-Patterns, Aimock Fixture Deployment, Dev Iteration Speed). The
--isolate mechanics + CLI rules were already duplicated in DEBUGGING.md.
- DELETE showcase/QA-COVERAGE.md — per-demo coverage matrix + starter hero
matrix + probe depth + infra locations + gaps folded into TESTING.md as
the 'Per-Demo Coverage Matrix' section.
Taglines added (no behavioral change to content): TESTING.md, DEBUGGING.md,
GOTCHAS.md, INTEGRATION-CHECKLIST.md, STYLING-GUIDE.md, FRONTEND-STRATEGY.md,
RAILWAY.md, bin/README.md, aimock/README.md, aimock/RAILWAY.md,
harness/README.md, harness/docs/rotation-drill.md.
Cross-link fixups: FRONTEND-STRATEGY.md (was QA-COVERAGE.md →
TESTING.md#per-demo-coverage-matrix), TESTING.md (removed dangling RUNBOOK
companion reference), README.md (rewritten as fanout entry + retained
from-scratch setup + dashboard SOPs below the fanout).
PARITY_NOTES.md × 12 left alone (per-slug context, not redundant).
(cherry picked from commit 75c9d9755c9118c8abc1fa52deda2012b768cab1)
(cherry picked from commit b64189bae0fe2c9e3a5e3ca440013deb4121f23b)
Mirrors A19b's BIA fix pattern for the Anthropic-family csdkts integration.
Root cause: csdkts uses Anthropic SDK which generates its own toolCallIds (toolu_*) rather than echoing aimock's prescribed call_d6_cc_*. The fixture's toolCallId-gated narration entries never matched on turn-2, causing fall-through to less-specific entries (or 503/no-match).
Fix: replace toolCallId discriminator with turnIndex (count of role:assistant messages). turnIndex is backend-id-invariant — it works regardless of how the backend rewrites tool_call_id values. Same shape as A19b BIA fix.
- Tokyo narration: toolCallId → turnIndex: 1
- AAPL narration: toolCallId → turnIndex: 3
- AAPL emit: added turnIndex: 2
- Tokyo emit: turnIndex: 0
response.content + canonical phrase ("rendered through the custom wildcard catchall") and response.toolCalls UNTOUCHED.
Verified locally on cr5495/fix-a20-csdkts-green at HEAD d178e6730 (post-A21b):
- /tmp/cr/a20v6-green-csdkts.log: 1 passed, INNER_EXIT=0
- iso2 slot, full infra healthy (aimock+pocketbase+dashboard+csdkts)
(cherry picked from commit e66e0eb0ce72c970348183eeb4f4b57c3f5b1d29)
Replace per-leg toolCallId pin with turnIndex (assistant-count) +
userMessage on the two narration fixtures, and add explicit turnIndex
to the AAPL-emit fixture, so first-match-wins partitions the four
request shapes BIA produces against the OpenAI Responses API.
Root cause:
- BIA uses @tanstack/ai-openai openaiText('gpt-4o') which calls
/v1/responses. aimock converts each /v1/responses request to a
chat-completions-shaped completionReq via responsesInputToMessages()
and matches with the same router. The matcher's toolCallId check is
strict equality against the last message's tool_call_id.
- BIA's TanStack runtime auto-generates tool_call_id at request time
(e.g. 'fc-fCgLtvquOtRpCJTM'), so the fixture-side literal
'call_d6_cc_weather_001' / 'call_d6_cc_stock_001' never matched.
Result: 503 STRICT no-fixture-match on the narration turns, BIA
agent looped on AAPL emit indefinitely.
Fix shape:
- Tokyo narration: toolCallId -> turnIndex: 1
- AAPL narration: toolCallId -> turnIndex: 3 (was off-by-one until I
accounted for the Tokyo-narration assistant message itself adding
to the assistant-count tally seen at AAPL emit time)
- AAPL emit: add turnIndex: 2 so first-match-wins partitions emit vs
narration on the second prompt's two turns
Verification (worktree wt-5495-a19-bia-record, slot iso6):
- RED: bin/showcase test built-in-agent:tool-rendering-custom-catchall
--d5 --isolate -> state=red, 0 passed/1 failed, INNER_EXIT=1
(aimock journal: 3x 503 'No fixture matched' on turn-2 narration
request; AAPL emit fixture matched repeatedly = infinite loop)
- GREEN: same command, post-fix and aimock-restart so the container
reloads the fixture -> state=green, 1 passed, INNER_EXIT=0; aimock
journal: 4 requests, all 200, clean progression
Tokyo-emit (asstCount=0) -> Tokyo-narrate (asstCount=1) ->
AAPL-emit (asstCount=2) -> AAPL-narrate (asstCount=3)
- LGP regression: bin/showcase test
langgraph-python:tool-rendering-custom-catchall --d5 --isolate ->
state=green, 1 passed, INNER_EXIT=0 (uses its own fixture under
aimock/d6/langgraph-python/ — untouched by this change)
Constraints honored: response.content and response.toolCalls
preserved verbatim; canonical narration phrases unchanged; only the
match keys (and their explanatory _comment fields) were modified;
other integrations' fixtures and the harness probe were not touched.
(cherry picked from commit 60d027a1376ba72309b5b3fcf94cc26c64757307)
Parent commit 9491b8934 (fix(showcase): disjoint catchall userMessages + content-asserting probes) changed the d5-tool-rendering-custom-catchall probe userMessages to 'Forecast Tokyo through the wildcard renderer' / 'Quote AAPL through the wildcard renderer' but did not add matching entries to LGP-gold's fixture. Result: aimock no-match -> agent_run_error_event -> SSE-missing -> probe RED on LGP with zero bubbles mounted.
This commit adds 4 entries (2 emit + 2 toolCallId-gated narration) following the canonical pattern used by the other 17 integrations. After this commit, LGP bubbles mount and narrations settle with the canonical phrase in bubble.textContent, matching BIA's end-state.
NOTE: A separate fleet-wide probe-layer bug (validateCustomCatchall's customContentPhrasePresent page-wide DOM scan returns false even when phrase is in bubble.textContent) keeps the probe RED in this session. That's a separate concern, to be fixed in a follow-on commit. This commit is a strict improvement: pre-fix LGP got zero bubbles + agent_run_error_event; post-fix LGP renders both bubbles and both narrations settle with the phrase.
(cherry picked from commit 62e04976c3b292a27e1b1cc0bf2bb6fda47db786)
After A2 (commit 6c596d8d6) stripped toolName from the primary emit entries, sibling turnIndex:0 fallback entries became unreachable under first-match-wins. The 7 A2-target siblings (ag2/google-adk/lgf/lgts/mastra/msdotnet/csdkts toolName strip) had equivalent dead fallbacks deleted in that commit; csdkts was inconsistently treated. Removing the 2 dead entries restores cross-fleet consistency. Probe contract unaffected — toolCallId-gated narration entries still emit the required content phrase.
(cherry picked from commit 9720636519f4cd858fcdc08ed84597be05604a2e)
crewai-crews/ms-agent-python/pydantic-ai catchall AAPL entries still carried stale $189.42/up 1.27% narration and toolCall args lacking price_usd/change_pct. Round 1 CR finding #3 was identified but never given a fix agent. Brought all 3 to canonical shape matching built-in-agent (A5) and csdkts narration (A6): price_usd=338.37, change_pct=-2.96, narration 'AAPL is trading at $338.37, down 2.96% on the day — rendered through the custom wildcard catchall.'
(cherry picked from commit df84657310451500278d0b3d0125c9c490042d2b)
ag2/claude-sdk-typescript/google-adk/langgraph-fastapi/langgraph-typescript/mastra/ms-agent-dotnet: AAPL is turn-1 so turnIndex:0 fallback unreachable; toolName gate fails on wildcard-renderer integrations that don't register get_stock_price. Aligned to LGP-gold pattern (userMessage+context discriminator, no toolName, no turnIndex:0 fallback). Preserved legitimate multi-pill matchers (SF/flights/d20/chain) on the 4 multi-pill integrations.
(cherry picked from commit c10821b5d904b31bee2ab2a39db3b1565aecba26)
built-in-agent shipped $189.42 vs the rest of the fleet's $338.37. Drift makes any content-asserting test on AAPL price brittle. Aligned.
(cherry picked from commit c0796da2a63abfeaa1d8ea06200df0754d30a2e6)
spring-ai retained hasToolResult:false on the Tokyo weather emit fixture — outlier vs LGP-gold and the other 16 catchall fixtures. Aligned: userMessage+context discriminator only.
(cherry picked from commit d268a88c4a3f405dfcc15b2aaccc190e61cd7ac7)
built-in-agent, crewai-crews, ms-agent-python, pydantic-ai: Tokyo turn-1 tool result makes hasToolResult permanently true → AAPL fixture never matched → 30s timeout. Aligned with agno/langroid/llamaindex/strands/claude-sdk-python pattern: rely on userMessage+context (and toolCallId where relevant) as the gate.
(cherry picked from commit 8e313cd1b043cf1c61efb82143171a28d7699c49)
Closed PR #5465 introduced cross-fixture leakage by stripping the
toolName discriminator on shared {userMessage, context} keys: with
aimock's alphabetical first-match-wins ordering,
tool-rendering-custom-catchall.json sorts before
tool-rendering-default-catchall.json, so default-catchall page requests
were served the custom file's content. The probe was structurally blind
because it asserted only DOM testids (copilot-tool-render +
data-tool-name=get_weather) — both fixtures emit get_weather, so the
testid signal passed regardless of which fixture won.
Real LGP-gold pattern is disjoint userMessages between default and
custom catchall fixtures (NOT shared keys discriminated by toolName).
This PR ports the LGP-gold pattern to the other 17 integrations and
adds page-text content assertions to both d5 catchall probes so this
class of regression can't recur silently:
- Probe prompts disjoint between default-catchall and custom-catchall
- Fixture userMessages updated to match the new disjoint prompts
- Page-text content assertions in both d5 catchall probes
(default negatively asserts the custom-catchall leak phrase;
custom positively asserts it)
Local gold-standard red-green proof captured:
- Step A: live staging Playwright baseline (RED-OF-RECORD)
- Step B: local control-plane on origin/main reproduces structural
fragility (probes pass at testid level, custom fixture wins on
default's userMessage path)
- Step C: local control-plane on this branch — both catchall probes
GREEN with the new content-asserting assertions
tool-rendering-default-catchall: pass=true (5443ms)
tool-rendering-custom-catchall: pass=true (8956ms)
cross-tool signature passed
LGP (langgraph-python) was already disjoint; it remains untouched.
PR #5459 added userMessage matchers ("forecast for Tokyo" + "current price
of AAPL") on the catchall fixtures, which flipped most integrations from
red to green. Six cells stayed red on staging because the FIXTURE SHAPE
itself was broken on the matched fixtures, not just the userMessage key.
Failing cells (all custom-catchall except strands default):
d5:agno/tool-rendering-custom-catchall
d5:llamaindex/tool-rendering-custom-catchall
d5:langroid/tool-rendering-custom-catchall
d5:claude-sdk-python/tool-rendering-custom-catchall
d5:strands/tool-rendering-custom-catchall
d5:strands/tool-rendering-default-catchall
PocketBase confirms the failure mode: turn 1 (Tokyo) completes; turn 2
(AAPL) times out at 30s or renders the wrong content. E.g. agno:
errorDesc: "timeout: assistant did not respond within 30000ms"
failure_turn: 2
turns_completed: 1 / 2
Root cause: the failing tool-emit fixtures used `hasToolResult: false`
(or `toolName: "<tool>"`) as their gate. aimock's hasToolResult check is
`messages.some(m => m.role === 'tool')` over the WHOLE thread — so once
turn 1's Tokyo tool result lands in the conversation, hasToolResult is
permanently true and `hasToolResult:false` can never match turn 2 → no
fixture → 30s timeout. `toolName:get_stock_price` likewise fails when an
integration backend doesn't forward the tool definition on turn 2.
The same trap is documented in
showcase/aimock/d6/langgraph-python/tool-rendering.json:
"_comment": "Gated on toolName:get_stock_price rather than
hasToolResult:false. The D5 tool-rendering-custom-catchall probe runs
'weather in Tokyo' first, which leaves a get_weather tool result in
the thread; the aimock router implements hasToolResult as
messages.some(m=>m.role==='tool'), so hasToolResult is permanently
true on the AAPL turn and a hasToolResult:false gate could never
match (→ no_fixture_match → 503 → 30s timeout)."
Fix: align all 6 files to the canonical pattern used by mastra/spring-
ai/built-in-agent on this probe:
1. toolCallId-keyed narration fixture FIRST
2. tool-emit fixture SECOND with ONLY `userMessage` + `context`
(no hasToolResult / toolName gate)
The toolCallId narration uses aimock's
`messages[last].role === 'tool' && tool_call_id === ...` check, so it
correctly wins on iteration 2 (post-tool-result) without being affected
by older turns' tool results. The tool-emit fixture matches turn 1 (last
message is user) and re-emits only when the narration above hasn't
matched.
Strands' two cells additionally needed REORDERING — they had the tool-
emit fixture before the toolCallId fixture, defeating first-match-wins.
Strands' staging backend was also returning 502 during testing; once it
recovers, the corrected fixtures should let the probe pass. The fixture
changes are necessary but may not be sufficient for strands if backend
remains down.
No probe-side, harness, or backend changes — pure fixture-content
alignment. Six fixture files modified; line totals: -87 / +68.
Jordan Ritter