The secondary-LLM prompt was far thinner than the canonical generation guidelines,
so it emitted trees that (correctly) failed the renderer's paint gate → surface-missing.
Port the canonical generation rules into the prompt, add output validation, add catalog
parity (DataTable + info-row), ground the planner with sales-context, and record
multi-turn aimock fixtures. Includes CR fixes: two-arg z.record for the DataTable rows
schema (zod@4 API), index-based DataTable row key, and Metric trendValue rendering for
neutral trend.
## Summary
`showcase/aimock/RAILWAY.md` described the aimock fixture infrastructure
as it worked during an earlier migration phase, not as it works today.
The stale claims actively misled debugging. This PR corrects them to
match the real, verified infra.
## Before → After
| Claim | Before (stale) | After (correct) |
| --- | --- | --- |
| Fixture delivery (§4/§5) | Three fixtures fetched at boot from
GitHub-raw URLs: `d5-all.json`, `smoke.json`, `feature-parity.json` |
Fixtures are **baked into the image** at build time under
`/fixtures/{shared,d4,d6}`; no remote fetch |
| `d5-all.json` | Authoritative D5 bundle, re-bundled from
`harness/fixtures/d5/*` | **Gone** — a one-time migration source split
into the per-slug `showcase/aimock/d6/<slug>/` tree (the source of
truth) |
| startCommand `--fixtures` (§5) | 3 `raw.githubusercontent.com` URLs |
`--fixtures /fixtures/shared --fixtures /fixtures/d4 --fixtures
/fixtures/d6` (the baked-in dirs). Added a warning that a single
`--fixtures /fixtures` parent loads nothing |
| Runtime image (§3) | Bare `ghcr.io/copilotkit/aimock:<version>` pulled
directly | `showcase-aimock` image, `FROM
ghcr.io/copilotkit/aimock:latest` + baked fixtures, built by
`showcase_build.yml` |
| Dockerfile (§8) | "Dead code / legacy wrapper, safe to remove" |
**LIVE** — `showcase_build.yml` builds it (`dockerfile:
showcase/aimock/Dockerfile`) and it bakes `shared/`, `d4/`, `d6/` into
`/fixtures/`. Do not remove |
| Env vars (§6), references (§9) | Referenced public GitHub-raw fixture
URLs and ~5 min raw-edge cache propagation | Local baked dirs;
propagation is via image rebuild + Railway deploy |
Also documents the working Railway mutation auth path by mechanism only
(§2): account-scoped `RAILWAY_TOKEN` from the DevOps `showcase`
1Password item via GraphQL `Authorization: Bearer` — the CLI session
token is not authorized for showcase-project mutations. No token value
is in the doc.
## Verification
- `showcase/aimock/d5-all.json` does not exist; the tree is `shared/`,
`d4/`, `d6/`.
- `showcase/aimock/Dockerfile` is `FROM
ghcr.io/copilotkit/aimock:latest` + `COPY shared|d4|d6 ->
/fixtures/...`.
- `.github/workflows/showcase_build.yml` matrix entry `showcase-aimock`
builds `dockerfile: showcase/aimock/Dockerfile`, context
`showcase/aimock`.
Docs-only change. Do not merge / do not enable auto-merge.
PR #5661 mis-templated this fixture from ms-agent-dotnet: the inner
secondary-LLM blocks used toolName "_design_a2ui_surface" (never matches
pydantic-ai, whose inner tool is render_a2ui) and only the sales-dashboard
pill had any block, so the other 3 declarative pills matched nothing and
the cell 503'd with no_fixture_match on turn 1.
Re-author 1:1 from the canonical langgraph-python fixture (identical inner
tool render_a2ui + per-pill surfaces), adapted for pydantic-ai: context
"pydantic-ai" on the outer generate_a2ui + narration entries, inner
render_a2ui entries matched by toolName alone (the agent's inner OpenAI()
client does not forward x-aimock-context). All 4 pills (sales-dashboard,
team-performance, at-risk, top-account) now get both an outer
(generate_a2ui) and inner (render_a2ui) block whose component payloads meet
each pill's probe assertion. Dead KPI/pie/bar/status blocks removed.
Real-probe proof: with this fixture the inner render_a2ui call matches and
the outer narration renders ("Here's your Q2 sales dashboard.") — the cell
advances from "Strict mode: no fixture matched / 503" to all LLM calls
matched. Remaining surface-missing failure is a non-fixture frontend/agent
A2UI delivery gap (see PR description), out of scope for this fixture fix.
The showcase-aimock RAILWAY.md described fixtures as fetched from GitHub-raw
URLs at boot and called the Dockerfile dead code. Both are false and actively
misled debugging: fixtures are baked into the image at build time under
/fixtures/{shared,d4,d6}, the Dockerfile is the live image builder driven by
showcase_build.yml, and the d5-all.json bundle no longer exists (split into
the per-slug d6/ tree). Corrects sections 3, 4, 5, 6, 7, 8, 9 to match the
real infra, fixes the startCommand to load the three baked-in subdirectories,
and documents the account-scoped RAILWAY_TOKEN mutation path by mechanism.
## Summary
- The pydantic-ai `generate_a2ui` declarative D6 turn was missing an
aimock fixture, producing HTTP 503 `no_fixture_match` on staging
(pydantic-ai 503 vs ms-agent-dotnet 200 for the same turn) — a source of
dashboard flapping.
- Adds the canonical mirror fixtures (outer `generate_a2ui` + matching
inner `_design_a2ui_surface`) to
`showcase/aimock/d6/pydantic-ai/gen-ui-declarative.json`. These are
deterministic canonical mirrors matching the langgraph-python convention
— **not** a non-deterministic real-LLM recording — preserving the
mandatory LGP 1:1 parity.
## Red-green proof
- **RED:** exact failing request (`POST /v1/responses`, gpt-4.1, "Show
me my sales dashboard for this quarter.", tools=[`generate_a2ui`],
header `x-aimock-context: pydantic-ai`, strict) against the pre-fix
fixture set → **HTTP 503 `no_fixture_match`** (reproduces staging
exactly; confirmed live on staging too).
- **GREEN:** same request against the new set → **HTTP 200** SSE
emitting the `generate_a2ui` tool call; the inner `_design_a2ui_surface`
turn also returns 200 with the dashboard surface.
- Independently re-verified. `validate-on-load` clean (no fixture
shadowing); existing pydantic-ai D6 turns (KPI/pie/bar/status) still
match identically — no regression.
## Notes
- No credentials in the committed fixture — the OpenAI key was never
even resolved (canonical mirror, not a recording). Credential scan of
the diff + full blob: zero matches.
## Test plan
- [ ] CI green
- [ ] After deploy, confirm the `generate_a2ui` declarative D6 cell
flips red→green on staging
Brings google-adk to 39/39 D6 (reproduced across two independent full-matrix
runs, zero regressions). Four changes:
- entrypoint.sh: remove ADK_DISABLE_PROGRESSIVE_SSE_STREAMING=1. That flag's
non-progressive aggregation path ended ADK's agentic loop after the first
tool round (no post-tool LLM re-invoke), which broke every demo needing a
second turn: the subagents chain (research -> writing -> critique),
tool-rendering-reasoning-chain (AAPL -> MSFT), shared-state-read-write's
confirmation, and the custom-catchall narration. The partial-event abort it
guarded against is already handled in-callback by stop_on_terminal_text.
- manifest.yaml: un-skip-list tool-rendering-reasoning-chain (now passes with
the loop restored).
- headless_complete_agent.py: add AGUIToolset() so the frontend highlight_note
tool is injected and routed to the browser. Removing the flag unmasked this
pre-existing gap — turn 3 dispatched highlight_note server-side and the
backend registry rejected it. langgraph-python auto-injects frontend tools;
ADK needs AGUIToolset() in the agent's tools list.
- aimock/d6/google-adk/gen-ui-interrupt.json: order each pill's narration leg
(toolCallId) before its emit leg and drop the thread-global hasToolResult
gate, so the alice pill no longer 503s after the sales pill leaves a tool
result in the thread.
Takes d6:strands and d6:strands-typescript from 32/35 to 34/35.
- shared-state-read: the turn-2 fixture leg wrongly pinned turnIndex:0, so the
aimock matcher skipped it on turn 2 -> 404 -> turn-2 sse-missing. Drop
turnIndex to mirror the langgraph-python gold-standard fixture.
- multimodal: sample.png/pdf/wav shipped as git-LFS pointers, so deploy/test
environments without 'git lfs pull' served the ~130-byte pointer text as the
upload -> the run never started (runsFinished=0). Ship them as regular
binaries via a per-integration .gitattributes lfs-unset + real bytes,
mirroring langgraph-python's convention.
Remaining red (gen-ui-declarative) is a Strands A2UI-dynamic run-completion bug
(reproduces on real-LLM staging too): the surface paints but generate_a2ui
never completes, so the run hangs 'Running'. Tracked separately.
Switch declarative-gen-ui to runtime auto-injection (injectA2UITool: true, plain
agent), matching the langgraph-python / AWS Strands gold standard. The ag-ui-adk
0.7.0 adapter auto-injects generate_a2ui via plan_a2ui_injection. Verified under
the canonical D6 driver: all four declarative pills pass (sales-dashboard,
team-performance, at-risk, top-account), plus a2ui-fixed-schema and
beautiful-chat.
Remove the stale known-failing mark from the a2ui-recovery heal e2e and the
OSS-374 inner-render-disambiguation notes (fixture _meta, QA doc, agent
docstring, route comment). Verified against the aimock journal: ag-ui-adk 0.7.0
forwards the run conversation into the inner render_a2ui call, so each pill
matches its own fixture by userMessage and both heal and exhaust pass. The
recovery demo stays backend-owned (the only path with the recovery loop) and
ADK-only (OSS-375 tracks langgraph-python parity).
Replace the "Tracked in Linear" placeholders with the filed issue IDs:
OSS-374 (showcase-aimock inner-subagent disambiguation) and OSS-375
(recovery-demo langgraph-python parity).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `heal` e2e test fails against the live showcase aimock because the harness
cannot disambiguate the two pills' inner render_a2ui sub-agent calls (the
backend-owned middleware issues the inner call with a generic render prompt +
shared suggestion context, so the "last user turn" aimock keys on is not the
pill prompt). Both pills match the same inner fixture (the EXHAUST one, by
first-match order), so the heal pill exhausts instead of healing.
This is an aimock harness limitation, not a middleware/demo bug: the middleware
heals free-form args correctly in the OSS-158 toolkit gate and against real
Gemini. Keep the test running (not skipped) as a deliberate demonstration for
the Showcase team; it does not red CI (these specs aren't run for google-adk in
CI). Update the spec header, the heal-test note, the fixture _note, the qa doc,
and the recovery_agent docstring to reflect this accurately (and drop the stale
sequenceIndex description). Tracked in Linear: aimock inner-subagent
disambiguation + recovery-demo LP parity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The heal fixture relied on per-attempt aimock response switching (sequenceIndex
0=invalid -> 1=valid), but sequenceIndex doesn't advance across the recovery
loop's intra-turn render_a2ui calls in the per-integration Playwright context
(no X-Test-Id), so aimock served invalid on every attempt and the demo never
healed (0 metrics).
Switch the heal pill to a single deterministic response: the inner render_a2ui
returns FREE-FORM args (components/data as JSON strings) that the middleware
heals via parse_and_fix into a valid surface in one pass — no per-attempt
switching. Verified in-sandbox: free-form args -> a2ui_operations with 2 Metric
components. The exhaust pill is unchanged (structurally invalid every attempt ->
a2ui_recovery_exhausted hard-fail, already verified e2e).
Updates the agent docstring, qa checklist, and spec test name/comment to reflect
heal = parse_and_fix healing (not invalid->valid retry). aimock-fixtures schema
test: 818 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements the Declarative Generative UI (A2UI dynamic schema) demo for both
the strands (Python) and strands-typescript integrations, bringing them to 1:1
parity with the canonical langgraph-python demo.
Backend:
- Add a dedicated dynamic-A2UI agent (a2ui_dynamic.py / buildA2uiDynamicAgent)
that wires no generate_a2ui tool. The runtime route sets
injectA2UITool: true + defaultCatalogId "declarative-gen-ui-catalog"; the
Strands adapter auto-injects generate_a2ui and drives a secondary render_a2ui
planner. StrandsAgentConfig.a2ui supplies the catalog id and a
composition_guide (sales dataset + composition rules) so the planner is
self-contained.
- Mount the agent on the /declarative-gen-ui sub-path and point the route's
HttpAgent at it, mirroring the a2ui-fixed-schema pattern.
- Bump ag_ui_strands 0.1.9 -> 0.2.1 (Python): the A2UI auto-injection config
(default_catalog_id, guidelines, render planner) landed in 0.2.0. Pulls
ag-ui-a2ui-toolkit transitively. strands-typescript already on 0.2.2.
Frontend:
- Port the langgraph-python declarative-gen-ui frontend (suggestions,
sales-context, definitions incl DataTable, renderers, chat) so the demo
matches the shared D5/D6 probe's sales-analyst pills and catalog components.
Fixtures:
- Author gen-ui-declarative D6 fixtures for both slugs from the langgraph-python
template (3 legs per pill: outer generate_a2ui, inner render_a2ui, narration).
Verified on real OpenAI (gpt-4o): both integrations paint a real A2UI surface
(4 Metric tiles + Revenue-by-Region PieChart + Monthly Revenue BarChart +
narration), matching the probe's expected testids and counts.
Note: the gen-ui-declarative D6 cell currently cannot go green because the
shared harness hits "ReferenceError: __name is not defined" in the probe's
page.evaluate (turn 1 preFill), before any agent call. That harness-tooling bug
is owned separately; the fixtures and wiring here are correct and the real-LLM
paint confirms the expected surface.
Strands mints UUID tool_call_ids for tool results (confirmed via the aimock
journal), so the langgraph-python toolCallId-keyed follow-up fixtures never
matched and the agent re-emitted the tool, looping (text-unstable) across
frontend-tools, gen-ui-agent, gen-ui-open, gen-ui-open-advanced,
gen-ui-headless-complete, reasoning-chain and the weather pills. Re-key the
affected multi-leg fixtures to the id- and thread-history-invariant
sequenceIndex pattern (as built-in-agent does), for both integrations.
Also:
- Add write_document poem/email/quantum fixtures for shared-state-streaming
(was a stale single fixture that 404d).
- Strip content from reasoning-chain tool legs (content+toolCalls in one
fixture is undefined behavior); reasoning rides toolCalls alone.
- Narrow the over-broad d4 summarize catch-all to "Summarize the sales
pipeline" so it stops shadowing gen-ui-agent competitor pill; matches
langgraph-python.
- Commit the real multimodal sample.png/sample.pdf (were git-LFS pointers
the harness could not resolve).
Add the missing "Show me my sales dashboard for this quarter." pill to
the pydantic-ai gen-ui-declarative D6 fixture set: an outer turn
(generate_a2ui, no args) plus the matching _design_a2ui_surface inner
turn carrying the dashboard component payload (KPI metrics row + revenue
pie + monthly-revenue bar), mirrored from the langgraph-python canonical
and the ms-agent-dotnet equivalent.
Closes the staging pydantic-ai D-chat 503 (no_fixture_match): the backend
hits aimock with tools=[generate_a2ui] for this userMessage under
x-aimock-context: pydantic-ai, but only ms-agent-dotnet had the fixture.
Deterministic canonical mirror (no real-LLM recording); contains only the
request-match shape and the A2UI response — no credentials.
The ag2 declarative-gen-ui route pointed its HttpAgent at the root
catch-all mount (agents/agent.py) instead of the dedicated
/declarative-gen-ui mount (a2ui_dynamic.py), and generate_a2ui declared
a required context arg that the model emits as {}. pydantic rejected
every call with "context Field required" and AG2 retried without bound —
a 630-iteration hot loop per pill that flooded logs and starved the
frontend.
Fix: route to the dedicated mount with injectA2UITool:false (the
dedicated agent owns generate_a2ui and emits a2ui_operations itself);
make generate_a2ui a no-arg tool matching the D6 fixtures and the
langgraph-python gold standard, with a constant inner system prompt
(per-pill distinctness comes from the captured user message). Regenerated
the gen-ui-declarative fixture and ported the LP definitions/renderers
catalog (all 7 driver testids) for parity. Eliminates the validation
loop: runsFinished=1, zero validation errors.
Bring the TypeScript AWS Strands integration to parity with the Python
strands sibling now that the @ag-ui/aws-strands TS adapter is confirmed to
support the same feature surface (per its examples/server):
- Restore A2UI: the declarative-gen-ui + a2ui-fixed-schema demos, their
routes, qa, specs, the @copilotkit/a2ui-renderer dep, beautiful-chat's
A2UI catalog, and the manifest entries (generative_ui / features / demos /
a2ui_pattern). manifest now matches strands-python feature-for-feature.
- Header forwarding: attach `x-aimock-context: strands-typescript` as a
static defaultHeader on the OpenAI client (model-factory + sub-agent
client) — the TS analog of the Python integration's _header_forwarding
shim — so aimock matches this integration's fixtures.
- aimock fixtures: add d6/strands-typescript + d4/strands-typescript
(ported from the Python sibling, context retargeted).
- playwright.config: X-AIMock-Context → strands-typescript.
Note: the raw tests/e2e Playwright suite is flaky and not a CI merge gate
(demo e2e / `/eval` D5 are comment-triggered, not required) — it fails the
same specs for strands-python too. The auto-gates (build, validate-
constraints, oxlint/oxfmt, unit) are green.
Mirrors LGP's `generate_a2ui` outer-emit fixture into 8 non-LGP slugs to close
the production 503 gap on the "Show me my sales dashboard for this quarter."
userMessage. Per /tmp/staging-journal-diff.md, aimock-staging returns 503 on
shape-B traffic (model=gpt-4.1, stream=true, tools=[generate_a2ui],
UA=AsyncOpenAI/Python) for all 8 slugs because no fixture matched.
Slugs patched (one entry each in d6/<slug>/gen-ui-declarative.json):
- llamaindex
- built-in-agent
- ag2
- langroid
- claude-sdk-typescript
- claude-sdk-python
- ms-agent-dotnet
- ms-agent-python
Each entry matches `userMessage` + `context: "<slug>"` and emits a
`generate_a2ui` toolcall with no args, mirroring LGP's sales-dashboard outer
entry. Per-slug unique toolCallId.
Red-green proof (local aimock, ghcr.io/copilotkit/aimock:latest):
RED (baseline main, 8 slugs): HTTP=404 no_fixture_match
GREEN (this branch, 8 slugs): HTTP=200 tool=generate_a2ui id=call_d6_decl_dash_outer_<slug>_001
LGP regression (baseline+fix): HTTP=200 (unchanged)
aimock fixture validation: 737/737 tests pass.
PR scope is intentionally narrow per CLAUDE.md "Scope PRs to flagged
findings": this closes ONE userMessage gap. Shape-A 503s (no tools key) and
other userMessage gaps remain as separate follow-ups.
Staging strict-mode replay confirmed ag2 was the only integration missing a
fixture for `Open Excalidraw and sketch a system diagram...` + `create_view`
(the D5 mcp-apps probe's turn-1). All 11 other contexts (langgraph-python,
google-adk, ms-agent-{dotnet,python}, strands, llamaindex, built-in-agent,
claude-sdk-{typescript,python}, langroid, mastra) returned 200 against
aimock-staging with the same payload; ag2 returned 503
`{"code":"no_fixture_match"}`.
Add the missing turn-1 entry to ag2/mcp-apps.json, mirroring the LGP
gold-standard at langgraph-python/tool-rendering-reasoning-chain.json (same
fixture id `call_d5_mcp_apps_create_view_001`, same arguments payload, same
chunkSize). Match keys: userMessage + toolName + context.
Local GREEN proof: iso9 aimock (port 6010) with new fixture mounted returns
200 with the canonical tool_call payload for the exact staging-replay request
body. LGP regression check (same probe with X-AIMock-Context: langgraph-python)
stays green.
Scope intentionally narrow per "scope PRs to the originally-flagged findings":
this PR closes only the one genuine fixture gap identified by the staging
verification audit. The 9 other 503s in that audit (sales-dashboard across 9
slugs + google-adk Excalidraw) are NOT fixture gaps — they are backend
tool-array forwarding issues and are tracked separately.
Re-tier the showcase docs tree to be an agent entry point: README.md
opens with a 'when X, see Y' fanout table that routes to the right
procedural doc; each procedural doc gets a one-line tagline answering
'what does this answer'.
Consolidation:
- DELETE showcase/RUNBOOK.md — operational content merged into DEBUGGING.md
(Integration Patterns, Docker Compose Environment, Production Debugging,
Anti-Patterns, Aimock Fixture Deployment, Dev Iteration Speed). The
--isolate mechanics + CLI rules were already duplicated in DEBUGGING.md.
- DELETE showcase/QA-COVERAGE.md — per-demo coverage matrix + starter hero
matrix + probe depth + infra locations + gaps folded into TESTING.md as
the 'Per-Demo Coverage Matrix' section.
Taglines added (no behavioral change to content): TESTING.md, DEBUGGING.md,
GOTCHAS.md, INTEGRATION-CHECKLIST.md, STYLING-GUIDE.md, FRONTEND-STRATEGY.md,
RAILWAY.md, bin/README.md, aimock/README.md, aimock/RAILWAY.md,
harness/README.md, harness/docs/rotation-drill.md.
Cross-link fixups: FRONTEND-STRATEGY.md (was QA-COVERAGE.md →
TESTING.md#per-demo-coverage-matrix), TESTING.md (removed dangling RUNBOOK
companion reference), README.md (rewritten as fanout entry + retained
from-scratch setup + dashboard SOPs below the fanout).
PARITY_NOTES.md × 12 left alone (per-slug context, not redundant).
(cherry picked from commit 75c9d9755c9118c8abc1fa52deda2012b768cab1)
(cherry picked from commit b64189bae0fe2c9e3a5e3ca440013deb4121f23b)
Mirrors A19b's BIA fix pattern for the Anthropic-family csdkts integration.
Root cause: csdkts uses Anthropic SDK which generates its own toolCallIds (toolu_*) rather than echoing aimock's prescribed call_d6_cc_*. The fixture's toolCallId-gated narration entries never matched on turn-2, causing fall-through to less-specific entries (or 503/no-match).
Fix: replace toolCallId discriminator with turnIndex (count of role:assistant messages). turnIndex is backend-id-invariant — it works regardless of how the backend rewrites tool_call_id values. Same shape as A19b BIA fix.
- Tokyo narration: toolCallId → turnIndex: 1
- AAPL narration: toolCallId → turnIndex: 3
- AAPL emit: added turnIndex: 2
- Tokyo emit: turnIndex: 0
response.content + canonical phrase ("rendered through the custom wildcard catchall") and response.toolCalls UNTOUCHED.
Verified locally on cr5495/fix-a20-csdkts-green at HEAD d178e6730 (post-A21b):
- /tmp/cr/a20v6-green-csdkts.log: 1 passed, INNER_EXIT=0
- iso2 slot, full infra healthy (aimock+pocketbase+dashboard+csdkts)
(cherry picked from commit e66e0eb0ce72c970348183eeb4f4b57c3f5b1d29)
Replace per-leg toolCallId pin with turnIndex (assistant-count) +
userMessage on the two narration fixtures, and add explicit turnIndex
to the AAPL-emit fixture, so first-match-wins partitions the four
request shapes BIA produces against the OpenAI Responses API.
Root cause:
- BIA uses @tanstack/ai-openai openaiText('gpt-4o') which calls
/v1/responses. aimock converts each /v1/responses request to a
chat-completions-shaped completionReq via responsesInputToMessages()
and matches with the same router. The matcher's toolCallId check is
strict equality against the last message's tool_call_id.
- BIA's TanStack runtime auto-generates tool_call_id at request time
(e.g. 'fc-fCgLtvquOtRpCJTM'), so the fixture-side literal
'call_d6_cc_weather_001' / 'call_d6_cc_stock_001' never matched.
Result: 503 STRICT no-fixture-match on the narration turns, BIA
agent looped on AAPL emit indefinitely.
Fix shape:
- Tokyo narration: toolCallId -> turnIndex: 1
- AAPL narration: toolCallId -> turnIndex: 3 (was off-by-one until I
accounted for the Tokyo-narration assistant message itself adding
to the assistant-count tally seen at AAPL emit time)
- AAPL emit: add turnIndex: 2 so first-match-wins partitions emit vs
narration on the second prompt's two turns
Verification (worktree wt-5495-a19-bia-record, slot iso6):
- RED: bin/showcase test built-in-agent:tool-rendering-custom-catchall
--d5 --isolate -> state=red, 0 passed/1 failed, INNER_EXIT=1
(aimock journal: 3x 503 'No fixture matched' on turn-2 narration
request; AAPL emit fixture matched repeatedly = infinite loop)
- GREEN: same command, post-fix and aimock-restart so the container
reloads the fixture -> state=green, 1 passed, INNER_EXIT=0; aimock
journal: 4 requests, all 200, clean progression
Tokyo-emit (asstCount=0) -> Tokyo-narrate (asstCount=1) ->
AAPL-emit (asstCount=2) -> AAPL-narrate (asstCount=3)
- LGP regression: bin/showcase test
langgraph-python:tool-rendering-custom-catchall --d5 --isolate ->
state=green, 1 passed, INNER_EXIT=0 (uses its own fixture under
aimock/d6/langgraph-python/ — untouched by this change)
Constraints honored: response.content and response.toolCalls
preserved verbatim; canonical narration phrases unchanged; only the
match keys (and their explanatory _comment fields) were modified;
other integrations' fixtures and the harness probe were not touched.
(cherry picked from commit 60d027a1376ba72309b5b3fcf94cc26c64757307)
Parent commit 9491b8934 (fix(showcase): disjoint catchall userMessages + content-asserting probes) changed the d5-tool-rendering-custom-catchall probe userMessages to 'Forecast Tokyo through the wildcard renderer' / 'Quote AAPL through the wildcard renderer' but did not add matching entries to LGP-gold's fixture. Result: aimock no-match -> agent_run_error_event -> SSE-missing -> probe RED on LGP with zero bubbles mounted.
This commit adds 4 entries (2 emit + 2 toolCallId-gated narration) following the canonical pattern used by the other 17 integrations. After this commit, LGP bubbles mount and narrations settle with the canonical phrase in bubble.textContent, matching BIA's end-state.
NOTE: A separate fleet-wide probe-layer bug (validateCustomCatchall's customContentPhrasePresent page-wide DOM scan returns false even when phrase is in bubble.textContent) keeps the probe RED in this session. That's a separate concern, to be fixed in a follow-on commit. This commit is a strict improvement: pre-fix LGP got zero bubbles + agent_run_error_event; post-fix LGP renders both bubbles and both narrations settle with the phrase.
(cherry picked from commit 62e04976c3b292a27e1b1cc0bf2bb6fda47db786)
After A2 (commit 6c596d8d6) stripped toolName from the primary emit entries, sibling turnIndex:0 fallback entries became unreachable under first-match-wins. The 7 A2-target siblings (ag2/google-adk/lgf/lgts/mastra/msdotnet/csdkts toolName strip) had equivalent dead fallbacks deleted in that commit; csdkts was inconsistently treated. Removing the 2 dead entries restores cross-fleet consistency. Probe contract unaffected — toolCallId-gated narration entries still emit the required content phrase.
(cherry picked from commit 9720636519f4cd858fcdc08ed84597be05604a2e)
crewai-crews/ms-agent-python/pydantic-ai catchall AAPL entries still carried stale $189.42/up 1.27% narration and toolCall args lacking price_usd/change_pct. Round 1 CR finding #3 was identified but never given a fix agent. Brought all 3 to canonical shape matching built-in-agent (A5) and csdkts narration (A6): price_usd=338.37, change_pct=-2.96, narration 'AAPL is trading at $338.37, down 2.96% on the day — rendered through the custom wildcard catchall.'
(cherry picked from commit df84657310451500278d0b3d0125c9c490042d2b)
ag2/claude-sdk-typescript/google-adk/langgraph-fastapi/langgraph-typescript/mastra/ms-agent-dotnet: AAPL is turn-1 so turnIndex:0 fallback unreachable; toolName gate fails on wildcard-renderer integrations that don't register get_stock_price. Aligned to LGP-gold pattern (userMessage+context discriminator, no toolName, no turnIndex:0 fallback). Preserved legitimate multi-pill matchers (SF/flights/d20/chain) on the 4 multi-pill integrations.
(cherry picked from commit c10821b5d904b31bee2ab2a39db3b1565aecba26)
built-in-agent shipped $189.42 vs the rest of the fleet's $338.37. Drift makes any content-asserting test on AAPL price brittle. Aligned.
(cherry picked from commit c0796da2a63abfeaa1d8ea06200df0754d30a2e6)
spring-ai retained hasToolResult:false on the Tokyo weather emit fixture — outlier vs LGP-gold and the other 16 catchall fixtures. Aligned: userMessage+context discriminator only.
(cherry picked from commit d268a88c4a3f405dfcc15b2aaccc190e61cd7ac7)
built-in-agent, crewai-crews, ms-agent-python, pydantic-ai: Tokyo turn-1 tool result makes hasToolResult permanently true → AAPL fixture never matched → 30s timeout. Aligned with agno/langroid/llamaindex/strands/claude-sdk-python pattern: rely on userMessage+context (and toolCallId where relevant) as the gate.
(cherry picked from commit 8e313cd1b043cf1c61efb82143171a28d7699c49)
Closed PR #5465 introduced cross-fixture leakage by stripping the
toolName discriminator on shared {userMessage, context} keys: with
aimock's alphabetical first-match-wins ordering,
tool-rendering-custom-catchall.json sorts before
tool-rendering-default-catchall.json, so default-catchall page requests
were served the custom file's content. The probe was structurally blind
because it asserted only DOM testids (copilot-tool-render +
data-tool-name=get_weather) — both fixtures emit get_weather, so the
testid signal passed regardless of which fixture won.
Real LGP-gold pattern is disjoint userMessages between default and
custom catchall fixtures (NOT shared keys discriminated by toolName).
This PR ports the LGP-gold pattern to the other 17 integrations and
adds page-text content assertions to both d5 catchall probes so this
class of regression can't recur silently:
- Probe prompts disjoint between default-catchall and custom-catchall
- Fixture userMessages updated to match the new disjoint prompts
- Page-text content assertions in both d5 catchall probes
(default negatively asserts the custom-catchall leak phrase;
custom positively asserts it)
Local gold-standard red-green proof captured:
- Step A: live staging Playwright baseline (RED-OF-RECORD)
- Step B: local control-plane on origin/main reproduces structural
fragility (probes pass at testid level, custom fixture wins on
default's userMessage path)
- Step C: local control-plane on this branch — both catchall probes
GREEN with the new content-asserting assertions
tool-rendering-default-catchall: pass=true (5443ms)
tool-rendering-custom-catchall: pass=true (8956ms)
cross-tool signature passed
LGP (langgraph-python) was already disjoint; it remains untouched.
PR #5459 added userMessage matchers ("forecast for Tokyo" + "current price
of AAPL") on the catchall fixtures, which flipped most integrations from
red to green. Six cells stayed red on staging because the FIXTURE SHAPE
itself was broken on the matched fixtures, not just the userMessage key.
Failing cells (all custom-catchall except strands default):
d5:agno/tool-rendering-custom-catchall
d5:llamaindex/tool-rendering-custom-catchall
d5:langroid/tool-rendering-custom-catchall
d5:claude-sdk-python/tool-rendering-custom-catchall
d5:strands/tool-rendering-custom-catchall
d5:strands/tool-rendering-default-catchall
PocketBase confirms the failure mode: turn 1 (Tokyo) completes; turn 2
(AAPL) times out at 30s or renders the wrong content. E.g. agno:
errorDesc: "timeout: assistant did not respond within 30000ms"
failure_turn: 2
turns_completed: 1 / 2
Root cause: the failing tool-emit fixtures used `hasToolResult: false`
(or `toolName: "<tool>"`) as their gate. aimock's hasToolResult check is
`messages.some(m => m.role === 'tool')` over the WHOLE thread — so once
turn 1's Tokyo tool result lands in the conversation, hasToolResult is
permanently true and `hasToolResult:false` can never match turn 2 → no
fixture → 30s timeout. `toolName:get_stock_price` likewise fails when an
integration backend doesn't forward the tool definition on turn 2.
The same trap is documented in
showcase/aimock/d6/langgraph-python/tool-rendering.json:
"_comment": "Gated on toolName:get_stock_price rather than
hasToolResult:false. The D5 tool-rendering-custom-catchall probe runs
'weather in Tokyo' first, which leaves a get_weather tool result in
the thread; the aimock router implements hasToolResult as
messages.some(m=>m.role==='tool'), so hasToolResult is permanently
true on the AAPL turn and a hasToolResult:false gate could never
match (→ no_fixture_match → 503 → 30s timeout)."
Fix: align all 6 files to the canonical pattern used by mastra/spring-
ai/built-in-agent on this probe:
1. toolCallId-keyed narration fixture FIRST
2. tool-emit fixture SECOND with ONLY `userMessage` + `context`
(no hasToolResult / toolName gate)
The toolCallId narration uses aimock's
`messages[last].role === 'tool' && tool_call_id === ...` check, so it
correctly wins on iteration 2 (post-tool-result) without being affected
by older turns' tool results. The tool-emit fixture matches turn 1 (last
message is user) and re-emits only when the narration above hasn't
matched.
Strands' two cells additionally needed REORDERING — they had the tool-
emit fixture before the toolCallId fixture, defeating first-match-wins.
Strands' staging backend was also returning 502 during testing; once it
recovers, the corrected fixtures should let the probe pass. The fixture
changes are necessary but may not be sufficient for strands if backend
remains down.
No probe-side, harness, or backend changes — pure fixture-content
alignment. Six fixture files modified; line totals: -87 / +68.
The catchall userMessage rename (#5453) only renamed STALE strings to 'forecast for Tokyo'. Integrations whose catchall fixtures used PILL-ALIGNED userMessages ('What's the weather in San Francisco?', 'Find flights from SFO to JFK.', 'Chain a few tools in this single turn', etc.) had no 'forecast for Tokyo' matcher to begin with — so the D5 catchall probe still fell through to live LLM and the cells stayed red on staging.
This PR ADDS the canonical 'forecast for Tokyo' (default+custom catchall) and 'What's the current price of AAPL?' (custom catchall only) emit+narrate fixture pairs to each integration that was missing them. Existing pill-aligned fixtures are preserved (pure prepend at the start of the fixtures array; first-match-wins means new fixtures match the D5 probe inputs without colliding with existing pill prompts).
Integrations touched:
- langgraph-typescript: both default + custom
- langgraph-fastapi: both default + custom
- google-adk: custom only (default already green)
- ms-agent-dotnet: custom only (default already green)
After merge, fleet-cp's e2e-deep probe will re-run within ≤6h staleness window and flip cells GREEN.
## Summary
- D5 e2e-deep probes for `tool-rendering-{default,custom}-catchall` send
`"forecast for Tokyo"` as the test input (see
`showcase/harness/src/probes/scripts/d5-tool-rendering-{default,custom}-catchall.ts`).
- aimock uses substring match on `userMessage`.
- The catchall fixtures on main had a stale `"check Tokyo weather
forecast"` string that couldn't substring-match the probe input →
fixture miss → probe falls through to live LLM → CV ✗ red D4 on the
dashboard.
- This PR renames the userMessage to the canonical `"forecast for
Tokyo"` across 28 catchall fixture files in 16 integrations.
## Scope
28 files × ~2 userMessage occurrences each = 67 line changes. **No code,
no agent, no page.tsx changes.** Pure fixture-data alignment.
Integrations covered (default-catchall and/or custom-catchall): ag2,
agno, built-in-agent, claude-sdk-python, claude-sdk-typescript,
crewai-crews, google-adk, langgraph-python, langroid, llamaindex,
mastra, ms-agent-dotnet, ms-agent-python, pydantic-ai, spring-ai,
strands.
## Commit history note
Commit b1f19bdc8 changes the rename target from the original `"weather
in Tokyo"` (commit 388c69e68) to `"forecast for Tokyo"`. This was a CR
Round 1 catch: `"weather in Tokyo"` was a substring of the chain pill
prompt (`"weather forecast chain in Tokyo"` / similar), which would have
caused the catchall fixture to incorrectly match chain-pill traffic.
`"forecast for Tokyo"` has no such substring collision with any other
probe input.
## Verification
- **Live red-green proof** was performed on the prior `"weather in
Tokyo"` rename (commit 388c69e68) against `ms-agent-python` via
`showcase/bin/showcase test ms-agent-python --d6 --isolate`:
- Before (origin/main): catchall featureTypes red (fixture miss → live
LLM → flaky)
- After: `d6:ms-agent-python/tool-rendering-default-catchall=green`,
`d6:ms-agent-python/tool-rendering-custom-catchall=green`
- **The current HEAD's `"forecast for Tokyo"` rename (b1f19bdc8) has NOT
been re-run live.** It is verified by static analysis only: substring
math (no collision with any known D5 probe input or chain pill prompt)
and a clean CR Round 2 across all reviewing agents.
- Post-merge dashboard re-probe will be the final runtime verification.
## Out of scope (separate follow-ups)
- LG-TS / LG-FastAPI catchall fixtures don't have the stale string — use
pill-aligned userMessages; need different fix
- `tool-rendering` (non-catchall) and `tool-rendering-reasoning-chain`
featureTypes have separate failure modes
- Other red cells in the dashboard (frontend-tools-cosmic timeout,
agent-config, auth, etc) are unrelated
The previous 'weather in Tokyo' rename (388c69e68) was a substring of the
main tool-rendering chain pill prompt 'Chain a few tools in this single
turn: get the weather in Tokyo, search flights from SFO to Tokyo, and roll
a d20.' Because aimock loads fixtures alphabetically per integration dir
and uses substring match with first-match-wins, the catchall fixture
(loaded before tool-rendering.json) was intercepting chain pill matches
across 16 integrations.
Rename catchall fixture userMessage to 'forecast for Tokyo' — a phrase
not contained in any other pill prompt. Update the corresponding D5
catchall probe inputs in d5-tool-rendering-{default,custom}-catchall.ts
and the test assertions that pin those inputs.
Call-Site Enumeration: 'weather in Tokyo' remains intentionally in
page.tsx suggestions.ts pills (user-visible UX) and inside the chain
pill prompt itself — neither is in the substring-match path now.
The D5 e2e-deep probe for tool-rendering-{default,custom}-catchall sends
"weather in Tokyo" as the test input (harness/src/probes/scripts/
d5-tool-rendering-{default,custom}-catchall.ts). The fixture userMessage
matcher uses substring match. Main's catchall fixtures had a stale
"check Tokyo weather forecast" string that could not substring-match
the probe input, causing the fixture to miss and the probe to fall
through to the live LLM — surfacing as CV x red D4 on the dashboard.
Rename to the canonical "weather in Tokyo" string across affected
integrations' tool-rendering-{default,custom}-catchall.json files.
No agent or page.tsx changes; fixture content otherwise unchanged.
The custom-catchall probe sends 'weather in Tokyo' then 'AAPL'. After the
weather tool runs in turn 1, hasToolResult is true across the rest of the
thread — which fires tool-rendering.json's AAPL 'hasToolResult:true' narration
prematurely on turn 2 iteration 1, returning prose without ever emitting the
get_stock_price tool. The custom-catchall assertion (both tools rendered
through the wildcard testid) then fails with missing get_stock_price.
Replace the (hasToolResult:true narration + hasToolResult:false/turnIndex:0
emitter) layered fallbacks with a sequenceIndex:0 emitter ordered before a
bare userMessage+context narration. The per-test fixture-match counter resets
each run, so the emitter fires exactly once on iteration 1 regardless of
prior pills' tool history, then falls through to the narration on
iteration 2. Applied symmetrically to the two AAPL blocks in
tool-rendering.json (the 'What\'s the current price of AAPL?' block at the
top and the legacy 'current price of AAPL' alias block lower down). The
toolCallId-keyed narration above each block is retained for the non-BIA
fast path.
The earlier partial fix to tool-rendering-custom-catchall.json is kept (it
adds toolCallId-scoped narration legs ordered before the existing
hasToolResult:true narrations); those fixtures never match real probe
traffic (the probe sends 'weather in Tokyo' / 'current price of AAPL', not
the unique 'check Tokyo weather forecast' substring in this file) but the
reordering is consistent with the cross-file pattern and harmless.
Verified locally: built-in-agent:tool-rendering and
built-in-agent:tool-rendering-custom-catchall both green via
`./bin/showcase test ... --d6 --direct`; built-in-agent:tool-rendering-default-catchall
also green; aimock-fixtures.test.ts (collision/shadow ceilings) unchanged.
Hero loses its surrounding card (bare KPI strip over the chart cards,
pinned to all six months); team performance pairs the rep table with a
quota-attainment bar chart; top account pairs the fact card with a
product-line pie (new dataset entry); at-risk becomes a risk panel — KPI
strip (ARR at risk / accounts / biggest exposure) over three side-by-side
severity cards with reason + next action. Fixtures re-captured from live
responses; D5 probe drops declarative-card from the hero set; e2e asserts
the accompanying charts and the risk panel; QA docs updated.
Replaces the hand-authored surface payloads with real gpt-5.4 responses
captured via the langgraph dev threads API during OSS-136 prompt
iteration (catalogId injected, since the replay path resolves the catalog
from recorded tool args). Verified: fixture suite 738/738, LGP container
e2e 6/6, ADK container e2e 6/6.
Jordan Ritter