## Status
WIP / not ready to merge. Preserves in-flight D6 work so it isn't lost
mid-rollout. LGP is fixture-complete; other integrations are
mid-rollout.
### Latest banked work
- **langgraph-python — 185 / 0 / 2** (green). Achieved by narrowing a d4
chat matcher that was shadowing the d6 beautiful-chat search_flights
fixture (load order is shared -> d4 -> d6, first-match-wins) plus
refreshing the d6 tool-rendering and tool-rendering-custom-catchall AAPL
fixtures (turnIndex:0 -> hasToolResult:false so the first leg fires in
multi-pill threads). 2 skips are the by-design mcp-apps iframe gap.
- **langgraph-typescript — 185 / 0 / 2** (green). Mirrored the LGP d4
narrowing on the LGT side (3 matchers) and across the
d6/langgraph-typescript suite: replaced fragile turnIndex:0 gates with
hasToolResult:false, fixed em-dash escaping that broke literal matches
in multi-pill threads, and added jsFunctions payloads to the three
sandboxed-ui fixtures (`_from-feature-parity`, `headless-complete`,
`gen-ui-open-advanced`). The final fix removed the chain-tools
`hasToolResult` match gate (which checked the whole thread and made the
chain pill fall through to the broad weather matcher mid-thread),
mirroring LGP.
- **google-adk: 174/7/2 (was 134/52/5)** — conveyance + test-parity +
fixtures + pill-wiring rebuild; 7 residual (6 default-catchall framework
default-renderer testid version question, 1 beautiful-chat fixture).
- **Pill-parity staged across 13 integrations** (ag2, agno, mastra,
pydantic-ai, claude-sdk-python, claude-sdk-typescript, llamaindex,
langroid, strands, spring-ai, built-in-agent, crewai-crews,
langgraph-fastapi). The canonical LGP suggestion pill set is now
mirrored as `src/app/demos/*/suggestions.ts` files in each integration,
with targeted edits to existing `open-gen-ui-advanced` and
`byoc-hashbrown` files. **These new files are currently UNWIRED** — each
integration's `page.tsx` still defines its pill list inline via
`useConfigureSuggestions`. Banked so the canonical source survives; a
follow-up will rewire `page.tsx` to import from `suggestions.ts` and
drop the inline copies.
- **Fleet test-parity sweep**: 576 e2e specs across 15 integrations
aligned to LGP canonical (SHA-verified); 2 orphan specs removed.
- ms-agent-dotnet 177/5/7, ms-agent-python 174/11/2 (post
fixture-mirror); default-catchall green (page-level renderer, not
react-core-gated).
## Scope
### Conveyance (foundation)
Inbound `x-aimock-context` (and friends) must ride along on outbound LLM
HTTP calls so aimock fixture matching sees the inflight test's context.
Without this the call lands on the default project's aimock and silently
picks the wrong fixture. New per-integration
`_header_forwarding.{py,ts}` shim plus matching `agent_server` / route /
factory wiring covers: ag2, agno, built-in-agent, claude-sdk-python,
claude-sdk-typescript, crewai-crews, google-adk, langgraph-fastapi,
langgraph-python, langgraph-typescript, langroid, llamaindex, mastra,
ms-agent-python, pydantic-ai, strands.
For ADK/Gemini the global httpx hook is installed BEFORE any `agents.*`
import (google-genai constructs its client at module-import time).
### langgraph-python — 185 / 0 / 2
Fixture-complete via the conveyance shim + refreshed d6/langgraph-python
fixtures + copilotkit 0.1.93 bump + the latest d4-matcher-narrowing fix
(see banked work above).
### Per-integration fixtures
Mid-rollout snapshot of d6 fixtures across the cohort plus narrowing of
`aimock/shared/common.json`'s generic 'hello' fixture to 'hello world'
so it no longer shadows D6 pills whose prompts contain 'hello' as a
substring.
### Harness `--isolate` patch
`scripts/cli/_common.sh apply_isolation` now rewrites compose-file
relative paths to absolute (build/context/dockerfile/volumes/env_file),
enforces the docker compose `[a-z0-9_-]` project-name rule, and exports
`SHOWCASE_COMPOSE_FILE` / `SHOWCASE_INFRA_PORT_OFFSET` plus offset host
URLs. The TS harness CLI (`aimock-rebuild` / `config` / `doctor` /
`lifecycle`) honors the new env so concurrent isolated stacks stop
reporting each other's services as healthy.
## Lockfile decision flagged
`showcase/integrations/langgraph-python/pnpm-lock.yaml` was deleted in
this branch. Decision: keep the deletion. Rationale:
- 03bed3b76 (fix(showcase): regenerate 18 lockfiles in isolation; switch
to npm ci) migrated all showcase integrations off pnpm onto npm ci.
- The integration's Dockerfile uses `npm ci --legacy-peer-deps`.
- Every sibling integration committed only `package-lock.json` after
03bed3b76.
- The orphan pnpm-lock.yaml only risks tooling drift.
If anyone wants it restored: `git checkout origin/main --
showcase/integrations/langgraph-python/pnpm-lock.yaml`.
## Commits
- feat(showcase): D6 conveyance — forward x-aimock-context headers to
LLM clients
- feat(showcase/langgraph-python): D6 conveyance shim + copilotkit
0.1.93 bump
- feat(showcase/langgraph-typescript): D6 conveyance — propagate request
headers into ChatOpenAI
- feat(showcase/built-in-agent): D6 conveyance — header-forwarding shim
+ factory wiring
- feat(showcase/harness): support concurrent --isolate runs
- test(showcase): D6 langgraph-python fixtures — drive to 180/5/2
- test(showcase): D6 per-integration aimock fixtures + shared narrowing
- docs(showcase): GOTCHAS entry for D6 conveyance + --isolate notes
- feat(showcase): D6 conveyance — wire header-forwarding shims into
remaining entrypoints
- fix(showcase): unblock LGP D6 beautiful-chat + custom-catchall via d4
matcher narrowing
- fix(showcase): narrow LGT D6 d4 shadows + wire sandboxed-ui
jsFunctions
- chore(showcase): copy LGP canonical suggestion pills into 13
integrations
- fix(showcase): forward x-aimock-context per-request in google-adk
routes
- test(showcase): align google-adk e2e specs to langgraph-python
canonical
- fix(showcase): align google-adk D6 fixtures to LGP contract
- fix(showcase): wire google-adk default-catchall to shared 4-pill
suggestions
## Summary
Two confirmed bugs in the showcase harness + operator dashboard that
together produced a false-green D3 on the dashboard while the e2e probe
pipeline was actually dead.
### Bug 1 — Browser-pool death spiral
(`showcase/harness/src/probes/helpers/browser-pool.ts`)
When a chromium process crashed, `recycleSlot` relaunched it; if that
launch threw, the slot was permanently removed via `slots.splice`. A
single transient launch failure (OOM spike, fd exhaustion) thus
monotonically shrank the pool to empty with no self-heal, after which
every e2e probe failed with `launcher-error` / `BrowserPool acquire
timeout`.
Fix:
- The recycle retries the relaunch with bounded backoff (3 attempts,
doubling backoff).
- On exhaustion the slot is **kept** (capacity preserved) and parked as
`relaunchPending`; the next `acquire()` lazily re-attempts the launch.
- `acquire()` re-initializes the pool as a backstop when every slot has
been lost (`slots.length === 0`).
- Existing acquire/release/recycle semantics and logging style
preserved.
### Bug 2 — Stale-green e2e rows read as healthy
(`shell-dashboard/src/lib/cell-model.ts`,
`src/components/depth-utils.ts`)
When the e2e driver stops writing `e2e:<slug>/<feature>` rows, the last
row freezes. A green row then reads as a healthy D3 forever, so the
depth ladder shows a false-green D3 that masks a dead probe pipeline
instead of surfacing it.
Fix:
- `buildCellModel` and `deriveDepth` treat a green e2e row whose
`observed_at` is older than `E2E_STALE_AFTER_MS` (6h, matching the
original stale-window model) as degraded/amber, so it no longer credits
D3.
- Only green is downgraded — a stale red/degraded row already signals a
problem and is left as-is.
- An unparseable timestamp is treated as not-stale, so staleness is
never inferred from bad data.
## Test plan
- [x] New unit test: pool recovers after a transient relaunch failure
(does NOT shrink to 0) and re-inits when emptied — confirmed red→green.
- [x] New unit tests: a stale green `e2e:` row downgrades to amber (not
green) and does not credit D3; a fresh green row stays green; a stale
red row stays red — confirmed red→green.
- [x] Existing fixtures with hardcoded old `observed_at` updated to
default to a recent timestamp so legitimate green rows aren't treated as
stale.
- [x] `showcase/harness` full suite: 1646 tests pass.
- [x] `shell-dashboard` full suite: 630 pass (1 env-gated build-spike
skipped).
- [x] `packages/**` test graph: 17 projects pass.
- [x] typecheck clean (both packages); oxlint 0 errors; oxfmt clean.
Note: there is a separate operational lever (reduce `BROWSER_POOL_SIZE`
/ e2e concurrency, raise harness memory) handled outside this PR; this
is the code fix only.
Re-validate each pending slot inside relaunchPendingSlots before claiming it so
two concurrent invocations can't double-launch a slot (leak + double-publish).
Contain recycle throws with a logged catch and gate recycleSlot entry on shutdown.
The deferred-relaunch background timer produced a new race in three review
rounds. Remove it entirely and rely on lazy, on-demand recovery: a failed
recycle parks the slot relaunchPending (never evicts), and the next acquire
re-attempts the launch via relaunchPendingSlots, serving any queued waiter
through handOff. The no-eviction + lazy-recovery design already prevents the
original drain-to-0 bug without a timer. Also clamp stats().inUse to >= 0.
Re-arm the deferred relaunch when a waiter parks after a recycle has
already failed with no waiter queued, so it is not stranded to timeout.
Make scheduleDeferredRelaunch idempotent via deferredRelaunchActive so
the acquire-kick and recycle-kick cannot spawn two timer loops. Guard
recycleSlot entry and the relaunchPending filter with a single
isSlotBusy predicate covering both recyclingSlots and relaunchingSlots,
so a late disconnect for a pending slot's old browser cannot double
launch and leak a process. Strengthen the double-publish test to fail
loud and pin exact-once, and add red-green tests for both fixes.
Reconcile the stale stats()/reinit()/acquire() comments: all-pending recovery
goes through relaunchPendingSlots, not the size-0 reinit backstop (which is the
truly-empty never-launched case). Rename the misleading reinit test to assert
the actual relaunchPendingSlots mechanism. Chain track()'s cleanup so a future
recovery throw cannot float as an unhandled rejection, and log a -1 sentinel
slotIndex for the never-added reinit close. Fix the self-contradicting
drainMicrotasks JSDoc.
loadAllFixtures silently skipped files lacking a fixtures array and let
JSON.parse throw without the path; throw a clear error naming the file in both
cases and attach the caught error as cause. Export the helper and add red-green
tests covering both throw branches.
relaunchPendingSlots only pushed recovered slots to `available` and never resolved
queued waiters, so the deferred relaunch scheduled by a failed recycle (scheduled
precisely because waiters were queued) left the waiter stranded for its full 30s
acquire timeout while a live browser sat idle. It also had no re-entry guard, so an
acquire()-driven call racing the deferred timer could both launch a browser for the
same pending slot — leaking one process and/or publishing the slot to `available`
twice (handing the same browser to two probes).
- Extract a single `handOff` helper (waiter-first, then `available` with an includes
guard) and use it in reinit, the recycle success path, and relaunchPendingSlots so
all three share one correct invariant.
- Add a `relaunchingSlots` re-entry guard: a slot is marked before its launch await
and skipped by concurrent invocations, cleared in finally.
- Replace the one-shot deferred relaunch with a self-rescheduling timer (bounded
backoff) that retries while pending slots and waiters coexist, so a waiter is no
longer stranded when a single deferred attempt also fails. Stops on shutdown.
- Track reinit/relaunch launch promises so shutdown() drains them, closing the leak
where a launch resolved after shutdown awaited the in-flight set.
- Route failure and close logging through the injected logger (with slot index)
instead of console.error / swallowed catches, so failures reach the harness pipeline.
- stats().size now counts only live (non-pending) capacity, making the empty-pool
backstop observable (size 0 when every slot is pending).
Tests: fix the vacuous reinit test (failAtCalls now exhausts all retries for every
slot and asserts size 0 before recovery), and add deferred-waiter recovery and
no-double-publish tests. Full harness suite green (1648 tests), typecheck green.
When a chromium process crashes, recycleSlot relaunches it; if that launch
threw, the slot was permanently removed via slots.splice. A single transient
launch failure (OOM spike, fd exhaustion) thus monotonically shrank the pool
to empty with no self-heal, after which every e2e probe failed with
launcher-error / BrowserPool acquire timeout.
The recycle now retries the relaunch with bounded backoff; on exhaustion it
keeps the slot (capacity preserved) and parks it as relaunchPending for a lazy
retry on the next acquire. acquire() also re-initializes the pool as a backstop
when every slot has been lost (slots.length === 0), so the pool always
self-heals rather than wedging.
- scripts/cli/_common.sh apply_isolation rewrites compose-file relative
paths to absolute (build/context/dockerfile/volumes/env_file), enforces
the docker compose [a-z0-9_-] project-name rule (normalize-with-warn or
hard fail), and exports SHOWCASE_COMPOSE_FILE / SHOWCASE_INFRA_PORT_OFFSET
plus offset host URLs (AIMOCK_URL_LOCAL / DASHBOARD_URL_LOCAL /
POCKETBASE_URL_LOCAL) so the TS harness CLI talks to THIS project's
aimock instead of the default :4010
- harness/src/cli/{aimock-rebuild,config,doctor,lifecycle}.ts honor the
new env vars; lifecycle picks up the offset infra port for health
probes so concurrent stacks no longer report each other's services as
healthy
- Commit a generated harness/package-lock.json (new file) so npm ci
resolves deterministically
Repoint the aimock fixture-coverage probe to the per-integration d4/d6/shared layout; align
d5-multimodal assertions to the current auto-send button sentinels.
the integration was added to the railway-services SSOT and smoke.yml nameExcludes; this
adds it to the sibling probe configs the probe-config-parity invariant requires to stay
in lockstep.
The "Traffic pie chart" suggestion ("Show me a pie chart of website traffic
by source.") names a subject but supplies no numbers, so the agent asked the
user for data instead of rendering. Add a system-prompt directive (LGP + ADK)
telling the agent to invent illustrative sample values and render on the first
turn, never asking for data. The suggestion copy stays clean — the behavior is
carried by the system prompt, not parenthetical UI hints.
Also retag the gen-ui-tool-based demo (LGP + ADK) from `generative-ui` to
`controlled-generative-ui` so the dojo sidebar pill reads "Controlled
Generative UI" — the established product taxonomy (already a category in
shared/feature-registry.json and the dashboard catalog).
Scoped to LGP and ADK per the ticket; the other 16 integrations keep the old
tag until the taxonomy rolls out wider.
Tests: add D5 aimock fixture entries mirroring all three suggestion chips
(bar/traffic-pie/market-share) so the suggestion-click path has deterministic
coverage. The existing "revenue by category" probe message is preserved, so
the dashboard D5 row stays green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- qa probe: manifest.yaml path corrected from showcase/packages/ to
showcase/integrations/ (Slice 3 moved files; probe code and test
fixture still referenced the old path).
- pin-drift probe: fail-baseline.json ENOENT in container — the 5-level
import.meta.url walk-up from dist/probes/drivers/ overshoots /app and
lands at /. Added PIN_DRIFT_REPO_ROOT=/app env var and COPY of the
baseline file into the runtime stage.
- aimock-wiring probe: showcase-ms-agent-harness-dotnet (deployed: false)
was not in the EXCLUDE_SERVICES set, causing it to be flagged as
unwired. Added to the exclude list in both aimock-wiring.ts and
smoke.yml.
Add d6-all-pills driver that exercises every pill on every integration
behind aimock context routing. Rename e2e-chat-tools/e2e-deep to
d4-chat-roundtrip/d5-single-pill to match depth convention. Wire
D5 representative map, D6 scoping helpers, dimension types, and
the orchestrator config for the new probe IDs.
Production-fix bundle for beautiful-chat + 3 other google-adk demos. All
issues either reported on Railway prod or unmasked by the catch-all
render_a2ui fallback I added in #4836. Local D5 stays 38/38 green.
- Missing copilotkit logo: `<img src="/copilotkit-logo-mark.svg">` returned
404 because the SVG never shipped in the integration's public/. Copy
copilotkit-logo-mark.svg + copilotkit-logo.svg over from langgraph-python.
- Multimodal sample.png / sample.pdf were LFS pointers in prod (Railway
build runs without `git lfs pull`), so the magic-byte check rejected them
on first click. Add an integration-scoped .gitattributes that exempts
these two paths from LFS (mirrors what every working sibling integration
already does) and re-stage the files as real binaries (10KB / 2.5KB).
- Sales Dashboard pill returned a generic "Step 1... Step 2..." narration
instead of rendering the A2UI surface. Root cause: the render_a2ui
catch-all fallback I added to aimock/d5-all.json in #4836 fired before
feature-parity.json's specific sales-dashboard fixture (load order
d5-all → smoke → feature-parity). Removing the catch-all from both
d5-all.json and the per-demo source so the specific fixture wins.
- Calculator App pill returned text only with a white iframe because
beautiful-chat's agent instruction never mentioned generateSandboxedUi —
Gemini saw the tool listed via AGUIToolset but had no nudge to use it.
Added a one-line "Interactive / sandboxed widgets" entry to the
instruction; verified Gemini now emits the tool call.
- hitl-in-app refund (#12345) and escalate (#12347) pills broke on the
second click because the 2nd-turn fixtures keyed on `sequenceIndex` 0/1
(a global thread-position counter that drifts when other pills land
tool messages in the same thread). Convert both to `toolCallId +
hasToolResult: true` and drop the reject branch — Gemini reasons
correctly from the tool's `approved: false` return without a fixture
override. Same pattern that fixed tool-rendering-reasoning-chain
previously.
- hitl-in-app downgrade (#12346) pill produced an unrelated
"Research / Outline / Draft / Review / Finalize" plan because the
prompt contains the substring "plan" and feature-parity.json has a
generic catch-all match on `userMessage: "plan"`. Add a specific
downgrade fixture in d5-all.json (loaded before feature-parity.json)
with hasToolResult: false / true branches.
- hitl-in-chat "Schedule a 1:1 with Alice" returned the wrong
"Nice to meet you, Alice in Tokyo" response when clicked AFTER another
pill in the same thread. The 2nd-turn fixture only matched on
`toolCallId` (no hasToolResult), so the bare "alice" / "Alice" greeting
fixtures further down won. Add `hasToolResult: true` to the 2nd-turn
fixture so it scopes correctly regardless of thread state.
- Voice manual recordings always returned "What is the weather in Tokyo?"
regardless of audio content. aimock had a catch-all transcription fixture
(`match: { endpoint: "transcription" }`) that returned the canned
Tokyo string for any audio input. The D5 voice probe uses the sample
audio button which bypasses /transcribe entirely (it injects text
directly into the composer), so removing the transcription fixture
drops aimock into --proxy-only fall-through to real OpenAI Whisper for
mic recordings while D5 stays green. Verified.
- readonly-state-agent-context and shared-state-read-write fixtures
returned hardcoded "Atai" / generic preferences responses even when
the user changed the state values in the UI inputs. Gate the
Who-am-I / Suggest-next-steps / Greet / Plan-a-weekend fixtures on
systemMessage substring matching the canonical default state values
("Atai" name for readonly-state-context; "tone: casual" for
shared-state-read-write). When the user changes state, the agent's
before-model callback rebuilds the system prompt with the new values,
the fixture's systemMessage substring no longer matches, and aimock
--proxy-only falls through to the real model so the response reflects
the actual state. Confirmed with paired curl tests (default state =
fixture match; alem name = real-LLM response).
Local verification: bin/showcase test google-adk --d5 finishes 38/38
green (137s). Calculator pill confirmed via direct ADK invocation
against real Gemini (GOOGLE_GEMINI_BASE_URL=) emits TOOL_CALL_START
toolCallName=generateSandboxedUi. Sales-dashboard pill confirmed end-to-end
returning the full Column / DashboardCards / PieChart / BarChart payload.
readonly-state-context confirmed with name=Atai matching fixture vs
name=alem falling through to real-LLM response that uses the actual
context.
Five distinct root causes were keeping google-adk from full D5 parity
with langgraph-python. Fixing them takes the integration to 38/38 D5
green under aimock locally (verified end-to-end with --live writing
to PocketBase).
- readonly-state-context: page.tsx asked for agent slug
readonly_state_agent_context (underscore) but the registry mounts
it kebab-case as readonly-state-agent-context. useAgent threw,
the demo layout never mounted, and the ctx-name input never rendered.
Align with the registry (and with langgraph-python).
- multimodal: the secondary failure was a backend Pydantic
ValidationError on HttpOptions.api_endpoint. google-genai 1.75
renamed the field to base_url; all three direct genai client
constructors (main.py, beautiful_chat_agent.py, subagents_agent.py)
needed the rename so the secondary A2UI / sub-agent LLM calls stop
crashing. (The pre-existing LFS-pointer issue on the bundled
sample.png/pdf was a worktree hydration problem, not a tracked
code change — git lfs pull handles it.)
- gen-ui-declarative (two bugs stacked):
1. The same api_endpoint -> base_url rename above. The secondary
generate_a2ui planner LLM was failing every request.
2. The D5 fixture emitted components in {id, type, props: {...}}
shape, but sanitize_a2ui_components requires component, so
every entry was dropped and the renderer received an empty
surface. Rewrite both the per-demo fixture and the d5-all.json
aggregate to the flat {id, component, ...props} shape that
langgraph-python's _design_a2ui_surface fixture already uses,
wrapping multi-child layouts in a basic-catalog Column (the
custom Card schema has a single child slot). Add
_design_a2ui_surface variants so LGP gets per-pill payloads too.
- shared-state-streaming: ADK's write_document took content and
the PredictStateMapping read tool_argument="content", but the
shared D5 fixture (and the LGP function signature) names the
argument document. Rename both sides so the fixture's tool_call
args plumb into the function and into PredictStateMapping's
state-key emission. STATE_DELTA now propagates and DocumentView
streams live.
- tool-rendering-reasoning-chain: in thinking mode
(include_thoughts=True), Gemini emits a turn as two separate
non-partial chunks — a text-only chunk with finish_reason=None
and a function-call-only chunk with finish_reason=FUNCTION_CALL.
stop_on_terminal_text fired on the first (text-only) chunk and
set end_invocation=True before the function-call chunk arrived,
which broke AAPL->MSFT chaining. Gate termination on
finish_reason=STOP; FUNCTION_CALL and None both mean "more
chunks inbound — defer". Applies to every agent that uses the
shared callback, so chain-aware behavior is uniform.
Local verification: bin/showcase test google-adk --d5 --live
finishes green for all 38 cells (~140s), dashboard reflects the
results from PocketBase. Manual real-Gemini click-through of the
five fixed demos also passes end-to-end via GOOGLE_GEMINI_BASE_URL=
(empty) recreate.
Wraps railwayServicesSource with withCache (24h TTL). Instantiates
DiscoveryAuthTracker with threshold 3. Adds system dimension. Caches
listServices in Railway adapter (60s TTL). Writes system status on
browser pool init failure so degradation is visible in the dashboard.
Tracks auth failures per source since last success. After 3 failures,
writes system:discovery-auth-failed to PocketBase. Sustained alerts
rate-limited to one PB write per 5 minutes. Auto-recovers on next
success. Non-auth errors are no-ops. 9 test cases.
Transparent wrapper at the DiscoverySource interface level. Caches
successful enumerate() results in memory (24h TTL), serves stale
data on upstream failure, collapses concurrent callers into a single
upstream request. Auth tracker side-effects are try-caught to never
block the primary data path. Evicts entries older than 2x TTL.
18 test cases covering success, failure, TTL, collapse, eviction,
non-JSON config guard, and tracker integration.
Three layers of regression guards for the runtime reasoning-role filter and
the chained demo behavior:
1. Runtime unit test — packages/runtime/.../run-message-filtering.test.ts:
- Verifies `LangGraphAgent.run` strips `role:"reasoning"` from
`input.messages` before delegating to super.run.
- Verifies user/assistant/system/tool messages pass through in order.
- Verifies empty + missing messages arrays are tolerated.
- Verifies pre-existing forwardedProps.streamSubgraphs default + override
behavior is preserved.
- 6/6 tests pass against the runtime package's vitest config.
2. D5 harness probe — showcase/harness/.../d5-tool-rendering-reasoning-chain.ts:
- Expanded from one chained turn (flights→weather) to all three chained
pills in a single thread (stocks AAPL→MSFT, dice d20→d6, flights→weather).
- This is the canonical multi-pill regression at the harness layer:
without the runtime reasoning-role filter, the second pill would crash
before the model was called.
- Each turn asserts the per-turn delta of reasoning-block mounts (idx+1),
the minimum card count for each tool group, and unique transcript
substrings that scope to that turn.
3. Playwright e2e spec — showcase/integrations/langgraph-python/tests/e2e/
tool-rendering-reasoning-chain.spec.ts:
- Mirrors the pattern of the sibling tool-rendering-default-catchall spec
(notably its multi-pill regression at lines 162-212).
- Page-loads test verifies the 3 pills mount and no cards leak from a
prior session.
- One test per chained pill (stocks, dice, flights+weather) asserts the
full chain renders with reasoning-block + correct per-tool cards +
narration matching the aimock fixture text.
- Sequential-pills regression test clicks all 3 pills in one thread,
asserts each chain renders independently AND the reasoning-block count
increases monotonically across turns.
Agent: extend `get_stock_price` to accept optional `price_usd` and
`change_pct` arguments (mirrors the basic tool-rendering agent's signature
introduced in #4770). The aimock fixtures script the chained AAPL/MSFT
comparison by passing deterministic prices via these args; without the
wider signature, pydantic rejects the tool call and the card never mounts.
The runtime unit test is the strongest guard — it would catch any
regression on the role-filter logic without depending on the full Docker
stack. The harness probe and Playwright spec catch end-to-end regressions
in the canonical CI environment.
The tool-rendering-reasoning-chain demo previously promised chained tool
calls in its pill titles but the agent and fixtures only delivered single
tools — clicking "Weather + flights to Tokyo" produced just a WeatherCard,
"Compare two stocks" only fetched AAPL, "Find flights from SFO to JFK"
showed flights but no destination weather. Three changes close the gap.
Agent: replace the soft "call 2+ tools when relevant" system prompt with
concrete per-pill chain examples mirroring the pattern already used by the
langgraph-typescript `tool-rendering` agent (weather→flights, ticker→peer,
roll→contrast die, flights→destination weather).
Pills: drop the redundant Tokyo pill (it was the SFO/JFK chain in reverse)
and reword each remaining pill message to PRE-DISCLOSE the chain so the
model commits to the follow-up call:
- "Compare AAPL and MSFT stocks for me."
- "Roll a 20-sided die for me and compare it to a smaller one."
- "Find flights from SFO to JFK and show me the weather there."
Fixtures: 9 fixtures (3 per pill: final-content → second-leg → first-leg,
ordered by toolCallId specificity for first-match-wins). Each fixture is
scoped by a langgraph-python-UNIQUE userMessage tail ("Compare AAPL and
MSFT stocks", "compare it to a smaller one", "show me the weather there").
Those substrings appear nowhere else across the 14+ integrations sharing
showcase-aimock on Railway, so the new fixtures cannot cross-contaminate
the other reasoning-chain demos that still ship the older prompt set.
A toolName-based gate was considered and rejected because most fleet
agents register `roll_dice` and aimock's `toolName` matcher is a tool-LIST
gate, not a tool-CALL gate — it would NOT have isolated this demo.
Probe: collapse the two-turn flow (Tokyo + SFO/JFK) into one chained turn
(SFO→JFK + JFK weather) that asserts BOTH per-tool renderers
(FlightListCard + WeatherCard) mount in a single response. Same coverage
at half the wall-clock and exercises the actual chained-tool path.
Four independent showcase production bugs Alem reported, plus the
D5 multimodal harness regression they unblocked.
Shared-state-read-write: "Greet me" ("Say hi and introduce yourself.")
and "Plan a weekend" ("Suggest a weekend plan based on my interests.")
were matching the bare `hi` and `plan` catch-alls in feature-parity.json
and returning the generic showcase-assistant blurb / 5-step content plan
instead of shared-state-aware responses. Added pill-specific fixtures in
shared-state.json (mirrored into d5-all.json) so the longer userMessage
substrings win first-match-wins ahead of feature-parity.
Auth sign-out: signing out unmounted CopilotKit entirely and bounced
the user back to the SignInCard, so the demo never showcased the
runtime returning 401 — its whole point. The QA contract in
qa/auth.md spelled out the intended UX. Restored it: CopilotKit stays
mounted after the first sign-in, the AuthBanner flips to an amber
"Signed out — the agent will reject your messages" state with a
re-Sign-in button, and CopilotKit's `onError` callback drives a
`data-testid="auth-demo-error"` surface that displays the runtime's
401 the moment the user sends an unauthenticated message. Updated the
e2e spec to match (the old "SignInCard re-mounts after sign-out" test
pinned the regression).
Gen-ui-agent: the aimock fixture short-circuited the 7-step
progression spelled out in `gen_ui_agent.py`'s SYSTEM_PROMPT to a
single set_steps call with all three steps already `completed`, so
the InlineAgentStateCard rendered the final 3/3 state instantly with
no sequential pending → in_progress → completed animation.
Regenerated as a 7-leg toolCallId chain per pill (8 fixtures × 3
pills): seed leg keyed on userMessage with NO `hasToolResult` gate
(matching PR #4770's pattern — `hasToolResult: false` would block the
seed from firing on the second pill in a multi-pill session), then
six toolCallId-keyed transitions, then a final narration. Fixture
order: toolCallId legs FIRST so the most specific match wins.
Multimodal D5: the sample-attachment buttons auto-send via
`agent.addMessage + copilotkit.runAgent` (restored in PR #4761), but
the D5 harness still typed `input` + pressed Enter via the runner
after `preFill`, sending a second user message that competed with the
in-flight image upload — the v1 LangGraph runtime SSE stream got
tangled (browser DevTools showed `statusCode: pending` indefinitely)
and the assistant message never rendered. Added `skipSend?: boolean`
to ConversationTurn (distinct from `skipFill`, which still presses
Enter once the textarea has content) and switched d5-multimodal.ts to
`skipSend: true` with `responseTimeoutMs: 60_000` so the runner waits
on the assistant response without poking the chat further. Bumped the
PDF auto-prompt fixture in feature-parity.json to include the word
"document" so the existing `buildModalityAssertion("document")` check
still lands.
D5 result: 37 → 39 of 40 features passing. Only
`tool-rendering-reasoning-chain` remains and is a separate
agent/runtime bug (Tokyo Responses-API `reasoning` message survives
into the next turn's conversation history, runtime returns
`RUN_ERROR: "message role is not supported"`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Probes now send X-AIMock-Strict: true on all requests via
extraHTTPHeaders, triggering strict fixture matching in aimock.
Live demo traffic (no header) continues to proxy to real LLMs.
The "Draw a flowchart" suggestion pill in the mcp-apps demo sent
"Use Excalidraw to draw a simple flowchart with three steps." which
had no matching create_view fixture in d5-all.json. aimock walked
through to feature-parity.json's `{userMessage: "steps"}` substring
fixture and returned a generic "Here is my plan..." content blurb
with no MCP tool call, so the runtime never invoked create_view, the
MCP middleware never fetched the UI resource, and the sandboxed
iframe never mounted.
Add a fixture pair in d5-all.json (and its harness mirror) keyed on
"draw a simple flowchart": turn 1 emits create_view with a three-
step Start -> Process -> End flowchart, turn 2 emits the narration
after the tool result. The distinctive substring beats the generic
feature-parity catch-alls under first-match-wins.
Adds a regression test that loads the same fixture files in the same
order as docker-compose.local.yml and asserts via aimock's matchFixture
that each mcp-apps pill routes to its create_view fixture on turn 1 and
its narration fixture on turn 2.
## Summary
Follow-up to #4723. The post-merge cycle showed
`gen-ui-headless-complete` still red with `page.click: Timeout` waiting
for the aria-label selector I added in #4723.
Root cause: the demo has TWO chip surfaces with diverging aria-label
shapes:
- **EmptyState** (first paint, before any messages): `aria-label="Try
suggestion: <message>"`, visible text = the message string.
- **SuggestionBar** (after first message lands):
`aria-label="Suggestion: <title>"`, visible text = the short title.
Which surface is mounted depends on `messages.length` —
timing-dependent. Both my prior selectors (`button >> text="Weather"`
and `button[aria-label="Suggestion: Weather"]`) matched only one surface
and timed out on the other.
Fix: drop the chip-click preFill entirely. The runner's normal
fill+press hits the textarea with the verbatim prompt; chip clicks and
textarea-Enter dispatch the same `runAgent`, so the fixture matcher
catches either route — and the textarea is always present.
Also includes a small refresh of `e2e-deep.test.ts` for a stale
assertion (Phase-2A REGISTRY_TO_D5 split renamed `tool-rendering` →
`tool-rendering-default-catchall`).
## Test plan
- [ ] Land + watch the next e2e-deep cycle
- [ ] Confirm `gen-ui-headless-complete` flips green
🤖 Generated with [Claude Code](https://claude.com/claude-code)
D5 probes were red for two demos because the aimock fixtures didn't
exist or were dispatched through the wrong Responses-API path.
- `harness/fixtures/d5/mcp-apps.json` (new) — the MCP Apps probe sent
"Open Excalidraw and sketch a system diagram" with no matching
fixture, so aimock 404'd and the agent threw `NotFoundError`.
Added a two-leg fixture: leg 1 emits a `create_view` MCP tool call
with Excalidraw element JSON; leg 2 is the `hasToolResult: true`
content reply. Mirrored verbatim into `aimock/d5-all.json` so the
Docker-baked aimock has the same matches.
- `harness/fixtures/d5/tool-rendering-reasoning-chain.json` — added
non-empty `content` fields to the first-leg fixtures (Tokyo weather
+ SFO/JFK flights). aimock's Responses-API dispatch routes
tool-call-only responses (no `content`) through
`buildToolCallResponse`, which silently drops the `reasoning`
payload; with a non-empty `content`, dispatch shifts to
`buildContentWithToolCallsResponse`, which emits the
reasoning_summary events the v2 `<ReasoningBlock>` needs to mount.
Mirrored into `d5-all.json`.
- `harness/fixtures/d5/reasoning-display.json` — added a second
matcher for the `reasoning-default` e2e pill prompt ("sky appears
blue"), keyed alongside the existing D5 probe matcher ("show your
reasoning step by step"). Both responses include `reasoning` fields
so the built-in `CopilotChatReasoningMessage` "Thinking…/Thought
for…" header lands.
- `aimock/feature-parity.json` — added a fixture for the agentic-chat
multi-turn name-recall test ("What name did I just give") replying
"You said your name is Alice." The matcher is intentionally narrow
to avoid colliding with the generic showcase-assistant catch-all.
Net D5 effect: `mcp-apps` and `tool-rendering-reasoning-chain` turn 1
go from red → green. The remaining red on `tool-rendering-reasoning-
chain` turn 2 is the deeper aimock+deepagents+Responses-API multi-
turn state-management issue (production with real OpenAI works); out
of scope per the original audit's Phase F.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User-facing renames so the showcase reads the way a cold visitor would
expect:
- `byoc-hashbrown` → `declarative-hashbrown` (and `byoc-json-render` →
`declarative-json-render`). The display titles already said
"Declarative UI: …"; only the URL slugs and folder paths still
leaked the internal BYOC ("Bring Your Own Components") jargon.
Renamed:
/demos/byoc-hashbrown → /demos/declarative-hashbrown
/demos/byoc-json-render → /demos/declarative-json-render
/api/copilotkit-byoc-* → /api/copilotkit-declarative-*
src/app/demos/byoc-* → src/app/demos/declarative-*
qa/byoc-*.md → qa/declarative-*.md
tests/e2e/byoc-*.spec.ts → tests/e2e/declarative-*.spec.ts
Internal Python module names + langgraph graph IDs stay legacy
(`byoc_hashbrown_agent.py`, `byoc_hashbrown`) — those are not
user-facing and renaming them is a separate cross-codebase pass.
- `a2ui-fixed-schema` slug intentionally unchanged.
- Tool Rendering trio parenthetical rename (Default → Catch-all →
Custom progression reads clearly as "how much do I customize?"):
Tool Rendering (Default) — unchanged
Tool Rendering (Custom default) → Tool Rendering (Catch-all)
Tool Rendering (Specific) → Tool Rendering (Custom)
- `tool-rendering-reasoning-chain` cell renamed from
"Generative UI: Rendering multiple tools" to
"Generative UI: Tool calls + reasoning" (the demo is about combining
reasoning + tool rendering, not about quantity of tools).
- `Open Generative UI: Default` / `Open Generative UI: Custom`
descriptions expanded so a visitor understands how Open Generative UI
differs from Tool Rendering (agent composes UI from a registered
library vs. attaching a renderer to a *named* backend tool).
- Showcase index now sorts demos within each tag by `manifest.features`
order. Previously demos appeared in manifest declaration order, which
ignored the team's curated "polished flagship → simplest start →
variants" arc.
Cross-cutting registry / harness / dashboard updates that fall out of
the rename:
- `shared/feature-registry.json` adds the two new IDs alongside the
legacy `byoc-*` (so the catalog stays valid; the other 17
integrations still declare `byoc-*` in their manifests).
- `shared/constraints.yaml` adds the new IDs to the
generative-ui-approach allow-list.
- `scripts/__tests__/generate-catalog.test.ts` updates the cell-count
expectations (45 features × 18 integrations = 810; 792 after docs-
only exclusion; 45 LGP cells = 38 wired + 1 stub + 6 unshipped).
- Harness probe `d5-byoc.ts` + `d5-byoc.test.ts` now route both slug
families through `preNavigateRoute` and exercise the new branches.
- `d5-feature-mapping.ts` and `shell-dashboard/live-status.ts` mirror
the dual-ID mapping so both legacy and renamed slugs roll up under
the same `byoc` D5 featureType.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
LGP probe hung on Turn 2 (Find flights from SFO to JFK) after Turn 1
(weather Tokyo) succeeded. Root cause: the SFO/JFK first-leg fixture
matched on `hasToolResult: false`, but Turn 1's tool result remains in
the conversation by Turn 2, so the matcher skipped it and fell through
to the second-leg fixture (`hasToolResult: true`), returning narration
without the search_flights tool call. The probe then timed out waiting
for the FlightListCard testid that never mounted.
Fix: switch the second-leg fixture to `toolCallId` matching (the last
message must be a tool result with the specific call_id) and drop
`hasToolResult` from the first-leg fixture. The second-leg fixture
must come BEFORE the first-leg in file order because the matcher is
first-match-wins; with `toolCallId`, it cannot accidentally swallow
first-leg requests (whose last message is the user prompt, not a
tool result).
Also adds a `content` field to the Tokyo first-leg fixtures
(`get_weather` and `get-weather` Mastra-hyphen variants) so they route
through `buildContentWithToolCallsStreamEvents` instead of
`buildToolCallStreamEvents` — the latter codepath silently drops the
`reasoning` field, which is why the reasoning-block testid never
mounted on Turn 1 either before this change.
Tokyo (Turn 1) keeps the simpler `hasToolResult` pattern because it has
no prior turns; only the SFO/JFK fixture pair needs the toolCallId
shape to survive multi-turn use.
The e2e-readiness probe waits for any of a compound CSS selector
covering chat-input testids. /demos/auth renders a <SignInCard>
BEFORE mounting the chat surface (the chat-input testids only appear
post-authentication), so the readiness probe was timing out at 30s
with `per-demo deadline exceeded` and the auth cell rolled up to D2
even though the demo itself works fine end-to-end.
Add `[data-testid="auth-sign-in-card"]` to READY_SELECTORS so the
probe accepts the SignInCard as a valid "demo mounted" signal. The
end-to-end auth flow (sign in -> sign out -> sign back in -> assistant
response) is still covered by auth.spec.ts under the e2e-demos
dimension, so the readiness probe shouldn't be doing that work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three fixes that follow up on PR #4743 to bring LGP closer to fully-green
on the dashboard:
1. tool-rendering-reasoning-chain probe: was failing with
`expected [data-testid="reasoning-block"] to mount within 30000ms`.
Root cause: the demo's `<ReasoningBlock>` slot only mounts when a
reasoning-role message lands in the transcript, which requires
aimock to emit REASONING_MESSAGE_* events, which in turn requires
the fixture's first-leg response to carry a `reasoning` field. The
weather/Tokyo and SFO/JFK first-leg fixtures were missing it.
Mirrors the convention documented in reasoning-display.json:2.
Patched both source (harness/fixtures/d5/) and bundle (aimock/d5-all.json).
2. gen-ui-interrupt source fixture: the source fixture file was missing
the resume-leg toolCallId entries that already existed in the bundle.
Cosmetic mirror so re-bundling stays consistent. Same chip prompts +
same toolCallIds as interrupt-headless.json (both probes share the
same agent and aimock fixture set; the difference is the FRONTEND
rendering — useInterrupt inline vs useHeadlessInterrupt separate-pane).
3. Dashboard gold-standard filter: 4 deprecated/legacy features
(agentic-chat-reasoning, hitl, hitl-in-chat-booking,
reasoning-default-render) used to render as X-marked rows in the
LGP gold-standard dashboard view because LGP intentionally does
NOT implement them — they were consolidated into the modern shape
(reasoning-custom + reasoning-default; hitl-in-chat with
useHumanInTheLoop). Other 17 integrations still serve those legacy
demos, so we don't yank the features from feature-registry.json
entirely. Instead: marked them `deprecated: true` and updated
generate-registry.ts to skip emitting cells when a deprecated
feature is unshipped for an integration. LGP cells: 43 → 39 (the
4 deprecated rows disappear). Other integrations: unchanged
(audit trail preserved). Catalog total: 774 → 770.
Tests:
- 1588/1588 harness vitest passing
- 19/19 generate-catalog + generate-registry tests passing
(counts updated for the 4 dropped LGP cells + new deprecated-
feature filter test)
- validate-fixture-tool-surface clean (282 fixtures × 627 demos)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the demo↔probe coverage gap for /demos/{interrupt-headless,
shared-state-read, tool-rendering-reasoning-chain} so every demo
under langgraph-python (the north-star integration) now has a D5
probe writing to its own PocketBase cell — not relying on cross-
demo umbrella records.
New probes (multi-turn, mirroring the agentic-chat structure):
- d5-interrupt-headless: exercises useHeadlessInterrupt — chip
prompt → backend interrupt(...) → app-surface popup → slot pick
→ resume → assistant confirmation. Distinct from gen-ui-interrupt
(which uses inline useInterrupt).
- d5-tool-rendering-reasoning-chain: combines reasoning-block slot
+ per-tool renderer (WeatherCard, FlightListCard) on the same
chat surface. Catches a regression in either side.
- d5-shared-state-read: recipe-editor demo (neutral default agent,
no tools) — verifies recipe-card form mounts AND agent reads
shared state across turns. Drops the dual-claim that
d5-shared-state.ts had on `shared-state-read` (now write-only).
Driver retry-once (e2e-deep.ts):
Probes that fail with a transient class (`goto-error` /
`conversation-error`) AND took ≥2s on the first attempt now retry
once before recording red. Persistent assertion-style failures
(sub-2s) and intentional aborts/feature-timeouts skip retry —
retrying a deterministic mismatch just burns clock and obscures
the signal. Cuts ~10× the dashboard flap rate.
Plumbing:
- D5FeatureType enum: +interrupt-headless, +tool-rendering-reasoning-chain.
- REGISTRY_TO_D5 (harness) + CATALOG_TO_D5_KEY (dashboard) mirror
the new mappings; d5-mapping-drift test enforces this.
- LGP manifest features + demos entries + constraints allowlist.
- feature-registry.json: +shared-state-read.
- aimock d5-all.json: +2 shared-state-read fixtures (interrupt-
headless + tool-rendering-reasoning-chain reuse existing fixtures
that already match their chip prompts).
Tests: 1588/1588 harness vitest green. validate-fixture-tool-surface
clean (282 fixtures × 627 demos, no drift). Two pre-existing test
fixes folded in — d5-gen-ui-interrupt assertion mock updated to
match the current evaluate-poll resume signal; conversation-runner
preFill ordering test now asserts the actual deferred-cascade
contract instead of a stricter pre-preFill ban that the runner
never enforced.
Known follow-up (not in this PR): auth.spec.ts test #5 ("signing
back in re-mounts a fresh chat surface") fails on Railway — second
sign-in's "Hello again" never produces an assistant response. Looks
like a react-core/v2 ref-handling regression on <CopilotKit>
unmount/remount; deserves its own focused investigation.
Other integrations may flip red on the new probes — that's
expected. We're treating LGP as the template; cross-integration
parity follows in a separate wave.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands the Bucket A framework fix and the fixture-correctness changes
needed to flip the remaining D4/D2 cells in the langgraph-python column
to D5. Full local D5 sweep is green (1 passed, 0 failed).
Framework: every Python `ToolMessage` constructed via `Command(update=...)`
now sets `name=` and `id=str(uuid.uuid4())`. Without these, @ag-ui/langgraph
synthesises TOOL_CALL_START events with `toolCallName: null` and
`parentMessageId: null`, which @ag-ui/client@0.0.53's Zod schema rejects;
the rejection is silently swallowed by `withAbortErrorHandling -> EMPTY`,
completing the SSE observable mid-stream so post-tool state never reaches
the consumer. Fix is applied across shared_state_streaming, shared_state_read_write,
gen_ui_agent, beautiful_chat, subagents (2 sites). Single-flag change in
a2ui_dynamic flips the secondary `_design_a2ui_surface` LLM call to
`streaming=True` so aimock's record/replay (SSE-only) sees it.
Fixtures (d5-all.json):
- toolCallId follow-ups for set_steps (3), display_flight, generate_a2ui (4),
schedule_meeting (2), generateSandboxedUi (7), and revenue chart so
multi-turn probes don't recurse into recursion-limit loops
- four hand-crafted secondary `_design_a2ui_surface` fixtures so A2UI
dynamic renders without a real LLM
- mcp-apps fixture rewritten to emit `create_view` tool call with a
minimal Excalidraw element payload; runtime middleware fetches the UI
resource and the iframe mounts
- AAPL and revenue `hasToolResult: true` follow-ups tightened to
`toolCallId` so they don't match cross-turn after prior turns' tool
results
- voice fast-path content-only fixture
- beautiful-chat-schedule-meeting first-turn fixture gains content so
the conversation runner sees an assistant message before the picker
click assertion
Probes: bumped per-card waitForSelector in d5-gen-ui-headless-complete from
15s to 60s — recharts ResponsiveContainer can be slow under 4 sequential
turns.
Shell-dojo: hide CLI Start Command from the dojo navigation via
`HIDDEN_DOJO_FEATURE_IDS`. Registry/manifests untouched so
harness/parity/dashboard still see it.
Jordan Ritter