the integration was added to the railway-services SSOT and smoke.yml nameExcludes; this
adds it to the sibling probe configs the probe-config-parity invariant requires to stay
in lockstep.
- qa probe: manifest.yaml path corrected from showcase/packages/ to
showcase/integrations/ (Slice 3 moved files; probe code and test
fixture still referenced the old path).
- pin-drift probe: fail-baseline.json ENOENT in container — the 5-level
import.meta.url walk-up from dist/probes/drivers/ overshoots /app and
lands at /. Added PIN_DRIFT_REPO_ROOT=/app env var and COPY of the
baseline file into the runtime stage.
- aimock-wiring probe: showcase-ms-agent-harness-dotnet (deployed: false)
was not in the EXCLUDE_SERVICES set, causing it to be flagged as
unwired. Added to the exclude list in both aimock-wiring.ts and
smoke.yml.
Add d6-all-pills driver that exercises every pill on every integration
behind aimock context routing. Rename e2e-chat-tools/e2e-deep to
d4-chat-roundtrip/d5-single-pill to match depth convention. Wire
D5 representative map, D6 scoping helpers, dimension types, and
the orchestrator config for the new probe IDs.
Production-fix bundle for beautiful-chat + 3 other google-adk demos. All
issues either reported on Railway prod or unmasked by the catch-all
render_a2ui fallback I added in #4836. Local D5 stays 38/38 green.
- Missing copilotkit logo: `<img src="/copilotkit-logo-mark.svg">` returned
404 because the SVG never shipped in the integration's public/. Copy
copilotkit-logo-mark.svg + copilotkit-logo.svg over from langgraph-python.
- Multimodal sample.png / sample.pdf were LFS pointers in prod (Railway
build runs without `git lfs pull`), so the magic-byte check rejected them
on first click. Add an integration-scoped .gitattributes that exempts
these two paths from LFS (mirrors what every working sibling integration
already does) and re-stage the files as real binaries (10KB / 2.5KB).
- Sales Dashboard pill returned a generic "Step 1... Step 2..." narration
instead of rendering the A2UI surface. Root cause: the render_a2ui
catch-all fallback I added to aimock/d5-all.json in #4836 fired before
feature-parity.json's specific sales-dashboard fixture (load order
d5-all → smoke → feature-parity). Removing the catch-all from both
d5-all.json and the per-demo source so the specific fixture wins.
- Calculator App pill returned text only with a white iframe because
beautiful-chat's agent instruction never mentioned generateSandboxedUi —
Gemini saw the tool listed via AGUIToolset but had no nudge to use it.
Added a one-line "Interactive / sandboxed widgets" entry to the
instruction; verified Gemini now emits the tool call.
- hitl-in-app refund (#12345) and escalate (#12347) pills broke on the
second click because the 2nd-turn fixtures keyed on `sequenceIndex` 0/1
(a global thread-position counter that drifts when other pills land
tool messages in the same thread). Convert both to `toolCallId +
hasToolResult: true` and drop the reject branch — Gemini reasons
correctly from the tool's `approved: false` return without a fixture
override. Same pattern that fixed tool-rendering-reasoning-chain
previously.
- hitl-in-app downgrade (#12346) pill produced an unrelated
"Research / Outline / Draft / Review / Finalize" plan because the
prompt contains the substring "plan" and feature-parity.json has a
generic catch-all match on `userMessage: "plan"`. Add a specific
downgrade fixture in d5-all.json (loaded before feature-parity.json)
with hasToolResult: false / true branches.
- hitl-in-chat "Schedule a 1:1 with Alice" returned the wrong
"Nice to meet you, Alice in Tokyo" response when clicked AFTER another
pill in the same thread. The 2nd-turn fixture only matched on
`toolCallId` (no hasToolResult), so the bare "alice" / "Alice" greeting
fixtures further down won. Add `hasToolResult: true` to the 2nd-turn
fixture so it scopes correctly regardless of thread state.
- Voice manual recordings always returned "What is the weather in Tokyo?"
regardless of audio content. aimock had a catch-all transcription fixture
(`match: { endpoint: "transcription" }`) that returned the canned
Tokyo string for any audio input. The D5 voice probe uses the sample
audio button which bypasses /transcribe entirely (it injects text
directly into the composer), so removing the transcription fixture
drops aimock into --proxy-only fall-through to real OpenAI Whisper for
mic recordings while D5 stays green. Verified.
- readonly-state-agent-context and shared-state-read-write fixtures
returned hardcoded "Atai" / generic preferences responses even when
the user changed the state values in the UI inputs. Gate the
Who-am-I / Suggest-next-steps / Greet / Plan-a-weekend fixtures on
systemMessage substring matching the canonical default state values
("Atai" name for readonly-state-context; "tone: casual" for
shared-state-read-write). When the user changes state, the agent's
before-model callback rebuilds the system prompt with the new values,
the fixture's systemMessage substring no longer matches, and aimock
--proxy-only falls through to the real model so the response reflects
the actual state. Confirmed with paired curl tests (default state =
fixture match; alem name = real-LLM response).
Local verification: bin/showcase test google-adk --d5 finishes 38/38
green (137s). Calculator pill confirmed via direct ADK invocation
against real Gemini (GOOGLE_GEMINI_BASE_URL=) emits TOOL_CALL_START
toolCallName=generateSandboxedUi. Sales-dashboard pill confirmed end-to-end
returning the full Column / DashboardCards / PieChart / BarChart payload.
readonly-state-context confirmed with name=Atai matching fixture vs
name=alem falling through to real-LLM response that uses the actual
context.
Five distinct root causes were keeping google-adk from full D5 parity
with langgraph-python. Fixing them takes the integration to 38/38 D5
green under aimock locally (verified end-to-end with --live writing
to PocketBase).
- readonly-state-context: page.tsx asked for agent slug
readonly_state_agent_context (underscore) but the registry mounts
it kebab-case as readonly-state-agent-context. useAgent threw,
the demo layout never mounted, and the ctx-name input never rendered.
Align with the registry (and with langgraph-python).
- multimodal: the secondary failure was a backend Pydantic
ValidationError on HttpOptions.api_endpoint. google-genai 1.75
renamed the field to base_url; all three direct genai client
constructors (main.py, beautiful_chat_agent.py, subagents_agent.py)
needed the rename so the secondary A2UI / sub-agent LLM calls stop
crashing. (The pre-existing LFS-pointer issue on the bundled
sample.png/pdf was a worktree hydration problem, not a tracked
code change — git lfs pull handles it.)
- gen-ui-declarative (two bugs stacked):
1. The same api_endpoint -> base_url rename above. The secondary
generate_a2ui planner LLM was failing every request.
2. The D5 fixture emitted components in {id, type, props: {...}}
shape, but sanitize_a2ui_components requires component, so
every entry was dropped and the renderer received an empty
surface. Rewrite both the per-demo fixture and the d5-all.json
aggregate to the flat {id, component, ...props} shape that
langgraph-python's _design_a2ui_surface fixture already uses,
wrapping multi-child layouts in a basic-catalog Column (the
custom Card schema has a single child slot). Add
_design_a2ui_surface variants so LGP gets per-pill payloads too.
- shared-state-streaming: ADK's write_document took content and
the PredictStateMapping read tool_argument="content", but the
shared D5 fixture (and the LGP function signature) names the
argument document. Rename both sides so the fixture's tool_call
args plumb into the function and into PredictStateMapping's
state-key emission. STATE_DELTA now propagates and DocumentView
streams live.
- tool-rendering-reasoning-chain: in thinking mode
(include_thoughts=True), Gemini emits a turn as two separate
non-partial chunks — a text-only chunk with finish_reason=None
and a function-call-only chunk with finish_reason=FUNCTION_CALL.
stop_on_terminal_text fired on the first (text-only) chunk and
set end_invocation=True before the function-call chunk arrived,
which broke AAPL->MSFT chaining. Gate termination on
finish_reason=STOP; FUNCTION_CALL and None both mean "more
chunks inbound — defer". Applies to every agent that uses the
shared callback, so chain-aware behavior is uniform.
Local verification: bin/showcase test google-adk --d5 --live
finishes green for all 38 cells (~140s), dashboard reflects the
results from PocketBase. Manual real-Gemini click-through of the
five fixed demos also passes end-to-end via GOOGLE_GEMINI_BASE_URL=
(empty) recreate.
Wraps railwayServicesSource with withCache (24h TTL). Instantiates
DiscoveryAuthTracker with threshold 3. Adds system dimension. Caches
listServices in Railway adapter (60s TTL). Writes system status on
browser pool init failure so degradation is visible in the dashboard.
Tracks auth failures per source since last success. After 3 failures,
writes system:discovery-auth-failed to PocketBase. Sustained alerts
rate-limited to one PB write per 5 minutes. Auto-recovers on next
success. Non-auth errors are no-ops. 9 test cases.
Transparent wrapper at the DiscoverySource interface level. Caches
successful enumerate() results in memory (24h TTL), serves stale
data on upstream failure, collapses concurrent callers into a single
upstream request. Auth tracker side-effects are try-caught to never
block the primary data path. Evicts entries older than 2x TTL.
18 test cases covering success, failure, TTL, collapse, eviction,
non-JSON config guard, and tracker integration.
Three layers of regression guards for the runtime reasoning-role filter and
the chained demo behavior:
1. Runtime unit test — packages/runtime/.../run-message-filtering.test.ts:
- Verifies `LangGraphAgent.run` strips `role:"reasoning"` from
`input.messages` before delegating to super.run.
- Verifies user/assistant/system/tool messages pass through in order.
- Verifies empty + missing messages arrays are tolerated.
- Verifies pre-existing forwardedProps.streamSubgraphs default + override
behavior is preserved.
- 6/6 tests pass against the runtime package's vitest config.
2. D5 harness probe — showcase/harness/.../d5-tool-rendering-reasoning-chain.ts:
- Expanded from one chained turn (flights→weather) to all three chained
pills in a single thread (stocks AAPL→MSFT, dice d20→d6, flights→weather).
- This is the canonical multi-pill regression at the harness layer:
without the runtime reasoning-role filter, the second pill would crash
before the model was called.
- Each turn asserts the per-turn delta of reasoning-block mounts (idx+1),
the minimum card count for each tool group, and unique transcript
substrings that scope to that turn.
3. Playwright e2e spec — showcase/integrations/langgraph-python/tests/e2e/
tool-rendering-reasoning-chain.spec.ts:
- Mirrors the pattern of the sibling tool-rendering-default-catchall spec
(notably its multi-pill regression at lines 162-212).
- Page-loads test verifies the 3 pills mount and no cards leak from a
prior session.
- One test per chained pill (stocks, dice, flights+weather) asserts the
full chain renders with reasoning-block + correct per-tool cards +
narration matching the aimock fixture text.
- Sequential-pills regression test clicks all 3 pills in one thread,
asserts each chain renders independently AND the reasoning-block count
increases monotonically across turns.
Agent: extend `get_stock_price` to accept optional `price_usd` and
`change_pct` arguments (mirrors the basic tool-rendering agent's signature
introduced in #4770). The aimock fixtures script the chained AAPL/MSFT
comparison by passing deterministic prices via these args; without the
wider signature, pydantic rejects the tool call and the card never mounts.
The runtime unit test is the strongest guard — it would catch any
regression on the role-filter logic without depending on the full Docker
stack. The harness probe and Playwright spec catch end-to-end regressions
in the canonical CI environment.
The tool-rendering-reasoning-chain demo previously promised chained tool
calls in its pill titles but the agent and fixtures only delivered single
tools — clicking "Weather + flights to Tokyo" produced just a WeatherCard,
"Compare two stocks" only fetched AAPL, "Find flights from SFO to JFK"
showed flights but no destination weather. Three changes close the gap.
Agent: replace the soft "call 2+ tools when relevant" system prompt with
concrete per-pill chain examples mirroring the pattern already used by the
langgraph-typescript `tool-rendering` agent (weather→flights, ticker→peer,
roll→contrast die, flights→destination weather).
Pills: drop the redundant Tokyo pill (it was the SFO/JFK chain in reverse)
and reword each remaining pill message to PRE-DISCLOSE the chain so the
model commits to the follow-up call:
- "Compare AAPL and MSFT stocks for me."
- "Roll a 20-sided die for me and compare it to a smaller one."
- "Find flights from SFO to JFK and show me the weather there."
Fixtures: 9 fixtures (3 per pill: final-content → second-leg → first-leg,
ordered by toolCallId specificity for first-match-wins). Each fixture is
scoped by a langgraph-python-UNIQUE userMessage tail ("Compare AAPL and
MSFT stocks", "compare it to a smaller one", "show me the weather there").
Those substrings appear nowhere else across the 14+ integrations sharing
showcase-aimock on Railway, so the new fixtures cannot cross-contaminate
the other reasoning-chain demos that still ship the older prompt set.
A toolName-based gate was considered and rejected because most fleet
agents register `roll_dice` and aimock's `toolName` matcher is a tool-LIST
gate, not a tool-CALL gate — it would NOT have isolated this demo.
Probe: collapse the two-turn flow (Tokyo + SFO/JFK) into one chained turn
(SFO→JFK + JFK weather) that asserts BOTH per-tool renderers
(FlightListCard + WeatherCard) mount in a single response. Same coverage
at half the wall-clock and exercises the actual chained-tool path.
Four independent showcase production bugs Alem reported, plus the
D5 multimodal harness regression they unblocked.
Shared-state-read-write: "Greet me" ("Say hi and introduce yourself.")
and "Plan a weekend" ("Suggest a weekend plan based on my interests.")
were matching the bare `hi` and `plan` catch-alls in feature-parity.json
and returning the generic showcase-assistant blurb / 5-step content plan
instead of shared-state-aware responses. Added pill-specific fixtures in
shared-state.json (mirrored into d5-all.json) so the longer userMessage
substrings win first-match-wins ahead of feature-parity.
Auth sign-out: signing out unmounted CopilotKit entirely and bounced
the user back to the SignInCard, so the demo never showcased the
runtime returning 401 — its whole point. The QA contract in
qa/auth.md spelled out the intended UX. Restored it: CopilotKit stays
mounted after the first sign-in, the AuthBanner flips to an amber
"Signed out — the agent will reject your messages" state with a
re-Sign-in button, and CopilotKit's `onError` callback drives a
`data-testid="auth-demo-error"` surface that displays the runtime's
401 the moment the user sends an unauthenticated message. Updated the
e2e spec to match (the old "SignInCard re-mounts after sign-out" test
pinned the regression).
Gen-ui-agent: the aimock fixture short-circuited the 7-step
progression spelled out in `gen_ui_agent.py`'s SYSTEM_PROMPT to a
single set_steps call with all three steps already `completed`, so
the InlineAgentStateCard rendered the final 3/3 state instantly with
no sequential pending → in_progress → completed animation.
Regenerated as a 7-leg toolCallId chain per pill (8 fixtures × 3
pills): seed leg keyed on userMessage with NO `hasToolResult` gate
(matching PR #4770's pattern — `hasToolResult: false` would block the
seed from firing on the second pill in a multi-pill session), then
six toolCallId-keyed transitions, then a final narration. Fixture
order: toolCallId legs FIRST so the most specific match wins.
Multimodal D5: the sample-attachment buttons auto-send via
`agent.addMessage + copilotkit.runAgent` (restored in PR #4761), but
the D5 harness still typed `input` + pressed Enter via the runner
after `preFill`, sending a second user message that competed with the
in-flight image upload — the v1 LangGraph runtime SSE stream got
tangled (browser DevTools showed `statusCode: pending` indefinitely)
and the assistant message never rendered. Added `skipSend?: boolean`
to ConversationTurn (distinct from `skipFill`, which still presses
Enter once the textarea has content) and switched d5-multimodal.ts to
`skipSend: true` with `responseTimeoutMs: 60_000` so the runner waits
on the assistant response without poking the chat further. Bumped the
PDF auto-prompt fixture in feature-parity.json to include the word
"document" so the existing `buildModalityAssertion("document")` check
still lands.
D5 result: 37 → 39 of 40 features passing. Only
`tool-rendering-reasoning-chain` remains and is a separate
agent/runtime bug (Tokyo Responses-API `reasoning` message survives
into the next turn's conversation history, runtime returns
`RUN_ERROR: "message role is not supported"`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Probes now send X-AIMock-Strict: true on all requests via
extraHTTPHeaders, triggering strict fixture matching in aimock.
Live demo traffic (no header) continues to proxy to real LLMs.
The "Draw a flowchart" suggestion pill in the mcp-apps demo sent
"Use Excalidraw to draw a simple flowchart with three steps." which
had no matching create_view fixture in d5-all.json. aimock walked
through to feature-parity.json's `{userMessage: "steps"}` substring
fixture and returned a generic "Here is my plan..." content blurb
with no MCP tool call, so the runtime never invoked create_view, the
MCP middleware never fetched the UI resource, and the sandboxed
iframe never mounted.
Add a fixture pair in d5-all.json (and its harness mirror) keyed on
"draw a simple flowchart": turn 1 emits create_view with a three-
step Start -> Process -> End flowchart, turn 2 emits the narration
after the tool result. The distinctive substring beats the generic
feature-parity catch-alls under first-match-wins.
Adds a regression test that loads the same fixture files in the same
order as docker-compose.local.yml and asserts via aimock's matchFixture
that each mcp-apps pill routes to its create_view fixture on turn 1 and
its narration fixture on turn 2.
## Summary
Follow-up to #4723. The post-merge cycle showed
`gen-ui-headless-complete` still red with `page.click: Timeout` waiting
for the aria-label selector I added in #4723.
Root cause: the demo has TWO chip surfaces with diverging aria-label
shapes:
- **EmptyState** (first paint, before any messages): `aria-label="Try
suggestion: <message>"`, visible text = the message string.
- **SuggestionBar** (after first message lands):
`aria-label="Suggestion: <title>"`, visible text = the short title.
Which surface is mounted depends on `messages.length` —
timing-dependent. Both my prior selectors (`button >> text="Weather"`
and `button[aria-label="Suggestion: Weather"]`) matched only one surface
and timed out on the other.
Fix: drop the chip-click preFill entirely. The runner's normal
fill+press hits the textarea with the verbatim prompt; chip clicks and
textarea-Enter dispatch the same `runAgent`, so the fixture matcher
catches either route — and the textarea is always present.
Also includes a small refresh of `e2e-deep.test.ts` for a stale
assertion (Phase-2A REGISTRY_TO_D5 split renamed `tool-rendering` →
`tool-rendering-default-catchall`).
## Test plan
- [ ] Land + watch the next e2e-deep cycle
- [ ] Confirm `gen-ui-headless-complete` flips green
🤖 Generated with [Claude Code](https://claude.com/claude-code)
D5 probes were red for two demos because the aimock fixtures didn't
exist or were dispatched through the wrong Responses-API path.
- `harness/fixtures/d5/mcp-apps.json` (new) — the MCP Apps probe sent
"Open Excalidraw and sketch a system diagram" with no matching
fixture, so aimock 404'd and the agent threw `NotFoundError`.
Added a two-leg fixture: leg 1 emits a `create_view` MCP tool call
with Excalidraw element JSON; leg 2 is the `hasToolResult: true`
content reply. Mirrored verbatim into `aimock/d5-all.json` so the
Docker-baked aimock has the same matches.
- `harness/fixtures/d5/tool-rendering-reasoning-chain.json` — added
non-empty `content` fields to the first-leg fixtures (Tokyo weather
+ SFO/JFK flights). aimock's Responses-API dispatch routes
tool-call-only responses (no `content`) through
`buildToolCallResponse`, which silently drops the `reasoning`
payload; with a non-empty `content`, dispatch shifts to
`buildContentWithToolCallsResponse`, which emits the
reasoning_summary events the v2 `<ReasoningBlock>` needs to mount.
Mirrored into `d5-all.json`.
- `harness/fixtures/d5/reasoning-display.json` — added a second
matcher for the `reasoning-default` e2e pill prompt ("sky appears
blue"), keyed alongside the existing D5 probe matcher ("show your
reasoning step by step"). Both responses include `reasoning` fields
so the built-in `CopilotChatReasoningMessage` "Thinking…/Thought
for…" header lands.
- `aimock/feature-parity.json` — added a fixture for the agentic-chat
multi-turn name-recall test ("What name did I just give") replying
"You said your name is Alice." The matcher is intentionally narrow
to avoid colliding with the generic showcase-assistant catch-all.
Net D5 effect: `mcp-apps` and `tool-rendering-reasoning-chain` turn 1
go from red → green. The remaining red on `tool-rendering-reasoning-
chain` turn 2 is the deeper aimock+deepagents+Responses-API multi-
turn state-management issue (production with real OpenAI works); out
of scope per the original audit's Phase F.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User-facing renames so the showcase reads the way a cold visitor would
expect:
- `byoc-hashbrown` → `declarative-hashbrown` (and `byoc-json-render` →
`declarative-json-render`). The display titles already said
"Declarative UI: …"; only the URL slugs and folder paths still
leaked the internal BYOC ("Bring Your Own Components") jargon.
Renamed:
/demos/byoc-hashbrown → /demos/declarative-hashbrown
/demos/byoc-json-render → /demos/declarative-json-render
/api/copilotkit-byoc-* → /api/copilotkit-declarative-*
src/app/demos/byoc-* → src/app/demos/declarative-*
qa/byoc-*.md → qa/declarative-*.md
tests/e2e/byoc-*.spec.ts → tests/e2e/declarative-*.spec.ts
Internal Python module names + langgraph graph IDs stay legacy
(`byoc_hashbrown_agent.py`, `byoc_hashbrown`) — those are not
user-facing and renaming them is a separate cross-codebase pass.
- `a2ui-fixed-schema` slug intentionally unchanged.
- Tool Rendering trio parenthetical rename (Default → Catch-all →
Custom progression reads clearly as "how much do I customize?"):
Tool Rendering (Default) — unchanged
Tool Rendering (Custom default) → Tool Rendering (Catch-all)
Tool Rendering (Specific) → Tool Rendering (Custom)
- `tool-rendering-reasoning-chain` cell renamed from
"Generative UI: Rendering multiple tools" to
"Generative UI: Tool calls + reasoning" (the demo is about combining
reasoning + tool rendering, not about quantity of tools).
- `Open Generative UI: Default` / `Open Generative UI: Custom`
descriptions expanded so a visitor understands how Open Generative UI
differs from Tool Rendering (agent composes UI from a registered
library vs. attaching a renderer to a *named* backend tool).
- Showcase index now sorts demos within each tag by `manifest.features`
order. Previously demos appeared in manifest declaration order, which
ignored the team's curated "polished flagship → simplest start →
variants" arc.
Cross-cutting registry / harness / dashboard updates that fall out of
the rename:
- `shared/feature-registry.json` adds the two new IDs alongside the
legacy `byoc-*` (so the catalog stays valid; the other 17
integrations still declare `byoc-*` in their manifests).
- `shared/constraints.yaml` adds the new IDs to the
generative-ui-approach allow-list.
- `scripts/__tests__/generate-catalog.test.ts` updates the cell-count
expectations (45 features × 18 integrations = 810; 792 after docs-
only exclusion; 45 LGP cells = 38 wired + 1 stub + 6 unshipped).
- Harness probe `d5-byoc.ts` + `d5-byoc.test.ts` now route both slug
families through `preNavigateRoute` and exercise the new branches.
- `d5-feature-mapping.ts` and `shell-dashboard/live-status.ts` mirror
the dual-ID mapping so both legacy and renamed slugs roll up under
the same `byoc` D5 featureType.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
LGP probe hung on Turn 2 (Find flights from SFO to JFK) after Turn 1
(weather Tokyo) succeeded. Root cause: the SFO/JFK first-leg fixture
matched on `hasToolResult: false`, but Turn 1's tool result remains in
the conversation by Turn 2, so the matcher skipped it and fell through
to the second-leg fixture (`hasToolResult: true`), returning narration
without the search_flights tool call. The probe then timed out waiting
for the FlightListCard testid that never mounted.
Fix: switch the second-leg fixture to `toolCallId` matching (the last
message must be a tool result with the specific call_id) and drop
`hasToolResult` from the first-leg fixture. The second-leg fixture
must come BEFORE the first-leg in file order because the matcher is
first-match-wins; with `toolCallId`, it cannot accidentally swallow
first-leg requests (whose last message is the user prompt, not a
tool result).
Also adds a `content` field to the Tokyo first-leg fixtures
(`get_weather` and `get-weather` Mastra-hyphen variants) so they route
through `buildContentWithToolCallsStreamEvents` instead of
`buildToolCallStreamEvents` — the latter codepath silently drops the
`reasoning` field, which is why the reasoning-block testid never
mounted on Turn 1 either before this change.
Tokyo (Turn 1) keeps the simpler `hasToolResult` pattern because it has
no prior turns; only the SFO/JFK fixture pair needs the toolCallId
shape to survive multi-turn use.
The e2e-readiness probe waits for any of a compound CSS selector
covering chat-input testids. /demos/auth renders a <SignInCard>
BEFORE mounting the chat surface (the chat-input testids only appear
post-authentication), so the readiness probe was timing out at 30s
with `per-demo deadline exceeded` and the auth cell rolled up to D2
even though the demo itself works fine end-to-end.
Add `[data-testid="auth-sign-in-card"]` to READY_SELECTORS so the
probe accepts the SignInCard as a valid "demo mounted" signal. The
end-to-end auth flow (sign in -> sign out -> sign back in -> assistant
response) is still covered by auth.spec.ts under the e2e-demos
dimension, so the readiness probe shouldn't be doing that work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three fixes that follow up on PR #4743 to bring LGP closer to fully-green
on the dashboard:
1. tool-rendering-reasoning-chain probe: was failing with
`expected [data-testid="reasoning-block"] to mount within 30000ms`.
Root cause: the demo's `<ReasoningBlock>` slot only mounts when a
reasoning-role message lands in the transcript, which requires
aimock to emit REASONING_MESSAGE_* events, which in turn requires
the fixture's first-leg response to carry a `reasoning` field. The
weather/Tokyo and SFO/JFK first-leg fixtures were missing it.
Mirrors the convention documented in reasoning-display.json:2.
Patched both source (harness/fixtures/d5/) and bundle (aimock/d5-all.json).
2. gen-ui-interrupt source fixture: the source fixture file was missing
the resume-leg toolCallId entries that already existed in the bundle.
Cosmetic mirror so re-bundling stays consistent. Same chip prompts +
same toolCallIds as interrupt-headless.json (both probes share the
same agent and aimock fixture set; the difference is the FRONTEND
rendering — useInterrupt inline vs useHeadlessInterrupt separate-pane).
3. Dashboard gold-standard filter: 4 deprecated/legacy features
(agentic-chat-reasoning, hitl, hitl-in-chat-booking,
reasoning-default-render) used to render as X-marked rows in the
LGP gold-standard dashboard view because LGP intentionally does
NOT implement them — they were consolidated into the modern shape
(reasoning-custom + reasoning-default; hitl-in-chat with
useHumanInTheLoop). Other 17 integrations still serve those legacy
demos, so we don't yank the features from feature-registry.json
entirely. Instead: marked them `deprecated: true` and updated
generate-registry.ts to skip emitting cells when a deprecated
feature is unshipped for an integration. LGP cells: 43 → 39 (the
4 deprecated rows disappear). Other integrations: unchanged
(audit trail preserved). Catalog total: 774 → 770.
Tests:
- 1588/1588 harness vitest passing
- 19/19 generate-catalog + generate-registry tests passing
(counts updated for the 4 dropped LGP cells + new deprecated-
feature filter test)
- validate-fixture-tool-surface clean (282 fixtures × 627 demos)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the demo↔probe coverage gap for /demos/{interrupt-headless,
shared-state-read, tool-rendering-reasoning-chain} so every demo
under langgraph-python (the north-star integration) now has a D5
probe writing to its own PocketBase cell — not relying on cross-
demo umbrella records.
New probes (multi-turn, mirroring the agentic-chat structure):
- d5-interrupt-headless: exercises useHeadlessInterrupt — chip
prompt → backend interrupt(...) → app-surface popup → slot pick
→ resume → assistant confirmation. Distinct from gen-ui-interrupt
(which uses inline useInterrupt).
- d5-tool-rendering-reasoning-chain: combines reasoning-block slot
+ per-tool renderer (WeatherCard, FlightListCard) on the same
chat surface. Catches a regression in either side.
- d5-shared-state-read: recipe-editor demo (neutral default agent,
no tools) — verifies recipe-card form mounts AND agent reads
shared state across turns. Drops the dual-claim that
d5-shared-state.ts had on `shared-state-read` (now write-only).
Driver retry-once (e2e-deep.ts):
Probes that fail with a transient class (`goto-error` /
`conversation-error`) AND took ≥2s on the first attempt now retry
once before recording red. Persistent assertion-style failures
(sub-2s) and intentional aborts/feature-timeouts skip retry —
retrying a deterministic mismatch just burns clock and obscures
the signal. Cuts ~10× the dashboard flap rate.
Plumbing:
- D5FeatureType enum: +interrupt-headless, +tool-rendering-reasoning-chain.
- REGISTRY_TO_D5 (harness) + CATALOG_TO_D5_KEY (dashboard) mirror
the new mappings; d5-mapping-drift test enforces this.
- LGP manifest features + demos entries + constraints allowlist.
- feature-registry.json: +shared-state-read.
- aimock d5-all.json: +2 shared-state-read fixtures (interrupt-
headless + tool-rendering-reasoning-chain reuse existing fixtures
that already match their chip prompts).
Tests: 1588/1588 harness vitest green. validate-fixture-tool-surface
clean (282 fixtures × 627 demos, no drift). Two pre-existing test
fixes folded in — d5-gen-ui-interrupt assertion mock updated to
match the current evaluate-poll resume signal; conversation-runner
preFill ordering test now asserts the actual deferred-cascade
contract instead of a stricter pre-preFill ban that the runner
never enforced.
Known follow-up (not in this PR): auth.spec.ts test #5 ("signing
back in re-mounts a fresh chat surface") fails on Railway — second
sign-in's "Hello again" never produces an assistant response. Looks
like a react-core/v2 ref-handling regression on <CopilotKit>
unmount/remount; deserves its own focused investigation.
Other integrations may flip red on the new probes — that's
expected. We're treating LGP as the template; cross-integration
parity follows in a separate wave.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands the Bucket A framework fix and the fixture-correctness changes
needed to flip the remaining D4/D2 cells in the langgraph-python column
to D5. Full local D5 sweep is green (1 passed, 0 failed).
Framework: every Python `ToolMessage` constructed via `Command(update=...)`
now sets `name=` and `id=str(uuid.uuid4())`. Without these, @ag-ui/langgraph
synthesises TOOL_CALL_START events with `toolCallName: null` and
`parentMessageId: null`, which @ag-ui/client@0.0.53's Zod schema rejects;
the rejection is silently swallowed by `withAbortErrorHandling -> EMPTY`,
completing the SSE observable mid-stream so post-tool state never reaches
the consumer. Fix is applied across shared_state_streaming, shared_state_read_write,
gen_ui_agent, beautiful_chat, subagents (2 sites). Single-flag change in
a2ui_dynamic flips the secondary `_design_a2ui_surface` LLM call to
`streaming=True` so aimock's record/replay (SSE-only) sees it.
Fixtures (d5-all.json):
- toolCallId follow-ups for set_steps (3), display_flight, generate_a2ui (4),
schedule_meeting (2), generateSandboxedUi (7), and revenue chart so
multi-turn probes don't recurse into recursion-limit loops
- four hand-crafted secondary `_design_a2ui_surface` fixtures so A2UI
dynamic renders without a real LLM
- mcp-apps fixture rewritten to emit `create_view` tool call with a
minimal Excalidraw element payload; runtime middleware fetches the UI
resource and the iframe mounts
- AAPL and revenue `hasToolResult: true` follow-ups tightened to
`toolCallId` so they don't match cross-turn after prior turns' tool
results
- voice fast-path content-only fixture
- beautiful-chat-schedule-meeting first-turn fixture gains content so
the conversation runner sees an assistant message before the picker
click assertion
Probes: bumped per-card waitForSelector in d5-gen-ui-headless-complete from
15s to 60s — recharts ResponsiveContainer can be slow under 4 sequential
turns.
Shell-dojo: hide CLI Start Command from the dojo navigation via
`HIDDEN_DOJO_FEATURE_IDS`. Registry/manifests untouched so
harness/parity/dashboard still see it.
agent-config: drop the "introduce yourself per your config" baseline — it
substring-matched every prefixed probe prompt (tone:, expertise:,
responseLength:), forcing byte-identical responses and masking the
per-knob behavior the cell is meant to demonstrate.
shared-state-streaming: add toolCallId-keyed follow-up fixtures
(poem/email/quantum) above each first-turn entry so the matcher returns
content-only after the tool result lands. Without these the unchanged
last user message kept re-matching the first-turn fixture, the agent
re-fired write_document, and langgraph hit its 100-step recursion
limit. The frontend state-binding regression that prevents the streamed
document from rendering is a separate framework-level issue.
Synced to showcase/aimock/d5-all.json so replay-mode picks up both
fixes.
Six probe-side fixes landing five additional D5 flips on the
langgraph-python column. None of the fixes touch demo or runtime
code — just the harness-side selectors / signal predicates so each
probe matches what the published langgraph-python image actually
renders today.
Per-probe changes:
- d5-tool-rendering-default-catchall: add a fallback path that
scans copilot-assistant-message bubbles for the literal tool name
+ a 'done'/'running' status label, alongside the strict testid
contract added in unreleased commit ba60df5d3. The strict path
re-engages once @copilotkit/react-core ships a release with the
testid; until then the bubble-text scan is the only stable hook
on 1.56.5.
- d5-reasoning-display: the published built-in
CopilotChatReasoningMessage carries no testid. Accept the verbatim
'Thought for' / 'Thinking…' header text that the slot emits, in
addition to the four pre-existing testid selectors used by the
reasoning-custom override.
- d5-gen-ui-headless-complete: the SuggestionBar renders
agent-generated chip phrasing alternately with the static
useConfigureSuggestions titles. Switch from clicking
'aria-label="Suggestion: <Title>"' to clicking by aria-label
substring with a per-chip alias list (e.g. ['stock', 'aapl'])
that matches BOTH forms. Selector also moves from a custom
text-walk to a CSS attribute-substring selector with the 'i'
modifier — self-contained, no dynamic-code-eval.
- d5-gen-ui-interrupt: widen the post-pick assertion to accept any
of (a) the 'time-picker-picked' testid, (b) the visible 'Booked'
badge text, or (c) the agent's 'scheduled / confirmed' resume
continuation. The picked-state Card unmounts as soon as
langgraph resumes after resolve, so a 5s testid wait races the
unmount; the OR catches whichever signal lands first.
- _beautiful-chat-shared (toggle-theme): two-track signal — pass on
EITHER the html.dark class flipping from its initial reading OR
the visible 'Theme toggled' assistant content rendering. The
class-flip is the strongest signal but useFrontendTool's dispatch
occasionally drops the handler call without dropping the agent's
follow-up content message; track (b) catches the 'tool semantics
reached the UI' state in the meantime.
- _beautiful-chat-shared (search-flights): swap the literal
'United Airlines' wait for the short brand label 'United', which
appears in BOTH the FlightCard a2ui surface's render (when it
paints) and the assistant's '49 / 89' narration. The price
literals ($349 / $289) stay as the canonical 2-flight
fingerprint.
Conversation-runner adjustment:
- Reverse the partial-fix from PR #4724: try chat-input cascade
resolution AT BOOT first (and read baseline at boot too); only
defer to post-preFill on auth-shape demos where the cascade
fails. Without this, demos where preFill itself fires the
message (chip clicks in headless-complete) had baseline read
AFTER the assistant bubble already appeared and the settle
waited indefinitely for further growth that never came.
The demo has TWO chip surfaces with diverging aria-label shapes:
EmptyState (first paint) emits aria-label='Try suggestion: <message>'
with the visible text being the message string, while SuggestionBar
(post-first-message) emits aria-label='Suggestion: <title>' with
visible text being the short title. Which surface is mounted at any
given turn depends on messages.length — timing-dependent. Both my
prior selectors matched only one surface and timed out on the other.
Drop the chip-click preFill entirely; let the runner's normal
fillAndPress hit the textarea with the verbatim prompt. Chip clicks
and textarea-Enter dispatch the same runAgent path, so the fixture
matcher catches either route — and the textarea is always present.
Also refresh e2e-deep.test.ts: a stale test assertion expected the
old REGISTRY_TO_D5 mapping shape (tool-rendering-default-catchall →
tool-rendering D5 type), but the Phase-2A split repointed it to its
own D5 type of the same name. Update the expected skipped list and
sideKeys to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
In replay mode (no real OpenAI), the existing frontend-tools fixtures
match by 'userMessage' substring only. Each pill (Sunset / Forest /
Cosmic) had a single fixture that returned both content + a
change_background toolCall. After the frontend tool ran, the agent's
follow-up LLM call still has the same last user message — substring
matches the same fixture again — agent re-fires change_background,
loops to langgraph's recursion limit, probe's settle window never
quiets, run reds out at the assistant-settle timeout.
Add a toolCallId-keyed follow-up per pill that returns content-only,
plus a static tool_call_id on the first-turn toolCall for the follow-
up to pin to. The follow-up combines userMessage + toolCallId so it
can't cross-match the next pill's first-turn request (which still has
a tool result in history from the prior pill, but a different last
user message).
Probe result: d5:langgraph-python/frontend-tools flips green again in
replay mode — was passing in PR #4724 only because the real-LLM run
that PR was tested against happened to settle without re-firing.
PILL_GRADIENT_HINTS pinned the fixture/aimock output deterministically
with a fixed substring list (color words + the exact hex codes from
the fixtures). Real OpenAI emits any chromatic-family hex (e.g.
forest as '#005f00 / #4caf50') that the fixed substring list can't
enumerate, so the assertion failed for valid green output.
Add a channel-dominance fallback: parse all 6-digit hex codes from
the gradient and accept when at least one satisfies the per-pill RGB
rule (g > r && g > b for forest, r > g && r > b for sunset, b > g
or purple-channel for cosmic). MIN_DOMINANCE_DELTA + MIN_DOMINANT_VALUE
prevent near-grey or near-black hexes from cross-matching. Word and
fixture-hex match still runs first; channel-dominance is the
fallback only.
Probe result: d5:langgraph-python/frontend-tools flips green.
Idiomatic auth demos (langgraph-python) render a SignInCard until
the user clicks 'Sign in with demo token' — the chat textarea only
mounts after that click, which preFill performs. The conversation
runner was resolving the chat-input cascade up front, before any
turn-level preFill ran, so the cascade timed out at 5s on
SignInCard and turn 1 failed with 'chat input not found'.
Move resolution and the initial assistant-message baseline into
the per-turn loop, lazy on first turn. Subsequent turns reuse the
cached selector.
Probe result: d5:langgraph-python/auth flips green.
After PR #4718 + the timeout bump in #4722 unblocked the abort-cascade,
post-merge cycles surfaced the real per-probe failures that were
previously hidden. Each fix here is targeted at one real failure mode
identified from PB error class + message:
* chat-css: HALCYON probe selector was `[class~="bg-muted"]`, but v2
emits PREFIXED utilities (`cpk:bg-muted`). Whole-token match on the
prefixed name (`[class~="cpk:bg-muted"]`) fixes both the static
USER_BUBBLE_INNER_SELECTOR and the page.evaluate selector — and is
unambiguous against `cpk:bg-muted-foreground` on nested children.
* gen-ui-agent: probe assumed `agent-step` DOM rows accumulated across
pills (delta ≥ 2 vs. pre-pill baseline). The backend's `set_steps`
reducer is `last-write-wins` (per `gen_ui_agent.py`), so each pill
REPLACES `state.steps` and the DOM swaps. delta was always 0 after
pill 1, even when the new steps rendered correctly. Drop the
baseline-capture preFill, assert ≥ 2 visible rows + per-pill
fingerprint dedup. Tests rewritten to match the swap model.
* gen-ui-headless-complete: chip click used `button >> text="<title>"`
(Playwright chained selector). The button has a stable
`aria-label="Suggestion: ${title}"` attribute set inline; switch to
`button[aria-label="Suggestion: ${title}"]` so we match the moment
the DOM exists rather than waiting for inner text-node hydration.
* shared-state-streaming: fixture sent `{"content": "..."}` but the
langgraph-python `write_document(document: str)` tool expects
`document`. Mismatch meant `state.document` never updated; probe saw
no content delta. Rename the argument key to `document` in both the
per-feature fixture and `d5-all.json` (the runtime fixture). The
middleware's `tool_argument="document"` config requires this name.
* reasoning-display: fixture only emitted `content` (text response).
Probe asserted the v2 `<ReasoningBlock>` (rendered behind
`[data-testid="reasoning-block"]`), which mounts only when a
reasoning-role message lands in the transcript. Add a `reasoning`
field so aimock emits REASONING_MESSAGE_* events.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #4718 added the LGP D5 coverage wave, taking langgraph-python's
declared D5 feature set to ~28. With FEATURE_CONCURRENCY=2 that's 14
features per worker × ~15s realistic wall-clock = ~210s — over the
prior 3-min cap before any slack. Post-merge cycles confirmed the math:
17 langgraph-python features ran cleanly and flipped green, but the
remaining ~10 hit `errorClass: "abort"` with `errorDesc: "aborted"`,
the abort-cascade signature where the global cap fires before the
per-feature loop reaches them. Per-feature timeout (5min) bounds wedged
features but doesn't bound the slug's total feature count against the
outer cap.
Bump to 10 min — gives ~5x headroom at current feature count. Lighter
integrations (5–10 features, ~60s) still exit early without sitting on
the cap, so cycle wall-clock barely changes (cron stays every 15 min,
concurrency=4 hides the slow integration's tail behind the others).
Revisit `FEATURE_CONCURRENCY` before raising further — Chromium context
count scales with it and the Railway pod's 2.4GB headroom budget assumes
the current value.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The shell-docs port (e2bef7a0b) updated R15/R17 sources to /integrations/built-in-agent
in showcase/shell-docs/src/lib/seo-redirects.ts and refreshed the snapshot fixture, but
left the legacy shell copy (showcase/shell/src/lib/seo-redirects.ts) and the harness
intentional-copy (showcase/harness/src/probes/drivers/seo-redirects.ts) on the old
/builtin-agent value. The redirect-decommission test imports from the legacy shell
file, so the snapshot diverged: fixture says /integrations/built-in-agent but source
generates /builtin-agent. Mirror the retargeting into both copies so all three files
(shell-docs source, shell legacy source, harness synced copy) agree with the fixture
and Validate Showcase passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Two-part redirect work for the docs.copilotkit.ai → shell-docs cutover,
in one PR.
### Part 1 — Retarget destinations in `seo-redirects.ts`
shell-docs serves canonical framework docs at `/{fw-slug}/...` from the
host root (no `/docs/` prefix) and uses different framework slugs from
the legacy SHELL surface. The redirect catalogue has been retargeted
accordingly:
- **Drop the `/docs/integrations/` prefix** from every destination.
- **Apply framework-slug renames** in destinations:
- `langgraph` → `langgraph-python`
- `adk` → `google-adk`
- `aws-strands` → `strands`
- `microsoft-agent-framework` → `ms-agent-dotnet`
- `crewai-flows` → `crewai-crews`
- **Re-flip the BIA → unselected rename** — `unselected/` was retired;
destinations now point at `/built-in-agent/`.
- **Slug-rename catch-alls** for the bare `/{old-slug}/*` form so legacy
upstream URLs (e.g. `/langgraph/quickstart`) 301 directly to the new
slug.
- **`/docs/integrations/*` and `/docs/*` catch-alls** so any URL still
carrying the legacy SHELL routing prefix lands at the shell-docs
equivalent.
- **`/migration-guides/*` → `/migrate/*`** (4 URLs).
- **Folder-index redirects** for shell-docs folders without an
`index.mdx` (`/troubleshooting`, `/migrate`, `/premium`, `/concepts`,
`/reference`).
390 redirect entries total in the new catalogue.
### Part 2 — Port middleware to shell-docs
- Copied the retargeted `seo-redirects.ts` to
`showcase/shell-docs/src/lib/`.
- Merged the SHELL redirect-middleware logic into shell-docs's existing
pageview-tracking middleware. Redirects fire first (with the
`seo_redirect` PostHog event); non-redirected requests still get the
`docs_pageview` capture and `distinct_id` cookie.
- Preserved the framework-scoped short-circuit so canonical
`/{fw-slug}/...` URLs are never hijacked by legacy patterns.
- Left the SHELL versions of `middleware.ts` and `seo-redirects.ts` in
place — the SHELL still serves `docs.showcase.copilotkit.ai` until DNS
flips.
## Test plan
- [x] `npm run build` clean in `showcase/shell-docs/`
- [x] `npm run build` clean in `showcase/shell/` (existing operation
unaffected)
- [x] `curl -sI
http://localhost:3099/docs/integrations/langgraph/quickstart` → 301 to
`/langgraph-python/quickstart`
- [x] `curl -sI http://localhost:3099/langgraph/quickstart` → 301 to
`/langgraph-python/quickstart`; `/aws-strands/quickstart` →
`/strands/quickstart`; `/migration-guides/v2` → `/migrate/v2`;
`/troubleshooting` → `/troubleshooting/common-issues`; `/coagents` →
`/langgraph-python`
- [ ] Validate full set against a running shell-docs instance with
`validate-redirects.ts` (run from `showcase/scripts/` against the
deployed preview)
Jordan Ritter