The validator was not considering the aimock match.toolName field when
checking for drift. When a fixture has toolName set, aimock only fires
it for agents that register that tool -- skip demos that don't have it.
The dashboard needs OPS_BASE_URL at build time to connect to the local
harness. Without this, the dashboard container builds with no API URL
and cannot display probe results locally.
- Add dedicated tool-free voice agents for strands, llamaindex,
ms-agent-python (aimock returns tool calls when tools are registered,
which the adapters don't loop on)
- Add sample_agent alias to langgraph-typescript langgraph.json
(was only in dev-mode config)
- Add SampleAudioButton and voice route to google-adk
- Add sample.wav to agno, ms-agent-dotnet, ms-agent-python, google-adk
V1 CopilotRuntime in single-route mode rejects multipart/form-data
with 415 Unsupported Media Type. Port all 9 integrations to V2
createCopilotRuntimeHandler which handles the /voice sub-route
natively.
Integrations: claude-sdk-python, claude-sdk-typescript, crewai-crews,
llamaindex, ms-agent-dotnet, ms-agent-python, pydantic-ai, spring-ai,
strands
When a Railway service deployed within the last 120 seconds, the D5
probe driver now skips all features for that service with green side
rows (note: "skipped: deploy in progress (Ns ago)") instead of
launching a browser and producing false reds from deploy churn.
The skip fires before the script loader and browser launch so recently
deployed services cost zero probe resources.
Discovery source: added deployedAt (latestDeployment.createdAt) to
RailwayServiceInfo and the GraphQL query.
Driver: added deployedAt to the input schema and DEPLOY_CHURN_GRACE_MS
constant (120_000ms). When deployedAt is within the grace window, all
features short-circuit to green with a descriptive note.
Tests: 8 new tests covering the skip path, boundary conditions
(exactly at grace window), backwards compat (absent/empty/unparseable
deployedAt), and the 0s-age edge case.
## Summary
- Never mutate originals -- temp overlay in $TMPDIR instead of in-place
.iso-bak
- Slot-based port allocation via atomic mkdir for parallel support (slot
0 = +200, slot 1 = +400, etc.)
- --project-name scoping prevents Docker container/network collisions
- Stale slot cleanup via PID check + 2-hour age fallback
- LOCAL_PORTS_FILE env var for TS harness to read offset ports
## Test plan
- [ ] Originals untouched after isolated run (md5 verified)
- [ ] CI green
- [ ] Two parallel --isolate runs don't conflict
The COMPOSE_CMD in apply_isolation was missing --project-name, so Docker
Compose would infer the project name from the directory and collide with
the base showcase stack (and other isolated runs). Adding --project-name
ensures containers, networks, and volumes are fully scoped to the
isolation slot.
apply_isolation previously mutated docker-compose.local.yml and
local-ports.json in-place with .iso-bak backups. If the process crashed
the originals stayed corrupted with +200 port offsets, breaking all
subsequent showcase commands.
Now writes modified copies to a temp directory and overrides
COMPOSE_FILE/PORTS_FILE shell variables so downstream code reads from
the overlay. Originals are never touched. restore_isolation just removes
the temp dir.
Also replaces hardcoded +200 port offset with atomic mkdir-based slot
allocation. Two parallel --isolate runs now get different port ranges
(slot 0 = +200, slot 1 = +400, etc.) instead of colliding on the same
ports. Container names include the slot number for collision-free Docker
naming. Stale slots from crashed runs are reclaimed via PID liveness
checks and a 2-hour age fallback.
TS harness files (config.ts, lifecycle.ts, doctor.ts) honor
LOCAL_PORTS_FILE env var so they read offset ports from the temp overlay.
D3/D4 was blue (accent), now amber/yellow to signal "not yet D5".
D1/D2 was amber, now red to signal "needs work".
Removed D6 references — D6 no longer exists.
spring-ai: StreamingToolAgent Phase 1 streaming was not including tool
definitions in the OpenAI API request. Without tool schemas, aimock
could not match the fixture requiring toolName:"get_weather" and fell
through to a generic text-only fixture. The tool call was never
emitted, so useRenderTool never triggered and the weather card never
rendered.
Attach toolCallbacks to the Phase 1 streaming request while keeping
internalToolExecutionEnabled=false. The LLM/aimock sees the tool
schemas and can return tool_calls, but Spring AI does not auto-execute
them. Phase 2 handles execution as before.
mastra: Mastra Agent class uses the object KEY in the tools config as
the OpenAI function name, not the createTool id. The weatherAgent
registered tools as { weatherTool, queryDataTool, ... } which sent
function names "weatherTool", "queryDataTool" etc. to aimock. The D5
fixture expects toolName:"get_weather", so it never matched. Also
aligned the createTool id from "get-weather" to "get_weather".
Use explicit keys matching the expected function names:
{ get_weather: weatherTool, query_data: queryDataTool, ... }
Verified locally: spring-ai 31/32 green (voice pre-existing), mastra
32/32 fully green.
When the harness process dies mid-probe (Railway redeploy, OOM),
PocketBase rows stay in state=running forever. These zombies block
the API's run list, show stale results, and confuse the dashboard.
sweepStaleRuns() marks any running row older than 15 minutes as
failed. Called once at orchestrator boot.
PR #4562 removed the referenceCount variable but left a log line
referencing it, causing ReferenceError at build time. The dashboard
never rebuilt with langgraph-python as REF because the build crashed.
The auto-detect logic picked whichever integration had the most wired
features, with ties broken alphabetically. This caused ag2 to appear
as the reference when it matched langgraph-python's feature count.
langgraph-python is always the gold standard reference.
## Summary
Adds a D5 voice test that exercises the voice transcription flow via the
sample audio button. Deterministic: aimock intercepts the Whisper
transcription call and returns a canned response.
### What's new
- **D5 voice script** (`d5-voice.ts`) — clicks sample audio button,
waits for transcription to fill textarea, sends, asserts weather
response
- **skipFill in conversation runner** — new `skipFill: true` option on
ConversationTurn that skips `page.fill()` when preFill already populated
the textarea (9 new tests)
- **Voice in D5 registry** — `voice` added to D5FeatureType enum +
REGISTRY_TO_D5 mapping
- **Aimock transcription fixture** — returns "What is the weather in
Tokyo?" for any `/v1/audio/transcriptions` request
- **Voice runtime docs** — confirmed OPENAI_BASE_URL routes Whisper
calls through aimock automatically
### How it works
1. preFill clicks `[data-testid="voice-sample-audio-button"]`
2. Aimock returns canned transcription → textarea fills with "What is
the weather in Tokyo?"
3. skipFill sends without overwriting → aimock handles the chat
completion
4. Assertion checks for weather/Tokyo in the assistant response
### Test plan
- [x] 1490 harness tests pass (91 files)
- [ ] CI green
- [ ] Local `showcase test langgraph-python --d5` with voice feature
Add D5 voice test that exercises sample-audio transcription via aimock.
Infrastructure: voice in D5 feature type registry + mapping, skipFill
support in conversation runner (9 new tests), inputValue forwarding
in e2e-deep Page wrappers, aimock transcription fixture, tool-free
weather fallback fixture for agents without tools. Verified locally:
D5 suite passes green on langgraph-python (60.4s).
Strategy 9: pull all D5 status records from PocketBase in one request,
categorize errors, cross-reference with deploy history, and compute
per-service flapping rates to distinguish real bugs from transient
production issues.
The chat input selector cascade gave each candidate only 2s to appear.
Under Railway production load, React hydration takes longer than 2s,
causing the probe to miss the input and mark the feature red. This was
the root cause of D5 flapping — all 18 integrations pass locally but
only 2-3 are stable green in production.
All 7 currently-red PocketBase D5 records showed the same error:
page.fill timeout waiting for the chat input selector.
The _normalize_part function in multimodal_agent.py checked
isinstance(part, dict) to gate all content-part processing.
When RunAgentInput is deserialized via Pydantic, ag_ui.core
types (TextInputContent, ImageInputContent, DocumentInputContent)
are model instances — not dicts — so every multimodal content
part was silently dropped.
This caused the D5 multimodal probe's PDF turn to fail: the user
message text was lost, the aimock matched the stale turn-1 fixture
instead of the turn-2 fixture, and the assertion saw the image
response where it expected the document response.
Convert Pydantic models to dicts via model_dump(by_alias=True)
before processing, preserving camelCase field names (mimeType)
that the rest of the function relies on.
The _MultimodalAgent.run() override used *args/**kwargs but
AgentFrameworkAgent.run() expects input_data: dict. The mismatch
caused TypeError at runtime. Changed to match the base signature
and yield events from the base generator.
gen-ui-interrupt and interrupt-headless pages were using V1 CopilotKit
with named agents (agent="gen-ui-interrupt", agent="interrupt-headless")
but the built-in-agent runtime only registers a single "default" agent
via CopilotRuntime V2. This caused 404/agent-not-found errors at D5.
Switch both pages to CopilotKitProvider with useSingleEndpoint and
remove agentId from CopilotChat, matching the pattern used by all
other built-in-agent demo pages.
## Summary
PR #4542 incorrectly changed the langgraph-python declarative-gen-ui
route from `graphId: "a2ui_dynamic"` to `graphId: "sample_agent"` and
removed `injectA2UITool: false`. This caused a regression from 31/31
green to red.
The `a2ui_dynamic` graph owns the `generate_a2ui` tool itself — the
runtime must NOT auto-inject its own A2UI tool on top (`injectA2UITool:
false`). The `sample_agent` graph is a generic chat agent with no tools,
which can never produce A2UI surfaces.
## Test plan
- [ ] langgraph-python returns to D5 green (31/31)
PR #4542 incorrectly changed graphId from "a2ui_dynamic" to
"sample_agent" and removed injectA2UITool: false. The a2ui_dynamic
graph owns the generate_a2ui tool itself — the runtime must NOT
auto-inject. This caused langgraph-python to regress from 31/31 to red.
## Summary
- Switch byoc-hashbrown and byoc-json-render agent factories from `type:
"tanstack"` to `type: "custom"` with a dedicated stream converter,
matching the proven pattern used by the main built-in-agent factory
(`tanstack-factory.ts`)
- Remove invalid `response_format: { type: "json_object" }` from
`modelOptions` -- TanStack AI v0.8.x uses the OpenAI Responses API which
does not support this Chat Completions parameter
- System prompts already enforce JSON-only output; `temperature: 0.2` is
retained as a valid Responses API parameter
The `type: "tanstack"` path routes through the runtime's
`convertTanStackStream` which has a `runFinished` flag (PR #4476) that
blocks all events after the first `RUN_FINISHED`. Combined with the
invalid `response_format` parameter being silently rejected by the
Responses API, this prevented text events from reaching the frontend,
causing the D5 probe to timeout waiting for
`[data-testid="copilot-assistant-message"]`.
## Test plan
- [ ] Verify byoc-hashbrown D5 passes locally with `bin/showcase test
built-in-agent --d5`
- [ ] Verify all other 27 built-in-agent features still pass D5
- [ ] Verify byoc-json-render D5 passes (same fix pattern)
The byoc-hashbrown and byoc-json-render agents used `type: "tanstack"`
which routes through the runtime's `convertTanStackStream`. That
converter has a `runFinished` flag (PR #4476) that blocks all events
after the first RUN_FINISHED, which can prevent text events from
reaching the frontend.
Additionally, both agents passed `response_format: { type: "json_object" }`
via `modelOptions`. TanStack AI's OpenAI adapter v0.8.x uses the
Responses API (`client.responses.create()`), not Chat Completions. The
Responses API does not support `response_format` (it uses `text.format`
instead), so this parameter was silently causing failures.
Fix both issues by:
- Switching from `type: "tanstack"` to `type: "custom"` with a dedicated
stream converter that skips RUN_FINISHED and forwards text events,
matching the proven pattern in tanstack-factory.ts
- Removing the invalid `response_format` from modelOptions (the system
prompt already enforces JSON-only output)
- Keeping `temperature: 0.2` which is valid for the Responses API
## Summary
- Remove `from __future__ import annotations` from 3 llamaindex agents
that were timing out at D5
- The deferred annotations import causes Pydantic to fail with
"class-not-fully-defined" when resolving `Annotated[str, "..."]` tool
parameters, producing a `RUN_ERROR` SSE event instead of streaming text
- The D5 conversation runner cannot detect `RUN_ERROR` as an assistant
response, so it times out after 30s
## Affected agents
- `a2ui_fixed.py` (gen-ui-a2ui-fixed / a2ui-fixed-schema feature)
- `a2ui_dynamic.py` (gen-ui-declarative / declarative-gen-ui feature)
- `tool_rendering_reasoning_chain_agent.py`
(tool-rendering-reasoning-chain feature)
## Root cause
`from __future__ import annotations` (PEP 563) defers evaluation of all
annotations to strings. When the LlamaIndex `AGUIChatWorkflow` passes
`backend_tools` to Pydantic for schema generation, `Annotated[str,
"Origin airport code"]` is stored as the string `"Annotated[str, 'Origin
airport code']"` instead of the actual type. Pydantic cannot resolve
this and raises `PydanticUserError: class-not-fully-defined`.
Agents without `backend_tools` (like `reasoning_agent.py`) were
unaffected because the code path that triggers Pydantic schema
validation is never reached with an empty tool list.
## Test plan
- [x] Built Docker image locally and curled all 3 endpoints directly
- [x] Before fix: all 3 returned `RUN_ERROR` with Pydantic
class-not-fully-defined
- [x] After fix: all 3 stream `TEXT_MESSAGE_CHUNK` events and finish
with `RUN_FINISHED`
- [ ] CI green
- [ ] D5 probe passes for all 3 features on production
The `from __future__ import annotations` import causes all type
annotations to be stored as strings rather than evaluated at
definition time. When the LlamaIndex AGUIChatWorkflow validates
backend_tools via Pydantic, `Annotated[str, "..."]` parameters
fail with "class-not-fully-defined" because Pydantic cannot
resolve the deferred string annotations.
This caused RUN_ERROR on every request to these three agents,
which the D5 conversation runner cannot detect as a response,
leading to the 30s timeout.
Affected agents:
- tool_rendering_reasoning_chain_agent.py (4 backend tools)
- a2ui_fixed.py (display_flight backend tool)
- a2ui_dynamic.py (generate_a2ui backend tool)
Verified locally: all three endpoints now stream
TEXT_MESSAGE_CHUNK events and finish with RUN_FINISHED.
## Summary
The docs-only early return in `ComposedCellInner` only rendered
`DocsLayer`, skipping `LinksLayer` which renders `CommandCell` for demos
with a `command` field. The `npx copilotkit@latest init` starter command
disappeared from the matrix.
Fix: render `LinksLayer` alongside `DocsLayer` when links overlay is
active.
## Test plan
- [ ] CLI Start Command row shows the `npx` command text (not just docs
links)
- [ ] Other docs-only features unaffected
## Summary
- Fix race condition in the D5 auth probe where the post-signout chat
message was sent before React's `useEffect` flushed `setHeaders()`,
causing the request to go out with valid auth headers (no 401, no error
surface, 8s timeout)
- After clicking sign-out, the probe now waits for the auth banner to
flip to `data-authenticated="false"` plus a 500ms settle delay before
triggering the probe send
- Add unit test covering the new "banner doesn't flip" failure mode
## Test plan
- [x] All 7 d5-auth unit tests pass (including new banner-flip test)
- [x] TypeScript compiles clean
- [x] Verified all 18 integrations already have
`data-testid="auth-banner"` with `data-authenticated` attribute
- [ ] CI green
The docs-only early return only rendered DocsLayer, skipping LinksLayer
which is responsible for rendering CommandCell when a demo has a
command field. The npx starter command disappeared from the matrix.
After clicking sign-out, React's useEffect that calls setHeaders() runs
async (after paint). The probe was immediately filling the textarea and
sending a message before useEffect flushed, so the request went out with
valid auth headers — no 401 occurred, no error surface rendered, and the
probe timed out at 8s.
Add a waitForSelector gate on the auth banner's data-authenticated="false"
attribute plus a 500ms settle delay to ensure setHeaders() has flushed
before the probe triggers the post-signout chat send.
The _stub_agent_server_deps fixture was missing stubs for
agents.interrupt_crew and agents.tool_rendering, which were added to
agent_server.py in d6b784ee9 (interrupt demos). The missing
interrupt_crew stub caused ModuleNotFoundError on import, failing 5
tests on both Python 3.10 and 3.12 in the Showcase: Validate workflow.
## Summary
- Add missing `data-testid="copilot-assistant-message"` and
`data-message-role="assistant"` to claude-sdk-python's BYOC hashbrown
renderer, matching langgraph-python and claude-sdk-typescript
- Fix D5 auth probe to use JS-level `element.click()` instead of
Playwright `force: true`, which `<cpk-web-inspector>` silently
intercepts
## Details
**byoc**: The custom `AssistantMessageRenderer` in the hashbrown
renderer was missing the harness testid attributes. Without them the
conversation runner sees 0 assistant messages and times out at 30s.
**auth**: Playwright's `page.click(sel, { force: true })` dispatches
pointer events that the `<cpk-web-inspector>` overlay absorbs before
React's synthetic event system fires the `onClick`. Switching to
`page.evaluate(() => document.querySelector(sel).click())` triggers the
DOM click directly, reliably flipping the auth state so the 401 error
surface appears.
## Test plan
- [x] `showcase test claude-sdk-python --d5 --verbose` — 29/29 green on
two consecutive runs
- [x] No changes to demo functionality — both fixes are test
infrastructure / harness plumbing
The custom _run_reasoning_agent handler had a bug where text messages
weren't rendered by the frontend despite the backend emitting correct
AG-UI events. The stock AGUI handler works with reasoning=False and
aimock fixtures — the D5 probe checks for reasoning keywords in the
transcript, not for REASONING_MESSAGE events specifically.
Locally verified: 28/29 D5 features pass (only auth fails — pre-existing
auth gate regression unrelated to this change).
The CLI Start Command feature (kind: "docs-only") was rendering empty
cells in the dashboard grid because the docs row only displayed when the
"docs" overlay was explicitly toggled on. Since docs are the only content
for docs-only features, show the docs row whenever any content-producing
overlay is active (links, depth, health, or docs), not just docs alone.
The dashboard Dockerfile uses COPY paths prefixed with `showcase/...`
which expect the repo root as build context. The compose entry was
using `build: ./shell-dashboard` which set context to
showcase/shell-dashboard/, causing COPY failures in worktrees.
Split into explicit context (../ relative to compose file = repo root)
and dockerfile path so COPY paths resolve correctly.
The _run_reasoning_agent handler's fallback path (for aimock fixtures
that return plain text with "Reasoning:" prefix) was setting
answer_text="" which skipped emitting any TEXT_MESSAGE events.
CopilotKit requires a text message to render an assistant bubble in
the conversation view -- reasoning events alone produce no visible
DOM element that the D5 probe selectors can match, causing both
reasoning-display and tool-rendering-reasoning-chain to timeout
with 0 assistant messages.
Fix: set answer_text = full_text so the response is emitted as both
a reasoning message (for the ReasoningBlock slot) and a text message
(for the conversation transcript the probe reads).
The production LangGraph server only registered 5 of 25 graphs from
langgraph.json, causing 404s for all unregistered graph endpoints.
Also adds 6 missing agent registrations to the main route and
normalizes deploymentUrl trailing slashes across dedicated routes.
Jordan Ritter