- agno: add default agent alias + per-request runtime
- langgraph-fastapi: add default agent alias
- llamaindex: fix agent name mismatch (byoc_hashbrown → byoc-hashbrown-demo)
- mastra: create dedicated byocHashbrownAgent with hashbrown system prompt
(was using weatherAgent which produced plain text instead of JSON)
- ms-agent-dotnet: upgrade byoc page to V2 CopilotKit import
The D5 conversation runner detects assistant responses via
data-testid="copilot-assistant-message". The byoc-hashbrown demo
overrides the assistantMessage slot with a custom HashBrown renderer,
which dropped that attribute. Without it the harness sees 0 messages
and times out.
## Summary
Adds `--isolate [name]` flag to `showcase test` so multiple
agents/sessions can run D5 probes without Docker container conflicts.
- `showcase test agno --d5 --isolate` — auto-names project
`isolate-<PID>`
- `showcase test agno --d5 --isolate d5verify` — explicit project name
When active, `apply_isolation()` offsets all host ports by +200, renames
containers from `showcase-*` to `<name>-*`, and sets
`COMPOSE_PROJECT_NAME`. The EXIT trap tears down the isolated group and
restores both `local-ports.json` and `docker-compose.local.yml` on exit.
## Test plan
- [x] Verified `--isolate d5verify` shows correct offset ports (3309 =
3109 + 200)
- [x] Containers start with `d5verify-*` names, don't conflict with
existing `showcase-*` group
- [x] Cleanup restores files and tears down containers on exit
- [ ] CI passes
When passed to `showcase test`, creates an isolated Docker Compose
project with offset ports (+200) and renamed containers, allowing
multiple agents/sessions to run showcase tests simultaneously without
container conflicts.
Usage:
showcase test agno --d5 --isolate # auto-names isolate-<PID>
showcase test agno --d5 --isolate d5verify # explicit name
## Summary
Follow-up to #4502. With matrix cells memoized and SSE deltas coalesced,
the dashboard is much smoother — but the **initial dashboard load**
still freezes for a few seconds because:
1. **Initial PB fetch is 10 sequential round-trips.** `fetchInitial`
walks `getList(page=1, 200) → getList(page=2, 200) → …` up to 10 pages,
each awaiting the previous. Network wall time stacks.
2. **First commit with real data is unavoidably a full-matrix
re-render.** The empty-map → populated-map transition invalidates
per-key memo checks on every cell — that's 720 cell renders in one
synchronous React commit, blocking the main thread.
## Changes
- **Parallelize the initial fetch.** Pull page 1 sequentially to learn
`totalItems`, then fire pages 2..N concurrently via `Promise.all`. Wall
time drops from `sum(rtt_per_page)` to roughly `max(rtt_per_page)`
modulo network parallelism. PocketBase reads are independent so this is
safe.
- **`startTransition` around the initial `setRows(initial)`.** The first
big commit is unavoidable, but marking it as a transition lets React 19
yield to user input mid-walk instead of blocking for the entire render.
`setStatus` stays urgent so the "connecting → live" indicator still
flips immediately.
- **`startTransition` around the SSE flush.** Bursts that the 16ms
coalescer can't fully absorb (large reconnect replays, simultaneous
probe completions) still need to yield rather than block.
## Test plan
- [x] `useLiveStatus.test.tsx` 15/15 pass with parallelized fetch (the
mock handles arbitrary page numbers).
- [x] `composed-cell.test.tsx` 9/10 pass (1 pre-existing failure, same
as #4502).
- [x] No new TypeScript errors in `useLiveStatus.ts`.
- [ ] **Manual perf trace before/after on the live dashboard.**
Expected: initial-load wall time drops sharply (sequential → parallel
pages), and any remaining heavy commit no longer blocks the main thread
(transition lets React yield).
- [ ] **Functional sanity** — confirm rows still arrive correctly when
pages return out of order, and the "connecting → live" status transition
still fires before the heavy render.
Add interrupt demos (Strategy B), byoc-json-render demo (zero-tool
agent + @json-render frontend), and shared-state-streaming (per-token
STATE_SNAPSHOT emission via tool-call argument interception).
Spring AI moves from 4 unsupported features to 0.
Replace gen-ui-interrupt and interrupt-headless "not supported" stubs
with working demos using useFrontendTool + async Promise pattern.
Backend agents use system prompt + tools=[] — CopilotKit runtime
routes tool calls to the frontend handler. Pattern proven by
ms-agent-python/dotnet, now extended to ag2, agno, built-in-agent,
claude-sdk-python, claude-sdk-typescript, crewai-crews, google-adk,
langroid, llamaindex, mastra, pydantic-ai, strands.
Split GET /trigger/eval into a two-step flow: GET renders a confirmation
page with a "Run Evaluation" button, POST performs the actual dispatch.
This prevents GitHub's link unfurling bot from auto-triggering evals
when it fetches the URL from PR comments.
The POST handler polls for the Actions run URL after dispatch and opens
it in a new tab via window.open, with a fallback link if the run isn't
found within 5 seconds.
The defaultClick implementation now uses page.evaluate() instead of
page.click(), so the test fakes need to inject the click via the
buildAuthAssertion({ click }) option rather than relying on a page.click()
method that the assertion no longer calls.
The <cpk-web-inspector> overlay intercepts Playwright pointer events even with
force: true, preventing the sign-out button's React onClick from firing. Switch
from page.click(selector, { force: true }) to page.evaluate() with a
document.querySelector(sel).click() call, which triggers the DOM click event
directly without pointer dispatch. This bypasses the overlay and reliably flips
the auth state.
Without this fix the auth D5 probe times out waiting for the error surface
that only appears after a successful sign-out + 401 request cycle.
The AssistantMessageRenderer was missing data-testid="copilot-assistant-message"
and data-message-role="assistant" attributes. Without these, the e2e-deep
conversation runner cannot detect assistant responses (it counts elements by
these selectors), causing a 30s timeout and D5 failure for the byoc feature.
Mirrors the same fix already applied to langgraph-python and
claude-sdk-typescript.
## Summary
- Added `data-testid="copilot-assistant-message"` and
`data-message-role="assistant"` to the BYOC hashbrown and json-render
custom assistant message renderers in claude-sdk-typescript
- These attributes were already present in the langgraph-python
gold-standard; this aligns the TS port
## Why
The byoc-hashbrown and byoc-json-render demos override CopilotChat's
`messageView.assistantMessage` slot with custom renderers. The e2e-deep
conversation runner counts assistant messages via
`[data-testid="copilot-assistant-message"]` to detect "response
settled." Without that attribute on the custom renderers, the count
stayed at 0 forever and the byoc feature timed out -- the only D5
failure blocking claude-sdk-typescript from green.
## Test plan
- `showcase test claude-sdk-typescript --d5 --verbose` goes from 28/29
(byoc timeout) to 29/29 green
- No demo functionality changed -- only test-harness attributes added to
wrapper divs
The byoc-hashbrown and byoc-json-render demos override the CopilotChat
assistantMessage slot with custom renderers, but the overrides were
missing the data-testid="copilot-assistant-message" attribute that the
e2e-deep conversation runner uses to detect settled assistant responses.
Without it, the runner's readMessageCount always returned 0 and the
byoc feature timed out at D5.
The langgraph-python gold-standard already had these attributes; this
aligns claude-sdk-typescript to match.
## Summary
- docs-only features (e.g. cli-start) exist purely for documentation
coverage tracking and have no route, depth probes, or health signals
- The catalog metadata was counting their 18 stub cells in the headline
wired/stub/unshipped/unsupported breakdown, inflating the total and
making the stats bar misleading
- Exclude docs-only cells from the headline counts; add a new
`docs_only` field to track them separately
**Before:** total_cells=720 wired=673 stub=18 unsupported=29
**After:** total_cells=702 wired=673 stub=0 unsupported=29 docs_only=18
## Test plan
- [x] `showcase/scripts` test suite: all 574 tests pass (including 13
catalog tests)
- [x] `showcase/shell-dashboard` test suite: same 3 pre-existing
failures, no regressions
- [x] TypeScript check clean (no new errors)
- [x] Invariant holds: wired + stub + unshipped + unsupported +
docs_only == cells.length
docs-only features (e.g. cli-start) exist purely for documentation
coverage tracking -- they have no route, no depth probes, and no
health signals. The catalog metadata was counting their 18 stub cells
in the headline wired/stub/unshipped/unsupported breakdown, inflating
the total and making the stats bar misleading.
Exclude docs-only cells from the headline counts. A new docs_only
field tracks the excluded count separately so the invariant
(wired + stub + unshipped + unsupported + docs_only == cells.length)
holds.
Before: total_cells=720 wired=673 stub=18 unsupported=29
After: total_cells=702 wired=673 stub=0 unsupported=29 docs_only=18
The llamaindex chat-slots demo only had the welcomeScreen slot,
missing the input.disclaimer and messageView.assistantMessage
overrides that langgraph-python (gold standard) provides. The D5
probe checks for data-testid="custom-assistant-message" after the
assistant responds, so the feature was stuck at D4.
Add CustomAssistantMessage and CustomDisclaimer components (matching
langgraph-python), wire them into CopilotChat props, update the e2e
spec to cover all three slots, and update manifest highlights.
Widen dialog from w-72 to w-[480px], extract key signal fields
(errorDesc, error, failureSummary, backendUrl) as readable key-value
pairs instead of buried JSON, make raw signal collapsible, remove
duplicate tooltip text that repeated badge color/status info.
Suppress RefDepthCell for docs-only features in the parity overlay,
showing '--' instead of a depth chip since docs-only features have no
probe coverage. Add depth distribution counts (D6..D1) to the
AdaptiveStatsBar when the depth overlay is active, giving operators
at-a-glance visibility into how many wired cells are at each depth.
Features with kind "docs-only" show only the docs row in the
dashboard -- no Dx badges, no status badges, no links or depth
layers. The row label is muted and tagged like "testing" rows.
- feature-registry.json: cli-start changed from primary to docs-only
- registry.ts: FeatureKind union extended with "docs-only"
- cell-pieces.tsx: CellStatus returns null for docs-only (hides all
badges)
- composed-cell.tsx: docs-only features render only DocsLayer, skip
links/depth/health
- feature-grid.tsx: docs-only rows use muted italic styling with a
"docs-only" tag
- Tests added for all three behavioral changes
## Summary
Adds a clickable "Run Evaluation" link in a bot comment on every PR. One
click triggers the showcase eval — no Checks tab navigation needed.
### How it works
1. `showcase_eval_check.yml` creates the Check Run AND posts a bot
comment with a signed link
2. The link points to
`eval-webhook-production.up.railway.app/trigger/eval?pr=N&check_run_id=M&sig=HMAC`
3. User clicks → webhook service verifies signature, dispatches eval
workflow, updates Check Run to in_progress, redirects user back to the
PR
4. Eval results appear in the Check Run (already implemented in #4517)
### Security
- Link is HMAC-signed using the webhook secret — can't be forged
- Signature covers `pr:check_run_id` — changing either invalidates the
link
- `EVAL_WEBHOOK_SECRET` added as repo secret for the signing step
### Changes
- `showcase/eval-webhook/src/server.ts` — new `GET /trigger/eval` route
with signature verification + redirect UX
- `.github/workflows/showcase_eval_check.yml` — posts bot comment with
signed link (updates on push, idempotent via HTML marker)
## Test plan
- [ ] Open a test PR → verify bot comment appears with "Run Evaluation"
link
- [ ] Click the link → verify redirect page shows, eval dispatches,
Check Run updates
- [ ] Tamper with the sig parameter → verify 403 response
deriveDepth() was computing D2 for not_supported_features because D1/D2
are integration-scoped. Add unsupported boolean to DepthResult, guard in
deriveDepth, update consumer components. 24/24 depth-utils tests pass.
Fix chat-slots agent name (dashes to underscores), prebuilt-popup
defaultOpen prop, auth demo defaults to unauthenticated state, and
add missing data-testid attributes to byoc renderers.
## Summary
Fixes 15 ag2 showcase files to bring all 29 ag2 features to D5-green
(verified locally with the D5 probe runner).
### Backend fixes
- **Starlette mount order**: `/open-gen-ui-advanced` must mount before
`/open-gen-ui` — Starlette resolves mounts via prefix matching in
registration order, so the shorter prefix was shadowing the longer one
- **`@tool` -> `@tool()` decorator**: bare `@tool` in
`agent_config_agent.py` silently fails to register the function as a
CopilotKit tool; needs the call form `@tool()`
- **headless-complete route wiring**: wired the headless-complete agent
into the `copilotkit-mcp-apps` runtime route so MCP Apps activity
messages render in the headless cell
- **Auth route cleanup**: removed stale `@ts-ignore`
- **Manifest update**: corrected headless-complete's route reference
from `copilotkit` to `copilotkit-mcp-apps`
### Demo page fixes
- **agent-config**: replaced broken `useAgent`/`setState` pattern with
CopilotKit `properties` prop (the correct way to pass config to AG2
ContextVariables)
- **multimodal**: rewrote the legacy-binary-shape converter to a content
flattener — AG2's AGUIStream validates message content as a plain string
and rejects content-part arrays with a 400
- **headless-complete**: changed `runtimeUrl` from `/api/copilotkit` to
`/api/copilotkit-mcp-apps`
- **byoc-hashbrown, byoc-json-render**: re-attached
`data-testid="copilot-assistant-message"` on custom slot overrides
(harness conversation runner needs this to count settled responses)
- **agentic-chat-reasoning, beautiful-chat**: added demo-level
`data-testid` wrappers
- **tool-rendering-reasoning-chain, voice**: fixed CopilotKit import
paths
### Railway rebuild note
Several demos (chat-slots, shared-state-*, subagents) were already
code-correct on `main` but showed D5 failures due to a stale Railway
image. A Railway rebuild (no code changes) will bring those green as
well.
## Test plan
- [ ] 29/29 ag2 features pass D5 probes locally (verified before PR)
- [ ] CI passes
- [ ] After merge + Railway rebuild, ag2 column should be fully D5-green
Rewrite the multimodal demo's message converter from a legacy-binary-shape
rewriter to a content flattener. AG2's AGUIStream validates message content
as a plain string and rejects arrays of content parts with a 400. The new
ContentFlattenerShim extracts text from multipart user messages before the
AG-UI run dispatches them to the AG2 backend.
Also adds defensive validation to sample-attachment-buttons: magic-byte
checks, LFS pointer detection, and actionable error messages.
- Fix Starlette mount order: /open-gen-ui-advanced before /open-gen-ui to
prevent prefix shadowing (open-gen-ui-advanced was unreachable)
- Fix @tool -> @tool() decorator in agent_config_agent.py (bare @tool
silently fails to register the function as a CopilotKit tool)
- Wire headless-complete agent into copilotkit-mcp-apps route so the
headless-complete demo can exercise MCP Apps rendering
- Update manifest to reference copilotkit-mcp-apps route for headless-complete
- Remove stale @ts-ignore in auth route
Add GET /trigger/eval route to eval-webhook with HMAC-signed URLs.
The showcase_eval_check.yml workflow now posts a bot comment with a
clickable "Run Evaluation" link. Clicking triggers the eval and
redirects back to the PR. The link is signed so it can't be forged.
## Summary
- Fix TOOL_CALL_RESULT events reusing parent assistant message ID across
StreamingToolAgent, SharedStateReadWriteController, and
SubagentsController —
React's deduplicateMessages() Map overwrote assistant messages with tool
messages when they shared the same ID
- Add `data-testid="copilot-assistant-message"` to BYOC hashbrown
renderer so
the D5 test harness can detect assistant messages
- Register AG-UI Jackson mixins via JacksonConfig postConfigurer,
normalize
array-format content via ContentNormalizingModule, remove ChatMemory
beans
that interfered with aimock fixture matching, inject Spring-managed
ObjectMapper into AgentConfigController
## Test plan
- [x] All 28/28 spring-ai D5 features pass locally (verified stable
across
multiple consecutive runs)
- [x] Docker build succeeds
- [x] No regressions — previously passing 22 features remain green
Root cause: TOOL_CALL_RESULT events reused the parent assistant
message ID, causing React deduplicateMessages() to overwrite
assistant messages with tool messages (Map keyed by ID). Fix uses
unique UUIDs for each tool result message across all three
controllers. Also adds data-testid to the BYOC hashbrown renderer
so the D5 harness can detect assistant messages, registers AG-UI
Jackson mixins via JacksonConfig postConfigurer, normalizes
array-format content via ContentNormalizingModule, removes ChatMemory
beans that interfered with aimock fixture matching, and injects
Spring-managed ObjectMapper into AgentConfigController.
Covers: failure triage (text-only vs tool features, probe text extraction,
message disappearing, chatMemory pollution), framework-specific gotchas
(LlamaIndex, Spring AI, Built-in Agent, MS Agent, AG2), and production
vs local parity checklist.
- Add buffer length check before timingSafeEqual to prevent RangeError
on missing/malformed X-Hub-Signature-256 headers
- Switch Dockerfile from pnpm to npm with package-lock.json (pnpm
lockfile lives at monorepo root, not in the package directory)
Add --ci flag to eval orchestrator that skips Docker lifecycle and
assumes services are already running. Add ci-native-eval.sh helper
that installs deps, starts next dev + agent servers natively, health-
waits, then runs showcase eval --ci. Fix on-demand E2E workflow with
langgraph-python support and agent-type detection.
Hono web server that receives check_run.requested_action webhooks from
GitHub, authenticates as the devops bot, updates the Check Run to
in_progress, and dispatches showcase_eval.yml via workflow_dispatch.
Includes GHCR build workflow and pnpm workspace registration.
The cell status row rendered RT (D4/e2e) and CV (D5) badges but was
missing the D2/API badge despite the legend already documenting it.
- Add `d2` field to `CellState` sourced from `agent:<slug>` rows
- Render API LiveBadge before RT in CellStatus (D2 < D4 ordering)
- Add API (Agent) to CellDrilldown DIMENSIONS array
- Fix stale FP/D6 references in tests left over from #4513
Jordan Ritter