agent-config: drop the "introduce yourself per your config" baseline — it
substring-matched every prefixed probe prompt (tone:, expertise:,
responseLength:), forcing byte-identical responses and masking the
per-knob behavior the cell is meant to demonstrate.
shared-state-streaming: add toolCallId-keyed follow-up fixtures
(poem/email/quantum) above each first-turn entry so the matcher returns
content-only after the tool result lands. Without these the unchanged
last user message kept re-matching the first-turn fixture, the agent
re-fired write_document, and langgraph hit its 100-step recursion
limit. The frontend state-binding regression that prevents the streamed
document from rendering is a separate framework-level issue.
Synced to showcase/aimock/d5-all.json so replay-mode picks up both
fixes.
Six probe-side fixes landing five additional D5 flips on the
langgraph-python column. None of the fixes touch demo or runtime
code — just the harness-side selectors / signal predicates so each
probe matches what the published langgraph-python image actually
renders today.
Per-probe changes:
- d5-tool-rendering-default-catchall: add a fallback path that
scans copilot-assistant-message bubbles for the literal tool name
+ a 'done'/'running' status label, alongside the strict testid
contract added in unreleased commit ba60df5d3. The strict path
re-engages once @copilotkit/react-core ships a release with the
testid; until then the bubble-text scan is the only stable hook
on 1.56.5.
- d5-reasoning-display: the published built-in
CopilotChatReasoningMessage carries no testid. Accept the verbatim
'Thought for' / 'Thinking…' header text that the slot emits, in
addition to the four pre-existing testid selectors used by the
reasoning-custom override.
- d5-gen-ui-headless-complete: the SuggestionBar renders
agent-generated chip phrasing alternately with the static
useConfigureSuggestions titles. Switch from clicking
'aria-label="Suggestion: <Title>"' to clicking by aria-label
substring with a per-chip alias list (e.g. ['stock', 'aapl'])
that matches BOTH forms. Selector also moves from a custom
text-walk to a CSS attribute-substring selector with the 'i'
modifier — self-contained, no dynamic-code-eval.
- d5-gen-ui-interrupt: widen the post-pick assertion to accept any
of (a) the 'time-picker-picked' testid, (b) the visible 'Booked'
badge text, or (c) the agent's 'scheduled / confirmed' resume
continuation. The picked-state Card unmounts as soon as
langgraph resumes after resolve, so a 5s testid wait races the
unmount; the OR catches whichever signal lands first.
- _beautiful-chat-shared (toggle-theme): two-track signal — pass on
EITHER the html.dark class flipping from its initial reading OR
the visible 'Theme toggled' assistant content rendering. The
class-flip is the strongest signal but useFrontendTool's dispatch
occasionally drops the handler call without dropping the agent's
follow-up content message; track (b) catches the 'tool semantics
reached the UI' state in the meantime.
- _beautiful-chat-shared (search-flights): swap the literal
'United Airlines' wait for the short brand label 'United', which
appears in BOTH the FlightCard a2ui surface's render (when it
paints) and the assistant's '49 / 89' narration. The price
literals ($349 / $289) stay as the canonical 2-flight
fingerprint.
Conversation-runner adjustment:
- Reverse the partial-fix from PR #4724: try chat-input cascade
resolution AT BOOT first (and read baseline at boot too); only
defer to post-preFill on auth-shape demos where the cascade
fails. Without this, demos where preFill itself fires the
message (chip clicks in headless-complete) had baseline read
AFTER the assistant bubble already appeared and the settle
waited indefinitely for further growth that never came.
The demo has TWO chip surfaces with diverging aria-label shapes:
EmptyState (first paint) emits aria-label='Try suggestion: <message>'
with the visible text being the message string, while SuggestionBar
(post-first-message) emits aria-label='Suggestion: <title>' with
visible text being the short title. Which surface is mounted at any
given turn depends on messages.length — timing-dependent. Both my
prior selectors matched only one surface and timed out on the other.
Drop the chip-click preFill entirely; let the runner's normal
fillAndPress hit the textarea with the verbatim prompt. Chip clicks
and textarea-Enter dispatch the same runAgent path, so the fixture
matcher catches either route — and the textarea is always present.
Also refresh e2e-deep.test.ts: a stale test assertion expected the
old REGISTRY_TO_D5 mapping shape (tool-rendering-default-catchall →
tool-rendering D5 type), but the Phase-2A split repointed it to its
own D5 type of the same name. Update the expected skipped list and
sideKeys to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
In replay mode (no real OpenAI), the existing frontend-tools fixtures
match by 'userMessage' substring only. Each pill (Sunset / Forest /
Cosmic) had a single fixture that returned both content + a
change_background toolCall. After the frontend tool ran, the agent's
follow-up LLM call still has the same last user message — substring
matches the same fixture again — agent re-fires change_background,
loops to langgraph's recursion limit, probe's settle window never
quiets, run reds out at the assistant-settle timeout.
Add a toolCallId-keyed follow-up per pill that returns content-only,
plus a static tool_call_id on the first-turn toolCall for the follow-
up to pin to. The follow-up combines userMessage + toolCallId so it
can't cross-match the next pill's first-turn request (which still has
a tool result in history from the prior pill, but a different last
user message).
Probe result: d5:langgraph-python/frontend-tools flips green again in
replay mode — was passing in PR #4724 only because the real-LLM run
that PR was tested against happened to settle without re-firing.
PILL_GRADIENT_HINTS pinned the fixture/aimock output deterministically
with a fixed substring list (color words + the exact hex codes from
the fixtures). Real OpenAI emits any chromatic-family hex (e.g.
forest as '#005f00 / #4caf50') that the fixed substring list can't
enumerate, so the assertion failed for valid green output.
Add a channel-dominance fallback: parse all 6-digit hex codes from
the gradient and accept when at least one satisfies the per-pill RGB
rule (g > r && g > b for forest, r > g && r > b for sunset, b > g
or purple-channel for cosmic). MIN_DOMINANCE_DELTA + MIN_DOMINANT_VALUE
prevent near-grey or near-black hexes from cross-matching. Word and
fixture-hex match still runs first; channel-dominance is the
fallback only.
Probe result: d5:langgraph-python/frontend-tools flips green.
Idiomatic auth demos (langgraph-python) render a SignInCard until
the user clicks 'Sign in with demo token' — the chat textarea only
mounts after that click, which preFill performs. The conversation
runner was resolving the chat-input cascade up front, before any
turn-level preFill ran, so the cascade timed out at 5s on
SignInCard and turn 1 failed with 'chat input not found'.
Move resolution and the initial assistant-message baseline into
the per-turn loop, lazy on first turn. Subsequent turns reuse the
cached selector.
Probe result: d5:langgraph-python/auth flips green.
After PR #4718 + the timeout bump in #4722 unblocked the abort-cascade,
post-merge cycles surfaced the real per-probe failures that were
previously hidden. Each fix here is targeted at one real failure mode
identified from PB error class + message:
* chat-css: HALCYON probe selector was `[class~="bg-muted"]`, but v2
emits PREFIXED utilities (`cpk:bg-muted`). Whole-token match on the
prefixed name (`[class~="cpk:bg-muted"]`) fixes both the static
USER_BUBBLE_INNER_SELECTOR and the page.evaluate selector — and is
unambiguous against `cpk:bg-muted-foreground` on nested children.
* gen-ui-agent: probe assumed `agent-step` DOM rows accumulated across
pills (delta ≥ 2 vs. pre-pill baseline). The backend's `set_steps`
reducer is `last-write-wins` (per `gen_ui_agent.py`), so each pill
REPLACES `state.steps` and the DOM swaps. delta was always 0 after
pill 1, even when the new steps rendered correctly. Drop the
baseline-capture preFill, assert ≥ 2 visible rows + per-pill
fingerprint dedup. Tests rewritten to match the swap model.
* gen-ui-headless-complete: chip click used `button >> text="<title>"`
(Playwright chained selector). The button has a stable
`aria-label="Suggestion: ${title}"` attribute set inline; switch to
`button[aria-label="Suggestion: ${title}"]` so we match the moment
the DOM exists rather than waiting for inner text-node hydration.
* shared-state-streaming: fixture sent `{"content": "..."}` but the
langgraph-python `write_document(document: str)` tool expects
`document`. Mismatch meant `state.document` never updated; probe saw
no content delta. Rename the argument key to `document` in both the
per-feature fixture and `d5-all.json` (the runtime fixture). The
middleware's `tool_argument="document"` config requires this name.
* reasoning-display: fixture only emitted `content` (text response).
Probe asserted the v2 `<ReasoningBlock>` (rendered behind
`[data-testid="reasoning-block"]`), which mounts only when a
reasoning-role message lands in the transcript. Add a `reasoning`
field so aimock emits REASONING_MESSAGE_* events.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #4718 added the LGP D5 coverage wave, taking langgraph-python's
declared D5 feature set to ~28. With FEATURE_CONCURRENCY=2 that's 14
features per worker × ~15s realistic wall-clock = ~210s — over the
prior 3-min cap before any slack. Post-merge cycles confirmed the math:
17 langgraph-python features ran cleanly and flipped green, but the
remaining ~10 hit `errorClass: "abort"` with `errorDesc: "aborted"`,
the abort-cascade signature where the global cap fires before the
per-feature loop reaches them. Per-feature timeout (5min) bounds wedged
features but doesn't bound the slug's total feature count against the
outer cap.
Bump to 10 min — gives ~5x headroom at current feature count. Lighter
integrations (5–10 features, ~60s) still exit early without sitting on
the cap, so cycle wall-clock barely changes (cron stays every 15 min,
concurrency=4 hides the slow integration's tail behind the others).
Revisit `FEATURE_CONCURRENCY` before raising further — Chromium context
count scales with it and the Railway pod's 2.4GB headroom budget assumes
the current value.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The shell-docs port (e2bef7a0b) updated R15/R17 sources to /integrations/built-in-agent
in showcase/shell-docs/src/lib/seo-redirects.ts and refreshed the snapshot fixture, but
left the legacy shell copy (showcase/shell/src/lib/seo-redirects.ts) and the harness
intentional-copy (showcase/harness/src/probes/drivers/seo-redirects.ts) on the old
/builtin-agent value. The redirect-decommission test imports from the legacy shell
file, so the snapshot diverged: fixture says /integrations/built-in-agent but source
generates /builtin-agent. Mirror the retargeting into both copies so all three files
(shell-docs source, shell legacy source, harness synced copy) agree with the fixture
and Validate Showcase passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Two-part redirect work for the docs.copilotkit.ai → shell-docs cutover,
in one PR.
### Part 1 — Retarget destinations in `seo-redirects.ts`
shell-docs serves canonical framework docs at `/{fw-slug}/...` from the
host root (no `/docs/` prefix) and uses different framework slugs from
the legacy SHELL surface. The redirect catalogue has been retargeted
accordingly:
- **Drop the `/docs/integrations/` prefix** from every destination.
- **Apply framework-slug renames** in destinations:
- `langgraph` → `langgraph-python`
- `adk` → `google-adk`
- `aws-strands` → `strands`
- `microsoft-agent-framework` → `ms-agent-dotnet`
- `crewai-flows` → `crewai-crews`
- **Re-flip the BIA → unselected rename** — `unselected/` was retired;
destinations now point at `/built-in-agent/`.
- **Slug-rename catch-alls** for the bare `/{old-slug}/*` form so legacy
upstream URLs (e.g. `/langgraph/quickstart`) 301 directly to the new
slug.
- **`/docs/integrations/*` and `/docs/*` catch-alls** so any URL still
carrying the legacy SHELL routing prefix lands at the shell-docs
equivalent.
- **`/migration-guides/*` → `/migrate/*`** (4 URLs).
- **Folder-index redirects** for shell-docs folders without an
`index.mdx` (`/troubleshooting`, `/migrate`, `/premium`, `/concepts`,
`/reference`).
390 redirect entries total in the new catalogue.
### Part 2 — Port middleware to shell-docs
- Copied the retargeted `seo-redirects.ts` to
`showcase/shell-docs/src/lib/`.
- Merged the SHELL redirect-middleware logic into shell-docs's existing
pageview-tracking middleware. Redirects fire first (with the
`seo_redirect` PostHog event); non-redirected requests still get the
`docs_pageview` capture and `distinct_id` cookie.
- Preserved the framework-scoped short-circuit so canonical
`/{fw-slug}/...` URLs are never hijacked by legacy patterns.
- Left the SHELL versions of `middleware.ts` and `seo-redirects.ts` in
place — the SHELL still serves `docs.showcase.copilotkit.ai` until DNS
flips.
## Test plan
- [x] `npm run build` clean in `showcase/shell-docs/`
- [x] `npm run build` clean in `showcase/shell/` (existing operation
unaffected)
- [x] `curl -sI
http://localhost:3099/docs/integrations/langgraph/quickstart` → 301 to
`/langgraph-python/quickstart`
- [x] `curl -sI http://localhost:3099/langgraph/quickstart` → 301 to
`/langgraph-python/quickstart`; `/aws-strands/quickstart` →
`/strands/quickstart`; `/migration-guides/v2` → `/migrate/v2`;
`/troubleshooting` → `/troubleshooting/common-issues`; `/coagents` →
`/langgraph-python`
- [ ] Validate full set against a running shell-docs instance with
`validate-redirects.ts` (run from `showcase/scripts/` against the
deployed preview)
Test name says "succeeds when SignInCard re-mounts after sign-out" with
signInCardRemounts: true, so the assertion should resolve, not reject. The
.rejects.toThrow(/SignInCard.*did not re-mount/) was a stale leftover from
when the older single-shape probe always failed this scenario; the multi-shape
probe added in d8d32fd38 succeeds on this fixture, but the test wasn't updated
in lock-step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CR Round 2 (CR1): the legacy-shape assertion path stacked hardcoded
sub-timeouts (idiomatic Math.min(3000, timeout) + 3000ms banner-flip
+ 500ms sleep + 2000ms fill + 2000ms press + Math.max(2000,
timeout-3000) error-poll) — total worst-case ~15.5s when the caller
asked for the default 8s. The doc-comment claimed "the total
wall-clock budget remains `timeout`" — false.
Replaced the hardcoded sub-timeouts with a single deadline computed
at function entry (`Date.now() + timeout`) and a helper `remainingMs()`
that bounds every subsequent wait. The assertion now genuinely
respects the caller's budget. Also bounded the idiomatic-detection
inner waitForSelector to `min(200, idiomaticDeadline - now())` so a
100ms-deadline test can't be overshot by a 200ms inner timeout.
Test changes:
- "fails when legacy banner flips but error surface never appears"
now passes signOutTimeoutMs: 4_000 — the prior 200ms test relied
on the buggy stacked-sub-timeouts to reach the legacy path. With
the budget actually honored, idiomatic-detection consumes a tight
budget before legacy can run; that's correct, fail-fast behavior.
- New test "fails fast with 'idiomatic detection consumed the budget'
when caller passes a tight timeout" pins the new fast-fail
behavior so future regressions on this contract are caught.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bucket (a) findings from CR Round 1, fixed inline:
A1. shell-dashboard resolveD5Row precedence
Multi-key D5 cells (beautiful-chat → 5 per-pill keys) returned
the first non-null row as worst, only upgrading on red. A degraded
row encountered after a green row was silently dropped → cell
rendered green when it should have been amber. Replaced the
"only red wins" check with a numeric rank table (red=3, degraded=2,
green=1) so red > degraded > green holds regardless of iteration
order. Added 5 multi-key fan-out tests covering: red-after-green,
red-before-green, degraded-vs-green order independence, red-beats-
degraded, all-green-stays-green.
Symbols touched: resolveD5Row (live-status.ts:176), new
D5_STATE_RANK constant. Call-site enumeration: resolveD5Row is
called only by resolveCell at live-status.ts:371 — same input/
output shape, no caller change needed.
A2. fixture _comment lies about aimock arg shape
Both gen-ui-agent.json and shared-state-streaming.json's _comment
claimed `arguments` MUST be JSON-stringified or aimock silently
drops the call. aimock's `normalizeResponse` (verified in
node_modules/@copilotkit/aimock/dist/fixture-loader.cjs:14-19)
auto-stringifies object-valued arguments at load time, so both
shapes work. Updated the comments to reflect reality and stop
misleading future fixture authors.
A3. e2e-deep per-feature timeout race orphaned runFeature
When the synthetic timer won the Promise.race, runFeature was
abandoned but never told to tear down. Browser context stayed
held until the global timeout eventually fired, while the outer
Semaphore.release ran immediately — a NEW feature could acquire
the slot while the orphan still held the context, silently
exceeding FEATURE_CONCURRENCY's pool budget. Now: a per-feature
AbortController forwards the parent abort signal to runFeature;
when the timer wins, .abort() fires so runFeature's finally chain
tears down its page/context. The setTimeout cleanup is in a
try/finally so a thrown rejection (defensive — runFeature's
contract says no) doesn't leak the timer. The parent-abort event
listener is removed on cleanup to prevent listener accumulation
over many feature iterations.
Symbols touched: per-feature loop body in executeE2eDeepDriver
(e2e-deep.ts:982). runFeature signature unchanged.
A5. d5-chat-css user-bubble inner selector substring too loose
`[class*="bg-muted"]` matches `bg-muted-foreground` too. Real
Tailwind output puts `bg-muted-foreground` on nested children of
the user bubble; the probe could read computed styles off the
wrong element and silently mis-validate. Switched to
`[class~="bg-muted"]` (whole-token match in space-separated
class lists), the standard CSS3 way to express "this exact class
is present on the element."
A7. auth legacy fill/press swallowed errors silently
Legacy-shape assertion's catch block dropped fill/press errors
so a chat-input cascade mismatch (or disabled textarea after
sign-out) produced a generic "error surface did not appear"
timeout instead of the real cause. Now captures the error
message and appends it to the eventual error string so the
failure record names what actually broke.
A9. d5-feature-mapping header listed removed `hitl-steps`
The header's "destinations" list still showed `hitl-steps : 1
demo` even though my PR's narrative says it was removed in
genuine-pass Phase 0 — falsifying a claim my own diff makes.
Updated the header to reflect the current REGISTRY_TO_D5 shape:
`hitl-text-input` covers the 3 in-chat HITL variants (including
the legacy `hitl` alias) and mcp-apps/subagents are split.
A6 reclassified to bucket (b) — the 6s waste on chip-driven probes is
sub-10% of the new 5-min per-feature timeout and not load-bearing for
convergence. Documented in the round summary; can be addressed in a
follow-up via a `noSend` ConversationTurn option.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The probe migrated from `clickable.click()` (Playwright pointer click)
to `clickByJs(page, selector)` (page.evaluate-driven JS click) to
bypass the cpk-web-inspector overlay. The test was still asserting
the old click-mock contract:
expect(click).toHaveBeenCalledWith(selector, opts)
That assertion can never fire because the probe no longer calls
page.click(). Updated to assert page.evaluate() is called with a
function whose source contains the slot selector — the indirect
shape of the new clickByJs path.
Also dropped the "fails when page is missing click()" test — the new
probe doesn't depend on page.click being on the structural Page
interface, so the runtime guard is gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The harness D5 probes run against ALL 18 integrations, but only
langgraph-python has the recent idiomatic refactors. Production PB
showed 16 integrations green on chat-css and chat-slots, and 3 green
on auth — shipping the previous pure-HALCYON / pure-SlotMarker /
pure-SignInCard probes would have regressed those green cells to red.
Made all three probes accept either shape, detected at runtime:
- d5-chat-css: tries HALCYON anchors (ember border + JetBrains Mono +
Fraunces) first, falls back to legacy hot-pink/amber background
anchors. Either passing satisfies the assertion. Combined error
message names both paths so operators can see WHICH theme the
integration was supposed to match against.
- d5-chat-slots: combined CSS-OR selector
`[data-slot-label="MessageView.AssistantMessage"], [data-testid="custom-assistant-message"]`.
Either marker on the page is sufficient evidence the slot wiring fired.
- d5-auth: preFill detects shape via brief SignInCard probe — present →
idiomatic (click sign-in to mount chat); absent → legacy (chat already
mounted, no preFill action). Assertion clicks sign-out then races
SignInCard re-mount (idiomatic pass) against banner-flip + error-surface
(legacy pass). Each path has a discrete error message so operators
can tell which contract failed.
Tests rewritten to exercise both paths for each probe.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The d5-tool-rendering-custom-catchall probe sends two prompts:
"weather in Tokyo" and "What's the current price of AAPL?". Only
the weather pair (tool-call + narration) was present in the fixture,
so the AAPL turn fell through to a non-deterministic response and
the custom-wildcard renderer mounted only one container instead of
two — surfacing as the production "rendered 1 container(s) but is
missing tool name(s)" error.
Add the matching get_stock_price fixture pair (call + narration)
keyed on the substring `current price of AAPL` with the same
hasToolResult disambiguation pattern as weather.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The probe was using Playwright's `page.click()` on the
[data-testid="time-picker-slot"] element. The cpk-web-inspector
overlay intercepts pointer events before React's synthetic event
system picks them up, so the slot's onClick (which calls the
LangGraph interrupt's resolve callback) never fires — the time-picker
never flips into the picked state and the probe times out at step 3.
Lift the JS-level click pattern out of d5-auth.ts:defaultClick into
a shared `clickByJs` helper in `_genuine-shared.ts`, then use it
from gen-ui-interrupt. The JS-level `.click()` triggers the DOM
click event without pointer dispatch, which the inspector overlay
doesn't intercept.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same root-cause pattern as gen-ui-agent: the three pill fixtures
(poem / email / quantum) emitted `arguments` as raw JSON objects.
aimock follows OpenAI's tool-call wire shape where `arguments` must
be a JSON-stringified string — raw objects silently drop the call,
write_document never executes, the shared-state document never
streams content, and the probe times out at 60s.
Stringify all three pill payloads, add explicit `id` fields, and
document the JSON-string requirement in the fixture _comment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The fixture was emitting `arguments` as a raw JSON object on each
set_steps tool call. aimock follows the OpenAI tool-call wire shape
where `arguments` is a JSON-stringified string — a raw object silently
drops the call so the demo's useAgent never sees a state mutation,
the agent-state-card never mounts, and the probe times out at 60s.
Stringify the arguments on all three pill fixtures (launch / offsite /
competitor research) and add explicit `id` fields per call to match
the convention in adjacent fixtures (gen-ui-headless-complete,
beautiful-chat-*). Documented the JSON-string requirement in the
fixture _comment so the next regression is obvious.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The d5 fixture for frontend-tools emits gradients as pure hex with
no English narration:
sunset: linear-gradient(135deg, #ff7e5f 0%, #feb47b 50%, #ff6b6b 100%)
forest: linear-gradient(135deg, #0a3d2e 0%, #166534 50%, #059669 100%)
cosmic: linear-gradient(135deg, #1e3a8a 0%, #6b21a8 50%, #9333ea 100%)
PILL_GRADIENT_HINTS only carried color-name keywords (sunset, orange,
green, etc.), so the keyword-includes check tripped fc=1 the moment
this fixture started shipping pure-hex gradients. Add the literal
6-digit hex codes from each fixture pill to the matching family.
Per the file's existing anti-prefix-collision comment, full 6-digit
codes preserve the cross-pill differentiation guarantee — a
regression that returns the same gradient for every pill still goes
red on turn 2 because the baseline ratchet rejects unchanged values
regardless of the hint family.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The auth demo refactor flipped its lifecycle: unauthenticated is now
the default state, <CopilotKit> only mounts after sign-in, and
sign-out unmounts the entire chat tree (instead of leaving stale
auth headers in a still-mounted chat). The old probe + e2e spec
chased a 401-error-banner surface that no longer exists in the new
demo, plus a brittle 500ms hardcoded `useEffect` flush wait.
Probe rewrite (`d5-auth.ts` + tests):
- Add `buildAuthPreFill` that clicks the SignInCard's sign-in button
before turn 1, then waits for the chat textarea to mount (proves
<CopilotKit> handshook with the runtime).
- `buildAuthAssertion` now clicks sign-out, then waits for SignInCard
to re-mount. The unmount marker IS the proof — no chat-send-and-401
dance is needed (or possible — there's no chat to send into).
- Drop the hardcoded 500ms setTimeout, the unauth-banner wait, and
the error-surface poll. None apply to the new flow.
E2E rewrite (`tests/e2e/auth.spec.ts`):
- "page loads unauthenticated with SignInCard visible"
- "signing in mounts the chat surface with AuthBanner"
- "authenticated send produces an assistant response"
- "signing out unmounts the chat tree and re-renders SignInCard"
- "signing back in re-mounts a fresh chat surface"
Fixture comment updated to reflect the new flow. The user message
("auth check turn 1") and content response are unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The headless-complete refactor replaced the old 5-chip set
(Weather/AAPL/Highlight/Sketch/Largest continent) with a new 4-chip
set wired through `useConfigureSuggestions`/SuggestionBar:
Weather → "What's the weather in Tokyo?" → WeatherCard
Stock price → "What's the price of AAPL right now?" → StockCard
Highlight a note → "Highlight this note for me: 'ship the demo on Friday'."
→ HighlightNote
Revenue chart → "Show me a chart of revenue over the last six months."
→ ChartCard
Probe rewrite:
- Drop the wrapper-scoped chip selector (`[data-testid=
"headless-suggestions"] >> text=...`) — the new SuggestionBar
doesn't have that container testid. Click via plain `button >>
text="<chip title>"` instead.
- Per-turn assert the matching tool card testid mounts
(headless-weather-card, headless-stock-card, headless-highlight-card,
headless-revenue-chart) plus a distinguishing text token. Driven
off a TURN_EXPECTATIONS table so adding a chip is one entry.
- Drop the readMessagesText helper that scoped to a missing
`[data-testid="headless-complete-messages"]` container; collect
text by concatenating all `headless-message-assistant` bubbles.
Fixture rewrite:
- Drop the Excalidraw + Largest-continent entries (no longer in the
chip set). Weather is already covered by feature-parity.json.
- Add a Revenue chart fixture for `get_revenue_chart` with inline
data so ChartCard renders without backend.
- Stock + Highlight fixtures preserved (still match the new chip
messages via substring).
Also restore `data-message-role` attributes on the headless-complete
UserBubble + AssistantBubble so the runner's settle plateau cascade
can resolve them — matches the runner's documented headless-template
contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The headless-simple demo was refactored to a deliberately minimal
"two hooks, one shadcn shell" template — text-in/text-out only, no
gen-UI. The D5 type literal `gen-ui-headless` no longer described
what the probe tests, and the old probe (Profile-card useComponent
+ continent fallback) was asserting against UI that no longer exists.
Three coordinated changes:
1. Rename `gen-ui-headless` D5FeatureType to `headless-simple` so the
slug matches the demo. Updated d5-registry, REGISTRY_TO_D5,
CATALOG_TO_D5_KEY (dashboard), and dependent tests/comments.
`headless-complete` keeps its existing literal because that demo
still drives the full gen-UI surface.
2. Replace d5-gen-ui-headless.{ts,test.ts,fixture} with
d5-headless-simple.ts + headless-simple.json fixture. New probe
clicks the "Say hello in one short sentence." chip and asserts the
`[data-testid="headless-message-assistant"]` bubble mounts with
non-empty content.
3. Restore `data-message-role` attributes on the headless-simple
UserBubble + AssistantBubble. The runner's chat-input cascade
documents these as the headless-template contract; the refactor
dropped them, breaking the runner's settle plateau detection.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The chat-slots refactor replaced the explicit `data-testid="custom-assistant-message"`
attribute with a SlotMarker wrapper component. The probe still asserted
the old testid which no longer exists, so the cell stuck at D4.
Add `data-slot-label={label}` to SlotMarker's outer span — idiomatic
data attribute that mirrors the existing `label` prop and gives the
probe a stable contract without restoring the legacy testid pattern.
The marker was already passing label="MessageView.AssistantMessage";
this just surfaces it in the DOM.
Update d5-chat-slots probe + test to assert
`[data-slot-label="MessageView.AssistantMessage"]`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The chat-customization-css demo was re-skinned to the HALCYON
parchment+ember theme; the probe still asserted the legacy hot-pink
(#ff006e) user bubble and amber (#fde047) assistant bubble. New
theme paints the user bubble's inner bg-muted element with the
halcyon-paper-elevated background plus an ember (#c44a1f) left
border in JetBrains Mono, and the assistant bubble in Fraunces
serif on a transparent background.
Replace the two background-color anchors with three independent
signals — ember rgb on the user inner border-left, JetBrains Mono
fragment on the user inner font-family (with fallback chain so
Linux runners without the webfont still match), and Fraunces fragment
on the assistant font-family. Each signal trips a distinct error so
operators can tell which CSS layer broke.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related changes targeting the fc=97 'Target page closed' D5 failure
on `beautiful-chat-toggle-theme`:
1. Probe (`_beautiful-chat-shared.ts:assertToggleTheme`) — replace the
manual 200ms `page.evaluate` poll over 30s with Playwright's
`waitForFunction`. The native polling is event-driven inside the
browser context, disconnects cleanly on page-close, and avoids
~150 round-trip evaluates per probe. The catch branch now checks
`page.isClosed()` first and surfaces a diagnostic message naming
the renderer-crash case explicitly, so operators don't have to
guess what 'Target page closed' meant.
2. Demo (`use-generative-ui-examples.tsx`) — drop `[theme, setTheme]`
from the `useFrontendTool` deps array. The handler reads `document`
directly and the setter is stable across renders, so the deps
array forced a re-registration after every theme flip. That race
could collide with an in-flight tool result and surface as a
renderer error during multi-turn sequences. Removing deps keeps
the tool registration stable for the duration of the conversation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three related fixes in the e2e-deep driver:
1. Add `isClosed()`, `route()`, `unroute()` to the wrapped E2eDeepPage
so probes that need them (d5-readonly-state-context, beautiful-chat
polling assertions) see Playwright's full surface. Optional in the
interface so test fakes don't need to track them.
2. Skip the captureDiagnostics page.evaluate when the page is closed —
the call would just throw "Target page closed" (already swallowed)
but skipping up-front keeps logs clean.
3. Raise DEFAULT_TIMEOUT_MS from 5min to 30min and add a per-feature
timeout race (default 5min). With ~30 features × ~45s avg /
FEATURE_CONCURRENCY, the previous 5-min ceiling tripped during
normal runs and aborted every still-pending feature, producing the
fc=1 cascade in production PB. The per-feature wall-clock prevents
any one wedged feature from monopolizing the new larger budget.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three orphan mappings in shell-dashboard read PB rows that nothing
emits (`hitl→hitl-steps`, `interrupt-headless`, `tool-rendering-reasoning-chain`),
and seven harness mappings have no dashboard counterpart so cells
that should advance to D5 stayed at D4. Aligned the dashboard map to
mirror REGISTRY_TO_D5 exactly and added a drift test in the harness
that asserts structural equality going forward.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
R15 and R17 sources were /builtin-agent (no hyphen), matching no real traffic. Real legacy URLs live under /integrations/built-in-agent/* (47 URLs in the upstream sitemap). Retarget the source patterns to match.
Two small hygiene fixes:
- frontend-tools cosmic-gradient pill matched bare token 'navy' which
could collide with any prompt mentioning navy. Tighten to
'navy → magenta cosmic gradient' — verbatim substring of the
pill prompt in frontend-tools/suggestions.ts. Mirror into d5-all.json.
- d5-all.json header comment claimed the first 7 entries are the
open-gen-ui pills, but ordering shifts as features land (headless-
simple/complete pills currently sit ahead of open-gen-ui after
recent merges). Rewrite the header to describe the ordering
convention generically: high-priority verbatim-prompt fixtures
appear first, first-match-wins; per-fixture _comments and the
per-feature source files are the source of truth.
The per-feature source file owned only 2 fixtures (one matcher,
'project planning') while the bundled d5-all.json had 6 fixtures
(3 pills × 2 turns each: project-planning, auth, reading). Because
d5-all.json is auto-merged from the per-feature sources, any future
regen would wipe the auth and reading fixtures from the bundle and
silently regress the frontend-tools-async probe.
Copy all 6 fixture entries from d5-all.json into
frontend-tools-async.json so the source is authoritative. No change
to d5-all.json — the bundle already has these entries.
The render_a2ui matcher emitted the same Card+Metric payload for every
pill, so the d5-gen-ui-declarative probe went red on the second pill —
its per-pill expectedTestIds map demands declarative-pie-chart for the
pie-chart pill, declarative-bar-chart for the bar-chart pill, and
declarative-status-badge for the status-report pill, none of which
were rendered.
Branch the render_a2ui response by combining userMessage substring with
toolName so each pill emits the catalog component its probe expects:
- KPI dashboard → Card + 3 Metric children
- pie chart → PieChart with regional sales data
- bar chart → BarChart with quarterly revenue data
- status report → Card + 3 StatusBadge children
Keep the bare toolName-only matcher at the bottom as a deterministic
fallback so unforeseen pills still render something instead of erroring.
Mirror all five fixture entries into the bundled d5-all.json.
Two pill mismatches were silently routing the headless-complete Stock and
Highlight pills to the showcase-assistant catch-all in feature-parity.json:
- The Stock fixture matched 'AAPL trading' but the SuggestionBar pill
configured by use-headless-suggestions.ts sends 'What's the price of
AAPL right now?' — substring 'AAPL trading' is not in that prompt.
- The Highlight fixture matched 'Highlight \'meeting at 3pm\'' but neither
the empty-state pill ('Highlight: ship the demo on Friday') nor the
SuggestionBar pill ('Highlight this note for me: ...ship the demo on
Friday...') contains that substring.
Switch both matchers to short distinctive substrings ('AAPL' and
'ship the demo on Friday') that appear verbatim in BOTH the empty-state
and SuggestionBar prompts. The tool-rendering AAPL fixture (matcher
'What\'s the current price of AAPL?') stays at higher priority via
array order — first-match-wins keeps it pinned to the tool-rendering
pill, so substring 'AAPL' here only catches headless-complete pills.
Update narration response text to reference the correct highlighted
phrase.
Two related issues in the per-pill assertions:
1. The status-report pill listed BOTH `declarative-status-badge` and
`declarative-card` as expected testids and used `.some()` to pass.
Pill 1 (kpi-dashboard) already mounts `declarative-card`, so the
status-report pill trivially passed via leftover even when it
produced no status badge at all.
2. All pills used `.some()` (any-of), so mounting a single satisfying
testid was enough — even one that was leftover from an earlier pill.
Switch the per-pill expected-testid check to `.every()` (conjunctive)
and require the status-report pill's distinguishing testid
(`declarative-status-badge`) explicitly. Add a per-pill baseline
captured via the conversation-runner's `preFill` hook so the assertion
also requires at least ONE expected testid to be NEWLY mounted (not
just present in the DOM as cross-pill leftover).
`[data-testid="agent-step"]` rows accumulate across pills — pill N's
rows persist into pill N+1's DOM. The previous absolute-count check
(`stepCount >= 2`) trivially passed on pill 2/3 even when the pill
produced no new rows; the cumulative-text duplication check was the
only signal catching cross-pill regressions.
Use the conversation-runner's `preFill` hook to capture the agent-step
DOM state BEFORE each pill is sent, then assert the count grew by at
least 2 NEW rows (delta-based, not absolute). The cumulative-text
fingerprint check is kept as a secondary signal to catch fixtures that
return the same canned step list across prompts.
`[data-testid="document-content"]` is sticky DOM that carries text from
pill N into pill N+1's pre-fill state. The previous assertion only
checked the absolute final char count, so pill 2/3 trivially passed on
pill 1's leftover document content even when the stream regressed.
Use the conversation-runner's `preFill` hook to capture the document
content baseline (text + char count) BEFORE each pill is sent, then
after settle assert either:
- delta from baseline >= STREAMING_MIN_FINAL_CHARS (substantive new
content appended), OR
- text differs from baseline AND final char count >= threshold
(document was REPLACED with substantive content of similar size).
Drop the misleading "guards against single-chunk emission" claim from
the docstring — the assertion runs after the runner's settle window,
so mid-stream chunking is NOT observable here. Documented as a
follow-up.
The probe input was "hello" with a misleading comment claiming it was a
no-op. The MCP-apps demo only mounts a sandboxed iframe AFTER an MCP
tool call returns a UI resource — `MCPAppsActivityRenderer` subscribes
to the runtime's MCP activity event and dynamically appends the iframe;
"hello" never triggers that path, so the iframe assertion was either
trivially red or trivially green via DOM leftovers.
Switch the input to the verbatim suggestion-pill prompt
"Open Excalidraw and sketch a system diagram with a client, server,
and database." (from `langgraph-python/src/app/demos/mcp-apps/suggestions.ts`)
and document honestly that the canned aimock fixture currently keyed to
"Use Excalidraw to sketch" returns content-only (no MCP tool call), so
the iframe assertion can only pass under a live agent + reachable MCP
endpoint until the fixture is updated. F4 owns the fixture file — TODO
left in the comment.
The probe sent "render the advanced gen-ui sandbox" — a string with no
fixture entry in `showcase/aimock/d5-all.json` or
`showcase/harness/fixtures/d5/gen-ui-open.json`, so the iframe never
rendered through a deterministic tool call. Switch to the verbatim
suggestion-pill prompt "Inline expression evaluator" which is keyed to
a `generateSandboxedUi` tool call in d5-all.json (line 77).
Also tighten the iframe selector cascade: the bare `iframe` fallback
matched ANY iframe on the page (e.g. third-party analytics frames).
Replace with `iframe[sandbox*="allow-scripts"]`, matching the advanced
demo's actual sandbox spec.
The probe returned at the first snapshot with critic count == 1, so a
regression that briefly shows 1 critic card before the supervisor
re-invokes critique_agent (count grows to 2) would silently pass.
After the validator passes, hold for CHAIN_SETTLE_DWELL_MS (3s) and
re-snapshot. If the second snapshot fails validation, including a
critic count > 1 or the new sub-agent-empty sentinel appearing in
the page text, throw with a clear destabilised-during-dwell message
that surfaces both initial and post-dwell counts.
Adds the empty-sub-agent sentinel from subagents.py to the probe
BOILERPLATE_MARKERS so an empty sub-agent result fails the test
instead of rendering as an empty card.
[first.status, second.status].sort() defaults to lexicographic
ordering — it returned [200, 429] correctly only by accident and
would mis-order [200, 1000]. Pass an explicit (a,b)=>a-b comparator
so the assertion holds for any future status-code pair.
The probe sent "show your reasoning step by step" and accepted any
assistant transcript containing "reasoning", "step", or "thinking".
Those tokens are typical assistant acknowledgements of the prompt
itself, so the keyword fallback always matched regardless of
whether the framework actually surfaced REASONING_MESSAGE_* events
to the frontend — making the assertion non-genuine.
Strict-only behavior now: the assertion requires one of the four
stable role/testid markers (reasoning-block / reasoning-content /
reasoning-default / [data-message-role="reasoning"]) to render
within the timeout. Integrations that rely on CopilotKit's default
reasoning slot without a stable testid will fail this probe — by
design.
Replace REASONING_KEYWORDS export with REASONING_SELECTORS, drop
the readAssistantTranscript helper (no longer needed), and update
the test to exercise the testid-presence success path and the
timeout failure path.
Alem Tuzlak