The probe returned at the first snapshot with critic count == 1, so a
regression that briefly shows 1 critic card before the supervisor
re-invokes critique_agent (count grows to 2) would silently pass.
After the validator passes, hold for CHAIN_SETTLE_DWELL_MS (3s) and
re-snapshot. If the second snapshot fails validation, including a
critic count > 1 or the new sub-agent-empty sentinel appearing in
the page text, throw with a clear destabilised-during-dwell message
that surfaces both initial and post-dwell counts.
Adds the empty-sub-agent sentinel from subagents.py to the probe
BOILERPLATE_MARKERS so an empty sub-agent result fails the test
instead of rendering as an empty card.
[first.status, second.status].sort() defaults to lexicographic
ordering — it returned [200, 429] correctly only by accident and
would mis-order [200, 1000]. Pass an explicit (a,b)=>a-b comparator
so the assertion holds for any future status-code pair.
_invoke_sub_agent's last-resort branch silently returned '' or a
Python repr like \"[{'type': 'text', ...}]\" for block-list content,
which the UI would render as a blank/garbled card. Return a stable
SUB_AGENT_EMPTY_SENTINEL ('<sub-agent produced no output>') instead
so the d5-subagents probe can match it against its boilerplate-marker
list and fail the genuine-pass test loudly when a sub-agent produces
no usable output.
The get_stock_price tool returned randint-based price/change every
call, so e2e specs asserting $338.37 / -2.96% would only pass when
aimock short-circuited the entire call. The Python tool body still
runs server-side under aimock — only the LLM call is mocked.
Mirror the deterministic-`value` pattern on roll_d20: accept optional
price_usd / change_pct arguments and echo them back when present.
Defaults to random mock data when the args are omitted, preserving
the legacy live-LLM behaviour.
The probe sent "show your reasoning step by step" and accepted any
assistant transcript containing "reasoning", "step", or "thinking".
Those tokens are typical assistant acknowledgements of the prompt
itself, so the keyword fallback always matched regardless of
whether the framework actually surfaced REASONING_MESSAGE_* events
to the frontend — making the assertion non-genuine.
Strict-only behavior now: the assertion requires one of the four
stable role/testid markers (reasoning-block / reasoning-content /
reasoning-default / [data-message-role="reasoning"]) to render
within the timeout. Integrations that rely on CopilotKit's default
reasoning slot without a stable testid will fail this probe — by
design.
Replace REASONING_KEYWORDS export with REASONING_SELECTORS, drop
the readAssistantTranscript helper (no longer needed), and update
the test to exercise the testid-presence success path and the
timeout failure path.
The HELLO_LEADING phrase was the showcase-assistant catch-all
boilerplate ('I can help you with weather lookups...') that other
tests in this PR explicitly guard AGAINST. The dedicated d5-all.json
fixture for 'Say hello in one short sentence' now returns a distinct
non-boilerplate greeting; the spec asserts that distinct phrase, so a
fixture-priority misroute fails loudly instead of passing by accident.
PILL_GRADIENT_HINTS contained naked hex prefixes (e.g. "#0", "#1",
"#2", "#ff", "#fb", "#3"-"#9") that accidentally cross-matched
across families — cosmic's #1e3a8a matches forest's "#1", so a
regression returning the same gradient for every pill could have
silently passed.
Drop ALL naked-hex prefix hints. Keep only color-name keywords:
- sunset: sunset, orange, red, rose, pink, amber, coral, peach
- forest: forest, green, emerald, lime, olive, teal
- cosmic: cosmic, space, navy, magenta, purple, violet, indigo,
fuchsia
If full-hex matching is ever needed, use full 6-digit codes from
the actual fixture rather than 1-2 character prefixes.
Also fix the broken "background did not change off baseline" test:
the previous assertion built a baseline-only page mock and then
discarded it via `void page`. Replaced with a fake-timer-driven
invocation that actually triggers the timeout branch, plus a
separate test for the wrong-family path.
The length-mode (responseLength) assertion compared the value-B
delta against the FULL CUMULATIVE pre-A transcript, so on the third
knob pair the math was strongly negative and the threshold check
always failed. Same shape mismatch existed on text-mode.
Thread a per-knob snapshot through each pair so both sides of the
comparison are per-turn deltas:
- Capture priorCumulative BEFORE turn A (chained from the previous
pair's value-B assertion via chainPriorCapture; pair 0 starts
with empty since the page is blank pre-run).
- Snapshot assertion stores aOnly = postA - priorCumulative and
postACumulative for the next subtraction.
- Diff assertion isolates onlyB = postB - postACumulative and
compares aOnly vs onlyB (text equality or length delta).
Tests updated to construct KnobSnapshot bundles via a snapshotFor
helper and exercise the cumulative-prior third-pair scenario for
length-mode.
The default-catchall probe queried `tool-status-{pending,executing,complete}`
but `DefaultToolCallRenderer` emits a single `copilot-tool-render-status`
pill with the lifecycle state on the container's `data-status` attribute.
Collapse the testid array to that single id and update the selector.
The custom-catchall probe queried `custom-catchall-render`, but the
production `CustomCatchallRenderer` emits `custom-wildcard-card`. Update
the constant + selector. Also swap the unmatched `"AAPL stock price"`
prompt for `"What's the current price of AAPL?"`, the verbatim key
already wired in `showcase/aimock/d5-all.json`, so the fixture
deterministically resolves instead of falling through to the live model.
Tests in both .test.ts files updated to match the new constants and
prompt; both files green locally.
The blitz removed the explicit useDefaultRenderTool() call expecting a
framework-level fallback to handle zero-hook registrations, but the
integration uses the published @copilotkit/react-core@1.56.5 which does
not yet ship that fallback. Without the call, useRenderToolCall has no
'*' renderer and tool calls render invisibly — the user only sees the
agent's final text summary instead of the OOTB tool card.
Restore the explicit useDefaultRenderTool() invocation. The framework
fallback (committed in this PR but inert until react-core publishes a
release that includes it) becomes a no-op once that ships.
The earlier V2 canonical sweep misread the brand-name guidance ("always
CopilotKit") as a directive on import paths and stripped /v2 from
@copilotkit/react-core specifiers. Inline review feedback clarified that
guidance applied only to the component name. This restores
"@copilotkit/react-core/v2" and "@copilotkit/react-core/v2/styles.css"
across the docs sweep scope (now including 8 framework quickstarts
inherited via rebase onto main); the <CopilotKit> rename and the drop
of @copilotkit/react-ui from install commands are kept.
Registry demo slugs are dash-form (`agentic-chat`); feature-viewer.copilotkit.ai
serves them at underscore-form (`agentic_chat`). The Code tab iframe was
404'ing because the slug was passed through verbatim. Replace `-` with `_`
when building the code URL.
Wraps the InlineDemo iframe in a Demo / Code tab strip mirroring the
IframeSwitcher component, so pages using <InlineDemo demo="..." /> now
expose both the live demo (integration backend) and a code view. The
code iframe points at feature-viewer.copilotkit.ai/<framework>/feature/
<demo>?view=code&sidebar=false&codeLayout=tabs, where <framework> is
translated through getDocsFolder() to map registry slugs like
langgraph-python down to their upstream folder name (langgraph) used by
the feature viewer.
The mdx-registry shipped a stub IframeSwitcher that took `src`/`title`
props and rendered a single iframe. MDX consumers actually pass
`exampleUrl`/`codeUrl`/`exampleLabel`/`codeLabel`, so the stub was
silently dropping those props and rendering an empty container. The
result: pages using `<IframeSwitcher>` showed no demo+code tabs.
Replace the stub with the real component from `@/components/content`,
which matches the props shape consumers actually use.
IframeSwitcher: forward the `id` prop to a wrapping `<div id={id}>` so MDX
authors can deep-link to a specific switcher instance. The prop was
declared but never consumed.
oss-vs-enterprise.mdx: fix the Frontend SDK bullet to reflect canonical
V2 — both hooks and prebuilt components ship from `@copilotkit/react-core`
(the standalone `@copilotkit/react-ui` package is V1-era).
Several MDX files import `IframeSwitcher` from `@/components/content`
(prebuilt-components, frontend-tools, interactive, tool-rendering, plus
integration overrides), but the component file was missing. This adds it
as a thin wrapper around the existing `<Tabs>` component, rendering a
demo iframe and a code iframe in a Tabs strip.
Props: `exampleUrl`, `codeUrl`, `exampleLabel` (default "Demo"),
`codeLabel` (default "Code"), `height` (default "600px"). Both iframes
are sandboxed and lazy-loaded.
Matches the upstream IframeSwitcher pattern used on docs.copilotkit.ai
to render embedded feature-viewer demos alongside their backing code.
backend/custom-agent.mdx (508 lines, structurally complete) is the canonical
Factory Mode page. The integrations/built-in-agent/custom-agent.mdx copy
(240 lines, missing 5 sections) was retired:
- Add 301 redirects in next.config.ts for the two historical paths
(/built-in-agent/custom-agent and /integrations/built-in-agent/custom-agent
→ /backend/custom-agent).
- Inbound link retargets to /backend/custom-agent landed in the previous
V2 normalization commit (5 files).
- Delete the divergent 240-line copy.
Apply the canonical V2 import form across all V2 docs:
- `<CopilotKit>` (not `<CopilotKitProvider>`)
- imports from `@copilotkit/react-core` (root, not `/v2`)
- styles from `@copilotkit/react-core/styles.css` (not `react-ui/v2/styles.css`)
- drop `@copilotkit/react-ui` from npm install commands
Fix V2 leaks in canonical pages: replace stale `useCopilotAction` and
`useCopilotReadable` references in agentic-protocols/a2a.mdx and
backend/custom-agent.mdx with their V2 equivalents (`useFrontendTool`,
`useAgentContext`).
Excludes intentional V1 references in migrate guides, V1 reference tree,
migration callouts, and V1 release notes.
## Summary
Six commits, batched into one PR for the May 12 docs cutover.
- `2c10c735d` Port upstream CLI walkthrough verbatim onto 8 framework
quickstarts (adk, agno, aws-strands, langgraph, llamaindex, mastra,
microsoft-agent-framework, pydantic-ai). Initial byte-perfect import
from upstream `docs/content/docs/integrations/<fw>/quickstart.mdx`.
- `f17c86867` Add `sample-audio-button` region marker on the
built-in-agent voice demo. Closes a B-cell snippet gap; the bundler now
reports 4 regions on `built-in-agent::voice`.
- `cebca9216` Remove AG-UI brand tab from desktop header for visual
parity with `docs.copilotkit.ai` (canonical has zero AG-UI elements in
the header).
- `abeb2792b` Also remove AG-UI link from mobile slide-out menu and drop
now-unused helpers (`AgUiIcon`, `AG_UI_LINKS`, `AG_UI_PREFIXES`, `Brand`
type, `activeBrandFromPath`, `usePathname`). Net 82-line cleanup of
`brand-nav.tsx`.
- `99d8ac7a5` Intermediate rewrite of the 8 quickstarts to use `npx
copilotkit@latest create --framework <id>` (after CLI accuracy audit
found `--intelligence` and `--framework` are mutually exclusive in
`copilotkit@2.0.3`). Also applies V2 canonical imports to the same 8
files: `<CopilotKit>` from `@copilotkit/react-core` (root), styles from
`@copilotkit/react-core/styles.css`, drop `@copilotkit/react-ui` from
npm install.
- `59a8a71a6` Final CLI-section shape: each quickstart now leads with
the upstream interactive walkthrough (`npx copilotkit@latest create`
plus a bulleted explanation of Project name / Enterprise Intelligence
Platform with sign-up link / Framework picker), followed by an "Or skip
the prompts and pin a framework directly" `--framework <id>` block.
LangGraph and Microsoft Agent Framework show both language variants in
the flag block.
## Test plan
- [ ] Visit each of the 8 framework quickstarts on the dev server.
Verify the CLI section shows the interactive walkthrough with the
EIP/sign-up bullet AND the `--framework <id>` flag block below it.
- [ ] LangGraph quickstart shows both `langgraph-py` and `langgraph-js`
flag commands. Microsoft Agent Framework shows both
`microsoft-agent-framework-dotnet` and `microsoft-agent-framework-py`.
- [ ] All 8 quickstarts: imports show `@copilotkit/react-core` (root),
styles from `@copilotkit/react-core/styles.css`. No `/v2` paths.
- [ ] Inspect shell-docs header at desktop and mobile (narrow viewport).
No AG-UI brand tab anywhere.
- [ ] Visit a built-in-agent voice page that consumes the
`sample-audio-button` region. The snippet renders.
The harness driver `showcase/harness/src/probes/drivers/seo-redirects.ts` is
an intentional inlined copy of `showcase/shell/src/lib/seo-redirects.ts`
(the harness runtime image is built hermetic from harness's own src/, so
the shell package's src/ isn't reachable at runtime). Both copies must
stay in sync per the file's own header — without this update the monthly
redirect-decommission report would surface ~480 phantom candidates after
the shell catalogue retargeting in the previous commits.
Body content is byte-identical to the new shell + shell-docs catalogue.
The "INTENTIONAL COPY" header comment block is preserved unchanged.
Phase 3 of the docs.copilotkit.ai cutover: surface a complete sitemap,
basic robots config, and per-framework self-canonical metadata so each
URL variant (bare /quickstart, /langgraph-python/quickstart,
/agno/quickstart, etc.) is indexed under its own canonical rather than
collapsing onto a single root.
- src/app/sitemap.ts: emit one entry per (root URL, framework variant)
pair plus reference and AG-UI sections. lastModified resolves from
MDX frontmatter `lastmod`, then file mtime, then now. Strips Next.js
route-group `(name)` segments and trailing `/index` so URLs match
what the routers actually serve.
- src/app/robots.ts: allow all, disallow /api/, sitemap pointer at
${NEXT_PUBLIC_BASE_URL}/sitemap.xml.
- src/lib/sitemap-helpers.ts: shared MDX walking + base-URL resolution
used by both metadata routes.
- generateMetadata() on the four catch-all docs routes
([[...slug]], [framework]/[[...slug]], reference/[...slug],
ag-ui/[[...slug]]) sets `alternates.canonical` to the page's own
full URL. Per-framework self-canonical, NOT root canonical.
- .env.example: document NEXT_PUBLIC_BASE_URL and NEXT_PUBLIC_SHELL_URL.
- next.config.ts: extend the existing NEXT_PUBLIC_BASE_URL doc comment
with the new sitemap/robots/canonical consumers.
Part 1 — Retarget seo-redirects.ts destinations:
- Drop the legacy /docs/integrations/ prefix everywhere; shell-docs
serves canonical framework docs at /<fw-slug>/<...> from the host
root.
- Apply registry-slug renames in destinations:
langgraph → langgraph-python
adk → google-adk
aws-strands → strands
microsoft-agent-framework → ms-agent-dotnet
crewai-flows → crewai-crews
unselected → built-in-agent (BIA canonical re-flip)
- Add slug-rename catch-alls for the bare /<old-slug>/* form so legacy
upstream URLs (e.g. /langgraph/quickstart) 301 to the new slug.
- Add /docs/integrations/* and /docs/* catch-alls so any URL still
carrying the legacy SHELL routing prefix lands at the shell-docs
equivalent.
- Add /migration-guides/* → /migrate/* (4 URLs).
- Add folder-index redirects for shell-docs folders without an
index.mdx (/troubleshooting, /migrate, /premium, /concepts,
/reference) so bare folder URLs land on a representative inner page.
Part 2 — Port the redirect middleware to shell-docs:
- Copy the retargeted seo-redirects.ts to shell-docs/src/lib/.
- Merge the SHELL redirect-middleware logic into shell-docs's existing
pageview-tracking middleware: redirects fire first (with seo_redirect
PostHog event), and non-redirected requests still get the
docs_pageview capture and distinct_id cookie.
- Preserve the framework-scoped short-circuit so canonical
/<fw-slug>/<...> URLs are never hijacked by legacy patterns.
- Leave the SHELL versions in place — the SHELL still serves
docs.showcase.copilotkit.ai until DNS flips.
Verified shell-docs and SHELL builds clean. Spot-checked redirects on a
local shell-docs server: /docs/integrations/langgraph/quickstart →
/langgraph-python/quickstart, /langgraph/quickstart →
/langgraph-python/quickstart, /migration-guides/v2 → /migrate/v2,
/troubleshooting → /troubleshooting/common-issues, /coagents →
/langgraph-python, /aws-strands/quickstart → /strands/quickstart.
Upstream renamed sdk-python/copilotkit/langgraph_agent.py to
langgraph_agui_agent.py and the class to LangGraphAGUIAgent. Update the V1
reference autogen entry in scripts/docs/lib/files.ts to point at the new
source and class name, and regenerate the shell-docs MDX. Pipeline now
reports 26/26 entries succeeding (was 25/26).
Resolved d5-all.json conflict by appending B6's headless-simple/complete fixtures (Say hello, joke, fun fact, Highlight, chart) before the tool-rendering and open-gen-ui entries already on integration.
Rewrites 10 D5 smoke probes from keyword-match-on-transcript checks
to side-effect / DOM-mount / network-payload assertions per
Phase-2B of .claude/specs/lgp-test-genuine-pass.md:
- d5-frontend-tools: 3 per-pill turns (Sunset/Forest/Cosmic) asserting
the page background data-attribute mutated to the pill's family-
specific gradient and differs from the prior pill's gradient.
- d5-frontend-tools-async: notes-card testid mount + settled-state
detection (list items OR empty-state copy).
- d5-agent-config: 3 knob pairs (tone/expertise/responseLength) sent
as 6 turns; assertion compares value-A vs value-B response text or
character-count delta. Aimock fixtures key on prompt-encoded knob
tokens since JSON fixtures don't expose context-keyed predicates.
- d5-gen-ui-agent: 3 per-pill turns asserting agent-state-card mounts
with ≥ 2 agent-step rows whose text fingerprint differs across
pills (catches a regression where fixtures stop differentiating).
- d5-gen-ui-declarative: 4 per-pill turns asserting at least one
catalog testid (declarative-card / metric / status-badge /
pie-chart / bar-chart) mounted.
- d5-gen-ui-a2ui-fixed: pill prompt + a2ui-fixed-card testid mount
assertion.
- d5-gen-ui-interrupt: 2 per-pill turns; click first time-picker-slot
after time-picker-card mounts; assert time-picker-picked appears.
- d5-gen-ui-open: pill prompt; iframe[srcdoc] mounts with srcdoc
length ≥ 100 chars.
- d5-shared-state-streaming: 3 per-pill turns asserting
document-content text length ≥ STREAMING_MIN_FINAL_CHARS.
- d5-readonly-state-context: seeds [data-testid=ctx-name] with a
sentinel; installs page.route() interceptor on /api/copilotkit;
asserts the captured outgoing request body contains the sentinel.
Adds _genuine-shared.ts with the structural Page extension
(GenuinePage), pill-click helper, testid waiter, and runtime guard
that mirrors the _beautiful-chat-shared.ts pattern.
All 10 *.test.ts files updated with focused assertions over fakes —
none rely on the production page being live, all 63 unit tests pass.
Adds per-pill aimock fixtures backing the Phase-2B genuine D5 probes:
- agent-config: 6 fixtures (3 knob pairs) so concise vs detailed
responses differ deterministically; satisfies the new probe's
text-diff + length-diff assertions.
- frontend-tools: 3 per-pill fixtures (sunset/forest/cosmic) emitting
change_background tool calls with family-specific gradient hexes.
- frontend-tools-async: query_notes tool call so the NotesCard mounts.
- gen-ui-agent: 3 per-pill set_steps tool calls with distinct step
content so the per-pill content-fingerprint assertion catches
fixture-drift.
- gen-ui-declarative: 4 per-pill generate_a2ui calls + render_a2ui
fixture for the secondary LLM call that paints the catalog.
- gen-ui-a2ui-fixed: SFO/JFK display_flight tool call.
- gen-ui-interrupt: 2 per-pill schedule_meeting tool calls with
distinct time-slot payloads.
- gen-ui-open: generateSandboxedUi tool call with a non-trivial
HTML payload so the iframe[srcdoc]-mount assertion has ≥ 100 chars
to observe.
- shared-state-streaming: 3 per-pill write_document tool calls with
substantive content payloads (≥ 100 chars each).
- readonly-state-context: pill-prompt fixture; the probe's network-
payload assertion checks the request body, not the response.
d5-all.json is updated with the new entries prepended so first-match
precedence routes specific pill prompts to their per-pill fixtures
ahead of the generic adjacent matches.
Adds production-code testids needed by the Phase-2B genuine D5 probes:
- frontend-tools/background.tsx: data-testid='frontend-tools-background'
+ data-background-value mirror so the probe can read the live gradient
off the DOM without computing styles.
- declarative-gen-ui/a2ui/renderers.tsx: data-testid for Card,
StatusBadge, Metric, PieChart, BarChart so per-pill probes can assert
the expected catalog component painted.
- a2ui-fixed-schema/a2ui/renderers.tsx: data-testid='a2ui-fixed-card'
on the Card override so the fixed-schema component-tree mount is
observable without keyword-matching the transcript.
Each cell's Layer 1 spec is already green; this is purely additive — no
spec rewrites.
All three tool-rendering cells now drive the suggestion pills directly
instead of typing free-form prompts that race fixture matchers:
- tool-rendering: 6 tests (page loads + 5 pills) — Weather in SF
asserts SF city + deterministic temp/humidity/wind; Find flights
asserts >=2 flight rows from the dedicated fixture; Stock price
asserts AAPL $338.37 / -2.96%; Roll a d20 asserts exactly 5 d20
cards with the 5th=20; Chain tools asserts weather+flights+d20
cards mount from a single pill click.
- tool-rendering-default-catchall: 6 tests asserting the OOTB default
tool-call renderer paints every tool call with data-testid=
copilot-tool-render plus data-tool-name. Branded sibling-cell
testids stay at zero. Test 6 asserts every card matches the
built-in renderer DOM signature.
- tool-rendering-custom-catchall: 6 tests asserting the same
custom-wildcard-card testid renders for every tool. Test 6 is the
cross-tool snapshot — every tool kind paints via the same shell.
Add stable testids and rendering surfaces for the three tool-rendering
cells so the e2e suite can distinguish each cell's strategy:
- tool-rendering: register useRenderTool for get_stock_price and
roll_d20; new StockCard / D20Card components with testids
stock-card / d20-card / stock-price / stock-change / d20-value.
Rename FlightListCard testid flight-list-card -> flights-card.
- tool-rendering-default-catchall: drop the custom shadcn
useDefaultRenderTool registration so the cell is truly zero
custom-render-hooks. The framework's built-in
DefaultToolCallRenderer now paints every tool call, with stable
data-testid='copilot-tool-render' wrapper plus data-tool-name,
data-args, and data-result attributes for inspection without
expanding the card.
- tool-rendering-custom-catchall: rename the wildcard renderer's
testids from custom-catchall-* to custom-wildcard-* so the cell
is distinguishable from the (now-OOTB) default-catchall demo.
- packages/react-core: when no per-tool / wildcard renderer is
registered, useRenderToolCall now falls back to the built-in
DefaultToolCallRenderer instead of returning null.
Add 14 fixtures at the top of d5-all.json so they win precedence over
the existing 'weather in Tokyo' / 'd5 beautiful-chat probe: search
flights' substring matches:
- Chain tools: 3 toolCalls in one assistant turn (get_weather Tokyo +
search_flights SFO->Tokyo + roll_d20=11) plus follow-up text.
- Weather in SF: get_weather(San Francisco) + text (deterministic).
- Find flights: dedicated search_flights(SFO,JFK) fixture, no longer
leaks the a2ui beautiful-chat shape.
- Stock price: get_stock_price(AAPL) returning $338.37 / -2.96%.
- Roll a 20-sided die: 5 sequential roll_d20 fixtures with
turnIndex 0..4 (stateless across runs), values [7,14,3,19,20].
Replace the random roll_dice tool with roll_d20(value), which echoes
the LLM-supplied value back as the result. Aimock fixtures script the
five sequential calls returning [7, 14, 3, 19, 20] so the e2e suite can
assert exact values rather than rolling until 20 lands.
Update SYSTEM_PROMPT to allow multi-tool chaining when the user
explicitly asks for it (Chain tools pill emits 3 tool calls in one
turn).
Restructure the "Run our CLI" step in all eight framework quickstarts to surface
two paths: the upstream interactive flow (which now covers the Enterprise
Intelligence Platform prompt and sign-up) and the `--framework <id>` flag fast
path. LangGraph and Microsoft Agent Framework keep both language variants in the
flag block.
Phase-2A wrong-target split. The old d5-gen-ui-open probe claimed BOTH
open-gen-ui and open-gen-ui-advanced via the same gen-ui-open literal,
which meant a regression on the advanced demo's iframe sandbox would
never surface (the basic-tier assertion didn't read iframe state). The
advanced demo now owns its own probe asserting iframe presence as the
distinguishing signal:
- d5-gen-ui-open-advanced: drives /demos/open-gen-ui-advanced, asserts
the [data-testid="gen-ui-open-advanced-iframe"] testid (with a
generic iframe fallback).
d5-gen-ui-open: scoped to the basic /demos/open-gen-ui route only;
preNavigateRoute no longer branches on the demos[] context.
d5-feature-mapping.ts: open-gen-ui-advanced now points at the new
gen-ui-open-advanced literal; d5-registry.ts adds the literal.
hitl-in-app — 7 explicit tests (1 skip):
- Page-load and pill-render tests retained.
- New refund #12345 approve/reject pair asserts the deterministic
fixture leading phrases ("I am processing the $50 refund" vs
"refund request was not approved").
- New escalate #12347 approve/reject pair asserts "Escalated ticket
#12347" vs "Not escalated".
- The describe block runs in serial mode so the approve test
(sequenceIndex 0 in the fixture) always runs before the reject test
(sequenceIndex 1) for each pill.
- Downgrade #12346 stays skipped per spec (broken upstream as of
2026-05-07).
frontend-tools-async — 4 explicit tests:
- Page-load test asserts composer + 3 pills.
- Project-planning, auth, and reading pills each click and assert the
Notes DB card renders with the correct keyword heading and the
per-note testid rows that the async handler returned. Anti-regression
assertions catch the previous fixture-priority bugs (generic-plan
boilerplate, showcase-assistant catch-all).
- Reading pill locks the full canonical shape per spec test #4: keyword,
match count, note title, content lines, tag chip, and the assistant
narration leading phrase.
No production-code testid changes — the existing dialog and notes-card
testids cover every assertion. Fixture work lives in d5-all.json (prior
commit).
Phase-2A wrong-target split. Both registry IDs previously pointed at
d5-tool-rendering, which asserts a per-tool WeatherCard — the wrong
signal for catchall renderers. Each catchall surface now owns a probe
that asserts its own testid contract:
- d5-tool-rendering-default-catchall: drives the built-in default
catchall renderer, asserts [data-testid="copilot-tool-render"] +
[data-tool-name="get_weather"] + a tool-status-* lifecycle pill.
- d5-tool-rendering-custom-catchall: drives the user-supplied wildcard
renderer, sends 2 distinct prompts (get_weather + get_stock_price)
and asserts BOTH render through the same [data-testid="custom-catchall-render"]
testid (cross-tool signature snapshot — the wildcard truly catches
all tools, not just one).
Mapping in d5-feature-mapping.ts now points each registry ID at its
own probe; d5-registry.ts adds the two new D5FeatureType literals.
Adds deterministic fixtures to d5-all.json (loaded before feature-parity.json
so it wins match precedence) for the Phase 1A genuine-pass cells:
- hitl-in-app refund #12345 and escalate #12347 pills: each emits a
request_user_approval tool call on the first turn, then branches via
sequenceIndex on the second turn (0 = approve response, 1 = reject
response). Test ordering (serial mode in the spec) drives which branch
is claimed first per pill pair.
- frontend-tools-async project-planning, auth, and reading pills: each
emits a query_notes tool call so the async client-side handler runs
against NOTES_DB and renders project-planning / auth / reading-tagged
matches. The reading pill's 2nd-turn narration locks the canonical
leading phrase per spec test #4.
Replaces nothing — purely additive. The dedicated pill-prompt entries
beat the broad 'plan' and showcase-assistant catch-all entries that were
silently winning before.
- Drop the 8 stale tests that asserted travel-planner shapes
(Current Itinerary / supervisor-indicator / .bg-gray-50). They
predated the supervisor + 3-subagent rewrite and could not catch
the 3 production bugs.
- New suite (5 tests, 0 skipped):
1) page loads with composer + 3 verbatim suggestion pills + 3
subagent role indicators (testid-based).
2-4) one test per pill (Write a blog post / Explain a topic /
Summarize a topic). Each clicks the pill, waits for all
three role-scoped subagent cards to reach data-status=
"complete", then asserts each card's subagent-result is
non-empty AND does not contain the showcase-assistant
boilerplate fragments. Test 4 also serves as a regression
gate on the delegations reducer fix — without it, that pill
returns HTTP 400 and the cards never reach complete.
5) clicks any pill, waits for terminal state, then asserts the
critic-card count is exactly 1 and stays at 1 with status
complete across a 5s dwell — catches any return of the
supervisor -> critic loop.
- Aimock fixtures: 3 verbatim pill chains (cold exposure training,
LLM tool calling, reusable rockets) added to both d5-all.json and
the harness source d5/mcp-subagents.json. Each chain drives the
full supervisor flow (turnIndex 0..3) plus three nested sub-agent
fixtures so every Researcher/Writer/Critic card surfaces real
prose instead of showcase boilerplate.
Drop the 5 cross-origin contentFrame() / page.on('console', ...) skipped
assertions across the two specs — sandbox=allow-scripts only blocks host
introspection of the iframe DOM, and console-spying on the host page
catches no inner-iframe logs. Replace with iframe-presence assertions:
each pill click must produce iframe[sandbox*='allow-scripts'] with a
non-empty srcdoc (or src) attribute. That is the load-bearing signal
that the open-generative-ui pipeline mounted SOMETHING.
Rewrite each suggestion message string as a short verbatim label that
doubles as a deterministic aimock fixture key (paired with the new
fixtures in showcase/aimock/d5-all.json). Drop pill-title parentheticals
per the cosmetic note in lgp-test-genuine-pass.md so titles read as
natural human prompts; keep the message field aligned with the fixture
key.
Final test counts: 5 minimal (page-load + 4 pill-iframe), 4 advanced
(page-load + 3 pill-iframe). All .skip() removed. The sandbox-function
round-trip (evaluateExpression / notifyHost) is intentionally not
asserted here — that requires a same-origin sandbox option or a
host-side spy on the runtime's sandbox-function-call event, both
deferred to a follow-up.
- subagent-activity-card: emit data-testid="subagent-card-<role>" on
each card wrapper (researcher | writer | critic), data-testid=
"subagent-result" on the Result content div, and data-testid=
"subagent-status" on the status pill. The previous testids
(subagent-activity-card, subagent-activity-result) collapsed across
roles, so the e2e suite couldn't count or content-assert per role.
- delegation-log: render a fixed row of 3 always-visible role
indicators (data-testid="subagent-indicator-<role>") so the page
exposes a stable hook for the load-state assertion regardless of
whether the supervisor has delegated yet.
Phase-2A wrong-target split. The old d5-mcp-subagents probe claimed both
mcp-apps and subagents feature types but routed both to /demos/subagents,
leaving mcp-apps wrong-targeted. Each feature now has its own probe:
- d5-mcp-apps: drives /demos/mcp-apps, asserts iframe presence via the
[data-testid="mcp-app-iframe"] testid (with iframe[sandbox] fallback).
- d5-subagents: drives /demos/subagents, asserts the 3 Phase-1D
subagent-card testids (researcher, writer, critic), enforces the
one-critic invariant, and rejects boilerplate placeholder text.
d5-feature-mapping.ts: comment block updated to reflect the split.
d5-registry.ts: no new literals needed (mcp-apps + subagents already
existed).
- Annotate AgentState.delegations with operator.add reducer so concurrent
sub-agent emissions in one supervisor step accumulate instead of
raising INVALID_CONCURRENT_GRAPH_UPDATE (HTTP 400 on the Summarize pill).
- Update _delegation_update to return only the new entry (the reducer
concatenates) instead of echoing the full prior list, which would
duplicate entries each step under operator.add.
- Cap supervisor -> critique_agent loop at _MAX_CRITIQUE_ITERATIONS
(default 1). Re-entrant critique calls short-circuit with a finish-now
ToolMessage and do not append a second delegation, so the UI shows
exactly one critic card per supervisor run.
- _invoke_sub_agent now walks messages newest-first and returns the
first non-empty AIMessage content (handles list-of-content-blocks
shape too). Prevents the previous failure mode where a final empty
AIMessage made the card Result blank or echoed the showcase-assistant
intro.
- Strengthen supervisor system prompt: each sub-agent must be called
exactly once, with no further calls after critique returns.
The 5 of 7 open-gen-ui pill prompts (3D axis, neural network, quicksort,
calculator, ping-the-host) currently match the broad userMessage: 'hi'
catch-all in feature-parity.json (substring match on 'hi' inside 'Say hi
to the host', 'this', etc.) and resolve to the showcase-assistant
boilerplate greeting instead of emitting the generateSandboxedUi tool
call. That kills the iframe path on 5/7 of the pills.
Add 7 high-priority verbatim-prompt fixtures in d5-all.json (and the
source-of-truth showcase/harness/fixtures/d5/gen-ui-open.json) that emit
deterministic generateSandboxedUi tool calls with inline HTML + CSS in
the tool result. d5-all.json loads BEFORE feature-parity.json so the
first-match-wins ordering naturally beats the catch-all. Pill fixtures
appear at the top of the file so they also win against intra-bundle
specifics. Inline HTML is intentionally minimal — the assertion bar is
iframe presence + non-empty srcdoc, not iframe DOM introspection
(cross-origin blocked under sandbox=allow-scripts only).
Alem Tuzlak