The specs and QA markdowns had drifted from the demos they describe.
This commit brings every test contract into line with the actual demo
source — eliminating false-greens, false-fails, and stale assertions.
False-fail spec assertions (would fail every run):
- `agentic-chat.spec.ts` — rewrote from the old `change_background` /
`weather-card` / `useAgentContext` flow that no longer exists. New
spec exercises the vanilla `<CopilotChat>` + three suggestion pills
contract the simplified demo actually exposes.
- `gen-ui-tool-based.spec.ts` — asserted on UI text ("Use the sidebar
to generate charts", "Chart Generator") that doesn't exist; switched
to suggestion-pill assertions and scoped the SVG check to inside the
assistant-message bubble (was matching CopilotChat's send-button
SVG).
- `agent-config.spec.ts` — asserted heading "Agent Config Object" but
the demo has "Agent Config".
- `multimodal.spec.ts` — asserted a non-existent "Multimodal
attachments" heading; switched to the `multimodal-demo-root` testid.
- `chat-slots.spec.ts` — asserted `[data-testid="custom-assistant-
message"]` and the bare text "slot" — neither exists. The actual
signal is `data-slot-label="MessageView.AssistantMessage"` from the
SlotMarker wrapper.
- `reasoning-default.spec.ts` — asserted `[data-testid="copilot-
reasoning-message"]` and `[data-message-role="reasoning"]`; neither
is emitted by `CopilotChatReasoningMessage`. Switched to the text-
based "Thinking…/Thought for…" header label.
False-green spec assertions (passed for the wrong reason):
- `shared-state-read.spec.ts` — was a complete false-green: asserted
on "Sales Pipeline", "Total Pipeline", "Active Deals" but the demo
has been a Recipe Editor for some time. Rewrote against the
recipe-card / ingredients-container / instructions-container testids.
- 11 specs (agent-config, beautiful-chat, frontend-tools-async,
gen-ui-tool-based, gen-ui-agent, gen-ui-interrupt, hitl-in-chat,
hitl-in-app, multimodal, readonly-state-agent-context, voice) used
`[data-role="assistant"]` to gate "agent responded" — but the v2
react-core bundle never emits that attribute (it ships
`data-testid="copilot-assistant-message"`). Mechanical sweep to the
correct testid.
- Deleted `shared-state-write.spec.ts` (route consolidated into
`shared-state-read-write` earlier on this branch — spec targeted a
removed demo) and `renderer-selector.spec.ts` (asserted on a radio-
pill UI that no longer exists; the four "Declarative UI" variants
are now separate manifest demos).
QA drift:
- `qa/gen-ui-tool-based.md` documented a "Haiku Generator" demo with
haiku-card / japanese-line / english-line / haiku-image testids — a
demo that doesn't exist anywhere on this branch. Rewrote to match
the chart-rendering demo's actual testids and pill prompts.
- `qa/chat-slots.md` referenced "Custom Slot" pill / "Welcome to the
Slots demo" heading / "This welcome card is rendered via the
welcomeScreen slot." body text — all of which the slot-wrappers
refactor on this branch removed. Updated to match the
`custom-welcome-message` sub-slot that's actually rendered. Also
fixed max-w-4xl → max-w-5xl to match the page.
- `qa/shared-state-read.md` said default instruction is "Preheat oven
to 350 F" but the source has "Preheat oven to 350°F (175°C)".
- `qa/agentic-chat.md` rewrote to match the simplified vanilla-chat
demo (the previous QA documented `change_background` / `WeatherCard`
flows that no longer exist).
- `qa/reasoning-default.md` cited `kind: "testing"` in feature-
registry.json for the `reasoning-default` entry; the registry entry
has no `kind` field. Rewrote without the false cross-file claim.
- Deleted 4 orphan QA files for demos that don't exist:
`agentic-chat-reasoning.md`, `hitl.md`, `hitl-in-chat-booking.md`,
`shared-state-write.md`.
- Renamed `qa/reasoning-default-render.md` → `qa/reasoning-default.md`
to match the manifest cell name.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug: in a single chat session, running both HITL booking flows
back-to-back (Alice 1:1 → then Sales call without refresh) used to
skip the time-picker on the second flow and jump straight to
"Booked ..." text.
Cause: confirmation fixtures were matched on `hasToolResult: true`,
which fires whenever the conversation has ANY tool message in
history. After the first flow finished, the second user message
short-circuited to a confirmation match before the second flow's
toolCall fixture (gated on `hasToolResult: false`) had a chance to
fire. The picker never rendered.
Fix: re-key the two confirmation fixtures on `toolCallId` (the
specific tool_call_id of the matching `book_call` invocation), which
only fires when the LAST conversation message is a tool result with
that id — exactly the moment we want the confirmation. Drop the
`hasToolResult: false` constraint on the toolCall fixtures so they
match a fresh user request regardless of prior tool history.
Add a back-to-back regression test to all 17 hitl-in-chat specs:
walk Alice flow to completion, then sales flow without refresh,
assert two `time-picker-card` elements rendered. If the multi-flow
regression returns, the second card never appears and the test
fails at `toHaveCount(2)`.
The hitl-in-chat demo ships in 17 integrations (langgraph-python plus
16 others — mastra, strands, ag2, agno, crewai-crews,
langgraph-typescript, langgraph-fastapi, pydantic-ai, llamaindex,
langroid, claude-sdk-python, claude-sdk-typescript, ms-agent-python,
ms-agent-dotnet, spring-ai, google-adk). All shipped placeholder e2e
specs that only checked the chat input was visible — none exercised
the actual booking flow.
Replace each with the full booking-flow spec written for
langgraph-python:
1. The "Schedule a 1:1 with Alice" suggestion renders the time-picker
card AND the Tokyo greeting is absent (regression guard against
the broad aimock `userMessage: "Alice"` matcher).
2. Picking a slot transitions to the picked-state card and produces
a "Booked … Alice" assistant follow-up.
3. The "Book a call with sales" suggestion runs the same flow with
the sales attendee.
Also add the matching aimock fixture pair for the sales suggestion
in feature-parity.json — without it, case 3 would only pass against
real OpenAI, not the aimock-backed CI deployments. The pair mirrors
the Alice fixture pair: `book_call` toolCall on first turn,
confirmation message after the picker resolves.
Per-integration coverage matters because each integration has its
own framework-specific HITL wiring (`useHumanInTheLoop` binding to
the agent, agent-side tool registration, run streaming protocol)
that can regress independently of the shared aimock fixture.
Pins the contract that the new full-sentence aimock fixture pair beats
the broad `userMessage: "Alice"` matcher:
1. Sending the suggestion `"Schedule a 1:1 with Alice next week to
review Q2 goals."` renders `[data-testid="time-picker-card"]`,
not the Tokyo greeting. The test explicitly asserts the Tokyo
greeting is absent — `toHaveCount(0)` against
`/Nice to meet you, Alice/i` — so any future broad-match
regression fails here loudly.
2. Clicking a slot transitions to `[data-testid="time-picker-picked"]`
and the assistant follow-up message contains "Booked ... Alice",
verifying the `hasToolResult: true` branch of the fixture pair
also wires through.
Tyler Slaton