Each non-LGP integration carried its own drifted/stale copy of the e2e specs, causing
inconsistent behavior and noisy diffs across the fleet. Copied langgraph-python's canonical
specs verbatim across ~15 integrations (576 spec files total, SHA-1-verified identical to
LGP) so every integration runs the same assertions.
Also removed 2 orphan specs whose underlying demo pages do not exist:
- showcase/integrations/agno/tests/e2e/hitl-in-chat-booking.spec.ts
- showcase/integrations/built-in-agent/tests/e2e/shared-state-write.spec.ts
Integration-specific variant specs were intentionally left as-is: reasoning-default-render,
byoc-*, agentic-chat-reasoning, and shared-state-write where the demo exists. google-adk and
langgraph-typescript were already in parity from earlier commits and show no new changes.
## Summary
The 6 beautiful-chat demos (spring-ai, strands, langroid, agno,
claude-sdk-typescript, claude-sdk-python) ship three identical
suggestion chips. Against the deployed aimock-backed showcase, all three
were broken:
| Suggestion | Symptom | Cause |
|---|---|---|
| Plan a 3-day Tokyo trip | Returned a generic "Hi there! I'm your
showcase assistant…" greeting | Substring `"hi"` matches inside
**arc*hi*tecture**, hijacked by the broad `userMessage: "hi"` fixture |
| Explain RAG like I'm 12 | aimock 4xx — `"No fixture matched"` | No
fixture |
| Draft a launch email | aimock 4xx — `"No fixture matched"` | No
fixture |
Verified locally against `showcase up spring-ai` in a headed browser —
all three now return on-topic content.
## Fix
Add three full-sentence fixtures before the broad `"hi"` matcher in
`feature-parity.json`. Aimock's matcher is substring + first-match-wins
by file order, so the long sentence matchers win first and the `"hi"`
fixture is never reached for these prompts. Each returns a plausible
markdown response (3-day Tokyo itinerary, open-book-test analogy,
3-paragraph launch email).
## Regression coverage
Replaced the 1-line beautiful-chat placeholders with a 4-test suite for
all 6 integrations:
- Page loads with heading + chat input
- Each suggestion's reply contains the expected keywords (`Day 1|Day
2|Day 3`, `open-book|retrieval|RAG`, `Subject:|co-pilot|launch`)
- Each test ALSO asserts `toHaveCount(0)` against `/I'm your showcase
assistant/i` — if the broad "hi" fixture re-broadens or the new fixtures
are reordered/removed, the tests fail with a useful message.
## Test plan
- [x] `validate-fixture-tool-surface` clean: 141 fixtures × 628 demos,
no drift
- [x] Manual headed-browser verification on `showcase up spring-ai` —
all 3 suggestions return their on-topic responses
- [ ] CI's `Validate Showcase` job stays green
- [ ] On-demand E2E (`/test-aimock <slug>`) passes for any of the 6
integrations
The 6 beautiful-chat demos (spring-ai, strands, langroid, agno,
claude-sdk-typescript, claude-sdk-python) ship three identical
suggestion chips: "Plan a 3-day Tokyo trip", "Explain RAG like I'm
12", and "Draft a launch email". Against the deployed aimock-backed
showcase, all three were broken:
- Tokyo trip: hijacked by the broad `userMessage: "hi"` fixture,
because the substring "hi" appears inside "arc**hi**tecture" in
the prompt. Returned a generic "Hi there! I'm your showcase
assistant..." greeting with nothing about Tokyo.
- RAG explain: no fixture matched, aimock returned an error.
- Launch email: same — no fixture, error.
Add three on-topic fixtures with the full suggestion sentence as
`userMessage` (effectively-exact substring match). Place them
before the broad "hi" fixture in the file so first-match-wins
routes each suggestion to the right response.
Add a `beautiful-chat.spec.ts` regression suite to all 6
integrations: send each suggestion, assert the right keywords
appear in the assistant reply ("Day 1/2/3" for Tokyo,
"open-book/RAG" for RAG, "Subject:/co-pilot" for email), AND
assert the hijacked greeting is absent. If the broad "hi" fixture
re-broadens or the new fixtures are reordered/removed, these
tests fail loudly.
Bug: in a single chat session, running both HITL booking flows
back-to-back (Alice 1:1 → then Sales call without refresh) used to
skip the time-picker on the second flow and jump straight to
"Booked ..." text.
Cause: confirmation fixtures were matched on `hasToolResult: true`,
which fires whenever the conversation has ANY tool message in
history. After the first flow finished, the second user message
short-circuited to a confirmation match before the second flow's
toolCall fixture (gated on `hasToolResult: false`) had a chance to
fire. The picker never rendered.
Fix: re-key the two confirmation fixtures on `toolCallId` (the
specific tool_call_id of the matching `book_call` invocation), which
only fires when the LAST conversation message is a tool result with
that id — exactly the moment we want the confirmation. Drop the
`hasToolResult: false` constraint on the toolCall fixtures so they
match a fresh user request regardless of prior tool history.
Add a back-to-back regression test to all 17 hitl-in-chat specs:
walk Alice flow to completion, then sales flow without refresh,
assert two `time-picker-card` elements rendered. If the multi-flow
regression returns, the second card never appears and the test
fails at `toHaveCount(2)`.
The hitl-in-chat demo ships in 17 integrations (langgraph-python plus
16 others — mastra, strands, ag2, agno, crewai-crews,
langgraph-typescript, langgraph-fastapi, pydantic-ai, llamaindex,
langroid, claude-sdk-python, claude-sdk-typescript, ms-agent-python,
ms-agent-dotnet, spring-ai, google-adk). All shipped placeholder e2e
specs that only checked the chat input was visible — none exercised
the actual booking flow.
Replace each with the full booking-flow spec written for
langgraph-python:
1. The "Schedule a 1:1 with Alice" suggestion renders the time-picker
card AND the Tokyo greeting is absent (regression guard against
the broad aimock `userMessage: "Alice"` matcher).
2. Picking a slot transitions to the picked-state card and produces
a "Booked … Alice" assistant follow-up.
3. The "Book a call with sales" suggestion runs the same flow with
the sales attendee.
Also add the matching aimock fixture pair for the sales suggestion
in feature-parity.json — without it, case 3 would only pass against
real OpenAI, not the aimock-backed CI deployments. The pair mirrors
the Alice fixture pair: `book_call` toolCall on first turn,
confirmation message after the picker resolves.
Per-integration coverage matters because each integration has its
own framework-specific HITL wiring (`useHumanInTheLoop` binding to
the agent, agent-side tool registration, run streaming protocol)
that can regress independently of the shared aimock fixture.
The showcase framework directories better reflect their role as
integration examples rather than distributable packages.
Renames showcase/packages/ -> showcase/integrations/ and updates
the test docker-compose file reference accordingly.
copilotkit-qa-bot[bot]