reasoning-default-render.spec.ts and shared-state-write.spec.ts navigate to
/demos/reasoning-default-render and /demos/shared-state-write respectively,
but neither demo directory exists in any integration (including the
langgraph-python gold reference) and neither is declared in any manifest.
They are stale, non-canonical leftovers from the #5127 page-mirror.
Removed from all baselines where present: reasoning-default-render from 8
(langgraph-fastapi never had it) and shared-state-write from all 9. The
parity validator stays green (0 fail) and langgraph-fastapi's prior
spec-under-coverage warning clears once its phantom spec is gone and the 5
canonical specs are added.
The 9 baseline integrations (ag2, agno, crewai-crews, langgraph-fastapi,
langroid, llamaindex, mastra, spring-ai, strands) were page-mirrored from
langgraph-python but the mirror omitted 5 LGP-canonical Playwright specs
whose demos are present on disk:
- declarative-hashbrown
- declarative-json-render
- reasoning-custom
- reasoning-default
- threadid-frontend-tool-roundtrip
Copied each spec verbatim (byte-identical) from langgraph-python, which the
baselines mirror. All 5 backing demo directories exist in every baseline.
The specs are framework-agnostic (navigate by route + testid), so no
per-integration edits are needed. Restores apples-to-apples spec parity.
Each non-LGP integration carried its own drifted/stale copy of the e2e specs, causing
inconsistent behavior and noisy diffs across the fleet. Copied langgraph-python's canonical
specs verbatim across ~15 integrations (576 spec files total, SHA-1-verified identical to
LGP) so every integration runs the same assertions.
Also removed 2 orphan specs whose underlying demo pages do not exist:
- showcase/integrations/agno/tests/e2e/hitl-in-chat-booking.spec.ts
- showcase/integrations/built-in-agent/tests/e2e/shared-state-write.spec.ts
Integration-specific variant specs were intentionally left as-is: reasoning-default-render,
byoc-*, agentic-chat-reasoning, and shared-state-write where the demo exists. google-adk and
langgraph-typescript were already in parity from earlier commits and show no new changes.
## Summary
The 6 beautiful-chat demos (spring-ai, strands, langroid, agno,
claude-sdk-typescript, claude-sdk-python) ship three identical
suggestion chips. Against the deployed aimock-backed showcase, all three
were broken:
| Suggestion | Symptom | Cause |
|---|---|---|
| Plan a 3-day Tokyo trip | Returned a generic "Hi there! I'm your
showcase assistant…" greeting | Substring `"hi"` matches inside
**arc*hi*tecture**, hijacked by the broad `userMessage: "hi"` fixture |
| Explain RAG like I'm 12 | aimock 4xx — `"No fixture matched"` | No
fixture |
| Draft a launch email | aimock 4xx — `"No fixture matched"` | No
fixture |
Verified locally against `showcase up spring-ai` in a headed browser —
all three now return on-topic content.
## Fix
Add three full-sentence fixtures before the broad `"hi"` matcher in
`feature-parity.json`. Aimock's matcher is substring + first-match-wins
by file order, so the long sentence matchers win first and the `"hi"`
fixture is never reached for these prompts. Each returns a plausible
markdown response (3-day Tokyo itinerary, open-book-test analogy,
3-paragraph launch email).
## Regression coverage
Replaced the 1-line beautiful-chat placeholders with a 4-test suite for
all 6 integrations:
- Page loads with heading + chat input
- Each suggestion's reply contains the expected keywords (`Day 1|Day
2|Day 3`, `open-book|retrieval|RAG`, `Subject:|co-pilot|launch`)
- Each test ALSO asserts `toHaveCount(0)` against `/I'm your showcase
assistant/i` — if the broad "hi" fixture re-broadens or the new fixtures
are reordered/removed, the tests fail with a useful message.
## Test plan
- [x] `validate-fixture-tool-surface` clean: 141 fixtures × 628 demos,
no drift
- [x] Manual headed-browser verification on `showcase up spring-ai` —
all 3 suggestions return their on-topic responses
- [ ] CI's `Validate Showcase` job stays green
- [ ] On-demand E2E (`/test-aimock <slug>`) passes for any of the 6
integrations
The 6 beautiful-chat demos (spring-ai, strands, langroid, agno,
claude-sdk-typescript, claude-sdk-python) ship three identical
suggestion chips: "Plan a 3-day Tokyo trip", "Explain RAG like I'm
12", and "Draft a launch email". Against the deployed aimock-backed
showcase, all three were broken:
- Tokyo trip: hijacked by the broad `userMessage: "hi"` fixture,
because the substring "hi" appears inside "arc**hi**tecture" in
the prompt. Returned a generic "Hi there! I'm your showcase
assistant..." greeting with nothing about Tokyo.
- RAG explain: no fixture matched, aimock returned an error.
- Launch email: same — no fixture, error.
Add three on-topic fixtures with the full suggestion sentence as
`userMessage` (effectively-exact substring match). Place them
before the broad "hi" fixture in the file so first-match-wins
routes each suggestion to the right response.
Add a `beautiful-chat.spec.ts` regression suite to all 6
integrations: send each suggestion, assert the right keywords
appear in the assistant reply ("Day 1/2/3" for Tokyo,
"open-book/RAG" for RAG, "Subject:/co-pilot" for email), AND
assert the hijacked greeting is absent. If the broad "hi" fixture
re-broadens or the new fixtures are reordered/removed, these
tests fail loudly.
Bug: in a single chat session, running both HITL booking flows
back-to-back (Alice 1:1 → then Sales call without refresh) used to
skip the time-picker on the second flow and jump straight to
"Booked ..." text.
Cause: confirmation fixtures were matched on `hasToolResult: true`,
which fires whenever the conversation has ANY tool message in
history. After the first flow finished, the second user message
short-circuited to a confirmation match before the second flow's
toolCall fixture (gated on `hasToolResult: false`) had a chance to
fire. The picker never rendered.
Fix: re-key the two confirmation fixtures on `toolCallId` (the
specific tool_call_id of the matching `book_call` invocation), which
only fires when the LAST conversation message is a tool result with
that id — exactly the moment we want the confirmation. Drop the
`hasToolResult: false` constraint on the toolCall fixtures so they
match a fresh user request regardless of prior tool history.
Add a back-to-back regression test to all 17 hitl-in-chat specs:
walk Alice flow to completion, then sales flow without refresh,
assert two `time-picker-card` elements rendered. If the multi-flow
regression returns, the second card never appears and the test
fails at `toHaveCount(2)`.
The hitl-in-chat demo ships in 17 integrations (langgraph-python plus
16 others — mastra, strands, ag2, agno, crewai-crews,
langgraph-typescript, langgraph-fastapi, pydantic-ai, llamaindex,
langroid, claude-sdk-python, claude-sdk-typescript, ms-agent-python,
ms-agent-dotnet, spring-ai, google-adk). All shipped placeholder e2e
specs that only checked the chat input was visible — none exercised
the actual booking flow.
Replace each with the full booking-flow spec written for
langgraph-python:
1. The "Schedule a 1:1 with Alice" suggestion renders the time-picker
card AND the Tokyo greeting is absent (regression guard against
the broad aimock `userMessage: "Alice"` matcher).
2. Picking a slot transitions to the picked-state card and produces
a "Booked … Alice" assistant follow-up.
3. The "Book a call with sales" suggestion runs the same flow with
the sales attendee.
Also add the matching aimock fixture pair for the sales suggestion
in feature-parity.json — without it, case 3 would only pass against
real OpenAI, not the aimock-backed CI deployments. The pair mirrors
the Alice fixture pair: `book_call` toolCall on first turn,
confirmation message after the picker resolves.
Per-integration coverage matters because each integration has its
own framework-specific HITL wiring (`useHumanInTheLoop` binding to
the agent, agent-side tool registration, run streaming protocol)
that can regress independently of the shared aimock fixture.
Replace sys.path.insert hacks in Python agent files with direct
imports via symlinks to shared/{python,typescript}/tools.
Update Dockerfiles, entrypoints, and configs to support the new
symlink-based tool resolution. Add PARITY_NOTES for frameworks
that have known gaps.
The showcase framework directories better reflect their role as
integration examples rather than distributable packages.
Renames showcase/packages/ -> showcase/integrations/ and updates
the test docker-compose file reference accordingly.
github-actions[bot]