Delete 9 recorded fixture files from showcase/aimock/d5-recorded/recorded/
that were captured during a previous real-API recording session. These
fixtures are not needed -- the existing feature-parity.json fixtures
already cover all 4 test cases (Task Manager, Search Flights, PieChart,
BarChart) for both LGP and LGT.
Investigation of the remaining bucket-C D4 cells uncovered that the
'recorded' fixtures shipped in PR #4725 are at best redundant with
hand-curated canonical fixtures already in d5-all.json, and at worst
override them with worse data — the recordings collapsed beautiful-chat's
canonical $349/$289 search_flights payload to the LLM-of-the-day's
$319/$289 (probe asserts $349 verbatim) and reasoning-custom's
canonical fixture (which carries a 'reasoning' field) with a content-only
recording (refused chain-of-thought disclosure).
Net D5 effect of removing the recordings:
- d5:langgraph-python/gen-ui-custom (catalog gen-ui-tool-based)
→ green (canonical d5-all.json #39/#40)
- d5:langgraph-python/reasoning-display (catalog reasoning-custom)
→ green (canonical d5-all.json #91 with 'reasoning' field that
cleanly satisfies the probe's reasoning-role testid contract)
Both flips confirmed live against PocketBase via --d5 --live.
The aimock record/replay infrastructure introduced alongside these
fixtures (showcase/docker-compose.{record,replay}.yml,
showcase/scripts/record-d5-fixtures.mjs) stays — it is reusable for
future demos where canonical fixtures don't yet cover the prompts —
just without the misleading initial recordings.
Bucket-C cells that remain at D4 after this change require fixes
outside the fixture/recording scope; see the PR description for
per-cell findings (release-blocked package testids, frontend-tool
dispatch defect, suggestion-bar misrender, resume-from-interrupt UI,
built-in reasoning-message testid).
Adds a record/replay loop on top of the existing local Docker stack so
each langgraph-python D5 demo can capture real-LLM responses once and
replay them deterministically thereafter. One fewer source of D5 flake
per demo: no more 'tests pass with real OpenAI / fail with aimock'
because the prompts have no fixture coverage.
What lands here:
- showcase/docker-compose.record.yml — overlays aimock with --record
+ --provider-openai/anthropic and a writable mount for the recording
dir. Drops the baseline d5-all.json/feature-parity.json/smoke.json
loads so prompts that already match a stale fixture can still proxy
through to the real provider.
- showcase/docker-compose.replay.yml — same writable mount, no
--record, no provider URLs; layers the per-demo fixtures alongside
the baseline ones for normal probe runs.
- showcase/scripts/record-d5-fixtures.mjs — orchestrator. For each
demo (catalog feature ID), drops any prior consolidated <slug>.json,
restarts aimock to clear in-memory recorded fixtures, snapshots
recorded/, runs the d5 probe through pnpm exec tsx, then merges
every per-call file written under recorded/ into a single
showcase/aimock/d5-recorded/<slug>.json (one fixture per LLM turn,
in chronological order).
- showcase/aimock/d5-recorded/<slug>.json × 6 — initial recordings
for the still-red bucket-C cells: beautiful-chat (8), gen-ui-interrupt
(2), gen-ui-tool-based (1), headless-complete (1), reasoning-custom
(1), tool-rendering-default-catchall (2). 15 fixtures total.
- showcase/aimock/d5-recorded/.gitignore — keeps the per-call
recorded/ scratch dir out of the tree (orchestrator deletes per-call
files after consolidation, this just guards against re-runs).
D5 impact: replaying the recordings flips
d5:langgraph-python/gen-ui-custom (catalog gen-ui-tool-based) from red
to green deterministically. The other five demos still fail their UI
or assertion-side checks, but their LLM-side responses are now fixed,
so the remaining work is probe/UI fixes against a stable baseline
rather than flake hunting.
Aimock recorder requires a one-line patch (turnIndex + hasToolResult
on each recorded fixture's match) for multi-turn flows to record
correctly. Upstream fix proposed for @copilotkit/aimock; until it
ships, the orchestrator probes for the patch and aborts loudly with
the missing-fields message rather than silently producing single-turn
fixtures. See the script's header comment for the exact patch payload.
Jordan Ritter