D5 probes hit real LLMs instead of fixtures because the 8 D5 fixture
files in showcase/ops/fixtures/d5/ were never loaded into aimock.
Railway fetches fixtures from GitHub raw URLs at boot — bundle all
D5 fixtures into d5-all.json and document the updated startCommand
with the new URL appearing before feature-parity.json for match
precedence.
Add 20 new aimock fixtures covering all D5 e2e-deep probe prompts:
- 9 toolCallId second-leg fixtures for tool-call round-trips
- 11 new userMessage fixtures for multi-turn and feature-specific flows
- Explicit tool call IDs on existing weather/pie-chart fixtures
- Correct Alice ordering to prevent substring collision
Without these fixtures, D5 probes hit real OpenAI (401 with test key)
or loop infinitely on tool calls with no second-leg response.
The aimock-fixture-coverage probe scans byoc-json-render.spec.ts and was
flagging two prompts without matching feature-parity.json fixtures:
- "Break down revenue by category as a pie chart"
- "Show me monthly expenses as a bar chart"
Existing pie/bar fixtures used substrings ("revenue distribution by
category", "expenses by category") that don't appear in these prompts,
so the substring matcher returned no hits.
Added two new fixtures with substrings that DO appear in the prompts
("revenue by category as a pie chart", "monthly expenses as a bar
chart") so the coverage probe stays green and aimock returns
deterministic chart data when the BYOC json-render specs run against
the sidecar.
Adds two feature-parity.json fixtures for the langgraph-python
byoc-json-render E2E spec prompts that the
aimock-fixture-coverage probe flagged as uncovered:
- "Break down revenue by category as a pie chart"
- "Show me monthly expenses as a bar chart"
Responses mirror the worked examples baked into the
byoc_json_render_agent system prompt verbatim, so the fixture
output matches what the live LLM would emit (a single
@json-render/react flat-spec JSON object) and the spec
assertions on [data-testid="pie-chart"] /
[data-testid="bar-chart"] resolve deterministically when the
suite runs against the aimock sidecar.
Wire CopilotChat AttachmentsConfig (image + PDF, inline base64) into a
dedicated /demos/multimodal cell backed by a vision-capable LangGraph
agent (gpt-4o). Scoped to its own runtime route so the vision cost
stays on this demo.
- Dedicated /api/copilotkit-multimodal route registering the new
multimodal graph under the multimodal-demo slug
- src/agents/multimodal_agent.py with an AgentMiddleware that flattens
PDF content parts to text server-side via pypdf
- Try with sample image / PDF buttons that drive CopilotChats own
hidden file input via DataTransfer + change event, so the sample and
paperclip paths exercise the same useAttachments pipeline
- public/demo-files/ placeholder — sample.png / sample.pdf binaries
are produced by the user at the end of the Wave 2b rollout
- Manifest + registry + constraints + docs-links wired; normalized
multi-modal -> multimodal in constraints to match the feature
registry id
- QA checklist + Playwright E2E spec (not run yet — pending deploy)
- aimock fixtures for sample prompts so CI runs are deterministic
- Extend the check-binaries hook allowlist to cover the shell-docs and
shell-dojo demo-content.json mirrors (same size as shell/src/data,
already allowed) so regenerated derived data can be committed
9 new userMessage->toolCall/content fixtures in feature-parity.json
for langgraph-python demos. New aimock-fixture-coverage.test.ts
enforces per-spec fixture presence — every E2E spec prompt must
have a matching aimock fixture or be explicitly skipped.
Tighten pie-chart fixture match to avoid beautiful-chat collision:
match specific gen-ui-tool-based prompt text instead of generic
substring that also hits beautiful-chat suggestion pills.
Railway now pulls ghcr.io/copilotkit/aimock:latest directly
via a startCommand that fetches fixtures from GitHub raw URLs.
The wrapper Dockerfile that copied fixtures into the image is
no longer needed.
Also updates README.md to reflect the new architecture:
- References Railway startCommand instead of Dockerfile
- Removes the Dockerfile entry from the fixtures listing
- Updates related workflow descriptions
Substring-match fixtures (pie chart, bar chart, schedule, trip, etc.)
cross-fired across demos with different tool surfaces and returned tool
names the target agent never registered, causing demos to render nothing
in prod when aimock handles traffic.
Fixture changes (showcase/aimock/feature-parity.json):
- Replace generic pie-chart / bar-chart matches with per-suggestion
specific phrases so gen-ui-tool-based gets render_pie_chart /
render_bar_chart directly and beautiful-chat gets pieChart / barChart
with real data (skipping the query_data two-step that caused the
infinite loop on re-matching prompts).
- Narrow schedule+meeting to the Beautiful Chat 30-minute prompt
returning scheduleTime.
- Narrow flight+fly to flights-from-SFO-to-JFK.
- Narrow background to sunset-themed-gradient.
- Remove trip, sales, pipeline, todo: substring-false-firing across
unrelated demos; interrupt/A2UI demos fall through to real LLM.
Guardrail (showcase/scripts/validate-fixture-tool-surface.ts):
- Pure validate() cross-references every fixture's tool-call name
against the tool surface of each demo whose suggestion prompt contains
the fixture's match substring. Loud failure when the fixture returns a
tool the demo's agent does not register.
- CLI walks packages/ collecting suggestions from page.tsx + hooks/,
frontend tools from useComponent / useHumanInTheLoop / useFrontendTool
/ useRenderTool / useDefaultRenderTool, and backend tools via route.ts
agentId->graphId map + langgraph.json graph->file + @tool decorators.
- 7 vitest cases written TDD-first covering the drift detection,
content-only fixtures, case-insensitivity, and multi-tool responses.
- Current state: 33 fixtures x 191 demos, no drift. Counterfactual
(reverting the pie-chart fix) correctly flags gen-ui-tool-based and
declarative-gen-ui.
Also fixes a separate runtime bug in the langgraph-python package
Dockerfile: WORKDIR /app left /app owned by root; the app user could
not create the .langgraph_api cache dir LangGraph's in-memory runtime
needs, so the agent crashed on boot. Added a non-recursive chown
app:app /app (preserves the original perf intent of the explicit
--chown on COPY, which avoided a recursive chown).
Follow-up to #3971. aimock supports fixture schema validation at startup via
--validate-on-load, but it's opt-in. The showcase Dockerfile did not pass
the flag, so fixtures with unrecognized response keys (e.g. "text" instead
of "content") loaded silently and only failed at request time with HTTP 500.
That's what crashed crewai-crews on startup.
Changes:
- showcase/aimock/Dockerfile: pass --validate-on-load so broken fixtures
fail the container boot, not individual requests.
- showcase/scripts/__tests__/aimock-fixtures.test.ts: new vitest spec that
loads feature-parity.json and smoke.json via @copilotkit/aimock's
loadFixtureFile + validateFixtures and asserts zero errors. Runs as part
of the existing showcase-validate CI workflow.
- showcase/scripts/package.json: add @copilotkit/aimock dependency for the
validator import.
Verified red-green: with the pre-#3971 broken "text" fixtures, validateFixtures
flags 5 errors; post-#3971 it returns zero. Docker red-green: container with
an intentionally broken fixture fails to start with "Validation failed: 1
error(s)" and non-zero exit.
The showcase aimock feature-parity fixture had five entries that used the
unrecognized field "text" instead of "content" in the response body.
aimock's /v1/chat/completions handler validates the response shape through
discriminators (isTextResponse, isToolCallResponse, etc.) and when none
match, returns 500 "Fixture response did not match any known type".
When CrewAI's ChatWithCrewFlow.__init__ calls generate_input_description_with_ai,
it sends a prompt containing "Reporting Analyst" / "reports" text (from the
LatestAiDevelopment crew definition). aimock substring-matches this against
the "report" fixture, which had the broken "text" shape, and returned 500.
CrewAI treats that as an InternalServerError, the crewai-crews container
crashes during module import (before uvicorn binds), Railway healthchecks
fail, and the deployment rolls back to the previous ACTIVE build.
Fix:
- Convert the 5 fixtures with "text" (plan, steps, mars, dashboard, report)
to the valid "content" field so they return 200 OK.
- Add a targeted fixture that matches CrewAI's exact startup prompt prefix
("Based on the following context, write a concise") and returns a clean
text description. This catches generate_input_description_with_ai and
generate_crew_description_with_ai before they can fall through to the
more generic fixtures, providing a more predictable response for CrewAI
startup.
Verified locally against @copilotkit/aimock@1.10.0:
- Before: the CrewAI-style prompt returns 500 with the exact error message
seen in Railway logs.
- After: returns 200 with a valid chat.completion envelope.
Note: CrewAI's blocking LLM call at module import remains fragile (any
aimock hiccup crashes the container before it can bind a port). That is
an upstream issue and will be tracked separately.
Jordan Ritter