Files
copilotkit__copilotkit/showcase/aimock/d5-all.json
Alem Tuzlak a3586fb62a fix(showcase): revert reasoning field on aimock d5 reasoning fixture
PR #4579 added a `reasoning` field to the "show your reasoning step by
step" fixture so aimock would emit response.reasoning_summary_* deltas
for the OpenAI Responses API path. Side effect: aimock's Chat
Completions handler also emits non-standard `reasoning_content` deltas
(DeepSeek/Qwen-style) ahead of the role/content chunks. Many
integrations' OpenAI client adapters don't expect those deltas and
either hang or fail to parse the stream — manifesting as "assistant
did not respond within 30000ms" across most reasoning cells in
production.

Restore the original content-only fixture. The langgraph-python /
langgraph-fastapi agent fixes from #4579 still work against real
OpenAI (gpt-5-mini + Responses API streams real reasoning summaries),
but the aimock-driven path no longer exercises the role-reasoning
render — keyword-only assertion in the d5 probe handles that.
2026-05-01 14:07:21 +02:00

531 lines
20 KiB
JSON

{
"_comment": "Bundled D5 (e2e-deep) fixtures. Source files: showcase/harness/fixtures/d5/*.json (auto-merged). Uses hasToolResult matching for multi-turn disambiguation. Loaded by aimock before feature-parity.json for match precedence.",
"fixtures": [
{
"match": {
"userMessage": "introduce yourself per your config"
},
"response": {
"content": "Hello — I'm operating with the configured tone, expertise, and responseLength forwarded by the CopilotKit provider. Adjust the controls above the chat and my system prompt will rebuild on the next turn."
}
},
{
"match": {
"userMessage": "good name for a goldfish"
},
"response": {
"content": "How about Bubbles? It is friendly, classic, and easy to call out at the tank. If you want alternatives: Goldie, Finley, or Mango."
}
},
{
"match": {
"userMessage": "name for its tank"
},
"response": {
"content": "Following the Bubbles theme, you could call the tank The Bubble Bowl. It pairs naturally with the goldfish's name and keeps the playful tone."
}
},
{
"match": {
"userMessage": "what we named the goldfish"
},
"response": {
"content": "We named the goldfish Bubbles, and the tank The Bubble Bowl."
}
},
{
"match": {
"userMessage": "auth check turn 1"
},
"response": {
"content": "Authenticated session is active. The runtime accepted your request because the Authorization header carried the demo bearer token."
}
},
{
"match": {
"userMessage": "verify the css theme rendering"
},
"response": {
"content": "The chat is themed with hot pink user bubbles and amber assistant bubbles. CSS variables are scoped to .chat-css-demo-scope so the theme does not leak."
}
},
{
"match": {
"userMessage": "verify chat slots are wired"
},
"response": {
"content": "Confirmed — the chat-slots demo is rendering through the custom slot wrappers. The CustomAssistantMessage component should be visible as a tinted card with a 'slot' badge in the corner."
}
},
{
"match": {
"userMessage": "fetch the async metric"
},
"response": {
"content": "The async tool resolved with the requested metric. The frontend handler awaited completion before forwarding the result back to the agent — async-streaming behavior confirmed."
}
},
{
"match": {
"userMessage": "switch theme to dark mode"
},
"response": {
"content": "Done — the change_theme tool was invoked and the page is now in dark mode. Frontend tools were registered via useCopilotAction and the agent dispatched the call client-side."
}
},
{
"match": {
"userMessage": "render the a2ui schema"
},
"response": {
"content": "The A2UI fixed-schema component was rendered. The schema-driven UI received the agent's payload and produced the corresponding UI element from the locked schema definition."
}
},
{
"match": {
"userMessage": "have the agent emit a ui"
},
"response": {
"content": "The agent emitted a UI block as part of its turn. The agent acts as the UI generator: its response payload describes the component and the renderer materialized it inline with the assistant message."
}
},
{
"match": {
"userMessage": "Show me a pie chart of revenue by category",
"hasToolResult": false
},
"response": {
"toolCalls": [
{
"id": "call_d5_render_pie_chart_001",
"name": "render_pie_chart",
"arguments": "{\"title\":\"Revenue by Category\",\"description\":\"Revenue breakdown by product category (Q4)\",\"data\":[{\"label\":\"Electronics\",\"value\":42000},{\"label\":\"Clothing\",\"value\":28000},{\"label\":\"Food\",\"value\":18000},{\"label\":\"Books\",\"value\":12000}]}"
}
]
}
},
{
"match": {
"userMessage": "Show me a pie chart of revenue by category",
"hasToolResult": true
},
"response": {
"content": "Pie chart rendered above — Electronics is the largest slice, followed by Clothing, Food, and Books."
}
},
{
"match": {
"userMessage": "Write me a haiku about nature",
"hasToolResult": false
},
"response": {
"toolCalls": [
{
"id": "call_d5_generate_haiku_001",
"name": "generate_haiku",
"arguments": "{\"japanese\":[\"古池や\",\"蛙飛び込む\",\"水の音\"],\"english\":[\"An old silent pond\",\"A frog jumps into the pond\",\"Splash! Silence again.\"],\"image_name\":\"Mount_Fuji_Lake_Reflection_Cherry_Blossoms_Sakura_Spring.jpg\",\"gradient\":\"linear-gradient(135deg, #f5f7fa 0%, #c3cfe2 100%)\"}"
}
]
}
},
{
"match": {
"userMessage": "Write me a haiku about nature",
"hasToolResult": true
},
"response": {
"content": "Haiku generated above — a beautiful verse about nature with an accompanying image."
}
},
{
"match": {
"userMessage": "render the declarative card"
},
"response": {
"content": "The declarative gen-UI specification has been resolved into a rendered card. The component descriptor was forwarded to the frontend renderer which materialized the card declaratively from the schema."
}
},
{
"match": {
"userMessage": "Show me a profile card for Ada Lovelace",
"hasToolResult": false
},
"response": {
"toolCalls": [
{
"id": "call_d5_show_card_001",
"name": "show_card",
"arguments": "{\"title\":\"Ada Lovelace\",\"body\":\"English mathematician (1815\\u20131852), credited as the first computer programmer for her notes on Charles Babbage's Analytical Engine \\u2014 including what is now recognized as the first algorithm intended to be carried out by a machine.\"}"
}
]
}
},
{
"match": {
"userMessage": "Show me a profile card for Ada Lovelace",
"hasToolResult": true
},
"response": {
"content": "Here is a quick card for Ada Lovelace — the rendered card above shows a short biography. Let me know if you want a deeper dive on her work or a different historical figure."
}
},
{
"match": {
"userMessage": "request the gen-ui interrupt"
},
"response": {
"content": "The agent paused at a gen-UI interrupt and rendered a choice component for the user. Choose to continue."
}
},
{
"match": {
"userMessage": "confirm the gen-ui choice"
},
"response": {
"content": "Gen-UI interrupt resolved. The agent received the user's choice and resumed, completing the workflow."
}
},
{
"match": {
"userMessage": "render an open gen-ui element"
},
"response": {
"content": "The open gen-UI element was rendered. The LLM produced an arbitrary-shape JSON payload and the renderer materialized it as a UI block."
}
},
{
"match": {
"userMessage": "continue the advanced gen-ui flow"
},
"response": {
"content": "The advanced gen-UI flow continued with the second-step component. The chained payloads from turns 1 and 2 form the complete advanced gen-UI sequence."
}
},
{
"match": {
"userMessage": "Issue a $50 refund to customer #12345",
"hasToolResult": false
},
"response": {
"toolCalls": [
{
"id": "call_d5_request_approval_001",
"name": "request_user_approval",
"arguments": "{\"message\":\"Issue a $50 refund to customer #12345.\",\"context\":\"Per the standard goodwill-credit policy for shipping delays.\"}"
}
]
}
},
{
"match": {
"userMessage": "Issue a $50 refund to customer #12345",
"hasToolResult": true
},
"response": {
"content": "Approved — processing the $50 refund to customer #12345 now."
}
},
{
"match": {
"userMessage": "trip to mars",
"hasToolResult": false
},
"response": {
"toolCalls": [
{
"id": "call_d5_generate_steps_001",
"name": "generate_task_steps",
"arguments": "{\"steps\":[{\"description\":\"Research Mars mission requirements and timeline\",\"status\":\"enabled\"},{\"description\":\"Design spacecraft and life support systems\",\"status\":\"enabled\"},{\"description\":\"Recruit and train the crew\",\"status\":\"enabled\"},{\"description\":\"Launch and navigate to Mars\",\"status\":\"enabled\"},{\"description\":\"Land and establish base camp\",\"status\":\"enabled\"}]}"
}
]
}
},
{
"match": {
"userMessage": "trip to mars",
"hasToolResult": true
},
"response": {
"content": "Great choices! I will proceed with executing the selected steps for your trip to Mars. Let me work through each one."
}
},
{
"match": {
"userMessage": "Book a 30-minute onboarding call for Alice",
"hasToolResult": false
},
"response": {
"toolCalls": [
{
"id": "call_d5_book_call_001",
"name": "book_call",
"arguments": "{\"topic\":\"Onboarding call\",\"attendee\":\"Alice\"}"
}
]
}
},
{
"match": {
"userMessage": "Book a 30-minute onboarding call for Alice",
"hasToolResult": true
},
"response": {
"content": "Booked Alice's onboarding call for the time you selected — calendar invite is on its way."
}
},
{
"match": {
"userMessage": "trigger the headless interrupt"
},
"response": {
"content": "Interrupt raised. The agent has paused at a graph-level interrupt and is waiting for user input. Reply with 'yes' or 'no' to resume."
}
},
{
"match": {
"userMessage": "resolve the interrupt with yes"
},
"response": {
"content": "Interrupt resolved with 'yes'. The agent resumed execution from the suspended node and produced its final answer."
}
},
{
"match": {
"userMessage": "Research the benefits of remote work and draft a one-paragraph summary",
"hasToolResult": false
},
"response": {
"toolCalls": [
{
"id": "call_d5_research_agent_001",
"name": "research_agent",
"arguments": "{\"task\":\"Benefits of remote work\"}"
}
]
}
},
{
"_comment": "Nested: research sub-agent single-turn LLM call",
"match": {
"userMessage": "Benefits of remote work"
},
"response": {
"content": "- Eliminates commute, returning ~10 hours/week to employees\n- Surveys consistently show higher job satisfaction among remote workers\n- Employers gain access to a geographically unbounded talent pool\n- Reduced office overhead (rent, utilities, maintenance)\n- Trade-offs: ad-hoc collaboration, mentorship of junior staff, and cultural cohesion degrade without intentional replacement rituals"
}
},
{
"match": {
"userMessage": "Research the benefits of remote work and draft a one-paragraph summary",
"turnIndex": 1
},
"response": {
"toolCalls": [
{
"id": "call_d5_writing_agent_001",
"name": "writing_agent",
"arguments": "{\"task\":\"One-paragraph summary on the benefits of remote work, grounded in the research facts.\\n\\nFacts:\\n- Eliminates commute, returning ~10 hours per week\\n- Higher reported job satisfaction in repeated surveys\\n- Wider talent pool for employers, no geographic limit\\n- Reduced office overhead\\n- Trade-offs: collaboration, mentorship, cultural cohesion\"}"
}
]
}
},
{
"_comment": "Nested: writing sub-agent single-turn LLM call",
"match": {
"userMessage": "One-paragraph summary on the benefits of remote work"
},
"response": {
"content": "Remote work returns roughly ten hours a week to employees by eliminating the commute, and repeated surveys show meaningfully higher job satisfaction among remote workers. Employers benefit too: a geographically unbounded talent pool and lower office overhead. The honest counterweight is that ad-hoc collaboration, mentorship of junior staff, and cultural cohesion all degrade without intentional rituals to replace what an office provided implicitly."
}
},
{
"match": {
"userMessage": "Research the benefits of remote work and draft a one-paragraph summary",
"turnIndex": 2
},
"response": {
"toolCalls": [
{
"id": "call_d5_critique_agent_001",
"name": "critique_agent",
"arguments": "{\"task\":\"Remote work returns roughly ten hours a week to employees by eliminating the commute, and repeated surveys show meaningfully higher job satisfaction among remote workers. Employers benefit too: a geographically unbounded talent pool and lower office overhead. The honest counterweight is that ad-hoc collaboration, mentorship of junior staff, and cultural cohesion all degrade without intentional rituals to replace what an office provided implicitly.\"}"
}
]
}
},
{
"_comment": "Nested: critique sub-agent single-turn LLM call",
"match": {
"userMessage": "Remote work returns roughly ten hours"
},
"response": {
"content": "1. Add a specific citation or date range for the surveys mentioned — 'repeated surveys' is vague and weakens the empirical claim.\n2. The ten-hours figure needs sourcing; without it the opening reads as an unsupported assertion rather than a research finding.\n3. The final sentence about 'intentional rituals' is abstract — one concrete example (e.g., scheduled pair-programming sessions, virtual coffee chats) would make the counterweight more actionable."
}
},
{
"match": {
"userMessage": "Research the benefits of remote work and draft a one-paragraph summary",
"turnIndex": 3
},
"response": {
"content": "Here is the summary, after research → drafting → critique:\n\nRemote work returns roughly ten hours a week to employees by eliminating the commute, and repeated surveys show meaningfully higher job satisfaction among remote workers. Employers benefit too: a geographically unbounded talent pool and lower office overhead. The honest counterweight is that ad-hoc collaboration, mentorship of junior staff, and cultural cohesion all degrade without intentional rituals to replace what an office provided implicitly."
}
},
{
"match": {
"userMessage": "describe the sample image"
},
"response": {
"content": "The image attachment shows a small abstract test pattern used by the multimodal demo to validate the image-upload pipeline. Successful render of this response confirms the binary attachment round-tripped through the runtime."
}
},
{
"match": {
"userMessage": "summarize the sample document"
},
"response": {
"content": "The PDF document contains a single test page used by the multimodal demo. Its text was flattened by pypdf on the Python side and forwarded as text content to the model. Receiving this response confirms the document upload path works."
}
},
{
"match": {
"userMessage": "hi from the popup test"
},
"response": {
"content": "Hello from the popup — the CopilotPopup prebuilt component is wired up and reachable. The launcher floats in the corner and the chat sits in an overlay panel."
}
},
{
"match": {
"userMessage": "hi from the sidebar test"
},
"response": {
"content": "Hello from the sidebar — the CopilotSidebar prebuilt component is wired up and reachable. Anything else you would like me to confirm from inside the sidebar?"
}
},
{
"match": {
"userMessage": "recall the user preference"
},
"response": {
"content": "Per the read-only context the user prefers concise responses. The agent received this preference via the shared-state context and is honoring it without writing back to state."
}
},
{
"match": {
"userMessage": "show your reasoning step by step"
},
"response": {
"content": "Reasoning: first, I identified the question requires step-by-step thinking. Then, I broke it into sub-steps and worked through each one. Finally, I aggregated the partial answers into a single response. The reasoning block above the answer demonstrates intermediate-thought rendering."
}
},
{
"match": {
"userMessage": "stream the counter to 5"
},
"response": {
"content": "Streaming state updates: counter advanced from 0 through 1, 2, 3, 4 to 5. Each intermediate value was reflected in the shared state and visible to the UI mid-stream."
}
},
{
"match": {
"userMessage": "remember that my favorite color is blue",
"hasToolResult": false
},
"response": {
"toolCalls": [
{
"id": "call_d5_set_notes_001",
"name": "set_notes",
"arguments": "{\"notes\":[\"Favorite color: blue\"]}"
}
]
}
},
{
"match": {
"userMessage": "remember that my favorite color is blue",
"hasToolResult": true
},
"response": {
"content": "Got it — I have noted that your favorite color is blue."
}
},
{
"match": {
"userMessage": "favorite color"
},
"response": {
"content": "Your favorite color is blue — I noted it earlier."
}
},
{
"match": {
"userMessage": "analyze data and call the tool"
},
"response": {
"content": "Reasoning step 1: I considered the query. Reasoning step 2: I decided to invoke the analysis tool. The tool returned its structured payload, and the reasoning chain wraps the rendered tool card. Both the reasoning block and the tool card should be visible in the transcript."
}
},
{
"match": {
"userMessage": "weather in Tokyo",
"hasToolResult": false,
"toolName": "get_weather"
},
"response": {
"toolCalls": [
{
"id": "call_d5_get_weather_001",
"name": "get_weather",
"arguments": "{\"location\":\"Tokyo\"}"
}
]
}
},
{
"_comment": "Mastra registers the weather tool as get-weather (hyphen); duplicate for compat",
"match": {
"userMessage": "weather in Tokyo",
"hasToolResult": false,
"toolName": "get-weather"
},
"response": {
"toolCalls": [
{
"id": "call_d5_get_weather_001",
"name": "get-weather",
"arguments": "{\"location\":\"Tokyo\"}"
}
]
}
},
{
"match": {
"userMessage": "weather in Tokyo",
"hasToolResult": true
},
"response": {
"content": "Tokyo is 22°C and partly cloudy."
}
},
{
"match": {
"userMessage": "weather in Tokyo"
},
"response": {
"content": "The weather in Tokyo is currently 22°C with partly cloudy skies and light easterly winds."
}
},
{
"match": {
"endpoint": "transcription"
},
"response": {
"transcription": {
"text": "What is the weather in Tokyo?"
}
}
}
]
}