mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
Jordan Ritter
9491b89340
Closed PR #5465 introduced cross-fixture leakage by stripping the toolName discriminator on shared {userMessage, context} keys: with aimock's alphabetical first-match-wins ordering, tool-rendering-custom-catchall.json sorts before tool-rendering-default-catchall.json, so default-catchall page requests were served the custom file's content. The probe was structurally blind because it asserted only DOM testids (copilot-tool-render + data-tool-name=get_weather) — both fixtures emit get_weather, so the testid signal passed regardless of which fixture won. Real LGP-gold pattern is disjoint userMessages between default and custom catchall fixtures (NOT shared keys discriminated by toolName). This PR ports the LGP-gold pattern to the other 17 integrations and adds page-text content assertions to both d5 catchall probes so this class of regression can't recur silently: - Probe prompts disjoint between default-catchall and custom-catchall - Fixture userMessages updated to match the new disjoint prompts - Page-text content assertions in both d5 catchall probes (default negatively asserts the custom-catchall leak phrase; custom positively asserts it) Local gold-standard red-green proof captured: - Step A: live staging Playwright baseline (RED-OF-RECORD) - Step B: local control-plane on origin/main reproduces structural fragility (probes pass at testid level, custom fixture wins on default's userMessage path) - Step C: local control-plane on this branch — both catchall probes GREEN with the new content-asserting assertions tool-rendering-default-catchall: pass=true (5443ms) tool-rendering-custom-catchall: pass=true (8956ms) cross-tool signature passed LGP (langgraph-python) was already disjoint; it remains untouched.
67 lines
3.1 KiB
JSON
67 lines
3.1 KiB
JSON
{
|
|
"_meta": {
|
|
"description": "D6 fixtures for claude-sdk-python / tool-rendering-custom-catchall",
|
|
"sourceScript": "d5-tool-rendering-custom-catchall.ts",
|
|
"copiedFrom": "langgraph-python",
|
|
"created": "2026-05-22",
|
|
"updated": "2026-06-15",
|
|
"note": "Dropped the toolName gate on tool-emit fixtures and the no-tool-registered fallback. Reason: when toolName:get_stock_price failed to match (claude-sdk-python backend appears not to forward the tool definition uniformly on the second turn), the userMessage-only fallback returned plain content with no toolCall \u2192 no custom-wildcard card \u2192 D5 assertion fails with 'missing tool name [get_stock_price]'. Canonical pattern (no gate on tool-emit, toolCallId narration above for first-match-wins) lets the tool call always emit; the toolCallId narration fires on iteration 2 regardless of which tools the backend forwards."
|
|
},
|
|
"fixtures": [
|
|
{
|
|
"_comment": "custom-catchall: weather in Tokyo follow-up after get_weather tool result. MUST come before the tool-emitting fixture (first-match-wins).",
|
|
"match": {
|
|
"userMessage": "Forecast Tokyo through the wildcard renderer",
|
|
"toolCallId": "call_d5_get_weather_001",
|
|
"context": "claude-sdk-python"
|
|
},
|
|
"response": {
|
|
"content": "Tokyo is 22\u00b0C and partly cloudy \u2014 rendered through the custom wildcard catchall."
|
|
}
|
|
},
|
|
{
|
|
"_comment": "custom-catchall: weather in Tokyo first leg \u2014 emit get_weather tool call. The custom wildcard renderer fires for this tool. Matches on userMessage+context alone; the toolCallId narration above wins after the tool result arrives.",
|
|
"match": {
|
|
"userMessage": "Forecast Tokyo through the wildcard renderer",
|
|
"context": "claude-sdk-python"
|
|
},
|
|
"response": {
|
|
"toolCalls": [
|
|
{
|
|
"id": "call_d5_get_weather_001",
|
|
"name": "get_weather",
|
|
"arguments": "{\"location\":\"Tokyo\"}"
|
|
}
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"_comment": "custom-catchall: AAPL stock price follow-up after get_stock_price tool result. MUST come before the tool-emitting fixture.",
|
|
"match": {
|
|
"userMessage": "Quote AAPL through the wildcard renderer",
|
|
"toolCallId": "call_d5_get_stock_price_001",
|
|
"context": "claude-sdk-python"
|
|
},
|
|
"response": {
|
|
"content": "AAPL is trading at $338.37, down 2.96% \u2014 rendered through the custom wildcard catchall."
|
|
}
|
|
},
|
|
{
|
|
"_comment": "custom-catchall: AAPL stock price first leg \u2014 emit get_stock_price tool call. No toolName/hasToolResult gate so the tool call emits regardless of which tools the backend forwards on the second turn. The toolCallId narration above wins on iteration 2.",
|
|
"match": {
|
|
"userMessage": "Quote AAPL through the wildcard renderer",
|
|
"context": "claude-sdk-python"
|
|
},
|
|
"response": {
|
|
"toolCalls": [
|
|
{
|
|
"id": "call_d5_get_stock_price_001",
|
|
"name": "get_stock_price",
|
|
"arguments": "{\"ticker\":\"AAPL\",\"price_usd\":338.37,\"change_pct\":-2.96}"
|
|
}
|
|
]
|
|
}
|
|
}
|
|
]
|
|
}
|