Commit Graph

3 Commits

Author SHA1 Message Date
Jordan Ritter b1f19bdc80 fix(showcase): rename catchall userMessage to 'forecast for Tokyo' to avoid chain pill substring collision
The previous 'weather in Tokyo' rename (388c69e68) was a substring of the
main tool-rendering chain pill prompt 'Chain a few tools in this single
turn: get the weather in Tokyo, search flights from SFO to Tokyo, and roll
a d20.' Because aimock loads fixtures alphabetically per integration dir
and uses substring match with first-match-wins, the catchall fixture
(loaded before tool-rendering.json) was intercepting chain pill matches
across 16 integrations.

Rename catchall fixture userMessage to 'forecast for Tokyo' — a phrase
not contained in any other pill prompt. Update the corresponding D5
catchall probe inputs in d5-tool-rendering-{default,custom}-catchall.ts
and the test assertions that pin those inputs.

Call-Site Enumeration: 'weather in Tokyo' remains intentionally in
page.tsx suggestions.ts pills (user-visible UX) and inside the chain
pill prompt itself — neither is in the substring-match path now.
2026-06-15 11:14:57 -07:00
Jordan Ritter 388c69e684 fix(showcase): align D6 tool-rendering catchall userMessage to D5 probe input
The D5 e2e-deep probe for tool-rendering-{default,custom}-catchall sends
"weather in Tokyo" as the test input (harness/src/probes/scripts/
d5-tool-rendering-{default,custom}-catchall.ts). The fixture userMessage
matcher uses substring match. Main's catchall fixtures had a stale
"check Tokyo weather forecast" string that could not substring-match
the probe input, causing the fixture to miss and the probe to fall
through to the live LLM — surfacing as CV x red D4 on the dashboard.

Rename to the canonical "weather in Tokyo" string across affected
integrations' tool-rendering-{default,custom}-catchall.json files.

No agent or page.tsx changes; fixture content otherwise unchanged.
2026-06-15 09:17:42 -07:00
Jordan Ritter abe69c3d56 feat(showcase-aimock): D6 fixtures for Microsoft Agent Framework integrations
Adds per-integration D6 fixtures for ms-agent-python and ms-agent-dotnet.
2026-05-26 14:02:17 -07:00