Jordan Ritter 9491b89340 fix(showcase): disjoint catchall userMessages + content-asserting probes (supersedes #5465) ...

Closed PR #5465 introduced cross-fixture leakage by stripping the
toolName discriminator on shared {userMessage, context} keys: with
aimock's alphabetical first-match-wins ordering,
tool-rendering-custom-catchall.json sorts before
tool-rendering-default-catchall.json, so default-catchall page requests
were served the custom file's content. The probe was structurally blind
because it asserted only DOM testids (copilot-tool-render +
data-tool-name=get_weather) — both fixtures emit get_weather, so the
testid signal passed regardless of which fixture won.

Real LGP-gold pattern is disjoint userMessages between default and
custom catchall fixtures (NOT shared keys discriminated by toolName).
This PR ports the LGP-gold pattern to the other 17 integrations and
adds page-text content assertions to both d5 catchall probes so this
class of regression can't recur silently:

- Probe prompts disjoint between default-catchall and custom-catchall
- Fixture userMessages updated to match the new disjoint prompts
- Page-text content assertions in both d5 catchall probes
  (default negatively asserts the custom-catchall leak phrase;
   custom positively asserts it)

Local gold-standard red-green proof captured:
- Step A: live staging Playwright baseline (RED-OF-RECORD)
- Step B: local control-plane on origin/main reproduces structural
  fragility (probes pass at testid level, custom fixture wins on
  default's userMessage path)
- Step C: local control-plane on this branch — both catchall probes
  GREEN with the new content-asserting assertions
    tool-rendering-default-catchall: pass=true (5443ms)
    tool-rendering-custom-catchall:  pass=true (8956ms)
                                     cross-tool signature passed

LGP (langgraph-python) was already disjoint; it remains untouched.

2026-06-16 09:37:51 -07:00

3.1 KiB

Raw Permalink Blame History

View Raw