Alem Tuzlak b691b63660 fix(showcase): narrow aimock fixture patterns + add drift guardrail ...

Substring-match fixtures (pie chart, bar chart, schedule, trip, etc.)
cross-fired across demos with different tool surfaces and returned tool
names the target agent never registered, causing demos to render nothing
in prod when aimock handles traffic.

Fixture changes (showcase/aimock/feature-parity.json):
- Replace generic pie-chart / bar-chart matches with per-suggestion
  specific phrases so gen-ui-tool-based gets render_pie_chart /
  render_bar_chart directly and beautiful-chat gets pieChart / barChart
  with real data (skipping the query_data two-step that caused the
  infinite loop on re-matching prompts).
- Narrow schedule+meeting to the Beautiful Chat 30-minute prompt
  returning scheduleTime.
- Narrow flight+fly to flights-from-SFO-to-JFK.
- Narrow background to sunset-themed-gradient.
- Remove trip, sales, pipeline, todo: substring-false-firing across
  unrelated demos; interrupt/A2UI demos fall through to real LLM.

Guardrail (showcase/scripts/validate-fixture-tool-surface.ts):
- Pure validate() cross-references every fixture's tool-call name
  against the tool surface of each demo whose suggestion prompt contains
  the fixture's match substring. Loud failure when the fixture returns a
  tool the demo's agent does not register.
- CLI walks packages/ collecting suggestions from page.tsx + hooks/,
  frontend tools from useComponent / useHumanInTheLoop / useFrontendTool
  / useRenderTool / useDefaultRenderTool, and backend tools via route.ts
  agentId->graphId map + langgraph.json graph->file + @tool decorators.
- 7 vitest cases written TDD-first covering the drift detection,
  content-only fixtures, case-insensitivity, and multi-tool responses.
- Current state: 33 fixtures x 191 demos, no drift. Counterfactual
  (reverting the pie-chart fix) correctly flags gen-ui-tool-based and
  declarative-gen-ui.

Also fixes a separate runtime bug in the langgraph-python package
Dockerfile: WORKDIR /app left /app owned by root; the app user could
not create the .langgraph_api cache dir LangGraph's in-memory runtime
needs, so the agent crashed on boot. Added a non-recursive chown
app:app /app (preserves the original perf intent of the explicit
--chown on COPY, which avoided a recursive chown).

2026-04-22 10:16:11 -05:00

3.1 KiB

Raw Permalink Blame History

View Raw