Implements the D6 reference-capture helper that drives D5 conversation
scripts against the LangGraph-Python (LGP) showcase to produce one
ParitySnapshot file per featureType under fixtures/d6-reference/.
The helper is dependency-injected end-to-end (browser launcher, SSE
interceptor, conversation runner, DOM serializer, snapshot writer) so
unit tests cover the orchestration without spinning up chromium or
touching the disk.
Per-turn SSE interceptor attach/stop lets multi-turn conversations
aggregate captures cleanly: tool calls concatenate in arrival order,
TTFT averages across non-empty-stream turns, and p50 chunk takes the
median-of-medians. Contract fields union across turns with first-seen
type winning on collision. JSON output is diff-stable: object keys
sorted, domElements sorted by (testId, tag, classes); toolCalls order
preserved (semantic).
Failure-mode contract is fail-closed: any failure (browser launch,
navigation, conversation failure_turn, DOM serialization, write)
returns failed without writing a partial snapshot. Browser is always
closed in finally.
Includes README documenting refresh cadence (weekly cron + manual
trigger) and the validation checklist for captured snapshots.
Adds a 5th "Status" tab to shell-dashboard that visualizes the probe
schedule (humanized cron, relative next/last run, duration, pass/fail
result), a "currently running" panel with progress bar and per-service
state grid, and a per-row Trigger control with "Run all" / "Run
specific..." menu.
Components live in src/components: status-tab, status-table,
status-running-panel, status-trigger-button. Local ProbeScheduleEntry
interface is a placeholder; B4b's lib/ops-api types replace it at
merge time. page.tsx wires the tab with an empty placeholder list and
no-op trigger; B4b's useProbes() hook lands in a follow-up.
19 new vitest cases cover humanized cron, relative-time tick, last-run
color tone (green/red/gray), trigger menu actions, and idle vs inflight
panel modes. Full dashboard suite green (150 tests, 1 skipped).
Per-invocation probe run history. One row per probe tick, distinct from
status/status_history (per-result state machine) — captures run-level
metadata (duration, triggered-vs-scheduled, pass/fail counts) for the
dashboard's "last N runs" widget consumed by B3 (status route) and B7
(probe-invoker hook).
PB migration creates the `probe_runs` collection with public-read /
superuser-write rules, mirroring `status` / `status_history`. Two
indexes: composite (probe_id, started_at DESC) for last-N lookups and
standalone (started_at DESC) for retention sweeps. Up/down migrations
are idempotent (no-op on re-apply, best-effort drop on rollback).
run-history.ts exports PROBE_RUNS_COLLECTION constant + ProbeRunWriter
with start/finish/recent. duration_ms is computed off the persisted
started_at (not a caller-supplied param) so the contract holds even if
clocks drift between start and finish. Filter values are JSON.stringify'd
to defend against probe ids containing PB filter metacharacters.
Tests use the same fakePb pattern as status-writer.test.ts; coverage
includes start state shape, finish state transitions (completed/failed),
duration computation off persisted row, recent() filter+sort+limit, and
empty-result handling.
Surfaces per-entry run metadata (started/finished/duration/summary), exposes
a typed getEntry snapshot, adds a manual trigger() that throws
InflightConflictError on overlap, and surfaces croner's nextRun() so the
forthcoming /api/probes routes can render schedule + last-run + next-run.
Handler return shape widened to RunSummary | void; existing void-returning
invokers keep working unchanged. ProbeRunTracker is a structural placeholder
local to scheduler.ts so this slot does not depend on B2's not-yet-existing
file.
Tests added (red-green): finished/duration populate after a scheduled run,
getEntry returns live state and undefined for unknown ids, trigger() runs
off-schedule and returns a runId, trigger() rejects when inflight,
nextRunAt returns a Date, triggeredRun toggles correctly, lastRunSummary
surfaces a handler-returned summary.
Hono middleware gating probe trigger endpoint. Reads OPS_TRIGGER_TOKEN at
construction (or accepts explicit expectedToken for tests). Fails loud if
no token configured — defaulting to "always allow" is the failure mode
this gate exists to prevent.
- 401 + JSON {error: "unauthorized"} on missing/malformed/wrong token
- Bearer scheme is case-insensitive per RFC 6750 §2.1
- Constant-time compare via crypto.timingSafeEqual; length-mismatch
returns false instead of throwing (raw timingSafeEqual leaks length)
- Empty tokens always rejected so a construction bug producing "" cannot
masquerade as a valid match
constantTimeEqual is exported for direct unit-test coverage.
Pure data class that tracks per-service progress during a probe
invocation. The probe-invoker registers services via enqueue/start
and finishes them via complete/fail; the HTTP API (GET /api/probes)
calls snapshot() to surface in-flight progress.
No I/O, no logging, no scheduler awareness. now() is injectable for
deterministic testing. Illegal transitions (e.g. complete on a
non-running service) are tolerated and overwrite state without
throwing — the invoker is the only writer, so out-of-order calls
are defensive cases, not real bugs.
Consumed at integration time by the probe-invoker (B7) and the
HTTP probes endpoint (B3).
Two D5 conversation scripts that probe gen-UI rendering against the
showcase's `headless-simple` and `gen-ui-tool-based` demos.
`d5-gen-ui-headless` mirrors the recorded fixture against
`/demos/headless-simple` (frontend `show_card` via useComponent).
Assertion: a custom-rendered component is present in the DOM (not
just text), the matched node is structurally non-trivial, and the
assistant's follow-up narration references the rendered card.
`d5-gen-ui-custom` mirrors the recorded fixture against
`/demos/gen-ui-tool-based` (frontend `render_pie_chart`). Stricter
than the headless tier: also asserts a STRUCTURAL match — an `<svg>`
with multiple `<circle>` (or `<path>`) drawing children, which is
what distinguishes "custom" from "headless".
Shared selector cascade and DOM walkers live in `_gen-ui-shared.ts`
(leading underscore so the D5 driver's `^d5-.*\.(js|ts)$` loader
skips it). Selector cascade matches CopilotKit testids first, falls
back to `[data-tool-name]`, `.copilotkit-render-component`, and
`[role="article"] svg` so showcases that don't yet wire dedicated
testids still get probed.
Tests cover registration shape, buildTurns matches the fixture's
userMessage, and assertions catch missing component / empty wrapper /
wrong SVG shape / missing narration tokens. Robustness: the custom
assertion accepts both `<circle>` and `<path>` slices so a future
showcase switching chart libraries doesn't false-fail.
Adds the D5 tool-rendering script under
showcase/ops/src/probes/scripts/d5-tool-rendering.ts. Drives the
/demos/tool-rendering page with a single 'weather in Tokyo' user
turn (matching the recorded LGP fixture verbatim) and asserts on
DOM CARD STRUCTURE rather than just response text — the heart of
D5 for this feature is verifying the per-tool renderer mounted a
structured weather card.
The assertion uses a 4-selector cascade so the script works across
the fleet without per-integration overrides:
1. [data-testid=weather-card] — LGP canonical
2. [data-tool-name=get_weather] — alternative convention
3. .copilotkit-tool-render — class-based fallback
4. [data-testid=copilot-tool-render] — generic CK testid
Fail modes carry distinct messages so triage can distinguish
framework regression (no card at all) from content regression
(card present but missing temperature/city/inner elements).
Tests cover registration, buildTurns shape, the cascade order, and
all four assertion failure paths plus two green paths (canonical
selector and fallback selector). Run via pnpm exec vitest run
src/probes/scripts/d5-tool-rendering.test.ts (14/14 pass).
One D5 script claims both `mcp-apps` and `subagents` feature types and
routes both to `/demos/subagents` via `preNavigateRoute`. The fixture
(`mcp-subagents.json`) was recorded against the LangGraph Python
supervisor agent which chains research → writing → critique sub-agents;
coupling `/demos/mcp-apps` to a public Excalidraw MCP server would have
made replay depend on an external service, so we reuse the subagents
chain for both.
Per-turn assertion scrapes the rendered reply and asserts five fragments
unique to each sub-agent's contribution (research facts, writing draft
language, critique framing). If any sub-agent had been skipped its
fragment would not have made it into the critique's final draft, so the
fixture-driven text check is sufficient signal that the entire chain
executed.
Tests cover registration under both feature types, fixture filename,
buildTurns shape (single turn matching fixture's userMessage matcher
verbatim), preNavigateRoute returning `/demos/subagents` for both feature
types, and the chained-reply assertion's pass/fail behaviour against
scripted page text.
Authors the agentic-chat probe script under
showcase/ops/src/probes/scripts/d5-agentic-chat.ts. Self-registers with
the D5 registry on import; the e2e-deep driver's dynamic loader picks
it up automatically.
Three-turn conversation mirrors the recorded fixture verbatim
(showcase/ops/fixtures/d5/agentic-chat.json) so aimock's first-match-
wins userMessage matcher routes each turn to the correct canned
response:
1. greeting / first ask — "good name for a goldfish"
2. follow-up referencing prior turn — "name for its tank"
3. context verification — "what we named the goldfish"
Per-turn assertions:
- turn 1: assistant non-empty.
- turn 2: assistant mentions a tank-name fragment (anyOf list to
survive plausible model rephrasings).
- turn 3: assistant recalls "Bubbles" from turn 1 — the actual
context-retention check (case-insensitive).
Tests cover registration-on-import, fixture/turn alignment with
cross-turn substring isolation, and turn-3's context-retention check
in both pass and fail modes.
Adds the D5 (e2e-deep multi-turn conversation) probe driver as the
runtime entry point for the D5/D6 complex-interact parity probes.
- helpers/d5-registry.ts: keyed-by-featureType registry with
registerD5Script() entry point. Single registered script may claim
multiple feature types; double-registration throws so script-file
collisions surface at boot.
- drivers/e2e-deep.ts: kind:"e2e_deep" ProbeDriver. Fans out per
declared feature type, runs runConversation() from
helpers/conversation-runner.ts on a fresh Playwright context per
feature, emits aggregate e2e-deep:<slug> + per-feature
d5:<slug>/<featureType> rows. Loads scripts dynamically from
src/probes/scripts/d5-*.{js,ts} so Wave 2b script authors can ship
in parallel without touching this file.
- config/probes/e2e-deep.yml: cadence */60, max_concurrency 2,
timeout 300_000ms per spec.
- types/index.ts: adds e2e_deep + d5 to DIMENSIONS so the loader and
rule schema accept the new kind / dimension at load time.
- Driver runs cleanly with an empty scripts/ directory: warns and
emits zero rows rather than throwing, so this lands ahead of the
Wave 2b script PRs without a chicken-and-egg problem.
Hand-authored against LangGraph Python (LGP) source as the reference
implementation for the eight D5 (complex interact) feature types:
agentic-chat, tool-rendering, shared-state, hitl-approve-deny,
hitl-text-input, gen-ui-headless, gen-ui-custom, mcp-subagents
(/demos/subagents — chosen over /demos/mcp-apps to avoid the external
Excalidraw MCP dependency).
aimock's match model is single-shot, so multi-turn behavior is expressed
as sibling fixtures: userMessage substring matches for distinct user
turns, plus toolCallId-routed fixtures for mid-loop re-invocations after
a tool result is appended. toolCallId fixtures appear above their
userMessage counterparts in each file so first-match-wins doesn't
re-route the agent loop into the original tool-call fixture and create
an infinite loop.
Per-feature-type-against-LGP-only — the same fixture is replayed across
all 17 integrations via aimock's fixture pool. Per-integration overrides
are deferred until D5 surfaces real divergences.
All eight files validate cleanly via @copilotkit/aimock loadFixtureFile +
validateFixtures (0 errors / 0 warnings, 21 fixtures total). All 20
conversation legs across the eight files replay-pass against a booted
aimock instance (chat completions short-circuit to the expected text or
tool_calls — no escape to a live LLM).
Notes are not picked up by the existing showcase aimock-fixtures.test.ts
(its globs do not include showcase/ops/fixtures/d5); extending those
globs is left for the PR that lands the e2e-deep driver.
Captures the three orthogonal axes the parity-vs-reference probe needs
from the runtime SSE stream: ordered tool-call names, stream timing
profile (TTFT, inter-chunk gaps, p50, total chunks, duration), and JSON
contract shape (field path to JS-native type).
Uses CDP Network.dataReceived for per-wire-chunk timestamps without
intercepting the response, then retrieves the full payload via
Network.getResponseBody once loadingFinished fires. Playwright route
APIs cannot stream cleanly — route.fulfill takes a complete body and
route.fetch buffers upstream, both of which collapse chunk-arrival
timing to a single point. CDP is the only path that preserves real
per-chunk arrival timestamps. The response itself flows naturally to
the page; we observe, we do not intercept.
Parser tolerates SSE multi-line data records, comments, CRLF endings,
and malformed half-line chunks (returns a non-json record rather than
throwing). Tool-call extraction walks the four likely field shapes
(toolCallName, name, tool_call.name, tool_use.name) so the parity
engine can compare against reference implementations that have not
migrated to ag-ui canonical naming.
28 unit tests cover the parser, contract walker, timing computation,
and capture assembly using captured raw SSE samples — no live browser.
Adds `runConversation(page, turns, opts)` — a multi-turn Playwright
helper that drives a CopilotKit chat surface through a sequence of
assertions. Returns a structured `ConversationResult` describing how far
the conversation got, where it failed, and per-turn wall-clock cost.
Behaviour:
- Resolves the chat input via the same 5-selector cascade used in
`e2e-demos.ts` (canonical testid → default placeholder → generic
text-input fallbacks). Cascade is short-circuited when the caller
supplies `chatInputSelector`.
- Detects "response complete" by polling the assistant-message DOM
count via `page.evaluate` and waiting for `assistantSettleMs`
(default 1500 ms) of no growth past the previous-turn baseline.
Uses `evaluate` rather than `textContent(selector)` to avoid
Playwright's auto-wait machinery defeating the poll cadence —
same rationale documented in `e2e-smoke.ts`.
- On any failure (selector miss, fill/press throw, response timeout,
assertion throw) records `failure_turn` (1-indexed) + `error` and
returns immediately — subsequent turns are NOT executed.
Tests inject scripted fakes for the Page surface so no real browser is
required. Coverage on the helper is 94% lines / 97% branches; the
uncovered tail is the browser-context `evaluate` callback that can't be
exercised without chromium (matches the pattern in `e2e-smoke.ts`).
The `Page` type is a structural minimal interface (NOT the playwright
`Page`) so consumers don't have to acquire DOM lib types just to call
the helper. Real `playwright.Page` satisfies it structurally.
Pure-logic comparator for the four D6 parity axes (DOM, tools, stream,
contract). Takes a reference + captured ParitySnapshot plus optional
tolerances and returns a per-axis verdict, aggregate verdict, and
structured details for the dashboard / Slack writer to render.
No I/O — Playwright, fetch, and capture logic live elsewhere; this
module is the canonical scoring implementation shared between the
reference-capture script and the live D6 driver. Default stream
tolerances (TTFT 2x, P50 chunk 3x) are pulled verbatim from the spec
and flagged in code as needing empirical calibration post-merge.
Includes 27 unit tests covering pass/fail/edge cases for each axis
(testId-vs-class fallback, double-claim prevention, ratio overrides,
zero-divisor + NaN guards, mixed-axis aggregate counting) and three
JSON fixtures under test/fixtures/parity-snapshots/ exercising the
all-pass and all-fail end-to-end paths.
## Summary
The e2e-demos probe writes PocketBase rows as `e2e:<slug>/<featureId>`
but the dashboard looked up `e2e_smoke:<slug>/<featureId>`. Keys never
matched, so per-demo depth was always null and cells were stuck at D2.
One-line fix in `resolveCell` (change dimension from `e2e_smoke` to
`e2e`), plus corresponding updates to all tests and the
cell-pieces/feature-grid components that referenced the old dimension
name.
## Test plan
- [x] All 130 unit tests pass (`npm test` in shell-dashboard)
- [ ] After deploy, dashboard cells show D3/D4 for demos the probe has
already visited
## Summary
Replace the hardcoded 200-line INTEGRATIONS array in
`integration-smoke.spec.ts` with a 6-line derivation from
`registry.json`. New demos automatically appear in smoke tests when
manifests are updated.
- Delete hardcoded Integration interface + 17-entry array
- Derive slug, name, backendUrl, deployed, hasToolRendering, demos from
registry
- Add unit tests verifying derivation correctness + regression guard
against re-hardcoding
## Test plan
- [x] New vitest tests pass (7/7)
- [x] Existing showcase build pipeline tests pass (1183/1183 across 26
files)
- [ ] CI green
The e2e-demos probe writes per-demo rows keyed as e2e:<slug>/<featureId>
but resolveCell looked up e2e_smoke:<slug>/<featureId>. The mismatch
caused every per-demo depth chip to show gray (no data), leaving cells
stuck at D2. Fix: read the e2e dimension the probe actually writes.
All 15 framework packages now have demo routes wired after the omnibus
parity merge. Update comments to reflect full-fleet scope — namePrefix
and max_concurrency were already set correctly in the config values.
Add missing feature IDs (hitl, hitl-in-chat-booking) to langgraph-python
manifest so it reclaims reference status from langgraph-fastapi. Update
catalog test expectations and validate-pins fail-baseline.json hash/count
after PR #4287 dependency changes.
Packages that gained beautiful-chat or byoc-json-render demos via the
omnibus parity merge were missing direct dependencies that work locally
(hoisted from root node_modules) but fail in Docker's isolated npm
install.
Added:
- @radix-ui/react-checkbox to crewai-crews, langgraph-fastapi,
ms-agent-dotnet, ms-agent-python
- @json-render/core + @json-render/react to ms-agent-python, strands
- @copilotkit/voice + openai to ms-agent-python
Replace the 200-line hardcoded INTEGRATIONS array in integration-smoke.spec.ts
with a 6-line derivation from registry.json. New demos automatically appear in
smoke tests when manifests are updated — no manual maintenance needed.
Jordan Ritter