Adds an optional `preFill` callback to `ConversationTurn` that runs
before the runner fills the chat input and presses Enter for each turn.
Failure semantics mirror the post-settle `assertions` callback: a thrown
error records the turn as failed and stops the conversation.
Rewires `d5-multimodal.ts` to use `preFill` to click the sample image /
PDF buttons before each turn — fixing the false-green where the
multimodal probe's transcript-keyword assertion passed without the
attachment ever being sent.
Adds unit tests covering preFill ordering, failure mode, and a
no-preFill regression guard, plus tests verifying the multimodal
script wires `preFill` to the right sample-button selectors.
Three changes that together restore D3 (e2e-readiness per-cell) emission for
half the dashboard cells after the pool sat on dead chromium instances and
the deploy webhook started 400-ing:
1. BrowserPool now detects dead browsers proactively. Once a chromium
process died (OOM, crash, network blip), the dead Browser instance
stayed in `available[]` and every probe that drew the slot failed with
"browser.newContext: Target page, context or browser has been closed"
— for hours/days, until the harness restarted or 100 release-cycles
tripped contextCount-based recycle. Adds a `disconnected` listener per
slot (registered in init() and recycleSlot()), an isConnected() check
at the top of acquire() that skips zombies and recycles them, a
release-time isConnected() check that catches the disconnect-event-
pending race, and a per-slot recyclingSlots guard preventing double-
relaunch when both paths fire concurrently. New unit tests use
fake browsers via the existing test-injection point on the constructor.
2. createPooledE2eDemosLauncher now honours the driver's abort signal,
mirroring the createPooledE2eDeepLauncher fix from ed0933e5c. Without
this, an outer-timeout kept the pooled browser held until the orphaned
driver promise drained all remaining demos — pool starvation across
ticks. Tracks open contexts so abort closes them before releasing,
uses a forceReleased flag so the driver's normal `browser.close()` in
the finally block doesn't double-release, and wires the logger
through the orchestrator registration.
3. /webhooks/deploy now accepts buildRunId / buildRunUrl. PR #4471 split
build and deploy into separate workflows and started co-sending these
fields, but `deployPayloadSchema.strict()` rejected them as unknown
keys, returning 400. Every Showcase: Verify Deploy run has 400'd
since. Adds both as optional, mirrors runUrl's http(s)-only
refinement on buildRunUrl, and propagates them onto DeployResultEvent
so downstream consumers can link a red deploy back to the build that
produced its images.
Pre-push gates: oxfmt --check clean; tsc --noEmit clean; harness vitest
suite passes the 1367 platform-portable tests (the 19 Windows-specific
pre-existing failures — path separators in test fixtures + boot-wiring
test timeouts — are present on origin/main with the same shape, untouched
by this change); tsc -p tsconfig.build.json clean.
agent-config: drop the AgentConfigLangGraphAgent subclass and use plain
LangGraphAgent. The subclass repacked CopilotKit provider properties
into forwardedProps.config.configurable.properties so the Python graph
could read them via RunnableConfig.configurable.properties — but
@ag-ui/langgraph@0.0.31 builds the LangGraph SDK request as
{ ..., config, context: { ...input.context, ...config.configurable } }
which merges configurable INTO context. LangGraph 0.6.0+ then rejects
with HTTP 400 'Cannot specify both configurable and context' on every
chat round-trip. Net effect: chat sent the user message, runtime 400'd,
no assistant response ever rendered. Removing the subclass unbreaks
the round-trip; the Python agent falls back to its DEFAULT_* constants
so the demo's frontend toggles no longer steer the system prompt
(known regression, tracked separately pending @ag-ui/langgraph fix
that decouples context from configurable).
byoc:
- D5 probe now sends the 'Sales dashboard' pill prompt (matches the
fixtures added in main:f0a89b843 in feature-parity.json) instead of
the previous generic 'render a byoc hashbrown' prompt that had no
matching JSON-shaped fixture. Removed the now-obsolete byoc.json D5
fixture file and regenerated the d5-all.json bundle (52 -> 50
fixtures).
- Added data-testid='copilot-assistant-message' + data-message-role=
'assistant' to the byoc-hashbrown and byoc-json-render renderer
wrapper divs. The CopilotChat default assistantMessage slot includes
these markers; overriding the slot with a custom JSON-rendering
component dropped them, so the e2e-deep conversation runner's
settle-detection cascade (which counts these selectors) never saw
the response and timed out at 30s. Re-attaching the markers is a
purely additive change that doesn't affect the renderers'
behavior.
- D5 byoc assertion now waits for [data-testid='metric-card'] AND a
chart (bar-chart or pie-chart) to render — a structural check on
the BYOC contract output, not a transcript-keyword check that the
custom renderer would never produce.
E2E status: 31/31 passing locally against
./bin/showcase up langgraph-python aimock with this branch's bundle.
Brings local D5 pass rate for langgraph-python from 24/31 to 29/31.
- preNavigateRoute added to chat-css, gen-ui-declarative,
gen-ui-a2ui-fixed, readonly-state-context — featureType literals
differ from registry IDs so default /demos/<featureType> 404'd.
- auth: <cpk-web-inspector> intercepts the sign-out click — added
force: true. The demo doesn't auto-refetch /info on header change,
so the post-sign-out 401 surface only appears after a probe send
— assertion now triggers one before polling.
- chat-css: inline-only page.evaluate body to avoid esbuild's __name
helper emit (undefined in the browser).
Remaining 2/31 known issues (documented in PR body for follow-up):
- byoc: agent runs but demo's hashbrown renderer expects streaming
JSON; aimock plain-text canned response leaves nothing to show.
- agent-config: runtime route never forwards to LangGraph (no
graph_id=agent_config_agent runs in the backend logs).
The gen-ui-headless D5 probe had two bugs:
1. esbuild's keepNames transform injects __name() wrappers around const
arrow functions. When those functions run inside page.evaluate() in
the browser context, __name is not defined. Replace with string-based
new Function() construction using function declarations.
2. findFirstNonTrivial and readChildCountForSelector used querySelector
which only returns the first match. On headless chat pages, the first
assistant message div is an empty wrapper (children=0), causing the
cascade to skip the selector even when a later match contains the
rendered gen-UI component. Switch to querySelectorAll and iterate all
matches.
## Summary
Adds comprehensive `console.debug` logging to the D5 probe execution
pipeline so production probe failures are diagnosable from logs alone,
without requiring reproduction.
**conversation-runner.ts** (the core multi-turn conversation engine):
- Logs conversation start/end with turn count and settle configuration
- Logs chat input selector cascade resolution -- each selector miss and
the final hit
- Logs per-turn lifecycle: message send, assistant settle wait, message
count changes, assertion execution, and failures with elapsed times
- Logs the fillAndVerifySend retry loop with attempt counts and user
message baseline vs current counts
- Logs waitForAssistantSettled polling with count changes, periodic
status during long waits (~every 5s), and timeout details
**e2e-deep.ts** (the driver that orchestrates per-feature runs):
- Logs page navigation URL and result
- Logs React hydration wait start, success, and timeout
- Logs conversation turn count after buildTurns
- Logs conversation success/failure with diagnostic capture summary
**All 9 D5 scripts** (agentic-chat, gen-ui-custom, gen-ui-headless,
hitl-approve-deny, hitl-steps, hitl-text-input, mcp-subagents,
shared-state, tool-rendering):
- Logs expected tokens/fragments and actual assistant text at each
assertion checkpoint
- Logs DOM element discovery (matched selectors, child counts, card
structure)
- Logs polling progress for long-running assertions (tool card probe,
chained reply fragments)
All logging uses `console.debug(...)` which is suppressed at INFO level
in production but available when operators set `LOG_LEVEL=debug` for
investigation.
## Test plan
- [x] All 1372 existing tests pass (`cd showcase/harness && npx vitest
run` -- 70 test files, 0 failures)
- [x] Debug output is visible in test stdout (vitest captures it
per-test)
- [x] No functional changes -- all logging is additive `console.debug`
calls
Instrument the conversation runner, e2e-deep driver, and all D5 scripts
with console.debug logging at every decision point so production probe
failures are diagnosable without reproduction.
conversation-runner.ts:
- Log conversation start/end with turn count and settle config
- Log chat input selector cascade resolution (each miss and final hit)
- Log per-turn lifecycle: message send, settle wait, count changes,
assertions, and failures with elapsed times
- Log fillAndVerifySend retry loop with attempt counts and user message
baseline/current counts
- Log waitForAssistantSettled polling with count changes, periodic
status during long waits, and timeout details
e2e-deep.ts (runFeature):
- Log page navigation URL and result
- Log React hydration wait (start, success, timeout)
- Log conversation turn count after buildTurns
- Log conversation success/failure with diagnostics
All D5 scripts (d5-agentic-chat, d5-gen-ui-custom, d5-gen-ui-headless,
d5-hitl-approve-deny, d5-hitl-steps, d5-hitl-text-input,
d5-mcp-subagents, d5-shared-state, d5-tool-rendering):
- Log expected tokens and actual assistant text at each assertion
- Log DOM element discovery (selectors matched, child counts)
- Log polling progress for long-running assertions (tool card probe,
chained reply fragments)
readLastAssistantText was reading textContent from the entire assistant
message wrapper, which includes rendered tool-component output (SVG
chart labels like "Electronics42,000Clothing28,000...") in addition to
the actual chat text. This caused gen-ui-custom probes to fail token
assertions because the extracted text was the rendered PieChart content
instead of the follow-up narration.
Scope the extraction to the prose/markdown child div inside the
canonical CopilotKit message structure, which contains only the
assistant's text response. Falls back gracefully when the expected
DOM structure isn't present (headless/custom composers).
Also adds debug logging so production traces show which element was
selected and what text was extracted.
Log pool stats and context counts when the abort signal fires and
force-releases a browser, so pool starvation events are visible in
Railway logs for diagnosis.
When the probe invoker's timeout fires, Promise.race returns the timeout
result but the driver promise keeps running with a pooled browser held.
The browser is not released back to the pool until the driver eventually
finishes (up to 5 minutes later). Since pool size (4) equals
max_concurrency (4), orphaned browsers accumulate across ticks and cause
cascading starvation -- each 15-minute probe run has fewer available
browsers until the pool is completely exhausted.
The fix wires the abort signal through to createPooledE2eDeepLauncher.
When abort fires, the launcher force-closes open browser contexts and
releases the browser back to the pool immediately. A forceReleased flag
prevents the driver's finally block from double-releasing.
Runner: inject --reporter=list,json and PLAYWRIGHT_JSON_OUTPUT_NAME
when spawning test subprocesses; read JSON file after exit for per-
test granularity; fall back to stdout then exit-code.
Baseline: restructure transformHarnessResponse to iterate
summary.services[] and produce slug-keyed results with _status
sub-keys matching collectResults format. Worst-status-wins when
same slug appears in multiple probes.
PR #4449 removed the eval tier system while unaware #4448 had just
merged. Restoring orchestrator, matrix, scope, config, GHA workflow,
and CLI wiring with all 8 CR fixes intact.
1328 test assertions across 4 test files covering scope classification,
baseline pull/load/capture, matrix aggregation/formatting/regressions,
and tiered parallel runner with concurrency control.
- scope.ts: git-diff-based scope detection classifying changes as
platform-wide, per-integration, or unrelated
- baseline.ts: harness API pull + local file load/capture for baseline
comparison with EvalBaseline type
- matrix.ts: result aggregation, coloured terminal matrix formatting,
regression detection (pass→fail), and verdict rendering. Canonical
SlugResult/TestResult types shared by runner
- runner.ts: tiered parallel test execution with concurrency control,
Playwright JSON parsing (future), and exit-code fallback
## Summary
Fixes the root causes for **30+ live-dashboard cells stuck at D2**
across 31 integration rows on https://dashboard.showcase.copilotkit.ai/.
Five surgical fixes plus one structural depth-walk fix; deploy-stale
cells (~40) are flagged separately for ops since they are not
code-fixable.
## Why these fixes ship together
A scrape of the live dashboard found 100 D2 chips. Bucketing them by
failure mode revealed the pattern is **structural + per-framework**, not
100 independent failures. Five frameworks ship genuine probe/integration
bugs we can fix in code; three frameworks are deploy-stale (Railway
image is older than the recent commit landings); one cell is a
per-feature slot-API contract bug. The dashboard itself also has a
structural bug that under-reports cells whose D5 is green when D3 has no
row yet.
## What ships
### Slot-agent fixes (Phase A/B/C of the blitz)
| Commit | What |
|---|---|
| `1de1556be` D5-green bypass for missing/red D3 in depth walk |
`deriveDepth` in
`showcase/shell-dashboard/src/components/depth-utils.ts` was
contiguous-walk-only, so 13 cells whose `d5:<slug>/<feature>` row is
green but whose `e2e:<slug>/<feature>` row is missing or red were stuck
at D2. Per the harness model, the e2e-deep driver only emits D5 after D3
+ D4 have passed at some point; D5-green is sufficient evidence. After
the D2 (agent) gate passes, `deriveDepth` now checks D5 first; if green,
achieved jumps to 5 and proceeds to D6. D1+D2 remain hard gates - D5
cannot bypass health/agent. 7 new tests pin every asymmetry. |
| `e0d89e0fe` fail loud on slug missing from registry in e2e-demos
driver | `E2eDemosResolver` previously returned `E2eDemoEntry[]`,
conflating "registry has the slug with zero demos" (legitimate brand-new
package) with "slug absent from registry" (operational fault - manifest
gap, stale mount, slug rename). Both collapsed silently into "aggregate
green / no per-feature rows" - exactly the symptom shape on the live
dashboard for several frameworks (`mastra`, `langroid`, `llamaindex`,
`langgraph-typescript`, `langgraph-fastapi`,
`ms-agent-{python,dotnet}`). Resolver now returns `{ present: boolean,
entries: E2eDemoEntry[] }`; when `present === false`, the driver emits a
synthetic red `e2e:<slug>/__missing-registry` side row + flips the
aggregate red with `errorClass: "registry-missing"`. Read-failure path
still goes through the existing green/silent branch to avoid swamping
the dashboard with N noisy red dots when the registry isn't mounted at
all. |
| `00e801209` claude-sdk-python missing openai dep crashed agent on
import | `src/agents/a2ui_dynamic.py` had a top-level `import openai`
but `openai` was never declared in `requirements.txt`. `agent_server.py`
imports `a2ui_dynamic` at module load, so `ModuleNotFoundError` cascaded
through `entrypoint.sh`'s "Agent failed to start - exiting" gate,
preventing Next.js from booting at all. Result: every `/demos/<id>`
route across the integration was unreachable - explaining all 9 red E2E
cells in the `claude-sdk-python` framework column uniformly. Two-line
fix: add `openai>=1.50.0` to `requirements.txt` and move `import openai`
to a lazy import inside `_generate_a2ui` (mirrors the existing pattern
in `agents/agent.py`). |
| `cb2ef9411` remove __future__ annotations from ag2 agent_config_agent
| Same class of bug as commit `e38fab4d6` (Apr 28) which fixed
`shared_state_read_write.py` and `subagents.py`: `from __future__ import
annotations` turns `ContextVariables` in the `@tool`-decorated
`get_current_config(context_variables: ContextVariables) -> str`
signature into a ForwardRef. Autogen's `TypeAdapter` can't resolve it,
registration raises `PydanticUserError` at module import,
`agent_server.py` fails to load, the whole AG2 integration goes
unreachable. Drop the `__future__` import. |
| `b377b78a6` render input slot in google-adk chat-slots welcome screen
| The custom welcome screen at
`src/app/demos/chat-slots/custom-welcome-screen.tsx` did not accept or
render the `input` ReactElement that `CopilotChatView` passes into the
`welcomeScreen` slot. `CopilotChatView` only mounts the chat composer
inside the welcome screen on the empty-thread path
(`CopilotChatView.tsx:301-348`), so when the empty-state welcome was
active no `<textarea>` ever reached the DOM. The e2e-readiness probe's
structural selectors all timed out, flipping the D2 chat-slots cell red.
Mirrors the canonical pattern used by 16+ other integrations (strands,
langgraph-python, claude-sdk-python). |
### cr-loop fixes (Round 1+2 review findings, applied during
convergence)
| Commit | What |
|---|---|
| `b1768f5db` return red aggregate when e2e-readiness resolver throws |
Round 1 finding (3 of 7 reviewers converged independently). When
`demosResolver(slug)` throws, the catch path was emitting a `__resolver`
red side row but `slugPresentInRegistry` defaulted to `true` and was
never flipped, so the fail-loud branch was skipped and the aggregate
fell through to green with `note: "no demos declared"`. Catch now
returns early with a red aggregate (`errorDesc: "resolver-error"`),
mirroring the missing-registry early-return shape. New C12b test pins
the aggregate state, since the existing C12 test only asserted the side
row. |
| `44037b9ce` align claude-sdk-python requirements.txt comment with
lazy-import implementation | Round 1 trivial - the comment block above
the `openai` dep claimed it was "imported at module top". The
`00e801209` fix moved the import to a lazy import; the comment was now
falsified by the diff. Updated to describe the actual two-layer
protection (lazy import as runtime safety net, requirements.txt as
authoritative dep declaration). |
| `070ecd051` empty in-band input.demos no longer suppresses
missing-registry red | Round 2 finding. The in-band fallback `if
(demos.length === 0 && Array.isArray(input.demos))` triggered even when
`input.demos === []`, forcing `slugPresentInRegistry = true` and
silently bypassing the fail-loud branch. A misconfigured probe YAML with
`demos: []` for a slug not in the registry would have rendered green.
Tightened to also require `input.demos.length > 0`. |
## Live-dashboard impact
After deploy:
- **13 cells** whose `d5:<slug>/<feature>` is already green flip to D5
immediately (purely from the depth-walk fix; no probe/agent change
required).
- **30+ cells** across `mastra` / `langroid` / `llamaindex` /
`langgraph-typescript` / `langgraph-fastapi` / `ms-agent-python` /
`ms-agent-dotnet` will surface as red `__missing-registry` rows on the
next probe tick (the previously-silent missing-registry fault becomes
loud and actionable).
- **9 cells** in `claude-sdk-python` flip green once the integration
container rebuilds with the openai dep.
- **15 cells** in `ag2` framework column should flip green once the
agent server boots cleanly (the `agent_config_agent` import crash was
preventing `agent_server.py` from loading at all).
- **1 cell** `google-adk` × `chat-slots` flips green on next deploy
(welcome screen now renders the chat composer).
## Deploy-stale cohort (NOT code-fixable, flagged for ops)
| Framework | D2 cells | Status |
|---|---|---|
| `built-in-agent` | 28 | **Needs Railway redeploy** - local `next
build` succeeds for all 53 routes; live
`https://showcase-built-in-agent-production.up.railway.app` returns HTTP
404 for every demo added in commits `e18ee1d25 e4afffd4f7abb1d07674e57483869cb5f51c7c44b71f6 53455bfeb`. Source healthy; deploy is
older than the demo-landing commits. |
| `claude-sdk-typescript` | 10 | **Needs Railway redeploy** - same
pattern: local build green, live URL returns 404 for newly-added demos
(`b9dbacd99 / 70522b1a6 / 417e3b397`). |
| `ms-agent-python` | 2 | Probable snapshot-stale - live HTML now
renders the canonical `data-testid="copilot-chat-textarea"` selector;
the readiness scan was probably running before the redeploy completed.
Should self-heal on next probe tick. |
Slack thread to Jordan with the `built-in-agent` deploy-stale evidence
already sent earlier in the run.
## Test plan
- [x] `pnpm --filter @copilotkit/showcase-harness exec vitest run
e2e-readiness` - 34/34 pass (32 pre-existing + 1 new C12b resolver-throw
aggregate-red + 1 new empty-in-band missing-registry)
- [x] `npx vitest run depth-utils` (in `showcase/shell-dashboard/`) -
30/30 pass (23 pre-existing + 7 new D5-bypass cohort)
- [x] `pnpm typecheck` on `showcase/harness` - clean
- [x] `npx tsc --noEmit` on `showcase/shell-dashboard` - touched files
(`depth-utils.ts`, `depth-utils.test.ts`) clean. Pre-existing TS errors
in `compute-tally-detail.test.tsx` are on `main`, not introduced by this
PR.
- [x] cr-loop converged on Round 3 (7 unbiased reviewers, 0 bucket (a)
findings, byte-identical verbatim prompts)
- [ ] CI green on PR HEAD
- [ ] After merge: ops triggers Railway redeploy of
`showcase-built-in-agent-production` and
`showcase-claude-sdk-typescript-production` - those 38 cells flip green
without any code change.
## Out of scope (deferred to follow-up PRs)
A 7-agent unbiased CR surfaced ~50 pre-existing concerns in files this
PR happens to touch but whose subject is distinct (bucket (d) per
cr-loop's classification rules). They have coherent, named subjects and
belong in their own PRs:
- **a2ui_dynamic.py agent observability audit** (claude-sdk-python):
broad `except Exception` leaks tracebacks to chat content; sync `openai`
call in async generator blocks the event loop; Anthropic-format messages
forwarded to OpenAI's `chat.completions.create` will 400 on the second
tool round; no max-iteration cap on the outer `while True` loop;
empty-string `ANTHROPIC_API_KEY` fallback masks misconfig;
`tool_call_id` falls back to empty string.
- **e2e-readiness probe robustness audit**: `Math.max(0, deadline -
Date.now())` returns `0` after deadline expiry, which Playwright treats
as "no timeout" instead of "fail immediately"; whitespace-only `route`
strings bypass the `config-invalid` guard; URL concatenation does not
normalize trailing/leading slashes; no abort check between `newPage()`
and `page.goto()`; `setTimeout` overflow on huge `E2E_DEMOS_TIMEOUT_MS`
env values; `__resolver` / `__missing-registry` sentinel namespacing
speculation.
- **e2e-readiness test hygiene**: the C7 "aborts mid-selector-loop
without walking all 6 selectors" test asserts behavior the
implementation no longer has (single compound selector replaced the
sequential loop); the assertion
`expect(selectorsTried.length).toBeLessThan(5)` is now trivially true.
- **depth-walk semantics audit**: D6 unreachable when `isD5Green` is
false (D6 was always gated on D5-green even before this PR; design
discussion on whether D6-green should imply D5-green - the same way
D5-green now implies D3/D4-pass).
- **agent_config_agent observability**: invalid frontend values
(`tone="snarky"`) silently coerce to defaults with no log; `logger`
instance imported but never used.
## Worktrees / cleanup
- Integration worktree at
`<repo>/.claude/worktrees/blitz-d2-to-d4-integration` retained alive
across cr-loop rounds per blitz protocol; will be removed after this PR
merges.
- 8 per-slot blitz worktrees and 2 ephemeral cr-fix worktrees were
created; 7 cleaned up successfully, 3 had Windows file-lock issues
(built-in-agent, claude-sdk-typescript, ms-agent-python) - git metadata
is gone, just stale dirs that need a manual `rm -rf`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Persist per-service probe results from the tracker snapshot into
PocketBase summary.services when a run finishes. Dashboard runs-list
rows with service data now show a chevron — clicking expands a detail
row with a service chip grid matching the inflight panel's pattern.
Existing runs (before this deploy) won't have service data and will
render without the expand affordance.
The in-band fallback condition triggered even for explicitly empty
input.demos arrays, forcing slugPresentInRegistry=true and suppressing
the fail-loud branch. A misconfigured probe with `demos: []` for a slug
absent from registry.json silently rendered aggregate green despite
the resolver reporting present:false.
Tighten the condition to require a non-empty array. Empty in-band arrays
now correctly fall through to fail-loud (or to the legitimate
"no demos declared" green branch when the slug IS in the registry).
Adds a unit test asserting empty input.demos + present:false resolver
yields aggregate red with errorDesc "registry-missing" and exactly one
synthetic e2e:<slug>/__missing-registry write.
Reported by cr-loop Round 2 confirmation slot 3.
The resolver-throw branch in createE2eDemosDriver emitted a synthetic
red `e2e:<slug>/__resolver` side row but left `slugPresentInRegistry`
at its default `true` and reset `demos = []`. On the production path
(no `input.demos` test fallback) control then fell through to the
empty-demos branch which returned an aggregate `state: "green"` with
`note: "no demos declared"`. Result: a red side row paired with a
green aggregate — alert rules pattern-matching `e2e-demos:*` red
never fired, defeating the entire fail-loud branch the catch was
added for.
Fix: in the catch block, after emitting the synthetic side row,
return early with a red aggregate carrying `errorDesc: "resolver-error"`
and the truncated message in `failureSummary`. This mirrors the
existing missing-registry early-return shape and centralizes the
resolver-throw aggregate next to its side-emit, eliminating the
fall-through entirely.
Tests: extend the existing C12 test to assert `result.state === "red"`
and `errorDesc === "resolver-error"` on the aggregate. Add a new C12b
test that exercises the production code path (no `input.demos`) — the
exact path the bug lived on — and asserts a red aggregate plus a
single synthetic side row.
Root cause: when registry.json is readable + parseable + shape-valid but
has no integration entry for the requested slug, the resolver returned
an empty array, indistinguishable from "registry knows the slug, no
demos declared yet" (the brand-new-package state). The driver collapsed
both cases into "aggregate green / no side rows", which on the live
dashboard rendered as GREEN agent:<slug> alongside MISSING
e2e:<slug>/<feature> rows for every cell of the affected service —
silent loss of signal across N dashboard cells per misconfigured slug.
This is a fail-loud violation per CopilotKit's fail-loud-discipline.
Fix: change the E2eDemosResolver return type from E2eDemoEntry[] to
{ present: boolean, entries: E2eDemoEntry[] } so the driver can
distinguish:
- present:true, entries:[..] -> normal fan-out
- present:true, entries:[] -> brand-new package (legitimately
green, "no demos declared")
- present:false, entries:[] -> slug missing from registry (NEW:
emit synthetic red side row keyed
e2e:<slug>/__missing-registry +
flip aggregate red)
When the registry can't be read at all (ENOENT, parse error, shape
error), the resolver forces present:true for every lookup so the
read-failure isn't decorated onto every service's dashboard cell —
the global failure is logged once per tick at warn and the existing
"no demos declared" green path takes over. The fail-loud branch fires
ONLY when the registry was successfully loaded but the requested slug
is absent from it.
Tests added (in showcase/harness/src/probes/drivers/e2e-readiness.test.ts):
- emits red __missing-registry side row when slug is absent from
registry — covers the synthetic side row + red aggregate +
structured warn log
- treats brand-new slug present with empty entries as legitimately
green — pins the boundary between "missing slug" and "empty demos"
- default registry resolver flags slug missing from registry.json as
not present — end-to-end exercise of the default resolver against
a fixture registry that names a different slug
- default resolver suppresses missing-registry red on global read
failure — pins the noise-suppression rule for ENOENT
- in-band input.demos suppresses missing-registry red — preserves
the test-injection escape hatch
With aimock backing all 18 integrations, there is zero LLM cost per
probe tick. Tightened schedules to catch regressions faster:
smoke: */15 → */5 min (L1/L2 HTTP health)
e2e-smoke: */30 → */15 min (L3/L4 Playwright chat round-trip)
e2e-deep: hourly → */15 min (D5 multi-turn conversations)
e2e-demos: 6-hourly → hourly (per-demo structural coverage)
Offsets preserved to avoid Chromium contention across probes.
Implements the 7 commands referenced by DEBUGGING.md:
build, ports, recreate, fixtures validate, doctor,
aimock-rebuild, diff-logs
Also adds .last-test-ts timestamp write to the test command
so diff-logs --since last-test works.
Dashboard: resolveCell() looked up D5 rows by catalog feature ID
(e.g. d5:slug/hitl-in-chat) but the harness writes them keyed by D5
feature type (e.g. d5:slug/hitl-text-input). Added CATALOG_TO_D5_KEY
mapping so the dashboard resolves the correct PB row for each cell.
CLI: buildDeepInputs() passed manifest registry IDs as `features` but
the D5 driver only maps them through demosToFeatureTypes() when they
arrive as `demos`. Changed to pass `demos` so the driver's mapping
from registry IDs to D5 feature types activates correctly.
Alem Tuzlak