Root cause (A11 investigation, PR #5495): the probe passed the canonical
phrase into the browser-side `page.evaluate` closure via the second-arg
form (`evaluate((expectedPhrase?: string) => …, phrase)`). Empirically
the arg arrives as `undefined` inside the closure — verified via
in-closure return diagnostics showing `needleLen === 0` while
`bubble.textContent` and `body.textContent` BOTH contained the canonical
phrase at the SAME poll moment. With `needle === """, the
`if (needle)` guard skipped the entire cascade, leaving
`customContentPhrasePresent` false forever and failing every
integration's `tool-rendering-custom-catchall` cell even when the
narration was reaching the DOM correctly (LGP gold).
Fix: inline the canonical phrase as a JS literal inside the closure —
no `page.evaluate(fn, arg)` second-arg dependency. Keep the existing
per-tier scoped cascade and `body.textContent` fallback semantics
unchanged. The shared constant `CUSTOM_CATCHALL_CONTENT_PHRASE` and
the inlined literal MUST stay in lock-step (existing
`exports the testid contract for cross-test reuse` unit test verifies
the constant; manual sync required on edit of the literal).
RED-GREEN proof:
RED (pre-fix, /tmp/cr/a11-red-lgp.log INNER_EXIT=1):
LGP `d5-single-pill-e2e:langgraph-python red`
settled text turn 2 = 'AAPL is trading at $338.37, down 2.96% —
rendered through the custom wildcard catchall renderer.' (phrase
IS in bubble.textContent) but
snap.customContentPhrasePresent: false ← THE BUG
GREEN (post-fix, /tmp/cr/a11-final-lgp.log INNER_EXIT=0):
LGP `d5-single-pill-e2e:langgraph-python green (8.9s) → 1 passed`
BIA remains red (separate fixture-routing concern — BIA's narration is
not reaching the DOM at all per `settled text` = pure tool-card text,
`assistantMsgCount: 2`, no canonical-phrase bubble; outside this
fix's scope per the worktree's modify-harness-only constraint).
(cherry picked from commit a980be48c514a2ab4972395e80e70e5b19247131)
Other dimension drivers under showcase/harness/src/probes/drivers/ follow
the dN-<purpose>.ts naming pattern (d4-chat-roundtrip.ts, d6-all-pills.ts).
D2 and D3 didn't — bring them in line.
drivers/liveness.ts -> drivers/d2-liveness.ts
drivers/liveness.test.ts -> drivers/d2-liveness.test.ts
drivers/e2e-readiness.ts -> drivers/d3-readiness.ts
drivers/e2e-readiness.test.ts -> drivers/d3-readiness.test.ts
The d3 driver loses the redundant 'e2e' prefix — the dimension is D3
and the corresponding dashboard badge is 'UI', so the file's job is best
described as 'is the demo frontend ready?'.
Imports updated in:
- showcase/harness/src/orchestrator.ts
- showcase/harness/src/orchestrator.test.ts
- showcase/harness/src/cli/runner.ts
- showcase/harness/src/probes/drivers/d2-liveness.test.ts (self-ref)
- showcase/harness/src/probes/drivers/d3-readiness.test.ts (self-ref)
Doc-comment references to the old file paths updated in:
- showcase/harness/src/cli/targets.ts
- showcase/harness/src/probes/liveness.ts (the helper, not the driver)
- showcase/harness/src/probes/loader/probe-invoker.ts
- showcase/harness/src/probes/discovery/railway-services.ts
- showcase/harness/src/probes/drivers/starter-smoke.ts
- showcase/harness/src/probes/drivers/starter-smoke.test.ts
- showcase/harness/src/types/dimensions.test.ts
- showcase/harness/config/probes/e2e-demos.yml
- showcase/harness/README.md (driver table)
Function/symbol exports unchanged (livenessDriver, e2eReadinessDriver,
createE2eDemosDriver, etc.) — file rename only. The parent helper module
showcase/harness/src/probes/liveness.ts is also untouched: it's a
separate, non-driver helper that exports deriveHealthUrl, livenessProbe,
and LIVENESS_SLACK_SAFE_FIELDS. Not renamed: drivers/e2e-parity.ts,
drivers/aimock-wiring.ts, drivers/pin-drift.ts, drivers/image-drift.ts
(non-dimensional probes).
validateCustomCatchall's requireContentPhrase branch was unreachable: assertCustomCatchall defaulted it to false and no caller overrode. The whole point of the PR is to catch cross-fixture leakage via content assertion — wiring it on. Turn-2 (Quote AAPL) now asserts the custom content phrase is present in the rendered bubbles, not just the testid.
(cherry picked from commit c9620c96f9addfee18f87fb1f8b2ae7fb040b3b8)
Probes registered fixtureFile: 'tool-rendering.json' but actual files are tool-rendering-{default,custom}-catchall.json. Field is signal-only (aimock loads directory-wide content-driven matching) but accurate metadata helps debugging. Test files updated to enforce the correct values.
(cherry picked from commit 3c9de26375546f2269870b563ecd1526ec5a3bc2)
Closed PR #5465 introduced cross-fixture leakage by stripping the
toolName discriminator on shared {userMessage, context} keys: with
aimock's alphabetical first-match-wins ordering,
tool-rendering-custom-catchall.json sorts before
tool-rendering-default-catchall.json, so default-catchall page requests
were served the custom file's content. The probe was structurally blind
because it asserted only DOM testids (copilot-tool-render +
data-tool-name=get_weather) — both fixtures emit get_weather, so the
testid signal passed regardless of which fixture won.
Real LGP-gold pattern is disjoint userMessages between default and
custom catchall fixtures (NOT shared keys discriminated by toolName).
This PR ports the LGP-gold pattern to the other 17 integrations and
adds page-text content assertions to both d5 catchall probes so this
class of regression can't recur silently:
- Probe prompts disjoint between default-catchall and custom-catchall
- Fixture userMessages updated to match the new disjoint prompts
- Page-text content assertions in both d5 catchall probes
(default negatively asserts the custom-catchall leak phrase;
custom positively asserts it)
Local gold-standard red-green proof captured:
- Step A: live staging Playwright baseline (RED-OF-RECORD)
- Step B: local control-plane on origin/main reproduces structural
fragility (probes pass at testid level, custom fixture wins on
default's userMessage path)
- Step C: local control-plane on this branch — both catchall probes
GREEN with the new content-asserting assertions
tool-rendering-default-catchall: pass=true (5443ms)
tool-rendering-custom-catchall: pass=true (8956ms)
cross-tool signature passed
LGP (langgraph-python) was already disjoint; it remains untouched.
## Summary
The smoke probe was the same HTTP contract as `/health` on the same
service (200-OK JSON body), so every tick paid two HTTP calls for the
same liveness signal. This PR:
- Drops the `/smoke` GET probe and the `smoke:<slug>` primary
`ProbeResult`. The driver now emits `health:<slug>` as the primary and
`agent:<slug>` via writer side-emit (half the per-tick cost).
- Renames the dashboard's D3 / e2e row label from `E2E (Demo)` to `UI
(Frontend)` (and the short cell badge from `E2E` to `UI`) — the row was
already about "the demo page renders in a browser", not E2E in the
traditional sense.
- Drops the `Smoke` row from the cell drilldown (CellState.smoke field
is retained for back-compat so historical rows still parse).
- Adds an explicit `Health` row to the legend (was implicit).
The driver's registry `kind` stays `"smoke"` so existing YAML configs +
orchestrator family wiring keep routing to this driver — the emission
key (`health:<slug>`) is the taxonomy contract that matters. The
underlying probe key (`e2e:<slug>/<feature>`) is preserved on PocketBase
so historical rows render correctly during the rename window.
## RED proof (BEFORE)
`smoke:<slug>` primary emission in `liveness.ts` (origin/main):
```
18: * 1. `smoke:<slug>` — the RETURN VALUE of `run()`. The invoker runs
56: * `smoke:<slug>`/`health:<slug>`/`agent:<slug>` triple the dashboard
241: input.mode === "discovery" ? `smoke:${slug}` : input.key;
```
`E2E (Demo)` / `Smoke` strings in dashboard render code (origin/main):
```
cell-drilldown.tsx:38: * Trip)" (chat+tools round-trip), D3/e2e = "E2E (Demo)" (the demo page loads
cell-drilldown.tsx:49: { key: "e2e", label: "E2E (Demo)" },
cell-drilldown.tsx:52: { key: "smoke", label: "Smoke" },
cell-pieces.tsx:382: name="E2E"
unified-cell.tsx:231: <TestBadge name="E2E" level={model.d3} />
adaptive-legend.tsx:59: E2E (Demo): demo page loads and round-trips in a browser
```
## GREEN proof (AFTER)
Primary emission rewritten — driver returns `health:<slug>` and no
`smoke:<slug>` ProbeResult is ever emitted:
```
33: * to `/smoke` and emitted a `smoke:<slug>` primary result, but that probe
230: // `kind: "smoke"` is the registry/family identifier the orchestrator
236: kind: "smoke", // registry-key back-compat; no smoke ProbeResult is emitted
367: if (input.mode === "discovery") return `health:${slug}`;
368: if (input.key.startsWith("smoke:")) {
369: return `health:${input.key.slice("smoke:".length)}`;
```
New labels in dashboard:
```
cell-drilldown.tsx:60: { key: "e2e", label: "UI (Frontend)" },
cell-pieces.tsx:382: name="UI"
unified-cell.tsx:231: <TestBadge name="UI" level={model.d3} />
adaptive-legend.tsx:65: UI (Frontend): demo page renders in browser (Playwright)
```
Confirmation that the old labels are gone from render code:
```
$ grep -rn 'E2E (Demo)\|"Smoke"' cell-drilldown.tsx cell-pieces.tsx unified-cell.tsx adaptive-legend.tsx
(no matches — labels removed)
```
## Test results
- `pnpm exec vitest run` in `showcase/shell-dashboard`: **63 files, 1089
tests passed, 1 skipped**
- `pnpm exec vitest run` in `showcase/harness`: **129 files, 2760 tests
passed**
- `pnpm exec tsc --noEmit` in both: clean
Liveness driver tests include a regression guard (`never emits a
smoke:<slug> key`) so the smoke contract cannot be re-introduced
silently.
## Test plan
- [x] Dashboard vitest green (1089/1089)
- [x] Harness vitest green (2760/2760)
- [x] tsc --noEmit green in both packages
- [x] Visual cross-check: drilldown renders 6 rows (no Smoke), e2e row
labelled `UI (Frontend)`; cell strip shows `UI` badge in place of `E2E`;
legend shows explicit `Health` row + `UI (Frontend)` D3 line
The smoke probe was the same HTTP contract as /health on the same
service (200-OK JSON body), so every tick paid two HTTP calls for the
same liveness signal. Drop the /smoke GET + the smoke:<slug> primary
ProbeResult; the driver now emits health:<slug> as the primary and
agent:<slug> via writer side-emit (half the per-tick cost).
The driver's registry kind stays "smoke" so existing YAML configs and
orchestrator family wiring keep routing to this driver — the
emission key (health:<slug>) is the taxonomy contract that matters.
Dashboard:
- Drilldown: D3/e2e row labelled "UI (Frontend)" (was "E2E (Demo)");
Smoke row dropped (CellState.smoke field retained for back-compat).
- Cell badges: short label "UI" (was "E2E") in cell-pieces + unified-cell.
- Legend: D3 = "UI (Frontend): demo page renders in browser (Playwright)"
plus an explicit Health row at the top.
Tests updated to assert new labels:
- cell-drilldown.test.tsx: 6 dimensions (no Smoke); UI (Frontend) label.
- cell-drilldown.lazy-signal.test.tsx: drilldown-badge-ui--frontend- testid.
- cell-pieces.test.tsx + .signal-degrade.test.tsx: badge name "UI".
- unified-cell.test.tsx: mock-badge-UI.
- overlay-selector-integration.test.tsx: "UI" in place of "E2E".
- dashboard-color-matrix.test.tsx: badge: "UI" case names.
- liveness.test.ts: two-call contract (health + agent), regression guard
asserting no smoke:<slug> ever emitted.
Underlying probe key (e2e:<slug>/<feature>) preserved on PocketBase so
historical rows render correctly during the rename window.
PR #5462's defect-2 fix assumed 1 bubble per turn (bubbleIndex =
turnIndex - 1). Multi-step agents (LangGraph, Mastra, CrewAI) emit
2-3+ bubbles per turn — tool-call + tool-render + final-text — so
the strict-index read lands on an intermediate tool-call bubble
whose scoped-text selectors (`.cpk:prose`, `[data-message-content]`,
`p`) are EMPTY (its content lives in a sibling card div outside the
scoped cascade). The cascade-pollution guard returns null forever
and the text-stable conjunct times out with `reason=text-unstable`
across virtually every multi-step demo (beautiful-chat-*, gen-ui-*,
shared-state-*, agentic-chat on LangGraph/Mastra/CrewAI/etc.).
Live DOM evidence from staging (langgraph-typescript :
beautiful-chat : pie chart prompt):
count = 3 bubbles
bubble[0]: tool-call "query_data" — `.cpk:prose` empty
bubble[1]: pie chart card — `.cpk:prose` empty
bubble[2]: final text "Here is the pie chart..." — 90 chars
Fix: read the LAST bubble in the matched cascade tier (count - 1)
instead of a fixed strict index. Defect-2 protection (don't read a
leftover bubble from a prior turn) is preserved via an explicit
pre-submit `baselineCount` sentinel snapshot in the runner; the
gate now requires `count > baselineCount` rather than
`count > turnIndex - 1`, which correctly rejects stale bubbles
without assuming 1 bubble = 1 turn.
Changes:
- assistant-message-count.ts: add `readCascadeStateLast(page)` —
same atomic single-evaluate contract as `readCascadeState` but
resolves the bubble index internally as `count - 1`.
- conversation-runner.ts: snapshot `baselineCount` before
sendTurnMessage; pass through to `waitForTurnComplete` via new
`baselineCount` option (defaults to `turnIndex - 1` for unit-
test fakes); `waitForTurnComplete` reads via
`readCascadeStateLast` and gates on `count > baselineCount`.
Cold-start retry re-snapshots `baselineCount` after page.reload.
Test plan:
- All 72 conversation-runner + assistant-message-count unit tests
pass (default baselineCount preserves count-progression scripts).
- Full harness suite 2758/2758 green, tsc clean, oxfmt/oxlint
clean on changed files.
- d5-* unit tests: update page fakes / assertion-call shapes to pass
the ctx bridge ({bubbleIndex, text}) instead of querying the live
bubble list
- d6-all-pills.test.ts: update page-fake dispatch + cascade-state
expectations to match the atomic readCascadeState return shape
({count, text} or null) — picks up the cross-tier consistency
guarantee in the unit layer
- d6-all-pills.ts: wire installPrePaintFromEnv and
attachSseInterceptor PRE-goto in both defaultLauncher and pooled
launcher paths, so first-paint state is deterministic and the SSE
counter is armed before any page navigation
- captureDiagnostics rewired through the Node-side helper
(readCascadeState) — single page.evaluate per call, no
within-snapshot tier drift
- Consume BUBBLE_RACE_MESSAGES override for defect-4 repro flows
- d5-gen-ui-custom, d5-gen-ui-open-advanced, d5-mcp-apps,
d5-subagents, d5-tool-rendering-default-catchall: wrap assertion
bodies in ctx bridge so they receive {bubbleIndex, text} for the
turn under test, replacing turn-leaky list[last] retrieval
- _gen-ui-shared: add readAssistantTextAt(page, bubbleIndex)
adapter; remove readLastAssistantText (no longer turn-scoped)
This is the read-side counterpart to the atomic readCascadeState
on the helper side: probes ask for the bubble at the runner's
chosen index instead of guessing.
R3-F1 (OPS_TRIGGER_TOKEN hoist):
Extract loadOpsTriggerToken() and hoist its invocation to the top of both
boot() and runControlPlane(), alongside loadWebhookSecrets() and
loadPocketbaseUrl(). Pre-fix the empty/whitespace check fired AFTER pb /
bus / scheduler / writer / S3 uploader allocations, so a typo'd
`OPS_TRIGGER_TOKEN=` allocated expensive resources before throwing. Now
all three fail-loud config predicates fire at the top, before any
allocation needing teardown. Behaviour is unchanged for valid tokens
(trimmed via R3-A.5 contract) and for the unset case (router omitted with
info log).
R3-F2 (sync-throw also emits deploy.writer.failed):
subscribeDeployResults() pre-fix only emitted `deploy.writer.failed` on
writer.write() promise rejection — a synchronous throw inside
deployEventToProbeResult() (malformed event, type drift) bypassed the
catch and we lost both the log AND the bus emit. Wrap the sync mapping
in try/catch and mirror the rejection path so alert rules / metrics
subscribers observe sync throws the same way they observe async write
failures.
Tests:
- 5 unit tests for loadOpsTriggerToken: undefined / empty-string fail-loud
/ whitespace-only fail-loud / R3-A.5 trim contract / verbatim value.
- 1 test for the sync-throw path: vi.doMock deployEventToProbeResult to
throw, assert bus emits deploy.writer.failed with the err message AND
writer.write was NOT called. Red-green verified locally (test fails
without the try/catch).
Suite: 2731 passing (+6 new). Typecheck clean.
CB-1 (Slot 2 #22): convert B2 deploy-webhook tests to vi.stubEnv with
`vi.unstubAllEnvs()` in afterEach instead of manual process.env mutation
+ try/finally restoration. Test runner now guarantees restoration and the
test bodies are shorter / less error-prone (CR Slot 2 #22).
CB-2 (Slot 2 #28): when `loadWebhookSecrets`' escape hatch fires with a
real-looking NODE_ENV (anything except "test"), log at `warn` instead of
`info` so a production typo (NODE_ENV=staging + HARNESS_ALLOW_NO_SECRET=1)
is visible in dashboards / log alerting. Pure local-dev (NODE_ENV=test)
stays at info level so a normal unit-test boot doesn't spam warnings.
CB-3 (Slot 4 #17): clarify `loadWebhookSecrets`' docstring + bypass-log
message — "set" is ambiguous (empty string would qualify pre-fix); use
"non-empty" to match the actual predicate.
R1-F2 (bucket b, defensive ordering): hoist fail-loud config validation
to the TOP of boot() and runControlPlane() — BEFORE any pb client, queue,
bus, scheduler, writer, S3 uploader, fleet-health, or aggregator
allocations. Pre-fix `loadWebhookSecrets` lived AFTER the entire
scheduler+writer (and CP queue+aggregator) setup, so a misconfigured boot
allocated expensive resources and mounted file watchers before throwing.
R1-F3 (bucket b, predicate symmetry): broaden the POCKETBASE_URL fail-loud
predicate to match SHARED_SECRET semantics. Pre-fix the POCKETBASE_URL
guard only fired on `NODE_ENV === "production"`, while `loadWebhookSecrets`
fired unless NODE_ENV=test or an explicit escape hatch — so staging /
unset / "development" deploys silently bound to http://localhost:8090.
Extracted `loadPocketbaseUrl(logger)` with the same test-or-escape-hatch
predicate as `loadWebhookSecrets`. A new `HARNESS_ALLOW_NO_PB_URL=1` env
flag mirrors `HARNESS_ALLOW_NO_SECRET=1` for local dev. Worker
`boot()` and the CP's `resolveFleetPbConfig` both call the helper.
Tests:
- HF13-A2 production-only assertion broadened (SHARED_SECRET set so the
test specifically exercises the POCKETBASE_URL guard).
- New R1-F3 tests cover NODE_ENV=development without POCKETBASE_URL
(throws) and the HARNESS_ALLOW_NO_PB_URL=1 escape hatch (succeeds).
R1-F1 (bucket a, load-bearing): the CP boot path now subscribes to
`deploy.result` events through its status writer. Pre-fix, B2 mounted POST
/webhooks/deploy on the CP role but only the worker boot path had a
`bus.on("deploy.result", ...)` listener — signed POSTs to the CP host
returned 202, the bus event fired with no subscriber, and the deploy-overall
dashboard row never landed.
Extracted the handler into a shared `subscribeDeployResults(bus, writer)`
helper exported from orchestrator.ts so both boot paths share the IDENTICAL
logic. Worker boot keeps its inline pattern (busUnsubs.push) so teardown is
unchanged; CP keeps the subscription alive for the lifetime of the bus.
Red-green tests in src/orchestrator.test.ts under "R1-F1" assert that
emitting deploy.result on the returned handle's bus drives one write keyed
"deploy:overall" through a stubbed status writer — once for runControlPlane
(the new path) and once for boot() (regression guard for the helper
extraction).
The 'Showcase: Verify Deploy' workflow's notify-harness step has been
POSTing to /webhooks/deploy and getting 404 on every main deploy since
at least 2026-06-12 (5+ consecutive failures). Root cause:
1. POST /webhooks/deploy is only registered when
webhookSecrets.length > 0 (gate at src/http/server.ts:119).
2. The CP buildServer call in runControlPlane omitted BOTH
webhookSecrets and metrics — so the public Railway host running
the CP role never mounted the route. The worker boot path had
them, but it isn't the host receiving notify-harness POSTs.
3. The FATAL-CONFIG guard on missing SHARED_SECRET was gated on
NODE_ENV === 'production' — any deploy with NODE_ENV unset or set
to something other than the literal 'production' silently shipped
with webhookSecrets=[] and no fatal error.
Part 1 — wire webhookSecrets + metrics into the CP buildServer call.
Lifted the env→secrets loader into a new exported helper
'loadWebhookSecrets()' (~orchestrator.ts:175-209) so BOTH boot paths
consume the same predicate. The worker 'boot()' call (~line 836) and
the CP 'runControlPlane' call (~line 2421) both invoke it, and the CP
buildServer call (~line 3047) now passes webhookSecrets+metrics
alongside the existing bus/probes/fleetRuns wiring.
Part 2 — tighten the FATAL-CONFIG predicate.
The new predicate throws unless EITHER a secret is set OR NODE_ENV ===
'test' OR HARNESS_ALLOW_NO_SECRET === '1' (narrow escape hatch for
local dev). The thrown error message names the env-vars, the gate
location ('src/http/server.ts:119'), the silent-404 symptom, and the
escape hatches, so an operator reading the deploy log can recover
without spelunking.
Updated the existing R5-G4 D5 regex (orchestrator.test.ts:1592) since
the error message changed; covered by the new B2 Tests A/C/D below.
Tests: 4 new red-green tests in 'B2: /webhooks/deploy registered on CP
+ fail-loud on missing SHARED_SECRET':
- Test A (CP /webhooks/deploy registration): RED 404 → GREEN 401.
- Test B (worker /webhooks/deploy regression): GREEN 401.
- Test C (FATAL-CONFIG when NODE_ENV=development): RED resolves →
GREEN rejects with /FATAL-CONFIG.*SHARED_SECRET/.
- Test D (NODE_ENV=test escape hatch): GREEN, no throw.
Full harness suite: 2719/2719 green. typecheck: clean.
Requires a CP-role redeploy on the public Railway host for the new
route registration to take effect.
Adds four red-green tests for the notify-harness 404 follow-up:
A. CP boot registers POST /webhooks/deploy when SHARED_SECRET is set
— RED pre-fix (404, route not mounted on CP); GREEN post-fix (401,
the route's own HMAC reject path fires on an unsigned POST).
B. Worker boot still registers POST /webhooks/deploy (regression
guard) — already green; pins the unchanged behavior.
C. boot throws FATAL-CONFIG when SHARED_SECRET unset AND NODE_ENV is
non-test — RED pre-fix (current guard fires only on
NODE_ENV='production'); GREEN post-fix once the predicate is
tightened.
D. boot succeeds when SHARED_SECRET unset AND NODE_ENV='test'
(test-mode escape hatch) — already green; pins the escape hatch.
Red capture (pre-fix run):
- Test A: AssertionError: expected 404 to be 401
- Test C: AssertionError: promise resolved instead of rejecting
## Summary
- D5 e2e-deep probes for `tool-rendering-{default,custom}-catchall` send
`"forecast for Tokyo"` as the test input (see
`showcase/harness/src/probes/scripts/d5-tool-rendering-{default,custom}-catchall.ts`).
- aimock uses substring match on `userMessage`.
- The catchall fixtures on main had a stale `"check Tokyo weather
forecast"` string that couldn't substring-match the probe input →
fixture miss → probe falls through to live LLM → CV ✗ red D4 on the
dashboard.
- This PR renames the userMessage to the canonical `"forecast for
Tokyo"` across 28 catchall fixture files in 16 integrations.
## Scope
28 files × ~2 userMessage occurrences each = 67 line changes. **No code,
no agent, no page.tsx changes.** Pure fixture-data alignment.
Integrations covered (default-catchall and/or custom-catchall): ag2,
agno, built-in-agent, claude-sdk-python, claude-sdk-typescript,
crewai-crews, google-adk, langgraph-python, langroid, llamaindex,
mastra, ms-agent-dotnet, ms-agent-python, pydantic-ai, spring-ai,
strands.
## Commit history note
Commit b1f19bdc8 changes the rename target from the original `"weather
in Tokyo"` (commit 388c69e68) to `"forecast for Tokyo"`. This was a CR
Round 1 catch: `"weather in Tokyo"` was a substring of the chain pill
prompt (`"weather forecast chain in Tokyo"` / similar), which would have
caused the catchall fixture to incorrectly match chain-pill traffic.
`"forecast for Tokyo"` has no such substring collision with any other
probe input.
## Verification
- **Live red-green proof** was performed on the prior `"weather in
Tokyo"` rename (commit 388c69e68) against `ms-agent-python` via
`showcase/bin/showcase test ms-agent-python --d6 --isolate`:
- Before (origin/main): catchall featureTypes red (fixture miss → live
LLM → flaky)
- After: `d6:ms-agent-python/tool-rendering-default-catchall=green`,
`d6:ms-agent-python/tool-rendering-custom-catchall=green`
- **The current HEAD's `"forecast for Tokyo"` rename (b1f19bdc8) has NOT
been re-run live.** It is verified by static analysis only: substring
math (no collision with any known D5 probe input or chain pill prompt)
and a clean CR Round 2 across all reviewing agents.
- Post-merge dashboard re-probe will be the final runtime verification.
## Out of scope (separate follow-ups)
- LG-TS / LG-FastAPI catchall fixtures don't have the stale string — use
pill-aligned userMessages; need different fix
- `tool-rendering` (non-catchall) and `tool-rendering-reasoning-chain`
featureTypes have separate failure modes
- Other red cells in the dashboard (frontend-tools-cosmic timeout,
agent-config, auth, etc) are unrelated
The previous 'weather in Tokyo' rename (388c69e68) was a substring of the
main tool-rendering chain pill prompt 'Chain a few tools in this single
turn: get the weather in Tokyo, search flights from SFO to Tokyo, and roll
a d20.' Because aimock loads fixtures alphabetically per integration dir
and uses substring match with first-match-wins, the catchall fixture
(loaded before tool-rendering.json) was intercepting chain pill matches
across 16 integrations.
Rename catchall fixture userMessage to 'forecast for Tokyo' — a phrase
not contained in any other pill prompt. Update the corresponding D5
catchall probe inputs in d5-tool-rendering-{default,custom}-catchall.ts
and the test assertions that pin those inputs.
Call-Site Enumeration: 'weather in Tokyo' remains intentionally in
page.tsx suggestions.ts pills (user-visible UX) and inside the chain
pill prompt itself — neither is in the substring-match path now.
- Migrate readDeclarativeTestIds from booleans to counts so leftover vs
newly-mounted is distinguishable
- everyNewlyMounted gate uses current[k] > baseline[k] (was boolean
!baseline[k] against a count, which falsely blocked at non-zero baseline)
- minCounts enforce newly-mounted delta, not raw current count
(fixes the cross-pill bleed: at-risk metric:3 floor used to pass on
hero's 3 leftover metrics with no fresh mount)
- Per-pill minCounts add the chart sibling asserts D5 was missing:
hero=4 metric+1 pie+1 bar, team=1 table+1 bar, top-account=1 info-row+1 pie
- 31/31 harness tests green
Hero loses its surrounding card (bare KPI strip over the chart cards,
pinned to all six months); team performance pairs the rep table with a
quota-attainment bar chart; top account pairs the fact card with a
product-line pie (new dataset entry); at-risk becomes a risk panel — KPI
strip (ARR at risk / accounts / biggest exposure) over three side-by-side
severity cards with reason + next action. Fixtures re-captured from live
responses; D5 probe drops declarative-card from the hero set; e2e asserts
the accompanying charts and the risk panel; QA docs updated.
Hero pill asserts the composed dashboard conjunctively (card + metric +
pie + bar); pills 2-4 each require a testid the hero is steered not to
mount (data-table / status-badge / info-row), preserving the
newly-mounted anti-masking gate.
Chronic content-reds in the D5/D6 worker families left `lastSuccessAt`
pinned to null on /api/runs, which tripped the dashboard "worker family X
has not completed successfully since Yh ago" silence banner even though
the workers were healthy. Each batch landed with jobs.done=1,
jobs.failed=17 (cells failing inside the probes), the §5.2.1 outcome
precedence derived "failed", findCompletedBatch returned null, and the
§9 family-silence monitor fell back to oldest-batch-enqueuedAt and fired
after 3 x period.
The fix is definitional: a batch counts as a "terminal completion" for
lastSuccessAt purposes when every job reaches a terminal state (done |
failed) with no `result.commError` on any row. Cell-level reds (rollup
counts) no longer block the timestamp; only worker-level outages
(crashed / reclaimed / lease-expired, surfaced via commError) do.
The strict §5.2.1 outcome=="completed" all-green semantic is unchanged
(it still drives lastRun.outcome and §6.2 dashboard rendering). No
caller of run-view consumed the old strict lastSuccessAt for logic
beyond the banner and monitor (grep verified — every external reference
is either a test fixture or the silence-banner display surface).
Tests added (RED -> GREEN):
- run-view.test.ts: lastSuccessAt advances when all jobs reach a terminal
state with no commError, even if cells failed (the regression case)
- run-view.test.ts: lastSuccessAt does NOT count a batch where any job
carries a commError (worker-outage signal stays loud)
- run-view.test.ts: lastSuccessAt does NOT count a batch with non-terminal
jobs (stalled / abandoned still excluded)
- family-silence-monitor.test.ts: does not alert when the family runs every
cycle with cell-level reds but no commError
- family-silence-monitor.test.ts: still alerts when the family stops
emitting results entirely (real outage)
Two existing tests updated to add commError to their "no completion"
fixtures so their original intent (walk-back across non-completion
batches) survives the new predicate:
- run-view.test.ts: lastSuccessAt walk-back through a commError batch
- fleet-runs.test.ts: T5 multi-page walk-back coverage
- queue-client.ts: add missing closing brace at EOF (TS1005 after rebase)
- job-producer.test.ts: add required `family: "d6"` to producer fixtures
and remove duplicate `logger` key in startedProducer
- result-aggregator.test.ts: align with per-row try/catch + dedup-lookup
behavior introduced in 0b2f613e0 — add `persisted: true` and
`writeOverlay` to test writers, update B6 contract expectations
- queue-client.test.ts: drop a stray blank line
Adds a 689-line integration test that exercises the full queue lifecycle
to the /api/runs projection (enqueue → claim → terminal → projection)
across all four families. Updates the railway-envs golden + verify-deploy
drivers regression test to account for the new fleet-runs route surface.
PRODUCER_FAMILY_WIRING (drift-locked set-equal to FLEET_FAMILIES via a unit
test) drives every buildJobProducer call site; the boot-resolved
worker-stale-after window is threaded through BOTH fleet-health and the
shared family-summary projection so they judge staleness against the same
window. Triggered CLI control-plane runs route through familyForLevel so a
registry rename breaks loudly instead of silently enqueueing jobs invisible
to the projection. Test queues across the harness gain a no-op pruneAged
for the new contract.
Read-only /api/runs routes mounted on the control-plane role, backed by
the SHARED memoized family-summary instance (one PB fan-out per TTL).
Bounds: per-route memo, request rate-limit. /health gains the
fleetRuns.lastEvaluatedAt stamp from the family-silence monitor as the
§9 compensating control for a wedged monitor (an external poll detects
the wedge — the monitor cannot report its own host's death).
job-producer takes a family option and stamps it on every enqueue; the
prune-ownership key is the d6 producer's family. family-silence-monitor
rides the existing fleet-health interval (no extra timer to tear down),
keys 6 h rate-limit + recovered one-shot per family, fails open on
PB-down (the meta-alert path must still fire), and renders alert text
from closed-vocabulary parts only (§5.2.1 redaction). control-plane
fire-and-forgets familySilence.tick(now) each fleet-health cycle.
The §5.1 FLEET_FAMILIES registry and the §5.2.1 family-summary projection
that derives per-family outcome / inflight / lastRun / lastSuccessAt from
the PB-backed batches. The memoized variant fan-outs PB reads at most once
per TTL regardless of viewer count — the SAME instance is shared by the
/api/runs routes and the family-silence monitor so a dashboard poll and a
monitor evaluation inside the same TTL cost one PB fan-out total.
queue-client stamps run_id/family onto every enqueue so downstream
projections can attribute jobs to a family-scoped batch; pruneAged
retention legs land here (the d6 producer owns the call, per §4.2).
result-aggregator computes redsIntroduced/redsCleared from claimed/
sequenced job state into probe_runs summary.
Adds run-id/family/worker-id columns to probe_jobs and resource_snapshots,
plus the EnqueueJobInput.family + FleetQueueClient.pruneAged contracts and
the hoisted deriveHealth primitive that downstream projections share. The
fleet-claim PB hook stamps run-id/family at claim time so every later
projection has a stable join key. probes/run-history is updated to read
the new columns.
The auth-middleware-presence regex in queue-client.test.ts hook-parity
suite was anchored to the single-line `}, $apis.requireAdminAuth());`
closer. After oxfmt rewrote fleet-claim.pb.js to its multi-line form
(`},\n $apis.requireAdminAuth(),\n);`) the regex stopped matching
and the test asserted 0 routes were guarded — a false alarm.
Broaden the regex to accept both the single-line closer and the
formatter's split form; the structural intent (each routerAdd
handler-end is followed by the requireAdminAuth middleware) is
unchanged.
PROMOTE_TO_A (defense-in-depth, exploit-class):
- fleet-health.ts reclaim list interpolated workerId raw into the PB filter
literal. workerId is DB-sourced (read back from the workers roster row),
not a compile-time constant, and the same field is escaped via JSON.stringify
at orchestrator.ts:3240 — but the reclaim path was missing the same
hardening. A double-quote in worker_id (corrupt row, buggy self-registration)
would either throw the list (silently skipping this worker's reclaim every
cycle) or widen the filter to claim other workers' jobs. Match the sibling
escape pattern.
PROMOTE_TO_B (doc reconciliations exposed by the CF round-8 audit):
- queue-client.ts COUNT-NAME CAVEAT was stale — the cross-referenced
contracts.ts doc was already updated (commit 80c5c940) but the queue-client
side still asked a future maintainer to make the edit that had already
landed.
- WarmHealthConfig + JobProducerOptions.warmHealth docs said the producer
warms 'every enumerated backend' / 'each enumerated spec'; the implementation
warms gate.specs (post-validation, post-backlog-gate). A fully-backlogged
tick warms nothing. Updated both interface-level docs to match.
- TickResult.reclaimedIndeterminate said 'Of reclaimed, the reclaims...' —
but the field is DISJOINT from reclaimed (sibling SweepResult contract +
the queue-client both state a thrown release lands here exclusively).
Rewrote to 'In addition to reclaimed, ...'.
- TickResult.skippedForBacklog doc named only the dedupe-gate contributor;
the in-function comment correctly documents the fail-CLOSED poisoned-count
fold-in. Expanded the field doc to name both contributors, and to clarify
that the fail-OPEN leg lands in backlogGateFailedOpen separately.
- SweepResult.commErrors pairing equation
(commErrors.length === reclaimed + reclaimedIndeterminate) was asserted
unconditionally on the shared contract, but reclaimedIndeterminate is
optional and fakes may not report the split. Scoped to 'implementations
that report the split'.
Sibling of the queue-client prose fix (CF7 #10), which flagged this
contract doc as describing only the drain phase: despite the name, the
lease phase's long-expired carve-out also claim-deletes claimed/running
rows (stale created-age, long-expired or unparseable lease) into this
count — no re-queue, no comm error, no reclaimed increment.
The S10 caller's ad-hoc raw Date.parse check could disagree with
isWorkerStale's PB-space-form-normalized parser (engine-lenient vs
anchored), so the unparseable-heartbeat warn did not fire precisely when
the staleness check was blind. Wire the F2-exported heartbeatParseable
companion (contracts.ts documents fleet-health as its intended caller)
and surface a per-cycle unparseableHeartbeats count on the
fleet.health.cycle log — red-green pinned (the count was the new
observable; the cycle log previously never fired for a blind-but-online
corrupt row).
CR finding recurring 3x; fixture lines pre-existing on the base (f8aee59b62 /
ef76c6d0ff / 2a9b38bbfe). The fleet contract (contracts.ts ServiceJobResult
.aggregateKey) forbids an e2e_d6:<slug> row on the fleet path - the dashboard
only ever reads d6:<slug> - so fixtures pinning e2e_d6:* keys mask the exact
regression class (success-path vs error-path aggregate-key drift) these
suites exist to catch. Behavior-neutral: every assertion pins the same
production logic, now over conformant keys.
- contracts.test.ts: makeResult probeKey/aggregateKey + the projection pin
-> d6:langgraph-python; fleetSurfaceState row key/dimension -> d6
- worker-loop.test.ts: makeDriver returns d6:<slug> (was e2e_d6:<slug>),
unified with the comm-error/driver-error paths that already pin d6:...;
all inline driver-fake fixtures + the buildServiceJobResult /
runClaimedJob aggregateKey assertions swept in lockstep. The
registry-routing `<kind>:routed` tagging keys are deliberately untouched
(they tag by DRIVER KIND to prove dispatch, symmetric across
e2e_smoke/e2e_demos)
- orchestrator.test.ts: driverInputs.key -> the d6:tracer-slug tracer,
slug-ALIGNED across probeKey/serviceSlug/probe_key because the d6 driver
derives its aggregate side-row slug from input.key (deriveSlug) while the
loop filters by d6:<serviceSlug> - a mismatched tracer would surface the
side row as a phantom cell; pass-through proof kept (aggregateKey echoes
the tracer) and strengthened with the "no D5 features declared" signal pin
CR finding on commErrorFromStatusSignal (contracts.ts:433-463, decode-guard
region touched on this branch): a malformed embedded value decodes to
undefined - indistinguishable from "absent" - so a REQ-B overlay written by a
NEWER producer (new PoolCommErrorKind rolled out write-side first, or a
renamed required field) is silently dropped by an older reader.
- document the version-skew hazard on the decode (return type unchanged:
every render path branches on presence)
- export statusSignalHasCommErrorKey(signal) so consumers can count/log
"key present but undecodable" drops; mirrors the decoder's wire-shape
guards (null/non-object/array are never valid signals)
- pinned with tests: unknown future kind, renamed required field, well-formed,
genuinely-absent, and array/non-object wire shapes
- the mirrored commErrorFromStatusSignal REGION is untouched: companion +
docs live OUTSIDE the byte-identity block; verified by running the
dashboard's commError-contract-drift.test.ts (12/12 green) - the dashboard
mirror (shell-dashboard live-status.ts) still needs its own sibling
companion, to be landed by the dashboard owner
Jordan Ritter