The ms-agent-harness-dotnet slug was excluded from per-cell D6/BE/smoke
probe enumeration by a placeholder fence added 2026-06-07, before the
real column existed. The column shipped in PR #5569 and its d6/d4 aimock
fixtures landed on main today (e10df0b4), so the fence is now stale.
Remove the slug from all 8 exclude SSOT sites so the column populates.
## Summary
Two follow-up fixes that complete the pinned-prod / floating-staging
contract for the showcase fleet (the contract enforced by the promote
CLI in #5566). Prod services must be digest-pinned (`@sha256`), staging
floats `:latest`.
**Fix 1 — `showcase/scripts/deploy-to-railway.ts`: provision prod
digest-pinned, not `:latest`.**
Prod services were being *born* on the mutable `:latest` tag, then later
pinned only at promote time. Now they are born pinned to a resolved
`@sha256` digest at create time, via a new TS GHCR resolver that mirrors
the Ruby promote CLI (`/token` exchange → manifest HEAD →
`Docker-Content-Digest`). Resolution failure is **fail-loud**
(`process.exit(1)`, never a `:latest` fallback). `goLive` asserts the
prod `source.image` is digest-pinned (`assertProdDigestPinned`,
refactored to be dependency-injectable and to throw a typed
`ProdPinError` instead of exiting inline).
**Fix 2 — `showcase/harness/.../image-drift.ts`: stop flagging pinned
prod red.**
Under the pinned-prod contract, prod is intentionally digest-pinned
behind `:latest`, so the image-drift probe was firing false-red on every
prod service. It now renders such prod services **green**
(`pinnedExpected`) while a genuinely missing digest stays **red**.
Staging behaviour is unchanged.
## Verification
- **Red-green proven locally** for both new test surfaces:
- `assertProdDigestPinned` guard: RED = 5 tests `assertProdDigestPinned
is not a function` (untestable inline-exit) → GREEN = 13 passed after DI
refactor; env-mismatch branch: RED = `promise resolved undefined instead
of rejecting` → GREEN after wiring the test's env control.
- Suites: `deploy-to-railway.digest-pin.test.ts` **14 passed**;
`image-drift.test.ts` **26 passed**. Typecheck (scripts + harness) 0
errors; oxfmt + oxlint clean; harness build green.
- **Empirical 6b against live prod** (settled two reviewer
masking-concerns as can't-happen under current config):
- prod harness `SHOWCASE_ENV` is *unset* (not `""`) and
`RAILWAY_ENVIRONMENT_NAME="production"`, so `isProductionEnv()`'s `??`
correctly resolves true → the prod-neutral fix **fires** in prod.
- prod image-drift is discovery-only; all 19 prod `showcase-*` services
are digest-pinned and tag-less, so `expectedTag` resolves to `latest`
for every prod service → a fixed tag cannot reach prod image-drift (no
false-green).
## Review
3-round cr-loop (7 agents/round) converged to zero actionable findings.
One fix-introduced test-scaffolding defect (dead `envId` helper option)
was caught in the confirmation round and fixed. Remaining reviewer notes
are pre-existing issues in untouched goLive/probe code or
theoretical-but-can't-happen-under-live-config items, tracked as
follow-ups (not in scope for this PR).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- remove the retired top-level `docs/` Next app and the disabled
docs-sync workflow/script
- add `docs -> showcase/shell-docs` as a top-level symlink for `cd docs`
muscle memory without restoring the old docs tree
- move the docs model allowlist into `showcase/shell-docs/` and retarget
docs validation/doctest extraction to shell-docs content
- update docs/agent guidance and CI path filters so `docs/` is treated
as an alias, not an active separate docs surface
- tighten the pre-commit package check so non-package docs/tooling
changes do not fan out into the full package matrix
## Validation
- `pnpm exec tsx scripts/validate-doc-model-names.ts`
- `pnpm exec tsx scripts/doc-tests/extract.ts`
- `pnpm exec vitest run
scripts/__tests__/validate-doc-model-names.test.ts
scripts/doc-tests/__tests__/extract.test.ts
showcase/harness/src/cli/eval/scope.test.ts`
- `pnpm exec oxlint showcase/harness/src/cli/eval/scope.test.ts
scripts/doc-tests/extract.ts`
- `git diff --cached --check` before follow-up commit
- `test "$(readlink docs)" = "showcase/shell-docs"`
- `test -f docs/package.json`
- `pnpm exec oxfmt --check .claude/docs/documentation.md
.claude/docs/hooks.md AGENTS.md CLAUDE.md CONTRIBUTING.md
showcase/shell-docs/README.md`
- commit hooks passed
## Notes
- historical docs remain recoverable from
`archive/docs-save-do-not-prune` and `archive/docs-retired-2026-06-17`
- I intentionally left Vercel/project teardown out of this PR; this is
repo cleanup only
Under the pinned-prod contract, prod is intentionally digest-pinned behind
:latest. image-drift now renders such prod services green (pinnedExpected)
instead of red; a genuinely missing digest stays red. Staging unchanged.
Adds an integration-style test file that pins the BOTH layers of the
2026-06-17 Cloudflare-WAF-burst incident fix together:
- the enumerator retries 3× on a 429+Cloudflare-1015 burst before
bubbling (the original bug let one 429 abort the whole enumerate);
- the silence monitor requires THREE consecutive silent evaluation
cycles before posting an alert (the original bug fired on tick #1).
These assertions were the LITERAL red proof on `main`:
- on main both `it` blocks PASSED while asserting the buggy behavior
(calls === 1, posts.length === 1 after a single silent tick),
- on this branch the same gates re-pin the fixed behavior (calls === 4
after retries; posts === [] until the third silent tick).
Run-time output captured for the PR body confirms the inversion.
Layer a per-family consecutive-silent-tick counter ON TOP of the existing
3×period elapsed-time gate (`SILENCE_PERIOD_MULTIPLIER`). The silence
alert now requires BOTH:
- `now - lastSuccessAt > 3 × period` (existing elapsed-time gate), AND
- `SILENCE_CONSECUTIVE_TICK_THRESHOLD = 3` consecutive evaluation cycles
observed silent (new — the counter resets on any successful evaluation).
Without the new gate a single bad cron tick on a family whose
`lastSuccessAt` was already stale (e.g. after a long quiet window or a
deploy gap) tripped the alert immediately — the failure mode the
2026-06-17 Cloudflare-WAF-burst incident exposed where one ~25 min flap
on backboard.railway.com/graphql/v2 paged every family at once.
The counter is NOT incremented during boot grace, so a cold-start cycle
can't alone push it to threshold. The meta-alert (`family-silence-eval`)
path keeps its own clock and is unaffected. Existing tests advance
through three consecutive silent ticks before asserting the post.
Three-retry exponential backoff (1s/4s/16s) on `source.enumerate` against
Railway-GQL when the underlying error is transient (HTTP 429, 5xx, or a
Cloudflare 1015/1020/1022 WAF marker, or a transport-level reject). On
persistent failure, fall back to the last successful catalog from a
per-enumerator in-memory cache — LOUDLY logged via
`fleet.producer.enumerate-failed-using-cache` so the cache-use shows up
in observability.
A fresh-boot process with no cached entry preserves the current
hard-fail behavior (the producer's `enumerate-failed` short-circuit) —
without a catalog there is nothing to enqueue. Real config errors
(`DiscoverySourceAuthError`, non-429 4xx, schema rot) are NOT retried so
operator-actionable failures surface immediately.
Context: 2026-06-17 Cloudflare WAF burst-blocked
backboard.railway.com/graphql/v2 for ~25 min, hard-failing the producer
enumerate on every cron tick and zeroing out D4/D5/D6 writes — the
entire staging dashboard went red within one tick. Retries + cache ride
out the burst on the same tick and preserve job production across
longer outages.
Pre-existing repo-wide lefthook test failures (@copilotkit/core,
@copilotkit/runtime, @copilotkit/shared, etc.) are unrelated to the
harness; verified by stashing my changes and running the same hook on
main with identical failures. Harness suite (131 files / 2812 tests),
typecheck, and build pass on this branch.
- Add control-plane-run.test.ts (17 tests) covering buildLocalServicesJson,
expectedKeys, dedupeScopes, and runViaControlPlane error surfacing
- Export SlugScope, buildLocalServicesJson, expectedKeys, dedupeScopes
for unit-test coverage (factored inline dedup loop into dedupeScopes
helper at the same time)
- runViaControlPlane: surface scopeLabel (demo-aware) in the 0-enqueue
error instead of the bare-slug join, with an empty-targets guard so
the error never renders with a double-space gap
- runViaControlPlane: treat tick.enqueueFailures > 0 as fatal — partial
enqueue used to silently proceed and either mask missing cells or
hang the poll loop to timeout
- Eliminate a stray literal NUL byte in the source by switching the
dedup key separator to a \x00 escape
- lifecycle.up(): name the compose call (infra-up vs target rebuild)
in the health-fail error so an operator can tell which call left a
service unhealthy
(cherry picked from commit 9f35c64adfdf7f5ff2bf0a5ae57ed03818cea607)
CR Finding 1 (BLOCKER): d5-tool-rendering-default-catchall.ts used the
broken page.evaluate(fn, arg) second-arg form to pass the leak-phrase
needle into the browser-side closure. A11 proved empirically that the
arg arrives as undefined inside the closure, making 'if (needle)' guard
the entire leak-detection cascade as dead code — customLeakPhrasePresent
stayed false forever, rendering validateDefaultCatchall's leak branch
dead code as well. Mirrors the A11 fix on the sibling custom-catchall
probe by inlining the needle as a JS string literal inside the closure;
no page.evaluate(fn, arg) dependency at all. Both probes now share the
same inline-needle pattern and keep the canonical literal in lock-step
with their exported phrase constant.
CR Finding 2 (MAJOR): the A11 inline-needle fix on the sibling
custom-catchall probe had no regression test — fake Page.evaluate in
makePageReturning never executes the probe closure, so reverting the
fix would not be caught. Added regression tests that capture the
probe's function source via toString() and assert (a) the canonical
phrase appears as a literal inside the page.evaluate(...) closure and
(b) the closure takes no parameter / the evaluate call has no
second arg. Added the same coverage to default-catchall to protect
the new A25a fix.
CR Finding 3 (MAJOR): A7's requireContentPhrase=true branch in
validateCustomCatchall / assertCustomCatchall had zero coverage —
tests omitted the third arg and exercised only the default false
branch. Added coverage for the true branch (pass on phrase present,
fail on phrase absent, fail on phrase undefined, default-branch
preserved) plus assertCustomCatchall plumbing through the options
form. Also added coverage for default-catchall's customLeakPhrasePresent
branch in validateDefaultCatchall for symmetry.
Local proof:
- RED (fix reverted via git stash): 2 inline-needle regression tests
fail on d5-tool-rendering-default-catchall.test.ts
- GREEN (fix restored): 34/34 tests pass across both files
Out of scope (NOT touched this commit): showcase/harness/src/cli/
control-plane-run.ts and lifecycle.ts (A25b's scope).
(cherry picked from commit c409e3a8d99e16ad0bb05ee3c2e2051e5792049f)
Re-tier the showcase docs tree to be an agent entry point: README.md
opens with a 'when X, see Y' fanout table that routes to the right
procedural doc; each procedural doc gets a one-line tagline answering
'what does this answer'.
Consolidation:
- DELETE showcase/RUNBOOK.md — operational content merged into DEBUGGING.md
(Integration Patterns, Docker Compose Environment, Production Debugging,
Anti-Patterns, Aimock Fixture Deployment, Dev Iteration Speed). The
--isolate mechanics + CLI rules were already duplicated in DEBUGGING.md.
- DELETE showcase/QA-COVERAGE.md — per-demo coverage matrix + starter hero
matrix + probe depth + infra locations + gaps folded into TESTING.md as
the 'Per-Demo Coverage Matrix' section.
Taglines added (no behavioral change to content): TESTING.md, DEBUGGING.md,
GOTCHAS.md, INTEGRATION-CHECKLIST.md, STYLING-GUIDE.md, FRONTEND-STRATEGY.md,
RAILWAY.md, bin/README.md, aimock/README.md, aimock/RAILWAY.md,
harness/README.md, harness/docs/rotation-drill.md.
Cross-link fixups: FRONTEND-STRATEGY.md (was QA-COVERAGE.md →
TESTING.md#per-demo-coverage-matrix), TESTING.md (removed dangling RUNBOOK
companion reference), README.md (rewritten as fanout entry + retained
from-scratch setup + dashboard SOPs below the fanout).
PARITY_NOTES.md × 12 left alone (per-slug context, not redundant).
(cherry picked from commit 75c9d9755c9118c8abc1fa52deda2012b768cab1)
(cherry picked from commit b64189bae0fe2c9e3a5e3ca440013deb4121f23b)
A21 (a9114a831) regression: positional slug after 'up' restricted which services start (only target+depends_on came up; infra profile services never started). With concurrent sibling stack on same host ports, health checks crossed to foreign pocketbase → cell results misrouted → 0.0s red. Fix: split into 2 compose calls — (1) compose infra up -d, then (2) compose --profile slug up -d --build slug. Preserves A21's target-only rebuild + restores full infra stack.
(cherry picked from commit d71ff46edb4a7b0a89f89d8126fd5fe3d5c7d24e)
Root cause
----------
Both the shell `cmd_up` (showcase/bin/showcase line 98) and the harness
auto-start path (showcase/harness/src/cli/lifecycle.ts up() line 268) emit
`docker compose up -d --build` with NO positional service filter after `up`.
Under `docker compose` semantics, an unfiltered `--build` rebuilds EVERY
service in every active profile — for `--isolate` runs that means the
target slug PLUS aimock + pocketbase + dashboard + harness-pool-worker
all rebuild on every isolated test invocation.
Concurrent `--isolate` runs (one per `SHOWCASE_ISO_SLOT=<N>`) contend on
the shared BuildKit instance, serializing what should be parallel work.
This is the BuildKit-contention stall that has been blocking csdkts (A20)
when it runs alongside other concurrent isolated cells: a small slug
rebuild waits on every other slot's full-stack rebuild.
Fix
---
Scope `--build` to the targeted slug(s) by passing them as positional
service args AFTER `up`. Compose then rebuilds ONLY those services and
falls through to cached images for infra services. First-time bootstrap
still works because `--build` only FORCES a rebuild of services with an
existing image — missing images are built automatically by compose.
Two call sites:
- showcase/bin/showcase (user-facing `showcase up <slug>...`)
- showcase/harness/src/cli/lifecycle.ts (npx tsx test path, the one
actually triggered under `--isolate`)
When no slugs are provided (infra-only bring-up), the blanket `--build`
is preserved so first-time infra bootstrap rebuilds whatever is missing.
`--rebuild` regression: the `--rebuild` flag goes through `rebuild()`,
not `up()`, and that path is unchanged — it still force-rebuilds and
force-recreates every targeted slug as before.
Red→Green proof
---------------
3 new vitest cases in showcase/harness/src/cli/lifecycle.test.ts assert
the compose argv shape:
- up(["langgraph-python"]): slug positional after `up` (was: missing)
- up([]): no slug positional (infra-only blanket)
- up(["a","b"]): both slugs as positionals after `up`
Pre-fix: 2 failed | 4 passed (6)
Post-fix: 0 failed | 6 passed (6)
The compose-argv assertion is the deterministic proof of behavior
change. Wall-clock timing improvement on concurrent --isolate runs
follows mechanically from the compose semantics: with the fix, slot
N's BuildKit work no longer blocks slot M's because they only rebuild
their respective slug images, not the shared infra stack.
(cherry picked from commit 8a47dbb0d2a2fe72e21c09a824485e1bb5afc463)
Before A18, `bin/showcase test <slug>:<demo> --d5/--d6` routed through the
control-plane runner but `runViaControlPlane()` collapsed every TestTarget
to its bare slug:
const slugs = [...new Set(targets.map((t) => t.slug))];
`target.demo` was silently dropped, and:
- `buildLocalServicesJson` hardcoded `demos: ["agentic-chat"]` for d5 and
the full demo set for d6 — regardless of what the operator typed.
- `expectedKeys` always emitted the level's DEFAULT-scope key
(`d5:<slug>/agentic-chat` for d5, `d6:<slug>` aggregate for d6).
The net effect on d5: typing `built-in-agent:tool-rendering-custom-catchall
--d5` enqueued the agentic-chat representative, ran agentic-chat in the
worker fleet, and wrote the side row `d5:built-in-agent/agentic-chat` —
green in ~0.0s — while the dashboard reported PASS for a cell the run
never exercised. Validation was dishonest by construction; the CLI claim
of per-demo coverage was substituted with the default representative.
Fix (additive):
- Introduce a per-call SlugScope = { slug, demo? } and thread it through
`buildLocalServicesJson` + `expectedKeys` + `runViaControlPlane`.
- When `target.demo` is set:
* `LOCAL_SERVICES_JSON` synthesizes `demos: [<demo>]` (the worker's
d6 driver reads `input.demos` → `demosToFeatureTypes` → the closed
featureType set, so the matrix narrows to exactly that demo's
featureType(s)).
* `expectedKeys` translates the demo ID into its featureType(s) via
the same `REGISTRY_TO_D5` mapping the driver uses, and waits on
`<level>:<slug>/<featureType>` — NOT the default-scope key. An
unmappable demo throws (would otherwise hang to timeout).
- When `target.demo` is absent: zero behavioral change. d5 still
enqueues the agentic-chat representative and waits on
`d5-single-pill-e2e:<slug>` + `d5:<slug>/agentic-chat`; d6 still
enqueues the full demo set and waits on the `d6:<slug>` aggregate.
- CLI banner now prints the per-scope label (`<slug>:<demo>` qualifier
when present), mirroring the legacy direct path's labeling.
Repro (built-in-agent:tool-rendering-custom-catchall, --d5 --isolate):
BEFORE
Waiting for worker fleet to produce cells:
d5-single-pill-e2e:built-in-agent, d5:built-in-agent/agentic-chat
✓ 2 passed (0.0s) ← agentic-chat representative ran;
catchall never touched. False-positive.
AFTER
Waiting for worker fleet to produce cells:
d5:built-in-agent/tool-rendering-custom-catchall
✗ 1 failed: state=red ← catchall actually ran. Cell is RED at
this HEAD (BIA narration→DOM is a
separate task A12). Validation is now
HONEST.
Regression: bare-slug d5 default UNCHANGED:
built-in-agent --d5 →
d5-single-pill-e2e:built-in-agent, d5:built-in-agent/agentic-chat
✓ 2 passed (0.0s)
Files:
showcase/harness/src/cli/control-plane-run.ts (+96 -17)
- import demosToFeatureTypes
- SlugScope interface
- buildLocalServicesJson: per-slug demo override
- expectedKeys: per-demo featureType-derived keys; refuse-loud
on unmappable demo
- runViaControlPlane: (slug, demo) dedup; scope-aware banner/log
(cherry picked from commit 82ae79279415b7eaf91759df4d8bc1625816a8c3)
Root cause (A11 investigation, PR #5495): the probe passed the canonical
phrase into the browser-side `page.evaluate` closure via the second-arg
form (`evaluate((expectedPhrase?: string) => …, phrase)`). Empirically
the arg arrives as `undefined` inside the closure — verified via
in-closure return diagnostics showing `needleLen === 0` while
`bubble.textContent` and `body.textContent` BOTH contained the canonical
phrase at the SAME poll moment. With `needle === """, the
`if (needle)` guard skipped the entire cascade, leaving
`customContentPhrasePresent` false forever and failing every
integration's `tool-rendering-custom-catchall` cell even when the
narration was reaching the DOM correctly (LGP gold).
Fix: inline the canonical phrase as a JS literal inside the closure —
no `page.evaluate(fn, arg)` second-arg dependency. Keep the existing
per-tier scoped cascade and `body.textContent` fallback semantics
unchanged. The shared constant `CUSTOM_CATCHALL_CONTENT_PHRASE` and
the inlined literal MUST stay in lock-step (existing
`exports the testid contract for cross-test reuse` unit test verifies
the constant; manual sync required on edit of the literal).
RED-GREEN proof:
RED (pre-fix, /tmp/cr/a11-red-lgp.log INNER_EXIT=1):
LGP `d5-single-pill-e2e:langgraph-python red`
settled text turn 2 = 'AAPL is trading at $338.37, down 2.96% —
rendered through the custom wildcard catchall renderer.' (phrase
IS in bubble.textContent) but
snap.customContentPhrasePresent: false ← THE BUG
GREEN (post-fix, /tmp/cr/a11-final-lgp.log INNER_EXIT=0):
LGP `d5-single-pill-e2e:langgraph-python green (8.9s) → 1 passed`
BIA remains red (separate fixture-routing concern — BIA's narration is
not reaching the DOM at all per `settled text` = pure tool-card text,
`assistantMsgCount: 2`, no canonical-phrase bubble; outside this
fix's scope per the worktree's modify-harness-only constraint).
(cherry picked from commit a980be48c514a2ab4972395e80e70e5b19247131)
Other dimension drivers under showcase/harness/src/probes/drivers/ follow
the dN-<purpose>.ts naming pattern (d4-chat-roundtrip.ts, d6-all-pills.ts).
D2 and D3 didn't — bring them in line.
drivers/liveness.ts -> drivers/d2-liveness.ts
drivers/liveness.test.ts -> drivers/d2-liveness.test.ts
drivers/e2e-readiness.ts -> drivers/d3-readiness.ts
drivers/e2e-readiness.test.ts -> drivers/d3-readiness.test.ts
The d3 driver loses the redundant 'e2e' prefix — the dimension is D3
and the corresponding dashboard badge is 'UI', so the file's job is best
described as 'is the demo frontend ready?'.
Imports updated in:
- showcase/harness/src/orchestrator.ts
- showcase/harness/src/orchestrator.test.ts
- showcase/harness/src/cli/runner.ts
- showcase/harness/src/probes/drivers/d2-liveness.test.ts (self-ref)
- showcase/harness/src/probes/drivers/d3-readiness.test.ts (self-ref)
Doc-comment references to the old file paths updated in:
- showcase/harness/src/cli/targets.ts
- showcase/harness/src/probes/liveness.ts (the helper, not the driver)
- showcase/harness/src/probes/loader/probe-invoker.ts
- showcase/harness/src/probes/discovery/railway-services.ts
- showcase/harness/src/probes/drivers/starter-smoke.ts
- showcase/harness/src/probes/drivers/starter-smoke.test.ts
- showcase/harness/src/types/dimensions.test.ts
- showcase/harness/config/probes/e2e-demos.yml
- showcase/harness/README.md (driver table)
Function/symbol exports unchanged (livenessDriver, e2eReadinessDriver,
createE2eDemosDriver, etc.) — file rename only. The parent helper module
showcase/harness/src/probes/liveness.ts is also untouched: it's a
separate, non-driver helper that exports deriveHealthUrl, livenessProbe,
and LIVENESS_SLACK_SAFE_FIELDS. Not renamed: drivers/e2e-parity.ts,
drivers/aimock-wiring.ts, drivers/pin-drift.ts, drivers/image-drift.ts
(non-dimensional probes).
validateCustomCatchall's requireContentPhrase branch was unreachable: assertCustomCatchall defaulted it to false and no caller overrode. The whole point of the PR is to catch cross-fixture leakage via content assertion — wiring it on. Turn-2 (Quote AAPL) now asserts the custom content phrase is present in the rendered bubbles, not just the testid.
(cherry picked from commit c9620c96f9addfee18f87fb1f8b2ae7fb040b3b8)
Probes registered fixtureFile: 'tool-rendering.json' but actual files are tool-rendering-{default,custom}-catchall.json. Field is signal-only (aimock loads directory-wide content-driven matching) but accurate metadata helps debugging. Test files updated to enforce the correct values.
(cherry picked from commit 3c9de26375546f2269870b563ecd1526ec5a3bc2)
Closed PR #5465 introduced cross-fixture leakage by stripping the
toolName discriminator on shared {userMessage, context} keys: with
aimock's alphabetical first-match-wins ordering,
tool-rendering-custom-catchall.json sorts before
tool-rendering-default-catchall.json, so default-catchall page requests
were served the custom file's content. The probe was structurally blind
because it asserted only DOM testids (copilot-tool-render +
data-tool-name=get_weather) — both fixtures emit get_weather, so the
testid signal passed regardless of which fixture won.
Real LGP-gold pattern is disjoint userMessages between default and
custom catchall fixtures (NOT shared keys discriminated by toolName).
This PR ports the LGP-gold pattern to the other 17 integrations and
adds page-text content assertions to both d5 catchall probes so this
class of regression can't recur silently:
- Probe prompts disjoint between default-catchall and custom-catchall
- Fixture userMessages updated to match the new disjoint prompts
- Page-text content assertions in both d5 catchall probes
(default negatively asserts the custom-catchall leak phrase;
custom positively asserts it)
Local gold-standard red-green proof captured:
- Step A: live staging Playwright baseline (RED-OF-RECORD)
- Step B: local control-plane on origin/main reproduces structural
fragility (probes pass at testid level, custom fixture wins on
default's userMessage path)
- Step C: local control-plane on this branch — both catchall probes
GREEN with the new content-asserting assertions
tool-rendering-default-catchall: pass=true (5443ms)
tool-rendering-custom-catchall: pass=true (8956ms)
cross-tool signature passed
LGP (langgraph-python) was already disjoint; it remains untouched.
## Summary
The smoke probe was the same HTTP contract as `/health` on the same
service (200-OK JSON body), so every tick paid two HTTP calls for the
same liveness signal. This PR:
- Drops the `/smoke` GET probe and the `smoke:<slug>` primary
`ProbeResult`. The driver now emits `health:<slug>` as the primary and
`agent:<slug>` via writer side-emit (half the per-tick cost).
- Renames the dashboard's D3 / e2e row label from `E2E (Demo)` to `UI
(Frontend)` (and the short cell badge from `E2E` to `UI`) — the row was
already about "the demo page renders in a browser", not E2E in the
traditional sense.
- Drops the `Smoke` row from the cell drilldown (CellState.smoke field
is retained for back-compat so historical rows still parse).
- Adds an explicit `Health` row to the legend (was implicit).
The driver's registry `kind` stays `"smoke"` so existing YAML configs +
orchestrator family wiring keep routing to this driver — the emission
key (`health:<slug>`) is the taxonomy contract that matters. The
underlying probe key (`e2e:<slug>/<feature>`) is preserved on PocketBase
so historical rows render correctly during the rename window.
## RED proof (BEFORE)
`smoke:<slug>` primary emission in `liveness.ts` (origin/main):
```
18: * 1. `smoke:<slug>` — the RETURN VALUE of `run()`. The invoker runs
56: * `smoke:<slug>`/`health:<slug>`/`agent:<slug>` triple the dashboard
241: input.mode === "discovery" ? `smoke:${slug}` : input.key;
```
`E2E (Demo)` / `Smoke` strings in dashboard render code (origin/main):
```
cell-drilldown.tsx:38: * Trip)" (chat+tools round-trip), D3/e2e = "E2E (Demo)" (the demo page loads
cell-drilldown.tsx:49: { key: "e2e", label: "E2E (Demo)" },
cell-drilldown.tsx:52: { key: "smoke", label: "Smoke" },
cell-pieces.tsx:382: name="E2E"
unified-cell.tsx:231: <TestBadge name="E2E" level={model.d3} />
adaptive-legend.tsx:59: E2E (Demo): demo page loads and round-trips in a browser
```
## GREEN proof (AFTER)
Primary emission rewritten — driver returns `health:<slug>` and no
`smoke:<slug>` ProbeResult is ever emitted:
```
33: * to `/smoke` and emitted a `smoke:<slug>` primary result, but that probe
230: // `kind: "smoke"` is the registry/family identifier the orchestrator
236: kind: "smoke", // registry-key back-compat; no smoke ProbeResult is emitted
367: if (input.mode === "discovery") return `health:${slug}`;
368: if (input.key.startsWith("smoke:")) {
369: return `health:${input.key.slice("smoke:".length)}`;
```
New labels in dashboard:
```
cell-drilldown.tsx:60: { key: "e2e", label: "UI (Frontend)" },
cell-pieces.tsx:382: name="UI"
unified-cell.tsx:231: <TestBadge name="UI" level={model.d3} />
adaptive-legend.tsx:65: UI (Frontend): demo page renders in browser (Playwright)
```
Confirmation that the old labels are gone from render code:
```
$ grep -rn 'E2E (Demo)\|"Smoke"' cell-drilldown.tsx cell-pieces.tsx unified-cell.tsx adaptive-legend.tsx
(no matches — labels removed)
```
## Test results
- `pnpm exec vitest run` in `showcase/shell-dashboard`: **63 files, 1089
tests passed, 1 skipped**
- `pnpm exec vitest run` in `showcase/harness`: **129 files, 2760 tests
passed**
- `pnpm exec tsc --noEmit` in both: clean
Liveness driver tests include a regression guard (`never emits a
smoke:<slug> key`) so the smoke contract cannot be re-introduced
silently.
## Test plan
- [x] Dashboard vitest green (1089/1089)
- [x] Harness vitest green (2760/2760)
- [x] tsc --noEmit green in both packages
- [x] Visual cross-check: drilldown renders 6 rows (no Smoke), e2e row
labelled `UI (Frontend)`; cell strip shows `UI` badge in place of `E2E`;
legend shows explicit `Health` row + `UI (Frontend)` D3 line
The smoke probe was the same HTTP contract as /health on the same
service (200-OK JSON body), so every tick paid two HTTP calls for the
same liveness signal. Drop the /smoke GET + the smoke:<slug> primary
ProbeResult; the driver now emits health:<slug> as the primary and
agent:<slug> via writer side-emit (half the per-tick cost).
The driver's registry kind stays "smoke" so existing YAML configs and
orchestrator family wiring keep routing to this driver — the
emission key (health:<slug>) is the taxonomy contract that matters.
Dashboard:
- Drilldown: D3/e2e row labelled "UI (Frontend)" (was "E2E (Demo)");
Smoke row dropped (CellState.smoke field retained for back-compat).
- Cell badges: short label "UI" (was "E2E") in cell-pieces + unified-cell.
- Legend: D3 = "UI (Frontend): demo page renders in browser (Playwright)"
plus an explicit Health row at the top.
Tests updated to assert new labels:
- cell-drilldown.test.tsx: 6 dimensions (no Smoke); UI (Frontend) label.
- cell-drilldown.lazy-signal.test.tsx: drilldown-badge-ui--frontend- testid.
- cell-pieces.test.tsx + .signal-degrade.test.tsx: badge name "UI".
- unified-cell.test.tsx: mock-badge-UI.
- overlay-selector-integration.test.tsx: "UI" in place of "E2E".
- dashboard-color-matrix.test.tsx: badge: "UI" case names.
- liveness.test.ts: two-call contract (health + agent), regression guard
asserting no smoke:<slug> ever emitted.
Underlying probe key (e2e:<slug>/<feature>) preserved on PocketBase so
historical rows render correctly during the rename window.
PR #5462's defect-2 fix assumed 1 bubble per turn (bubbleIndex =
turnIndex - 1). Multi-step agents (LangGraph, Mastra, CrewAI) emit
2-3+ bubbles per turn — tool-call + tool-render + final-text — so
the strict-index read lands on an intermediate tool-call bubble
whose scoped-text selectors (`.cpk:prose`, `[data-message-content]`,
`p`) are EMPTY (its content lives in a sibling card div outside the
scoped cascade). The cascade-pollution guard returns null forever
and the text-stable conjunct times out with `reason=text-unstable`
across virtually every multi-step demo (beautiful-chat-*, gen-ui-*,
shared-state-*, agentic-chat on LangGraph/Mastra/CrewAI/etc.).
Live DOM evidence from staging (langgraph-typescript :
beautiful-chat : pie chart prompt):
count = 3 bubbles
bubble[0]: tool-call "query_data" — `.cpk:prose` empty
bubble[1]: pie chart card — `.cpk:prose` empty
bubble[2]: final text "Here is the pie chart..." — 90 chars
Fix: read the LAST bubble in the matched cascade tier (count - 1)
instead of a fixed strict index. Defect-2 protection (don't read a
leftover bubble from a prior turn) is preserved via an explicit
pre-submit `baselineCount` sentinel snapshot in the runner; the
gate now requires `count > baselineCount` rather than
`count > turnIndex - 1`, which correctly rejects stale bubbles
without assuming 1 bubble = 1 turn.
Changes:
- assistant-message-count.ts: add `readCascadeStateLast(page)` —
same atomic single-evaluate contract as `readCascadeState` but
resolves the bubble index internally as `count - 1`.
- conversation-runner.ts: snapshot `baselineCount` before
sendTurnMessage; pass through to `waitForTurnComplete` via new
`baselineCount` option (defaults to `turnIndex - 1` for unit-
test fakes); `waitForTurnComplete` reads via
`readCascadeStateLast` and gates on `count > baselineCount`.
Cold-start retry re-snapshots `baselineCount` after page.reload.
Test plan:
- All 72 conversation-runner + assistant-message-count unit tests
pass (default baselineCount preserves count-progression scripts).
- Full harness suite 2758/2758 green, tsc clean, oxfmt/oxlint
clean on changed files.
- d5-* unit tests: update page fakes / assertion-call shapes to pass
the ctx bridge ({bubbleIndex, text}) instead of querying the live
bubble list
- d6-all-pills.test.ts: update page-fake dispatch + cascade-state
expectations to match the atomic readCascadeState return shape
({count, text} or null) — picks up the cross-tier consistency
guarantee in the unit layer
- d6-all-pills.ts: wire installPrePaintFromEnv and
attachSseInterceptor PRE-goto in both defaultLauncher and pooled
launcher paths, so first-paint state is deterministic and the SSE
counter is armed before any page navigation
- captureDiagnostics rewired through the Node-side helper
(readCascadeState) — single page.evaluate per call, no
within-snapshot tier drift
- Consume BUBBLE_RACE_MESSAGES override for defect-4 repro flows
- d5-gen-ui-custom, d5-gen-ui-open-advanced, d5-mcp-apps,
d5-subagents, d5-tool-rendering-default-catchall: wrap assertion
bodies in ctx bridge so they receive {bubbleIndex, text} for the
turn under test, replacing turn-leaky list[last] retrieval
- _gen-ui-shared: add readAssistantTextAt(page, bubbleIndex)
adapter; remove readLastAssistantText (no longer turn-scoped)
This is the read-side counterpart to the atomic readCascadeState
on the helper side: probes ask for the bubble at the runner's
chosen index instead of guessing.
R3-F1 (OPS_TRIGGER_TOKEN hoist):
Extract loadOpsTriggerToken() and hoist its invocation to the top of both
boot() and runControlPlane(), alongside loadWebhookSecrets() and
loadPocketbaseUrl(). Pre-fix the empty/whitespace check fired AFTER pb /
bus / scheduler / writer / S3 uploader allocations, so a typo'd
`OPS_TRIGGER_TOKEN=` allocated expensive resources before throwing. Now
all three fail-loud config predicates fire at the top, before any
allocation needing teardown. Behaviour is unchanged for valid tokens
(trimmed via R3-A.5 contract) and for the unset case (router omitted with
info log).
R3-F2 (sync-throw also emits deploy.writer.failed):
subscribeDeployResults() pre-fix only emitted `deploy.writer.failed` on
writer.write() promise rejection — a synchronous throw inside
deployEventToProbeResult() (malformed event, type drift) bypassed the
catch and we lost both the log AND the bus emit. Wrap the sync mapping
in try/catch and mirror the rejection path so alert rules / metrics
subscribers observe sync throws the same way they observe async write
failures.
Tests:
- 5 unit tests for loadOpsTriggerToken: undefined / empty-string fail-loud
/ whitespace-only fail-loud / R3-A.5 trim contract / verbatim value.
- 1 test for the sync-throw path: vi.doMock deployEventToProbeResult to
throw, assert bus emits deploy.writer.failed with the err message AND
writer.write was NOT called. Red-green verified locally (test fails
without the try/catch).
Suite: 2731 passing (+6 new). Typecheck clean.
CB-1 (Slot 2 #22): convert B2 deploy-webhook tests to vi.stubEnv with
`vi.unstubAllEnvs()` in afterEach instead of manual process.env mutation
+ try/finally restoration. Test runner now guarantees restoration and the
test bodies are shorter / less error-prone (CR Slot 2 #22).
CB-2 (Slot 2 #28): when `loadWebhookSecrets`' escape hatch fires with a
real-looking NODE_ENV (anything except "test"), log at `warn` instead of
`info` so a production typo (NODE_ENV=staging + HARNESS_ALLOW_NO_SECRET=1)
is visible in dashboards / log alerting. Pure local-dev (NODE_ENV=test)
stays at info level so a normal unit-test boot doesn't spam warnings.
CB-3 (Slot 4 #17): clarify `loadWebhookSecrets`' docstring + bypass-log
message — "set" is ambiguous (empty string would qualify pre-fix); use
"non-empty" to match the actual predicate.
R1-F2 (bucket b, defensive ordering): hoist fail-loud config validation
to the TOP of boot() and runControlPlane() — BEFORE any pb client, queue,
bus, scheduler, writer, S3 uploader, fleet-health, or aggregator
allocations. Pre-fix `loadWebhookSecrets` lived AFTER the entire
scheduler+writer (and CP queue+aggregator) setup, so a misconfigured boot
allocated expensive resources and mounted file watchers before throwing.
R1-F3 (bucket b, predicate symmetry): broaden the POCKETBASE_URL fail-loud
predicate to match SHARED_SECRET semantics. Pre-fix the POCKETBASE_URL
guard only fired on `NODE_ENV === "production"`, while `loadWebhookSecrets`
fired unless NODE_ENV=test or an explicit escape hatch — so staging /
unset / "development" deploys silently bound to http://localhost:8090.
Extracted `loadPocketbaseUrl(logger)` with the same test-or-escape-hatch
predicate as `loadWebhookSecrets`. A new `HARNESS_ALLOW_NO_PB_URL=1` env
flag mirrors `HARNESS_ALLOW_NO_SECRET=1` for local dev. Worker
`boot()` and the CP's `resolveFleetPbConfig` both call the helper.
Tests:
- HF13-A2 production-only assertion broadened (SHARED_SECRET set so the
test specifically exercises the POCKETBASE_URL guard).
- New R1-F3 tests cover NODE_ENV=development without POCKETBASE_URL
(throws) and the HARNESS_ALLOW_NO_PB_URL=1 escape hatch (succeeds).
R1-F1 (bucket a, load-bearing): the CP boot path now subscribes to
`deploy.result` events through its status writer. Pre-fix, B2 mounted POST
/webhooks/deploy on the CP role but only the worker boot path had a
`bus.on("deploy.result", ...)` listener — signed POSTs to the CP host
returned 202, the bus event fired with no subscriber, and the deploy-overall
dashboard row never landed.
Extracted the handler into a shared `subscribeDeployResults(bus, writer)`
helper exported from orchestrator.ts so both boot paths share the IDENTICAL
logic. Worker boot keeps its inline pattern (busUnsubs.push) so teardown is
unchanged; CP keeps the subscription alive for the lifetime of the bus.
Red-green tests in src/orchestrator.test.ts under "R1-F1" assert that
emitting deploy.result on the returned handle's bus drives one write keyed
"deploy:overall" through a stubbed status writer — once for runControlPlane
(the new path) and once for boot() (regression guard for the helper
extraction).
The 'Showcase: Verify Deploy' workflow's notify-harness step has been
POSTing to /webhooks/deploy and getting 404 on every main deploy since
at least 2026-06-12 (5+ consecutive failures). Root cause:
1. POST /webhooks/deploy is only registered when
webhookSecrets.length > 0 (gate at src/http/server.ts:119).
2. The CP buildServer call in runControlPlane omitted BOTH
webhookSecrets and metrics — so the public Railway host running
the CP role never mounted the route. The worker boot path had
them, but it isn't the host receiving notify-harness POSTs.
3. The FATAL-CONFIG guard on missing SHARED_SECRET was gated on
NODE_ENV === 'production' — any deploy with NODE_ENV unset or set
to something other than the literal 'production' silently shipped
with webhookSecrets=[] and no fatal error.
Part 1 — wire webhookSecrets + metrics into the CP buildServer call.
Lifted the env→secrets loader into a new exported helper
'loadWebhookSecrets()' (~orchestrator.ts:175-209) so BOTH boot paths
consume the same predicate. The worker 'boot()' call (~line 836) and
the CP 'runControlPlane' call (~line 2421) both invoke it, and the CP
buildServer call (~line 3047) now passes webhookSecrets+metrics
alongside the existing bus/probes/fleetRuns wiring.
Part 2 — tighten the FATAL-CONFIG predicate.
The new predicate throws unless EITHER a secret is set OR NODE_ENV ===
'test' OR HARNESS_ALLOW_NO_SECRET === '1' (narrow escape hatch for
local dev). The thrown error message names the env-vars, the gate
location ('src/http/server.ts:119'), the silent-404 symptom, and the
escape hatches, so an operator reading the deploy log can recover
without spelunking.
Updated the existing R5-G4 D5 regex (orchestrator.test.ts:1592) since
the error message changed; covered by the new B2 Tests A/C/D below.
Tests: 4 new red-green tests in 'B2: /webhooks/deploy registered on CP
+ fail-loud on missing SHARED_SECRET':
- Test A (CP /webhooks/deploy registration): RED 404 → GREEN 401.
- Test B (worker /webhooks/deploy regression): GREEN 401.
- Test C (FATAL-CONFIG when NODE_ENV=development): RED resolves →
GREEN rejects with /FATAL-CONFIG.*SHARED_SECRET/.
- Test D (NODE_ENV=test escape hatch): GREEN, no throw.
Full harness suite: 2719/2719 green. typecheck: clean.
Requires a CP-role redeploy on the public Railway host for the new
route registration to take effect.
Adds four red-green tests for the notify-harness 404 follow-up:
A. CP boot registers POST /webhooks/deploy when SHARED_SECRET is set
— RED pre-fix (404, route not mounted on CP); GREEN post-fix (401,
the route's own HMAC reject path fires on an unsigned POST).
B. Worker boot still registers POST /webhooks/deploy (regression
guard) — already green; pins the unchanged behavior.
C. boot throws FATAL-CONFIG when SHARED_SECRET unset AND NODE_ENV is
non-test — RED pre-fix (current guard fires only on
NODE_ENV='production'); GREEN post-fix once the predicate is
tightened.
D. boot succeeds when SHARED_SECRET unset AND NODE_ENV='test'
(test-mode escape hatch) — already green; pins the escape hatch.
Red capture (pre-fix run):
- Test A: AssertionError: expected 404 to be 401
- Test C: AssertionError: promise resolved instead of rejecting
## Summary
- D5 e2e-deep probes for `tool-rendering-{default,custom}-catchall` send
`"forecast for Tokyo"` as the test input (see
`showcase/harness/src/probes/scripts/d5-tool-rendering-{default,custom}-catchall.ts`).
- aimock uses substring match on `userMessage`.
- The catchall fixtures on main had a stale `"check Tokyo weather
forecast"` string that couldn't substring-match the probe input →
fixture miss → probe falls through to live LLM → CV ✗ red D4 on the
dashboard.
- This PR renames the userMessage to the canonical `"forecast for
Tokyo"` across 28 catchall fixture files in 16 integrations.
## Scope
28 files × ~2 userMessage occurrences each = 67 line changes. **No code,
no agent, no page.tsx changes.** Pure fixture-data alignment.
Integrations covered (default-catchall and/or custom-catchall): ag2,
agno, built-in-agent, claude-sdk-python, claude-sdk-typescript,
crewai-crews, google-adk, langgraph-python, langroid, llamaindex,
mastra, ms-agent-dotnet, ms-agent-python, pydantic-ai, spring-ai,
strands.
## Commit history note
Commit b1f19bdc8 changes the rename target from the original `"weather
in Tokyo"` (commit 388c69e68) to `"forecast for Tokyo"`. This was a CR
Round 1 catch: `"weather in Tokyo"` was a substring of the chain pill
prompt (`"weather forecast chain in Tokyo"` / similar), which would have
caused the catchall fixture to incorrectly match chain-pill traffic.
`"forecast for Tokyo"` has no such substring collision with any other
probe input.
## Verification
- **Live red-green proof** was performed on the prior `"weather in
Tokyo"` rename (commit 388c69e68) against `ms-agent-python` via
`showcase/bin/showcase test ms-agent-python --d6 --isolate`:
- Before (origin/main): catchall featureTypes red (fixture miss → live
LLM → flaky)
- After: `d6:ms-agent-python/tool-rendering-default-catchall=green`,
`d6:ms-agent-python/tool-rendering-custom-catchall=green`
- **The current HEAD's `"forecast for Tokyo"` rename (b1f19bdc8) has NOT
been re-run live.** It is verified by static analysis only: substring
math (no collision with any known D5 probe input or chain pill prompt)
and a clean CR Round 2 across all reviewing agents.
- Post-merge dashboard re-probe will be the final runtime verification.
## Out of scope (separate follow-ups)
- LG-TS / LG-FastAPI catchall fixtures don't have the stale string — use
pill-aligned userMessages; need different fix
- `tool-rendering` (non-catchall) and `tool-rendering-reasoning-chain`
featureTypes have separate failure modes
- Other red cells in the dashboard (frontend-tools-cosmic timeout,
agent-config, auth, etc) are unrelated
The previous 'weather in Tokyo' rename (388c69e68) was a substring of the
main tool-rendering chain pill prompt 'Chain a few tools in this single
turn: get the weather in Tokyo, search flights from SFO to Tokyo, and roll
a d20.' Because aimock loads fixtures alphabetically per integration dir
and uses substring match with first-match-wins, the catchall fixture
(loaded before tool-rendering.json) was intercepting chain pill matches
across 16 integrations.
Rename catchall fixture userMessage to 'forecast for Tokyo' — a phrase
not contained in any other pill prompt. Update the corresponding D5
catchall probe inputs in d5-tool-rendering-{default,custom}-catchall.ts
and the test assertions that pin those inputs.
Call-Site Enumeration: 'weather in Tokyo' remains intentionally in
page.tsx suggestions.ts pills (user-visible UX) and inside the chain
pill prompt itself — neither is in the substring-match path now.
- Migrate readDeclarativeTestIds from booleans to counts so leftover vs
newly-mounted is distinguishable
- everyNewlyMounted gate uses current[k] > baseline[k] (was boolean
!baseline[k] against a count, which falsely blocked at non-zero baseline)
- minCounts enforce newly-mounted delta, not raw current count
(fixes the cross-pill bleed: at-risk metric:3 floor used to pass on
hero's 3 leftover metrics with no fresh mount)
- Per-pill minCounts add the chart sibling asserts D5 was missing:
hero=4 metric+1 pie+1 bar, team=1 table+1 bar, top-account=1 info-row+1 pie
- 31/31 harness tests green
Hero loses its surrounding card (bare KPI strip over the chart cards,
pinned to all six months); team performance pairs the rep table with a
quota-attainment bar chart; top account pairs the fact card with a
product-line pie (new dataset entry); at-risk becomes a risk panel — KPI
strip (ARR at risk / accounts / biggest exposure) over three side-by-side
severity cards with reason + next action. Fixtures re-captured from live
responses; D5 probe drops declarative-card from the hero set; e2e asserts
the accompanying charts and the risk panel; QA docs updated.
Hero pill asserts the composed dashboard conjunctively (card + metric +
pie + bar); pills 2-4 each require a testid the hero is steered not to
mount (data-table / status-badge / info-row), preserving the
newly-mounted anti-masking gate.
Chronic content-reds in the D5/D6 worker families left `lastSuccessAt`
pinned to null on /api/runs, which tripped the dashboard "worker family X
has not completed successfully since Yh ago" silence banner even though
the workers were healthy. Each batch landed with jobs.done=1,
jobs.failed=17 (cells failing inside the probes), the §5.2.1 outcome
precedence derived "failed", findCompletedBatch returned null, and the
§9 family-silence monitor fell back to oldest-batch-enqueuedAt and fired
after 3 x period.
The fix is definitional: a batch counts as a "terminal completion" for
lastSuccessAt purposes when every job reaches a terminal state (done |
failed) with no `result.commError` on any row. Cell-level reds (rollup
counts) no longer block the timestamp; only worker-level outages
(crashed / reclaimed / lease-expired, surfaced via commError) do.
The strict §5.2.1 outcome=="completed" all-green semantic is unchanged
(it still drives lastRun.outcome and §6.2 dashboard rendering). No
caller of run-view consumed the old strict lastSuccessAt for logic
beyond the banner and monitor (grep verified — every external reference
is either a test fixture or the silence-banner display surface).
Tests added (RED -> GREEN):
- run-view.test.ts: lastSuccessAt advances when all jobs reach a terminal
state with no commError, even if cells failed (the regression case)
- run-view.test.ts: lastSuccessAt does NOT count a batch where any job
carries a commError (worker-outage signal stays loud)
- run-view.test.ts: lastSuccessAt does NOT count a batch with non-terminal
jobs (stalled / abandoned still excluded)
- family-silence-monitor.test.ts: does not alert when the family runs every
cycle with cell-level reds but no commError
- family-silence-monitor.test.ts: still alerts when the family stops
emitting results entirely (real outage)
Two existing tests updated to add commError to their "no completion"
fixtures so their original intent (walk-back across non-completion
batches) survives the new predicate:
- run-view.test.ts: lastSuccessAt walk-back through a commError batch
- fleet-runs.test.ts: T5 multi-page walk-back coverage
Jordan Ritter