Approach (A) reorder: decode -> html-strip -> headTailCap -> scrub(head)+scrub(tail).
The prior order scrubbed the FULL decoded body before the cap, so a body >32KB hit
scrubSecrets' bounded-prefix path: it truncated to a ...[unscanned:N] prefix BEFORE
headTailCap, which (1) lost the real tail, (2) never scanned the real tail for secrets,
and (3) zeroed elided_count. Reordering scrubs each retained <=16KB segment AFTER the
cap: input is bounded per-segment (linear regex x bounded length = ReDoS-impossible, no
scan-budget truncation), the captured head AND tail are the REAL ends of the body, and
elided_count comes from headTailCap on the FULL body. The unrelated 2KB metadata-hot-path
scan guard (SCRUB_MAX_SCAN_LEN) is untouched; the now-unneeded RAW_BYTE_SCAN_MAX export
is removed.
The emit-hardening redesign added a 2KB SCRUB_MAX_SCAN_LEN guard inside
scrubSecrets, correct for the DEFAULT-tier metadata hot path (legit values
<=512B). But raw-byte-capture.ts also calls scrubSecrets on the DECODED wire
body, which is legitimately large (<=16KB head + <=16KB tail = up to 32KB).
The 2KB guard truncated the body to ~2KB before headTailCap ran, so for an
>32KB body elided_count became 0 and the head+tail elision never triggered.
Approach: parameterize the scan cap (chosen over the reorder-and-scrub-segments
alternative because it is a minimal, local change that preserves the documented
decode->scrub->html-strip->headtail pipeline order and keeps the metadata
default behavior byte-identical for all existing callers).
- scrub.ts: scrubSecrets gains an optional maxScanLen param defaulting to the
exported SCRUB_MAX_SCAN_LEN (2KB). Metadata/validateMetadata callers are
unchanged; the ReDoS guard on the hot path is untouched.
- raw-byte-capture.ts: passes RAW_BYTE_SCAN_MAX (head+tail cap = 32KB) so the
full retained sample is scanned (no secret in the kept head/tail escapes) and
headTailCap sees the un-truncated body. The three scrub regexes are linear at
any bounded length, so the larger bound stays ReDoS-safe.
- scrub.test.ts: asserts the default param still applies the 2KB guard AND an
explicit larger cap does not truncate, proving both contexts.
The redesigned scrub regexes are linear, but the previous hard <50ms bound
flaked under JIT warm-up and shared-runner load (observed 77ms on a loaded
machine). Add a timeScrub helper that discards a warm-up call before timing
and assert against a 500ms ReDoS ceiling — wide enough to be jitter-immune,
still ~3x under the legacy ~1.4s catastrophic-backtracking blowup it guards
against.
applyByteCap's "hard guarantee" docstring was false: it trimmed only the
metadata bag (Steps 1-3), so caller-supplied variable-length STRING fields
that the caller controls — `slug`, `demo`, `parent_span_id`, `trace_id` —
were never trimmed. A 5000-char `demo`/`slug` enqueued ~5552B over the 2048B
default cap while stamped `_truncated:true` as if bounded (spec §7 R5-F3
violation).
Add Step 4: after the metadata bag is exhausted (dropped to {}), if the
envelope is STILL over cap the excess can only be in those four caller string
fields. Clamp them longest-first to a short prefix + `…[clamped]` marker until
under cap. Minted ids (`test_id`, `span_id`) are left intact — they are
fixed-width by construction and already bounded; `trace_id` mirrors `test_id`
(fixed-width UUIDv7) so it is included only for completeness so the
post-condition holds even for a hand-built oversized `trace_id`. Format-
constrained / enum / numeric fields (`boundary`, `layer`, `outcome`, `ts`,
`mono_ns`, `duration_ms`, `schema_version`, 9-key `edge_headers`) are bounded
by construction and never touched. Post-condition: serializedSize(env) <= cap
for ANY realistic envelope.
Fix `_truncated` semantics: stamp it only when trimming ACTUALLY occurred (a
field was modified), per its documented meaning, rather than on bare over-cap
detection. A detection that finds nothing trimmable until Step 4 ends with
Step 4 trimming and stamping.
applyByteCap is private; its sole caller is buildEnvelope (which both emit()
and flush() route through). Pure instrumentation preserved — never throws.
Adds a durable bytecap-invariant matrix test over pathological shapes (huge
slug/demo/parent_span_id, huge nested + scalar-heavy metadata, default + debug
tiers) asserting serializedSize(env) <= BYTE_CAP_BY_TIER[tier] for every case.
The §6 PII guarantee was incomplete in three ways that three CR rounds kept
re-finding in the same class. This completes the scrub.
1. Nested-value scrub. `validateMetadata` scrubbed ONLY top-level string
values; secrets buried in allow-listed array/object values (e.g.
`backend.error.caught.stack_brief`, `aimock.match.decision.reject_reasons`)
leaked. New `scrubDeep(value)` in scrub.ts (leaf module; called from
schema.ts which already imports from ./scrub.js — no cycle) applies
`scrubSecrets` to every string LEAF at any depth, preserving structure and
non-string leaves. validateMetadata deep-scrubs object/array values on a
`structuredClone` (cycle-preserving; no caller mutation), falling back to
in-place scrub on the rare unclonable input (correctness > no-mutation).
Walker cycle-safety: ITERATIVE (explicit work stack, never the call stack)
with a WeakSet visited-guard — a self-referential/cyclic or 10k-deep object
is visited once and never re-entered, so no stack-overflow/hang on untrusted
metadata. Only plain objects (Object.prototype/null proto) and arrays are
descended; Date/RegExp/etc. are left as-is.
2. base64url key alphabet. SK_KEY_REGEX
`/sk-(?:[A-Za-z0-9_-]*[A-Za-z0-9]{12,})[A-Za-z0-9_-]*/g` redacts modern keys
whose body uses base64url incl. `_`/`-` (sk-ant-api03-AbCd_Ef-…, sk-proj-…),
with a ≥12-char contiguous-alnum entropy gate (shortest real tail
`0123456789xyzAB` = 15) so prose ("ask-me-later" → me/later; "task_list_items"
uses `sk_` not `sk-`) is NOT redacted. Legacy `sk-<16+ alnum>` still matches.
3. multi-@ URL userinfo. URL_USERINFO_REGEX `/([a-z][a-z0-9+.-]*:\/\/)[^/\s]*@/gi`
redacts the FULL authority up to the LAST `@` before the host/path; excluding
`/` keeps it from crossing into the path. Fixes `https://a@b@c.com/x` leaking
`b@c.com`; user:pass@ and bare-token tok@ still work.
scrubSecrets signature + SCRUB_REPLACEMENT unchanged. Call sites of the deep
scrub: schema.ts `validateMetadata` (object/array branch) → scrubDeep; emit.ts
buildEnvelope routes data-plane metadata through validateMetadata unchanged.
Red→green: scrub-coverage corpus (schema.test.ts) + nested array-of-objects
sweep & cyclic-input safety (emit.test.ts) observed RED (base64url key
unredacted; multi-@ tail leaked; nested secret survived envelope JSON), GREEN
after fix. Full cvdiag suite 119/119 (pb-writer live-PB test skipped sans
PB_BIN). tsc clean; oxfmt + oxlint 0 errors on touched files.
buildEnvelope's accounting branch aliased args.metadata into the
envelope; applyByteCap's in-place trims (Steps 1/2) then mutated the
caller's object as a side effect of pure instrumentation. Shallow-clone
the accounting bag before it enters the envelope (data-plane events
already get a fresh `survivor` from validateMetadata, so only the
accounting branch needed the clone).
applyByteCap Step 3's full-bag size-drop stamped _metadata_dropped, but
that flag is the §6 PII closed-world signal (set by buildEnvelope when
validateMetadata drops unknown keys); overloading it for a SIZE drop
pollutes PB drift queries. A size-drop is already observable via the
_truncated flag stamped earlier, so Step 3 now drops the bag without
touching _metadata_dropped. The genuine PII stamp is unchanged.
Call sites (both private, in-module only):
buildEnvelope: emit() (emit.ts:309), flush() accounting (emit.ts:514).
applyByteCap: buildEnvelope() (emit.ts:388).
No external callers.
The old SK_KEY_REGEX /sk-[A-Za-z0-9]{16,}/ stopped its character class at
the first hyphen, so modern hyphenated-prefix keys leaked into PocketBase:
the `proj` (4) / `ant` (3) / `api03` (5) prefix segments never reached the
required 16-char run, so `sk-proj-…` and `sk-ant-api03-…` never matched and
were stored verbatim — defeating the §6 secret-scrub guarantee for the two
most common real-world key formats.
New pattern: /sk-(?:[A-Za-z0-9]+-)*[A-Za-z0-9]{16,}/g
`sk-` then zero-or-more hyphen-terminated alnum SEGMENTS (the prefix words)
followed by a ≥16-char alnum entropy TAIL.
Why it does not over-redact ordinary hyphenated prose: the mandatory 16+
alphanumeric tail is the entropy gate. Words like "ask-me-later" or
"task-list" have no 16+ contiguous alnum run after an `sk-`, so they never
match (negative test asserts the phrase is returned unchanged). The legacy
`sk-<16+ alnum>` form is preserved as the zero-segment case.
SCRUB_REPLACEMENT and scrubSecrets structure unchanged. Four new tests
(sk-proj-, sk-ant-api03-, legacy sk-<16+>, negative over-redaction);
RED on the two hyphenated cases pre-fix, GREEN after. Full cvdiag suite
110/110.
Move the `pbWriter === undefined` early-return to the TOP of flush(), before
the queue.splice and the drop-accounting block. Previously, with no writer,
flush() spliced the queue into `batch` and discarded it on the trailing
early-return while resetting droppedSinceFlush to 0 — silently losing all
queued telemetry every flush window. Under { autoFlush: true } with no writer
this was continuous data loss, contradicting the documented pbWriter contract
("When absent, events stay queued").
flush() callers: the startBackgroundFlush() interval timer and any manual
flush() call. With a writer present, the A3 drop-accounting + reset-only-
after-landing behavior is unchanged.
flush() previously zeroed droppedSinceFlush BEFORE confirming the accounting
emit() landed, then recovered the just-enqueued event via a fragile second
splice of the whole queue (which APPENDS despite the "prepend" docstring). If
that emit() returned null (validateEnvelope failure / exception) the drop count
AND the cvdiag.queue_dropped record were permanently lost.
Fix: extract the envelope-construction + validateEnvelope + applyByteCap logic
of emit() into a private buildEnvelope() helper (no enqueue side effect). flush()
now calls buildEnvelope() for the accounting event and pushes the result DIRECTLY
into the outgoing batch (no queue round-trip / re-splice). droppedSinceFlush is
reset to 0 ONLY after the accounting envelope is in the batch, so a null/throw
retains the count for the next flush. Docstring updated to "append" (event order
in a best-effort batch is not load-bearing — the classifier re-sorts by mono_ns/ts).
Call sites of the extracted buildEnvelope(): emit() (enqueues result, still
resolves/returns CvdiagEnvelope|null) and flush() (appends to batch, still
resolves/never-rejects). Public method signatures unchanged.
Red-green: src/cvdiag/emit.test.ts. Loss-path test forces the accounting build
to return null and asserts the drop count is retained (lands on the next flush);
RED on original = 0 queue_dropped events land (count silently lost), GREEN after
fix = exactly 1 lands with _dropped_count preserved.
applyByteCap previously stamped _truncated:true and trimmed only >64-char
string values / nested objects in metadata. An envelope over the tier byte
cap due to many short-string or numeric/boolean metadata values (none
reachable by that pass) was stamped truncated yet still enqueued OVER cap,
violating spec §7 R5-F3 ("the whole serialized envelope must fit the tier
byte cap").
Fix escalates while still over cap: (1) legacy >64-char/object trim, then
(2) clamp ALL string values progressively, then (3) drop metadata to {}
(stamping _metadata_dropped). Fixed scalar envelope fields are bounded by
construction, so an empty metadata bag is the hard guarantee the enqueued
row fits any realistic cap. Method stays pure-instrumentation: it never
throws.
Call sites: applyByteCap is private; emit() (emit.ts:361) is the sole caller
and already wraps it in try/catch. Post-condition holds: on return the
serialized envelope is <= cap OR metadata is already {}.
validateMetadata kept surviving metadata VALUES verbatim and never scrubbed
them, so Bearer/sk-/URL-userinfo secrets in free-text fields (e.g.
*.message_scrubbed, probe.*.url) reached PocketBase unscrubbed — violating
spec §6 (the field is literally named message_scrubbed). Now every SURVIVING
string metadata value is run through scrubSecrets; non-string values are left
untouched.
Also tighten URL_USERINFO_REGEX: the old pattern required a user:password@
colon, so a bare-token userinfo (scheme://token@, e.g. https://ghp_xxx@host)
was NOT redacted. The new pattern matches any non-/, non-@, non-ws run before
@, covering BOTH the colon and no-colon forms; the $1[REDACTED]@ replacement
shape is unchanged and the / exclusion confines the match to the authority.
Import-cycle avoidance (approach a): moved scrubSecrets + its regex constants
into a new leaf module scrub.ts that has NO intra-cvdiag imports. Applying the
scrub inside validateMetadata required schema.ts to call scrubSecrets, but
edge-headers.ts already imports EDGE_HEADER_KEYS from schema.ts, so importing
scrubSecrets from edge-headers into schema would form a schema -> edge-headers
-> schema cycle. The leaf module is imported by both; edge-headers re-exports
the symbols so the historical public surface is preserved.
scrubSecrets call-site enumeration (all still resolve):
- src/probes/drivers/d4-chat-roundtrip.ts (imports from ./cvdiag/index.js;
index.ts does `export * from ./edge-headers.js`, which re-exports scrub.ts)
- src/cvdiag/raw-byte-capture.ts (imports from ./edge-headers.js — re-export)
- src/cvdiag/index.ts (`export *` from edge-headers — re-export intact)
Adds an integration-style test file that pins the BOTH layers of the
2026-06-17 Cloudflare-WAF-burst incident fix together:
- the enumerator retries 3× on a 429+Cloudflare-1015 burst before
bubbling (the original bug let one 429 abort the whole enumerate);
- the silence monitor requires THREE consecutive silent evaluation
cycles before posting an alert (the original bug fired on tick #1).
These assertions were the LITERAL red proof on `main`:
- on main both `it` blocks PASSED while asserting the buggy behavior
(calls === 1, posts.length === 1 after a single silent tick),
- on this branch the same gates re-pin the fixed behavior (calls === 4
after retries; posts === [] until the third silent tick).
Run-time output captured for the PR body confirms the inversion.
Layer a per-family consecutive-silent-tick counter ON TOP of the existing
3×period elapsed-time gate (`SILENCE_PERIOD_MULTIPLIER`). The silence
alert now requires BOTH:
- `now - lastSuccessAt > 3 × period` (existing elapsed-time gate), AND
- `SILENCE_CONSECUTIVE_TICK_THRESHOLD = 3` consecutive evaluation cycles
observed silent (new — the counter resets on any successful evaluation).
Without the new gate a single bad cron tick on a family whose
`lastSuccessAt` was already stale (e.g. after a long quiet window or a
deploy gap) tripped the alert immediately — the failure mode the
2026-06-17 Cloudflare-WAF-burst incident exposed where one ~25 min flap
on backboard.railway.com/graphql/v2 paged every family at once.
The counter is NOT incremented during boot grace, so a cold-start cycle
can't alone push it to threshold. The meta-alert (`family-silence-eval`)
path keeps its own clock and is unaffected. Existing tests advance
through three consecutive silent ticks before asserting the post.
Three-retry exponential backoff (1s/4s/16s) on `source.enumerate` against
Railway-GQL when the underlying error is transient (HTTP 429, 5xx, or a
Cloudflare 1015/1020/1022 WAF marker, or a transport-level reject). On
persistent failure, fall back to the last successful catalog from a
per-enumerator in-memory cache — LOUDLY logged via
`fleet.producer.enumerate-failed-using-cache` so the cache-use shows up
in observability.
A fresh-boot process with no cached entry preserves the current
hard-fail behavior (the producer's `enumerate-failed` short-circuit) —
without a catalog there is nothing to enqueue. Real config errors
(`DiscoverySourceAuthError`, non-429 4xx, schema rot) are NOT retried so
operator-actionable failures surface immediately.
Context: 2026-06-17 Cloudflare WAF burst-blocked
backboard.railway.com/graphql/v2 for ~25 min, hard-failing the producer
enumerate on every cron tick and zeroing out D4/D5/D6 writes — the
entire staging dashboard went red within one tick. Retries + cache ride
out the burst on the same tick and preserve job production across
longer outages.
Pre-existing repo-wide lefthook test failures (@copilotkit/core,
@copilotkit/runtime, @copilotkit/shared, etc.) are unrelated to the
harness; verified by stashing my changes and running the same hook on
main with identical failures. Harness suite (131 files / 2812 tests),
typecheck, and build pass on this branch.
- Add control-plane-run.test.ts (17 tests) covering buildLocalServicesJson,
expectedKeys, dedupeScopes, and runViaControlPlane error surfacing
- Export SlugScope, buildLocalServicesJson, expectedKeys, dedupeScopes
for unit-test coverage (factored inline dedup loop into dedupeScopes
helper at the same time)
- runViaControlPlane: surface scopeLabel (demo-aware) in the 0-enqueue
error instead of the bare-slug join, with an empty-targets guard so
the error never renders with a double-space gap
- runViaControlPlane: treat tick.enqueueFailures > 0 as fatal — partial
enqueue used to silently proceed and either mask missing cells or
hang the poll loop to timeout
- Eliminate a stray literal NUL byte in the source by switching the
dedup key separator to a \x00 escape
- lifecycle.up(): name the compose call (infra-up vs target rebuild)
in the health-fail error so an operator can tell which call left a
service unhealthy
(cherry picked from commit 9f35c64adfdf7f5ff2bf0a5ae57ed03818cea607)
CR Finding 1 (BLOCKER): d5-tool-rendering-default-catchall.ts used the
broken page.evaluate(fn, arg) second-arg form to pass the leak-phrase
needle into the browser-side closure. A11 proved empirically that the
arg arrives as undefined inside the closure, making 'if (needle)' guard
the entire leak-detection cascade as dead code — customLeakPhrasePresent
stayed false forever, rendering validateDefaultCatchall's leak branch
dead code as well. Mirrors the A11 fix on the sibling custom-catchall
probe by inlining the needle as a JS string literal inside the closure;
no page.evaluate(fn, arg) dependency at all. Both probes now share the
same inline-needle pattern and keep the canonical literal in lock-step
with their exported phrase constant.
CR Finding 2 (MAJOR): the A11 inline-needle fix on the sibling
custom-catchall probe had no regression test — fake Page.evaluate in
makePageReturning never executes the probe closure, so reverting the
fix would not be caught. Added regression tests that capture the
probe's function source via toString() and assert (a) the canonical
phrase appears as a literal inside the page.evaluate(...) closure and
(b) the closure takes no parameter / the evaluate call has no
second arg. Added the same coverage to default-catchall to protect
the new A25a fix.
CR Finding 3 (MAJOR): A7's requireContentPhrase=true branch in
validateCustomCatchall / assertCustomCatchall had zero coverage —
tests omitted the third arg and exercised only the default false
branch. Added coverage for the true branch (pass on phrase present,
fail on phrase absent, fail on phrase undefined, default-branch
preserved) plus assertCustomCatchall plumbing through the options
form. Also added coverage for default-catchall's customLeakPhrasePresent
branch in validateDefaultCatchall for symmetry.
Local proof:
- RED (fix reverted via git stash): 2 inline-needle regression tests
fail on d5-tool-rendering-default-catchall.test.ts
- GREEN (fix restored): 34/34 tests pass across both files
Out of scope (NOT touched this commit): showcase/harness/src/cli/
control-plane-run.ts and lifecycle.ts (A25b's scope).
(cherry picked from commit c409e3a8d99e16ad0bb05ee3c2e2051e5792049f)
Re-tier the showcase docs tree to be an agent entry point: README.md
opens with a 'when X, see Y' fanout table that routes to the right
procedural doc; each procedural doc gets a one-line tagline answering
'what does this answer'.
Consolidation:
- DELETE showcase/RUNBOOK.md — operational content merged into DEBUGGING.md
(Integration Patterns, Docker Compose Environment, Production Debugging,
Anti-Patterns, Aimock Fixture Deployment, Dev Iteration Speed). The
--isolate mechanics + CLI rules were already duplicated in DEBUGGING.md.
- DELETE showcase/QA-COVERAGE.md — per-demo coverage matrix + starter hero
matrix + probe depth + infra locations + gaps folded into TESTING.md as
the 'Per-Demo Coverage Matrix' section.
Taglines added (no behavioral change to content): TESTING.md, DEBUGGING.md,
GOTCHAS.md, INTEGRATION-CHECKLIST.md, STYLING-GUIDE.md, FRONTEND-STRATEGY.md,
RAILWAY.md, bin/README.md, aimock/README.md, aimock/RAILWAY.md,
harness/README.md, harness/docs/rotation-drill.md.
Cross-link fixups: FRONTEND-STRATEGY.md (was QA-COVERAGE.md →
TESTING.md#per-demo-coverage-matrix), TESTING.md (removed dangling RUNBOOK
companion reference), README.md (rewritten as fanout entry + retained
from-scratch setup + dashboard SOPs below the fanout).
PARITY_NOTES.md × 12 left alone (per-slug context, not redundant).
(cherry picked from commit 75c9d9755c9118c8abc1fa52deda2012b768cab1)
(cherry picked from commit b64189bae0fe2c9e3a5e3ca440013deb4121f23b)
A21 (a9114a831) regression: positional slug after 'up' restricted which services start (only target+depends_on came up; infra profile services never started). With concurrent sibling stack on same host ports, health checks crossed to foreign pocketbase → cell results misrouted → 0.0s red. Fix: split into 2 compose calls — (1) compose infra up -d, then (2) compose --profile slug up -d --build slug. Preserves A21's target-only rebuild + restores full infra stack.
(cherry picked from commit d71ff46edb4a7b0a89f89d8126fd5fe3d5c7d24e)
Root cause
----------
Both the shell `cmd_up` (showcase/bin/showcase line 98) and the harness
auto-start path (showcase/harness/src/cli/lifecycle.ts up() line 268) emit
`docker compose up -d --build` with NO positional service filter after `up`.
Under `docker compose` semantics, an unfiltered `--build` rebuilds EVERY
service in every active profile — for `--isolate` runs that means the
target slug PLUS aimock + pocketbase + dashboard + harness-pool-worker
all rebuild on every isolated test invocation.
Concurrent `--isolate` runs (one per `SHOWCASE_ISO_SLOT=<N>`) contend on
the shared BuildKit instance, serializing what should be parallel work.
This is the BuildKit-contention stall that has been blocking csdkts (A20)
when it runs alongside other concurrent isolated cells: a small slug
rebuild waits on every other slot's full-stack rebuild.
Fix
---
Scope `--build` to the targeted slug(s) by passing them as positional
service args AFTER `up`. Compose then rebuilds ONLY those services and
falls through to cached images for infra services. First-time bootstrap
still works because `--build` only FORCES a rebuild of services with an
existing image — missing images are built automatically by compose.
Two call sites:
- showcase/bin/showcase (user-facing `showcase up <slug>...`)
- showcase/harness/src/cli/lifecycle.ts (npx tsx test path, the one
actually triggered under `--isolate`)
When no slugs are provided (infra-only bring-up), the blanket `--build`
is preserved so first-time infra bootstrap rebuilds whatever is missing.
`--rebuild` regression: the `--rebuild` flag goes through `rebuild()`,
not `up()`, and that path is unchanged — it still force-rebuilds and
force-recreates every targeted slug as before.
Red→Green proof
---------------
3 new vitest cases in showcase/harness/src/cli/lifecycle.test.ts assert
the compose argv shape:
- up(["langgraph-python"]): slug positional after `up` (was: missing)
- up([]): no slug positional (infra-only blanket)
- up(["a","b"]): both slugs as positionals after `up`
Pre-fix: 2 failed | 4 passed (6)
Post-fix: 0 failed | 6 passed (6)
The compose-argv assertion is the deterministic proof of behavior
change. Wall-clock timing improvement on concurrent --isolate runs
follows mechanically from the compose semantics: with the fix, slot
N's BuildKit work no longer blocks slot M's because they only rebuild
their respective slug images, not the shared infra stack.
(cherry picked from commit 8a47dbb0d2a2fe72e21c09a824485e1bb5afc463)
Before A18, `bin/showcase test <slug>:<demo> --d5/--d6` routed through the
control-plane runner but `runViaControlPlane()` collapsed every TestTarget
to its bare slug:
const slugs = [...new Set(targets.map((t) => t.slug))];
`target.demo` was silently dropped, and:
- `buildLocalServicesJson` hardcoded `demos: ["agentic-chat"]` for d5 and
the full demo set for d6 — regardless of what the operator typed.
- `expectedKeys` always emitted the level's DEFAULT-scope key
(`d5:<slug>/agentic-chat` for d5, `d6:<slug>` aggregate for d6).
The net effect on d5: typing `built-in-agent:tool-rendering-custom-catchall
--d5` enqueued the agentic-chat representative, ran agentic-chat in the
worker fleet, and wrote the side row `d5:built-in-agent/agentic-chat` —
green in ~0.0s — while the dashboard reported PASS for a cell the run
never exercised. Validation was dishonest by construction; the CLI claim
of per-demo coverage was substituted with the default representative.
Fix (additive):
- Introduce a per-call SlugScope = { slug, demo? } and thread it through
`buildLocalServicesJson` + `expectedKeys` + `runViaControlPlane`.
- When `target.demo` is set:
* `LOCAL_SERVICES_JSON` synthesizes `demos: [<demo>]` (the worker's
d6 driver reads `input.demos` → `demosToFeatureTypes` → the closed
featureType set, so the matrix narrows to exactly that demo's
featureType(s)).
* `expectedKeys` translates the demo ID into its featureType(s) via
the same `REGISTRY_TO_D5` mapping the driver uses, and waits on
`<level>:<slug>/<featureType>` — NOT the default-scope key. An
unmappable demo throws (would otherwise hang to timeout).
- When `target.demo` is absent: zero behavioral change. d5 still
enqueues the agentic-chat representative and waits on
`d5-single-pill-e2e:<slug>` + `d5:<slug>/agentic-chat`; d6 still
enqueues the full demo set and waits on the `d6:<slug>` aggregate.
- CLI banner now prints the per-scope label (`<slug>:<demo>` qualifier
when present), mirroring the legacy direct path's labeling.
Repro (built-in-agent:tool-rendering-custom-catchall, --d5 --isolate):
BEFORE
Waiting for worker fleet to produce cells:
d5-single-pill-e2e:built-in-agent, d5:built-in-agent/agentic-chat
✓ 2 passed (0.0s) ← agentic-chat representative ran;
catchall never touched. False-positive.
AFTER
Waiting for worker fleet to produce cells:
d5:built-in-agent/tool-rendering-custom-catchall
✗ 1 failed: state=red ← catchall actually ran. Cell is RED at
this HEAD (BIA narration→DOM is a
separate task A12). Validation is now
HONEST.
Regression: bare-slug d5 default UNCHANGED:
built-in-agent --d5 →
d5-single-pill-e2e:built-in-agent, d5:built-in-agent/agentic-chat
✓ 2 passed (0.0s)
Files:
showcase/harness/src/cli/control-plane-run.ts (+96 -17)
- import demosToFeatureTypes
- SlugScope interface
- buildLocalServicesJson: per-slug demo override
- expectedKeys: per-demo featureType-derived keys; refuse-loud
on unmappable demo
- runViaControlPlane: (slug, demo) dedup; scope-aware banner/log
(cherry picked from commit 82ae79279415b7eaf91759df4d8bc1625816a8c3)
Jordan Ritter