## Summary
Makes the showcase harness probe's turn-done signal **reliable**,
killing the dominant class of dashboard false-red flaps without ever
hiding a real failure.
`waitForTurnComplete` previously relied on a fragile SSE fetch-counter
conjunct that false-reds healthy demos whenever the page-side fetch
wrapper missed the runtime URL/transport. This change makes the
**`data-copilot-running` DOM attribute** (driven directly by the agent
run lifecycle, `RUN_STARTED`→true / `RUN_FINISHED`→false,
transport-independent) the **PRIMARY** done-signal, with the SSE counter
demoted to a **headless-only fallback** (headless demos never render
`CopilotChatView`, so the attribute is absent).
Design (all three preserved — no false-green, no false-red, hangs still
red):
- **Primary signal** = the `data-copilot-running` true→false
**transition** with a **stayed-stopped quiescence window** (a stop must
persist on the same run-start count for `settleMs`; a new sub-run resets
it) — so it cannot complete on an intermediate stop in a multi-step
turn.
- **SSE counter** = headless fallback only; never an OR-trigger when the
DOM signal is present.
- **`done-signal-missing` backstop** (gated on `attrPresent===true` +
`runningNow!==true`) reds a genuine painted-but-never-finished DOM turn
before the hard timeout; headless turns use their full timeout for their
only signal.
## How it was reviewed
A full 4-round `cr-loop` (7 unbiased agents/round + confirmation rounds
+ a Procedure-3 promotion audit) caught and fixed **5 distinct
correctness defects** in the implementation before merge:
- **F1** — SSE OR-trigger could complete a multi-step turn early on an
intermediate stop (false-GREEN), in both the loop and the post-loop
classifier.
- **F2** — the run-start baseline was captured *after* the message send,
killing the primary signal on fast turns (false-RED).
- **F3** — non-atomic double `surfaceReady` read per poll (latent hazard
+ wasted round-trip).
- **F4** — the surface-mount (`completeOnMount`) path had no quiescence
window (false-GREEN on intermediate stop + false-RED on a still-running
gen-UI turn).
- **F5** — the early backstop false-redded slow-but-healthy **headless**
turns (now gated on the DOM signal).
Bidirectional red-green tests for F1–F5 plus a systematic `{DOM,
headless} × {completes, lagging-recovers, genuine-hang} × {text,
surface}` completion/backstop matrix. Full harness unit suite: **3173
passed / 18 skipped / 0 failed**; `tsc --noEmit` clean; lint 0 errors;
build clean.
## Known follow-ups (NOT in this PR — pre-existing / non-blocking)
- **Theoretical edge (not reachable on real or realistically-streamed
turns):** if a run completed within a single synchronous microtask
(zero-duration), the page-side MutationObserver could miss the true edge
while `attrPresent===true` → false-red. Real LLM turns and aimock
realistic-streaming hold the attribute true across many event-loop
ticks, so the observer reliably latches it. A naive "re-add SSE fallback
for DOM-present" fix would reintroduce F1's multi-step false-green, so
it's intentionally not done here.
- **Recommended quick follow-up (latency only, no wrong verdict):**
capture `baselineBannerText` pre-`sendTurnMessage` (mirroring the
run-start/count baselines) so a fast-erroring cold-start turn fast-fails
(#5142) instead of burning the full timeout.
- **Pre-existing sse-interceptor capture/counter internals** (none
load-bearing for the new done-signal; verified STAY_IN_C by the
Procedure-3 audit): page-side counter soft-nav/multi-capture reset,
`__hk_fetchWrapped` pattern reuse + hardcoded fallback, g/y-flag
stateful RegExp, TextDecoder end-of-stream flush, bare-catch
reader-error swallow, framenav payload discard/TOCTOU,
CDP-wallTime-vs-Date.now TTFT, addInitScript/close-listener
re-registration accumulation.
## Test plan
- [x] `pnpm test` (harness) — 3173 passed / 18 skipped / 0 failed
- [x] `tsc --noEmit` exit 0, lint 0 errors, build exit 0
- [ ] Verify on staging that auth / prebuilt-sidebar / claude-sdk-tools
(and other previously-flapping cells) stop false-redding while
genuinely-broken cells stay red
Please review the replay/primary-signal approach. Not auto-merging.
The D6 fleet worker drives each integration over its insecure Docker
origin (http://<slug>:10000), where crypto.randomUUID is undefined so
hand-rolled headless chats threw and never mounted (sse-missing). tsx/
esbuild also wraps named inner functions in __name(...) calls that leak
into page.evaluate and throw __name is not defined.
Add installBrowserContextShims (init-scripts.ts): a __name no-op helper
and a crypto.randomUUID secure-context polyfill, registered via
addInitScript at document_start of every D6 page; wired into the
d6-all-pills newPage goto path.
Cover the data-copilot-running turn-done signal in waitForTurnComplete:
true->false transition completion, stayed-stopped quiescence, the
attr-gated early backstop, pre-send run-start baseline, and the
integration wait-for-turn-complete behavior.
Make waitForTurnComplete use the page-side data-copilot-running attribute
as the primary turn-done signal: detect the running true->false transition,
require stayed-stopped quiescence, and gate the early backstop on
attrPresent + runningNow to avoid headless false-RED. Capture a pre-send
run-start baseline so fast turns keep the primary signal alive, read
surfaceReady once per poll, and add computeMaxTurnDurationMs.
Add buildCopilotRunningObserverScript to sse-interceptor.ts and wire it
via addInitScript so the page exposes a data-copilot-running attribute
that the harness can observe for turn-completion signaling.
Adds a durable persistence layer for @copilotkit/bot, replacing the
in-memory-only ActionStore with a pluggable StateStore.
- StateStore interface (kv/list/lock/dedup/queue) with a shared
conformance suite; MemoryStore default plus @copilotkit/bot-store-redis
and @copilotkit/bot-store-postgres backends.
- createBot({ store }): typed per-thread state via Standard Schema,
action snapshots persisted through the store, per-conversation turn
lock (onLockConflict drop|force), and inbound-event dedup keyed on a
stable eventId. ActionStore is kept as a deprecated alias.
- Cross-platform transcripts (bot.transcripts + identity resolver) with
age-bounded retention (prune on append + filter on read), and
runAgent({ transcript: true }) to auto-inject history and capture the
reply.
- createBot({ components }) re-registers components so durable actions
re-fire after a restart; restart-durability demo in examples/slack.
- Dedup is marked seen only after the turn lock is acquired, so a turn
dropped on lock-conflict does not burn its eventId (no lost retries).
- Release lockstep: bot-store-redis/postgres version with bot + bot-ui.
- open a per-feature CvdiagProbeSession for each d5/d6 pill probe
- emit exactly-once probe.exit and failure_classifier per session
- join probe-session output to its run via the X-Test-Id header
- thread cvdiagPbWriter through the orchestrator and CLI runner
Call-Site Enumeration: FAILURE_CLASSIFIER_SET is exported from
cvdiag/probe-session and consumed by d6-all-pills (classifier validation
against the canonical set). The export has no other call sites; any future
classifier addition must update the canonical set in probe-session and the
validation in d6-all-pills together.
Behavior-preserving extraction of the CvdiagProbeSession lifecycle from the
d4 chat-roundtrip driver into a shared cvdiag/probe-session module, so the
d5/d6 probe path can reuse the same session boundaries. d4-chat-roundtrip
now imports the extracted session instead of defining it inline.
schema.json regenerated from the merged canonical schema.ts (failure_classifier
probe.exit additions UNION backend request.ingress/sse.first_byte/llm.call.*
boundaries + test_id adoption). Per-integration staged schema.ts copies
re-derived via 'showcase cvdiag-stage-ts' so codegen --check and stage --check
are both in sync. No hand-merge of generated artifacts.
## What
Adds **cvdiag** — a permanent, always-available observability subsystem
for the showcase, built to diagnose the red↔green cell flap on the
staging dashboard and to make that diagnosis a dashboard query rather
than a multi-day forensic hunt in the future.
Captures the full request path with `X-Test-Id` correlation across
**probe → backend → aimock → edge**, across every integration
(TypeScript, Python, Java/spring-ai, .NET):
- Per-language backend emitters (canonical + staged/compile-linked
mirrors), all sharing one schema (`schema.json`, closed-world
`additionalProperties:false`).
- CREATE-only writes to two new PocketBase collections: `cvdiag_events`
and `cvdiag_raw_byte_samples` (additive migrations — no existing data
touched).
- An 8-class flap classifier mapping to the observed failure signatures
(`sse-missing` / `text-unstable` / `dom-missing`).
- DEBUG-tier raw-byte capture (secret-scrubbed) and HMAC-guarded A/B
edge-interference detection.
## Why
The runId flap-fix (`cdc1e90e`, 2026-06-09) did **not** fully resolve
the flap — it was still observed 2026-06-19. cvdiag exists so the
*remaining* cause is observed live with full correlation instead of
inferred.
## Safety / enablement
- **Inert by default.** With `CVDIAG_BACKEND_EMITTER` unset the
subsystem performs zero host mutation (no logging-config changes, no
threads/tasks, no stdout) — verified by
`test_cvdiag_inert_when_disabled`. **To accumulate data, set
`CVDIAG_BACKEND_EMITTER=1` on the showcase services.**
- All per-language scrubbers match the canonical `scrubSecrets`
(sk-/base64url, Bearer, colon-less URL userinfo, size-guard) — verified
with real toolchains (vitest / mvn / dotnet).
- Merged latest `main` (only conflict: a clean `.csproj` include union).
## Verification
- harness `tsc --noEmit` ✓ · `src/cvdiag` vitest 251/251 ✓ ·
`cvdiag-stage-ts --check` in-sync ✓
- Java MessageScrubber 17/17 (mvn) ✓ · .NET CvdiagBackend 5/5 (dotnet
sdk:9.0) ✓ · Python emitters 93/93 (3.12) ✓
🤖 Generated with [Claude Code](https://claude.com/claude-code)
> **DRAFT / WIP — not reviewed, not ready to merge.** Checkpoint per
request. The mandatory 7-agent cr-loop + CI-green gate runs before this
leaves draft. LGP and ADK ship together in this PR.
## Problem (a false-D6 in both directions)
Declarative A2UI demos wire `a2ui.injectA2UITool: true`, so the response
is a rendered `render_a2ui` surface with **no assistant text bubble**.
The D6 conversation-runner's turn-completion gate required the assistant
**text** to stabilize — so on a working declarative demo the run
finished and the dashboard painted, but text never settled →
`waitForTurnComplete` timed out (`reason=text-unstable`) **before the
render assertion ran**.
Result: `langgraph-python:declarative-gen-ui` (the gold standard)
reported **false-RED while rendering correctly** (all 4 pills verified
live on staging), while `google-adk` reported **false-GREEN**.
## Fix
Opt-in `ConversationTurn.completeOnMount` (set only by
`d5-gen-ui-declarative.ts`). For those turns the text-stability
completion conjunct is **replaced** by a surface-mount predicate:
run-finished (sseOk) + a new assistant bubble + the expected declarative
testids **newly mounting**. A non-rendering surface now yields a new
`surface-missing` failure reason (truthful RED). Text-based demos are
byte-for-byte unchanged (opt-in, per-turn).
## Proof (both directions, live D6)
- RED (before): `text-unstable` timeout; dashboard text painted.
- GREEN (after): passes in ~5s; `buildDeclarativeAssertion` actually
runs and verifies testids mount for all 4 pills.
- INTEGRITY: forced a broken render (renamed testids) → test goes
**red** (`surface-missing`). Not "always green now."
- Unit: 89/89 + 3 new (green-on-mount, red-on-surface-missing).
## Scope: LGP + ADK (ship together) — both truthful GREEN
- **LGP** gold-standard cell: false-RED → truthful GREEN (all 4 pills
assert).
- **ADK** realignment: **test-only, complete.** The shared-script fix
auto-applies; verified all 4 ADK pills truthfully GREEN (each surface
mounts from baseline 0 via surface-mount completion). Prior false-green
closed; **no ADK backend gap**.
## Out of scope (someone else's problem)
This shared-script change re-evaluates **every** declarative-gen-ui cell
truthfully. Integrations beyond LGP/ADK that don't actually render will
flip to **truthful RED** — e.g. `langgraph-typescript` pill 2
(team-performance `declarative-data-table` doesn't mount). Those are
real per-demo render gaps for their owners; **not fixed here.**
## Follow-up (not in this PR)
The `render_a2ui` call returns a ~5.9 MB SSE for a ~2 KB surface (LGT
worse) — a separate runtime amplification concern in the
`injectA2UITool:true` middleware path.
## Before ready/merge
- [x] ADK empirical verdict — all 4 pills truthful GREEN, realignment
test-only, no backend gap
- [ ] mandatory cr-loop → zero findings
- [ ] CI green
Opts each declarative-gen-ui pill into the new `completeOnMount` turn
completion so these surface-rendering demos are gated on their expected
declarative testids mounting rather than assistant-text stability.
Declarative A2UI demos render a surface (mounted testids) with no assistant
text bubble, so the assistant-text-stability completion gate never settled and
timed out on working demos — a false-RED.
This adds an opt-in `completeOnMount` turn-completion path that replaces the
assistant-text-stability conjunct with a surface-mount predicate: run-finished
+ a new assistant bubble + the expected declarative testids newly mounting.
A new `surface-missing` failure reason reports when the run finishes but the
expected surface never mounts. Turns that do not opt in keep the existing
text-stability behavior unchanged.
The ms-agent-harness-dotnet slug was excluded from per-cell D6/BE/smoke
probe enumeration by a placeholder fence added 2026-06-07, before the
real column existed. The column shipped in PR #5569 and its d6/d4 aimock
fixtures landed on main today (e10df0b4), so the fence is now stale.
Remove the slug from all 8 exclude SSOT sites so the column populates.
## Summary
Two follow-up fixes that complete the pinned-prod / floating-staging
contract for the showcase fleet (the contract enforced by the promote
CLI in #5566). Prod services must be digest-pinned (`@sha256`), staging
floats `:latest`.
**Fix 1 — `showcase/scripts/deploy-to-railway.ts`: provision prod
digest-pinned, not `:latest`.**
Prod services were being *born* on the mutable `:latest` tag, then later
pinned only at promote time. Now they are born pinned to a resolved
`@sha256` digest at create time, via a new TS GHCR resolver that mirrors
the Ruby promote CLI (`/token` exchange → manifest HEAD →
`Docker-Content-Digest`). Resolution failure is **fail-loud**
(`process.exit(1)`, never a `:latest` fallback). `goLive` asserts the
prod `source.image` is digest-pinned (`assertProdDigestPinned`,
refactored to be dependency-injectable and to throw a typed
`ProdPinError` instead of exiting inline).
**Fix 2 — `showcase/harness/.../image-drift.ts`: stop flagging pinned
prod red.**
Under the pinned-prod contract, prod is intentionally digest-pinned
behind `:latest`, so the image-drift probe was firing false-red on every
prod service. It now renders such prod services **green**
(`pinnedExpected`) while a genuinely missing digest stays **red**.
Staging behaviour is unchanged.
## Verification
- **Red-green proven locally** for both new test surfaces:
- `assertProdDigestPinned` guard: RED = 5 tests `assertProdDigestPinned
is not a function` (untestable inline-exit) → GREEN = 13 passed after DI
refactor; env-mismatch branch: RED = `promise resolved undefined instead
of rejecting` → GREEN after wiring the test's env control.
- Suites: `deploy-to-railway.digest-pin.test.ts` **14 passed**;
`image-drift.test.ts` **26 passed**. Typecheck (scripts + harness) 0
errors; oxfmt + oxlint clean; harness build green.
- **Empirical 6b against live prod** (settled two reviewer
masking-concerns as can't-happen under current config):
- prod harness `SHOWCASE_ENV` is *unset* (not `""`) and
`RAILWAY_ENVIRONMENT_NAME="production"`, so `isProductionEnv()`'s `??`
correctly resolves true → the prod-neutral fix **fires** in prod.
- prod image-drift is discovery-only; all 19 prod `showcase-*` services
are digest-pinned and tag-less, so `expectedTag` resolves to `latest`
for every prod service → a fixed tag cannot reach prod image-drift (no
false-green).
## Review
3-round cr-loop (7 agents/round) converged to zero actionable findings.
One fix-introduced test-scaffolding defect (dead `envId` helper option)
was caught in the confirmation round and fixed. Remaining reviewer notes
are pre-existing issues in untouched goLive/probe code or
theoretical-but-can't-happen-under-live-config items, tracked as
follow-ups (not in scope for this PR).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- remove the retired top-level `docs/` Next app and the disabled
docs-sync workflow/script
- add `docs -> showcase/shell-docs` as a top-level symlink for `cd docs`
muscle memory without restoring the old docs tree
- move the docs model allowlist into `showcase/shell-docs/` and retarget
docs validation/doctest extraction to shell-docs content
- update docs/agent guidance and CI path filters so `docs/` is treated
as an alias, not an active separate docs surface
- tighten the pre-commit package check so non-package docs/tooling
changes do not fan out into the full package matrix
## Validation
- `pnpm exec tsx scripts/validate-doc-model-names.ts`
- `pnpm exec tsx scripts/doc-tests/extract.ts`
- `pnpm exec vitest run
scripts/__tests__/validate-doc-model-names.test.ts
scripts/doc-tests/__tests__/extract.test.ts
showcase/harness/src/cli/eval/scope.test.ts`
- `pnpm exec oxlint showcase/harness/src/cli/eval/scope.test.ts
scripts/doc-tests/extract.ts`
- `git diff --cached --check` before follow-up commit
- `test "$(readlink docs)" = "showcase/shell-docs"`
- `test -f docs/package.json`
- `pnpm exec oxfmt --check .claude/docs/documentation.md
.claude/docs/hooks.md AGENTS.md CLAUDE.md CONTRIBUTING.md
showcase/shell-docs/README.md`
- commit hooks passed
## Notes
- historical docs remain recoverable from
`archive/docs-save-do-not-prune` and `archive/docs-retired-2026-06-17`
- I intentionally left Vercel/project teardown out of this PR; this is
repo cleanup only
Under the pinned-prod contract, prod is intentionally digest-pinned behind
:latest. image-drift now renders such prod services green (pinnedExpected)
instead of red; a genuinely missing digest stays red. Staging unchanged.
Jordan Ritter