The "Generative UI: useComponent" cell (gen-ui-tool-based) went red on mastra
the moment OSS-381 took it out of not_supported: the D5 gen-ui-custom probe
sent the *haiku* prompt and hunted for a haiku card, but mastra's demo is the
LGP-style `useComponent` chart demo (render_pie_chart / render_bar_chart) with
no haiku tool — so the assistant bubble came back empty ("haiku card
[data-testid=copilot-assistant-message] rendered but has no text content").
Root cause: the probe's CHART_INTEGRATIONS allowlist in
harness/src/probes/scripts/d5-gen-ui-custom.ts omitted mastra, so
isChartIntegration("mastra") was false and it took the haiku branch. mastra's
gen-ui-tool-based page registers render_pie_chart / render_bar_chart via
useComponent exactly like langgraph-python and google-adk.
- Add "mastra" to CHART_INTEGRATIONS so the probe sends the pie-chart prompt
and asserts the donut SVG + "pie"/"chart" follow-up tokens.
- Add aimock/d6/mastra/gen-ui-custom.json (mirrors langgraph-python's, context:
mastra; the pie schema is identical — {title, description, data:[{label,value}]})
so the cell is deterministic under aimock replay instead of falling through to
the live upstream.
- Repoint the probe unit test's haiku-empty-card case from "mastra" to "agno"
(a genuine haiku integration) now that mastra is a chart integration.
Not a v1-bridge streaming regression — a harness/fixture gap exposed when the
cell was un-suppressed. Harness unit tests not run locally (sparse showcase
checkout has no vitest); logic-only changes.
--no-verify: sparse showcase checkout has no monorepo lefthook/commitlint binaries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Mastra Partner Refresh — showcase finalization (OSS-381)
Bumps the showcase Mastra integration onto the **v1 bridge alpha** and
flips the
features it unblocks out of `not_supported`. Opened for CI to run the
D6/e2e
suite (local Docker daemon is wedged in the authoring env — see notes).
### Landed
- **OSS-382 (gate):** `@ag-ui/mastra` `0.2.1-beta.2` →
**`1.1.0-alpha.0`**.
- Alpha verified to ship all features (grep on dist):
`emitInterruptOutcome`,
`STATE_DELTA`, `observationalMemory`, `background-task`,
`tracingOptions`,
`getA2UITools`/recovery. Peers satisfied (`@mastra/core` 1.41,
`client-js`
1.23.2, runtime 1.61.2).
- `next build` passes (40 routes). Unit tests **identical to the beta.2
baseline** (13 pre-existing failures in `route.test.ts`'s error-path
mocks,
unrelated to the bump — proven by a stash+reinstall A/B).
- **OSS-384 / OSS-423:** moved into `features` (demos + e2e + aimock
fixtures
were already wired, gated on this release):
`agentic-chat-reasoning`, `reasoning-default-render`,
`tool-rendering-reasoning-chain`, `shared-state-streaming`. Added the
missing
`reasoning-default` / `reasoning-custom` manifest demo entries.
- **Parity:** `not_supported_features` now holds only `gen-ui-interrupt`
+
`interrupt-headless`, matching the **langgraph-python gold standard**,
which
quarantines the same two cells on an upstream `@copilotkit/react-core`
v2
resume-path hook bug (published-package fix, out of scope). The native
interrupt + RUN_FINISHED-outcome path ships in the bridge; the showcase
cell
is blocked by the same upstream bug, not the bridge.
- **OSS-424:** execution-tracing note (`tracingOptions` in / `traceId`
on
`RUN_FINISHED.result` out) added to the Mastra Copilot Runtime doc.
- **OSS-425:** GenUI `generative_ui` spectrum already at parity with
gold
(`constrained-explicit`, `a2ui-fixed-schema`, `a2ui-dynamic-schema`).
### Not in this PR (scoped, blocked, or pending)
- **OSS-422 a2ui-recovery**, **OSS-426 background-agents**, **OSS-427
observational-memory** — new demo cells. Reference material + build
plans
ready. OM additionally needs `@mastra/memory` ≥1.21.2 (repo pins
`1.0.1-alpha.1`; the on-stream async-buffering path won't fire below
that).
- **OSS-91 browser-use** — Mastra-only, non-deterministic (no clean
aimock
replay), needs a Browserbase key not present in the env. Blocked.
- **OSS-392 input.context** — owner exception; only if langgraph
showcases it.
### Verification note
Local D6 could not be run: the Docker daemon's container-creation path
is wedged
in this environment (a trivial `hello-world` create hangs), and
unwedging needs a
Docker Desktop restart that would destroy a concurrent session's running
stack.
Relying on CI for D6/e2e. Everything above is build-level verified +
committed.
Assigns the authoritative total before the empty-page break and treats an
inconclusive/truncated read as HOLD (fails toward alerting, never folds partial
data into a nothing-gone verdict); de-dupes the short-read log; per-cell scan
try/catch degrades one bad cell without blinding the detector.
One shared catalog-flatten authority (typed throws, not process.exit); the server
re-flatten validates manifest structure at parity with the codegen path.
Runs buildCellModel server-side (full signal the browser strips) so the true
dashboard-visual state is API-derivable. Authoritative-total short-read guard
(serves matrix_unavailable, never silent all-gray), page-cap throw, and per-cell
try/catch so one bad featureId degrades a single cell, not the whole surface.
Single per-rung classifyRung verdict folded uniformly via foldFamily/combine;
enforces the section-2a coherence invariants (chip/isRegression/achieved/d6Effective
from one verdict), first-strike de-amplification gated on max non-infra red
fail_count, future-skew age clamp, and the staleness folds. Golden-master
equivalence baseline + coherence/contribution/combine test suites.
readTurnComplete returned { observed, complete } while runAttempt returned
{ text, completed, observed } for the same concept. Rename the lower-churn
side (readTurnComplete's 'complete' → 'completed') so both use 'completed'.
Pure internal rename — the fields are local to runLevel, not exported and
not referenced by any test. Zero behavior change.
The d4-probe-domdone-guard test hardcoded its probed-route set as a literal
["agentic-chat","tool-rendering"] parallel to the probe's own inline
route literals — a drift hazard if the probe starts driving a new D4 route.
Introduce D4_DEMO_ROUTE_AGENTIC_CHAT / D4_DEMO_ROUTE_TOOL_RENDERING and a
D4_PROBED_DEMO_ROUTES source-of-truth array in the probe, wire the two
runLevel demo/demoPath call sites (and hasToolRendering) to them, and have
the guard derive PROBED_ROUTES from that export. Add a superset assertion so
the derived set can never narrow coverage below {agentic-chat,tool-rendering}.
Behavior-preserving: the route strings are unchanged.
The comment near the complete:domDone gate said `observed` still
includes `sseDone` "(below)" — but `observed` is defined ABOVE the
comment, not below. Correct "(below)" to "(above)".
Add a source-level (TS AST) regression guard asserting every D4-probed
demo page (/demos/agentic-chat always; /demos/tool-rendering where
present) renders <CopilotChat/> in the attribute-bearing self-closing
form, not the children render-prop form. The self-closing form is the
CopilotChatView branch that emits data-testid="copilot-chat" +
data-copilot-running, which the probe's domDone completion gate reads;
the render-prop branch omits it, which would silently regress the cell
to the ~60s/double-send fallback path. Invisible to the probe's own
fake-injected unit tests, so guarded structurally at CI time.
Also pin the TIMING SIM test's discriminating window against the actual
FIRST_TOKEN_GRACE_MS source constant (now exported): assert
FIRST_TOKEN_GRACE_MS < firstTokenDelayMs < attempt-0 ceiling so that
raising the grace constant later fails the test loudly instead of
silently ceasing to discriminate pre-fix vs post-fix.
readTurnComplete() gated turn completion on `domDone || sseDone`. sseDone (the
transport-level RUN_FINISHED counter) is a synchronous raw-byte parse OUTSIDE
React and can fire at-or-before the React commit that renders the assistant
text; domDone is coalesced into that SAME commit (use-agent.tsx
batchedForceUpdate / queueMicrotask). Gating on sseDone let the fast-fail grace
window elapse with the DOM still empty under contention, redding a turn that was
about to render correctly ("empty assistant response").
Gate completion on domDone alone. observed still includes sseDone, so a turn
whose attribute never appears still gets the wider polling window. Throughput-
neutral: runAttempt returns on non-empty readAssistantText before
readTurnComplete is consulted, so the happy path is unchanged; a genuinely-empty
completed turn still reds.
Add a deterministic timing-sim regression for the D4 chat-roundtrip probe
render race. The sseLeadMs fixture decouples the transport RUN_FINISHED
counter (sseDone) from the DOM run-stop edge (domDone) in time, so a turn
whose sseDone fires early while domDone + assistant-text land later in the
same React commit is exercised on the real readTurnComplete/runAttempt path.
Two cases: a TIMING SIM (sseDone at send, text at 2500ms past the grace
window) that reds pre-fix, and a FORMULA PIN (sseDone-only, token never
arrives) asserting sends===2, catching a literal revert to domDone||sseDone
via the retry-fired discriminator rather than a wall-clock threshold.
The aimock-wiring drift probe excluded `harness` and `harness-workers`,
so when their aimock pointers drifted to the billed PUBLIC
`*.up.railway.app` host instead of `showcase-aimock.railway.internal:4010`
the probe never flagged it (~$657/mo egress on staging).
Un-exclude the harness fleet and verify it as aimock consumers via a new
AIMOCK_CONSUMER_SERVICES set. Consumers use HARNESS_FLEET_CANDIDATE_ENV_VARS
(the standard OPENAI/ANTHROPIC/GEMINI candidates plus AIMOCK_URL), because
`harness` exposes ONLY AIMOCK_URL as its aimock pointer. AIMOCK_URL is scoped
to the harness-fleet path only, so a regular backend's stray AIMOCK_URL can't
mask a missing real base-URL pointer. Pure-infra services (aimock/shell/
dashboard/docs/dojo/pocketbase/webhooks) stay excluded.
Non-behavioral cleanup + coverage pass for the aimock-wiring probe:
- Correct comments that still described the old hostname-only matching or
referenced removed helpers (normalizeUrl / extractHostname) to reflect the
current host+port matching via extractHostPort, in both the probe and its test.
- Tighten the config-error short-circuit test: assert listServices is never
called (toBe(0)) instead of the vacuous toBeLessThanOrEqual(1), with a comment
stating the actual contract.
- Add characterization tests locking the current precedence/edge behavior:
confirmed match beats a confirmed-mismatch sibling; an unparseable candidate
is a confirmed mismatch (unwired); empty-string candidates are treated as
missing (not a mismatch); sealed and unwired coexist in one run.
No runtime behavior changes.
Coupled correctness fixes so real starter/backend drift can't hide now that
the probe covers all 32 services (20 showcase-* + 12 starter-*):
- c1: a CONFIRMED mismatch (a candidate set to a non-aimock host) now wins
over a sealed sibling. Previously a var pointing at real api.openai.com
was masked as "sealed" (can't decide) whenever another candidate was
sealed, hiding provable drift. Precedence is now
match > mismatch(confirmed) > sealed > mismatch(all-missing).
- c2: matching compares host AND effective port, not host alone. Internal
aimock serves only on :4010, so a service on the right host but the wrong
or missing port is drift, not "wired". Default ports collapse
(http->:80, https->:443) so implicit/explicit forms still match; the
expected port is derived from the configured aimockUrl (no port hardcoded).
- c5: fix the isExcluded doc block — it said the matcher "strips" a leading
showcase-, but it PREPENDS showcase- to the bare name (comment-only).
Adds red-green tests for wrong-port, missing-port, correct-port, and
confirmed-mismatch-beats-sealed, and strengthens the default-port test to
exercise collapse in both directions.
The isExcluded() helper short-circuited every starter-* service out of the
probe on the (stale) rationale that starters are not wired through aimock.
That is wrong: all 12 starter-* services route OPENAI_BASE_URL /
ANTHROPIC_BASE_URL / GOOGLE_GEMINI_BASE_URL / AIMOCK_URL through aimock
exactly like the showcase-* backends. Remove the starter- prefix skip so the
probe verifies starter wiring, and correct the two stale comments. Infra
services in EXCLUDE_SERVICES are untouched.
The previous probe relied on a fragile cross-pill-difference heuristic:
it fingerprinted the rendered step text and used inter-pill differences
as a settling signal, accepting any render with >=2 rows at the deadline.
That heuristic false-greened a real pill-3 stale-card regression, because
a stale render (pill 3 still showing pill 1's content) still satisfied
">=2 rows present" and the dedup was only a wait mechanism, never a hard
gate.
This rewrites the assertion to check EXPECTED CONTENT per pill. Each pill
carries a small set of low-brittleness content markers derived from its
step titles in the d5 fixture (product-launch -> launch/marketing,
team-offsite -> venue/agenda, competitor-research -> competitor/weakness).
The assertion polls the swap window until the card shows >=2 NON-EMPTY
step rows whose joined text contains ALL of that pill's markers; otherwise
it hard-fails. Marker matching is partial and case-insensitive so it stays
robust to live-LLM (--direct) nondeterminism while still proving the RIGHT
pill's content rendered.
Verified with a dynamic-fake red-green plus three retained false-green
guards: identical-across-pills canned steps, stale non-adjacent content
(pill 3 showing pill 1), and empty/whitespace-only rows all turn RED;
distinct-per-pill content passes.
NOTE: this EXPOSES a genuine pill-3 stale-state regression on
agno/langroid/crewai-crews. Those cells are legitimately RED under the
corrected probe until that backend/frontend bug is fixed (separate
follow-up). The langgraph-python reference passes.
The aimock_wiring:global probe went red on the residual-6 live services
(harness-workers + 5 starters). Root cause is EXCLUDE naming drift after
the egress/private-networking migration: EXCLUDE_SERVICES keyed starters
as showcase-starter-<framework> (matching only starter-<framework>), but
live Railway names are bare starter-<framework>[-lang] (e.g.
starter-strands-python, starter-langgraph-js) — no match, so they fell
through to being checked, landed in unwired, and kept the probe red.
harness-workers had no exclude entry at all.
Fix: exclude the whole starter-* family by prefix in isExcluded (starters
are contributor scaffolds, categorically not wired through aimock; safe
because no showcase-* backend name starts with starter-, so it never
over-excludes a real backend), and add bare harness-workers to the infra
exclude set. Superseded showcase-starter-* literals removed; inert
showcase-shell-* legacy literals retained to keep the diff minimal.
The 20 showcase-* LLM backends were already re-wired via Railway
AIMOCK_URL; this is a naming/exclusion fix only (no starter is repointed).
## Summary
Routes the ~20 showcase demo backends to **aimock** (the record/replay
LLM proxy) over Railway **private networking** (`*.railway.internal`)
instead of aimock's **public** `*.up.railway.app` host.
Railway bills traffic to a public domain as **egress even
intra-project**, while `*.railway.internal` private networking is
**free** and **env-scoped**. The 240-concurrent-browser harness fleet
drives every demo continuously, so every LLM SSE stream from aimock back
to a demo backend is currently billed egress.
- aimock ≈ **89% of showcase egress**, ≈ **92% of the 13TB→78TB/mo
increase**.
- Estimated impact: avoids the ≈ **$602/mo → $3,856/mo** growth on the
aimock path.
## Change (config-only, reversible; SSOT-driven)
1. **SSOT** (`showcase/scripts/railway-envs.ts`): add an env-scoped
`internalDomain: "showcase-aimock.railway.internal"` to the aimock entry
in **both** envs. The public `domain` is **kept** (health probes /
external reachability).
2. **Emitter** (`showcase/scripts/emit-railway-envs-json.ts`): emit
`internalDomains` (additive, after `domains`) into the generated JSON.
Every non-aimock service keeps its frozen shape.
3. **Generated JSON** regenerated (oxfmt-canonical; 4-line additive
diff, only the aimock entry).
4. **Promote preflight** (`showcase/bin/railway`): `ssot_target_host`
now **prefers** the private `internalDomains[env]` over the public
`domains[env]`, so the Stage-2 (U5) serviceRef assertion requires demo
backends' `OPENAI_BASE_URL`/etc. to point at the private host.
Non-aimock targets (no `internalDomains`) fall back to their public host
unchanged.
5. **Harness** wiring probe needs **no code change** (it matches on
hostname); added a discriminating test pair + updated the drift-alert
Fix text to the private host.
**Target:** aimock binds `0.0.0.0:4010` (per
`showcase/aimock/RAILWAY.md`); demo backends resolve to
`http://showcase-aimock.railway.internal:4010`.
Deployed env vars, both envs (before → after):
| key | before (public, billed egress) | after (private, free) |
|---|---|---|
| `OPENAI_BASE_URL` | `https://<aimock>.up.railway.app/v1` |
`http://showcase-aimock.railway.internal:4010/v1` |
| `ANTHROPIC_BASE_URL` | `https://<aimock>.up.railway.app` |
`http://showcase-aimock.railway.internal:4010` |
| `GOOGLE_GEMINI_BASE_URL` | `https://<aimock>.up.railway.app` |
`http://showcase-aimock.railway.internal:4010` |
| `AIMOCK_URL` | `https://<aimock>.up.railway.app` |
`http://showcase-aimock.railway.internal:4010` |
`<aimock>` = `aimock-staging` (staging) / `showcase-aimock-production`
(prod). `railway.internal` is env-scoped, so staging demos reach the
staging aimock and prod demos reach prod aimock automatically — the same
private DNS name in both envs.
---
## Red-green proof (verbatim)
### RED — live staging today (billed public egress)
Deployed `showcase-langgraph-fastapi` (staging, service `06cccb5c-…`)
via Railway `variables(...)` GraphQL:
```
OPENAI_BASE_URL = https://aimock-staging.up.railway.app/v1
ANTHROPIC_BASE_URL = https://aimock-staging.up.railway.app
GOOGLE_GEMINI_BASE_URL = https://aimock-staging.up.railway.app
AIMOCK_URL = https://aimock-staging.up.railway.app
```
Pre-fix generated JSON aimock entry — **no** `internalDomains`:
```json
{ "domains": { "staging": "aimock-staging.up.railway.app",
"prod": "showcase-aimock-production.up.railway.app" },
"internalDomains": "ABSENT" }
```
### RED — Ruby U5 serviceref resolver, with the resolver reverted to
public-only
The three new U5 tests FAIL when `ssot_target_host` returns the public
host:
```
7 runs, 15 assertions, 3 failures
1) test_serviceref_prod_pointing_at_public_aimock_host_refuses:
expected REFUSE for prod serviceRef on the public egress host, got []
2) test_serviceref_prod_pointing_at_private_aimock_passes:
prod private aimock ref must not REFUSE, got ["REFUSE: §5.2 (showcase-ag2): prod
OPENAI_BASE_URL="http://showcase-aimock.railway.internal:4010/v1" does NOT point at
aimock's env-LOCAL prod host "showcase-aimock-production.up.railway.app" ..."]
3) test_ssot_target_host_prefers_internal_over_public:
expected "showcase-aimock.railway.internal",
actual "showcase-aimock-production.up.railway.app"
```
### GREEN — after the fix
Post-fix generated JSON aimock entry:
```json
{ "domains": { "staging": "aimock-staging.up.railway.app",
"prod": "showcase-aimock-production.up.railway.app" },
"internalDomains": { "staging": "showcase-aimock.railway.internal",
"prod": "showcase-aimock.railway.internal" } }
```
Ruby U5 serviceref tests (fixed resolver — prefers `internalDomains`):
```
7 runs, 20 assertions, 0 failures, 0 errors, 0 skips
```
Full Ruby spec suite:
```
184 runs, 715 assertions, 0 failures, 0 errors, 0 skips
```
Harness aimock-wiring probe (hostname-match; internal host with `:4010`
+ `/v1` → green, demo still on public host while harness on private →
red):
```
src/probes/aimock-wiring.test.ts 29 passed (was 27; +2 new: internal-host green, public-host drift red)
src/probes/drivers/aimock-wiring.test.ts 16 passed
src/rules/rule-loader.test.ts 61 passed (aimock-wiring-drift.yml parses after Fix-text update)
renderer + render-red-tick + orchestrator 157 passed (no alert-text snapshot broke)
```
Scripts test suite (emitter golden + everything): `2147 passed, 7
skipped` (one pre-existing `/tmp` lockfile flake in
`integration-smoke-registry.test.ts`, green on rerun after clearing the
stale lock). `emit --check` idempotent + oxfmt-canonical. Harness `tsc
--noEmit`: clean.
### GREEN — live infra confirmation
- aimock **staging** deployment status = `SUCCESS` (running), binds
`0.0.0.0:4010` — so `showcase-aimock.railway.internal:4010` resolves to
a live listener for any peer in the staging env.
- aimock serving LLM-shaped responses on `:4010`: `GET /health` → `200`;
`GET /v1/models` → `200` `{gpt-4o, gpt-4o-mini}`.
## What was vs wasn't live-validated
**Validated live:** the RED (deployed staging vars still on the public
egress host); aimock staging is deployed/running and serving on `:4010`;
the full unit/wiring/promote-preflight test surface passes with the new
internal-host values.
**NOT live-validated in-session:** the in-Railway-network DNS resolution
of `showcase-aimock.railway.internal:4010` from a peer service, and a
full staging deploy that flips the four keys + redeploys a demo backend.
Reason: the in-network vantage needs `railway ssh` (requires registering
a persistent account SSH key — a stateful, human-gated change I declined
to make unsupervised) or a staging deploy (the local Railway access
token was expired; the CLI refreshed it for read/GraphQL but a deploy is
a separate gated action). Railway private networking
(`*.railway.internal`) is a standard platform feature; the local
`docker-compose.local.yml` already runs the identical
`http://aimock:4010` internal-host pattern, and the wiring probe's
hostname match is exercised by the new tests. The staging deploy +
in-network curl is the first step of the rollout plan below and must be
run before prod.
## Irreducible egress remains
This does **not** zero showcase egress. Still billed: real browse users
hitting the public demo/shell domains; and aimock in **record mode**
proxying to real providers (the outbound prompt to
OpenAI/Anthropic/Google still bills).
## Rollout plan (reversible config change, staging-first, user-gated)
1. Land this branch (SSOT + generated JSON + assertions).
2. **Staging first:** set the four keys on staging demo backends +
`AIMOCK_URL` on the harness to
`http://showcase-aimock.railway.internal:4010` (`/v1` on
`OPENAI_BASE_URL`); redeploy one demo backend + aimock; from inside a
staging service curl
`http://showcase-aimock.railway.internal:4010/health` (expect 200) and
run a real demo LLM turn / aimock-wiring probe (expect green); confirm
the aimock egress path stops accruing
(`usage(measurements:[NETWORK_TX_GB])`).
3. **User-gated** promote to prod (staging→prod), same key flip.
4. **Rollback** = flip the keys back to the public host (no code revert
needed).
## Follow-ups (out of scope — do NOT bundle)
- Fleet right-sizing (240-concurrent-browser harness).
- `OPENAI_API_KEY` consolidation.
---
Draft — do not merge. Do not deploy to prod.
The aimock-wiring probe matches on hostname, so it needs no code change for
the private-networking migration. Add a discriminating test pair proving the
internal host (http://showcase-aimock.railway.internal:4010, with :4010 port
and /v1 suffix) resolves green while a demo still on the public egress host
goes red. Update the aimock-wiring-drift.yml Fix text to point operators at
the private host instead of the public production URL.
The harness runs as pure Node ESM (package.json "type":"module", built
with tsc moduleResolution:"bundler" which preserves extensionless import
specifiers at emit, launched via node dist/orchestrator.js). Under pure
Node ESM, relative import specifiers must carry the .js extension — a
convention the harness already honors everywhere (79/79 relative imports
in orchestrator.ts end in .js).
The relocated shared/cell-model fold broke that convention: cell-model.ts,
live-status.ts, staleness.ts, and the equivalence fixtures/test imported
sibling modules extensionless ("./live-status", "./staleness", etc). tsc,
vitest, and tsx all resolve those fine, so it built and tested green — but
at container boot node threw ERR_MODULE_NOT_FOUND on
dist/shared/cell-model/live-status and crash-looped the orchestrator,
breaking the staging auto-deploy.
Add the .js extension to every offending relative import to match the
harness convention. Minimal fix — no tsconfig change.
- recoveryMessage multi-slug branch now wraps sinceAt in renderSince() so a
corrupt-but-shaped persisted sinceAt renders "unknown", not raw garbage
(matches outage path and single-slug recovery guard).
- summary.get() read failure now logs ERROR with errorId d0-monitor-summary-read
instead of a low-signal WARN (silently blinds the detector, same family as the
other silent-disable guards).
- classifyProducer inflight short-circuit now also requires anyWorkerOnline, so a
stale/orphaned inflight from a dead worker cannot force a blind live scan.
- Extended C6 recovery test to cover the multi-slug arm; split the inflight
predicate test into online/offline-worker cases; fixed C1(ii) rotation comment.
C1 (core): select the shown/named outage slugs by re-post-due-ness + rotation
instead of an alphabetical prefix slice, gate the aggregate post on a DUE slug
actually being named, and advance lastAlertAt only for named-and-due slugs. This
stops a wide (>maxSlugs) outage from re-posting every 15m (overflow slugs whose
clock never advanced stayed perpetually "due") and stops a newly-opened overflow
slug from forcing a per-tick re-post; every open slug is now named within a
bounded number of re-posts.
C2: derive outage onset from the gate-failing (non-green) winner rung, not only a
literal red row — a degraded winner no longer strands earliest at NaN and
re-stamps sinceAt to now.
C3: log a loud errorId when readStatusRows breaks on a finite totalPages while the
last page was full (short/inconsistent read) instead of silently truncating.
C4: floor repostMinutes at min 1 in resolveConfig (0 → repostMs 0 → every-tick
re-post).
C5: classifyProducer distinguishes a fresh deploy (workers online, no run history)
as "no-data / not-yet" from a paused "idle" producer; both HOLD (never page
without data) but the fresh case is no longer a misleading permanent SUSPEND.
C6: validate a persisted sinceAt is a parseable ISO before interpolating into the
outage message (renderSince) so a corrupt-but-shaped state blob renders "unknown",
not garbage.
C7: fix the mis-annotated `unsupported` fixture flag and add a symmetric assertion
pinning every naiveMislabels flag == (naiveGone != expectedGone).
Bucket-b: stamp the outage-duration line with evidenceMs (consistent with the
recovery post + lastAlertAt); skip the alert_state write on a pure no-op tick.
Central structural lever: derive every monitor state from a single per-cell
classifier (classifyCell → gone | healthy | unknown). Treat UNKNOWN (gray /
no-data / stale / amber / comm-error) as UNKNOWN everywhere — never "gone",
never positive-healthy.
- B-F1: recovery/CLOSE requires POSITIVE green cells (cellHealthy: chipColor
green, achievedDepth>=3, fresh), not the mere absence of red. A gone column
decaying to no-data no longer auto-recovers. Fixed the healthyRows test
fixture to emit a genuine green D5/D6 ladder.
- B-onset: derive sinceAt from the folded verdict's contributing ladder rows,
not a raw row.state==="red" re-scan (removed keyBelongsToSlug).
- B-A5gap: guard self-heal/loud-log on "no slug has any wired cell", not
map.size (every integration slug is keyed even with zero wired cells).
- B-env: normalize the prod gate (trim+lowercase, empty-as-unset) so an empty
SHOWCASE_ENV no longer shadows a prod Railway env and a mis-cased/padded
value no longer silently disables the monitor. Added resolveMonitorEnv +
shouldRegister; orchestrator + gate test both use the real predicate.
- B-flap: an already-open outage keeps its hourly re-post even when a later
confirm scan is inconclusive (confirm gates only OPEN and CLOSE).
- B-cadence: advance lastAlertAt only for slugs actually named in the message
(respect maxSlugsInMessage); overflow slugs keep their clock.
- Cheap: no-wired-cells logs once per tick; MAX_SLUGS floors at 1; lastAlertAt
stamps evidenceMs; resolveConfig negative/NaN/empty coverage.
Bucket-(a) fixes, each with a local red-green test:
- A1: replace the substring `:${slug}` onset match with an anchored
exact slug-segment match (`keyBelongsToSlug`) so a prefix-colliding
sibling (`strands` vs `strands-typescript`) no longer mis-attributes
the earlier sibling's red onset. RED: strands' sinceAt was pulled to
the strands-typescript onset; GREEN: each slug gets its own onset.
- A2: recovery/CLOSE is now SYMMETRIC with OPEN — a recovery requires a
second agreeing fresh-healthy read (confirm scan). RED: a single
transient healthy read fired a false "recovered"; GREEN: held until
two reads agree.
- A3: guard an empty/degenerate schedule set — longestPeriodMs 0/NaN
would make idleWindowMs 0 → isProducerLive permanently false → the
monitor SUSPENDS forever and never pages. Falls back to a 45m default
window (DEFAULT_IDLE_WINDOW_MS) and logs at error.
- A4: bound readStatusRows — guard NaN/undefined totalPages (a `page >=
NaN` break never trips) and add a hard MAX_STATUS_PAGES cap so a full
page + bad totalPages cannot infinite-loop/OOM. RED: OOM; GREEN:
terminates at the cap.
- A5: a registry-load failure logs at error with a stable errorId (not a
silent warn-once permanent no-op), and the monitor accepts a loader
thunk so it re-reads registry.json each tick while the wired-cell set
is empty — a transiently-missing file self-heals without a redeploy.
- A6 (verified, no code change): createSlackWebhookTarget already throws
on every non-2xx (4xx/5xx/429/3xx/network-exhausted); added a test
asserting the monitor does NOT delete recovery state when the post
throws.
Bucket-(b): stamp the recovery message with the confirm-scan instant
(evidenceMs) not tick-start; wrap the scheduler tick handler in a
catch+errorId; log the prod env-gate skip at warn with a reason; add a
clarifying comment that the aggregate lastAlertAt reset is intentional
one-message-one-clock cadence; simplify the three dashboard barrel-shim
comments (drop the rot-prone enumerated symbol lists).
Add a harness-native monitor that pages #oss-alerts when a whole
integration column collapses to red-D0 ("completely gone" / backend
unreachable) in production — the incident class the per-cell alert rules
miss (LGT went fully gone on 2026-07-13 and nothing paged).
Detection runs the dashboard's OWN buildCellModel fold (the shared
cell-model module both the dashboard and the monitor import) over the
same PocketBase status rows and applies a column-gone predicate over the
resulting CellModel fields, so the monitor's verdict equals the DepthChip
the dashboard renders by construction — no parallel re-derivation.
- d0-gone-predicate.ts: pure cellGone/columnGone/columnFreshHealthy over
buildCellModel outputs + registry-derived wired-cell enumeration
(mirrors the dashboard page-stats iteration / determineCellStatus rule).
- d0-gone-monitor.ts: createD0GoneMonitor factory — producer-liveness
SUSPENDED gate (reuses the family-silence inflight-aware /api/runs
reasoning, 3x-longest-period idle window), 60s confirm re-read (never a
re-probe), 15m-detect vs 1h-repost state machine, positive-fresh-healthy
CLOSE gate, ONE aggregated outage / consolidated recovery Slack message,
durable per-slug JSON map in alert_state (getSet/putSet).
- orchestrator.ts: register internal:prod-d0-gone-monitor @ */15, gated on
SHOWCASE_ENV ?? RAILWAY_ENVIRONMENT_NAME === production + kill-switch,
control-plane-only (inside runControlPlane), reusing the oss_alerts
webhook target + shared memoized family summary.
- unified-cell.test.tsx: add the required isStaleCell/observedAtAgeMs
fields to the CellModel test literal (Phase-1 dashboard tsc gate).
Red-green: a frozen test-only naiveGone (achievedDepth===0 alone)
mislabels gray-D0-no-data and stale columns as gone on committed
fixtures (RED); the real predicate fires only on red-D0-fresh and matches
buildCellModel's own outputs (GREEN). Producer-idle SUSPENDED proven
load-bearing (disabling the gate flips both F1 tests red). Plus
confirm-scan blip-rejection, hourly dedup, recovery-clear, failure modes,
and the prod-only/kill-switch registration gate.
Move the pure cell-classification fold cluster (cell-model, live-status,
staleness, format-ts) out of showcase/shell-dashboard/src/lib/ into
showcase/harness/src/shared/cell-model/ so BOTH the dashboard and a new
harness monitor import ONE copy with zero duplication and no behavior change.
The harness builds via tsc -p tsconfig.build.json with rootDir:"src" and
cannot import outside its own src/, so the harness is the correct library
home. The dashboard consumes the cluster via relative path across the package
boundary (established precedent, e.g. d5-cadence-banner.redgreen.test.ts).
- git mv the four files into harness shared/cell-model/; their intra-cluster
relative imports stay valid (they move together, no external coupling).
- Replace the four original shell-dashboard paths with thin export-* barrels
so all ~51 existing dashboard import sites resolve unchanged.
- Repoint commError-contract-drift.test.ts's source-text drift parse at the
new canonical harness location (the barrels carry no derivation body).
- Add cell-model.equivalence.test.ts + committed fixtures + a pre-move
baseline JSON (generated from the original git-HEAD code) proving the move
is byte-identical across a red-D0, gray no-data, stale, mixed, all-green,
and unsupported column.
The aggregate e2e-smoke:<slug> red signal omitted errorDesc on the normal
return path, so an abort/timeout/send-budget-exhausted red that runLevel
RETURNS (not throws) showed on the PRIMARY dashboard tick as an unclassified
content-shaped red — only the side chat:/tools: rows kept the classifier.
Thread the failing level's errorDesc (L3 precedence, L4 fallback) onto the
aggregate so the primary tick matches the side row and the launcher-phase
abort path. Does not change red/green — only carries the classifier.
Also reorder the aborted-and-empty short-circuit ABOVE the alternate-content
/ raw-byte evaluate reads: an aborted run's page is tearing down, so those
reads were swallowed against a dead page and emitted an ambiguous empty
histogram. Non-aborted runs still perform the alternate-content salvage.
A mid-poll abort — the external ctx.abortSignal firing, or the driver's
own hard-timeout landing during the first-token poll — makes runAttempt
return empty WITHOUT throwing. The retry loop breaks and control falls to
the clean-exit path, where the level was misclassified as a generic
content red ("empty assistant response", probe.exit outcome "err", no
errorDesc). That masqueraded a teardown/abort/timeout as a CONTENT
failure on the dashboard + CVDIAG.
Add an aborted-AND-empty guard before the content-red gate that
short-circuits to the same abort classification the other paths use
(errorDesc "abort", probe.exit outcome "timeout"). Discriminator is
abortSignal.aborted, not emptiness alone: a genuinely-completed-empty
turn (not aborted) stays the content-red "empty assistant response".
Harmonize the three readTurnState() consumers in the d4 chat-roundtrip probe
through one guarded safeReadTurnState() wrapper so a mid-poll readTurnState()
throw is handled consistently everywhere: it means "no reliable signal" ->
degraded widen + observable telemetry, never a silent false-red (the prior
readDegraded swallow) nor a spurious level-error (the prior unguarded
readBaseline/readTurnComplete escape). A genuinely-empty degraded turn still
reds at the ceiling.
Folds completing the PR's own items:
- item-1 first-send cap: guard the in-send press against SEND_PRESS_MIN_BUDGET_MS
so a near-hang type can't floor press to ~1ms and produce a generic
level-error; classify distinctly as send-budget-exhausted. Only press is
guarded (type opens the envelope), so a legitimately-small pageTimeoutMs still
issues a healthy first send.
- item-3 null-header: suppress the finally-block fallback probe.message.send once
a real-header boundary already fired, so a retry whose winning resend lands no
POST no longer emits a second null-header boundary (mis-attributed
edge_interference_signal).
Also add "abort" to the errorDesc JSDoc enumeration (zero-risk).
Red-green covered for all three behavioral items against the real
runLevel/readTurnComplete path with a faithful fake.
Tests for the #5882 bucket-(b) follow-up hardening:
- Budget-exhaustion retry guard (red-green): a near-exhausted-budget retry no
longer attempts a doomed ~1ms-floored resend (type invoked exactly once).
- Degraded-path floor (red-green): a degraded page (sseAttachFailed, no
completion signal) with a late-but-present token resolves GREEN instead of a
base-floor false-red; a genuinely-empty degraded run still reds.
- Retry telemetry re-attribution (focused test): a retry-rescued GREEN turn
records the winning attempt's edge headers on probe.message.send.
- Coverage: FIFO-cap CVDIAG_MAX_OUTSTANDING_STARTS_PER_URL eviction backstop;
DEBUG-auto-disarm fail-closed (disarmed => no raw-byte capture);
alternate-content / raw-byte block SKIPPED on the container-success path.
- Fake fix: makeLateTokenBrowser now mints per-page state so L3 and L4 each run
an independent stall+retry cycle (was a shared-page singleton that leaked
sendCount from L3 into L4, so the L4 retry path was never genuinely exercised).
Follow-up to #5882 (bucket-(b) CR items). Three behavioral/telemetry fixes
plus one documentation clarification, all in d4-chat-roundtrip.ts:
- Budget-exhaustion retry guard: skip a non-completion retry resend when the
remaining wall-clock budget is below RETRY_MIN_BUDGET_MS (750ms). A late
resend previously floored its type/press action timeout to ~1ms, throwing a
page-fault-shaped error that mis-classified the stall as a generic red — a
spurious-red flap source. The stall now reds on its own terms.
- Degraded-path floor: when the SSE interceptor silently no-ops
(sseAttachFailed), no completion signal ever arrives, so the poll could only
fall into the never-observed branch and pin the deadline to the base floor —
reintroducing the slow-first-token false-red #5882 targets. Consult
sseAttachFailed to WIDEN the never-observed wait to the per-attempt ceiling so
a late-but-present token on a degraded page is still captured.
- Retry edge-header re-attribution: on a retry-rescued turn, re-arm the
message-POST edge-header capture (messageSendEdge / lastMessagePostResp /
emitMessageSend latch) so probe.message.send / edge_interference_signal / the
DEBUG raw-byte sample reflect the WINNING attempt, not the stalled first one.
- Document why lastStoppedAtMs is retained on the d4 TurnState (write-only in
d4; part of the shared attachSseInterceptor global shape the d6 run-signal
snapshot also mirrors) so it does not read as dead code.
The sseOnlyStaleStop guard in makeLateTokenBrowser was ineffective: the
driver reads the per-attempt baseline via readTurnState() BEFORE the
first send, and at that point lastSendAtMs===0 made elapsed (~epoch ms)
exceed completeAt, spuriously flipping complete=true at the baseline
read. That inflated the baseline runsFinished to prior+1, so the current
turn's real finished edge never rose PAST the baseline and the driver's
sseDone = runsFinished > baseline.runsFinished could never fire. The
SSE-only-stale-grace path was therefore never entered — the test passed
only because the token rendered directly, so it did NOT guard the
Node-stamp fix.
Gate the fake's complete on started (a turn has been sent) so the
pre-send baseline read is a TRUE baseline (runsFinished = prior +
priorSendsDone). The finished edge is now a genuine THIS-turn transition
the driver observes via sseDone, and the SSE-only completion path with a
stale lastStoppedAtMs is genuinely exercised.
Proof the guard now bites (temporary production revert, not committed):
- pre-fix stamp (completeAtMs = stoppedAtMs > 0 ? stoppedAtMs : now):
test FAILS, expected 'red' to be 'green' (grace collapses).
- restored Node-stamp (completeAtMs = Date.now()): test PASSES.
Also corrects two production comments (3 reviewers flagged): the
completed-empty deadline comment claimed the base floor is always
respected, but Math.min(..., fastFailEnd, attemptCeiling) intentionally
clamps below the floor (fast-fail); and the FIRST_TOKEN_FAST_FAIL_MS doc
now spells out the full Math.min term. Comment-only, no logic change.
On an SSE-only completion (turn detected complete via runsFinished>baseline
with no fresh DOM stop-edge for THIS turn), readTurnState().lastStoppedAtMs
still held a stale prior-run value, and it is stamped on the browser-page
clock while graceEnd/deadline math runs on the Node clock. Feeding it into
Node-clock arithmetic pushed graceEnd into the past and collapsed the
FIRST_TOKEN_GRACE_MS window to the base floor, false-REDing a late-but-present
first token.
Stamp completeAtMs from Date.now() (Node) at the first poll that observes
THIS turn complete; readTurnComplete no longer threads stoppedAtMs. Also DRY
the attempt-0 baseline onto the existing readBaseline helper and fix stale
comments (fallback emit is in finally; eviction rides the onResponse wiring;
lastStoppedAtMs doc). completed-empty still fast-reds; base floor and
hardCeiling caps preserved.
Adds a red-green unit test modelling an SSE-only completion with a stale
lastStoppedAtMs (grace collapses pre-fix, honored post-fix).
The first-token poll keyed turn-complete off the page-GLOBAL monotonic
`runsFinished >= 1` / latched `sawRunningTrue`. A PRIOR run on the page
(auto-greeting / initial-mount run) leaves those already satisfied when the
user's turn starts, so the poll treated THIS turn as already complete, saw the
still-empty container, and spuriously fast-failed RED — the a1 false-red.
Fix: capture a per-attempt BASELINE (`runsFinished` + `runStartCount`) at send
time and treat the turn complete only on a NEW edge past that baseline
(`runsFinished > baseline` / a new `runStartCount` DOM run-start). A fresh
baseline is taken before each retry resend, so a stale prior edge can no longer
defeat the retry. The grace window is now stamped from the REAL finished edge
(`lastStoppedAtMs`) rather than the poll's local clock (fixes the ~500ms-short
grace). `TurnState` / `readTurnState` are extended to surface `runStartCount`
and `lastStoppedAtMs` (the sse-interceptor already latches them).
Also:
- Bound the retry resend's type/press action timeouts by the remaining budget
to `hardCeiling` (was flat `pageTimeoutMs`, letting a stalled resend push
poll-phase wall-clock to ~2x past the ceiling).
- Surface an interceptor-attach fault (`wirePlaywrightPage.goto`) via an
injectable `onAttachFault` marker + `TurnState.sseAttachFailed` so a silent
regression to the inert base-floor path is detectable, not invisible.
- Move the `probe.message.send` fallback emit into the `finally` (idempotent)
so it fires on nav/send throw paths too.
- Label external `ctx.abortSignal` aborts as `"abort"` (not `"driver-error"`)
in the aggregate, matching the per-level classification.
- Correct the fast-fail-floor / hardCeiling / poll-deadline doc comments.
Tests: a1 regression (prior finished run + in-flight turn → not false-red) at
L3 and L4; completed-empty does exactly ONE send (retry does not fire); L3
coverage for the grace/fast-fail/retry path; attach-fault telemetry surfaces.
The prior D4 first-token fix (ceae0c2c9) was INERT in production: it keyed the
turn-complete decision off the `onSseEvent` Node-side seam, which the real
launchers never wire (Playwright has no per-SSE-event signal). `sseObserved`
therefore stayed false on the real path and the whole extension collapsed to the
pre-fix base budget floor. Its red-green used a fake page that invoked the seam
synthetically, so the deadness was never caught.
Root cause of the flap: a STALLED turn (RUN_FINISHED served by aimock but the
page never rendered it — the real 20:16:52Z failure), NOT a mere client render
race. So we need a real completion signal AND a retry for never-completed turns.
Three-part fix (mirrors d6-all-pills' production-wired signal):
1. Wire the real signal. `wirePlaywrightPage.goto` now calls
`attachSseInterceptor(page)` before navigation (injectable for tests), seeding
the page-side `__hk_runsFinished` / `__hk_copilotRunning` turn-lifecycle
globals at document_start. A new `readTurnState()` E2ePage seam reads them via
`page.evaluate`; the first-token poll keys off THAT — the same
transport-level + DOM run-stop edge d6 trusts — not the dead onSseEvent seam.
2. Fast-fail genuinely-empty turns. With a real turn-complete edge, a turn that
completes with empty assistant text reds in ~completion+grace (bounded by
FIRST_TOKEN_FAST_FAIL_MS ~15s) instead of burning the flat 60s. A
completed-empty turn still reds (no masking) and is never retried.
3. Retry-on-non-completion. A turn OBSERVED in-flight that never signals
completion within budget (stalled/dropped stream) retries once before red.
Never-observed (dead/no-turn) runs stop at the base floor, no retry. Total
wall-clock is bounded by pageTimeoutMs (per-attempt budget split).
CR findings resolved: abortSignal.aborted checked inside the poll loop; the
body-scrape fallback keeps fromAssistantContainer=false AND no longer clears
cvdiagResponseEmpty, so a fallback-salvaged red can't emit terminal_outcome=ok;
red/green never gated on cvdiag (telemetry-only); the no-turn budget is capped by
the per-attempt ceiling; the fallback `tail.length>20` floor and
`split("\n")[0]` truncation removed (false-red on short/multiline answers);
stale "not started" / dead-seam comments corrected.
Real-surface red-green (real chromium + real attachSseInterceptor + real driver
against a local fixture serving SSE /api/copilotkit with injected stream delay):
- RED (pre-fix, interceptor unwired): late-token turn -> red "empty assistant
response" in ~1.4s; sseObserved-on-real-path = FALSE (dead seam proven).
- GREEN (post-fix, interceptor wired): same late-token -> green;
readTurnState on real path = {attrPresent:true,sawRunningTrue:true,...};
sseObserved-on-real-path = TRUE.
- completed-empty (wired): still RED, fast-fail ~2.5s/level, runsFinished:1.
- recoverable-stall (wired): attempt 1 never completes -> retry -> green.
Unit suite (56 tests) rewritten to exercise the real readTurnState path plus the
retry/fast-fail behaviors; tsc + build + vitest all pass.
D4's L4 "tools" probe read the assistant-message container by polling
textContent for a fixed textPollTimeoutMs. On a run where the first token
rendered into the DOM slightly later than that budget — on a turn that
genuinely produced content — the poll exhausted and read the container as
empty, yielding a spurious "L4: empty assistant response" red (a client-side
first-token render race, not a real-LLM/fixture issue).
Harden the wait to key off the AG-UI SSE turn lifecycle rather than a fixed
timeout: track RUN_FINISHED/RUN_ERROR on the already-wired onSseEvent seam,
keep polling while a turn is in-flight (up to the pageTimeoutMs hard ceiling),
and after completion allow a small bounded first-token grace window for the
DOM to paint. A turn that completes with no content ever still fails, and when
no SSE stream is observed at all the poll falls back to the base budget floor
(unchanged pre-fix behavior, no hangs).
Adds red-green tests exercising the real runLevel wait path: a late-but-present
first token now passes; a genuinely-empty completed turn still fails.
Brings the 499-commit-stale foundations branch up to date with main so #5761
has a clean diff and no stale reverts (e.g. forwardHeaders). Conflicts:
- CopilotThreadsDrawer.tsx: took main's (main renamed CopilotDrawer -> ThreadsDrawer
+ added the collapse feature; the branch's edit was a no-op import-type split).
- pnpm-lock.yaml: regenerated with the pinned pnpm 10.33.4 (adds @copilotkit/bot-intelligence).
The fleet pnpm-workspace.yaml carries a multi-segment glob
(`examples/v2/*/apps/*`) that the strict matcher rejects with a
SchemaError. Because that throw happens during enumeration — before the
probe's `pathPrefix` filter applies — it aborted the entire version_drift
discovery, surfacing as probe.discovery-enumerate-failed / discoveryFailed
with 0 PB rows.
Skip patterns whose static (wildcard-free) prefix cannot intersect the
requested `pathPrefix` BEFORE validating their glob shape, so a deep-glob
for an unrelated subtree no longer aborts a probe that only wants
`packages/`. An unsupported pattern that DOES overlap the requested prefix
still surfaces the strict-shape SchemaError, and behavior with no
pathPrefix is unchanged.
Mike Ryan