The pool checked the cap and mutated liveContextCount AROUND await points
with no synchronous reservation, so under concurrent acquire/release
(pooled drivers run FEATURE_CONCURRENCY_D6=4 x parallel services) the cap
invariant broke and capacity bled. Fix as one cohesive reservation model:
- reserveSlot(): synchronous check-and-increment with NO await between them,
so concurrent acquires can never collectively pass the check and overshoot
maxContexts during one another's newContext() awaits.
- openContextOn() now opens against a caller-held reservation; on
newContext() failure it rolls the reservation back (clamped) and rethrows.
- acquire() reserves before any await and HOLDS the reservation across the
newContext-failure retry (re-reserves explicitly), so the retry path no
longer bypasses the cap. Rolls back when it falls through to a waiter.
- serveNextWaiter() reserves synchronously before opening and is consistent
with the model so concurrent release + recycle-drain cannot overshoot.
- Dead-waiter orphan LEAK fixed: a settled flag is set by BOTH the
timeout-reject and resolve wrappers; serveNextWaiter detects a waiter that
timed out mid-open, CLOSES the freshly-opened context and rolls back the
count instead of orphaning it (the leak that permanently bled capacity and
re-created pool starvation).
- release() does its bookkeeping (delete + clamped decrement) and evaluates
the idle-recycle decision off purely SYNCHRONOUS state, then awaits
context.close() AFTER, so no await straddles the decrement and the size check.
Hardening (same file):
- recycleBrowser .finally() no longer resets entry.recycling (the success
path already cleared it; the eviction path spliced the entry out, so
resetting would resurrect a dead entry). Abandoned-context decrements now
go through the clamped releaseReservation().
- liveContextCount decrements are clamped at 0 (releaseReservation) as
insurance against any future double-decrement.
Public API (acquire/release/stats/shutdown/constructor) and stats()
semantics (size=maxContexts, available=max-live, inUse=live) preserved.
Tests: added CONCURRENT-path regressions (the existing suite was all
sequential, which is why these bugs passed CI). Red-green receipts:
- "N concurrent acquire() never exceed maxContexts": RED expected 5 to be
<= 2 -> GREEN.
- "newContext-failure retry path respects maxContexts": RED expected 2 to
be <= 1 -> GREEN.
- "does not leak a context when a waiter times out mid-serveNextWaiter":
RED expected 1 to be 0 -> GREEN.
- release close/accounting reorder: forward regression guard (the pre-fix
order kept the released context in the live set across its close-await, so
the reorder is behavior-preserving but makes the no-await-gap invariant
machine-checkable).
Wire the orchestrator to the options-object BrowserPool: a fixed set of
browser processes (BROWSER_POOL_BROWSERS, default 3, legacy BROWSER_POOL_SIZE
fallback) with a global live-context cap (BROWSER_POOL_MAX_CONTEXTS, default
24). registerAllProbeDrivers call sites are unchanged (launcher ctor args
unchanged).
Rollout note: BROWSER_POOL_SIZE's meaning shifts from "concurrent browsers"
to "base browser process count". The deploy env should move to
BROWSER_POOL_BROWSERS=3 + BROWSER_POOL_MAX_CONTEXTS=24.
Update all five pooled probe launchers for the context-pooled BrowserPool.
Each launcher's newContext() now checks out a pooled BrowserContext via
pool.acquire(opts) and the returned context-wrapper's close() releases it
via pool.release(ctx); the launcher-level close() is a no-op (no Browser is
held). The X-AIMock-Strict literal is dropped from every launcher (now
centralized in the pool); drivers still pass their per-probe X-AIMock-Context
/ X-Test-Id headers, which now flow through to pool.acquire. Driver run()
bodies are unchanged.
- d4 (createPooledE2eSmokeLauncher), e2e-parity
(createPooledE2eParityLauncher): minimal transform.
- e2e-readiness (createPooledE2eDemosLauncher), d5
(createPooledE2eDeepLauncher), d6 (createPooledE2eFullLauncher): the abort
closure is re-targeted to close each open context (each releasing its
pooled context) instead of force-releasing a held browser. d6's
dead-browser re-acquire dance is removed, the pool only opens contexts on
live browsers. Per-service Semaphore(FEATURE_CONCURRENCY/_D6) is preserved
as the orthogonal per-service fan-out bound.
Driver tests reinterpret POOL_SIZE as maxContexts and assert per-context
acquire/release moves inUse by 1, abort closes open contexts with no browser
fork, and newContext(opts).extraHTTPHeaders forwards into pool.acquire.
Re-architect BrowserPool to check out BrowserContexts from a fixed, small
set of long-lived browser PROCESSES instead of recycling whole browsers.
The old model recycled a browser (close + relaunch = a process fork) every
recycleAfter releases, so a steady-state run forked chromium repeatedly on
the hot path, the EAGAIN / PID-ceiling churn source ("Zygote could not
fork", 0/18 d6 runs).
New model:
- Checkout unit is a BrowserContext: acquire(opts?, timeoutMs?) opens a
context on the least-loaded live browser; release(ctx) closes it. No
process fork on the hot path.
- Options-object constructor: new BrowserPool({ browsers, maxContexts,
recycleAfter, logger, launchBrowser, launchStaggerMs }). Defaults
browsers=3 (env BROWSER_POOL_BROWSERS, legacy BROWSER_POOL_SIZE fallback),
maxContexts=24 (env BROWSER_POOL_MAX_CONTEXTS), recycleAfter=300 (env
BROWSER_POOL_RECYCLE_AFTER, now a per-browser served-context hygiene
threshold).
- Browser processes launch only at init() (the fixed set, staggered), on
crash recovery, and on the rare served-context hygiene recycle.
- X-AIMock-Strict default header centralized in acquire(); FIFO waiters past
the cap carry their context options; crash blast radius is bounded to the
dead browser's contexts.
- stats() keeps its field names with context semantics (size=maxContexts,
available/inUse derived from live-context count).
Tests rewritten red-green; the load-bearing assertion is that the fake
launcher's launched count stays == N across many acquire/release cycles
(zero hot-path forks).
An aborted D6 run (orchestrator process killed during a pool-churn burst)
left its `probe_runs` row orphaned in `running` state with a null summary.
The boot-time `sweepStaleRuns` then stamped it `state:failed` with
`{total:0,passed:0,failed:0}` and `duration_ms:null`, discarding the
partial per-service results the run had actually computed — a 578s run with
dozens of green features surfaced as `failed / total:0`.
Two minimal, success-path-consistent changes:
- probe-invoker: incrementally persist the running partial tally onto the
probe_runs row via a new `runWriter.update()` as each fan-out target
completes, so an orphaned row already carries real partial progress.
Best-effort — never tanks the tick; `finish()` remains the authoritative
final write.
- run-history: `sweepStaleRuns` now PRESERVES an existing partial summary
(and derives a real duration from the persisted started_at) instead of
clobbering it to zeros. Rows that died before any target completed still
fall back to an explicit empty rollup.
Purely the result-aggregation-on-abort path; pool sizing and
launch/recycle logic are untouched (churn root cause handled separately).
## Summary
Three fixes hardening BrowserPool slot publication:
1. **Idempotent `release()` via checked-out tracking** — prevents a
double-`release()` from double-serving one browser to two waiters. The
waiter-path `checkedOut` add is deferred one microtask to defeat a
same-frame double-release; this ordering is FIFO-proven safe and covered
by a control-mutant test.
2. **`handOff` liveness gate** — never hand a disconnected browser to a
waiter; recycle (idle slot) or park `relaunchPending` (busy slot)
instead, keeping the waiter queued for a live browser.
3. **`track()` registers the same wrapper it removes** — so
`shutdown()`'s drain awaits the recovery's cleanup rather than
under-waiting the raw promise.
These harden pre-existing latent bugs. The launch-gate (#5137) already
fixed the d6 infra issue, so this is robustness hardening.
## Test plan
- [x] red-green: double-release-idempotency (double `release()` is a
no-op)
- [x] red-green: dead-browser-liveness-gate (disconnected browser never
handed to a waiter)
- [x] red-green: track-wrapper-drain (shutdown awaits recovery cleanup)
- [x] red-green: waiter-served-release-no-leak control-mutant (deferred
microtask add ordering)
- [x] full harness suite green (99 files, 1681 tests)
- [x] `tsc --noEmit` clean, build clean, oxfmt + oxlint clean
Three fixes:
(1) idempotent release() via checked-out tracking (prevents double-release
double-serving one browser to two waiters; the waiter-path checked-out add is
deferred one microtask to defeat same-frame double-release, FIFO-proven safe
and covered by a control-mutant test),
(2) handOff liveness gate (never hand a disconnected browser to a waiter —
recycle/park instead),
(3) track() registers the same wrapper it removes so shutdown's drain awaits
cleanup.
These harden pre-existing latent bugs; the launch-gate (#5137) already fixed
the d6 infra issue, so this is robustness hardening.
`waitForAssistantSettled` now fast-fails (throws `AssistantErroredError`,
classified `conversation-error`, in ~2s) instead of burning the full 30s
`responseTimeoutMs` when the chat surfaces a genuine error banner.
Each poll snapshots the `copilot-error-banner` testid (shipped in #5110)
and computes a single boolean — an error banner is present in a state that
DIFFERS from the turn's baseline (a brand-new banner, OR a persisted banner
whose text changed). The turn fast-fails only when ALL of:
- that differs-from-baseline state is SUSTAINED across 2 consecutive polls
(a single isolated flicker — a transient toast, a one-poll re-render
glitch — is debounced away and does NOT fire), AND
- the assistant produced NO response this turn (the message count has not
grown past baseline). Success-in-flight wins: a non-fatal warning banner
alongside a real answer never force-fails the turn; the settle path
governs instead.
Because it only acts when the turn would otherwise time out (no response +
a sustained error), this is a strict improvement: it can only turn a
would-be timeout-fail into a fast fail of the SAME verdict — it can never
flip a verdict. The full (untruncated) banner text is compared across polls
so two errors diverging only after the first 300 chars still differ;
truncation applies only to the thrown message.
Two intentional safe-degrade edges fall back to the normal timeout (correct
verdict, just not sped up):
- count-oscillation: a response count that briefly grows past baseline and
then reverts disarms the debounce, so an error that only sustains after
that bounce settles via timeout.
- synchronous-error-at-baseline: a brand-new error whose banner text is
byte-identical to a persisted stale banner already visible at baseline
cannot be distinguished from it, so it settles via timeout.
The BrowserPool init() fill loop launches browsers one at a time through the
serializing launch gate. If launch iteration N throws -- the exact PID-ceiling
failure (pthread_create EAGAIN / "Zygote could not fork") this file exists to
survive -- init() rejected and propagated, but the browsers already launched on
iterations 0..N-1 stayed live in this.slots/available/browserToSlot and were
never closed. The sole production caller (orchestrator.ts boot) catches the
rejection, marks the pool degraded, and does NOT call shutdown(), so those
chromium processes (~50 PIDs each) leaked permanently -- accelerating the very
PID exhaustion the launch gate is meant to prevent. The gate serializes the
fill, which makes a partial-then-throw fill MORE reachable.
Wrap the fill loop so a mid-fill launch failure resets the pool's internal
state and closes every browser already launched before re-throwing. State is
cleared BEFORE closing so the synchronous `disconnected` fire from a close
cannot re-enter the recycle path via the slot's disconnect handler (the
!this.slots.includes(slot) guard short-circuits it) -- the same ordering
intent shutdown() relies on, without adding an isShutdown re-check. init()'s
reject-on-failure contract is preserved.
Call sites: init() signature is unchanged. The only production caller
(showcase/harness/src/orchestrator.ts:292) and all test callers continue to
rely on reject-on-failure, which still holds.
Adds a multi-slot partial-fill test (pool size 4, launch 3 throws): asserts
init() rejects, the 2 browsers launched before the throw were closed, and
stats() reports an empty pool. The prior init-failure tests only covered
1-slot / first-launch-fails (nothing launched yet), which is why the leak was
missed.
The browser-pool launch-stagger resolution honored the "negative or
non-numeric value falls back to the default rather than disabling the
stagger silently" contract only for the BROWSER_LAUNCH_STAGGER_MS env
var, NOT for an explicit launchStaggerMs constructor arg. A negative
explicit arg (e.g. -50) was taken by the `??` (non-null), skipped the
env/default branch, then got clamped to 0 by `resolvedStagger >= 0 ? ... : 0`
— silently DISABLING the stagger and reintroducing the launch-burst PID
spike (pthread_create EAGAIN / "Zygote could not fork") the gate exists
to prevent. A NaN explicit arg behaved the same way.
Validate the explicit arg the same way as the env var BEFORE it wins: a
valid explicit arg (>= 0, not NaN) wins; else a valid env value; else the
default (150ms). A negative/NaN explicit arg now falls back to the default,
not 0. An explicit 0 is still respected (tests rely on it to stay fast)
because `validExplicit ?? validEnv` keeps a literal 0.
Call sites: this only changes how this.launchStaggerMs is computed inside
the constructor. The sole consumer is launchBrowser()'s
`this.launchStaggerMs > 0 ? delay(...) : undefined` gate — its semantics
are unchanged (the field is still always a valid non-negative number). The
only production constructor (orchestrator.ts: `new BrowserPool(poolSize,
undefined, logger)`) passes no stagger arg, so its behavior (undefined →
env → default) is identical before and after.
Adds two red-green tests: a negative explicit arg and a NaN explicit arg
must each space launches by ~the default stagger (gap >= 140ms), proving
the stagger is not silently disabled.
The harness runs every e2e probe through one process-wide BrowserPool whose
chromium launches happen in BURSTS (initial fill, recycle relaunches, lazy
relaunchPending recovery, reinit backstop). On the Railway staging container
(~1000-PID ceiling, ~50 PIDs per headless chromium) a burst transiently
spikes PID demand past the ceiling and trips `pthread_create: Resource
temporarily unavailable` / "Zygote could not fork", so browsers fail to launch
and a d6 run goes 0/18.
Funnel EVERY launch through a single concurrency-1 serialization gate that
also waits BROWSER_LAUNCH_STAGGER_MS (default 150ms, env-tunable) after each
launch settles before the next may start. The caller still receives its
browser the instant the process is up; only the NEXT launch is gated. This
spaces process spawns so the transient PID spike never exceeds the ceiling
WITHOUT reducing the eventual pool size. All launch paths (init fill,
acquire-time/recycle relaunch, reinit, relaunchPending recovery) now route
through the gated `launchBrowser` wrapper over the raw launcher.
Adapts the two deterministic-race guard tests that previously required two
recovery launches to be in flight simultaneously — incompatible with the
serialization invariant — to assert the same re-entry guards under serialized
launches (verified by mutation: disabling the in-loop guard still fails the
test). Existing tests pin stagger to 0 to stay fast.
- cli/targets.ts: read not_supported_features from manifest YAML, pass through
buildFullInputs as notSupportedFeatures on FullInput
- probes/discovery/railway-services.ts: extract not_supported_features from
registry.json per integration via new RegistryIntegrationInfo type and emit
on RailwayServiceInfo records so the D6 driver receives it at probe time
When an integration's manifest lists a feature in not_supported_features (NSF),
the D6 e2e-full driver now partitions requestedFeatures into capable + incapable
sets BEFORE script resolution. Incapable features get a green side-row with
errorClass='skipped-incapable' and surface in the aggregate skipped[] list plus
a new incapable[] field. They no longer count as red.
Includes red-green test that asserts NSF features without a registered script
emit state=green (was red prior to this change).
The probe's EXCLUDE_SERVICES set listed all infra services with a
`showcase-` prefix (e.g. `showcase-harness`, `showcase-shell`,
`showcase-aimock`), but the actual deployed Railway service names in
the production project are BARE (`harness`, `shell`, `dashboard`,
`docs`, `dojo`, `pocketbase`, `webhooks`, `aimock`). The exact-match
`isExcluded` check therefore failed to exclude them, the non-LLM
infra services were counted as unwired (correctly missing
`*_BASE_URL` overrides), and the probe stayed red on staging/prod
even though aimock IS correctly wired for every real LLM-calling
integration service.
Fix `isExcluded` to also match the `showcase-`-prefixed form of a
bare input (`isExcluded("harness")` matches `showcase-harness` in the
set). Both naming conventions now resolve to the same canonical
entry, no parallel lists to drift. Added `showcase-webhooks`,
`showcase-dashboard`, `showcase-docs`, `showcase-dojo` to round out
the infra roster. Hypothetical pinger-style names like
`showcase-aimock-pinger-mock-for-test` still correctly surface as
unwired (existing regression test continues to pass).
Driver wrapper (`drivers/aimock-wiring.ts`) delegates entirely to the
probe and needed no change.
Red-green: two new tests assert bare names (`harness`, `shell`,
`dashboard`, `docs`, `dojo`, `pocketbase`, `webhooks`, `aimock`,
`ms-agent-harness-dotnet`) are excluded; failed pre-fix, pass
post-fix. Full file: 27/27 probe + 16/16 driver tests pass.
The showcase dashboard reads integration-scoped aggregate rows keyed
`d6:<slug>` (see shell-dashboard/src/lib/live-status.ts and
shell-dashboard/src/components/depth-utils.ts) to populate the D6
column. The CLI driver path (cli/targets.ts) invokes the e2e-full
driver with `key: d6:<slug>` so its primary return matches that
shape — but the cron probe driver runs with `key:
d6-all-pills-e2e:<name>`, so its primary return never lands under
`d6:<slug>`. The driver only emits per-feature side rows keyed
`d6:<slug>/<featureType>`, never the integration-scoped aggregate.
Result: in staging/prod the D6 column is permanently blank even
though the docstring and probe YAML promise the aggregate row.
Fix: add an `emitAggregate` helper next to the existing `sideEmit`
that writes a `d6:<slug>` row carrying the same
`E2eFullAggregateSignal` shape as the primary return, and call it
at every primary-return point in `d6-all-pills.ts`:
- post-loop aggregate (placed AFTER the feature loop so a
per-feature timeout cannot skip it; emits red when any feature
failed, so the dashboard shows red rather than blank)
- no-features-declared early return
- deploy-churn grace-window skip
- launcher-error early return
- all-runnable-filtered-out early returns (both the missing-script
red and the all-filtered green branches)
`emitAggregate` is best-effort — writer errors are logged but never
propagate to the caller, matching `sideEmit` semantics.
Tests: two new regression tests in d6-all-pills.test.ts under
`aggregate d6:<slug> side row (dashboard read contract)` exercise
the cron-shape key (`d6-all-pills-e2e:<slug>`) and assert that the
emitted `d6:<slug>` side row carries the right state (green when
all features pass; red when any fails) and aggregate signal.
Red-green: confirmed both tests fail before the fix and pass after.
Full harness suite (1648 tests / 99 files) and typecheck both clean.
Widen PoolLogger to optional warn/error and route capacity-loss events
(recycle-relaunch-failed, recovery-failed, recycle-failed, new reinit-empty)
to error and per-attempt failures (relaunch-failed, reinit-failed,
close-failed) to warn so Sentry sees real outages instead of buried info.
Add a reiniting concurrent-entry guard so two acquires on an empty pool
launch at most poolSize browsers.
## Status
WIP / not ready to merge. Preserves in-flight D6 work so it isn't lost
mid-rollout. LGP is fixture-complete; other integrations are
mid-rollout.
### Latest banked work
- **langgraph-python — 185 / 0 / 2** (green). Achieved by narrowing a d4
chat matcher that was shadowing the d6 beautiful-chat search_flights
fixture (load order is shared -> d4 -> d6, first-match-wins) plus
refreshing the d6 tool-rendering and tool-rendering-custom-catchall AAPL
fixtures (turnIndex:0 -> hasToolResult:false so the first leg fires in
multi-pill threads). 2 skips are the by-design mcp-apps iframe gap.
- **langgraph-typescript — 185 / 0 / 2** (green). Mirrored the LGP d4
narrowing on the LGT side (3 matchers) and across the
d6/langgraph-typescript suite: replaced fragile turnIndex:0 gates with
hasToolResult:false, fixed em-dash escaping that broke literal matches
in multi-pill threads, and added jsFunctions payloads to the three
sandboxed-ui fixtures (`_from-feature-parity`, `headless-complete`,
`gen-ui-open-advanced`). The final fix removed the chain-tools
`hasToolResult` match gate (which checked the whole thread and made the
chain pill fall through to the broad weather matcher mid-thread),
mirroring LGP.
- **google-adk: 174/7/2 (was 134/52/5)** — conveyance + test-parity +
fixtures + pill-wiring rebuild; 7 residual (6 default-catchall framework
default-renderer testid version question, 1 beautiful-chat fixture).
- **Pill-parity staged across 13 integrations** (ag2, agno, mastra,
pydantic-ai, claude-sdk-python, claude-sdk-typescript, llamaindex,
langroid, strands, spring-ai, built-in-agent, crewai-crews,
langgraph-fastapi). The canonical LGP suggestion pill set is now
mirrored as `src/app/demos/*/suggestions.ts` files in each integration,
with targeted edits to existing `open-gen-ui-advanced` and
`byoc-hashbrown` files. **These new files are currently UNWIRED** — each
integration's `page.tsx` still defines its pill list inline via
`useConfigureSuggestions`. Banked so the canonical source survives; a
follow-up will rewire `page.tsx` to import from `suggestions.ts` and
drop the inline copies.
- **Fleet test-parity sweep**: 576 e2e specs across 15 integrations
aligned to LGP canonical (SHA-verified); 2 orphan specs removed.
- ms-agent-dotnet 177/5/7, ms-agent-python 174/11/2 (post
fixture-mirror); default-catchall green (page-level renderer, not
react-core-gated).
## Scope
### Conveyance (foundation)
Inbound `x-aimock-context` (and friends) must ride along on outbound LLM
HTTP calls so aimock fixture matching sees the inflight test's context.
Without this the call lands on the default project's aimock and silently
picks the wrong fixture. New per-integration
`_header_forwarding.{py,ts}` shim plus matching `agent_server` / route /
factory wiring covers: ag2, agno, built-in-agent, claude-sdk-python,
claude-sdk-typescript, crewai-crews, google-adk, langgraph-fastapi,
langgraph-python, langgraph-typescript, langroid, llamaindex, mastra,
ms-agent-python, pydantic-ai, strands.
For ADK/Gemini the global httpx hook is installed BEFORE any `agents.*`
import (google-genai constructs its client at module-import time).
### langgraph-python — 185 / 0 / 2
Fixture-complete via the conveyance shim + refreshed d6/langgraph-python
fixtures + copilotkit 0.1.93 bump + the latest d4-matcher-narrowing fix
(see banked work above).
### Per-integration fixtures
Mid-rollout snapshot of d6 fixtures across the cohort plus narrowing of
`aimock/shared/common.json`'s generic 'hello' fixture to 'hello world'
so it no longer shadows D6 pills whose prompts contain 'hello' as a
substring.
### Harness `--isolate` patch
`scripts/cli/_common.sh apply_isolation` now rewrites compose-file
relative paths to absolute (build/context/dockerfile/volumes/env_file),
enforces the docker compose `[a-z0-9_-]` project-name rule, and exports
`SHOWCASE_COMPOSE_FILE` / `SHOWCASE_INFRA_PORT_OFFSET` plus offset host
URLs. The TS harness CLI (`aimock-rebuild` / `config` / `doctor` /
`lifecycle`) honors the new env so concurrent isolated stacks stop
reporting each other's services as healthy.
## Lockfile decision flagged
`showcase/integrations/langgraph-python/pnpm-lock.yaml` was deleted in
this branch. Decision: keep the deletion. Rationale:
- 03bed3b76 (fix(showcase): regenerate 18 lockfiles in isolation; switch
to npm ci) migrated all showcase integrations off pnpm onto npm ci.
- The integration's Dockerfile uses `npm ci --legacy-peer-deps`.
- Every sibling integration committed only `package-lock.json` after
03bed3b76.
- The orphan pnpm-lock.yaml only risks tooling drift.
If anyone wants it restored: `git checkout origin/main --
showcase/integrations/langgraph-python/pnpm-lock.yaml`.
## Commits
- feat(showcase): D6 conveyance — forward x-aimock-context headers to
LLM clients
- feat(showcase/langgraph-python): D6 conveyance shim + copilotkit
0.1.93 bump
- feat(showcase/langgraph-typescript): D6 conveyance — propagate request
headers into ChatOpenAI
- feat(showcase/built-in-agent): D6 conveyance — header-forwarding shim
+ factory wiring
- feat(showcase/harness): support concurrent --isolate runs
- test(showcase): D6 langgraph-python fixtures — drive to 180/5/2
- test(showcase): D6 per-integration aimock fixtures + shared narrowing
- docs(showcase): GOTCHAS entry for D6 conveyance + --isolate notes
- feat(showcase): D6 conveyance — wire header-forwarding shims into
remaining entrypoints
- fix(showcase): unblock LGP D6 beautiful-chat + custom-catchall via d4
matcher narrowing
- fix(showcase): narrow LGT D6 d4 shadows + wire sandboxed-ui
jsFunctions
- chore(showcase): copy LGP canonical suggestion pills into 13
integrations
- fix(showcase): forward x-aimock-context per-request in google-adk
routes
- test(showcase): align google-adk e2e specs to langgraph-python
canonical
- fix(showcase): align google-adk D6 fixtures to LGP contract
- fix(showcase): wire google-adk default-catchall to shared 4-pill
suggestions
## Summary
Two confirmed bugs in the showcase harness + operator dashboard that
together produced a false-green D3 on the dashboard while the e2e probe
pipeline was actually dead.
### Bug 1 — Browser-pool death spiral
(`showcase/harness/src/probes/helpers/browser-pool.ts`)
When a chromium process crashed, `recycleSlot` relaunched it; if that
launch threw, the slot was permanently removed via `slots.splice`. A
single transient launch failure (OOM spike, fd exhaustion) thus
monotonically shrank the pool to empty with no self-heal, after which
every e2e probe failed with `launcher-error` / `BrowserPool acquire
timeout`.
Fix:
- The recycle retries the relaunch with bounded backoff (3 attempts,
doubling backoff).
- On exhaustion the slot is **kept** (capacity preserved) and parked as
`relaunchPending`; the next `acquire()` lazily re-attempts the launch.
- `acquire()` re-initializes the pool as a backstop when every slot has
been lost (`slots.length === 0`).
- Existing acquire/release/recycle semantics and logging style
preserved.
### Bug 2 — Stale-green e2e rows read as healthy
(`shell-dashboard/src/lib/cell-model.ts`,
`src/components/depth-utils.ts`)
When the e2e driver stops writing `e2e:<slug>/<feature>` rows, the last
row freezes. A green row then reads as a healthy D3 forever, so the
depth ladder shows a false-green D3 that masks a dead probe pipeline
instead of surfacing it.
Fix:
- `buildCellModel` and `deriveDepth` treat a green e2e row whose
`observed_at` is older than `E2E_STALE_AFTER_MS` (6h, matching the
original stale-window model) as degraded/amber, so it no longer credits
D3.
- Only green is downgraded — a stale red/degraded row already signals a
problem and is left as-is.
- An unparseable timestamp is treated as not-stale, so staleness is
never inferred from bad data.
## Test plan
- [x] New unit test: pool recovers after a transient relaunch failure
(does NOT shrink to 0) and re-inits when emptied — confirmed red→green.
- [x] New unit tests: a stale green `e2e:` row downgrades to amber (not
green) and does not credit D3; a fresh green row stays green; a stale
red row stays red — confirmed red→green.
- [x] Existing fixtures with hardcoded old `observed_at` updated to
default to a recent timestamp so legitimate green rows aren't treated as
stale.
- [x] `showcase/harness` full suite: 1646 tests pass.
- [x] `shell-dashboard` full suite: 630 pass (1 env-gated build-spike
skipped).
- [x] `packages/**` test graph: 17 projects pass.
- [x] typecheck clean (both packages); oxlint 0 errors; oxfmt clean.
Note: there is a separate operational lever (reduce `BROWSER_POOL_SIZE`
/ e2e concurrency, raise harness memory) handled outside this PR; this
is the code fix only.
Re-validate each pending slot inside relaunchPendingSlots before claiming it so
two concurrent invocations can't double-launch a slot (leak + double-publish).
Contain recycle throws with a logged catch and gate recycleSlot entry on shutdown.
The deferred-relaunch background timer produced a new race in three review
rounds. Remove it entirely and rely on lazy, on-demand recovery: a failed
recycle parks the slot relaunchPending (never evicts), and the next acquire
re-attempts the launch via relaunchPendingSlots, serving any queued waiter
through handOff. The no-eviction + lazy-recovery design already prevents the
original drain-to-0 bug without a timer. Also clamp stats().inUse to >= 0.
Re-arm the deferred relaunch when a waiter parks after a recycle has
already failed with no waiter queued, so it is not stranded to timeout.
Make scheduleDeferredRelaunch idempotent via deferredRelaunchActive so
the acquire-kick and recycle-kick cannot spawn two timer loops. Guard
recycleSlot entry and the relaunchPending filter with a single
isSlotBusy predicate covering both recyclingSlots and relaunchingSlots,
so a late disconnect for a pending slot's old browser cannot double
launch and leak a process. Strengthen the double-publish test to fail
loud and pin exact-once, and add red-green tests for both fixes.
Reconcile the stale stats()/reinit()/acquire() comments: all-pending recovery
goes through relaunchPendingSlots, not the size-0 reinit backstop (which is the
truly-empty never-launched case). Rename the misleading reinit test to assert
the actual relaunchPendingSlots mechanism. Chain track()'s cleanup so a future
recovery throw cannot float as an unhandled rejection, and log a -1 sentinel
slotIndex for the never-added reinit close. Fix the self-contradicting
drainMicrotasks JSDoc.
loadAllFixtures silently skipped files lacking a fixtures array and let
JSON.parse throw without the path; throw a clear error naming the file in both
cases and attach the caught error as cause. Export the helper and add red-green
tests covering both throw branches.
relaunchPendingSlots only pushed recovered slots to `available` and never resolved
queued waiters, so the deferred relaunch scheduled by a failed recycle (scheduled
precisely because waiters were queued) left the waiter stranded for its full 30s
acquire timeout while a live browser sat idle. It also had no re-entry guard, so an
acquire()-driven call racing the deferred timer could both launch a browser for the
same pending slot — leaking one process and/or publishing the slot to `available`
twice (handing the same browser to two probes).
- Extract a single `handOff` helper (waiter-first, then `available` with an includes
guard) and use it in reinit, the recycle success path, and relaunchPendingSlots so
all three share one correct invariant.
- Add a `relaunchingSlots` re-entry guard: a slot is marked before its launch await
and skipped by concurrent invocations, cleared in finally.
- Replace the one-shot deferred relaunch with a self-rescheduling timer (bounded
backoff) that retries while pending slots and waiters coexist, so a waiter is no
longer stranded when a single deferred attempt also fails. Stops on shutdown.
- Track reinit/relaunch launch promises so shutdown() drains them, closing the leak
where a launch resolved after shutdown awaited the in-flight set.
- Route failure and close logging through the injected logger (with slot index)
instead of console.error / swallowed catches, so failures reach the harness pipeline.
- stats().size now counts only live (non-pending) capacity, making the empty-pool
backstop observable (size 0 when every slot is pending).
Tests: fix the vacuous reinit test (failAtCalls now exhausts all retries for every
slot and asserts size 0 before recovery), and add deferred-waiter recovery and
no-double-publish tests. Full harness suite green (1648 tests), typecheck green.
When a chromium process crashes, recycleSlot relaunches it; if that launch
threw, the slot was permanently removed via slots.splice. A single transient
launch failure (OOM spike, fd exhaustion) thus monotonically shrank the pool
to empty with no self-heal, after which every e2e probe failed with
launcher-error / BrowserPool acquire timeout.
The recycle now retries the relaunch with bounded backoff; on exhaustion it
keeps the slot (capacity preserved) and parks it as relaunchPending for a lazy
retry on the next acquire. acquire() also re-initializes the pool as a backstop
when every slot has been lost (slots.length === 0), so the pool always
self-heals rather than wedging.
- scripts/cli/_common.sh apply_isolation rewrites compose-file relative
paths to absolute (build/context/dockerfile/volumes/env_file), enforces
the docker compose [a-z0-9_-] project-name rule (normalize-with-warn or
hard fail), and exports SHOWCASE_COMPOSE_FILE / SHOWCASE_INFRA_PORT_OFFSET
plus offset host URLs (AIMOCK_URL_LOCAL / DASHBOARD_URL_LOCAL /
POCKETBASE_URL_LOCAL) so the TS harness CLI talks to THIS project's
aimock instead of the default :4010
- harness/src/cli/{aimock-rebuild,config,doctor,lifecycle}.ts honor the
new env vars; lifecycle picks up the offset infra port for health
probes so concurrent stacks no longer report each other's services as
healthy
- Commit a generated harness/package-lock.json (new file) so npm ci
resolves deterministically
Repoint the aimock fixture-coverage probe to the per-integration d4/d6/shared layout; align
d5-multimodal assertions to the current auto-send button sentinels.
the integration was added to the railway-services SSOT and smoke.yml nameExcludes; this
adds it to the sibling probe configs the probe-config-parity invariant requires to stay
in lockstep.
The "Traffic pie chart" suggestion ("Show me a pie chart of website traffic
by source.") names a subject but supplies no numbers, so the agent asked the
user for data instead of rendering. Add a system-prompt directive (LGP + ADK)
telling the agent to invent illustrative sample values and render on the first
turn, never asking for data. The suggestion copy stays clean — the behavior is
carried by the system prompt, not parenthetical UI hints.
Also retag the gen-ui-tool-based demo (LGP + ADK) from `generative-ui` to
`controlled-generative-ui` so the dojo sidebar pill reads "Controlled
Generative UI" — the established product taxonomy (already a category in
shared/feature-registry.json and the dashboard catalog).
Scoped to LGP and ADK per the ticket; the other 16 integrations keep the old
tag until the taxonomy rolls out wider.
Tests: add D5 aimock fixture entries mirroring all three suggestion chips
(bar/traffic-pie/market-share) so the suggestion-click path has deterministic
coverage. The existing "revenue by category" probe message is preserved, so
the dashboard D5 row stays green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- qa probe: manifest.yaml path corrected from showcase/packages/ to
showcase/integrations/ (Slice 3 moved files; probe code and test
fixture still referenced the old path).
- pin-drift probe: fail-baseline.json ENOENT in container — the 5-level
import.meta.url walk-up from dist/probes/drivers/ overshoots /app and
lands at /. Added PIN_DRIFT_REPO_ROOT=/app env var and COPY of the
baseline file into the runtime stage.
- aimock-wiring probe: showcase-ms-agent-harness-dotnet (deployed: false)
was not in the EXCLUDE_SERVICES set, causing it to be flagged as
unwired. Added to the exclude list in both aimock-wiring.ts and
smoke.yml.
Add d6-all-pills driver that exercises every pill on every integration
behind aimock context routing. Rename e2e-chat-tools/e2e-deep to
d4-chat-roundtrip/d5-single-pill to match depth convention. Wire
D5 representative map, D6 scoping helpers, dimension types, and
the orchestrator config for the new probe IDs.
Production-fix bundle for beautiful-chat + 3 other google-adk demos. All
issues either reported on Railway prod or unmasked by the catch-all
render_a2ui fallback I added in #4836. Local D5 stays 38/38 green.
- Missing copilotkit logo: `<img src="/copilotkit-logo-mark.svg">` returned
404 because the SVG never shipped in the integration's public/. Copy
copilotkit-logo-mark.svg + copilotkit-logo.svg over from langgraph-python.
- Multimodal sample.png / sample.pdf were LFS pointers in prod (Railway
build runs without `git lfs pull`), so the magic-byte check rejected them
on first click. Add an integration-scoped .gitattributes that exempts
these two paths from LFS (mirrors what every working sibling integration
already does) and re-stage the files as real binaries (10KB / 2.5KB).
- Sales Dashboard pill returned a generic "Step 1... Step 2..." narration
instead of rendering the A2UI surface. Root cause: the render_a2ui
catch-all fallback I added to aimock/d5-all.json in #4836 fired before
feature-parity.json's specific sales-dashboard fixture (load order
d5-all → smoke → feature-parity). Removing the catch-all from both
d5-all.json and the per-demo source so the specific fixture wins.
- Calculator App pill returned text only with a white iframe because
beautiful-chat's agent instruction never mentioned generateSandboxedUi —
Gemini saw the tool listed via AGUIToolset but had no nudge to use it.
Added a one-line "Interactive / sandboxed widgets" entry to the
instruction; verified Gemini now emits the tool call.
- hitl-in-app refund (#12345) and escalate (#12347) pills broke on the
second click because the 2nd-turn fixtures keyed on `sequenceIndex` 0/1
(a global thread-position counter that drifts when other pills land
tool messages in the same thread). Convert both to `toolCallId +
hasToolResult: true` and drop the reject branch — Gemini reasons
correctly from the tool's `approved: false` return without a fixture
override. Same pattern that fixed tool-rendering-reasoning-chain
previously.
- hitl-in-app downgrade (#12346) pill produced an unrelated
"Research / Outline / Draft / Review / Finalize" plan because the
prompt contains the substring "plan" and feature-parity.json has a
generic catch-all match on `userMessage: "plan"`. Add a specific
downgrade fixture in d5-all.json (loaded before feature-parity.json)
with hasToolResult: false / true branches.
- hitl-in-chat "Schedule a 1:1 with Alice" returned the wrong
"Nice to meet you, Alice in Tokyo" response when clicked AFTER another
pill in the same thread. The 2nd-turn fixture only matched on
`toolCallId` (no hasToolResult), so the bare "alice" / "Alice" greeting
fixtures further down won. Add `hasToolResult: true` to the 2nd-turn
fixture so it scopes correctly regardless of thread state.
- Voice manual recordings always returned "What is the weather in Tokyo?"
regardless of audio content. aimock had a catch-all transcription fixture
(`match: { endpoint: "transcription" }`) that returned the canned
Tokyo string for any audio input. The D5 voice probe uses the sample
audio button which bypasses /transcribe entirely (it injects text
directly into the composer), so removing the transcription fixture
drops aimock into --proxy-only fall-through to real OpenAI Whisper for
mic recordings while D5 stays green. Verified.
- readonly-state-agent-context and shared-state-read-write fixtures
returned hardcoded "Atai" / generic preferences responses even when
the user changed the state values in the UI inputs. Gate the
Who-am-I / Suggest-next-steps / Greet / Plan-a-weekend fixtures on
systemMessage substring matching the canonical default state values
("Atai" name for readonly-state-context; "tone: casual" for
shared-state-read-write). When the user changes state, the agent's
before-model callback rebuilds the system prompt with the new values,
the fixture's systemMessage substring no longer matches, and aimock
--proxy-only falls through to the real model so the response reflects
the actual state. Confirmed with paired curl tests (default state =
fixture match; alem name = real-LLM response).
Local verification: bin/showcase test google-adk --d5 finishes 38/38
green (137s). Calculator pill confirmed via direct ADK invocation
against real Gemini (GOOGLE_GEMINI_BASE_URL=) emits TOOL_CALL_START
toolCallName=generateSandboxedUi. Sales-dashboard pill confirmed end-to-end
returning the full Column / DashboardCards / PieChart / BarChart payload.
readonly-state-context confirmed with name=Atai matching fixture vs
name=alem falling through to real-LLM response that uses the actual
context.
Five distinct root causes were keeping google-adk from full D5 parity
with langgraph-python. Fixing them takes the integration to 38/38 D5
green under aimock locally (verified end-to-end with --live writing
to PocketBase).
- readonly-state-context: page.tsx asked for agent slug
readonly_state_agent_context (underscore) but the registry mounts
it kebab-case as readonly-state-agent-context. useAgent threw,
the demo layout never mounted, and the ctx-name input never rendered.
Align with the registry (and with langgraph-python).
- multimodal: the secondary failure was a backend Pydantic
ValidationError on HttpOptions.api_endpoint. google-genai 1.75
renamed the field to base_url; all three direct genai client
constructors (main.py, beautiful_chat_agent.py, subagents_agent.py)
needed the rename so the secondary A2UI / sub-agent LLM calls stop
crashing. (The pre-existing LFS-pointer issue on the bundled
sample.png/pdf was a worktree hydration problem, not a tracked
code change — git lfs pull handles it.)
- gen-ui-declarative (two bugs stacked):
1. The same api_endpoint -> base_url rename above. The secondary
generate_a2ui planner LLM was failing every request.
2. The D5 fixture emitted components in {id, type, props: {...}}
shape, but sanitize_a2ui_components requires component, so
every entry was dropped and the renderer received an empty
surface. Rewrite both the per-demo fixture and the d5-all.json
aggregate to the flat {id, component, ...props} shape that
langgraph-python's _design_a2ui_surface fixture already uses,
wrapping multi-child layouts in a basic-catalog Column (the
custom Card schema has a single child slot). Add
_design_a2ui_surface variants so LGP gets per-pill payloads too.
- shared-state-streaming: ADK's write_document took content and
the PredictStateMapping read tool_argument="content", but the
shared D5 fixture (and the LGP function signature) names the
argument document. Rename both sides so the fixture's tool_call
args plumb into the function and into PredictStateMapping's
state-key emission. STATE_DELTA now propagates and DocumentView
streams live.
- tool-rendering-reasoning-chain: in thinking mode
(include_thoughts=True), Gemini emits a turn as two separate
non-partial chunks — a text-only chunk with finish_reason=None
and a function-call-only chunk with finish_reason=FUNCTION_CALL.
stop_on_terminal_text fired on the first (text-only) chunk and
set end_invocation=True before the function-call chunk arrived,
which broke AAPL->MSFT chaining. Gate termination on
finish_reason=STOP; FUNCTION_CALL and None both mean "more
chunks inbound — defer". Applies to every agent that uses the
shared callback, so chain-aware behavior is uniform.
Local verification: bin/showcase test google-adk --d5 --live
finishes green for all 38 cells (~140s), dashboard reflects the
results from PocketBase. Manual real-Gemini click-through of the
five fixed demos also passes end-to-end via GOOGLE_GEMINI_BASE_URL=
(empty) recreate.
Wraps railwayServicesSource with withCache (24h TTL). Instantiates
DiscoveryAuthTracker with threshold 3. Adds system dimension. Caches
listServices in Railway adapter (60s TTL). Writes system status on
browser pool init failure so degradation is visible in the dashboard.
Tracks auth failures per source since last success. After 3 failures,
writes system:discovery-auth-failed to PocketBase. Sustained alerts
rate-limited to one PB write per 5 minutes. Auto-recovers on next
success. Non-auth errors are no-ops. 9 test cases.
Transparent wrapper at the DiscoverySource interface level. Caches
successful enumerate() results in memory (24h TTL), serves stale
data on upstream failure, collapses concurrent callers into a single
upstream request. Auth tracker side-effects are try-caught to never
block the primary data path. Evicts entries older than 2x TTL.
18 test cases covering success, failure, TTL, collapse, eviction,
non-JSON config guard, and tracker integration.
Jordan Ritter