## Summary
Producer-side foundation for fleet framework item 3b. Two
**behavior-preserving** generalizations that make the seam capable of
multiple browser families and multiple producer cadences, while keeping
the d6 case **byte-identical**. No wiring is flipped on yet (see Out of
Scope).
### 1. Parameterized service enumerator
`showcase/harness/src/fleet/control-plane/catalog-enumerator.ts`
- New generic `createServiceEnumerator(params)`
(catalog-enumerator.ts:215) takes the service-set `filter`, the
`driverKind`, and a `probeKeyPrefix` (string prefix → `<prefix>:<slug>`,
or a builder fn).
- `createD6ServiceEnumerator` (catalog-enumerator.ts:280) is
re-expressed as a thin call passing the d6 params: `D6_DRIVER_KIND`
(`e2e_d6`), prefix `"d6"` (→ `d6:<slug>`), and `D6_DISCOVERY_FILTER`. d6
output is unchanged — same services, same filter, same kind, same keys.
### 2. Control-plane accepts an array of producer schedules
`showcase/harness/src/fleet/control-plane/control-plane.ts`
- New `ProducerSchedule` type (`{ scheduleId, cron, producer }`) + a
`schedules?` dep on `ControlPlaneDeps`.
- `createControlPlane` normalizes to an array (control-plane.ts:~232);
omitting `schedules` degenerates to the single d6 schedule on
`FLEET_PRODUCER_SCHEDULE_ID` (`fleet-job-producer`) @ `40 * * * *` —
current behavior preserved exactly.
- `start()` / `stop()` iterate the array, registering/unregistering each
scheduler entry and starting/stopping each producer.
## Out of scope (deferred — gated on other in-flight PRs)
- **No `runControlPlane` wiring** to actually PASS multiple schedules —
that edit conflicts with in-flight **#5284** (which edits
`runControlPlane`) and is deferred. This PR only makes
`control-plane.ts` *capable* of N schedules + generalizes the enumerator
seam; the wiring lands later.
- No `e2e_smoke` / `e2e_demos` / `e2e_deep` enumerators or producers
(Phase 2).
- No changes to `worker-loop.ts` / `payload-mapper.ts` /
`probe-loader.ts` (other PRs own those).
- No driverKind constant / contract changes.
## Test plan
- [x] Red→green TDD: 3 new enumerator tests (generic
kind/keys/filter/fn-prefix) + 2 new control-plane tests (N entries
registered with distinct crons; stop tears all down) failed before impl,
pass after.
- [x] Equivalence: all pre-existing d6-enumerator + single-schedule
control-plane tests pass unchanged.
- [x] Full harness suite green: **2078 passed** (119 files).
- [x] `tsc -p tsconfig.build.json` clean (exit 0).
- [x] Only the 4 intended files changed; no lockfile drift.
Do not merge — producer-side foundation only; wiring follows after #5284
lands.
The multi-schedule seam (createServiceEnumerator + the schedules[] capability
in control-plane) didn't meet the file's own best-effort/fail-loud bar. Harden
it now since Phase 2 builds on it (no production caller yet):
- stop(): guard each producer.stop() per-entry so one rejection no longer aborts
teardown of later schedules (leaked cron handlers + running producers).
- start(): pre-validate every schedule's cron up-front before starting any
producer, throwing an aggregated error naming the offending scheduleId — no
more half-started plane with `started` latched true.
- normalization: throw on duplicate scheduleId (replace-semantics would silently
collapse two producers onto one entry) and on an explicitly-empty schedules:[]
(distinct from omitted, which keeps the d6 default).
- createServiceEnumerator: require a non-empty filter.namePrefix (an absent
prefix would enumerate ALL services) and reject an empty probeKey from a
function-form prefix, naming the slug.
Also: narrow the d6 "byte-identical" docstring (specs identical; the
catalog-enumerated log adds driverKind), pluralize the start()/stop() +
module-header producer comments, drop the Phase 2 marker, and extract the shared
ServiceSetFilter type.
Producer-side foundation for fleet item 3b — two behavior-preserving
generalizations, byte-identical for the d6 case:
1. Generalize the d6 service enumerator into a parameterized
`createServiceEnumerator(params)` carrying the service-set filter, the
driverKind, and the probeKey prefix builder. `createD6ServiceEnumerator`
is now a thin wrapper passing the d6 params (e2e_d6 kind, d6:<slug> keys,
D6_DISCOVERY_FILTER), so d6 behavior is unchanged.
2. Generalize createControlPlane to accept an array of
{ scheduleId, cron, producer } entries and register each on the scheduler.
The single-d6 case degenerates to a one-element array on
fleet-job-producer @ 40 * * * *, preserving current behavior.
Out of scope (gated on in-flight PRs): orchestrator runControlPlane wiring
to pass multiple schedules (conflicts with #5284), the e2e_smoke/demos/deep
families (Phase 2), and worker-loop/payload-mapper/probe-loader.
CR fixes for the fleet worker driverKind→driver registry (PR #5283):
- Fix default (self-contained) worker boot: build the default d6 as a
registry entry { driver, payloadToInput, aggregateSlugKey } instead of a
bare driver with no mapper, so startWorkerLoop's construction guard no
longer throws "Fleet worker has no drivers".
- Thread aggregate-key derivation through DriverRegistryEntry
(aggregateSlugKey?), defaulting to d6:<slug> so the d6 cell-capture filter
stays byte-identical while non-d6 kinds can supply their own scheme.
- Add construction-time fail-loud assert that each registry entry's factory
kind matches its key; raise unknown-driver-kind log to error to match the
sibling protocol-violation logs.
- Extract shared buildPooledBrowserDrivers consumed by both
registerAllProbeDrivers and the worker registry; collapse no-op per-kind
payload-mapper aliases to the single createPayloadToInput.
- Introduce typed DriverKind union (contained to worker/payload-mapper);
keep contracts.driverKind a string wire boundary.
- Tests: new fleet/orchestrator.test.ts (default-boot equivalence), driver
construction-guard + custom aggregateSlugKey coverage, and lock-step +
registry-wiring pins (factory kind == constant).
Generalize the fleet worker from a single hardwired d6 driver into a
driver REGISTRY keyed by payload.driverKind, so one worker can host all
four browser driver families (e2e_d6, e2e_deep, e2e_demos, e2e_smoke).
- worker-loop: accept a `drivers: Map<kind, { driver, payloadToInput }>`
and dispatch each claimed job by `payload.driverKind`. Unknown kind →
terminal `worker-protocol-violation` (same shape as an unmappable
payload), never a worker crash. Legacy single `driver`+`payloadToInput`
pair retained as a fallback for back-compat. Fail-loud at construction
when neither a registry nor the legacy pair is supplied.
- payload-mapper: generalize createD6PayloadToInput into a shared
createPayloadToInput plus per-kind aliases and the four driver-kind
constants (the input re-hydration is identical across families; each
driver's own zod schema is the validation gate).
- orchestrator runWorker: build all four pooled drivers on the shared
BrowserPool and register them by kind, lifted from the legacy
registerAllProbeDrivers pooled construction.
d6 routing is unchanged (equivalence gate). Red-green tests cover
routing e2e_smoke/e2e_deep/e2e_d6 to their drivers and unknown-kind →
protocol-violation. Full harness suite green (2085 tests); tsc clean
except the known pre-existing toReversed error.
The harness starter-smoke CHAT rung 503'd on staging because its raw fetch()
POST sent only Content-Type — omitting X-AIMock-Context. The scoped
per-integration "Hello" fixtures in showcase/aimock/d4/<integration>/chat.json
only match (under aimock strict mode) when the request carries
X-AIMock-Context:<context>, and that context token IS the dashboard column
slug (verified against each integration's
showcase/integrations/<col>/playwright.config.ts extraHTTPHeaders and the
fixture's match.context). The local browser e2e passes only because Playwright
injects the header, forwarded by HeaderForwardingMiddleware to aimock.
Send X-AIMock-Context: <columnSlug> on the chat POST only (matching the
browser; the GET rungs hit the runtime /info route, not aimock). columnSlug is
already resolved via starterToColumnSlug before the chat rung runs, so the skew
cases (langgraph-js->langgraph-typescript, adk->google-adk,
strands-python->strands, ms-agent-framework-*->ms-agent-*) get the correct
context for free.
Proven RED->GREEN against live staging aimock
(https://aimock-staging.up.railway.app): without the header -> 503
no_fixture_match; with the mapped column-slug context -> 200 matching the
scoped fixture.
## Summary
Rewrites the `starter-smoke` probe to the **v2 path-based multi-route**
runtime protocol the deployed starters actually speak — proven via a
local build+curl gate against `examples/integrations/crewai-crews` at
1.59.5 (the runtime mounts `createCopilotEndpoint` in default
`mode:"multi-route"`, identical at 1.59.3 and 1.59.5).
- **agent** → `GET /api/copilotkit/info` requiring 200 + `version` (was:
"any non-404", which a health-JSON/HTML error page wrongly passed)
- **chat** → `POST /api/copilotkit/agent/<id>/run` (`Accept:
text/event-stream`), asserting ≥1 `TEXT_MESSAGE_CONTENT` delta **+**
terminal `RUN_FINISHED` **+** no `RUN_ERROR` (was: "non-empty body")
- **health** → repointed to `GET /api/copilotkit/info` (the starters
serve no `/api/health` route)
- **interaction** → `GET /` (unchanged)
- **drop the trailing slash** on runtime POSTs — `POST /api/copilotkit/`
308-redirects and drops the POST → 404, the core cause of the red rungs
Plus a **unified abort/body-read error classification** (one
`abortOutcome()` helper keyed on the local `externallyAborted` flag): a
self-timeout or a non-abort error → `transport-error` (soft), only a
genuine external abort → `aborted`; a cut-short body read softens to
`transport-error` instead of hard `smoke-failed`; the level loop
short-circuits on external abort. Reviewed across 3 CR rounds
(path-based rewrite + 2 abort-classification fix rounds + confirmation).
> **NOTE:** the content-level chat rung will correctly read **RED**
against the deployed starters until the Python-side `ag-ui-crewai`
`RUN_ERROR` (a real-LLM, in-process agent failure — *not* a probe bug)
is fixed. That is the truthful signal.
## Deferred follow-ups (tracked, not in scope)
- **Verify the alert/staleness consumer branches on `errorClass`, not
`state`** — else the soft/hard (`transport-error` vs `smoke-failed`)
split is cosmetic at the dashboard layer
- `res.text()` resolve-partial (truncated-but-resolved body) precision
edge → could mis-class as `smoke-failed`
- `redirect:"follow"` POST→GET edge; health/interaction success-path
body drain; health/agent share `/info` (correlated signals)
- `CLASS_RANK` ranks `aborted` lowest (design call); unmapped-starter
`columnSlug` invariant + prototype-key collision
- test-coverage gaps: `resolveTimeoutMs` env parsing,
`deriveStarterSlug` fallback, real-timer abort path
## Test plan
- [x] Harness unit suite — 1829 tests pass (105 files); starter-smoke
35/35; red-green verified for protocol + all abort/body-read fixes
- [x] Typecheck (no new errors), oxlint (0 errors), oxfmt, harness build
- [ ] Post-merge: redeploy harness, re-probe; chat rung greens once the
Python-side RUN_ERROR is fixed (separate workstream)
The chat rung hardcoded `/api/copilotkit/agent/default/run`, which 404s for
mastra — it registers dynamic non-`default` agent keys via
`MastraAgent.getLocalAgents` rather than `agents:{default}`. Resolve the
agent id per-starter from the first key of the `/info` `agents` map (the same
info response the health + agent rungs already fetch), falling back to
`default` only when that map is empty/unreadable, and surface the resolved id
on the chat row signal for drilldown.
Rewrite the starter-smoke probe to the v2 path-based multi-route runtime
protocol the deployed starters actually speak (proven via a local
build+curl gate against examples/integrations/crewai-crews at 1.59.5):
- agent rung: GET /api/copilotkit/info -> 200 + version (was: any non-404)
- chat rung: POST /api/copilotkit/agent/<id>/run (Accept: text/event-stream)
asserting >=1 TEXT_MESSAGE_CONTENT delta + terminal RUN_FINISHED + no
RUN_ERROR (was: non-empty body)
- health rung: repointed to GET /api/copilotkit/info (the starters serve
no /api/health route)
- interaction rung: GET / (unchanged)
- drop trailing slash on all runtime POSTs (the old /api/copilotkit/ 308s
and drops the POST -> 404, the core cause of the red rung)
Plus a unified abort/body-read error classification (one abortOutcome()
helper keyed on the local externallyAborted flag): a self-timeout or a
non-abort error is transport-error (soft), only a genuine external abort
is 'aborted'; a cut-short body read softens to transport-error instead of
hard smoke-failed; the level loop short-circuits on external abort.
NOTE: the content-level chat rung will correctly read RED against the
deployed starters until the Python-side ag-ui-crewai RUN_ERROR (a
real-LLM, in-process agent failure, NOT a probe bug) is fixed.
Deferred follow-ups: verify the alert/staleness consumer branches on
errorClass (not state) so the soft/hard split is actually consumed;
res.text() resolve-partial precision edge; redirect:follow POST edge;
health/interaction body-drain; health/agent shared /info; CLASS_RANK
aborted-lowest; unmapped-starter columnSlug/prototype-key; test-coverage
gaps (resolveTimeoutMs, deriveStarterSlug, real-timer abort).
The starter_smoke probe registers its driver under kind `starter_smoke`
(starterSmokeDriver, orchestrator.ts) and config/probes/starter_smoke.yml
declares `kind: starter_smoke`, but the probe-config Zod `kind` enum is
built from DIMENSIONS, which only listed the `starter` emit-prefix and not
the `starter_smoke` probe kind. The enum therefore rejected the YAML at
parse time; because `kind` is shared across all three union variants the
failure surfaced as the misleading "Unrecognized key(s) in object:
'discovery'" union error, the probe never loaded, AND probe-loader.test.ts's
shipped-config assertion failed (the lone recurring failure).
Add `starter_smoke` alongside `starter`, mirroring the existing kind/emit
pairs (e2e_d6/d6, e2e_deep/d5, e2e_demos/e2e). `discovery` was a red herring
— it is a valid key the DiscoveryBlockSchema already accepts; the working
d6-all-pills-e2e.yml uses the identical discovery shape.
Red-green coverage for the writer-hardening fixes:
- in-flight cap DROP emits dropped-over-inflight and the counter does not leak
- a hung create times out, counts as a failure, and escalates after the threshold
- once writes are timing out, concurrent drops contribute to the escalation
- a drop while a write succeeds (healthy backpressure) does NOT escalate
- a no-progress deleteByFilter over cap backs off (gated by interval, not per insert)
- a cold start far over cap prunes in bounded batches (<= 200), never a thousands-row list/filter
- a hung Slack webhook is aborted at the timeout so the serialized health-signal chain isn't stalled
Three durable-logging robustness fixes in the resource-snapshot writer:
- PRUNE BATCH CAP + NO-PROGRESS BACKOFF: a cold start over a large backlog
(or lowering RESOURCE_SNAPSHOT_MAX_ROWS on a full collection) made
`surplus` thousands large, used directly as a `perPage` and a
one-clause-per-row id-delete filter — a giant request/URL that PB
rejects (400/414) so the prune never converged. Cap each sweep to
PRUNE_BATCH_MAX (200) oldest rows and let knownOverCap drain the rest
across inserts. When a sweep makes NO structural progress while still
over cap (PB delete-rule 403, filter 400, or 0 matched), back off via a
suppression window so the retry is gated to once-per-interval instead of
re-arming knownOverCap and re-running the failing sweep on every insert.
- DROP→ESCALATION SEAM: an in-flight-cap DROP did not count toward the
consecutive-failure escalation, so a 100%-broken PB whose writes hang
could keep new snapshots on the DROP path and delay/dodge the loud
`write-failing-systematically` error. A drop now contributes to the
failure signal when writes are already failing (consecutiveFailures > 0),
with no false positive on healthy backpressure (a drop while writes
succeed does not escalate).
- ORPHAN-CREATE REJECTION: on a write timeout the race rejects but the
orphaned `pb.create` keeps running; a later rejection from a client that
ignores the AbortSignal would be unhandled. Attach a no-op catch.
formatGauges had no production callers after the fix round removed the
gauge-string logging; only its own test referenced it. Delete the dead
function and the describe("formatGauges") block.
Add coverage (slot-6 F1) that `writeUnrecoverable` NAMES the proven wedge signal
in the Slack/alarm message when the give-up snapshot measured the cgroup PID
ceiling (cgroupPidsCurrent >= 0) — pids.current/pids.max + threads — and OMITS
the clause when the gauges are the off-Linux `-1` sentinel.
Documentation the diff invalidated, swept to match reality:
- browser-pool.ts JSDoc/comments + browser-pool.test.ts comments claimed
hard-recovery "purges the stale /tmp/playwright_* profile dirs". The code is
purge-free — hardRecover is a PACED cold relaunch that gives the
thread-exhausted kernel time to relax. Rewrite to match: paced cold relaunch,
NO /tmp purge (the wedge is the cgroup pids ceiling, mitigated demand-side;
/tmp is a proven non-cause).
- probe-invoker.ts stagger comment said "8 simultaneous workers", "~400 thread
spawns", "10-minute D6 budget" → updated to the current 5 / 20-min reality.
- e2e-deep.yml: the global cap note (40, "D6 peak 32 + D5 peak 8") → 24, with
D6 peak now 5×4=20; 20+8=28 > 24 so a d6+d5 overlap serializes under the
global cap (intended demand-side back-pressure).
- d6-all-pills-e2e.yml: the worked example (max_concurrency 8, ceil(18/8)=3
rounds) → 5 / ceil(18/5)=4 rounds, cross-referencing the overlap block that
justifies the concurrency itself.
Robustness/bounding nits a 7-agent CR flagged:
- browser-pool: the hot acquire/release path emitted a SECOND `hot-gauges` info
log duplicating inUse/available already on the acquire/release line. Fold the
two cheap cgroup-pid fields INTO the existing acquire/release log line (one
line, not two); keep the cheap two-file-read sampling (now `readHotGauges`).
- probe-invoker: `logTickGauges` double-encoded the sample as both a formatted
`gauges:` string AND the structured `...g` spread. Drop the redundant string;
keep the queryable spread.
- orchestrator: the best-effort Slack ping in `writeUnrecoverable` had no
timeout, so a hung webhook could stall the serialized health-signal write
chain. Add an AbortController timeout on the fetch.
The snapshot writer had several robustness gaps a 7-agent CR flagged:
- PRUNE TIES: the ring prune deleted `observed_at < cutoff` (strict), but
observed_at is ms-resolution and several snapshots can fire in the same
millisecond (degraded+heartbeat+crash). If surplus rows tied the boundary
timestamp, strict `<` deleted nothing → knownOverCap stayed true → every
insert re-ran a full no-op sweep → unbounded growth + per-insert cost. Now
the prune deletes the oldest `surplus` rows by stable row id, which converges
regardless of ties, with a deleted==0 safety valve that can't spin.
- RATE-LIMIT CATCH-UP: a local row-count estimate (re-anchored to the real
total every sweep) forces a catch-up prune the moment we cross the cap, so the
rate-limit only suppresses sweeps while UNDER cap and never strands us over.
- FAILURE ESCALATION: swallow+warn alone hid a systematic PB rejection (unrun
migration / outage / schema drift). Now consecutive failures are counted, the
per-failure warn is throttled, and after a threshold the writer escalates ONCE
to logger.error (latched, reset on first success).
- BOUNDING: a best-effort in-flight cap drops snapshots past N concurrent
unawaited creates (launch-crash-loop burst), and a Promise.race write timeout
frees the in-flight slot if a PB write hangs.
- NULL SENTINELS: the `-1` "unavailable" gauge sentinel is written as `null`
(PB number fields are nullable) so post-wedge queries separate measured-vs-
unavailable.
- DOCS: cadence prose + JSDoc updated to the real 45s heartbeat, and a
single-writer note that the fleet case needs an elected pruner.
Red-green tests added: identical-boundary-timestamp prune converges to maxRows;
rate-limit keeps prune off the steady-state path but knownOverCap catches up
over cap; N consecutive failures escalate to error once and reset on success;
`-1` gauge written as null.
browser-pool.ts imported `./resource-gauges` without a `.js` extension while
sibling files use `.js`. The harness builds with tsc (module: esnext) and runs
the emitted ESM directly (`node dist/orchestrator.js`); moduleResolution:
bundler lets it typecheck but tsc emits the specifier verbatim and Node ESM
does NO extension resolution, so the dist would fail at boot with
ERR_MODULE_NOT_FOUND. Append `.js` to the extensionless relative specifiers in
browser-pool.ts and resource-gauges.test.ts. Verified by building the dist and
boot-importing dist/orchestrator.js (BOOT-IMPORT-OK, no module-not-found).
The live wedge peaked 998/1000 against the cgroup pids.max=1000 ceiling
specifically during a d6+d5 overlap — the simultaneous chromium renderers of
both probes summed past the ceiling. Fewer concurrent d6 feature-workers cut
the simultaneous-renderer thread peak, leaving headroom for a co-firing d5 (or
a recovery relaunch) without crossing 1000. Complementary to the
MAX_CONTEXTS 40->24 lever already landed. Comment updated to the pids-ceiling
reasoning; 5 x 4 = 20 concurrent contexts stays under the 24-context cap.
Previously full gauges were logged on only 3 sites (launch, self-heal-launch-
failed, probe-tick). Extend forensic gauge coverage across the whole pool
lifecycle so a recurring wedge has copious history to look back at.
- Full forensic snapshot (sample + log + durable onSnapshot hook) on every
meaningful transition: init, recycle (crash/hygiene), crash/disconnect,
set-empty degraded, self-heal-launch-failed, recovered, unrecoverable, and
shutdown.
- Periodic heartbeat (~45s, env BROWSER_POOL_HEARTBEAT_MS) via a
shutdown-signal-gated self-rescheduling loop — NOT a raw setInterval that
leaks past shutdown — for a baseline trend between events.
- Cheap cgroup-PID-only subset (logHotGauges) on the hot acquire/release path:
high-resolution view of the headline pids.current/pids.max signal without
the full-sample /proc-walk + df cost, and no durable write on the hot path.
- onSnapshot is best-effort: a throwing hook is caught + logged, never crashes
the pool.
The browser-pool wedge ends in a container restart that clears in-memory
state, and Railway's stdout window rolls off — so stdout/in-memory gauges
are not retrievable post-wedge. Persist the OS resource gauges to a new
`resource_snapshots` PB collection so the forensic history survives the
restart and the PID/thread-ceiling exhaustion is reconstructable after.
- New PB migration `1779989300_create_resource_snapshots.js` (mirrors the
probe_runs idempotent create + public-read pattern).
- `resource-snapshot-writer.ts`: best-effort writer (swallows + logs every PB
error so a missing migration / PB hiccup never breaks the pool) with
ring-style retention (cap last N rows, default 5000, env-overridable).
- Orchestrator wires the writer to the pool's onSnapshot hook, reusing the
shared pb client.
The Dockerfile FIX#3 comment claimed the in-image `ulimit -u` raised the cgroup
pids.max (target 16384). It cannot: `ulimit -u` only lifts this process's
RLIMIT_NPROC; the cgroup pids.max is set by the container runtime
(--pids-limit), is platform-fixed at 1000 on staging, and is not raisable from
the image. Rewrite the comment to state this accurately and point at the real
demand-side mitigation (fewer concurrent contexts + gauge/alarm). The harmless
`ulimit -u` line itself is retained. Also update the env test to assert the new
MAX_CONTEXTS default of 24.
The wedge was reproduced as cgroup PID/thread-ceiling exhaustion (pthread EAGAIN
-> "has been closed" -> crash-loop with pids.current pegged at pids.max=1000).
Every /tmp candidate was refuted by measurement, so this removes the /tmp
machinery that targeted a non-cause and replaces it with the demand-side fix +
observability:
- Remove the proactive stale-profile sweep (init + recycle), defaultPurgeProfileDirs,
the PurgeProfileDirs type, the staleProfileDirTtlMs / PURGE_MIN_AGE_MS options
and env, and the breaker's /tmp purge — the hard recovery is now just the paced
cold relaunch. Corresponding purge/sweep tests removed.
- KEEP the circuit-breaker failure-counting -> give-up -> onUnrecoverable alarm
(root-cause-agnostic backstop) and the footgun clamp; the breaker tests are
adapted to drive the wedge via the launcher instead of the purge.
- Log the resource gauges on every launchBrowser(), on the self-heal-launch-failed
path, and at probe-tick start/end so a burst approaching pids.max is observable
and an EAGAIN correlates to a measured pids.current.
- Include pids.current/pids.max + thread count in the onUnrecoverable payload and
the orchestrator's terminal alarm message so the alert names the real signal.
- Lower the BROWSER_POOL_MAX_CONTEXTS default 40 -> 24 to cap thread demand under
the platform-fixed ceiling (still env-overridable).
Standalone, unit-tested gauge sampler that reads the cgroup PID controller
(pids.current / pids.max, v2 with v1 fallback) plus the process-tree thread/proc
counts, FDs, RSS, /dev/shm, and /tmp inode/space from /proc + /sys/fs/cgroup.
The cgroup PID counters are the headline signal for the proven browser-pool
wedge (PID/thread-ceiling exhaustion). Degrades gracefully to -1 off-Linux so it
is safe to import anywhere. Consumed by the browser pool and probe invoker in
follow-up commits.
The recurring browser-pool wedge's ROOT CAUSE: every chromium.launch() leaves
a /tmp/playwright_* profile dir + playwright-artifacts-* dir whose Playwright
close-time cleanup is best-effort (swallowed on failure). In a long-lived
harness container under sustained d6 cron load these orphans accumulate
monotonically until /tmp exhausts inodes/space — then every launch throws
"...has been closed" and self-heal relaunches into the same exhausted /tmp
forever (only a restart, which gives a fresh /tmp, clears it).
PREVENTION: sweep the age-gated stale dirs PROACTIVELY on a cadence — once at
init() and on EVERY recycle relaunch (crash + hygiene) — so /tmp never reaches
exhaustion. This is the durable complement to the circuit-breaker ESCAPE: the
sweep prevents the wedge; the breaker escapes it if it ever still happens.
Unifies the purge into ONE canonical helper — defaultPurgeProfileDirs now takes
an age threshold (minAgeMs) as its first arg. The proactive sweep passes the
conservative staleProfileDirTtlMs (5min default, env
BROWSER_POOL_STALE_PROFILE_DIR_TTL_MS); the breaker's hard-recovery keeps its
aggressive PURGE_MIN_AGE_MS (60s). Same hardened (symlink-safe, realpath-checked)
implementation, different age — no duplicate purge logic.
Adds 4 prevention tests (init-sweep, every-recycle-sweep, bounded-accumulation
over 30 cycles, real-FS age gate) and adapts the existing breaker tests to gate
their purge-count assertions on the breaker's age so the shared cadence does not
inflate them.
defaultPurgeProfileDirs (the self-heal hard-recovery profile-dir purge) ran a
blind readdir + rm(recursive, force) against the world-writable OS temp dir:
- A planted playwright_* SYMLINK had its TARGET recursively deleted. Now read
dirents (withFileTypes), SKIP symlinks + non-directories, and verify the
realpath stays under the scan dir before removing.
- A browser launching CONCURRENTLY with the purge had its FRESH profile dir
nuked, re-wedging the very launch being revived. Now only dirs older than
PURGE_MIN_AGE_MS (60s) are eligible (mtime via lstat).
The real purge previously had ZERO coverage (the pool tests inject a no-op).
Parameterize the scan dir so it is unit-testable against a temp fixture, export
it, and add tests: removes old playwright_*/playwright-artifacts-* dirs, leaves
non-matching dirs, SKIPS symlinks (target untouched), leaves too-fresh dirs, and
swallows ENOENT. (Also pulls in the resolvePositive helper used by the breaker
clamp, which lives adjacent to this function.)
Two circuit-breaker robustness fixes:
Latch second-episode silence: this.unrecoverable was cleared ONLY on a
successful launch, so a permanently-wedged container that re-empties after the
first give-up hit `if (this.unrecoverable) return` and went silent on every
subsequent degraded episode — the exact silent-spin the breaker exists to kill.
Remove the instance latch entirely; the loop-local consecutiveHardRecoveries
guard already gives once-per-episode (the loop returns immediately after firing),
and each fresh degraded episode spawns its own self-heal loop that can alarm.
Stale JSDoc updated to match. The onUnrecoverable hook now carries breaker
counters so the wired alarm can report how hard the pool tried.
Disable footgun: a selfHealHardRecoveryThreshold / selfHealMaxHardRecoveries of
0 silently disabled both the hard-recovery escape AND the give-up (guards are
`> 0`), reverting to the infinite silent spin. Clamp resolved breaker thresholds
to >= 1 (resolvePositive) so a config typo can't disable the safety net.
Tests: a second degraded episode ALSO fires onUnrecoverable (red→green: prior
latch silenced it); a 0 threshold stays armed (red→green: prior it spun
forever); plus a browsers:2 multi-browser test locking in the partial-revive
counter-reset / breaker pacing that was previously unverified.
The self-heal circuit-breaker's terminal give-up shipped a mechanism with no
operator signal — the pool's onUnrecoverable hook was never wired, so a
breaker give-up silently stopped spinning without telling anyone a redeploy is
required. Add createBrowserPoolHealthSignals.writeUnrecoverable and wire it to
the pool's onUnrecoverable hook:
- Unconditional health-signal writes: a DISTINCT system:browser-pool-unrecoverable
key (red, terminal, critical, redeployRequired + breaker counters) AND an
escalation of the shared degraded key to severity=critical/terminal, so a
give-up is distinguishable from a transient self-heal-degraded.
- Best-effort, env-guarded Slack ping on SLACK_WEBHOOK_BROWSER_POOL_UNRECOVERABLE
(unset by default per alerting discipline — deploy the code, wire the URL
separately); a failing/absent webhook never blocks the health-signal write.
The pool's onUnrecoverable hook now passes breaker counters so the alarm can
report how hard the pool tried before giving up.
The recurring staging BrowserPool collapse (#5185/#5221/#5225 each chipped at it but it kept recurring): after the long-lived harness container runs ~hours under sustained d6 cron load, chromium enters a launch crash-loop where every chromium.launch() throws 'Target page, context or browser has been closed'. The set empties, startSelfHeal() kicks in, and its loop just relaunches into the SAME wedged state forever (self-heal-launch-failed repeating) without ever escaping — the stale /tmp/playwright_* profile dirs / FD-shm pressure persist across every relaunch. acquire() therefore has no contexts forever and blocks to timeout fleet-wide; only a container restart cleared it (reactive).
Adds a circuit-breaker to the self-heal loop: after selfHealHardRecoveryThreshold (default 4) consecutive launch failures, instead of looping another identical relaunch the pool performs a HARD recovery — purges the stale ${tmpdir()}/playwright_* profile/temp dirs the crash-looping processes left behind, then cold-launches fresh. Any successful launch resets the counter. If selfHealMaxHardRecoveries (default 3) consecutive hard recoveries also fail, it fires a loud browser-pool.pool-unrecoverable alarm (onUnrecoverable hook) and stops the heal loop rather than spinning silently — the operator signal that a redeploy is genuinely required. All tunable via BROWSER_POOL_SELF_HEAL_* env vars; the purge is injectable for tests and defaults to a hermetic no-op when a fake launcher is injected.
`bin/showcase test --rebuild` was either a silent no-op or an error
depending on container state:
- runner.run() only rebuilt services in `autoStarted` (those NOT already
running), so --rebuild against an already-up service did nothing and a
stale container was silently reused (the 36h-stale-image false-positive).
- lifecycle.rebuild([slug]) ran `compose --profile <slug> build` WITHOUT
`--profile infra`, leaving `aimock` undefined → "service <slug> depends
on undefined service aimock: invalid compose project" when the service
was down.
Fix both:
- runner.run() now rebuilds every targeted slug when --rebuild is set,
regardless of running state, and health-checks rebuilt-but-already-running
services before probing.
- rebuild() includes `--profile infra` alongside the slug profiles (mirrors
up()) and force-recreates the targeted containers so the freshly-built
image is actually adopted.
Adds red-green unit coverage asserting the infra profile and force-recreate
in the rebuild compose invocation.
The d6-all-pills-e2e probe header was sized for a legacy "8 services x 4
features" single-round fleet (timeout_ms: 600000 / 10 min). Discovery now
enumerates ~18 showcase demo services, so at max_concurrency: 8 the fan-out
runs ceil(18/8) = 3 serialized rounds. Rounds 2-3 start late and execute under
CPU contention, stretching each service toward ~200s and blowing the 10-min
per-service budget — all 18 services went red with `driver timeout after
600000ms`.
Raise the outer cap to 1200000 (20 min), matching the sibling 18-service
e2e-demos probe (same fleet, same budget). 3 rounds x ~200s fits comfortably.
Timeout-only change: max_concurrency stays at 8, so D6 peak stays at 32
contexts (under BROWSER_POOL_MAX_CONTEXTS=40, leaving 8 for the offset D5
tick). Header comment rewritten to document the real fleet size and the
fan-out math.
The unmapped-starter aggregate emitted `total: 0` while `failed` listed all
four levels — a broken count invariant. Set `total: STARTER_LEVELS.length`
to match. Add the two missing red-green unit tests: the `aborted` errorClass
branch (already-aborted ctx.abortSignal → levels classified `aborted`, not
`transport-error`) and the makeSideEmit writer-throw resilience (a writer
whose write() rejects is swallowed at error-level and must not fail the
aggregate tick).
The starter_smoke probe runs hourly (`schedule: "40 * * * *"`), but
STARTER_STALE_AFTER_MS was set to 13h with a comment claiming a 6h cadence
— ~13 missed hourly ticks before amber, defeating the intended two-miss
flip. Re-derive the window from the 1h probe period to 2.5h: strictly
> 2 periods (so two consecutive misses, last row ~3h old, flip amber) yet
< 3h (so a single missed/slow-wake tick, last row ~2h old, stays green,
absorbing a scale-to-zero cold-start). Reconcile both the staleness.ts and
starter_smoke.yml comments to the same hourly basis + 2.5h window in
lockstep (no more "6h" in the starter context). Extend the staleness tests
with explicit hourly-tick boundaries (1 miss → green, 2 misses → amber).
Also drop the inaccurate "hollow" from the not-supported ✗ comments — there
is no hollow render variant; the state renders as grey ✗ text.
Adds the `starter_smoke` probe driver + config/probes/starter_smoke.yml that
fans out per-starter HTTP health/agent/chat/interaction levels, registers it
in the orchestrator, and exempts it from probe-config parity (its matrix
shape differs from the depth-dimension probes). Includes 13 driver unit tests.
Probe slot (S2) of the starter-row-group spec (model B): harness HTTP-probes
the deployed (sleepable) Railway starter services.
Introduces the `starter` probe dimension in harness types and the single
source of truth for the starter-slug→dashboard-column-slug remap
(STARTER_TO_COLUMN, starterToColumnSlug, STARTER_LEVELS) plus a drift-lint
test asserting every smoke-matrix starter is mapped or explicitly excluded
and every mapped column slug resolves to a real manifest directory.
Foundation slot (S1) of the starter-row-group spec (model B).
Commit 7ac3e59a5 ("D6 all-pills probe driver") replaced the
`e2eParityDriver` registration (kind `e2e_parity`) with `e2eFullDriver`
(kind `e2e_d6`) in BOTH branches of `registerAllProbeDrivers`. After that
commit NO driver registers kind `e2e_parity`, yet
`config/probes/e2e-parity.yml` still shipped.
The probe-loader hard-rejects any YAML whose `kind` has no registered
driver (`no driver registered for kind 'e2e_parity'`), so `e2e-parity.yml`
failed to load on every boot and was never scheduled — silently dropping
that probe family. The D6/parity dashboard dimension is now produced by the
`e2e_d6` (all-pills) driver, which emits the identical
`d6:<slug>/<featureType>` rows the dashboard reads by key prefix (plus a
`d6:<slug>` aggregate), so `e2e_parity` is genuinely superseded, not just
temporarily unregistered.
Delete the orphaned config so the loader no longer rejects it and the full
probe-config set loads clean. Add a loader test that exercises the REAL
shipped `config/probes` set against the REAL driver registry and asserts
every YAML loads with zero `probes.reload.failed` errors — the existing
orchestrator guard only checks a hardcoded kind list and never read the
on-disk YAMLs, so it missed this orphan.
RACE3b's post-shutdown `liveContextCount === 0` assertion is tautological —
shutdown() sets liveContextCount to 0 unconditionally — so it proves nothing.
Drop it; the load-bearing assertions remain (acquire rejects with the shutdown
sentinel, contextToBrowser.size === 0, the orphan context was closed).
Correct the RACE3a/RACE3b comments that claimed shutdown() awaits the tracked
orphan close (reverted): the straddled orphan is closed by the orphan guard's
fire-and-forget close and never lands in contextToBrowser.
Jordan Ritter