Mint a run_id per D5/D6 feature run, inject x-diag-run-id + seed x-diag-hops=harness alongside x-aimock-context, and do a post-run aimock-journal join (timeout-bounded) that records a cv-verdict row (tagged harness-d5/harness-d6) localizing whether the context header reached aimock. Add claim/worker_id/snapshot/pool-condition breadcrumbs and surface previously-swallowed catches.
Shared single-line CVDIAG log format (redacted 12-char header prefix), x-diag-run-id/x-diag-hops correlation header constants, and a best-effort PocketBase diag_events sink (anonymously HTTP-readable) so the CV/x-aimock-context propagation chain can be traced mid-incident without Railway log access.
Wire the resource-snapshot writer back into the fleet worker path with
per-replica attribution, add the worker_id column migration, and stamp
the legacy single-process path's worker_id as "" (empty string) to match
PocketBase storage and the surrounding codebase convention.
Make the showcase dev tool faithful to staging by construction. Two changes:
1. `showcase test --d5/--d6` now drives the fleet CONTROL-PLANE (producer ->
probe_jobs queue -> worker -> result-aggregator) instead of the legacy
in-process runLevel() driver. The new cli/control-plane-run.ts replicates
the deep/full producer tick exactly as runControlPlane wires it
(createE2eDeepServiceEnumerator / createServiceEnumerator over
createJobProducer + createFleetQueueClient), enqueues one operator-triggered
tick, and polls local PocketBase for the run's terminal cells. The running
worker fleet claims + runs the driver + the aggregator writes the d5/d6
status cells, so the dev tool exercises the IDENTICAL wiring + concurrency
as staging. The old in-process path stays available behind `--direct`.
2. `showcase up --dev` adds a docker-compose.dev.yml overlay that bind-mounts
each integration's source and overrides the run command with a stack-aware
hot-reload entrypoint (shared/dev/dev-entrypoint.sh: uvicorn --reload for
FastAPI agents, langgraph dev for graphs, next dev for the frontend). Edit a
source file and the component reloads in place with no image rebuild. The
built-image mode remains the faithful/staging-equivalent default.
The e2e_deep kind was removed (D5 now runs the D6 driver), leaving three
browser driver families: e2e_d6, e2e_demos, e2e_smoke. Update the stale
"four browser driver families" docstring and its test comment.
Asserts the real D5 invocation shape buildDeepInputs stamps (both knobs
together): only D5_REPRESENTATIVES featureTypes run AND every emitted key
(per-cell d5:<slug>/<ft> + aggregate d5:<slug>) uses the d5: prefix. The
existing tests cover the knobs in isolation only.
buildDeepInputs now forwards manifest.not_supported_features (matching D6's
buildFullInputs) so local CLI D5 runs don't false-red architecturally-
unsupported features. The D5 thrown-error terminal key changes from
d5-single-pill-e2e:<slug> to d5:<slug> (the driver's own emitAggregate key
shape) so a hard driver throw surfaces as a RED D5 cell, not a blank row.
Also corrects D5-scope docs: representativeOnly keeps the representative
featureTypes per D5_REPRESENTATIVES, not "one pill per category".
Removes E2E_DEEP_DRIVER_KIND and "e2e_deep" from the worker-internal
closed driver-kind set (D5 runs the e2e_d6 driver now), updates the
x-test-id-headers guard to assert on the surviving d6-all-pills driver
(d5-single-pill.ts was deleted), and repoints a stale doc comment.
Repoints the D5 probe at the unified D6 driver. The fleet D5
enumerator now stamps driverKind=e2e_d6 with representativeOnly + a
"d5" rowPrefix; the CLI's buildDeepInputs carries the same inputs;
config/probes/e2e-deep.yml declares kind=e2e_d6. Drops the e2e_deep
driver registration (orchestrator + worker registry + BROWSER_KINDS +
the worker-internal kind set), keeping the D5 producer schedule/cadence
intact. The worker now honors driverInputs.rowPrefix when filtering the
aggregate side-row out of captured cells so a "d5:<slug>" aggregate
doesn't leak into the D6 entry's "d6:<slug>" cell capture. This
eliminates the separate D5 launcher path whose own launcher instance +
cadence systematically dropped x-aimock-context against the shared fleet
pool (aimock strict 503 -> red).
Adds two input knobs to the d6-all-pills driver so it can run as "D5
take-one": `representativeOnly` filters the feature matrix to the
D5_REPRESENTATIVES set, and `rowPrefix` ("d5" | "d6", default "d6")
threads the dashboard key prefix through every emitted per-cell and
aggregate PB row. The representatives map is injectable for testing.
Everything else (route, headers, conversation, pooled launcher) is
unchanged. Red-green unit tests cover both knobs.
The control-plane runs the 8 in-process HTTP probe families but the
on-demand trigger endpoint (POST /api/probes/:id/trigger) was only mounted
on the legacy boot() path, so operators had no way to fire a family
immediately and had to wait on the slow cron. Wire the same
registerProbesRoutes onto the control-plane's buildServer using its own
httpProbeRegistry/httpProbeConfigs/scheduler/httpRunWriter and
OPS_TRIGGER_TOKEN. Only the prefixed in-process probe ids (probe:<id>) are
triggerable; browser-only and unknown ids 404. Token handling mirrors
boot() (unset -> router omitted; set-but-empty -> fail-loud).
Phase 2 of the harness pool-fleet migration — wire the three remaining BROWSER
probe families into the control-plane PRODUCER side so they actually run on the
fleet (the worker DriverRegistry for all four kinds already landed in #5283).
Unit A — three catalog-enumerator factories mirroring createD6ServiceEnumerator,
each delegating to the generic createServiceEnumerator with its own driverKind +
dashboard probeKey prefix and the shared D6_DISCOVERY_FILTER:
- createE2eSmokeServiceEnumerator → e2e_smoke / d4:<slug>
- createE2eDemosServiceEnumerator → e2e_demos / e2e-demos:<slug>
- createE2eDeepServiceEnumerator → e2e_deep / d5-single-pill-e2e:<slug>
R-timeout (demos): the demos driver's 20-min outer cap is threaded in-process via
the legacy E2E_DEMOS_TIMEOUT_MS env, which the fleet worker never sets. The demos
enumerator now conveys the YAML timeout_ms per-job in driverInputs.timeout_ms
(new E2E_DEMOS_TIMEOUT_MS SSOT const mirroring e2e-demos.yml), and the demos
driver reads input.timeout_ms as a resolution source (env > input > deps >
default) so the 38-demo service no longer blows the 5-min default to all-red.
Unit B — runControlPlane now builds four producers and passes a multi-schedule
manifest (buildProducerSchedules) to createControlPlane, each family on its own
cron read literally from the config YAMLs (the deliberate offsets stagger the
families' Playwright fan-outs on the shared BrowserPool):
- fleet-job-producer 40 * * * * (d6, unchanged; honors FLEET_PRODUCER_CRON)
- fleet-producer-e2e-smoke */15 * * * *
- fleet-producer-e2e-demos 10 * * * *
- fleet-producer-e2e-deep 5,20,35,50 * * * *
The in-process HTTP probe runner and the d6 producer's REQ-B sweep leg are left
intact (additive). The worker registry is untouched.
R1 (verified, no change): both createPooledE2eSmokeLauncher and
createPooledE2eDeepLauncher thread contextOpts.extraHTTPHeaders, so smoke + deep
set their per-slug X-AIMock-Context themselves.
Address the CR gaps on the in-process HTTP-probe control-plane:
- Sweep orphaned `running` probe_runs at control-plane boot (boot()'s
sweepStaleRuns never ran in fleet mode → leaked rows forever). Best-effort;
a sweep failure does not abort boot.
- Add a fail-loud BROWSER_KINDS / HTTP-driver disjointness assert at boot plus
a drift-lock test mirroring registerAllProbeDrivers, so a mis-added kind
can't silently go dark.
- Assert the PR's headline guarantees that were unasserted: cron-from-YAML
(exact values), /health loader-failure boot-survival + probes.reload.failed
emit + rules=0 observability, hot-reload add/remove + watcher teardown, a
discovery-backed family (qa) in the in-process schedule, and tightened
/health rule/job counts to exact equality.
- Bump the includeKind skip log to info; make diffHttpProbeSchedules'
unregister-failure post-state explicit (keep config observable); correct the
/health "no longer masks" comment, the discovery-source comment
(image_drift uses railway-services; version_drift uses pnpm-packages), the
reload-failed surface comment (no bus subscriber on the control-plane), and
drop transitional-rot tags.
The fleet control-plane previously ran only the d6 producer, leaving the 8
HTTP-only probe families (smoke, starter_smoke, image_drift, qa, aimock_wiring,
version_drift, pin_drift, redirect_decommission) dark on the fleet. Lift the
legacy boot() probe-loader machinery into runControlPlane so those families run
in-process:
- Add BROWSER_KINDS = {e2e_d6, e2e_smoke, e2e_demos, e2e_deep}; HTTP = every
other kind. Add registerHttpProbeDrivers (HTTP-only driver set, no BrowserPool
drivers).
- In runControlPlane, build an HTTP-only probeRegistry + discovery registry
(railway-services cached + pnpm-packages, mirroring boot()), a probe-loader
scoped to HTTP kinds, and the same diffProbeSchedules/buildProbeInvoker loop
boot() uses — registering one probe:<id> scheduler entry per YAML config.
Crons are driven from the YAML schedule. Browser e2e_* YAMLs route to the
worker producer path and are NOT scheduled in-process.
- Add an includeKind predicate to createProbeLoader so a browser YAML on disk is
SKIPPED (not rejected) against the HTTP-only registry — no spurious
probes.reload.failed.
- /health: ruleCount now reflects the in-process HTTP probe count (was a
hardcoded 0). The control-plane role still drops the rules>0 gate, but the
real count means a silent probe-loader failure is visible on /health rather
than masked. schedulerJobCount already counts the new probe entries.
- Tear down the HTTP-probe file watcher on stop() and on bind failure.
## Summary
Producer-side foundation for fleet framework item 3b. Two
**behavior-preserving** generalizations that make the seam capable of
multiple browser families and multiple producer cadences, while keeping
the d6 case **byte-identical**. No wiring is flipped on yet (see Out of
Scope).
### 1. Parameterized service enumerator
`showcase/harness/src/fleet/control-plane/catalog-enumerator.ts`
- New generic `createServiceEnumerator(params)`
(catalog-enumerator.ts:215) takes the service-set `filter`, the
`driverKind`, and a `probeKeyPrefix` (string prefix → `<prefix>:<slug>`,
or a builder fn).
- `createD6ServiceEnumerator` (catalog-enumerator.ts:280) is
re-expressed as a thin call passing the d6 params: `D6_DRIVER_KIND`
(`e2e_d6`), prefix `"d6"` (→ `d6:<slug>`), and `D6_DISCOVERY_FILTER`. d6
output is unchanged — same services, same filter, same kind, same keys.
### 2. Control-plane accepts an array of producer schedules
`showcase/harness/src/fleet/control-plane/control-plane.ts`
- New `ProducerSchedule` type (`{ scheduleId, cron, producer }`) + a
`schedules?` dep on `ControlPlaneDeps`.
- `createControlPlane` normalizes to an array (control-plane.ts:~232);
omitting `schedules` degenerates to the single d6 schedule on
`FLEET_PRODUCER_SCHEDULE_ID` (`fleet-job-producer`) @ `40 * * * *` —
current behavior preserved exactly.
- `start()` / `stop()` iterate the array, registering/unregistering each
scheduler entry and starting/stopping each producer.
## Out of scope (deferred — gated on other in-flight PRs)
- **No `runControlPlane` wiring** to actually PASS multiple schedules —
that edit conflicts with in-flight **#5284** (which edits
`runControlPlane`) and is deferred. This PR only makes
`control-plane.ts` *capable* of N schedules + generalizes the enumerator
seam; the wiring lands later.
- No `e2e_smoke` / `e2e_demos` / `e2e_deep` enumerators or producers
(Phase 2).
- No changes to `worker-loop.ts` / `payload-mapper.ts` /
`probe-loader.ts` (other PRs own those).
- No driverKind constant / contract changes.
## Test plan
- [x] Red→green TDD: 3 new enumerator tests (generic
kind/keys/filter/fn-prefix) + 2 new control-plane tests (N entries
registered with distinct crons; stop tears all down) failed before impl,
pass after.
- [x] Equivalence: all pre-existing d6-enumerator + single-schedule
control-plane tests pass unchanged.
- [x] Full harness suite green: **2078 passed** (119 files).
- [x] `tsc -p tsconfig.build.json` clean (exit 0).
- [x] Only the 4 intended files changed; no lockfile drift.
Do not merge — producer-side foundation only; wiring follows after #5284
lands.
The multi-schedule seam (createServiceEnumerator + the schedules[] capability
in control-plane) didn't meet the file's own best-effort/fail-loud bar. Harden
it now since Phase 2 builds on it (no production caller yet):
- stop(): guard each producer.stop() per-entry so one rejection no longer aborts
teardown of later schedules (leaked cron handlers + running producers).
- start(): pre-validate every schedule's cron up-front before starting any
producer, throwing an aggregated error naming the offending scheduleId — no
more half-started plane with `started` latched true.
- normalization: throw on duplicate scheduleId (replace-semantics would silently
collapse two producers onto one entry) and on an explicitly-empty schedules:[]
(distinct from omitted, which keeps the d6 default).
- createServiceEnumerator: require a non-empty filter.namePrefix (an absent
prefix would enumerate ALL services) and reject an empty probeKey from a
function-form prefix, naming the slug.
Also: narrow the d6 "byte-identical" docstring (specs identical; the
catalog-enumerated log adds driverKind), pluralize the start()/stop() +
module-header producer comments, drop the Phase 2 marker, and extract the shared
ServiceSetFilter type.
Producer-side foundation for fleet item 3b — two behavior-preserving
generalizations, byte-identical for the d6 case:
1. Generalize the d6 service enumerator into a parameterized
`createServiceEnumerator(params)` carrying the service-set filter, the
driverKind, and the probeKey prefix builder. `createD6ServiceEnumerator`
is now a thin wrapper passing the d6 params (e2e_d6 kind, d6:<slug> keys,
D6_DISCOVERY_FILTER), so d6 behavior is unchanged.
2. Generalize createControlPlane to accept an array of
{ scheduleId, cron, producer } entries and register each on the scheduler.
The single-d6 case degenerates to a one-element array on
fleet-job-producer @ 40 * * * *, preserving current behavior.
Out of scope (gated on in-flight PRs): orchestrator runControlPlane wiring
to pass multiple schedules (conflicts with #5284), the e2e_smoke/demos/deep
families (Phase 2), and worker-loop/payload-mapper/probe-loader.
CR fixes for the fleet worker driverKind→driver registry (PR #5283):
- Fix default (self-contained) worker boot: build the default d6 as a
registry entry { driver, payloadToInput, aggregateSlugKey } instead of a
bare driver with no mapper, so startWorkerLoop's construction guard no
longer throws "Fleet worker has no drivers".
- Thread aggregate-key derivation through DriverRegistryEntry
(aggregateSlugKey?), defaulting to d6:<slug> so the d6 cell-capture filter
stays byte-identical while non-d6 kinds can supply their own scheme.
- Add construction-time fail-loud assert that each registry entry's factory
kind matches its key; raise unknown-driver-kind log to error to match the
sibling protocol-violation logs.
- Extract shared buildPooledBrowserDrivers consumed by both
registerAllProbeDrivers and the worker registry; collapse no-op per-kind
payload-mapper aliases to the single createPayloadToInput.
- Introduce typed DriverKind union (contained to worker/payload-mapper);
keep contracts.driverKind a string wire boundary.
- Tests: new fleet/orchestrator.test.ts (default-boot equivalence), driver
construction-guard + custom aggregateSlugKey coverage, and lock-step +
registry-wiring pins (factory kind == constant).
Generalize the fleet worker from a single hardwired d6 driver into a
driver REGISTRY keyed by payload.driverKind, so one worker can host all
four browser driver families (e2e_d6, e2e_deep, e2e_demos, e2e_smoke).
- worker-loop: accept a `drivers: Map<kind, { driver, payloadToInput }>`
and dispatch each claimed job by `payload.driverKind`. Unknown kind →
terminal `worker-protocol-violation` (same shape as an unmappable
payload), never a worker crash. Legacy single `driver`+`payloadToInput`
pair retained as a fallback for back-compat. Fail-loud at construction
when neither a registry nor the legacy pair is supplied.
- payload-mapper: generalize createD6PayloadToInput into a shared
createPayloadToInput plus per-kind aliases and the four driver-kind
constants (the input re-hydration is identical across families; each
driver's own zod schema is the validation gate).
- orchestrator runWorker: build all four pooled drivers on the shared
BrowserPool and register them by kind, lifted from the legacy
registerAllProbeDrivers pooled construction.
d6 routing is unchanged (equivalence gate). Red-green tests cover
routing e2e_smoke/e2e_deep/e2e_d6 to their drivers and unknown-kind →
protocol-violation. Full harness suite green (2085 tests); tsc clean
except the known pre-existing toReversed error.
The harness starter-smoke CHAT rung 503'd on staging because its raw fetch()
POST sent only Content-Type — omitting X-AIMock-Context. The scoped
per-integration "Hello" fixtures in showcase/aimock/d4/<integration>/chat.json
only match (under aimock strict mode) when the request carries
X-AIMock-Context:<context>, and that context token IS the dashboard column
slug (verified against each integration's
showcase/integrations/<col>/playwright.config.ts extraHTTPHeaders and the
fixture's match.context). The local browser e2e passes only because Playwright
injects the header, forwarded by HeaderForwardingMiddleware to aimock.
Send X-AIMock-Context: <columnSlug> on the chat POST only (matching the
browser; the GET rungs hit the runtime /info route, not aimock). columnSlug is
already resolved via starterToColumnSlug before the chat rung runs, so the skew
cases (langgraph-js->langgraph-typescript, adk->google-adk,
strands-python->strands, ms-agent-framework-*->ms-agent-*) get the correct
context for free.
Proven RED->GREEN against live staging aimock
(https://aimock-staging.up.railway.app): without the header -> 503
no_fixture_match; with the mapped column-slug context -> 200 matching the
scoped fixture.
## Summary
Rewrites the `starter-smoke` probe to the **v2 path-based multi-route**
runtime protocol the deployed starters actually speak — proven via a
local build+curl gate against `examples/integrations/crewai-crews` at
1.59.5 (the runtime mounts `createCopilotEndpoint` in default
`mode:"multi-route"`, identical at 1.59.3 and 1.59.5).
- **agent** → `GET /api/copilotkit/info` requiring 200 + `version` (was:
"any non-404", which a health-JSON/HTML error page wrongly passed)
- **chat** → `POST /api/copilotkit/agent/<id>/run` (`Accept:
text/event-stream`), asserting ≥1 `TEXT_MESSAGE_CONTENT` delta **+**
terminal `RUN_FINISHED` **+** no `RUN_ERROR` (was: "non-empty body")
- **health** → repointed to `GET /api/copilotkit/info` (the starters
serve no `/api/health` route)
- **interaction** → `GET /` (unchanged)
- **drop the trailing slash** on runtime POSTs — `POST /api/copilotkit/`
308-redirects and drops the POST → 404, the core cause of the red rungs
Plus a **unified abort/body-read error classification** (one
`abortOutcome()` helper keyed on the local `externallyAborted` flag): a
self-timeout or a non-abort error → `transport-error` (soft), only a
genuine external abort → `aborted`; a cut-short body read softens to
`transport-error` instead of hard `smoke-failed`; the level loop
short-circuits on external abort. Reviewed across 3 CR rounds
(path-based rewrite + 2 abort-classification fix rounds + confirmation).
> **NOTE:** the content-level chat rung will correctly read **RED**
against the deployed starters until the Python-side `ag-ui-crewai`
`RUN_ERROR` (a real-LLM, in-process agent failure — *not* a probe bug)
is fixed. That is the truthful signal.
## Deferred follow-ups (tracked, not in scope)
- **Verify the alert/staleness consumer branches on `errorClass`, not
`state`** — else the soft/hard (`transport-error` vs `smoke-failed`)
split is cosmetic at the dashboard layer
- `res.text()` resolve-partial (truncated-but-resolved body) precision
edge → could mis-class as `smoke-failed`
- `redirect:"follow"` POST→GET edge; health/interaction success-path
body drain; health/agent share `/info` (correlated signals)
- `CLASS_RANK` ranks `aborted` lowest (design call); unmapped-starter
`columnSlug` invariant + prototype-key collision
- test-coverage gaps: `resolveTimeoutMs` env parsing,
`deriveStarterSlug` fallback, real-timer abort path
## Test plan
- [x] Harness unit suite — 1829 tests pass (105 files); starter-smoke
35/35; red-green verified for protocol + all abort/body-read fixes
- [x] Typecheck (no new errors), oxlint (0 errors), oxfmt, harness build
- [ ] Post-merge: redeploy harness, re-probe; chat rung greens once the
Python-side RUN_ERROR is fixed (separate workstream)
The chat rung hardcoded `/api/copilotkit/agent/default/run`, which 404s for
mastra — it registers dynamic non-`default` agent keys via
`MastraAgent.getLocalAgents` rather than `agents:{default}`. Resolve the
agent id per-starter from the first key of the `/info` `agents` map (the same
info response the health + agent rungs already fetch), falling back to
`default` only when that map is empty/unreadable, and surface the resolved id
on the chat row signal for drilldown.
Rewrite the starter-smoke probe to the v2 path-based multi-route runtime
protocol the deployed starters actually speak (proven via a local
build+curl gate against examples/integrations/crewai-crews at 1.59.5):
- agent rung: GET /api/copilotkit/info -> 200 + version (was: any non-404)
- chat rung: POST /api/copilotkit/agent/<id>/run (Accept: text/event-stream)
asserting >=1 TEXT_MESSAGE_CONTENT delta + terminal RUN_FINISHED + no
RUN_ERROR (was: non-empty body)
- health rung: repointed to GET /api/copilotkit/info (the starters serve
no /api/health route)
- interaction rung: GET / (unchanged)
- drop trailing slash on all runtime POSTs (the old /api/copilotkit/ 308s
and drops the POST -> 404, the core cause of the red rung)
Plus a unified abort/body-read error classification (one abortOutcome()
helper keyed on the local externallyAborted flag): a self-timeout or a
non-abort error is transport-error (soft), only a genuine external abort
is 'aborted'; a cut-short body read softens to transport-error instead of
hard smoke-failed; the level loop short-circuits on external abort.
NOTE: the content-level chat rung will correctly read RED against the
deployed starters until the Python-side ag-ui-crewai RUN_ERROR (a
real-LLM, in-process agent failure, NOT a probe bug) is fixed.
Deferred follow-ups: verify the alert/staleness consumer branches on
errorClass (not state) so the soft/hard split is actually consumed;
res.text() resolve-partial precision edge; redirect:follow POST edge;
health/interaction body-drain; health/agent shared /info; CLASS_RANK
aborted-lowest; unmapped-starter columnSlug/prototype-key; test-coverage
gaps (resolveTimeoutMs, deriveStarterSlug, real-timer abort).
The starter_smoke probe registers its driver under kind `starter_smoke`
(starterSmokeDriver, orchestrator.ts) and config/probes/starter_smoke.yml
declares `kind: starter_smoke`, but the probe-config Zod `kind` enum is
built from DIMENSIONS, which only listed the `starter` emit-prefix and not
the `starter_smoke` probe kind. The enum therefore rejected the YAML at
parse time; because `kind` is shared across all three union variants the
failure surfaced as the misleading "Unrecognized key(s) in object:
'discovery'" union error, the probe never loaded, AND probe-loader.test.ts's
shipped-config assertion failed (the lone recurring failure).
Add `starter_smoke` alongside `starter`, mirroring the existing kind/emit
pairs (e2e_d6/d6, e2e_deep/d5, e2e_demos/e2e). `discovery` was a red herring
— it is a valid key the DiscoveryBlockSchema already accepts; the working
d6-all-pills-e2e.yml uses the identical discovery shape.
Red-green coverage for the writer-hardening fixes:
- in-flight cap DROP emits dropped-over-inflight and the counter does not leak
- a hung create times out, counts as a failure, and escalates after the threshold
- once writes are timing out, concurrent drops contribute to the escalation
- a drop while a write succeeds (healthy backpressure) does NOT escalate
- a no-progress deleteByFilter over cap backs off (gated by interval, not per insert)
- a cold start far over cap prunes in bounded batches (<= 200), never a thousands-row list/filter
- a hung Slack webhook is aborted at the timeout so the serialized health-signal chain isn't stalled
Three durable-logging robustness fixes in the resource-snapshot writer:
- PRUNE BATCH CAP + NO-PROGRESS BACKOFF: a cold start over a large backlog
(or lowering RESOURCE_SNAPSHOT_MAX_ROWS on a full collection) made
`surplus` thousands large, used directly as a `perPage` and a
one-clause-per-row id-delete filter — a giant request/URL that PB
rejects (400/414) so the prune never converged. Cap each sweep to
PRUNE_BATCH_MAX (200) oldest rows and let knownOverCap drain the rest
across inserts. When a sweep makes NO structural progress while still
over cap (PB delete-rule 403, filter 400, or 0 matched), back off via a
suppression window so the retry is gated to once-per-interval instead of
re-arming knownOverCap and re-running the failing sweep on every insert.
- DROP→ESCALATION SEAM: an in-flight-cap DROP did not count toward the
consecutive-failure escalation, so a 100%-broken PB whose writes hang
could keep new snapshots on the DROP path and delay/dodge the loud
`write-failing-systematically` error. A drop now contributes to the
failure signal when writes are already failing (consecutiveFailures > 0),
with no false positive on healthy backpressure (a drop while writes
succeed does not escalate).
- ORPHAN-CREATE REJECTION: on a write timeout the race rejects but the
orphaned `pb.create` keeps running; a later rejection from a client that
ignores the AbortSignal would be unhandled. Attach a no-op catch.
formatGauges had no production callers after the fix round removed the
gauge-string logging; only its own test referenced it. Delete the dead
function and the describe("formatGauges") block.
Jordan Ritter