A single 50-row page per sweep was far slower than the incident the
stale-pending drain exists for: against the 3,734-row staging backlog at
~10 sweeps/hour that is ~7.5 hours of drain. The sweep now loops candidate
pages (re-listing page 1 — deletes shift pagination) up to a cap of 10
pages / 500 rows per sweep, draining the same backlog in well under an
hour while bounding a single sweep's PB load. CAS-claim-then-delete per
row is unchanged; a pass that expires nothing terminates the loop.
The sweepExpired lease phase listed claimed/running rows with perPage 50
and NO sort: with >50 such rows (mass worker crash), PB's unspecified
default order could return the same 50 live-lease rows every sweep,
leaving expired leases beyond the page permanently orphaned with zero
signal. Sort by lease_expires_at ascending (indexed) so the most-expired
rows always head the page, and WARN when the page is full so truncation
is observable. Single page per sweep is kept deliberately — the sort
guarantees progressive forward drain.
When the stale-pending sweep CAS-claims a row under stale-pending-sweeper
and the delete fails, the next lease sweep treated the expired sweeper
lease like a crashed worker's: it re-queued the row AND synthesized a
worker-reclaimed-pending comm error — a gray "re-queued / back in flight"
dashboard overlay for stale garbage mid-deletion, attributed to a
non-existent worker. The lease sweep now special-cases rows held by the
stale-pending sweeper: still re-queued (the self-healing delete-retry
contract is unchanged) but silently — no comm error, no reclaimed count,
just a stale-sweeper-retry-requeue debug line. The lease holder is
snapshotted before the release CAS so attribution reflects who held the
expired lease, not the post-release row.
sweepExpired's lease phase re-queues an expired-lease row to pending and
emits worker-reclaimed-pending ("back in flight"), but the stale-pending
phase of the SAME call lists pending fresh and ages rows off PB's system
`created` (the ORIGINAL enqueue time) — so a long-claimed job was
re-queued then immediately claimed-and-deleted, falsifying the comm
error and nulling downstream aggregate-key resolution on the deleted
row. Track the ids re-queued in this sweep and exclude them from this
call's stale phase; a truly stale job ages out on the next sweep. The
`created` anchor is kept (re-anchoring needs a column; out of scope).
sweepExpired only reclaimed claimed/running leases — a pending row had
no terminal path, so an accumulated backlog (staging: 3,734 pending,
oldest 22h) could only drain through 2 serial workers and effectively
never did.
The sweep now also expires pending jobs older than expiryPeriods x
their family's production period (default 3 periods; the family has
enqueued fresher batches since, so the stale job's result would be
ancient data). Each stale row is first CLAIMED via the S0 CAS under a
synthetic stale-pending-sweeper id — so the delete can never race a
worker (exactly-one-winner) — then deleted; a failed delete self-heals
via normal lease expiry + re-queue. Unparseable created timestamps are
conservatively skipped (delete is destructive). Policy is configurable
via FleetQueueClientConfig.stalePending; the control-plane wires the
real per-family cadences (FLEET_FAMILY_PERIODS_MS: d4/d5 15min, d6/
e2e-demos hourly) so 15min families expire on a 45min window. No comm
error is synthesized for expired-pending rows (they never ran); the
count surfaces as SweepResult.expiredPending and in the producer's
sweep log.
Producers enqueued a fresh batch every scheduled tick regardless of
whether the family's previous batch had even been claimed — against 2
serial browser workers the queue compounded without bound (staging:
3,734 pending), feeding the claim-page starvation.
A scheduled tick now skips its family's batch when that family already
has pending (unclaimed) jobs, bounding the per-family backlog to one
batch, with a structured fleet.producer.skipped-for-backlog log
(family, pendingCount, skippedJobs) and a skippedForBacklog count on
TickResult. The check is per family, fails OPEN on a count blip (a PB
read failure must never stop production), and is BYPASSED by
operator-triggered ticks (explicit intent wins; the trigger CLI treats
0 enqueued as failure). Backed by a new
FleetQueueClient.countPendingForFamily (server-side totalItems count of
a family's pending rows).
claimNext listed ONE global oldest-50 pending page; with the d4+d5
producers ticking every 15min against 2 serial browser workers, a
persistent backlog permanently saturated that page and e2e-demos jobs
never entered the candidate set (prod: all 18 e2e-demos jobs pending
forever; staging: 3,734 pending, oldest 22h).
claimNext now discovers the distinct families present in pending
(oldest first, one perPage=1 query per family) and tries them in
round-robin rotation across calls, listing a per-family candidate page
for each. Every discovered family is attempted before giving up, so no
family starves while any of its jobs are claimable. The S0 CAS
exactly-one-winner semantics and the per-page anti-herd shuffle are
unchanged; only candidate SELECTION changed.
## Summary
End-to-end hardening of the showcase shell's URL plane:
- Carries `backendHostPattern` + `docsHost` in the shell runtime config
(no longer baked from `registry.json` at Docker build time). Derives
demo backend URLs at runtime from the pattern, so a new pattern via env
reconfigures every integration on the next deploy without a registry
rebuild.
- Issues docs-host 301/308 redirects from middleware with a runtime
`DOCS_HOST`; misconfigured values no longer 500 every docs route — they
fall back to a sentinel that disables the docs-redirect step.
- Hardens the redirect table builder + matcher: first-match-wins for
duplicate exact sources, deduped wildcard prefixes with warn, malformed
entries rejected at lookup-build time, case-insensitive matching parity,
trailing-slash normalization, structural `/integrations` namespace guard
above the docs-host redirect (closes R15/R17 hijack class).
- Hardens runtime-config + backend-url env readers:
scheme/whitespace/control-char normalization, query/fragment/userinfo
rejection on `DOCS_HOST` / `POSTHOG_HOST` / backend-host pattern /
local-override URLs, present-but-empty `posthogKey` rejection, prod
loopback `BASE_URL`/`DOCS_HOST` rejection (no silent `http://` prepend),
once-guarded FATAL logging (no per-request spam).
- Brings the build-time twin in `showcase/scripts/generate-registry.ts`
to parity with the runtime normalizer (scheme/trailing-slash strip,
degenerate fallback, `NEXT_PUBLIC` fallback, slug validation).
- Surfaces PostHog capture failures once per failure class; keeps
capture alive across the redirect via `event.waitUntil`; missing
`POSTHOG_KEY` is `console.error` in production and surfaces at
config-resolution time (not first redirect).
- Open-redirect hardening on `/shared//evil.com` (SU-18); `//` rejected
at both source and destination; root `/` exact-source and `/:path*`
wildcard sources rejected; non-printable-ASCII source/destination
rejected.
Stream: SU (shell-runtime-urls). Subject groups
SU-2/8/11/13/14/15/16/17/18/19/20 (initial), SU2-A/B (env-hardening),
CR2-C (test infra), SU5-A1..A7 (registry/builder/lint/matcher boundary),
SU6-A1..A6 (request-time normalization parity), SU6-B1..B7
(parsed-normalized return forms, SHOWCASE_LOCAL states, generator
`{slug}` validation), SU7-F1..F3 (final-round backend-pattern + POSTHOG
+ table + script-side parity).
## Test plan
- [x] `pnpm test` in `showcase/shell` — 7 files, 235/235 pass
- [x] `pnpm test` in `showcase/scripts` — 51 files, 1857/1857 pass (one
pre-existing flake in `generate-registry.test.ts` unrelated to this
diff; passes on subsequent runs, classic vitest fork-reuse cross-file
pollution)
- [x] `pnpm exec tsc --noEmit` in `showcase/shell` — clean
- [x] `oxlint` on every changed file — 0 warnings, 0 errors
- [x] `oxfmt --check` on every changed file — clean
- [x] `pnpm build` in `showcase/shell` (full Next 15.x build with
registry+demo-content+starter-content+search-index generation) — clean
- [ ] CI green on the PR — confirm via `gh pr checks` after push
Round-by-round CR convergence covering the redirect builder, the middleware
matcher, the docs-host self-loop guard, and the runtime-config env readers.
Highlights:
- Clear module-load warns after fresh middleware import
- Validate SET BASE_URL values (scheme-less/degenerate/garbage) with
sentinel fallback + once-guarded FATAL log
- Normalize path/query/fragment-bearing DOCS_HOST to origin; reject
non-http(s) schemes; branch rejection reasons
- Harden POSTHOG_HOST (degenerate-host/scheme rejection); expose
posthogKey via readEnvPair semantics
- Reject a DOCS_HOST equal to the shell's own host (redirect-loop guard,
authority compare)
- Warn on missing local-ports.json under SHOWCASE_LOCAL=1 and validate
TCP port range; extract helper for tests
- backend-url hardening — slug charset guard, frozen local-backends memo,
pattern path-segment warn, local-override URL validation
- Client config fail-loud covers all four URL fields with type checks
- Make RuntimeConfig.posthogKey optional — absence is a valid state, not
a wiring bug
- Drop R15/R17 and guard /integrations from SEO redirects
- Dedup duplicate wildcard prefixes with first-match-wins warn
- Validate malformed SEO entries at lookup-build time
- Restore case-insensitive redirect matching parity
- Normalize trailing slashes before redirect matching
- Keep the framework segment on F13, pin MG3 case fix
- Read posthogKey from runtime config in middleware, not raw process.env
- Fall back to the default backend host pattern for degenerate values
- Disable docs redirects when the default fallback collides with the
shell host
- Bring validateBaseUrl to parity with its sibling readers
- Strip query/fragment from POSTHOG_HOST while keeping reverse-proxy paths
- Restrict local backend overrides to http(s) URLs
- Add server-only guard to runtime-config
- Harden localBackendsEnv failure posture
- Hoist /integrations namespace guard above the docs-host redirect
- Validate seo-redirect sources and cross-kind shadowing in
buildRedirectLookup
- Unify slash normalization for middleware matching
- Lowercase-normalize REGISTRY_FRAMEWORK_SLUGS at construction
- Escalate missing POSTHOG_KEY to console.error in production
- Skip all redirect steps when docs redirects are disabled (sentinel
consumer)
- Reject userinfo credentials in DOCS_HOST, POSTHOG_HOST, and the backend
host pattern
- Branch dev-vs-prod logging in readDocsHost and fatalPatternOnce
- Prepend http:// (not https://) to scheme-less loopback hosts
- Round-5 micro-finding batch across the URL config libs
SU5-A1..A7 — registry safety, // reject, builder lint batch (case-
insensitive :path*, same-destination twin allowlist, original-case
divergence remainder), matcher api boundary, generator+vitest infra, test
hygiene + empty docs-host guard, comment batch.
SU6-A1..A6 — reject miscased :path* tokens, warn on tokenless wildcards,
normalize redirect-destination comparisons like request time, reject
destinations containing "//", surface missing POSTHOG_KEY at config-
resolution time, compile matcher harness like Next's runtime, type
parse/tokensToRegexp in the path-to-regexp shim, keep buildRedirectLookup
JSDoc attached.
SU6-B1..B7 — reject query/fragment/userinfo in pattern and local-override
URL gates, return parsed-normalized URL form from validation success
paths, distinguish unset/blank/padded SHOWCASE_LOCAL states, warn when
SHOWCASE_LOCAL is set to a value other than 1, validate {slug} placeholder
in generate-registry, mirror middleware drop semantics in the wiring
test's registry re-derivation, pin the noStore spy and calls to one fresh
module instance in the Edge-path test.
Wire the legacy monolith scheduler's status writer with an explicit
writtenBy:"legacy" identity, correct the dual-writer dedupe comments
to describe the real upsert-collapse (and why it is not safe), stamp
the alert engine's synthesized cron outcome persisted:false honestly,
and align alert/orchestrator/probe test fixtures with the real
StatusWriter and OverlayWriteOutcome contracts.
Persist thrown driver errors to PB in the runDriverInputs catch,
carry WriteOutcome discriminators through the CLI summary (dropped
write counts with correct pluralization, no duplicate cause
clauses), and pin the runner/results behavior with StatusWriter
contract-typed stubs so writer contract drift is compile-checked
at the stub site.
Normalize padded projected keys to trimmed canonical form, skip
blank keys loudly, and replace ambiguous outcomes with explicit
discriminators: honest outage-skip/empty-projection semantics,
droppedCommError surfaced whenever the comm error misses the
aggregate row, trusted-negative duplicate preference scoped to
cell-vs-cell (no aggregate-row impersonation), and comm-error
identity asserts converted to discriminators so the consumer
cannot hot-loop.
Pin the writer-identity stamping, flip/foreign-write warn paths
(TTL re-warns, self-write memory cap eviction), date normalization
shapes, overlay write outcomes, idempotency latches, and error
classification against fail-loud fake-PB fixtures hardened against
silent divergence from real PocketBase behavior.
Add written_by/state_written_at columns (PB migrations) and stamp every
status write with a stable host-derived writer identity. The status
writer now detects cross-writer state flips and foreign writes (the
anti-dual-writer flap-comb defense), normalizes observedAt to PB-safe
RFC-3339 shapes before date-field writes, and classifies writer errors
honestly (401 auth vs 403 permission split, new pb_not_found reason).
host CLI loads no env file so isolated runs use the config.ts default superuser; that default (admin@localhost.dev) mismatched the compose-seeded admin@example.com (docker-compose.local.yml POCKETBASE_SUPERUSER_EMAIL), so a fresh isolated PB 400'd pb-auth → 0 jobs enqueued → d6-all-pills no-op. Aligned the default to admin@example.com (password already matched). Added config.test.ts deriving the expected value from docker-compose.local.yml so host CLI and container seed can't drift. Verified llamaindex+agno --isolate: no 400, Enqueued 1 job, worker runs d6-all-pills per-pill.
Mint a run_id per D5/D6 feature run, inject x-diag-run-id + seed x-diag-hops=harness alongside x-aimock-context, and do a post-run aimock-journal join (timeout-bounded) that records a cv-verdict row (tagged harness-d5/harness-d6) localizing whether the context header reached aimock. Add claim/worker_id/snapshot/pool-condition breadcrumbs and surface previously-swallowed catches.
Shared single-line CVDIAG log format (redacted 12-char header prefix), x-diag-run-id/x-diag-hops correlation header constants, and a best-effort PocketBase diag_events sink (anonymously HTTP-readable) so the CV/x-aimock-context propagation chain can be traced mid-incident without Railway log access.
Wire the resource-snapshot writer back into the fleet worker path with
per-replica attribution, add the worker_id column migration, and stamp
the legacy single-process path's worker_id as "" (empty string) to match
PocketBase storage and the surrounding codebase convention.
Make the showcase dev tool faithful to staging by construction. Two changes:
1. `showcase test --d5/--d6` now drives the fleet CONTROL-PLANE (producer ->
probe_jobs queue -> worker -> result-aggregator) instead of the legacy
in-process runLevel() driver. The new cli/control-plane-run.ts replicates
the deep/full producer tick exactly as runControlPlane wires it
(createE2eDeepServiceEnumerator / createServiceEnumerator over
createJobProducer + createFleetQueueClient), enqueues one operator-triggered
tick, and polls local PocketBase for the run's terminal cells. The running
worker fleet claims + runs the driver + the aggregator writes the d5/d6
status cells, so the dev tool exercises the IDENTICAL wiring + concurrency
as staging. The old in-process path stays available behind `--direct`.
2. `showcase up --dev` adds a docker-compose.dev.yml overlay that bind-mounts
each integration's source and overrides the run command with a stack-aware
hot-reload entrypoint (shared/dev/dev-entrypoint.sh: uvicorn --reload for
FastAPI agents, langgraph dev for graphs, next dev for the frontend). Edit a
source file and the component reloads in place with no image rebuild. The
built-image mode remains the faithful/staging-equivalent default.
The e2e_deep kind was removed (D5 now runs the D6 driver), leaving three
browser driver families: e2e_d6, e2e_demos, e2e_smoke. Update the stale
"four browser driver families" docstring and its test comment.
Asserts the real D5 invocation shape buildDeepInputs stamps (both knobs
together): only D5_REPRESENTATIVES featureTypes run AND every emitted key
(per-cell d5:<slug>/<ft> + aggregate d5:<slug>) uses the d5: prefix. The
existing tests cover the knobs in isolation only.
buildDeepInputs now forwards manifest.not_supported_features (matching D6's
buildFullInputs) so local CLI D5 runs don't false-red architecturally-
unsupported features. The D5 thrown-error terminal key changes from
d5-single-pill-e2e:<slug> to d5:<slug> (the driver's own emitAggregate key
shape) so a hard driver throw surfaces as a RED D5 cell, not a blank row.
Also corrects D5-scope docs: representativeOnly keeps the representative
featureTypes per D5_REPRESENTATIVES, not "one pill per category".
Removes E2E_DEEP_DRIVER_KIND and "e2e_deep" from the worker-internal
closed driver-kind set (D5 runs the e2e_d6 driver now), updates the
x-test-id-headers guard to assert on the surviving d6-all-pills driver
(d5-single-pill.ts was deleted), and repoints a stale doc comment.
Repoints the D5 probe at the unified D6 driver. The fleet D5
enumerator now stamps driverKind=e2e_d6 with representativeOnly + a
"d5" rowPrefix; the CLI's buildDeepInputs carries the same inputs;
config/probes/e2e-deep.yml declares kind=e2e_d6. Drops the e2e_deep
driver registration (orchestrator + worker registry + BROWSER_KINDS +
the worker-internal kind set), keeping the D5 producer schedule/cadence
intact. The worker now honors driverInputs.rowPrefix when filtering the
aggregate side-row out of captured cells so a "d5:<slug>" aggregate
doesn't leak into the D6 entry's "d6:<slug>" cell capture. This
eliminates the separate D5 launcher path whose own launcher instance +
cadence systematically dropped x-aimock-context against the shared fleet
pool (aimock strict 503 -> red).
Adds two input knobs to the d6-all-pills driver so it can run as "D5
take-one": `representativeOnly` filters the feature matrix to the
D5_REPRESENTATIVES set, and `rowPrefix` ("d5" | "d6", default "d6")
threads the dashboard key prefix through every emitted per-cell and
aggregate PB row. The representatives map is injectable for testing.
Everything else (route, headers, conversation, pooled launcher) is
unchanged. Red-green unit tests cover both knobs.
The control-plane runs the 8 in-process HTTP probe families but the
on-demand trigger endpoint (POST /api/probes/:id/trigger) was only mounted
on the legacy boot() path, so operators had no way to fire a family
immediately and had to wait on the slow cron. Wire the same
registerProbesRoutes onto the control-plane's buildServer using its own
httpProbeRegistry/httpProbeConfigs/scheduler/httpRunWriter and
OPS_TRIGGER_TOKEN. Only the prefixed in-process probe ids (probe:<id>) are
triggerable; browser-only and unknown ids 404. Token handling mirrors
boot() (unset -> router omitted; set-but-empty -> fail-loud).
Phase 2 of the harness pool-fleet migration — wire the three remaining BROWSER
probe families into the control-plane PRODUCER side so they actually run on the
fleet (the worker DriverRegistry for all four kinds already landed in #5283).
Unit A — three catalog-enumerator factories mirroring createD6ServiceEnumerator,
each delegating to the generic createServiceEnumerator with its own driverKind +
dashboard probeKey prefix and the shared D6_DISCOVERY_FILTER:
- createE2eSmokeServiceEnumerator → e2e_smoke / d4:<slug>
- createE2eDemosServiceEnumerator → e2e_demos / e2e-demos:<slug>
- createE2eDeepServiceEnumerator → e2e_deep / d5-single-pill-e2e:<slug>
R-timeout (demos): the demos driver's 20-min outer cap is threaded in-process via
the legacy E2E_DEMOS_TIMEOUT_MS env, which the fleet worker never sets. The demos
enumerator now conveys the YAML timeout_ms per-job in driverInputs.timeout_ms
(new E2E_DEMOS_TIMEOUT_MS SSOT const mirroring e2e-demos.yml), and the demos
driver reads input.timeout_ms as a resolution source (env > input > deps >
default) so the 38-demo service no longer blows the 5-min default to all-red.
Unit B — runControlPlane now builds four producers and passes a multi-schedule
manifest (buildProducerSchedules) to createControlPlane, each family on its own
cron read literally from the config YAMLs (the deliberate offsets stagger the
families' Playwright fan-outs on the shared BrowserPool):
- fleet-job-producer 40 * * * * (d6, unchanged; honors FLEET_PRODUCER_CRON)
- fleet-producer-e2e-smoke */15 * * * *
- fleet-producer-e2e-demos 10 * * * *
- fleet-producer-e2e-deep 5,20,35,50 * * * *
The in-process HTTP probe runner and the d6 producer's REQ-B sweep leg are left
intact (additive). The worker registry is untouched.
R1 (verified, no change): both createPooledE2eSmokeLauncher and
createPooledE2eDeepLauncher thread contextOpts.extraHTTPHeaders, so smoke + deep
set their per-slug X-AIMock-Context themselves.
Address the CR gaps on the in-process HTTP-probe control-plane:
- Sweep orphaned `running` probe_runs at control-plane boot (boot()'s
sweepStaleRuns never ran in fleet mode → leaked rows forever). Best-effort;
a sweep failure does not abort boot.
- Add a fail-loud BROWSER_KINDS / HTTP-driver disjointness assert at boot plus
a drift-lock test mirroring registerAllProbeDrivers, so a mis-added kind
can't silently go dark.
- Assert the PR's headline guarantees that were unasserted: cron-from-YAML
(exact values), /health loader-failure boot-survival + probes.reload.failed
emit + rules=0 observability, hot-reload add/remove + watcher teardown, a
discovery-backed family (qa) in the in-process schedule, and tightened
/health rule/job counts to exact equality.
- Bump the includeKind skip log to info; make diffHttpProbeSchedules'
unregister-failure post-state explicit (keep config observable); correct the
/health "no longer masks" comment, the discovery-source comment
(image_drift uses railway-services; version_drift uses pnpm-packages), the
reload-failed surface comment (no bus subscriber on the control-plane), and
drop transitional-rot tags.
The fleet control-plane previously ran only the d6 producer, leaving the 8
HTTP-only probe families (smoke, starter_smoke, image_drift, qa, aimock_wiring,
version_drift, pin_drift, redirect_decommission) dark on the fleet. Lift the
legacy boot() probe-loader machinery into runControlPlane so those families run
in-process:
- Add BROWSER_KINDS = {e2e_d6, e2e_smoke, e2e_demos, e2e_deep}; HTTP = every
other kind. Add registerHttpProbeDrivers (HTTP-only driver set, no BrowserPool
drivers).
- In runControlPlane, build an HTTP-only probeRegistry + discovery registry
(railway-services cached + pnpm-packages, mirroring boot()), a probe-loader
scoped to HTTP kinds, and the same diffProbeSchedules/buildProbeInvoker loop
boot() uses — registering one probe:<id> scheduler entry per YAML config.
Crons are driven from the YAML schedule. Browser e2e_* YAMLs route to the
worker producer path and are NOT scheduled in-process.
- Add an includeKind predicate to createProbeLoader so a browser YAML on disk is
SKIPPED (not rejected) against the HTTP-only registry — no spurious
probes.reload.failed.
- /health: ruleCount now reflects the in-process HTTP probe count (was a
hardcoded 0). The control-plane role still drops the rules>0 gate, but the
real count means a silent probe-loader failure is visible on /health rather
than masked. schedulerJobCount already counts the new probe entries.
- Tear down the HTTP-probe file watcher on stop() and on bind failure.
Jordan Ritter