Backfills the canonical LGP testid time-picker-cancel on the 'None of these
work' button in the time picker card. Testid-only change — no behavior
modification.
useLangGraphInterrupt is the LangGraph-specific hook from @copilotkit/react-core
and is the wrong framework primitive on a Claude SDK Python backend. The canonical
LGP demo uses useInterrupt from @copilotkit/react-core/v2 — the framework-agnostic
CopilotKit v2 interrupt hook — which is the correct surface for non-LangGraph
backends. Swap the import and call site.
## Summary
Adds end-to-end **run-visibility** to the showcase fleet — a new data
path from queue enqueue → per-family projection → dashboard + Slack
alerting:
- **Contracts + PB schema**: `EnqueueJobInput.family`,
`FleetQueueClient.pruneAged`, run-id/family/worker-id columns on
`probe_jobs` / `resource_snapshots`, fleet-claim hook stamps at claim
time.
- **Queue + aggregator**: queue-client stamps run-id/family on enqueue;
d6 producer owns `pruneAged` retention; aggregator computes
`redsIntroduced`/`redsCleared` into `probe_runs`.
- **Run-view projection (`run-view.ts`)**: §5.1 `FLEET_FAMILIES`
registry + §5.2.1 family-summary projection.
`createMemoizedFamilySummary` bounds PB load at ~one fan-out per TTL
regardless of viewer count.
- **Producer wiring**: `PRODUCER_FAMILY_WIRING` drift-locks the four
producers' family ids set-equal to the registry (unit-tested);
CLI-triggered runs route through `familyForLevel` so a registry rename
breaks loudly.
- **Family-silence monitor (§9)**: Slack alerting for the one incident
class transition-keyed rules are blind to (a silent family produces no
row transitions). Rides the existing fleet-health interval (no extra
timer), per-family 6h rate-limit + recovered one-shot + boot-grace,
fails open on PB-down for the meta-alert. Closed-vocabulary redaction.
- **HTTP `/api/runs`** + **`/health.fleetRuns.lastEvaluatedAt`**:
read-only fleet-runs routes mounted unconditionally on the CP role,
backed by the SAME memoized summary the monitor uses. `/health` stamp is
the §9 compensating control so an external poll can detect a wedged
monitor.
- **Orchestrator wiring**: boot-resolved `workerStaleAfterMs` threaded
through both fleet-health and the projection so they judge staleness
against the same window; test queues across the harness gain a no-op
`pruneAged`.
- **Dashboard**: worker-runs Ops section (family table, worker strip,
run-history drill-down), D0-from-staleness vs D0-from-failure family
annotation, per-family silence banner on the coverage tab; data layer
(DTOs, fetchers, polling hook, context provider).
- **Integration test**: 689-line end-to-end test exercising the full
queue lifecycle → /api/runs projection across all four families.
This is the merged scope of the **runviz blitz** lanes T1-T15.
## Test plan
- [x] `pnpm typecheck` on `showcase/harness` and
`showcase/shell-dashboard` — clean
- [x] `vitest run` on full harness suite — 2276 pass / 3 pre-existing
probe-pool timing flakes (NOT in diff; subject is "probe-pool timing
fragility", a different PR's concern)
- [x] `oxfmt --check showcase/` — all green
- [x] Integration test `run-visibility.integration.test.ts` passes —
exercises queue → projection across all four families
- [ ] CI on PR HEAD — pending push, monitored after open
## Known follow-up (bucket-d)
- Probe-pool timing-sensitive flakes in
`src/probes/helpers/browser-pool.test.ts` (FIX#4a self-heal) and
`src/probes/loader/probe-invoker.test.ts` (timeout assertions) —
different test fails on each run; subject is probe-pool timing
fragility, not runviz.
## Deviation note
The runviz ship-finisher session ran in a Claude Agent SDK harness
without an Agent dispatch tool, so the standard 7-agent CR-loop could
not be dispatched. Inline review was performed against the cr-loop
subject-manifest + four-bucket partition: T1-T15 each landed via their
own per-task review on the blitz integration branch before reaching this
PR, and T8/T15 (the two final-merged streams) were validated via
typecheck-clean + targeted-tests-pass + cross-module wiring inspection.
A heavyweight 7-agent CR can be run against this PR if desired.
- queue-client.ts: add missing closing brace at EOF (TS1005 after rebase)
- job-producer.test.ts: add required `family: "d6"` to producer fixtures
and remove duplicate `logger` key in startedProducer
- result-aggregator.test.ts: align with per-row try/catch + dedup-lookup
behavior introduced in 0b2f613e0 — add `persisted: true` and
`writeOverlay` to test writers, update B6 contract expectations
- queue-client.test.ts: drop a stray blank line
Adds a 689-line integration test that exercises the full queue lifecycle
to the /api/runs projection (enqueue → claim → terminal → projection)
across all four families. Updates the railway-envs golden + verify-deploy
drivers regression test to account for the new fleet-runs route surface.
Adds the dashboard Ops worker-runs section — family table, worker strip,
run-history drill-down, D0-from-staleness vs D0-from-failure family
annotation with clock glyph, and the per-family silence banner on the
coverage tab. Wires the data layer: DTOs, /api/runs fetchers, polling
hook, and a worker-runs context provider. cell-drilldown / cell-pieces
gain family-aware rendering.
PRODUCER_FAMILY_WIRING (drift-locked set-equal to FLEET_FAMILIES via a unit
test) drives every buildJobProducer call site; the boot-resolved
worker-stale-after window is threaded through BOTH fleet-health and the
shared family-summary projection so they judge staleness against the same
window. Triggered CLI control-plane runs route through familyForLevel so a
registry rename breaks loudly instead of silently enqueueing jobs invisible
to the projection. Test queues across the harness gain a no-op pruneAged
for the new contract.
Read-only /api/runs routes mounted on the control-plane role, backed by
the SHARED memoized family-summary instance (one PB fan-out per TTL).
Bounds: per-route memo, request rate-limit. /health gains the
fleetRuns.lastEvaluatedAt stamp from the family-silence monitor as the
§9 compensating control for a wedged monitor (an external poll detects
the wedge — the monitor cannot report its own host's death).
job-producer takes a family option and stamps it on every enqueue; the
prune-ownership key is the d6 producer's family. family-silence-monitor
rides the existing fleet-health interval (no extra timer to tear down),
keys 6 h rate-limit + recovered one-shot per family, fails open on
PB-down (the meta-alert path must still fire), and renders alert text
from closed-vocabulary parts only (§5.2.1 redaction). control-plane
fire-and-forgets familySilence.tick(now) each fleet-health cycle.
The §5.1 FLEET_FAMILIES registry and the §5.2.1 family-summary projection
that derives per-family outcome / inflight / lastRun / lastSuccessAt from
the PB-backed batches. The memoized variant fan-outs PB reads at most once
per TTL regardless of viewer count — the SAME instance is shared by the
/api/runs routes and the family-silence monitor so a dashboard poll and a
monitor evaluation inside the same TTL cost one PB fan-out total.
queue-client stamps run_id/family onto every enqueue so downstream
projections can attribute jobs to a family-scoped batch; pruneAged
retention legs land here (the d6 producer owns the call, per §4.2).
result-aggregator computes redsIntroduced/redsCleared from claimed/
sequenced job state into probe_runs summary.
Adds run-id/family/worker-id columns to probe_jobs and resource_snapshots,
plus the EnqueueJobInput.family + FleetQueueClient.pruneAged contracts and
the hoisted deriveHealth primitive that downstream projections share. The
fleet-claim PB hook stamps run-id/family at claim time so every later
projection has a stable join key. probes/run-history is updated to read
the new columns.
## Summary
Bundles two related dashboard/harness lanes that landed together once
both reached green:
1. **CF #18 fairness / claim-fair lane** — the 23-commit base
(`fix/fleet-claim-fairness` rebased onto its CF7 integration tip
`80c5c9402`) carrying claim-spike, cf3/cf4/cf5/cf6/cf7 hardening waves,
plus the **CF8 micro-fix** for the round-8 supplemental-merge finding:
- **CF8 F3 supplemental-merge freshness guard** (`useLiveStatus.ts`):
the cold-load comm-error supplemental fetch runs CONCURRENTLY with the
bulk pages, so the bulk copy of an aggregate row can be NEWER than the
supplemental snapshot. The previous merge replaced the bulk row
unconditionally, regressing `state`/`observed_at` to stale values until
the row's next SSE delta (long for slow-cadence aggregates). Added
`supplementalRowIsOlder` — when the supplemental row is strictly older
the newer bulk row stays INTACT (signal-less) rather than being grafted
into a chimera (newer core + stale signal) that the reducer's
signal-PRESENCE no-op check would silently swallow.
- **Procedure 3 promotions** (bucket-c/d audit over the CF round-8
ledger): one functional fix (escape `workerId` in fleet-health's reclaim
list filter — matches the `JSON.stringify` pattern used at
orchestrator.ts:3240 for the same field; a `"`-bearing worker_id would
otherwise break out of the literal) plus five doc reconciliations on the
producer/contract surfaces flagged across slot1/slot2/slot4/slot5 (stale
queue-client cross-reference, WarmHealthConfig doc vs `gate.specs`
reality, TickResult.reclaimedIndeterminate "Of reclaimed" → "In addition
to reclaimed", TickResult.skippedForBacklog poisoned-count contributor,
SweepResult.commErrors equation scoping).
2. **Dashboard drilldown D4 parity** — three commits making the
dashboard drilldown surface the D4 rung the same way the grid does:
- `feat(showcase): add resolveD4Row + CellState.d4 so the drilldown can
see the D4 rung` — exposes the D4 row on `CellState`, modeled after
`resolveD3Row`.
- `fix(showcase): drilldown shows the D4 row, de-crosses the e2e label,
and scopes the rollup line honestly` — renders D4 in the drilldown
panel, fixes the crossed e2e label, scopes the rollup line to its real
source set.
- `fix(showcase): unify dimension naming on the legend taxonomy across
grid, legacy cells, and legend` — taxonomy cleanup so legend, grid, and
legacy cells use one set of labels.
## Verification
- Dashboard `npx tsc --noEmit`: clean (only the pre-existing
missing-`@/data/*.json` errors that exist on `main`).
- Dashboard `vitest run`: 59 files / 1012 tests passing, 1 skipped.
- Harness `npx tsc --noEmit`: clean.
- Harness `vitest run`: 121/122 files passing; 1 failed = `probe-invoker
times out invoker-level even when driver ignores abortSignal` — the
NAMED known wall-clock flake (per CF7 integration verification), re-run
in isolation: 67/67 PASS.
- Rebase of drilldown-parity onto the updated `fix/cf8-m1` tip: ZERO
conflicts (the two lanes touch disjoint files).
## Test plan
- [ ] CI green on the PR
- [ ] Dashboard drilldown shows D4 rung in addition to D3/D5/D6
- [ ] Legend taxonomy reads the same in legend, grid, drilldown
- [ ] (post-merge, in showcase) cold-load comm-error overlay still
paints; a stale supplemental no longer regresses a freshly-failed
aggregate row
The auth-middleware-presence regex in queue-client.test.ts hook-parity
suite was anchored to the single-line `}, $apis.requireAdminAuth());`
closer. After oxfmt rewrote fleet-claim.pb.js to its multi-line form
(`},\n $apis.requireAdminAuth(),\n);`) the regex stopped matching
and the test asserted 0 routes were guarded — a false alarm.
Broaden the regex to accept both the single-line closer and the
formatter's split form; the structural intent (each routerAdd
handler-end is followed by the requireAdminAuth middleware) is
unchanged.
PROMOTE_TO_A (defense-in-depth, exploit-class):
- fleet-health.ts reclaim list interpolated workerId raw into the PB filter
literal. workerId is DB-sourced (read back from the workers roster row),
not a compile-time constant, and the same field is escaped via JSON.stringify
at orchestrator.ts:3240 — but the reclaim path was missing the same
hardening. A double-quote in worker_id (corrupt row, buggy self-registration)
would either throw the list (silently skipping this worker's reclaim every
cycle) or widen the filter to claim other workers' jobs. Match the sibling
escape pattern.
PROMOTE_TO_B (doc reconciliations exposed by the CF round-8 audit):
- queue-client.ts COUNT-NAME CAVEAT was stale — the cross-referenced
contracts.ts doc was already updated (commit 80c5c940) but the queue-client
side still asked a future maintainer to make the edit that had already
landed.
- WarmHealthConfig + JobProducerOptions.warmHealth docs said the producer
warms 'every enumerated backend' / 'each enumerated spec'; the implementation
warms gate.specs (post-validation, post-backlog-gate). A fully-backlogged
tick warms nothing. Updated both interface-level docs to match.
- TickResult.reclaimedIndeterminate said 'Of reclaimed, the reclaims...' —
but the field is DISJOINT from reclaimed (sibling SweepResult contract +
the queue-client both state a thrown release lands here exclusively).
Rewrote to 'In addition to reclaimed, ...'.
- TickResult.skippedForBacklog doc named only the dedupe-gate contributor;
the in-function comment correctly documents the fail-CLOSED poisoned-count
fold-in. Expanded the field doc to name both contributors, and to clarify
that the fail-OPEN leg lands in backlogGateFailedOpen separately.
- SweepResult.commErrors pairing equation
(commErrors.length === reclaimed + reclaimedIndeterminate) was asserted
unconditionally on the shared contract, but reclaimedIndeterminate is
optional and fakes may not report the split. Scoped to 'implementations
that report the split'.
The cold-load comm-error supplemental fetch runs CONCURRENTLY with the bulk
pages, so the bulk copy of an aggregate row can be NEWER than the supplemental
snapshot (the row's state changed between the two reads). The previous merge
replaced the bulk row unconditionally — regressing state/observed_at and
potentially fail_count back to the older supplemental values until the row's
next SSE delta (long for slow-cadence aggregates).
Add a freshness guard: when the supplemental row is strictly older (by
observed_at), keep the newer bulk row INTACT — signal-less rather than
chimera (newer core + stale signal). A chimera row would be silently swallowed
by the reducer's signal-PRESENCE no-op check; a signal-less bulk row lets
the next SSE delta restore the real current signal via the
undefined→defined presence flip.
Equal timestamps and unparseable timestamps both prefer the supplemental
(signal-bearing) row — only POSITIVELY-stale supplemental is suppressed,
preserving the cold-load comm-error overlay intent of CF7-F3 #1.
Sibling of the queue-client prose fix (CF7 #10), which flagged this
contract doc as describing only the drain phase: despite the name, the
lease phase's long-expired carve-out also claim-deletes claimed/running
rows (stale created-age, long-expired or unparseable lease) into this
count — no re-queue, no comm error, no reclaimed increment.
The harness contract gained statusSignalHasCommErrorKey (REQ-B
version-skew observability) with its dashboard sibling explicitly left
to the dashboard owner. Add the byte-identity-safe companion to
live-status.ts — placed OUTSIDE the commErrorFromStatusSignal region
pinned by commError-contract-drift.test.ts (only the decode function
source is mirrored; proven by the drift suite staying green) — so
dashboard consumers can distinguish a present-but-undecodable overlay
from a genuinely absent one. Unit-pinned: unknown future kind,
well-formed, absent, and array-expando wire shapes.
The S10 caller's ad-hoc raw Date.parse check could disagree with
isWorkerStale's PB-space-form-normalized parser (engine-lenient vs
anchored), so the unparseable-heartbeat warn did not fire precisely when
the staleness check was blind. Wire the F2-exported heartbeatParseable
companion (contracts.ts documents fleet-health as its intended caller)
and surface a per-cycle unparseableHeartbeats count on the
fleet.health.cycle log — red-green pinned (the count was the new
observable; the cycle log previously never fired for a blind-but-online
corrupt row).
- mergeRowsToMap's disjoint-key divergence warn no longer fires on a
signal undefined⇄defined flip between row groups (live-status.ts:
coreRowFieldsEqual split out of rowsAreNoop): the initial fetch
projects signal away while SSE deltas deliver full rows, so the flip
is expected provenance, not a keyspace violation. upsertByKey's
reducer keeps treating the flip as observable (existing tests pin it).
- __tests__/cell-model.test.ts destructure comment no longer claims
noUncheckedIndexedAccess is enabled (it is not in this package's
tsconfig) nor that the sibling matched (it did not).
- src/lib/cell-model.test.ts row() helper aligned to the
destructure-with-fallback shape the comment describes.
- cell-model.ts:29-31 re-export rationale now cites the actual
importer (__tests__/cell-model.test.ts).
staleness.ts isStale treats an unparseable observed_at as NOT stale
(false-stale would downgrade a live green row), while the FF7 gate in
decodeCellCommError treats unparseable as stale/skip (false-not-stale
would pin an uncleared overlay forever). isStale predates this branch
(blame: d0da6e357, pre-existing on main; untouched here), so per the
blame gate it is left as-is and the divergence is documented at the
FF7 site instead. Reported for the ledger.
decodeCellCommError's staleness gate (cell-model.ts:697) compared only
`now - parsed > staleAfterMs`, which is never true for a future-dated
observedAt — clock skew or a corrupt producer timestamp would pin the
unreachable/pending overlay indefinitely, the same permanent-phantom
failure mode the FF7 unparseable-timestamp skip prevents. A timestamp
more than COMM_ERROR_FUTURE_SKEW_TOLERANCE_MS (5min) ahead of now is
now skipped like an unparseable one; skew within tolerance still
surfaces (pinned by a companion test so the guard can't over-correct).
The D1-D4 gate fires only on d3.exists/d4.exists, so a cell with ONLY
green D5/D6 rows (no e2e/chat/tools rows at all) slipped past it and
rendered a green chip + green d6Effective at achievedDepth=0/
ceilingDepth=0 — a false top-of-ladder claim contradicting the
strictness doctrine (PRESENT-but-null D4 grays; D5-no-data grays the
ladder). A wholly absent D3/D4 family now collapses to the gray
"unverified" chip (same shape as the d4NoData collapse) with red-D5/D6
dominance preserved, and d6Effective stays blocked (null).
cell-model.ts:847-852 (gate) / :905-921 (d6Effective).
One existing fixture (amber pass-through under reclaimed-pending) built
its amber from an absent-D3/D4 map; it now carries green e2e/chat rows
so the chip is genuinely amber through an intact ladder — the test's
never-mask assertion is unchanged.
decodeCellCommError (cell-model.ts) derives the REQ-B unreachable/
pending overlay from row.signal, but the bulk initial fetch projects
STATUS_LIST_FIELDS, which omits signal (useLiveStatus.ts /
live-status.ts:STATUS_LIST_FIELDS) — so on every page refresh rows
materialized with signal undefined and ACTIVE overlays vanished until
an SSE delta happened to re-deliver that row.
Fix option (a) — matching the projection's data-volume rationale (the
signal blob is ~61% of the bulk payload): useLiveStatus now issues a
SUPPLEMENTAL initial fetch, concurrent with the bulk pages, of ONLY
the comm-error candidate aggregate rows (key has no /<featureId>
segment) for the four mirror dimensions, WITH signal, and merges them
over their projected bulk twins by key. That is ~4 rows per
integration, so the bulk projection's first-paint win is preserved.
Per-cell rows under the same dimensions carry heavy parity-diff
signals and are deliberately not re-fetched (a stale per-cell comm
error is only a same/lower-severity tie-break candidate against the
aggregate mirror).
- live-status.ts: new FLEET_COMM_AGGREGATE_DIMENSIONS single source of
truth (d6/d4/e2e-demos/d5-single-pill-e2e); STATUS_LIST_FIELDS doc
updated to the truth (buildCellModel reads signal per cell at render;
the old "only ever read in the drilldown" claim was false).
- cell-model.ts: decodeCellCommError derives its aggregate candidates
from the shared constant (scan order preserved; pinned by the
equal-timestamp tie-break tests).
- useLiveStatus.ts: fetchCommAggregateRows + by-key merge; skipped
entirely for dimension scopes outside the aggregate set; supplemental
failure retries through the same connect() chain (fail loud).
- useLiveStatus.test.tsx: the PB mock now honours the fields projection
(returning full rows for the projected bulk fetch is exactly how this
bug stayed invisible to the suite) and serves the supplemental fetch
from a dedicated fixture; new cold-fetch red-green tests assert the
overlay renders end-to-end with NO SSE delta, the narrow filter
shape, the out-of-set skip, and the in-set narrowing.
- live-status.test.ts: formatter pass on the CF7-F3 #5 test added in
the previous commit (oxfmt).
CR finding recurring 3x; fixture lines pre-existing on the base (f8aee59b62 /
ef76c6d0ff / 2a9b38bbfe). The fleet contract (contracts.ts ServiceJobResult
.aggregateKey) forbids an e2e_d6:<slug> row on the fleet path - the dashboard
only ever reads d6:<slug> - so fixtures pinning e2e_d6:* keys mask the exact
regression class (success-path vs error-path aggregate-key drift) these
suites exist to catch. Behavior-neutral: every assertion pins the same
production logic, now over conformant keys.
- contracts.test.ts: makeResult probeKey/aggregateKey + the projection pin
-> d6:langgraph-python; fleetSurfaceState row key/dimension -> d6
- worker-loop.test.ts: makeDriver returns d6:<slug> (was e2e_d6:<slug>),
unified with the comm-error/driver-error paths that already pin d6:...;
all inline driver-fake fixtures + the buildServiceJobResult /
runClaimedJob aggregateKey assertions swept in lockstep. The
registry-routing `<kind>:routed` tagging keys are deliberately untouched
(they tag by DRIVER KIND to prove dispatch, symmetric across
e2e_smoke/e2e_demos)
- orchestrator.test.ts: driverInputs.key -> the d6:tracer-slug tracer,
slug-ALIGNED across probeKey/serviceSlug/probe_key because the d6 driver
derives its aggregate side-row slug from input.key (deriveSlug) while the
loop filters by d6:<serviceSlug> - a mismatched tracer would surface the
side row as a phantom cell; pass-through proof kept (aggregateKey echoes
the tracer) and strengthened with the "no D5 features declared" signal pin
CR finding on commErrorFromStatusSignal (contracts.ts:433-463, decode-guard
region touched on this branch): a malformed embedded value decodes to
undefined - indistinguishable from "absent" - so a REQ-B overlay written by a
NEWER producer (new PoolCommErrorKind rolled out write-side first, or a
renamed required field) is silently dropped by an older reader.
- document the version-skew hazard on the decode (return type unchanged:
every render path branches on presence)
- export statusSignalHasCommErrorKey(signal) so consumers can count/log
"key present but undecodable" drops; mirrors the decoder's wire-shape
guards (null/non-object/array are never valid signals)
- pinned with tests: unknown future kind, renamed required field, well-formed,
genuinely-absent, and array/non-object wire shapes
- the mirrored commErrorFromStatusSignal REGION is untouched: companion +
docs live OUTSIDE the byte-identity block; verified by running the
dashboard's commError-contract-drift.test.ts (12/12 green) - the dashboard
mirror (shell-dashboard live-status.ts) still needs its own sibling
companion, to be landed by the dashboard owner
CR finding flagged 4x across 2 rounds (lines pre-existing on the base, f8aee59b62):
isWorkerStale (contracts.ts:610-618) did a bare Date.parse with NaN -> false,
making a corrupt heartbeat indistinguishable from a fresh one ("never stale
forever" = silent fleet-health blindness), and lacked the anchored PB
space->"T" normalization the queue-client treats as load-bearing for lease
timestamps (PB_DATE_SEP_RE, replicated locally - not imported, since
queue-client imports contracts).
- normalize the PB date form with the anchored regex before parsing
- keep isWorkerStale(...): boolean compat (unparseable stays not-yet-stale)
- export heartbeatParseable(lastHeartbeatAt) so fleet-health (S10) can
warn/count unparseable heartbeats (caller wiring is a follow-up; the S10
caller is outside this fix's file scope)
- red-green: PB space-form timestamps, corrupt timestamp, fresh/stale ISO
boundary pins (strict > at exactly-the-window)
The single-callback-slot fake let the fleet-health registration silently
overwrite the consumer's, making consumer-loop assertions vacuous with
fleetHealth injected and hiding leaked intervals from the cleared flag.
- assertServiceJobPayload rejects a non-number meta.priority when present
- protocolViolationResult falls back to a jobId-derived aggregateKey
(defense-in-depth; empty probe_key is unreachable via claimNext's G1c gate)
- job-claim memoizes the working auth route so a v0.22 backend pays the
/_superusers 404 probe once per process, not per token expiry
- raceCandidates' deterministic-4xx per-poll re-warn documented as accepted
- deterministicEndpointRejection notes why 401 is deterministic (postFleet
already re-authed once)
shuffleInPlace rng clamp NOT done: lines are pre-existing (d408766cce on
origin/main) — blame-gated out.
The SINGLE-SWEEPER comment claimed a singleton, but runControlPlane wires
four producers over one queue client, so cron overrun can overlap sweeps;
a concurrent sweep cannot see the first sweep's grace set and could
claim-delete a row it just re-queued. Overlapping callers now piggyback on
the in-flight sweep; cross-process replicas remain documented as uncovered
(fleet deploys one control-plane).
The finally's unconditional eviction also fired on the malformed-input
refusal thrown BEFORE the release CAS, dropping the assumed-live lease the
indeterminate-renew containment depends on and killing the heartbeat of a
live job. The eviction is now scoped; the 'DONE either way' header claim is
corrected.
Attempt N can commit server-side then throw; a blind attempt N+1 re-seeded
result_processed:false, un-latching what the consumer aggregated in the
250ms pause window. The cross-call retry path already guarded exactly this;
the same read-before-write guard now applies inside the retry loop, and a
failed pre-read refuses the blind rewrite.
A family kept because countPendingForFamily failed (non-poisoned) was
indistinguishable from a clean zero-backlog read in the tick outcome —
the documented sweepFailed/enumerateFailed ambiguity class. Counted per
family and included in the tick-complete log meta.
A null/undefined/primitive enumerator element threw a TypeError out of the
tick body, rejecting the tick promise and violating the 'tick never
rejects' invariant. Non-object elements are now dropped loudly and counted
in the existing invalid-spec accounting.
Jordan Ritter