PROMOTE_TO_A (defense-in-depth, exploit-class):
- fleet-health.ts reclaim list interpolated workerId raw into the PB filter
literal. workerId is DB-sourced (read back from the workers roster row),
not a compile-time constant, and the same field is escaped via JSON.stringify
at orchestrator.ts:3240 — but the reclaim path was missing the same
hardening. A double-quote in worker_id (corrupt row, buggy self-registration)
would either throw the list (silently skipping this worker's reclaim every
cycle) or widen the filter to claim other workers' jobs. Match the sibling
escape pattern.
PROMOTE_TO_B (doc reconciliations exposed by the CF round-8 audit):
- queue-client.ts COUNT-NAME CAVEAT was stale — the cross-referenced
contracts.ts doc was already updated (commit 80c5c940) but the queue-client
side still asked a future maintainer to make the edit that had already
landed.
- WarmHealthConfig + JobProducerOptions.warmHealth docs said the producer
warms 'every enumerated backend' / 'each enumerated spec'; the implementation
warms gate.specs (post-validation, post-backlog-gate). A fully-backlogged
tick warms nothing. Updated both interface-level docs to match.
- TickResult.reclaimedIndeterminate said 'Of reclaimed, the reclaims...' —
but the field is DISJOINT from reclaimed (sibling SweepResult contract +
the queue-client both state a thrown release lands here exclusively).
Rewrote to 'In addition to reclaimed, ...'.
- TickResult.skippedForBacklog doc named only the dedupe-gate contributor;
the in-function comment correctly documents the fail-CLOSED poisoned-count
fold-in. Expanded the field doc to name both contributors, and to clarify
that the fail-OPEN leg lands in backlogGateFailedOpen separately.
- SweepResult.commErrors pairing equation
(commErrors.length === reclaimed + reclaimedIndeterminate) was asserted
unconditionally on the shared contract, but reclaimedIndeterminate is
optional and fakes may not report the split. Scoped to 'implementations
that report the split'.
The cold-load comm-error supplemental fetch runs CONCURRENTLY with the bulk
pages, so the bulk copy of an aggregate row can be NEWER than the supplemental
snapshot (the row's state changed between the two reads). The previous merge
replaced the bulk row unconditionally — regressing state/observed_at and
potentially fail_count back to the older supplemental values until the row's
next SSE delta (long for slow-cadence aggregates).
Add a freshness guard: when the supplemental row is strictly older (by
observed_at), keep the newer bulk row INTACT — signal-less rather than
chimera (newer core + stale signal). A chimera row would be silently swallowed
by the reducer's signal-PRESENCE no-op check; a signal-less bulk row lets
the next SSE delta restore the real current signal via the
undefined→defined presence flip.
Equal timestamps and unparseable timestamps both prefer the supplemental
(signal-bearing) row — only POSITIVELY-stale supplemental is suppressed,
preserving the cold-load comm-error overlay intent of CF7-F3 #1.
Sibling of the queue-client prose fix (CF7 #10), which flagged this
contract doc as describing only the drain phase: despite the name, the
lease phase's long-expired carve-out also claim-deletes claimed/running
rows (stale created-age, long-expired or unparseable lease) into this
count — no re-queue, no comm error, no reclaimed increment.
The harness contract gained statusSignalHasCommErrorKey (REQ-B
version-skew observability) with its dashboard sibling explicitly left
to the dashboard owner. Add the byte-identity-safe companion to
live-status.ts — placed OUTSIDE the commErrorFromStatusSignal region
pinned by commError-contract-drift.test.ts (only the decode function
source is mirrored; proven by the drift suite staying green) — so
dashboard consumers can distinguish a present-but-undecodable overlay
from a genuinely absent one. Unit-pinned: unknown future kind,
well-formed, absent, and array-expando wire shapes.
The S10 caller's ad-hoc raw Date.parse check could disagree with
isWorkerStale's PB-space-form-normalized parser (engine-lenient vs
anchored), so the unparseable-heartbeat warn did not fire precisely when
the staleness check was blind. Wire the F2-exported heartbeatParseable
companion (contracts.ts documents fleet-health as its intended caller)
and surface a per-cycle unparseableHeartbeats count on the
fleet.health.cycle log — red-green pinned (the count was the new
observable; the cycle log previously never fired for a blind-but-online
corrupt row).
- mergeRowsToMap's disjoint-key divergence warn no longer fires on a
signal undefined⇄defined flip between row groups (live-status.ts:
coreRowFieldsEqual split out of rowsAreNoop): the initial fetch
projects signal away while SSE deltas deliver full rows, so the flip
is expected provenance, not a keyspace violation. upsertByKey's
reducer keeps treating the flip as observable (existing tests pin it).
- __tests__/cell-model.test.ts destructure comment no longer claims
noUncheckedIndexedAccess is enabled (it is not in this package's
tsconfig) nor that the sibling matched (it did not).
- src/lib/cell-model.test.ts row() helper aligned to the
destructure-with-fallback shape the comment describes.
- cell-model.ts:29-31 re-export rationale now cites the actual
importer (__tests__/cell-model.test.ts).
staleness.ts isStale treats an unparseable observed_at as NOT stale
(false-stale would downgrade a live green row), while the FF7 gate in
decodeCellCommError treats unparseable as stale/skip (false-not-stale
would pin an uncleared overlay forever). isStale predates this branch
(blame: d0da6e357, pre-existing on main; untouched here), so per the
blame gate it is left as-is and the divergence is documented at the
FF7 site instead. Reported for the ledger.
decodeCellCommError's staleness gate (cell-model.ts:697) compared only
`now - parsed > staleAfterMs`, which is never true for a future-dated
observedAt — clock skew or a corrupt producer timestamp would pin the
unreachable/pending overlay indefinitely, the same permanent-phantom
failure mode the FF7 unparseable-timestamp skip prevents. A timestamp
more than COMM_ERROR_FUTURE_SKEW_TOLERANCE_MS (5min) ahead of now is
now skipped like an unparseable one; skew within tolerance still
surfaces (pinned by a companion test so the guard can't over-correct).
The D1-D4 gate fires only on d3.exists/d4.exists, so a cell with ONLY
green D5/D6 rows (no e2e/chat/tools rows at all) slipped past it and
rendered a green chip + green d6Effective at achievedDepth=0/
ceilingDepth=0 — a false top-of-ladder claim contradicting the
strictness doctrine (PRESENT-but-null D4 grays; D5-no-data grays the
ladder). A wholly absent D3/D4 family now collapses to the gray
"unverified" chip (same shape as the d4NoData collapse) with red-D5/D6
dominance preserved, and d6Effective stays blocked (null).
cell-model.ts:847-852 (gate) / :905-921 (d6Effective).
One existing fixture (amber pass-through under reclaimed-pending) built
its amber from an absent-D3/D4 map; it now carries green e2e/chat rows
so the chip is genuinely amber through an intact ladder — the test's
never-mask assertion is unchanged.
decodeCellCommError (cell-model.ts) derives the REQ-B unreachable/
pending overlay from row.signal, but the bulk initial fetch projects
STATUS_LIST_FIELDS, which omits signal (useLiveStatus.ts /
live-status.ts:STATUS_LIST_FIELDS) — so on every page refresh rows
materialized with signal undefined and ACTIVE overlays vanished until
an SSE delta happened to re-deliver that row.
Fix option (a) — matching the projection's data-volume rationale (the
signal blob is ~61% of the bulk payload): useLiveStatus now issues a
SUPPLEMENTAL initial fetch, concurrent with the bulk pages, of ONLY
the comm-error candidate aggregate rows (key has no /<featureId>
segment) for the four mirror dimensions, WITH signal, and merges them
over their projected bulk twins by key. That is ~4 rows per
integration, so the bulk projection's first-paint win is preserved.
Per-cell rows under the same dimensions carry heavy parity-diff
signals and are deliberately not re-fetched (a stale per-cell comm
error is only a same/lower-severity tie-break candidate against the
aggregate mirror).
- live-status.ts: new FLEET_COMM_AGGREGATE_DIMENSIONS single source of
truth (d6/d4/e2e-demos/d5-single-pill-e2e); STATUS_LIST_FIELDS doc
updated to the truth (buildCellModel reads signal per cell at render;
the old "only ever read in the drilldown" claim was false).
- cell-model.ts: decodeCellCommError derives its aggregate candidates
from the shared constant (scan order preserved; pinned by the
equal-timestamp tie-break tests).
- useLiveStatus.ts: fetchCommAggregateRows + by-key merge; skipped
entirely for dimension scopes outside the aggregate set; supplemental
failure retries through the same connect() chain (fail loud).
- useLiveStatus.test.tsx: the PB mock now honours the fields projection
(returning full rows for the projected bulk fetch is exactly how this
bug stayed invisible to the suite) and serves the supplemental fetch
from a dedicated fixture; new cold-fetch red-green tests assert the
overlay renders end-to-end with NO SSE delta, the narrow filter
shape, the out-of-set skip, and the in-set narrowing.
- live-status.test.ts: formatter pass on the CF7-F3 #5 test added in
the previous commit (oxfmt).
CR finding recurring 3x; fixture lines pre-existing on the base (f8aee59b62 /
ef76c6d0ff / 2a9b38bbfe). The fleet contract (contracts.ts ServiceJobResult
.aggregateKey) forbids an e2e_d6:<slug> row on the fleet path - the dashboard
only ever reads d6:<slug> - so fixtures pinning e2e_d6:* keys mask the exact
regression class (success-path vs error-path aggregate-key drift) these
suites exist to catch. Behavior-neutral: every assertion pins the same
production logic, now over conformant keys.
- contracts.test.ts: makeResult probeKey/aggregateKey + the projection pin
-> d6:langgraph-python; fleetSurfaceState row key/dimension -> d6
- worker-loop.test.ts: makeDriver returns d6:<slug> (was e2e_d6:<slug>),
unified with the comm-error/driver-error paths that already pin d6:...;
all inline driver-fake fixtures + the buildServiceJobResult /
runClaimedJob aggregateKey assertions swept in lockstep. The
registry-routing `<kind>:routed` tagging keys are deliberately untouched
(they tag by DRIVER KIND to prove dispatch, symmetric across
e2e_smoke/e2e_demos)
- orchestrator.test.ts: driverInputs.key -> the d6:tracer-slug tracer,
slug-ALIGNED across probeKey/serviceSlug/probe_key because the d6 driver
derives its aggregate side-row slug from input.key (deriveSlug) while the
loop filters by d6:<serviceSlug> - a mismatched tracer would surface the
side row as a phantom cell; pass-through proof kept (aggregateKey echoes
the tracer) and strengthened with the "no D5 features declared" signal pin
CR finding on commErrorFromStatusSignal (contracts.ts:433-463, decode-guard
region touched on this branch): a malformed embedded value decodes to
undefined - indistinguishable from "absent" - so a REQ-B overlay written by a
NEWER producer (new PoolCommErrorKind rolled out write-side first, or a
renamed required field) is silently dropped by an older reader.
- document the version-skew hazard on the decode (return type unchanged:
every render path branches on presence)
- export statusSignalHasCommErrorKey(signal) so consumers can count/log
"key present but undecodable" drops; mirrors the decoder's wire-shape
guards (null/non-object/array are never valid signals)
- pinned with tests: unknown future kind, renamed required field, well-formed,
genuinely-absent, and array/non-object wire shapes
- the mirrored commErrorFromStatusSignal REGION is untouched: companion +
docs live OUTSIDE the byte-identity block; verified by running the
dashboard's commError-contract-drift.test.ts (12/12 green) - the dashboard
mirror (shell-dashboard live-status.ts) still needs its own sibling
companion, to be landed by the dashboard owner
CR finding flagged 4x across 2 rounds (lines pre-existing on the base, f8aee59b62):
isWorkerStale (contracts.ts:610-618) did a bare Date.parse with NaN -> false,
making a corrupt heartbeat indistinguishable from a fresh one ("never stale
forever" = silent fleet-health blindness), and lacked the anchored PB
space->"T" normalization the queue-client treats as load-bearing for lease
timestamps (PB_DATE_SEP_RE, replicated locally - not imported, since
queue-client imports contracts).
- normalize the PB date form with the anchored regex before parsing
- keep isWorkerStale(...): boolean compat (unparseable stays not-yet-stale)
- export heartbeatParseable(lastHeartbeatAt) so fleet-health (S10) can
warn/count unparseable heartbeats (caller wiring is a follow-up; the S10
caller is outside this fix's file scope)
- red-green: PB space-form timestamps, corrupt timestamp, fresh/stale ISO
boundary pins (strict > at exactly-the-window)
The single-callback-slot fake let the fleet-health registration silently
overwrite the consumer's, making consumer-loop assertions vacuous with
fleetHealth injected and hiding leaked intervals from the cleared flag.
- assertServiceJobPayload rejects a non-number meta.priority when present
- protocolViolationResult falls back to a jobId-derived aggregateKey
(defense-in-depth; empty probe_key is unreachable via claimNext's G1c gate)
- job-claim memoizes the working auth route so a v0.22 backend pays the
/_superusers 404 probe once per process, not per token expiry
- raceCandidates' deterministic-4xx per-poll re-warn documented as accepted
- deterministicEndpointRejection notes why 401 is deterministic (postFleet
already re-authed once)
shuffleInPlace rng clamp NOT done: lines are pre-existing (d408766cce on
origin/main) — blame-gated out.
The SINGLE-SWEEPER comment claimed a singleton, but runControlPlane wires
four producers over one queue client, so cron overrun can overlap sweeps;
a concurrent sweep cannot see the first sweep's grace set and could
claim-delete a row it just re-queued. Overlapping callers now piggyback on
the in-flight sweep; cross-process replicas remain documented as uncovered
(fleet deploys one control-plane).
The finally's unconditional eviction also fired on the malformed-input
refusal thrown BEFORE the release CAS, dropping the assumed-live lease the
indeterminate-renew containment depends on and killing the heartbeat of a
live job. The eviction is now scoped; the 'DONE either way' header claim is
corrected.
Attempt N can commit server-side then throw; a blind attempt N+1 re-seeded
result_processed:false, un-latching what the consumer aggregated in the
250ms pause window. The cross-call retry path already guarded exactly this;
the same read-before-write guard now applies inside the retry loop, and a
failed pre-read refuses the blind rewrite.
A family kept because countPendingForFamily failed (non-poisoned) was
indistinguishable from a clean zero-backlog read in the tick outcome —
the documented sweepFailed/enumerateFailed ambiguity class. Counted per
family and included in the tick-complete log meta.
A null/undefined/primitive enumerator element threw a TypeError out of the
tick body, rejecting the tick promise and violating the 'tick never
rejects' invariant. Non-object elements are now dropped loudly and counted
in the existing invalid-spec accounting.
The fail-closed gate matched a message substring duplicated in 3 places;
any rewording silently flipped it fail-closed -> fail-open. The refusal is
now a dedicated exported class, the producer gates via instanceof, the test
literal copies use the real class, and a drift test drives the REAL
queue-client refusal through the producer gate.
A resolved sink call clears the producer's undelivered comm-error buffer
(deliverSweepCommErrors' at-least-once contract), so the late-bound sink
silently 'succeeding' before controlPlaneRef was assigned dropped the
batch permanently. Extract buildSweepCommErrorSink (exported for tests)
and reject on the unbound window so the producer re-buffers and
redelivers after assembly; pin with red-green unit tests.
- assertServiceJobPayload rejects an empty meta.enqueuedAt — the
emptyPayloadForLease never-aggregate sentinel, same forbidden class as
the other empties (the minted fallback never crosses this boundary)
- 408/429 are excluded from the deterministic-4xx class (shared
deterministicEndpointRejection helper): transient by meaning, they keep
the assumed-live renew and at-least-once sweep containments
- the family-discovery truncation warn threads nextFamilyClauseSafe so a
clause-unsafe garbage head can't masquerade as a hidden claimable family
- fix 'CONTROl' comment typo in queue-client.test.ts
Enforce the documented jobId/workerId equality between the top-level
input and the echoed result (a mismatched caller would release one row
while filing the result under another), and move the terminalJobStatus
computation inside the try so a malformed result can no longer throw
past the finally and leak the claim-time cache entry.
NOTE: contracts.ts's INTEGRATOR NOTE on ReportJobInput still says report()
does not validate this — contracts.ts is sibling-owned this round and is
NOT touched here; the note needs a follow-up edit.
The long-expired carve-out required a finite parsed lease, so an expired
claimed/running row with an unparseable lease_expires_at was re-queued
with a 'back in flight' worker-reclaimed-pending signal — which the next
sweep falsified by claim-deleting the row (the recent-lease protection
also requires a finite lease). An unparseable lease carries no
recent-flight evidence either phase can honor: treat it as long-expired
and delete directly so the emitted signal matches the outcome. Pinned
with a two-sweep test.
An array is typeof 'object' and can carry expando fields that satisfy
every per-field check — the same hole the nested meta check's
Array.isArray guard already closes one level down. Reject it at the
shared enqueue/decode boundary.
authenticate() threw plain Errors for every non-2xx, so the renew and
sweep deterministic-4xx carve-outs (instanceof JobClaimEndpointError +
4xx) never fired for auth failures — rotated creds rode the indeterminate
path forever: a phantom assumed-live lease per renew beat and a false
worker-reclaimed-pending comm error per sweep. 4xx auth responses now
carry the discriminable class (auth path + status); 5xx/network stay
plain (indeterminate). End-to-end pins go through the real job-claim
client for both the renew and sweep halves.
The pending-target release could never succeed: the hook refuses every
pending-target release on a live lease (refused_lease_live, no holder
exemption) and refused_not_holder otherwise, so the containment was inert
— the row wedged a lease window and got a false worker-reclaimed-pending
overlay. Mirror the decode-failure containment (terminal target passes
the live-lease gate) and share its synthetic-result builder. Tests now
use hook-faithful release fakes instead of unconditional released:true.
Jordan Ritter