The S10 caller's ad-hoc raw Date.parse check could disagree with
isWorkerStale's PB-space-form-normalized parser (engine-lenient vs
anchored), so the unparseable-heartbeat warn did not fire precisely when
the staleness check was blind. Wire the F2-exported heartbeatParseable
companion (contracts.ts documents fleet-health as its intended caller)
and surface a per-cycle unparseableHeartbeats count on the
fleet.health.cycle log — red-green pinned (the count was the new
observable; the cycle log previously never fired for a blind-but-online
corrupt row).
- mergeRowsToMap's disjoint-key divergence warn no longer fires on a
signal undefined⇄defined flip between row groups (live-status.ts:
coreRowFieldsEqual split out of rowsAreNoop): the initial fetch
projects signal away while SSE deltas deliver full rows, so the flip
is expected provenance, not a keyspace violation. upsertByKey's
reducer keeps treating the flip as observable (existing tests pin it).
- __tests__/cell-model.test.ts destructure comment no longer claims
noUncheckedIndexedAccess is enabled (it is not in this package's
tsconfig) nor that the sibling matched (it did not).
- src/lib/cell-model.test.ts row() helper aligned to the
destructure-with-fallback shape the comment describes.
- cell-model.ts:29-31 re-export rationale now cites the actual
importer (__tests__/cell-model.test.ts).
staleness.ts isStale treats an unparseable observed_at as NOT stale
(false-stale would downgrade a live green row), while the FF7 gate in
decodeCellCommError treats unparseable as stale/skip (false-not-stale
would pin an uncleared overlay forever). isStale predates this branch
(blame: d0da6e357, pre-existing on main; untouched here), so per the
blame gate it is left as-is and the divergence is documented at the
FF7 site instead. Reported for the ledger.
decodeCellCommError's staleness gate (cell-model.ts:697) compared only
`now - parsed > staleAfterMs`, which is never true for a future-dated
observedAt — clock skew or a corrupt producer timestamp would pin the
unreachable/pending overlay indefinitely, the same permanent-phantom
failure mode the FF7 unparseable-timestamp skip prevents. A timestamp
more than COMM_ERROR_FUTURE_SKEW_TOLERANCE_MS (5min) ahead of now is
now skipped like an unparseable one; skew within tolerance still
surfaces (pinned by a companion test so the guard can't over-correct).
The D1-D4 gate fires only on d3.exists/d4.exists, so a cell with ONLY
green D5/D6 rows (no e2e/chat/tools rows at all) slipped past it and
rendered a green chip + green d6Effective at achievedDepth=0/
ceilingDepth=0 — a false top-of-ladder claim contradicting the
strictness doctrine (PRESENT-but-null D4 grays; D5-no-data grays the
ladder). A wholly absent D3/D4 family now collapses to the gray
"unverified" chip (same shape as the d4NoData collapse) with red-D5/D6
dominance preserved, and d6Effective stays blocked (null).
cell-model.ts:847-852 (gate) / :905-921 (d6Effective).
One existing fixture (amber pass-through under reclaimed-pending) built
its amber from an absent-D3/D4 map; it now carries green e2e/chat rows
so the chip is genuinely amber through an intact ladder — the test's
never-mask assertion is unchanged.
decodeCellCommError (cell-model.ts) derives the REQ-B unreachable/
pending overlay from row.signal, but the bulk initial fetch projects
STATUS_LIST_FIELDS, which omits signal (useLiveStatus.ts /
live-status.ts:STATUS_LIST_FIELDS) — so on every page refresh rows
materialized with signal undefined and ACTIVE overlays vanished until
an SSE delta happened to re-deliver that row.
Fix option (a) — matching the projection's data-volume rationale (the
signal blob is ~61% of the bulk payload): useLiveStatus now issues a
SUPPLEMENTAL initial fetch, concurrent with the bulk pages, of ONLY
the comm-error candidate aggregate rows (key has no /<featureId>
segment) for the four mirror dimensions, WITH signal, and merges them
over their projected bulk twins by key. That is ~4 rows per
integration, so the bulk projection's first-paint win is preserved.
Per-cell rows under the same dimensions carry heavy parity-diff
signals and are deliberately not re-fetched (a stale per-cell comm
error is only a same/lower-severity tie-break candidate against the
aggregate mirror).
- live-status.ts: new FLEET_COMM_AGGREGATE_DIMENSIONS single source of
truth (d6/d4/e2e-demos/d5-single-pill-e2e); STATUS_LIST_FIELDS doc
updated to the truth (buildCellModel reads signal per cell at render;
the old "only ever read in the drilldown" claim was false).
- cell-model.ts: decodeCellCommError derives its aggregate candidates
from the shared constant (scan order preserved; pinned by the
equal-timestamp tie-break tests).
- useLiveStatus.ts: fetchCommAggregateRows + by-key merge; skipped
entirely for dimension scopes outside the aggregate set; supplemental
failure retries through the same connect() chain (fail loud).
- useLiveStatus.test.tsx: the PB mock now honours the fields projection
(returning full rows for the projected bulk fetch is exactly how this
bug stayed invisible to the suite) and serves the supplemental fetch
from a dedicated fixture; new cold-fetch red-green tests assert the
overlay renders end-to-end with NO SSE delta, the narrow filter
shape, the out-of-set skip, and the in-set narrowing.
- live-status.test.ts: formatter pass on the CF7-F3 #5 test added in
the previous commit (oxfmt).
CR finding recurring 3x; fixture lines pre-existing on the base (f8aee59b62 /
ef76c6d0ff / 2a9b38bbfe). The fleet contract (contracts.ts ServiceJobResult
.aggregateKey) forbids an e2e_d6:<slug> row on the fleet path - the dashboard
only ever reads d6:<slug> - so fixtures pinning e2e_d6:* keys mask the exact
regression class (success-path vs error-path aggregate-key drift) these
suites exist to catch. Behavior-neutral: every assertion pins the same
production logic, now over conformant keys.
- contracts.test.ts: makeResult probeKey/aggregateKey + the projection pin
-> d6:langgraph-python; fleetSurfaceState row key/dimension -> d6
- worker-loop.test.ts: makeDriver returns d6:<slug> (was e2e_d6:<slug>),
unified with the comm-error/driver-error paths that already pin d6:...;
all inline driver-fake fixtures + the buildServiceJobResult /
runClaimedJob aggregateKey assertions swept in lockstep. The
registry-routing `<kind>:routed` tagging keys are deliberately untouched
(they tag by DRIVER KIND to prove dispatch, symmetric across
e2e_smoke/e2e_demos)
- orchestrator.test.ts: driverInputs.key -> the d6:tracer-slug tracer,
slug-ALIGNED across probeKey/serviceSlug/probe_key because the d6 driver
derives its aggregate side-row slug from input.key (deriveSlug) while the
loop filters by d6:<serviceSlug> - a mismatched tracer would surface the
side row as a phantom cell; pass-through proof kept (aggregateKey echoes
the tracer) and strengthened with the "no D5 features declared" signal pin
CR finding on commErrorFromStatusSignal (contracts.ts:433-463, decode-guard
region touched on this branch): a malformed embedded value decodes to
undefined - indistinguishable from "absent" - so a REQ-B overlay written by a
NEWER producer (new PoolCommErrorKind rolled out write-side first, or a
renamed required field) is silently dropped by an older reader.
- document the version-skew hazard on the decode (return type unchanged:
every render path branches on presence)
- export statusSignalHasCommErrorKey(signal) so consumers can count/log
"key present but undecodable" drops; mirrors the decoder's wire-shape
guards (null/non-object/array are never valid signals)
- pinned with tests: unknown future kind, renamed required field, well-formed,
genuinely-absent, and array/non-object wire shapes
- the mirrored commErrorFromStatusSignal REGION is untouched: companion +
docs live OUTSIDE the byte-identity block; verified by running the
dashboard's commError-contract-drift.test.ts (12/12 green) - the dashboard
mirror (shell-dashboard live-status.ts) still needs its own sibling
companion, to be landed by the dashboard owner
CR finding flagged 4x across 2 rounds (lines pre-existing on the base, f8aee59b62):
isWorkerStale (contracts.ts:610-618) did a bare Date.parse with NaN -> false,
making a corrupt heartbeat indistinguishable from a fresh one ("never stale
forever" = silent fleet-health blindness), and lacked the anchored PB
space->"T" normalization the queue-client treats as load-bearing for lease
timestamps (PB_DATE_SEP_RE, replicated locally - not imported, since
queue-client imports contracts).
- normalize the PB date form with the anchored regex before parsing
- keep isWorkerStale(...): boolean compat (unparseable stays not-yet-stale)
- export heartbeatParseable(lastHeartbeatAt) so fleet-health (S10) can
warn/count unparseable heartbeats (caller wiring is a follow-up; the S10
caller is outside this fix's file scope)
- red-green: PB space-form timestamps, corrupt timestamp, fresh/stale ISO
boundary pins (strict > at exactly-the-window)
The single-callback-slot fake let the fleet-health registration silently
overwrite the consumer's, making consumer-loop assertions vacuous with
fleetHealth injected and hiding leaked intervals from the cleared flag.
- assertServiceJobPayload rejects a non-number meta.priority when present
- protocolViolationResult falls back to a jobId-derived aggregateKey
(defense-in-depth; empty probe_key is unreachable via claimNext's G1c gate)
- job-claim memoizes the working auth route so a v0.22 backend pays the
/_superusers 404 probe once per process, not per token expiry
- raceCandidates' deterministic-4xx per-poll re-warn documented as accepted
- deterministicEndpointRejection notes why 401 is deterministic (postFleet
already re-authed once)
shuffleInPlace rng clamp NOT done: lines are pre-existing (d408766cce on
origin/main) — blame-gated out.
The SINGLE-SWEEPER comment claimed a singleton, but runControlPlane wires
four producers over one queue client, so cron overrun can overlap sweeps;
a concurrent sweep cannot see the first sweep's grace set and could
claim-delete a row it just re-queued. Overlapping callers now piggyback on
the in-flight sweep; cross-process replicas remain documented as uncovered
(fleet deploys one control-plane).
The finally's unconditional eviction also fired on the malformed-input
refusal thrown BEFORE the release CAS, dropping the assumed-live lease the
indeterminate-renew containment depends on and killing the heartbeat of a
live job. The eviction is now scoped; the 'DONE either way' header claim is
corrected.
Attempt N can commit server-side then throw; a blind attempt N+1 re-seeded
result_processed:false, un-latching what the consumer aggregated in the
250ms pause window. The cross-call retry path already guarded exactly this;
the same read-before-write guard now applies inside the retry loop, and a
failed pre-read refuses the blind rewrite.
A family kept because countPendingForFamily failed (non-poisoned) was
indistinguishable from a clean zero-backlog read in the tick outcome —
the documented sweepFailed/enumerateFailed ambiguity class. Counted per
family and included in the tick-complete log meta.
A null/undefined/primitive enumerator element threw a TypeError out of the
tick body, rejecting the tick promise and violating the 'tick never
rejects' invariant. Non-object elements are now dropped loudly and counted
in the existing invalid-spec accounting.
The fail-closed gate matched a message substring duplicated in 3 places;
any rewording silently flipped it fail-closed -> fail-open. The refusal is
now a dedicated exported class, the producer gates via instanceof, the test
literal copies use the real class, and a drift test drives the REAL
queue-client refusal through the producer gate.
A resolved sink call clears the producer's undelivered comm-error buffer
(deliverSweepCommErrors' at-least-once contract), so the late-bound sink
silently 'succeeding' before controlPlaneRef was assigned dropped the
batch permanently. Extract buildSweepCommErrorSink (exported for tests)
and reject on the unbound window so the producer re-buffers and
redelivers after assembly; pin with red-green unit tests.
- assertServiceJobPayload rejects an empty meta.enqueuedAt — the
emptyPayloadForLease never-aggregate sentinel, same forbidden class as
the other empties (the minted fallback never crosses this boundary)
- 408/429 are excluded from the deterministic-4xx class (shared
deterministicEndpointRejection helper): transient by meaning, they keep
the assumed-live renew and at-least-once sweep containments
- the family-discovery truncation warn threads nextFamilyClauseSafe so a
clause-unsafe garbage head can't masquerade as a hidden claimable family
- fix 'CONTROl' comment typo in queue-client.test.ts
Enforce the documented jobId/workerId equality between the top-level
input and the echoed result (a mismatched caller would release one row
while filing the result under another), and move the terminalJobStatus
computation inside the try so a malformed result can no longer throw
past the finally and leak the claim-time cache entry.
NOTE: contracts.ts's INTEGRATOR NOTE on ReportJobInput still says report()
does not validate this — contracts.ts is sibling-owned this round and is
NOT touched here; the note needs a follow-up edit.
The long-expired carve-out required a finite parsed lease, so an expired
claimed/running row with an unparseable lease_expires_at was re-queued
with a 'back in flight' worker-reclaimed-pending signal — which the next
sweep falsified by claim-deleting the row (the recent-lease protection
also requires a finite lease). An unparseable lease carries no
recent-flight evidence either phase can honor: treat it as long-expired
and delete directly so the emitted signal matches the outcome. Pinned
with a two-sweep test.
An array is typeof 'object' and can carry expando fields that satisfy
every per-field check — the same hole the nested meta check's
Array.isArray guard already closes one level down. Reject it at the
shared enqueue/decode boundary.
authenticate() threw plain Errors for every non-2xx, so the renew and
sweep deterministic-4xx carve-outs (instanceof JobClaimEndpointError +
4xx) never fired for auth failures — rotated creds rode the indeterminate
path forever: a phantom assumed-live lease per renew beat and a false
worker-reclaimed-pending comm error per sweep. 4xx auth responses now
carry the discriminable class (auth path + status); 5xx/network stay
plain (indeterminate). End-to-end pins go through the real job-claim
client for both the renew and sweep halves.
The pending-target release could never succeed: the hook refuses every
pending-target release on a live lease (refused_lease_live, no holder
exemption) and refused_not_holder otherwise, so the containment was inert
— the row wedged a lease window and got a false worker-reclaimed-pending
overlay. Mirror the decode-failure containment (terminal target passes
the live-lease gate) and share its synthetic-result builder. Tests now
use hook-faithful release fakes instead of unconditional released:true.
- report ordering assertion made non-vacuous: capture rows[0].result INSIDE
the releaseJob fake and assert undefined at release time.
- empty-slug-segment fixture carries probe_key 'd6:' on the claim fake's
returned view too (no pinning of the internal key source).
- job-claim.test.ts: file-local silent logger replacing the shared logger
import (spy-leak class under fork-reuse).
- auth-order tests: auth + endpoint URLs recorded in one call log, order
asserted by index; fallback test pins _superusers-first order and the
Authorization header carried to the endpoint.
- makeFakePb.list honesty: models created/lease_expires_at sorts (throws on
unmodeled keys) and honors perPage/page truncation; self-tested in the
fake-honesty suite.
- documented the interleaved-|| connective-guard limitation in
rowMatchesFilter.
- renamed the renew 'convenience re-read returns null' test to the cache-hit
pin it actually covers, with expect(getOneSpy).not.toHaveBeenCalled().
- report() retry: a null getOne resolution is a FAILED read (throw, no blind
write) and a "" result is PB's unset-JSON shape (absent → write proceeds).
- sweepExpired: thrown-release conservative maybes now counted on a separate
reclaimedIndeterminate (SweepResultWithIndeterminate); reclaimed counts only
CAS-confirmed re-queues. Producer one-liner documented for when the
sibling-owned TickResult gains the field.
- job-claim 401 retry: snapshot the token the failed request used; only null
authToken if unchanged (no clobbering a concurrently refreshed token).
- decode-failure synthetic result write: single attempt, no 250ms retry pacing
inside the claim race (consumer crash-synthesis is the documented backstop).
- fleet-claim.pb.js: typeof jobId !== "string" → 400 in all three handlers.
- docs: recent-lease bound is expiryPeriods × period (not one window); claim
5xx→won:false bounded false-overlay source; report retryability deploy-skew
note; drainStalePending page-advance indeterminacy note.
The won:true/no-job protocol breach may still have committed the claim, so
falling through abandoned a row this worker owns — wedging a full lease window
and producing a false worker-reclaimed-pending overlay on the next sweep.
Mirror the decode-failure containment: best-effort releaseJob(id, worker,
"pending") (no work happened) before continuing, with refusals and throws
swallowed+warned.
Jordan Ritter