- assertServiceJobPayload rejects an empty meta.enqueuedAt — the
emptyPayloadForLease never-aggregate sentinel, same forbidden class as
the other empties (the minted fallback never crosses this boundary)
- 408/429 are excluded from the deterministic-4xx class (shared
deterministicEndpointRejection helper): transient by meaning, they keep
the assumed-live renew and at-least-once sweep containments
- the family-discovery truncation warn threads nextFamilyClauseSafe so a
clause-unsafe garbage head can't masquerade as a hidden claimable family
- fix 'CONTROl' comment typo in queue-client.test.ts
Enforce the documented jobId/workerId equality between the top-level
input and the echoed result (a mismatched caller would release one row
while filing the result under another), and move the terminalJobStatus
computation inside the try so a malformed result can no longer throw
past the finally and leak the claim-time cache entry.
NOTE: contracts.ts's INTEGRATOR NOTE on ReportJobInput still says report()
does not validate this — contracts.ts is sibling-owned this round and is
NOT touched here; the note needs a follow-up edit.
The long-expired carve-out required a finite parsed lease, so an expired
claimed/running row with an unparseable lease_expires_at was re-queued
with a 'back in flight' worker-reclaimed-pending signal — which the next
sweep falsified by claim-deleting the row (the recent-lease protection
also requires a finite lease). An unparseable lease carries no
recent-flight evidence either phase can honor: treat it as long-expired
and delete directly so the emitted signal matches the outcome. Pinned
with a two-sweep test.
An array is typeof 'object' and can carry expando fields that satisfy
every per-field check — the same hole the nested meta check's
Array.isArray guard already closes one level down. Reject it at the
shared enqueue/decode boundary.
authenticate() threw plain Errors for every non-2xx, so the renew and
sweep deterministic-4xx carve-outs (instanceof JobClaimEndpointError +
4xx) never fired for auth failures — rotated creds rode the indeterminate
path forever: a phantom assumed-live lease per renew beat and a false
worker-reclaimed-pending comm error per sweep. 4xx auth responses now
carry the discriminable class (auth path + status); 5xx/network stay
plain (indeterminate). End-to-end pins go through the real job-claim
client for both the renew and sweep halves.
The pending-target release could never succeed: the hook refuses every
pending-target release on a live lease (refused_lease_live, no holder
exemption) and refused_not_holder otherwise, so the containment was inert
— the row wedged a lease window and got a false worker-reclaimed-pending
overlay. Mirror the decode-failure containment (terminal target passes
the live-lease gate) and share its synthetic-result builder. Tests now
use hook-faithful release fakes instead of unconditional released:true.
- report ordering assertion made non-vacuous: capture rows[0].result INSIDE
the releaseJob fake and assert undefined at release time.
- empty-slug-segment fixture carries probe_key 'd6:' on the claim fake's
returned view too (no pinning of the internal key source).
- job-claim.test.ts: file-local silent logger replacing the shared logger
import (spy-leak class under fork-reuse).
- auth-order tests: auth + endpoint URLs recorded in one call log, order
asserted by index; fallback test pins _superusers-first order and the
Authorization header carried to the endpoint.
- makeFakePb.list honesty: models created/lease_expires_at sorts (throws on
unmodeled keys) and honors perPage/page truncation; self-tested in the
fake-honesty suite.
- documented the interleaved-|| connective-guard limitation in
rowMatchesFilter.
- renamed the renew 'convenience re-read returns null' test to the cache-hit
pin it actually covers, with expect(getOneSpy).not.toHaveBeenCalled().
- report() retry: a null getOne resolution is a FAILED read (throw, no blind
write) and a "" result is PB's unset-JSON shape (absent → write proceeds).
- sweepExpired: thrown-release conservative maybes now counted on a separate
reclaimedIndeterminate (SweepResultWithIndeterminate); reclaimed counts only
CAS-confirmed re-queues. Producer one-liner documented for when the
sibling-owned TickResult gains the field.
- job-claim 401 retry: snapshot the token the failed request used; only null
authToken if unchanged (no clobbering a concurrently refreshed token).
- decode-failure synthetic result write: single attempt, no 250ms retry pacing
inside the claim race (consumer crash-synthesis is the documented backstop).
- fleet-claim.pb.js: typeof jobId !== "string" → 400 in all three handlers.
- docs: recent-lease bound is expiryPeriods × period (not one window); claim
5xx→won:false bounded false-overlay source; report retryability deploy-skew
note; drainStalePending page-advance indeterminacy note.
The won:true/no-job protocol breach may still have committed the claim, so
falling through abandoned a row this worker owns — wedging a full lease window
and producing a false worker-reclaimed-pending overlay on the next sweep.
Mirror the decode-failure containment: best-effort releaseJob(id, worker,
"pending") (no work happened) before continuing, with refusals and throws
swallowed+warned.
A row whose lease expired beyond its family's stale window is past the
recent-lease heuristic's cross-sweep protection: re-queueing it emitted a
'back in flight' worker-reclaimed-pending signal that the NEXT sweep falsified
by stale-deleting the row. When a lease-phase row is already stale-expirable
(parseable created past maxAgeMs AND lease expired longer than the window),
claim it under the sweeper id and delete it — no comm error, counted in
expiredPending. Recently-expired rows keep today's re-queue path; header's
cross-call-protection claim corrected.
A backslash (or empty) family used to BREAK family discovery, starving every
younger family behind the offending row for up to 3h. The offending ROW needs
no family-charset semantics to exclude — push an id != "<rowId>" clause and
continue, keeping the warn. Younger families are still discovered; the unsafe
row stays unclaimable.
probeKeyFamily("") takes the prefix-LIKE clause leg, so the empty family's
inclusion clause matched every leading-colon key and its exclusion clause hid
ALL leading-colon families from discovery. Treat family "" as clause-unsafe
in familyClauseSafe (skip+warn, like the backslash class), and require
non-empty probeKey/serviceSlug/driverKind/meta.runId in assertServiceJobPayload
so the documented forbidden empty sentinels fail loud at both the enqueue and
claim-decode boundaries.
A renewed:true response missing its job view is a protocol violation on a
SUCCESSFUL renew — the worker still holds the live lease. Evicting + returning
null stopped the heartbeat and let the sweeper falsely reclaim the live job.
Return the cached lease assumed-live (like the indeterminate path); only
evict+null when no cached lease exists. Also corrects the stale 'We ONLY
return null when the CAS itself failed' comment.
A JobClaimEndpointError 4xx from the renew endpoint fails identically every
beat, so the indeterminate assumed-live containment never converged — the
worker held a phantom lease forever while the server lease lapsed and another
worker double-ran the job. Mirror the sweep's 4xx carve-out: error-log
renew-rejected, evict payload+lease caches, return null. 5xx/transport/
2xx-unreadable keep the assumed-live path.
The global lease sweep mirrors comm errors onto the status row keyed by the
reclaimed job's probe_key (resolveSweepAggregateKey → aggregateCommError) for
ALL four fleet families, but the dashboard's decodeCellCommError candidate
scan only covered the d6 family's d6:<slug> aggregate. The smoke/demos/deep
families land on d4:<slug>, e2e-demos:<slug>, and d5-single-pill-e2e:<slug> —
rows the dashboard reads nowhere else — so reclaim/crash overlays on those
families were invisible. Add the three candidate keys (e2e window).
- Object.freeze the UNSUPPORTED CellModel + NOT_WIRED_LEVEL singletons —
they are returned by reference to every caller, so one consumer mutating
them would corrupt every unsupported/not-wired cell.
- Remove the unfirable !d5.exists chip arm (d5.exists and d6.exists derive
from the SAME CATALOG_TO_D5_KEY entry, so !d5.exists && d6.exists is
impossible); fix the decision-table and d6Effective docs that described
the unfirable behavior, keeping a note for a future map split.
- Constrain upsertByKey's generic to StatusRow so rowsAreNoop can never
vacuously match a non-StatusRow type whose fields are all undefined.
- keyFor: validate the dimension segment for ':'/'/' like slug/featureId.
- decodeCellCommError: scope the comm-error staleness window per row family
(e2e 6h for d5/d6/e2e, D4 1h for chat/tools, liveness 45m for health)
instead of applying the 6h E2E window to liveness-cadence rows.
The three-way contradiction: harness fleetSurfaceState routes green-only →
pending, cell-model routes green OR gray (non-regression) → pending while its
own comment claimed 'only green', and both files claimed 'exact mirror'.
gray→pending IS intended dashboard-side (gray is the dashboard-only no-data
colour ProbeState cannot represent). Fix the comments on both sides to state
the asymmetry precisely, extend the drift test to pin the derivation
difference itself (harness green-equality + no gray; dashboard
failure-passthrough + isRegression), add behavioral gray→pending and
red-passthrough tests, and fix chipColorToSurface's stale union doc
(now ChipColor | unreachable | pending).
The stale-green branch returned the RAW row (status "amber" with row.state
still "green"), violating the .row.state ↔ .status invariant resolveD4/D5/D6
maintain. Return { ...row, state: "degraded" } like the other resolvers.
stateToTestStatus mapped unknown runtime states (e.g. "error") to null,
swallowing the A2 rank-fold winner one step after the fold surfaced it — the
D5/D6 chip went benign gray no-data while live-status rendered the loud
"error" tone for the same row. Add foldStateToTestStatus (out-of-vocab →
"red") for the D5/D6 resolvers; D3/D4 keep the base mapping since the
chip's D1-D4 gate check already rescues their null (pinned by tests).
The strict missing-sub-row collapse used the literal `worstState !== "red"`,
which silently swallowed out-of-vocabulary runtime states (e.g. "error") that
the A2 rank machinery deliberately ranks ABOVE red — collapsing exactly the
state the rank fold exists to surface into benign gray no-data. Use
`rankOfState(worstState) < STATE_RANK.red`, mirroring the already-fixed
resolveD5Row/resolveD6Row in live-status.ts.
Follow-up to the TickResult.truncatedByStop accounting field — the
control-plane wiring test's producer fake builds a literal TickResult
and needed the new required field for tsc to stay clean.
A failed assertion in the loop-crash test leaked the worker's bound
/health server (and the default-boot pool) across tests because stop()
was only reached via the in-body rejects.toThrow. Mirror the sibling
default-boot test's try/finally pattern with a rejection-swallowing
safety-net stop.
- derive the buffer-cap overflow boundary from
MAX_BUFFERED_SWEEP_COMM_ERRORS instead of hardcoding 'old100'
- race the re-entrancy-guard's overlapping tick against a short timer so
a broken guard fails diagnostically instead of deadlocking
- pin the warm-up abort test's AbortController+setTimeout coupling
(AbortSignal.timeout would evade fake timers)
- document makeFakeQueue.enqueued as recording enqueue ATTEMPTS
- align queue-client samplePayload driverKind with the probeKey family
so future payload-coherence cross-validation doesn't mass-fail
Stop-truncated specs vanished from the tick outcome: tick-complete's
'services' could not be reconciled against enqueued + enqueueFailures +
skippedForBacklog. Add truncatedByStop to TickResult and the
tick-complete log (invariant: services == enqueued + enqueueFailures +
skippedForBacklog + truncatedByStop) and fix the 'enqueued' doc to
mention stop truncation.
The one-shot no-sink warn kept a sink-less deployment from burying its
logs, but made every subsequent drop completely traceless. Later drops
now log at debug with the dropped jobIds and a running dropped-total
counter; the counter also rides the first full warn.
The first stop() flips 'running' synchronously, so a second concurrent
stop() hit the !running early-return and resolved immediately while the
tick the first stop() was quiescing on kept enqueueing. Both stop()
paths now loop-await the in-flight tick AND the queued trigger before
resolving.
The producer's re-entrancy guard skipped TRIGGERED ticks too, silently
losing an explicit operator 'run it NOW' whenever a slow scheduled tick
was in flight — contradicting the operator-intent-wins rationale that
already lets triggers bypass the backlog gate. A triggered tick that
hits the guard now waits for the in-flight tick and then runs; scheduled
ticks keep the skip. Capped at one queued trigger: a second concurrent
trigger gets the skip + warn.
- rowMatchesFilter throws when multiple positive clauses for one field
are ANDed (the OR-model would evaluate them wrong → vacuous pass)
- makePagingPb throws on unmodeled sort keys instead of silently
returning insertion order
- documented the escaped-quote limitation of the clause value-extraction
regex and likeToRegExp's dangling-backslash fallback
(buffer-cap boundary in job-producer.test.ts deliberately untouched —
sibling slot owns that file)
- release: status is REQUIRED (the old '|| done' fallback silently
finished a job whose caller omitted/emptied status) — 400 like
jobId/workerId
- claim+renew: leaseSeconds floored at 1s (0.001 previously yielded a
1ms lease — instantly stealable, renew thrash)
- all handlers: reject non-string workerId (a JSON number coerces into
the text claimed_by column and the holder can never renew/release)
- claim: same-holder live-lease re-claim answers claimed:true with an
alreadyHeld:true marker (timeout-after-commit retry no longer abandons
a row the worker actually holds); client treats it as a plain win
(ClaimEndpointBody documents the no-op marker)
- hook-parity pins updated/added for every change
- decode-failure REFUSED release now warns with the hook's refusal reason
- probeKeySlug empty-slug ('d6:') falls back to the whole probe_key; a
fully-empty key falls back to unknown-<jobId> (never the forbidden
empty serviceSlug sentinel)
- countPendingForFamily throws on non-numeric/negative totalItems instead
of failing the producer backlog gate open with -1
- stale-pending drain contains a thrown claim CAS per row (a 4xx no
longer aborts the whole drain) and documents the 5xx->won:false
bounded page-stall on passAttempted
- writeResult paces retries (RESULT_WRITE_RETRY_DELAY_MS, injectable
sleep) instead of burning attempts back-to-back
- protocol-violation warns for won&&!job / renewed&&!job before falling
through
- ensureAuth memoizes the in-flight auth promise (no concurrent stampede)
and wraps the auth-body JSON.parse with status context
- releaseJob interface doc corrected: the hook admits claimed AND running
rows (the decode-failure cleanup depends on claimed->failed); the doc's
'running state only' was the wrong side
- documented the LIKE-vs-= ASCII case-sensitivity divergence for
mixed-case families (unreachable: keys are lowercase slugs)
The conservative thrown-release path treated EVERY throw as
may-have-committed, so a wedge row (e.g. empty claimed_by -> hook 400 on
the missing workerId) produced a permanent per-sweep false
worker-reclaimed-pending overlay. job-claim now threads the HTTP status
onto thrown endpoint errors (JobClaimEndpointError); the sweep maps a
4xx (hook rejected, nothing committed) to an error-level log with no
grace, no comm error, no reclaimed++, while 5xx/transport/2xx-unreadable
keep the conservative at-least-once handling.
On the refused_terminal_same_holder retry path, writeResult's
result_processed:false seed could overwrite a row the consumer had
already aggregated (and latched), un-latching it for a second
aggregation — a double-count. The retry now reads the row first: a
present+processed result is skipped (info 'result already aggregated'),
a present+unprocessed result is skipped too (idempotent), only a
missing result falls through to writeResult. A failed pre-write read
throws instead of blind-writing.
With job-claim now throwing on unreadable-2xx (and renew already throwing
on 5xx), a thrown renew escaped queue-client.renewLease into the worker
heartbeat, which treats any renewLease throw as fatal — the heartbeat
died and the sweeper reclaimed a live job (false worker-crashed-mid-job).
renewLease now catches throws from claim.renewLease, warns, and returns
the last-known lease unchanged (no payloadCache eviction, never null) so
the heartbeat retries next beat; only a definitive renewed:false stops
it. Documented at-most-one-lease-duration duplicate-execution risk; the
release CAS still arbitrates terminal writes.
A 2xx response whose body fails to read, is empty, or fails to parse was
mapped to {} — i.e. claimed/renewed/released: false — fabricating a CAS
LOSS for a transition the server had already COMMITTED (the renew flavor
recreated the false worker-crashed-mid-job class; the release flavor made
report() falsely declare the result discarded). Now any unreadable 2xx
body THROWS with path context; callers contain the throw.
probeKeyFamily treats a leading-colon probe_key as its own whole-key
family, so the family value itself contains colons — expanding it with
the <family>:% LIKE leg folded the unrelated family ":foo:bar" under
":foo" in the inclusion/count legs and hid it from discovery via the
exclusion leg. Special-case colon-bearing families to the equality leg
only, implementing the invariant pinned on probeKeyFamily (G3e).
countPendingForFamily filtered on status = "pending" only, so a family
whose whole batch was claimed/running stopped gating and a scheduled
tick could enqueue a fresh batch on top of the in-flight one, doubling
the family's concurrency. Broaden the filter to non-terminal
(pending || claimed || running) and align the gate docs in the contract
and job-producer.
probeKeyFamily(':foo:bar') returns the WHOLE key as its own family (the
round-2 leading-colon fix), but the queue-client clause builders
(familyInclusionClause/familyExclusionClause) expand any family with a
prefix-LIKE <family>:% leg, which would include/exclude ':foo:bar' under
family ':foo' — disagreeing with the fairness partition. The clause
builders live in queue-client.ts (out of scope here), so this documents the
invariant on the contracts side — whole-key (colon-bearing) families must
match by equality only — and pins the partition behavior in the test suite.
The queue-client special-case is reported to the integrator separately.
The taxonomy doc defined the kind as exclusively worker-self-monitor-
reported, but the control-plane result consumer ALSO synthesizes it for
terminal-but-resultless rows past the grace window (the queue-client's
bounded result-write retry exhausted — see RESULT_WRITE_MAX_ATTEMPTS).
Both the contracts.ts kind doc and the dashboard mirror now enumerate
the two sources.
Jordan Ritter