A row whose lease expired beyond its family's stale window is past the
recent-lease heuristic's cross-sweep protection: re-queueing it emitted a
'back in flight' worker-reclaimed-pending signal that the NEXT sweep falsified
by stale-deleting the row. When a lease-phase row is already stale-expirable
(parseable created past maxAgeMs AND lease expired longer than the window),
claim it under the sweeper id and delete it — no comm error, counted in
expiredPending. Recently-expired rows keep today's re-queue path; header's
cross-call-protection claim corrected.
A backslash (or empty) family used to BREAK family discovery, starving every
younger family behind the offending row for up to 3h. The offending ROW needs
no family-charset semantics to exclude — push an id != "<rowId>" clause and
continue, keeping the warn. Younger families are still discovered; the unsafe
row stays unclaimable.
probeKeyFamily("") takes the prefix-LIKE clause leg, so the empty family's
inclusion clause matched every leading-colon key and its exclusion clause hid
ALL leading-colon families from discovery. Treat family "" as clause-unsafe
in familyClauseSafe (skip+warn, like the backslash class), and require
non-empty probeKey/serviceSlug/driverKind/meta.runId in assertServiceJobPayload
so the documented forbidden empty sentinels fail loud at both the enqueue and
claim-decode boundaries.
A renewed:true response missing its job view is a protocol violation on a
SUCCESSFUL renew — the worker still holds the live lease. Evicting + returning
null stopped the heartbeat and let the sweeper falsely reclaim the live job.
Return the cached lease assumed-live (like the indeterminate path); only
evict+null when no cached lease exists. Also corrects the stale 'We ONLY
return null when the CAS itself failed' comment.
A JobClaimEndpointError 4xx from the renew endpoint fails identically every
beat, so the indeterminate assumed-live containment never converged — the
worker held a phantom lease forever while the server lease lapsed and another
worker double-ran the job. Mirror the sweep's 4xx carve-out: error-log
renew-rejected, evict payload+lease caches, return null. 5xx/transport/
2xx-unreadable keep the assumed-live path.
The global lease sweep mirrors comm errors onto the status row keyed by the
reclaimed job's probe_key (resolveSweepAggregateKey → aggregateCommError) for
ALL four fleet families, but the dashboard's decodeCellCommError candidate
scan only covered the d6 family's d6:<slug> aggregate. The smoke/demos/deep
families land on d4:<slug>, e2e-demos:<slug>, and d5-single-pill-e2e:<slug> —
rows the dashboard reads nowhere else — so reclaim/crash overlays on those
families were invisible. Add the three candidate keys (e2e window).
- Object.freeze the UNSUPPORTED CellModel + NOT_WIRED_LEVEL singletons —
they are returned by reference to every caller, so one consumer mutating
them would corrupt every unsupported/not-wired cell.
- Remove the unfirable !d5.exists chip arm (d5.exists and d6.exists derive
from the SAME CATALOG_TO_D5_KEY entry, so !d5.exists && d6.exists is
impossible); fix the decision-table and d6Effective docs that described
the unfirable behavior, keeping a note for a future map split.
- Constrain upsertByKey's generic to StatusRow so rowsAreNoop can never
vacuously match a non-StatusRow type whose fields are all undefined.
- keyFor: validate the dimension segment for ':'/'/' like slug/featureId.
- decodeCellCommError: scope the comm-error staleness window per row family
(e2e 6h for d5/d6/e2e, D4 1h for chat/tools, liveness 45m for health)
instead of applying the 6h E2E window to liveness-cadence rows.
The three-way contradiction: harness fleetSurfaceState routes green-only →
pending, cell-model routes green OR gray (non-regression) → pending while its
own comment claimed 'only green', and both files claimed 'exact mirror'.
gray→pending IS intended dashboard-side (gray is the dashboard-only no-data
colour ProbeState cannot represent). Fix the comments on both sides to state
the asymmetry precisely, extend the drift test to pin the derivation
difference itself (harness green-equality + no gray; dashboard
failure-passthrough + isRegression), add behavioral gray→pending and
red-passthrough tests, and fix chipColorToSurface's stale union doc
(now ChipColor | unreachable | pending).
The stale-green branch returned the RAW row (status "amber" with row.state
still "green"), violating the .row.state ↔ .status invariant resolveD4/D5/D6
maintain. Return { ...row, state: "degraded" } like the other resolvers.
stateToTestStatus mapped unknown runtime states (e.g. "error") to null,
swallowing the A2 rank-fold winner one step after the fold surfaced it — the
D5/D6 chip went benign gray no-data while live-status rendered the loud
"error" tone for the same row. Add foldStateToTestStatus (out-of-vocab →
"red") for the D5/D6 resolvers; D3/D4 keep the base mapping since the
chip's D1-D4 gate check already rescues their null (pinned by tests).
The strict missing-sub-row collapse used the literal `worstState !== "red"`,
which silently swallowed out-of-vocabulary runtime states (e.g. "error") that
the A2 rank machinery deliberately ranks ABOVE red — collapsing exactly the
state the rank fold exists to surface into benign gray no-data. Use
`rankOfState(worstState) < STATE_RANK.red`, mirroring the already-fixed
resolveD5Row/resolveD6Row in live-status.ts.
Follow-up to the TickResult.truncatedByStop accounting field — the
control-plane wiring test's producer fake builds a literal TickResult
and needed the new required field for tsc to stay clean.
A failed assertion in the loop-crash test leaked the worker's bound
/health server (and the default-boot pool) across tests because stop()
was only reached via the in-body rejects.toThrow. Mirror the sibling
default-boot test's try/finally pattern with a rejection-swallowing
safety-net stop.
- derive the buffer-cap overflow boundary from
MAX_BUFFERED_SWEEP_COMM_ERRORS instead of hardcoding 'old100'
- race the re-entrancy-guard's overlapping tick against a short timer so
a broken guard fails diagnostically instead of deadlocking
- pin the warm-up abort test's AbortController+setTimeout coupling
(AbortSignal.timeout would evade fake timers)
- document makeFakeQueue.enqueued as recording enqueue ATTEMPTS
- align queue-client samplePayload driverKind with the probeKey family
so future payload-coherence cross-validation doesn't mass-fail
Stop-truncated specs vanished from the tick outcome: tick-complete's
'services' could not be reconciled against enqueued + enqueueFailures +
skippedForBacklog. Add truncatedByStop to TickResult and the
tick-complete log (invariant: services == enqueued + enqueueFailures +
skippedForBacklog + truncatedByStop) and fix the 'enqueued' doc to
mention stop truncation.
The one-shot no-sink warn kept a sink-less deployment from burying its
logs, but made every subsequent drop completely traceless. Later drops
now log at debug with the dropped jobIds and a running dropped-total
counter; the counter also rides the first full warn.
The first stop() flips 'running' synchronously, so a second concurrent
stop() hit the !running early-return and resolved immediately while the
tick the first stop() was quiescing on kept enqueueing. Both stop()
paths now loop-await the in-flight tick AND the queued trigger before
resolving.
The producer's re-entrancy guard skipped TRIGGERED ticks too, silently
losing an explicit operator 'run it NOW' whenever a slow scheduled tick
was in flight — contradicting the operator-intent-wins rationale that
already lets triggers bypass the backlog gate. A triggered tick that
hits the guard now waits for the in-flight tick and then runs; scheduled
ticks keep the skip. Capped at one queued trigger: a second concurrent
trigger gets the skip + warn.
- rowMatchesFilter throws when multiple positive clauses for one field
are ANDed (the OR-model would evaluate them wrong → vacuous pass)
- makePagingPb throws on unmodeled sort keys instead of silently
returning insertion order
- documented the escaped-quote limitation of the clause value-extraction
regex and likeToRegExp's dangling-backslash fallback
(buffer-cap boundary in job-producer.test.ts deliberately untouched —
sibling slot owns that file)
- release: status is REQUIRED (the old '|| done' fallback silently
finished a job whose caller omitted/emptied status) — 400 like
jobId/workerId
- claim+renew: leaseSeconds floored at 1s (0.001 previously yielded a
1ms lease — instantly stealable, renew thrash)
- all handlers: reject non-string workerId (a JSON number coerces into
the text claimed_by column and the holder can never renew/release)
- claim: same-holder live-lease re-claim answers claimed:true with an
alreadyHeld:true marker (timeout-after-commit retry no longer abandons
a row the worker actually holds); client treats it as a plain win
(ClaimEndpointBody documents the no-op marker)
- hook-parity pins updated/added for every change
- decode-failure REFUSED release now warns with the hook's refusal reason
- probeKeySlug empty-slug ('d6:') falls back to the whole probe_key; a
fully-empty key falls back to unknown-<jobId> (never the forbidden
empty serviceSlug sentinel)
- countPendingForFamily throws on non-numeric/negative totalItems instead
of failing the producer backlog gate open with -1
- stale-pending drain contains a thrown claim CAS per row (a 4xx no
longer aborts the whole drain) and documents the 5xx->won:false
bounded page-stall on passAttempted
- writeResult paces retries (RESULT_WRITE_RETRY_DELAY_MS, injectable
sleep) instead of burning attempts back-to-back
- protocol-violation warns for won&&!job / renewed&&!job before falling
through
- ensureAuth memoizes the in-flight auth promise (no concurrent stampede)
and wraps the auth-body JSON.parse with status context
- releaseJob interface doc corrected: the hook admits claimed AND running
rows (the decode-failure cleanup depends on claimed->failed); the doc's
'running state only' was the wrong side
- documented the LIKE-vs-= ASCII case-sensitivity divergence for
mixed-case families (unreachable: keys are lowercase slugs)
The conservative thrown-release path treated EVERY throw as
may-have-committed, so a wedge row (e.g. empty claimed_by -> hook 400 on
the missing workerId) produced a permanent per-sweep false
worker-reclaimed-pending overlay. job-claim now threads the HTTP status
onto thrown endpoint errors (JobClaimEndpointError); the sweep maps a
4xx (hook rejected, nothing committed) to an error-level log with no
grace, no comm error, no reclaimed++, while 5xx/transport/2xx-unreadable
keep the conservative at-least-once handling.
On the refused_terminal_same_holder retry path, writeResult's
result_processed:false seed could overwrite a row the consumer had
already aggregated (and latched), un-latching it for a second
aggregation — a double-count. The retry now reads the row first: a
present+processed result is skipped (info 'result already aggregated'),
a present+unprocessed result is skipped too (idempotent), only a
missing result falls through to writeResult. A failed pre-write read
throws instead of blind-writing.
With job-claim now throwing on unreadable-2xx (and renew already throwing
on 5xx), a thrown renew escaped queue-client.renewLease into the worker
heartbeat, which treats any renewLease throw as fatal — the heartbeat
died and the sweeper reclaimed a live job (false worker-crashed-mid-job).
renewLease now catches throws from claim.renewLease, warns, and returns
the last-known lease unchanged (no payloadCache eviction, never null) so
the heartbeat retries next beat; only a definitive renewed:false stops
it. Documented at-most-one-lease-duration duplicate-execution risk; the
release CAS still arbitrates terminal writes.
A 2xx response whose body fails to read, is empty, or fails to parse was
mapped to {} — i.e. claimed/renewed/released: false — fabricating a CAS
LOSS for a transition the server had already COMMITTED (the renew flavor
recreated the false worker-crashed-mid-job class; the release flavor made
report() falsely declare the result discarded). Now any unreadable 2xx
body THROWS with path context; callers contain the throw.
probeKeyFamily treats a leading-colon probe_key as its own whole-key
family, so the family value itself contains colons — expanding it with
the <family>:% LIKE leg folded the unrelated family ":foo:bar" under
":foo" in the inclusion/count legs and hid it from discovery via the
exclusion leg. Special-case colon-bearing families to the equality leg
only, implementing the invariant pinned on probeKeyFamily (G3e).
countPendingForFamily filtered on status = "pending" only, so a family
whose whole batch was claimed/running stopped gating and a scheduled
tick could enqueue a fresh batch on top of the in-flight one, doubling
the family's concurrency. Broaden the filter to non-terminal
(pending || claimed || running) and align the gate docs in the contract
and job-producer.
probeKeyFamily(':foo:bar') returns the WHOLE key as its own family (the
round-2 leading-colon fix), but the queue-client clause builders
(familyInclusionClause/familyExclusionClause) expand any family with a
prefix-LIKE <family>:% leg, which would include/exclude ':foo:bar' under
family ':foo' — disagreeing with the fairness partition. The clause
builders live in queue-client.ts (out of scope here), so this documents the
invariant on the contracts side — whole-key (colon-bearing) families must
match by equality only — and pins the partition behavior in the test suite.
The queue-client special-case is reported to the integrator separately.
The taxonomy doc defined the kind as exclusively worker-self-monitor-
reported, but the control-plane result consumer ALSO synthesizes it for
terminal-but-resultless rows past the grace window (the queue-client's
bounded result-write retry exhausted — see RESULT_WRITE_MAX_ATTEMPTS).
Both the contracts.ts kind doc and the dashboard mirror now enumerate
the two sources.
typeof raw !== "object" admits arrays: an array carrying comm-error fields
as expando properties decoded as a well-formed PoolCommError. Both copies
(harness contracts.ts and the dashboard live-status.ts mirror) now reject
arrays explicitly and stay byte-identical — the drift suite gains a
byte-identity pin on the full function source so a one-sided fix can never
silently re-open the gap.
The rollup fold used literal includes("red")/includes("degraded") checks,
so an out-of-vocabulary contributor state (e.g. "error", which the harness
can persist at runtime) matched neither bucket and rolled the cell up GRAY
(benign no-data) while its own badge rendered the loud error tone —
violating the documented precedence red > degraded > green > error >
unknown and the A2 never-swallow rule. Contributor states now route through
worstStateRank, so an unknown state rolls up at least red-severity.
The pass-through gate in fleetSurfaceState was a literal === "red" check, so
a row whose last-known state was degraded/error/out-of-vocab got masked by
the neutral pending overlay — violating the function's own never-mask-a-
genuine-failure invariant (A2: error ranks ABOVE red). Only green now becomes
pending; the dashboard cell-model derivation mirrors it over its ChipColor
vocabulary (red AND amber pass through). Fixes the self-contradicting JSDoc
closing sentence, the stale live-status.ts mirror doc (union was missing
pending), and extends the drift test to pin the pass-through semantics on
both sides.
- count a warm GET as fired only after successful dispatch (sync-throwing
fetchImpl no longer inflates the warmed log)
- route an enumerator resolving to a non-array through the same
enumerate-failure handling as a throw
- mint the runId AFTER the running check (stopped ticks no longer burn
the factory counter or log phantom runIds; empty-runId sentinel)
- log tick-start BEFORE the enumerate await so a hung discovery still
leaves a tick trace (services count moved to tick-complete)
- warn once, with jobIds, when swept comm errors exist but no sink is
configured (previously dropped with only a count)
- terminal .catch on the fire-and-forget warm chain (a throwing logger
no longer raises an unhandled rejection)
- document the SweepCommErrorSink at-least-once contract (aggregator
must be idempotent per jobId+observedAt)
- tests: fix stale "Four producers" comment; replace the fragile
two-shot Math.random spy with a never-exhausting deterministic
sequence plus a call-count guard
A slow tick overlapping the next cron tick let both read the backlog
gate's pending count before either enqueued, so both produced a batch
for the same family — the motivating concurrent same-service-run
incident class. An overlapping tick is now skipped (warned) via the
tracked in-flight tick promise; it returns the empty-runId no-op result
without minting a run.
NOTE for the integrator: the gate also under-counts — queue-client's
countPendingForFamily filters status="pending" only, so a fully-claimed
but-still-running batch doesn't gate. Broadening the filter to
non-terminal rows (pending || claimed || running) is a one-line edit in
queue-client.ts (plus the contracts.ts doc), both sibling-owned —
deferred to the integration pass.
stop() used to flip flags and resolve while an in-flight tick kept
enumerating/sweeping/enqueueing, and buffered sweep comm errors were
silently dropped at shutdown. Now stop() awaits the tracked in-flight
tick promise, the enqueue loop re-checks `running` and truncates loudly
when stopped mid-batch, and stop() makes one final sink delivery of the
buffered comm errors — logging dropped count + jobIds at error level if
that last attempt fails too.
The undelivered comm-error buffer was drained only on maybeSweep's try
SUCCESS path, so a persistently-throwing sweepExpired — the exact
failure mode the buffer exists to ride out — never handed the buffered
batch to a now-healthy sink. Extract the delivery into
deliverSweepCommErrors and run it on every sweep attempt, including the
catch arm.
rowMatchesFilter now THROWS on operators it can't honor (<,>,<=,>=,?=,
?~) and on status clauses with !=/~/!~ instead of silently matching all
rows, with the quoted-literal false-positive limitation documented;
both fakes return totalItems:-1 (and -1 totalPages) under skipTotal —
faithful to PB, closing the fail-open count class beyond the single
countPendingForFamily pin; makePagingPb honors the '-' sort-direction
prefix and computes totalPages honestly; makeOrderedPb's
ignore-all-filters behavior is documented as deliberate coupling to the
duplicate-family defensive break; the result-write retry bound is
pinned with the exported RESULT_WRITE_MAX_ATTEMPTS constant.
(i) per-candidate try/catch in claimNext's CAS race — a thrown transport
claim no longer aborts the whole rotation (warn + next candidate);
(ii) discoverPendingFamilies' duplicate-family defensive break now warns
before breaking; (iii) renew re-read triage split — a decodePayload
failure logs as a protocol violation, not a read blip; (iv) the
decode-failure synthetic result uses an injected clock (new config.now)
instead of new Date(); (v) backslash charset guard for family clause
building (the equality/LIKE escape contracts contradict for backslash —
skip such families with a warn) and the over-claiming VERIFIED comment
scoped to %/_ only; (vi) single-sweeper assumption documented as
load-bearing at the grace-set declaration; (vii) hook comment misname
fixed (worker-crashed-mid-job -> worker-reclaimed-pending); (viii)
claimJob maps a 5xx claim response to a lost CAS (won:false, warn) —
a WAL serialization error escaping runInTransaction surfaces as 500 —
while 4xx and renew/release 5xx still throw loud.
After a release-CAS success + result-write exhaustion, report() throws;
a natural retry got REFUSED (row already terminal) and emitted an error
claiming the result is discarded and the job re-runs — both false (the
result is still writable by this holder; terminal rows never re-run).
The hook's release response now carries a refusal reason
(refused_terminal_same_holder / refused_not_holder /
refused_lease_live), threaded through job-claim's ReleaseResult.
report() treats refused-terminal-under-my-workerId as the second leg of
a timeout-after-commit retry and proceeds to writeResult; the
not-holder error is reworded to 're-runs only if reclaimed to pending'.
A reason-less refusal still fails closed.
Stale-pending age is anchored on PB created and never re-anchored on
re-queue: a job running longer than its family expiry window got
lease-reclaimed with a 'back in flight' comm error, then claim-deleted
by the NEXT sweep before any plausible re-run — the dashboard
permanently showed 're-queued' for silently-discarded work. Schema-free
fix: the release hook now RETAINS the expired lease_expires_at on a
pending re-queue (claim admits pending rows regardless of lease), and
the stale phase skips rows whose retained lease is recent (parseable
and within now - familyExpiryWindow) — recently in flight means the
created-based age is stale evidence. Comments cover the heuristic, the
requeued_at-column alternative, and the sweeper-garbage lingering
tradeoff; hook parity test pins the retention.
The decode-failure synthesis persisted serviceSlug: "" and runId: "" —
the exact empty sentinels this file's own emptyPayloadForLease warning
forbids feeding aggregation (an empty runId groups into nothing, an
empty serviceSlug corrupts the per-service rollup). Recover the slug
from the probe_key's slug segment (new probeKeySlug, the complement of
contracts.ts probeKeyFamily) and mint a non-colliding pviol_<jobId>
runId; reconcile the contradictory comments on both sides. Verified
downstream: result-consumer/aggregator key on aggregateKey + dedupe on
jobId, with serviceSlug/runId riding into logs and batch grouping.
A release that THROWS in the lease sweep may have COMMITTED server-side
(timeout-after-commit): the row is then pending but absent from the
grace set — the same call's stale phase could claim-and-delete it — and
its worker-reclaimed-pending comm error was never synthesized, losing
the gray 're-queued' surface forever (no later sweep re-emits pending
rows). Conservatively add the row to the grace set and synthesize the
comm error on a throw (at-least-once: a duplicate gray overlay is
harmless, a missing one is not); sweeper-held rows stay silent,
mirroring the committed path.
The three /api/fleet/* routerAdd handlers carried no auth middleware —
a middleware-less PB 0.22 routerAdd handler is PUBLIC, so any
unauthenticated caller could claim/renew/release arbitrary jobs despite
the header claiming superuser auth was required. Append
$apis.requireAdminAuth() (verified against PB 0.22.21 JSVM types) to
all three routes; the client already authenticates as superuser with a
401-reauth retry, so enforcement is compat-safe. Also clamp
leaseSeconds in claim+renew (numeric only, 3600s ceiling, 30s default
on garbage) and pin both contracts in the hook-source parity tests.
A probe key beginning with ":" yielded the empty-string family, which
then flowed into countPendingForFamily and the claimNext fairness
partition as a phantom real bucket. Treat such a key as having no family
prefix (the whole key is its own family), matching the no-colon case.
The truthiness guard (`if (featureId && ...)`) let an empty-string
featureId bypass the delimiter validation and fall through to the
integration-aggregate key shape, silently fabricating `<dim>:<slug>` for
what the caller meant as a per-feature lookup. Throw loudly instead,
preserving the function's defensive posture.
The JSDoc said data-bearing states are delegated to buildBadge "under
the `health` dimension label", but the code passes "starter" — and must:
the starter ✓/✗/~ glyph vocabulary requires formatLabel's non-health
branch (the health branch renders up/down/stale word labels instead).
rowsAreNoop ignored fail_count, first_failure_at, and id, so an SSE
delta moving only one of them was swallowed as a no-op: first_failure_at
is load-bearing in formatTooltip ("red since ..."), fail_count feeds the
drilldown/alerting surfaces, and a deleted-and-recreated PB row (same
key, fresh id) kept the stale id in the map. Add the three fields to the
comparison and correct the comparator's field-list doc claims.
Jordan Ritter