The three-way contradiction: harness fleetSurfaceState routes green-only →
pending, cell-model routes green OR gray (non-regression) → pending while its
own comment claimed 'only green', and both files claimed 'exact mirror'.
gray→pending IS intended dashboard-side (gray is the dashboard-only no-data
colour ProbeState cannot represent). Fix the comments on both sides to state
the asymmetry precisely, extend the drift test to pin the derivation
difference itself (harness green-equality + no gray; dashboard
failure-passthrough + isRegression), add behavioral gray→pending and
red-passthrough tests, and fix chipColorToSurface's stale union doc
(now ChipColor | unreachable | pending).
The stale-green branch returned the RAW row (status "amber" with row.state
still "green"), violating the .row.state ↔ .status invariant resolveD4/D5/D6
maintain. Return { ...row, state: "degraded" } like the other resolvers.
stateToTestStatus mapped unknown runtime states (e.g. "error") to null,
swallowing the A2 rank-fold winner one step after the fold surfaced it — the
D5/D6 chip went benign gray no-data while live-status rendered the loud
"error" tone for the same row. Add foldStateToTestStatus (out-of-vocab →
"red") for the D5/D6 resolvers; D3/D4 keep the base mapping since the
chip's D1-D4 gate check already rescues their null (pinned by tests).
The strict missing-sub-row collapse used the literal `worstState !== "red"`,
which silently swallowed out-of-vocabulary runtime states (e.g. "error") that
the A2 rank machinery deliberately ranks ABOVE red — collapsing exactly the
state the rank fold exists to surface into benign gray no-data. Use
`rankOfState(worstState) < STATE_RANK.red`, mirroring the already-fixed
resolveD5Row/resolveD6Row in live-status.ts.
Follow-up to the TickResult.truncatedByStop accounting field — the
control-plane wiring test's producer fake builds a literal TickResult
and needed the new required field for tsc to stay clean.
A failed assertion in the loop-crash test leaked the worker's bound
/health server (and the default-boot pool) across tests because stop()
was only reached via the in-body rejects.toThrow. Mirror the sibling
default-boot test's try/finally pattern with a rejection-swallowing
safety-net stop.
- derive the buffer-cap overflow boundary from
MAX_BUFFERED_SWEEP_COMM_ERRORS instead of hardcoding 'old100'
- race the re-entrancy-guard's overlapping tick against a short timer so
a broken guard fails diagnostically instead of deadlocking
- pin the warm-up abort test's AbortController+setTimeout coupling
(AbortSignal.timeout would evade fake timers)
- document makeFakeQueue.enqueued as recording enqueue ATTEMPTS
- align queue-client samplePayload driverKind with the probeKey family
so future payload-coherence cross-validation doesn't mass-fail
Stop-truncated specs vanished from the tick outcome: tick-complete's
'services' could not be reconciled against enqueued + enqueueFailures +
skippedForBacklog. Add truncatedByStop to TickResult and the
tick-complete log (invariant: services == enqueued + enqueueFailures +
skippedForBacklog + truncatedByStop) and fix the 'enqueued' doc to
mention stop truncation.
The one-shot no-sink warn kept a sink-less deployment from burying its
logs, but made every subsequent drop completely traceless. Later drops
now log at debug with the dropped jobIds and a running dropped-total
counter; the counter also rides the first full warn.
The first stop() flips 'running' synchronously, so a second concurrent
stop() hit the !running early-return and resolved immediately while the
tick the first stop() was quiescing on kept enqueueing. Both stop()
paths now loop-await the in-flight tick AND the queued trigger before
resolving.
The producer's re-entrancy guard skipped TRIGGERED ticks too, silently
losing an explicit operator 'run it NOW' whenever a slow scheduled tick
was in flight — contradicting the operator-intent-wins rationale that
already lets triggers bypass the backlog gate. A triggered tick that
hits the guard now waits for the in-flight tick and then runs; scheduled
ticks keep the skip. Capped at one queued trigger: a second concurrent
trigger gets the skip + warn.
- rowMatchesFilter throws when multiple positive clauses for one field
are ANDed (the OR-model would evaluate them wrong → vacuous pass)
- makePagingPb throws on unmodeled sort keys instead of silently
returning insertion order
- documented the escaped-quote limitation of the clause value-extraction
regex and likeToRegExp's dangling-backslash fallback
(buffer-cap boundary in job-producer.test.ts deliberately untouched —
sibling slot owns that file)
- release: status is REQUIRED (the old '|| done' fallback silently
finished a job whose caller omitted/emptied status) — 400 like
jobId/workerId
- claim+renew: leaseSeconds floored at 1s (0.001 previously yielded a
1ms lease — instantly stealable, renew thrash)
- all handlers: reject non-string workerId (a JSON number coerces into
the text claimed_by column and the holder can never renew/release)
- claim: same-holder live-lease re-claim answers claimed:true with an
alreadyHeld:true marker (timeout-after-commit retry no longer abandons
a row the worker actually holds); client treats it as a plain win
(ClaimEndpointBody documents the no-op marker)
- hook-parity pins updated/added for every change
- decode-failure REFUSED release now warns with the hook's refusal reason
- probeKeySlug empty-slug ('d6:') falls back to the whole probe_key; a
fully-empty key falls back to unknown-<jobId> (never the forbidden
empty serviceSlug sentinel)
- countPendingForFamily throws on non-numeric/negative totalItems instead
of failing the producer backlog gate open with -1
- stale-pending drain contains a thrown claim CAS per row (a 4xx no
longer aborts the whole drain) and documents the 5xx->won:false
bounded page-stall on passAttempted
- writeResult paces retries (RESULT_WRITE_RETRY_DELAY_MS, injectable
sleep) instead of burning attempts back-to-back
- protocol-violation warns for won&&!job / renewed&&!job before falling
through
- ensureAuth memoizes the in-flight auth promise (no concurrent stampede)
and wraps the auth-body JSON.parse with status context
- releaseJob interface doc corrected: the hook admits claimed AND running
rows (the decode-failure cleanup depends on claimed->failed); the doc's
'running state only' was the wrong side
- documented the LIKE-vs-= ASCII case-sensitivity divergence for
mixed-case families (unreachable: keys are lowercase slugs)
The conservative thrown-release path treated EVERY throw as
may-have-committed, so a wedge row (e.g. empty claimed_by -> hook 400 on
the missing workerId) produced a permanent per-sweep false
worker-reclaimed-pending overlay. job-claim now threads the HTTP status
onto thrown endpoint errors (JobClaimEndpointError); the sweep maps a
4xx (hook rejected, nothing committed) to an error-level log with no
grace, no comm error, no reclaimed++, while 5xx/transport/2xx-unreadable
keep the conservative at-least-once handling.
On the refused_terminal_same_holder retry path, writeResult's
result_processed:false seed could overwrite a row the consumer had
already aggregated (and latched), un-latching it for a second
aggregation — a double-count. The retry now reads the row first: a
present+processed result is skipped (info 'result already aggregated'),
a present+unprocessed result is skipped too (idempotent), only a
missing result falls through to writeResult. A failed pre-write read
throws instead of blind-writing.
With job-claim now throwing on unreadable-2xx (and renew already throwing
on 5xx), a thrown renew escaped queue-client.renewLease into the worker
heartbeat, which treats any renewLease throw as fatal — the heartbeat
died and the sweeper reclaimed a live job (false worker-crashed-mid-job).
renewLease now catches throws from claim.renewLease, warns, and returns
the last-known lease unchanged (no payloadCache eviction, never null) so
the heartbeat retries next beat; only a definitive renewed:false stops
it. Documented at-most-one-lease-duration duplicate-execution risk; the
release CAS still arbitrates terminal writes.
A 2xx response whose body fails to read, is empty, or fails to parse was
mapped to {} — i.e. claimed/renewed/released: false — fabricating a CAS
LOSS for a transition the server had already COMMITTED (the renew flavor
recreated the false worker-crashed-mid-job class; the release flavor made
report() falsely declare the result discarded). Now any unreadable 2xx
body THROWS with path context; callers contain the throw.
probeKeyFamily treats a leading-colon probe_key as its own whole-key
family, so the family value itself contains colons — expanding it with
the <family>:% LIKE leg folded the unrelated family ":foo:bar" under
":foo" in the inclusion/count legs and hid it from discovery via the
exclusion leg. Special-case colon-bearing families to the equality leg
only, implementing the invariant pinned on probeKeyFamily (G3e).
countPendingForFamily filtered on status = "pending" only, so a family
whose whole batch was claimed/running stopped gating and a scheduled
tick could enqueue a fresh batch on top of the in-flight one, doubling
the family's concurrency. Broaden the filter to non-terminal
(pending || claimed || running) and align the gate docs in the contract
and job-producer.
probeKeyFamily(':foo:bar') returns the WHOLE key as its own family (the
round-2 leading-colon fix), but the queue-client clause builders
(familyInclusionClause/familyExclusionClause) expand any family with a
prefix-LIKE <family>:% leg, which would include/exclude ':foo:bar' under
family ':foo' — disagreeing with the fairness partition. The clause
builders live in queue-client.ts (out of scope here), so this documents the
invariant on the contracts side — whole-key (colon-bearing) families must
match by equality only — and pins the partition behavior in the test suite.
The queue-client special-case is reported to the integrator separately.
The taxonomy doc defined the kind as exclusively worker-self-monitor-
reported, but the control-plane result consumer ALSO synthesizes it for
terminal-but-resultless rows past the grace window (the queue-client's
bounded result-write retry exhausted — see RESULT_WRITE_MAX_ATTEMPTS).
Both the contracts.ts kind doc and the dashboard mirror now enumerate
the two sources.
typeof raw !== "object" admits arrays: an array carrying comm-error fields
as expando properties decoded as a well-formed PoolCommError. Both copies
(harness contracts.ts and the dashboard live-status.ts mirror) now reject
arrays explicitly and stay byte-identical — the drift suite gains a
byte-identity pin on the full function source so a one-sided fix can never
silently re-open the gap.
The rollup fold used literal includes("red")/includes("degraded") checks,
so an out-of-vocabulary contributor state (e.g. "error", which the harness
can persist at runtime) matched neither bucket and rolled the cell up GRAY
(benign no-data) while its own badge rendered the loud error tone —
violating the documented precedence red > degraded > green > error >
unknown and the A2 never-swallow rule. Contributor states now route through
worstStateRank, so an unknown state rolls up at least red-severity.
The pass-through gate in fleetSurfaceState was a literal === "red" check, so
a row whose last-known state was degraded/error/out-of-vocab got masked by
the neutral pending overlay — violating the function's own never-mask-a-
genuine-failure invariant (A2: error ranks ABOVE red). Only green now becomes
pending; the dashboard cell-model derivation mirrors it over its ChipColor
vocabulary (red AND amber pass through). Fixes the self-contradicting JSDoc
closing sentence, the stale live-status.ts mirror doc (union was missing
pending), and extends the drift test to pin the pass-through semantics on
both sides.
- count a warm GET as fired only after successful dispatch (sync-throwing
fetchImpl no longer inflates the warmed log)
- route an enumerator resolving to a non-array through the same
enumerate-failure handling as a throw
- mint the runId AFTER the running check (stopped ticks no longer burn
the factory counter or log phantom runIds; empty-runId sentinel)
- log tick-start BEFORE the enumerate await so a hung discovery still
leaves a tick trace (services count moved to tick-complete)
- warn once, with jobIds, when swept comm errors exist but no sink is
configured (previously dropped with only a count)
- terminal .catch on the fire-and-forget warm chain (a throwing logger
no longer raises an unhandled rejection)
- document the SweepCommErrorSink at-least-once contract (aggregator
must be idempotent per jobId+observedAt)
- tests: fix stale "Four producers" comment; replace the fragile
two-shot Math.random spy with a never-exhausting deterministic
sequence plus a call-count guard
A slow tick overlapping the next cron tick let both read the backlog
gate's pending count before either enqueued, so both produced a batch
for the same family — the motivating concurrent same-service-run
incident class. An overlapping tick is now skipped (warned) via the
tracked in-flight tick promise; it returns the empty-runId no-op result
without minting a run.
NOTE for the integrator: the gate also under-counts — queue-client's
countPendingForFamily filters status="pending" only, so a fully-claimed
but-still-running batch doesn't gate. Broadening the filter to
non-terminal rows (pending || claimed || running) is a one-line edit in
queue-client.ts (plus the contracts.ts doc), both sibling-owned —
deferred to the integration pass.
stop() used to flip flags and resolve while an in-flight tick kept
enumerating/sweeping/enqueueing, and buffered sweep comm errors were
silently dropped at shutdown. Now stop() awaits the tracked in-flight
tick promise, the enqueue loop re-checks `running` and truncates loudly
when stopped mid-batch, and stop() makes one final sink delivery of the
buffered comm errors — logging dropped count + jobIds at error level if
that last attempt fails too.
The undelivered comm-error buffer was drained only on maybeSweep's try
SUCCESS path, so a persistently-throwing sweepExpired — the exact
failure mode the buffer exists to ride out — never handed the buffered
batch to a now-healthy sink. Extract the delivery into
deliverSweepCommErrors and run it on every sweep attempt, including the
catch arm.
rowMatchesFilter now THROWS on operators it can't honor (<,>,<=,>=,?=,
?~) and on status clauses with !=/~/!~ instead of silently matching all
rows, with the quoted-literal false-positive limitation documented;
both fakes return totalItems:-1 (and -1 totalPages) under skipTotal —
faithful to PB, closing the fail-open count class beyond the single
countPendingForFamily pin; makePagingPb honors the '-' sort-direction
prefix and computes totalPages honestly; makeOrderedPb's
ignore-all-filters behavior is documented as deliberate coupling to the
duplicate-family defensive break; the result-write retry bound is
pinned with the exported RESULT_WRITE_MAX_ATTEMPTS constant.
(i) per-candidate try/catch in claimNext's CAS race — a thrown transport
claim no longer aborts the whole rotation (warn + next candidate);
(ii) discoverPendingFamilies' duplicate-family defensive break now warns
before breaking; (iii) renew re-read triage split — a decodePayload
failure logs as a protocol violation, not a read blip; (iv) the
decode-failure synthetic result uses an injected clock (new config.now)
instead of new Date(); (v) backslash charset guard for family clause
building (the equality/LIKE escape contracts contradict for backslash —
skip such families with a warn) and the over-claiming VERIFIED comment
scoped to %/_ only; (vi) single-sweeper assumption documented as
load-bearing at the grace-set declaration; (vii) hook comment misname
fixed (worker-crashed-mid-job -> worker-reclaimed-pending); (viii)
claimJob maps a 5xx claim response to a lost CAS (won:false, warn) —
a WAL serialization error escaping runInTransaction surfaces as 500 —
while 4xx and renew/release 5xx still throw loud.
After a release-CAS success + result-write exhaustion, report() throws;
a natural retry got REFUSED (row already terminal) and emitted an error
claiming the result is discarded and the job re-runs — both false (the
result is still writable by this holder; terminal rows never re-run).
The hook's release response now carries a refusal reason
(refused_terminal_same_holder / refused_not_holder /
refused_lease_live), threaded through job-claim's ReleaseResult.
report() treats refused-terminal-under-my-workerId as the second leg of
a timeout-after-commit retry and proceeds to writeResult; the
not-holder error is reworded to 're-runs only if reclaimed to pending'.
A reason-less refusal still fails closed.
Stale-pending age is anchored on PB created and never re-anchored on
re-queue: a job running longer than its family expiry window got
lease-reclaimed with a 'back in flight' comm error, then claim-deleted
by the NEXT sweep before any plausible re-run — the dashboard
permanently showed 're-queued' for silently-discarded work. Schema-free
fix: the release hook now RETAINS the expired lease_expires_at on a
pending re-queue (claim admits pending rows regardless of lease), and
the stale phase skips rows whose retained lease is recent (parseable
and within now - familyExpiryWindow) — recently in flight means the
created-based age is stale evidence. Comments cover the heuristic, the
requeued_at-column alternative, and the sweeper-garbage lingering
tradeoff; hook parity test pins the retention.
The decode-failure synthesis persisted serviceSlug: "" and runId: "" —
the exact empty sentinels this file's own emptyPayloadForLease warning
forbids feeding aggregation (an empty runId groups into nothing, an
empty serviceSlug corrupts the per-service rollup). Recover the slug
from the probe_key's slug segment (new probeKeySlug, the complement of
contracts.ts probeKeyFamily) and mint a non-colliding pviol_<jobId>
runId; reconcile the contradictory comments on both sides. Verified
downstream: result-consumer/aggregator key on aggregateKey + dedupe on
jobId, with serviceSlug/runId riding into logs and batch grouping.
A release that THROWS in the lease sweep may have COMMITTED server-side
(timeout-after-commit): the row is then pending but absent from the
grace set — the same call's stale phase could claim-and-delete it — and
its worker-reclaimed-pending comm error was never synthesized, losing
the gray 're-queued' surface forever (no later sweep re-emits pending
rows). Conservatively add the row to the grace set and synthesize the
comm error on a throw (at-least-once: a duplicate gray overlay is
harmless, a missing one is not); sweeper-held rows stay silent,
mirroring the committed path.
The three /api/fleet/* routerAdd handlers carried no auth middleware —
a middleware-less PB 0.22 routerAdd handler is PUBLIC, so any
unauthenticated caller could claim/renew/release arbitrary jobs despite
the header claiming superuser auth was required. Append
$apis.requireAdminAuth() (verified against PB 0.22.21 JSVM types) to
all three routes; the client already authenticates as superuser with a
401-reauth retry, so enforcement is compat-safe. Also clamp
leaseSeconds in claim+renew (numeric only, 3600s ceiling, 30s default
on garbage) and pin both contracts in the hook-source parity tests.
A probe key beginning with ":" yielded the empty-string family, which
then flowed into countPendingForFamily and the claimNext fairness
partition as a phantom real bucket. Treat such a key as having no family
prefix (the whole key is its own family), matching the no-colon case.
The truthiness guard (`if (featureId && ...)`) let an empty-string
featureId bypass the delimiter validation and fall through to the
integration-aggregate key shape, silently fabricating `<dim>:<slug>` for
what the caller meant as a per-feature lookup. Throw loudly instead,
preserving the function's defensive posture.
The JSDoc said data-bearing states are delegated to buildBadge "under
the `health` dimension label", but the code passes "starter" — and must:
the starter ✓/✗/~ glyph vocabulary requires formatLabel's non-health
branch (the health branch renders up/down/stale word labels instead).
rowsAreNoop ignored fail_count, first_failure_at, and id, so an SSE
delta moving only one of them was swallowed as a no-op: first_failure_at
is load-bearing in formatTooltip ("red since ..."), fail_count feeds the
drilldown/alerting surfaces, and a deleted-and-recreated PB row (same
key, fresh id) kept the stale id in the map. Add the three fields to the
comparison and correct the comparator's field-list doc claims.
Both resolvers collapsed a family with a missing sub-row via
`worstState !== "red"` literal equality, silently swallowing
out-of-vocabulary states (e.g. "error") that the A2 worstStateRank
machinery deliberately ranks ABOVE red. Compare by rank instead so a
red-or-worse fold dominates no-data; worstState is typed State but can
hold raw strings at runtime, so the rank form is the honest comparison.
The harness FleetSurfaceState union had no "pending" member and the
derivation mapped EVERY comm error to the red "unreachable" overlay,
contradicting the worker-reclaimed-pending neutral-surface contract that
POOL_COMM_ERROR_KINDS mandates and the dashboard already implements.
Mirror the dashboard cell-model derivation exactly: reclaimed-pending on
a non-red row -> "pending"; a red row passes through unmasked; every
other kind -> "unreachable". Fix the stale binary-derivation docstring
and extend the cross-package drift test to pin the union overlay
members AND the derivation shape on both sides.
TickOptions.filter / EnumerateContext.filter were documented trigger-only
but tick() forwarded the filter unconditionally — a scheduled tick could
be scoped by a stray operator filter. The filter is now forwarded only
when triggered (dropped with a warn otherwise). ServiceJobSpec.priority's
'higher pulls first' doc described behavior that does not exist (claimNext
never reads it) — rewritten as reserved/not-consulted; no priority
behavior added.
nowMs was captured before the potentially-seconds-long enumerate() await,
back-dating lease-expiry decisions and lastSweepAt, and stamping
meta.enqueuedAt (documented 'ISO timestamp the control-plane enqueued the
job') with the tick-START time. The clock is now re-read immediately
before maybeSweep and enqueuedAt is stamped per job at enqueue time. The
default runId factory also reads the injected now() for
injection-discipline consistency (behavior otherwise identical).
Each producer gets an independent default runIdFactory with its counter
starting at 0, so two producers ticking in the same ms with equal tick
counts minted the SAME runId — and the aggregator groups results by
meta.runId. The factory now bakes in a per-factory random discriminator
segment (generated once at creation); ids stay sortable-prefixed by
timestamp.
The enumerate-throw path returned the same shape as a legitimately empty
run (enqueued: 0), so a discovery outage was indistinguishable from an
empty catalog in the tick outcome — the exact ambiguity class sweepFailed
was added to remove. TickResult now carries enumerateFailed, set on the
enumerate-throw path. (control-plane.test.ts fake TickResult gains the
new required field.)
A synchronously-throwing injected fetchImpl escaped the warm loop and
aborted the whole tick before any job was enqueued, violating the 'never
block or fail job production' contract — the per-spec dispatch is now
try/caught. The success handler also discarded the Response without
consuming it (an unread body pins the socket under undici) — the body is
now cancelled best-effort — and the abort timers are unref()'d (guarded)
so a pending warm timer can't hold the process open.
Jordan Ritter