P1-A (cap semantics): the MAX_RECLAIM_ATTEMPTS cap was keyed on
`reclaim_count`, a LIFETIME tally bumped by BOTH the sweeper re-queue
path AND the peer-worker expired-lease steal (claim CAS). A long-lived
job that accrues benign peer steals could exhaust its 3-budget and then
get claim-DELETED on its first real orphan rather than re-queued.
Fix: introduce a dedicated `consecutive_orphan_count` column (migration
1779990400) that is bumped ONLY by the sweeper re-queue path in the
fleet-claim release CAS, and reset to 0 on every terminal done|failed
release. The peer-worker steal (claim CAS wasExpiredSteal branch) does
NOT touch this counter. The reaper's cap check now uses
`consecutive_orphan_count` instead of `reclaim_count`. `reclaim_count`
is left intact as the lifetime dashboard diagnostic (jobs.reclaimed).
P1-B (low boundary): adds a test at consecutive_orphan_count = MAX-1
(= 2) asserting the row is RE-QUEUED, not deleted. The off-by-one
mutation `>= MAX` -> `>= MAX-1` causes this test to go RED.
Test-fake honesty: `makeReclaimClaim`'s claimJob now explicitly models
the steal-bump on `reclaim_count` (matching the real hook) while
intentionally NOT bumping `consecutive_orphan_count`, and adds an
explicit pin test confirming steals do not consume the reclaim budget.
JSDoc on MAX_RECLAIM_ATTEMPTS updated to describe the correct semantics:
consecutive re-orphans scoped by sweeper re-queue, reset on terminal.
Red-green proof:
- P1-A RED: revert cap to reclaim_count → "P1-A CAP SCOPE" fails with
`expect(undefined).toBeDefined()` (job deleted instead of re-queued)
- P1-A GREEN: consecutive_orphan_count cap → test passes (re-queued)
- P1-B RED: mutate `>= MAX` to `>= MAX-1` → low-boundary test fails
- P1-B GREEN: revert mutation → low-boundary test passes
Suite: 145 queue-client + 103 producer = 248 total, all green.
Layer (a) of the worker reclamation+rollover redesign: make a worker bounce
non-lossy. The reaper's long-expired carve-out (G1d) used to claim-DELETE an
orphaned in-flight (claimed/running) row whose lease expired beyond its
family's stale window AND whose created-age was past that window —
`reclaimed=0, expiredPending++` — silently dropping work an abrupt bounce
(SIGKILL past grace / OOM / crash) left mid-flight.
Invert it to RECLAIM-WINS-UNTIL-CAP: re-queue the orphan to pending (it
re-runs; idempotent probes make at-least-once safe) until its durable
`reclaim_count` reaches MAX_RECLAIM_ATTEMPTS (3), only then claim-deleting a
row that keeps re-orphaning so a poison job cannot loop forever.
The carve-out existed to dodge an honesty bind: re-queueing a `created`-stale
row emitted a "back in flight" gray the next sweep falsified by claim-deleting
it off the renewal-immune `created` age. Dissolve the bind with a new
`requeued_at` column (migration 1779990300) the release CAS stamps on every
pending re-queue; both stale phases now age off `staleAgeAnchorMs`
(`requeued_at ?? created`), so a reclaimed row is genuinely young again and the
next sweep does not delete it. `reclaim_count` (migration 1779990200) is reused
as the attempt counter — no second tally to drift.
Red-green proven on the real reaper (queue-client.test.ts): a stale-aged
long-expired orphan below the cap goes RED (deleted, reclaimed=0,
expiredPending=1) on delete-wins and GREEN (re-queued, reclaimed=1,
expiredPending=0, requeued_at stamped) on reclaim-wins; plus an attempt-cap
deletion test and a next-sweep no-falsification test.
Adds run-id/family/worker-id columns to probe_jobs and resource_snapshots,
plus the EnqueueJobInput.family + FleetQueueClient.pruneAged contracts and
the hoisted deriveHealth primitive that downstream projections share. The
fleet-claim PB hook stamps run-id/family at claim time so every later
projection has a stable join key. probes/run-history is updated to read
the new columns.
- report() retry: a null getOne resolution is a FAILED read (throw, no blind
write) and a "" result is PB's unset-JSON shape (absent → write proceeds).
- sweepExpired: thrown-release conservative maybes now counted on a separate
reclaimedIndeterminate (SweepResultWithIndeterminate); reclaimed counts only
CAS-confirmed re-queues. Producer one-liner documented for when the
sibling-owned TickResult gains the field.
- job-claim 401 retry: snapshot the token the failed request used; only null
authToken if unchanged (no clobbering a concurrently refreshed token).
- decode-failure synthetic result write: single attempt, no 250ms retry pacing
inside the claim race (consumer crash-synthesis is the documented backstop).
- fleet-claim.pb.js: typeof jobId !== "string" → 400 in all three handlers.
- docs: recent-lease bound is expiryPeriods × period (not one window); claim
5xx→won:false bounded false-overlay source; report retryability deploy-skew
note; drainStalePending page-advance indeterminacy note.
- release: status is REQUIRED (the old '|| done' fallback silently
finished a job whose caller omitted/emptied status) — 400 like
jobId/workerId
- claim+renew: leaseSeconds floored at 1s (0.001 previously yielded a
1ms lease — instantly stealable, renew thrash)
- all handlers: reject non-string workerId (a JSON number coerces into
the text claimed_by column and the holder can never renew/release)
- claim: same-holder live-lease re-claim answers claimed:true with an
alreadyHeld:true marker (timeout-after-commit retry no longer abandons
a row the worker actually holds); client treats it as a plain win
(ClaimEndpointBody documents the no-op marker)
- hook-parity pins updated/added for every change
(i) per-candidate try/catch in claimNext's CAS race — a thrown transport
claim no longer aborts the whole rotation (warn + next candidate);
(ii) discoverPendingFamilies' duplicate-family defensive break now warns
before breaking; (iii) renew re-read triage split — a decodePayload
failure logs as a protocol violation, not a read blip; (iv) the
decode-failure synthetic result uses an injected clock (new config.now)
instead of new Date(); (v) backslash charset guard for family clause
building (the equality/LIKE escape contracts contradict for backslash —
skip such families with a warn) and the over-claiming VERIFIED comment
scoped to %/_ only; (vi) single-sweeper assumption documented as
load-bearing at the grace-set declaration; (vii) hook comment misname
fixed (worker-crashed-mid-job -> worker-reclaimed-pending); (viii)
claimJob maps a 5xx claim response to a lost CAS (won:false, warn) —
a WAL serialization error escaping runInTransaction surfaces as 500 —
while 4xx and renew/release 5xx still throw loud.
After a release-CAS success + result-write exhaustion, report() throws;
a natural retry got REFUSED (row already terminal) and emitted an error
claiming the result is discarded and the job re-runs — both false (the
result is still writable by this holder; terminal rows never re-run).
The hook's release response now carries a refusal reason
(refused_terminal_same_holder / refused_not_holder /
refused_lease_live), threaded through job-claim's ReleaseResult.
report() treats refused-terminal-under-my-workerId as the second leg of
a timeout-after-commit retry and proceeds to writeResult; the
not-holder error is reworded to 're-runs only if reclaimed to pending'.
A reason-less refusal still fails closed.
Stale-pending age is anchored on PB created and never re-anchored on
re-queue: a job running longer than its family expiry window got
lease-reclaimed with a 'back in flight' comm error, then claim-deleted
by the NEXT sweep before any plausible re-run — the dashboard
permanently showed 're-queued' for silently-discarded work. Schema-free
fix: the release hook now RETAINS the expired lease_expires_at on a
pending re-queue (claim admits pending rows regardless of lease), and
the stale phase skips rows whose retained lease is recent (parseable
and within now - familyExpiryWindow) — recently in flight means the
created-based age is stale evidence. Comments cover the heuristic, the
requeued_at-column alternative, and the sweeper-garbage lingering
tradeoff; hook parity test pins the retention.
The three /api/fleet/* routerAdd handlers carried no auth middleware —
a middleware-less PB 0.22 routerAdd handler is PUBLIC, so any
unauthenticated caller could claim/renew/release arbitrary jobs despite
the header claiming superuser auth was required. Append
$apis.requireAdminAuth() (verified against PB 0.22.21 JSVM types) to
all three routes; the client already authenticates as superuser with a
401-reauth retry, so enforcement is compat-safe. Also clamp
leaseSeconds in claim+renew (numeric only, 3600s ceiling, 30s default
on garbage) and pin both contracts in the hook-source parity tests.
sweepExpired (and fleet-health's reclaim) decide 'expired' from a listed
SNAPSHOT, then releaseJob(jobId, holder, 'pending') authorizes on
claimed_by alone — a worker renewing between the list and the release
still matches claimed_by, so a live just-renewed job was yanked back to
pending (duplicate execution + a false worker-reclaimed-pending comm
error). The /api/fleet/release hook now refuses a pending-target release
while the row's CURRENT lease is still live, re-checked inside the same
transaction with the leaseExpired helper that stays byte-equivalent to
the client's anchored parse. The client needs no change: released:false
already maps to the sweep's skip path. Pinned by a hook-source parity
test plus a renewed-after-list race test against a hook-faithful fake.
A PB image freshly built from main crash-loops staging PocketBase (502s)
because of two latent defects, both verified by booting the built image
against a real PB 0.22.21 binary on a volume that already has the
collections but has NOT recorded their migrations in `_migrations`.
1. Hook API. `pb_hooks/main.pb.js` registered its CORS middleware via the
bare global `onBeforeServe(...)`, which is undefined in PB 0.22.x JSVM
(only the `$app.onBeforeServe()` Go method exists) — it throws
`ReferenceError: onBeforeServe is not defined` at hook load and crashes
the server. Switch to the documented global `routerUse((next) => (c) =>
…)` entry point. Separately, the per-request closure runs in PB's pooled
goja runtime where top-level helpers/consts are out of scope, so calling
them throws per request and the router returns HTTP 400 on EVERY route;
inline the entire allowlist/env/match logic into the closure to fix that
second regression. Verified: health 200, collection reads 200, the
allowlisted origin is echoed on `Access-Control-Allow-Origin`, a
non-allowlisted origin is not, and OPTIONS preflight returns 204.
2. Migration idempotency. `1777700000_create_baseline.js` and the three
original `1745193*` creators (status, status_history, alert_state) called
`saveCollection(new Collection(...))` unconditionally, so on a volume
where the collection already exists they throw
`UNIQUE constraint failed: _collections.name`, aborting the ENTIRE
migration chain before later migrations (resource_snapshots, future fleet
collections) can run. Guard each with the proven find-or-skip pattern
already used by probe_runs / resource_snapshots, and harden their down
arms to tolerate an already-absent collection. Verified end to end:
deleting those migrations' `_migrations` rows while leaving the
collections in place (the exact staging state), then rebooting the built
image — boots healthy, re-records the migrations cleanly with no UNIQUE
abort and no duplicate collections, and a brand-new collection migration
still applies through the now-clean chain (the pool-fleet path).
PocketBase backend for showcase-ops: Dockerfile + entrypoint for the
Railway-hosted instance, JSVM main.pb.js hook for CORS + request
shaping, migration sequence creating status / status_history /
alert_state collections, CORS config, and the recreate_collections
v1/v2 + drop_history_fail_count schema drift corrections.
Jordan Ritter