P1-A (cap semantics): the MAX_RECLAIM_ATTEMPTS cap was keyed on
`reclaim_count`, a LIFETIME tally bumped by BOTH the sweeper re-queue
path AND the peer-worker expired-lease steal (claim CAS). A long-lived
job that accrues benign peer steals could exhaust its 3-budget and then
get claim-DELETED on its first real orphan rather than re-queued.
Fix: introduce a dedicated `consecutive_orphan_count` column (migration
1779990400) that is bumped ONLY by the sweeper re-queue path in the
fleet-claim release CAS, and reset to 0 on every terminal done|failed
release. The peer-worker steal (claim CAS wasExpiredSteal branch) does
NOT touch this counter. The reaper's cap check now uses
`consecutive_orphan_count` instead of `reclaim_count`. `reclaim_count`
is left intact as the lifetime dashboard diagnostic (jobs.reclaimed).
P1-B (low boundary): adds a test at consecutive_orphan_count = MAX-1
(= 2) asserting the row is RE-QUEUED, not deleted. The off-by-one
mutation `>= MAX` -> `>= MAX-1` causes this test to go RED.
Test-fake honesty: `makeReclaimClaim`'s claimJob now explicitly models
the steal-bump on `reclaim_count` (matching the real hook) while
intentionally NOT bumping `consecutive_orphan_count`, and adds an
explicit pin test confirming steals do not consume the reclaim budget.
JSDoc on MAX_RECLAIM_ATTEMPTS updated to describe the correct semantics:
consecutive re-orphans scoped by sweeper re-queue, reset on terminal.
Red-green proof:
- P1-A RED: revert cap to reclaim_count → "P1-A CAP SCOPE" fails with
`expect(undefined).toBeDefined()` (job deleted instead of re-queued)
- P1-A GREEN: consecutive_orphan_count cap → test passes (re-queued)
- P1-B RED: mutate `>= MAX` to `>= MAX-1` → low-boundary test fails
- P1-B GREEN: revert mutation → low-boundary test passes
Suite: 145 queue-client + 103 producer = 248 total, all green.
Layer (a) of the worker reclamation+rollover redesign: make a worker bounce
non-lossy. The reaper's long-expired carve-out (G1d) used to claim-DELETE an
orphaned in-flight (claimed/running) row whose lease expired beyond its
family's stale window AND whose created-age was past that window —
`reclaimed=0, expiredPending++` — silently dropping work an abrupt bounce
(SIGKILL past grace / OOM / crash) left mid-flight.
Invert it to RECLAIM-WINS-UNTIL-CAP: re-queue the orphan to pending (it
re-runs; idempotent probes make at-least-once safe) until its durable
`reclaim_count` reaches MAX_RECLAIM_ATTEMPTS (3), only then claim-deleting a
row that keeps re-orphaning so a poison job cannot loop forever.
The carve-out existed to dodge an honesty bind: re-queueing a `created`-stale
row emitted a "back in flight" gray the next sweep falsified by claim-deleting
it off the renewal-immune `created` age. Dissolve the bind with a new
`requeued_at` column (migration 1779990300) the release CAS stamps on every
pending re-queue; both stale phases now age off `staleAgeAnchorMs`
(`requeued_at ?? created`), so a reclaimed row is genuinely young again and the
next sweep does not delete it. `reclaim_count` (migration 1779990200) is reused
as the attempt counter — no second tally to drift.
Red-green proven on the real reaper (queue-client.test.ts): a stale-aged
long-expired orphan below the cap goes RED (deleted, reclaimed=0,
expiredPending=1) on delete-wins and GREEN (re-queued, reclaimed=1,
expiredPending=0, requeued_at stamped) on reclaim-wins; plus an attempt-cap
deletion test and a next-sweep no-falsification test.
Adds run-id/family/worker-id columns to probe_jobs and resource_snapshots,
plus the EnqueueJobInput.family + FleetQueueClient.pruneAged contracts and
the hoisted deriveHealth primitive that downstream projections share. The
fleet-claim PB hook stamps run-id/family at claim time so every later
projection has a stable join key. probes/run-history is updated to read
the new columns.
- report() retry: a null getOne resolution is a FAILED read (throw, no blind
write) and a "" result is PB's unset-JSON shape (absent → write proceeds).
- sweepExpired: thrown-release conservative maybes now counted on a separate
reclaimedIndeterminate (SweepResultWithIndeterminate); reclaimed counts only
CAS-confirmed re-queues. Producer one-liner documented for when the
sibling-owned TickResult gains the field.
- job-claim 401 retry: snapshot the token the failed request used; only null
authToken if unchanged (no clobbering a concurrently refreshed token).
- decode-failure synthetic result write: single attempt, no 250ms retry pacing
inside the claim race (consumer crash-synthesis is the documented backstop).
- fleet-claim.pb.js: typeof jobId !== "string" → 400 in all three handlers.
- docs: recent-lease bound is expiryPeriods × period (not one window); claim
5xx→won:false bounded false-overlay source; report retryability deploy-skew
note; drainStalePending page-advance indeterminacy note.
- release: status is REQUIRED (the old '|| done' fallback silently
finished a job whose caller omitted/emptied status) — 400 like
jobId/workerId
- claim+renew: leaseSeconds floored at 1s (0.001 previously yielded a
1ms lease — instantly stealable, renew thrash)
- all handlers: reject non-string workerId (a JSON number coerces into
the text claimed_by column and the holder can never renew/release)
- claim: same-holder live-lease re-claim answers claimed:true with an
alreadyHeld:true marker (timeout-after-commit retry no longer abandons
a row the worker actually holds); client treats it as a plain win
(ClaimEndpointBody documents the no-op marker)
- hook-parity pins updated/added for every change
(i) per-candidate try/catch in claimNext's CAS race — a thrown transport
claim no longer aborts the whole rotation (warn + next candidate);
(ii) discoverPendingFamilies' duplicate-family defensive break now warns
before breaking; (iii) renew re-read triage split — a decodePayload
failure logs as a protocol violation, not a read blip; (iv) the
decode-failure synthetic result uses an injected clock (new config.now)
instead of new Date(); (v) backslash charset guard for family clause
building (the equality/LIKE escape contracts contradict for backslash —
skip such families with a warn) and the over-claiming VERIFIED comment
scoped to %/_ only; (vi) single-sweeper assumption documented as
load-bearing at the grace-set declaration; (vii) hook comment misname
fixed (worker-crashed-mid-job -> worker-reclaimed-pending); (viii)
claimJob maps a 5xx claim response to a lost CAS (won:false, warn) —
a WAL serialization error escaping runInTransaction surfaces as 500 —
while 4xx and renew/release 5xx still throw loud.
After a release-CAS success + result-write exhaustion, report() throws;
a natural retry got REFUSED (row already terminal) and emitted an error
claiming the result is discarded and the job re-runs — both false (the
result is still writable by this holder; terminal rows never re-run).
The hook's release response now carries a refusal reason
(refused_terminal_same_holder / refused_not_holder /
refused_lease_live), threaded through job-claim's ReleaseResult.
report() treats refused-terminal-under-my-workerId as the second leg of
a timeout-after-commit retry and proceeds to writeResult; the
not-holder error is reworded to 're-runs only if reclaimed to pending'.
A reason-less refusal still fails closed.
Stale-pending age is anchored on PB created and never re-anchored on
re-queue: a job running longer than its family expiry window got
lease-reclaimed with a 'back in flight' comm error, then claim-deleted
by the NEXT sweep before any plausible re-run — the dashboard
permanently showed 're-queued' for silently-discarded work. Schema-free
fix: the release hook now RETAINS the expired lease_expires_at on a
pending re-queue (claim admits pending rows regardless of lease), and
the stale phase skips rows whose retained lease is recent (parseable
and within now - familyExpiryWindow) — recently in flight means the
created-based age is stale evidence. Comments cover the heuristic, the
requeued_at-column alternative, and the sweeper-garbage lingering
tradeoff; hook parity test pins the retention.
The three /api/fleet/* routerAdd handlers carried no auth middleware —
a middleware-less PB 0.22 routerAdd handler is PUBLIC, so any
unauthenticated caller could claim/renew/release arbitrary jobs despite
the header claiming superuser auth was required. Append
$apis.requireAdminAuth() (verified against PB 0.22.21 JSVM types) to
all three routes; the client already authenticates as superuser with a
401-reauth retry, so enforcement is compat-safe. Also clamp
leaseSeconds in claim+renew (numeric only, 3600s ceiling, 30s default
on garbage) and pin both contracts in the hook-source parity tests.
sweepExpired (and fleet-health's reclaim) decide 'expired' from a listed
SNAPSHOT, then releaseJob(jobId, holder, 'pending') authorizes on
claimed_by alone — a worker renewing between the list and the release
still matches claimed_by, so a live just-renewed job was yanked back to
pending (duplicate execution + a false worker-reclaimed-pending comm
error). The /api/fleet/release hook now refuses a pending-target release
while the row's CURRENT lease is still live, re-checked inside the same
transaction with the leaseExpired helper that stays byte-equivalent to
the client's anchored parse. The client needs no change: released:false
already maps to the sweep's skip path. Pinned by a hook-source parity
test plus a renewed-after-list race test against a hook-faithful fake.
Add written_by/state_written_at columns (PB migrations) and stamp every
status write with a stable host-derived writer identity. The status
writer now detects cross-writer state flips and foreign writes (the
anti-dual-writer flap-comb defense), normalizes observedAt to PB-safe
RFC-3339 shapes before date-field writes, and classifies writer errors
honestly (401 auth vs 403 permission split, new pb_not_found reason).
Shared single-line CVDIAG log format (redacted 12-char header prefix), x-diag-run-id/x-diag-hops correlation header constants, and a best-effort PocketBase diag_events sink (anonymously HTTP-readable) so the CV/x-aimock-context propagation chain can be traced mid-incident without Railway log access.
Wire the resource-snapshot writer back into the fleet worker path with
per-replica attribution, add the worker_id column migration, and stamp
the legacy single-process path's worker_id as "" (empty string) to match
PocketBase storage and the surrounding codebase convention.
probe_runs.triggered was `required: true`, and PocketBase rejects the
boolean `false` as empty (validation_required). The fleet aggregator opens
every run-history row with `triggered: false` (scheduled, not ad-hoc), so
run-history start() failed on every non-triggered run. Add an idempotent
migration that ALTERs the existing collection's field to optional (the
create migration already ran on staging/prod volumes, so we alter rather
than recreate).
A PB image freshly built from main crash-loops staging PocketBase (502s)
because of two latent defects, both verified by booting the built image
against a real PB 0.22.21 binary on a volume that already has the
collections but has NOT recorded their migrations in `_migrations`.
1. Hook API. `pb_hooks/main.pb.js` registered its CORS middleware via the
bare global `onBeforeServe(...)`, which is undefined in PB 0.22.x JSVM
(only the `$app.onBeforeServe()` Go method exists) — it throws
`ReferenceError: onBeforeServe is not defined` at hook load and crashes
the server. Switch to the documented global `routerUse((next) => (c) =>
…)` entry point. Separately, the per-request closure runs in PB's pooled
goja runtime where top-level helpers/consts are out of scope, so calling
them throws per request and the router returns HTTP 400 on EVERY route;
inline the entire allowlist/env/match logic into the closure to fix that
second regression. Verified: health 200, collection reads 200, the
allowlisted origin is echoed on `Access-Control-Allow-Origin`, a
non-allowlisted origin is not, and OPTIONS preflight returns 204.
2. Migration idempotency. `1777700000_create_baseline.js` and the three
original `1745193*` creators (status, status_history, alert_state) called
`saveCollection(new Collection(...))` unconditionally, so on a volume
where the collection already exists they throw
`UNIQUE constraint failed: _collections.name`, aborting the ENTIRE
migration chain before later migrations (resource_snapshots, future fleet
collections) can run. Guard each with the proven find-or-skip pattern
already used by probe_runs / resource_snapshots, and harden their down
arms to tolerate an already-absent collection. Verified end to end:
deleting those migrations' `_migrations` rows while leaving the
collections in place (the exact staging state), then rebooting the built
image — boots healthy, re-records the migrations cleanly with no UNIQUE
abort and no duplicate collections, and a brand-new collection migration
still applies through the now-clean chain (the pool-fleet path).
Normalize the migration schema comments to reality: the heartbeat is 45s (not
the stale "~30-60s"), retention prunes by stable row id (robust to
same-millisecond observed_at ties) rather than a bare timestamp cutoff, and
document the null-vs-unavailable convention — the nullable number fields store
`null` for the `-1` "unavailable" sentinel so post-wedge queries cleanly
separate a measured reading from an unavailable one.
The browser-pool wedge ends in a container restart that clears in-memory
state, and Railway's stdout window rolls off — so stdout/in-memory gauges
are not retrievable post-wedge. Persist the OS resource gauges to a new
`resource_snapshots` PB collection so the forensic history survives the
restart and the PID/thread-ceiling exhaustion is reconstructable after.
- New PB migration `1779989300_create_resource_snapshots.js` (mirrors the
probe_runs idempotent create + public-read pattern).
- `resource-snapshot-writer.ts`: best-effort writer (swallows + logs every PB
error so a missing migration / PB hiccup never breaks the pool) with
ring-style retention (cap last N rows, default 5000, env-overridable).
- Orchestrator wires the writer to the pool's onSnapshot hook, reusing the
shared pb client.
Audit of staging + production PocketBase rules turned up two open holes:
- users.createRule = "" allowed any anonymous client to POST to
/api/collections/users/records and create a real account. The
dashboard exposes no signup UX — operators authenticate via the
superuser credentials in PbAuthPrompt — so create should be
admin-only.
- baseline.updateRule = "" (production only) allowed any anonymous
client to PATCH baseline rows, including silently flipping status
cells and overwriting the updated_by/updated_at audit fields. The
dashboard's baseline edit flow already gates writes behind
PbAuthPrompt, so locking updateRule to admin-only keeps the existing
operator workflow intact while removing the open hole.
Both changes were applied via the PocketBase admin API to staging first
(verified dashboard still renders and live status SSE still flows),
then to production (same verification, plus 403 confirmations on the
anonymous signup + anonymous baseline PATCH requests that previously
returned 200).
All other collection rules already had the right shape: status,
status_history, probe_runs, baseline retain listRule/viewRule = ""
because the dashboard reads them unauthenticated via the PocketBase JS
SDK; create/update/delete rules stayed null because the harness writes
with the superuser JWT.
GHCR auto-links container packages to their source repository when the
image includes org.opencontainers.image.source. Without this label the
package's repository field stays null on the API, which triggers the
daily drift audit alert even though Actions access is configured.
R2-A.11: two robustness improvements to 1777165230_create_probe_runs.js:
- CREATE INDEX statements now use IF NOT EXISTS so a partial-apply
doesn't trip the next migration run on the index DDL step. The
collection-presence gate above already covers saveCollection
idempotency; this covers the per-index path in case PB ever runs
index DDL separately from the schema commit.
- Document why the up-migration's findCollectionByNameOrId catch must
remain broad: PB JSVM does not expose typed error discrimination
(no ErrCollectionNotFound), so we cannot narrow without a runtime
feature change. The down-migration's catch is already narrowed
because it operates on a resolved-or-skip path.
CR-A1.6: four related tightenings to the probe_runs collection migration.
- Down-migration's catch now wraps only findCollectionByNameOrId, not
deleteCollection. A real delete failure (FK constraint, permission,
etc.) must propagate so the migration framework can roll back.
Swallowing both calls would have left a half-deleted collection live
in PB while looking like a clean down-migration.
- triggered field is now required:true. The writer always sets it
explicitly (running rows pass true|false), so the schema should match
the contract and fail loud at insert time on a forgetful caller.
- duration_ms field gains options.min:0 to reject negative durations
from clock skew at the storage layer.
- summary maxSize tightened from 2MB to 64KB. The shape is
{total, passed, failed, services?} — well under 64KB. The 2MB
ceiling was an exfiltration sink given the public listRule below;
budget now matches realistic max.
Per-invocation probe run history. One row per probe tick, distinct from
status/status_history (per-result state machine) — captures run-level
metadata (duration, triggered-vs-scheduled, pass/fail counts) for the
dashboard's "last N runs" widget consumed by B3 (status route) and B7
(probe-invoker hook).
PB migration creates the `probe_runs` collection with public-read /
superuser-write rules, mirroring `status` / `status_history`. Two
indexes: composite (probe_id, started_at DESC) for last-N lookups and
standalone (started_at DESC) for retention sweeps. Up/down migrations
are idempotent (no-op on re-apply, best-effort drop on rollback).
run-history.ts exports PROBE_RUNS_COLLECTION constant + ProbeRunWriter
with start/finish/recent. duration_ms is computed off the persisted
started_at (not a caller-supplied param) so the contract holds even if
clocks drift between start and finish. Filter values are JSON.stringify'd
to defend against probe ids containing PB filter metacharacters.
Tests use the same fakePb pattern as status-writer.test.ts; coverage
includes start state shape, finish state transitions (completed/failed),
duration computation off persisted row, recent() filter+sort+limit, and
empty-result handling.
PocketBase backend for showcase-ops: Dockerfile + entrypoint for the
Railway-hosted instance, JSVM main.pb.js hook for CORS + request
shaping, migration sequence creating status / status_history /
alert_state collections, CORS config, and the recreate_collections
v1/v2 + drop_history_fail_count schema drift corrections.
Alem Tuzlak