Shared single-line CVDIAG log format (redacted 12-char header prefix), x-diag-run-id/x-diag-hops correlation header constants, and a best-effort PocketBase diag_events sink (anonymously HTTP-readable) so the CV/x-aimock-context propagation chain can be traced mid-incident without Railway log access.
Wire the resource-snapshot writer back into the fleet worker path with
per-replica attribution, add the worker_id column migration, and stamp
the legacy single-process path's worker_id as "" (empty string) to match
PocketBase storage and the surrounding codebase convention.
probe_runs.triggered was `required: true`, and PocketBase rejects the
boolean `false` as empty (validation_required). The fleet aggregator opens
every run-history row with `triggered: false` (scheduled, not ad-hoc), so
run-history start() failed on every non-triggered run. Add an idempotent
migration that ALTERs the existing collection's field to optional (the
create migration already ran on staging/prod volumes, so we alter rather
than recreate).
A PB image freshly built from main crash-loops staging PocketBase (502s)
because of two latent defects, both verified by booting the built image
against a real PB 0.22.21 binary on a volume that already has the
collections but has NOT recorded their migrations in `_migrations`.
1. Hook API. `pb_hooks/main.pb.js` registered its CORS middleware via the
bare global `onBeforeServe(...)`, which is undefined in PB 0.22.x JSVM
(only the `$app.onBeforeServe()` Go method exists) — it throws
`ReferenceError: onBeforeServe is not defined` at hook load and crashes
the server. Switch to the documented global `routerUse((next) => (c) =>
…)` entry point. Separately, the per-request closure runs in PB's pooled
goja runtime where top-level helpers/consts are out of scope, so calling
them throws per request and the router returns HTTP 400 on EVERY route;
inline the entire allowlist/env/match logic into the closure to fix that
second regression. Verified: health 200, collection reads 200, the
allowlisted origin is echoed on `Access-Control-Allow-Origin`, a
non-allowlisted origin is not, and OPTIONS preflight returns 204.
2. Migration idempotency. `1777700000_create_baseline.js` and the three
original `1745193*` creators (status, status_history, alert_state) called
`saveCollection(new Collection(...))` unconditionally, so on a volume
where the collection already exists they throw
`UNIQUE constraint failed: _collections.name`, aborting the ENTIRE
migration chain before later migrations (resource_snapshots, future fleet
collections) can run. Guard each with the proven find-or-skip pattern
already used by probe_runs / resource_snapshots, and harden their down
arms to tolerate an already-absent collection. Verified end to end:
deleting those migrations' `_migrations` rows while leaving the
collections in place (the exact staging state), then rebooting the built
image — boots healthy, re-records the migrations cleanly with no UNIQUE
abort and no duplicate collections, and a brand-new collection migration
still applies through the now-clean chain (the pool-fleet path).
Normalize the migration schema comments to reality: the heartbeat is 45s (not
the stale "~30-60s"), retention prunes by stable row id (robust to
same-millisecond observed_at ties) rather than a bare timestamp cutoff, and
document the null-vs-unavailable convention — the nullable number fields store
`null` for the `-1` "unavailable" sentinel so post-wedge queries cleanly
separate a measured reading from an unavailable one.
The browser-pool wedge ends in a container restart that clears in-memory
state, and Railway's stdout window rolls off — so stdout/in-memory gauges
are not retrievable post-wedge. Persist the OS resource gauges to a new
`resource_snapshots` PB collection so the forensic history survives the
restart and the PID/thread-ceiling exhaustion is reconstructable after.
- New PB migration `1779989300_create_resource_snapshots.js` (mirrors the
probe_runs idempotent create + public-read pattern).
- `resource-snapshot-writer.ts`: best-effort writer (swallows + logs every PB
error so a missing migration / PB hiccup never breaks the pool) with
ring-style retention (cap last N rows, default 5000, env-overridable).
- Orchestrator wires the writer to the pool's onSnapshot hook, reusing the
shared pb client.
Audit of staging + production PocketBase rules turned up two open holes:
- users.createRule = "" allowed any anonymous client to POST to
/api/collections/users/records and create a real account. The
dashboard exposes no signup UX — operators authenticate via the
superuser credentials in PbAuthPrompt — so create should be
admin-only.
- baseline.updateRule = "" (production only) allowed any anonymous
client to PATCH baseline rows, including silently flipping status
cells and overwriting the updated_by/updated_at audit fields. The
dashboard's baseline edit flow already gates writes behind
PbAuthPrompt, so locking updateRule to admin-only keeps the existing
operator workflow intact while removing the open hole.
Both changes were applied via the PocketBase admin API to staging first
(verified dashboard still renders and live status SSE still flows),
then to production (same verification, plus 403 confirmations on the
anonymous signup + anonymous baseline PATCH requests that previously
returned 200).
All other collection rules already had the right shape: status,
status_history, probe_runs, baseline retain listRule/viewRule = ""
because the dashboard reads them unauthenticated via the PocketBase JS
SDK; create/update/delete rules stayed null because the harness writes
with the superuser JWT.
GHCR auto-links container packages to their source repository when the
image includes org.opencontainers.image.source. Without this label the
package's repository field stays null on the API, which triggers the
daily drift audit alert even though Actions access is configured.
R2-A.11: two robustness improvements to 1777165230_create_probe_runs.js:
- CREATE INDEX statements now use IF NOT EXISTS so a partial-apply
doesn't trip the next migration run on the index DDL step. The
collection-presence gate above already covers saveCollection
idempotency; this covers the per-index path in case PB ever runs
index DDL separately from the schema commit.
- Document why the up-migration's findCollectionByNameOrId catch must
remain broad: PB JSVM does not expose typed error discrimination
(no ErrCollectionNotFound), so we cannot narrow without a runtime
feature change. The down-migration's catch is already narrowed
because it operates on a resolved-or-skip path.
CR-A1.6: four related tightenings to the probe_runs collection migration.
- Down-migration's catch now wraps only findCollectionByNameOrId, not
deleteCollection. A real delete failure (FK constraint, permission,
etc.) must propagate so the migration framework can roll back.
Swallowing both calls would have left a half-deleted collection live
in PB while looking like a clean down-migration.
- triggered field is now required:true. The writer always sets it
explicitly (running rows pass true|false), so the schema should match
the contract and fail loud at insert time on a forgetful caller.
- duration_ms field gains options.min:0 to reject negative durations
from clock skew at the storage layer.
- summary maxSize tightened from 2MB to 64KB. The shape is
{total, passed, failed, services?} — well under 64KB. The 2MB
ceiling was an exfiltration sink given the public listRule below;
budget now matches realistic max.
Per-invocation probe run history. One row per probe tick, distinct from
status/status_history (per-result state machine) — captures run-level
metadata (duration, triggered-vs-scheduled, pass/fail counts) for the
dashboard's "last N runs" widget consumed by B3 (status route) and B7
(probe-invoker hook).
PB migration creates the `probe_runs` collection with public-read /
superuser-write rules, mirroring `status` / `status_history`. Two
indexes: composite (probe_id, started_at DESC) for last-N lookups and
standalone (started_at DESC) for retention sweeps. Up/down migrations
are idempotent (no-op on re-apply, best-effort drop on rollback).
run-history.ts exports PROBE_RUNS_COLLECTION constant + ProbeRunWriter
with start/finish/recent. duration_ms is computed off the persisted
started_at (not a caller-supplied param) so the contract holds even if
clocks drift between start and finish. Filter values are JSON.stringify'd
to defend against probe ids containing PB filter metacharacters.
Tests use the same fakePb pattern as status-writer.test.ts; coverage
includes start state shape, finish state transitions (completed/failed),
duration computation off persisted row, recent() filter+sort+limit, and
empty-result handling.
PocketBase backend for showcase-ops: Dockerfile + entrypoint for the
Railway-hosted instance, JSVM main.pb.js hook for CORS + request
shaping, migration sequence creating status / status_history /
alert_state collections, CORS config, and the recreate_collections
v1/v2 + drop_history_fail_count schema drift corrections.
Jordan Ritter