Backfills the canonical LGP testid time-picker-cancel on the 'None of these
work' button in the time picker card. Testid-only change — no behavior
modification.
useLangGraphInterrupt is the LangGraph-specific hook from @copilotkit/react-core
and is the wrong framework primitive on a Claude SDK Python backend. The canonical
LGP demo uses useInterrupt from @copilotkit/react-core/v2 — the framework-agnostic
CopilotKit v2 interrupt hook — which is the correct surface for non-LangGraph
backends. Swap the import and call site.
- queue-client.ts: add missing closing brace at EOF (TS1005 after rebase)
- job-producer.test.ts: add required `family: "d6"` to producer fixtures
and remove duplicate `logger` key in startedProducer
- result-aggregator.test.ts: align with per-row try/catch + dedup-lookup
behavior introduced in 0b2f613e0 — add `persisted: true` and
`writeOverlay` to test writers, update B6 contract expectations
- queue-client.test.ts: drop a stray blank line
Adds a 689-line integration test that exercises the full queue lifecycle
to the /api/runs projection (enqueue → claim → terminal → projection)
across all four families. Updates the railway-envs golden + verify-deploy
drivers regression test to account for the new fleet-runs route surface.
Adds the dashboard Ops worker-runs section — family table, worker strip,
run-history drill-down, D0-from-staleness vs D0-from-failure family
annotation with clock glyph, and the per-family silence banner on the
coverage tab. Wires the data layer: DTOs, /api/runs fetchers, polling
hook, and a worker-runs context provider. cell-drilldown / cell-pieces
gain family-aware rendering.
PRODUCER_FAMILY_WIRING (drift-locked set-equal to FLEET_FAMILIES via a unit
test) drives every buildJobProducer call site; the boot-resolved
worker-stale-after window is threaded through BOTH fleet-health and the
shared family-summary projection so they judge staleness against the same
window. Triggered CLI control-plane runs route through familyForLevel so a
registry rename breaks loudly instead of silently enqueueing jobs invisible
to the projection. Test queues across the harness gain a no-op pruneAged
for the new contract.
Read-only /api/runs routes mounted on the control-plane role, backed by
the SHARED memoized family-summary instance (one PB fan-out per TTL).
Bounds: per-route memo, request rate-limit. /health gains the
fleetRuns.lastEvaluatedAt stamp from the family-silence monitor as the
§9 compensating control for a wedged monitor (an external poll detects
the wedge — the monitor cannot report its own host's death).
job-producer takes a family option and stamps it on every enqueue; the
prune-ownership key is the d6 producer's family. family-silence-monitor
rides the existing fleet-health interval (no extra timer to tear down),
keys 6 h rate-limit + recovered one-shot per family, fails open on
PB-down (the meta-alert path must still fire), and renders alert text
from closed-vocabulary parts only (§5.2.1 redaction). control-plane
fire-and-forgets familySilence.tick(now) each fleet-health cycle.
The §5.1 FLEET_FAMILIES registry and the §5.2.1 family-summary projection
that derives per-family outcome / inflight / lastRun / lastSuccessAt from
the PB-backed batches. The memoized variant fan-outs PB reads at most once
per TTL regardless of viewer count — the SAME instance is shared by the
/api/runs routes and the family-silence monitor so a dashboard poll and a
monitor evaluation inside the same TTL cost one PB fan-out total.
queue-client stamps run_id/family onto every enqueue so downstream
projections can attribute jobs to a family-scoped batch; pruneAged
retention legs land here (the d6 producer owns the call, per §4.2).
result-aggregator computes redsIntroduced/redsCleared from claimed/
sequenced job state into probe_runs summary.
Adds run-id/family/worker-id columns to probe_jobs and resource_snapshots,
plus the EnqueueJobInput.family + FleetQueueClient.pruneAged contracts and
the hoisted deriveHealth primitive that downstream projections share. The
fleet-claim PB hook stamps run-id/family at claim time so every later
projection has a stable join key. probes/run-history is updated to read
the new columns.
The auth-middleware-presence regex in queue-client.test.ts hook-parity
suite was anchored to the single-line `}, $apis.requireAdminAuth());`
closer. After oxfmt rewrote fleet-claim.pb.js to its multi-line form
(`},\n $apis.requireAdminAuth(),\n);`) the regex stopped matching
and the test asserted 0 routes were guarded — a false alarm.
Broaden the regex to accept both the single-line closer and the
formatter's split form; the structural intent (each routerAdd
handler-end is followed by the requireAdminAuth middleware) is
unchanged.
PROMOTE_TO_A (defense-in-depth, exploit-class):
- fleet-health.ts reclaim list interpolated workerId raw into the PB filter
literal. workerId is DB-sourced (read back from the workers roster row),
not a compile-time constant, and the same field is escaped via JSON.stringify
at orchestrator.ts:3240 — but the reclaim path was missing the same
hardening. A double-quote in worker_id (corrupt row, buggy self-registration)
would either throw the list (silently skipping this worker's reclaim every
cycle) or widen the filter to claim other workers' jobs. Match the sibling
escape pattern.
PROMOTE_TO_B (doc reconciliations exposed by the CF round-8 audit):
- queue-client.ts COUNT-NAME CAVEAT was stale — the cross-referenced
contracts.ts doc was already updated (commit 80c5c940) but the queue-client
side still asked a future maintainer to make the edit that had already
landed.
- WarmHealthConfig + JobProducerOptions.warmHealth docs said the producer
warms 'every enumerated backend' / 'each enumerated spec'; the implementation
warms gate.specs (post-validation, post-backlog-gate). A fully-backlogged
tick warms nothing. Updated both interface-level docs to match.
- TickResult.reclaimedIndeterminate said 'Of reclaimed, the reclaims...' —
but the field is DISJOINT from reclaimed (sibling SweepResult contract +
the queue-client both state a thrown release lands here exclusively).
Rewrote to 'In addition to reclaimed, ...'.
- TickResult.skippedForBacklog doc named only the dedupe-gate contributor;
the in-function comment correctly documents the fail-CLOSED poisoned-count
fold-in. Expanded the field doc to name both contributors, and to clarify
that the fail-OPEN leg lands in backlogGateFailedOpen separately.
- SweepResult.commErrors pairing equation
(commErrors.length === reclaimed + reclaimedIndeterminate) was asserted
unconditionally on the shared contract, but reclaimedIndeterminate is
optional and fakes may not report the split. Scoped to 'implementations
that report the split'.
The cold-load comm-error supplemental fetch runs CONCURRENTLY with the bulk
pages, so the bulk copy of an aggregate row can be NEWER than the supplemental
snapshot (the row's state changed between the two reads). The previous merge
replaced the bulk row unconditionally — regressing state/observed_at and
potentially fail_count back to the older supplemental values until the row's
next SSE delta (long for slow-cadence aggregates).
Add a freshness guard: when the supplemental row is strictly older (by
observed_at), keep the newer bulk row INTACT — signal-less rather than
chimera (newer core + stale signal). A chimera row would be silently swallowed
by the reducer's signal-PRESENCE no-op check; a signal-less bulk row lets
the next SSE delta restore the real current signal via the
undefined→defined presence flip.
Equal timestamps and unparseable timestamps both prefer the supplemental
(signal-bearing) row — only POSITIVELY-stale supplemental is suppressed,
preserving the cold-load comm-error overlay intent of CF7-F3 #1.
Sibling of the queue-client prose fix (CF7 #10), which flagged this
contract doc as describing only the drain phase: despite the name, the
lease phase's long-expired carve-out also claim-deletes claimed/running
rows (stale created-age, long-expired or unparseable lease) into this
count — no re-queue, no comm error, no reclaimed increment.
The harness contract gained statusSignalHasCommErrorKey (REQ-B
version-skew observability) with its dashboard sibling explicitly left
to the dashboard owner. Add the byte-identity-safe companion to
live-status.ts — placed OUTSIDE the commErrorFromStatusSignal region
pinned by commError-contract-drift.test.ts (only the decode function
source is mirrored; proven by the drift suite staying green) — so
dashboard consumers can distinguish a present-but-undecodable overlay
from a genuinely absent one. Unit-pinned: unknown future kind,
well-formed, absent, and array-expando wire shapes.
The S10 caller's ad-hoc raw Date.parse check could disagree with
isWorkerStale's PB-space-form-normalized parser (engine-lenient vs
anchored), so the unparseable-heartbeat warn did not fire precisely when
the staleness check was blind. Wire the F2-exported heartbeatParseable
companion (contracts.ts documents fleet-health as its intended caller)
and surface a per-cycle unparseableHeartbeats count on the
fleet.health.cycle log — red-green pinned (the count was the new
observable; the cycle log previously never fired for a blind-but-online
corrupt row).
- mergeRowsToMap's disjoint-key divergence warn no longer fires on a
signal undefined⇄defined flip between row groups (live-status.ts:
coreRowFieldsEqual split out of rowsAreNoop): the initial fetch
projects signal away while SSE deltas deliver full rows, so the flip
is expected provenance, not a keyspace violation. upsertByKey's
reducer keeps treating the flip as observable (existing tests pin it).
- __tests__/cell-model.test.ts destructure comment no longer claims
noUncheckedIndexedAccess is enabled (it is not in this package's
tsconfig) nor that the sibling matched (it did not).
- src/lib/cell-model.test.ts row() helper aligned to the
destructure-with-fallback shape the comment describes.
- cell-model.ts:29-31 re-export rationale now cites the actual
importer (__tests__/cell-model.test.ts).
staleness.ts isStale treats an unparseable observed_at as NOT stale
(false-stale would downgrade a live green row), while the FF7 gate in
decodeCellCommError treats unparseable as stale/skip (false-not-stale
would pin an uncleared overlay forever). isStale predates this branch
(blame: d0da6e357, pre-existing on main; untouched here), so per the
blame gate it is left as-is and the divergence is documented at the
FF7 site instead. Reported for the ledger.
decodeCellCommError's staleness gate (cell-model.ts:697) compared only
`now - parsed > staleAfterMs`, which is never true for a future-dated
observedAt — clock skew or a corrupt producer timestamp would pin the
unreachable/pending overlay indefinitely, the same permanent-phantom
failure mode the FF7 unparseable-timestamp skip prevents. A timestamp
more than COMM_ERROR_FUTURE_SKEW_TOLERANCE_MS (5min) ahead of now is
now skipped like an unparseable one; skew within tolerance still
surfaces (pinned by a companion test so the guard can't over-correct).
The D1-D4 gate fires only on d3.exists/d4.exists, so a cell with ONLY
green D5/D6 rows (no e2e/chat/tools rows at all) slipped past it and
rendered a green chip + green d6Effective at achievedDepth=0/
ceilingDepth=0 — a false top-of-ladder claim contradicting the
strictness doctrine (PRESENT-but-null D4 grays; D5-no-data grays the
ladder). A wholly absent D3/D4 family now collapses to the gray
"unverified" chip (same shape as the d4NoData collapse) with red-D5/D6
dominance preserved, and d6Effective stays blocked (null).
cell-model.ts:847-852 (gate) / :905-921 (d6Effective).
One existing fixture (amber pass-through under reclaimed-pending) built
its amber from an absent-D3/D4 map; it now carries green e2e/chat rows
so the chip is genuinely amber through an intact ladder — the test's
never-mask assertion is unchanged.
decodeCellCommError (cell-model.ts) derives the REQ-B unreachable/
pending overlay from row.signal, but the bulk initial fetch projects
STATUS_LIST_FIELDS, which omits signal (useLiveStatus.ts /
live-status.ts:STATUS_LIST_FIELDS) — so on every page refresh rows
materialized with signal undefined and ACTIVE overlays vanished until
an SSE delta happened to re-deliver that row.
Fix option (a) — matching the projection's data-volume rationale (the
signal blob is ~61% of the bulk payload): useLiveStatus now issues a
SUPPLEMENTAL initial fetch, concurrent with the bulk pages, of ONLY
the comm-error candidate aggregate rows (key has no /<featureId>
segment) for the four mirror dimensions, WITH signal, and merges them
over their projected bulk twins by key. That is ~4 rows per
integration, so the bulk projection's first-paint win is preserved.
Per-cell rows under the same dimensions carry heavy parity-diff
signals and are deliberately not re-fetched (a stale per-cell comm
error is only a same/lower-severity tie-break candidate against the
aggregate mirror).
- live-status.ts: new FLEET_COMM_AGGREGATE_DIMENSIONS single source of
truth (d6/d4/e2e-demos/d5-single-pill-e2e); STATUS_LIST_FIELDS doc
updated to the truth (buildCellModel reads signal per cell at render;
the old "only ever read in the drilldown" claim was false).
- cell-model.ts: decodeCellCommError derives its aggregate candidates
from the shared constant (scan order preserved; pinned by the
equal-timestamp tie-break tests).
- useLiveStatus.ts: fetchCommAggregateRows + by-key merge; skipped
entirely for dimension scopes outside the aggregate set; supplemental
failure retries through the same connect() chain (fail loud).
- useLiveStatus.test.tsx: the PB mock now honours the fields projection
(returning full rows for the projected bulk fetch is exactly how this
bug stayed invisible to the suite) and serves the supplemental fetch
from a dedicated fixture; new cold-fetch red-green tests assert the
overlay renders end-to-end with NO SSE delta, the narrow filter
shape, the out-of-set skip, and the in-set narrowing.
- live-status.test.ts: formatter pass on the CF7-F3 #5 test added in
the previous commit (oxfmt).
CR finding recurring 3x; fixture lines pre-existing on the base (f8aee59b62 /
ef76c6d0ff / 2a9b38bbfe). The fleet contract (contracts.ts ServiceJobResult
.aggregateKey) forbids an e2e_d6:<slug> row on the fleet path - the dashboard
only ever reads d6:<slug> - so fixtures pinning e2e_d6:* keys mask the exact
regression class (success-path vs error-path aggregate-key drift) these
suites exist to catch. Behavior-neutral: every assertion pins the same
production logic, now over conformant keys.
- contracts.test.ts: makeResult probeKey/aggregateKey + the projection pin
-> d6:langgraph-python; fleetSurfaceState row key/dimension -> d6
- worker-loop.test.ts: makeDriver returns d6:<slug> (was e2e_d6:<slug>),
unified with the comm-error/driver-error paths that already pin d6:...;
all inline driver-fake fixtures + the buildServiceJobResult /
runClaimedJob aggregateKey assertions swept in lockstep. The
registry-routing `<kind>:routed` tagging keys are deliberately untouched
(they tag by DRIVER KIND to prove dispatch, symmetric across
e2e_smoke/e2e_demos)
- orchestrator.test.ts: driverInputs.key -> the d6:tracer-slug tracer,
slug-ALIGNED across probeKey/serviceSlug/probe_key because the d6 driver
derives its aggregate side-row slug from input.key (deriveSlug) while the
loop filters by d6:<serviceSlug> - a mismatched tracer would surface the
side row as a phantom cell; pass-through proof kept (aggregateKey echoes
the tracer) and strengthened with the "no D5 features declared" signal pin
CR finding on commErrorFromStatusSignal (contracts.ts:433-463, decode-guard
region touched on this branch): a malformed embedded value decodes to
undefined - indistinguishable from "absent" - so a REQ-B overlay written by a
NEWER producer (new PoolCommErrorKind rolled out write-side first, or a
renamed required field) is silently dropped by an older reader.
- document the version-skew hazard on the decode (return type unchanged:
every render path branches on presence)
- export statusSignalHasCommErrorKey(signal) so consumers can count/log
"key present but undecodable" drops; mirrors the decoder's wire-shape
guards (null/non-object/array are never valid signals)
- pinned with tests: unknown future kind, renamed required field, well-formed,
genuinely-absent, and array/non-object wire shapes
- the mirrored commErrorFromStatusSignal REGION is untouched: companion +
docs live OUTSIDE the byte-identity block; verified by running the
dashboard's commError-contract-drift.test.ts (12/12 green) - the dashboard
mirror (shell-dashboard live-status.ts) still needs its own sibling
companion, to be landed by the dashboard owner
CR finding flagged 4x across 2 rounds (lines pre-existing on the base, f8aee59b62):
isWorkerStale (contracts.ts:610-618) did a bare Date.parse with NaN -> false,
making a corrupt heartbeat indistinguishable from a fresh one ("never stale
forever" = silent fleet-health blindness), and lacked the anchored PB
space->"T" normalization the queue-client treats as load-bearing for lease
timestamps (PB_DATE_SEP_RE, replicated locally - not imported, since
queue-client imports contracts).
- normalize the PB date form with the anchored regex before parsing
- keep isWorkerStale(...): boolean compat (unparseable stays not-yet-stale)
- export heartbeatParseable(lastHeartbeatAt) so fleet-health (S10) can
warn/count unparseable heartbeats (caller wiring is a follow-up; the S10
caller is outside this fix's file scope)
- red-green: PB space-form timestamps, corrupt timestamp, fresh/stale ISO
boundary pins (strict > at exactly-the-window)
The single-callback-slot fake let the fleet-health registration silently
overwrite the consumer's, making consumer-loop assertions vacuous with
fleetHealth injected and hiding leaked intervals from the cleared flag.
- assertServiceJobPayload rejects a non-number meta.priority when present
- protocolViolationResult falls back to a jobId-derived aggregateKey
(defense-in-depth; empty probe_key is unreachable via claimNext's G1c gate)
- job-claim memoizes the working auth route so a v0.22 backend pays the
/_superusers 404 probe once per process, not per token expiry
- raceCandidates' deterministic-4xx per-poll re-warn documented as accepted
- deterministicEndpointRejection notes why 401 is deterministic (postFleet
already re-authed once)
shuffleInPlace rng clamp NOT done: lines are pre-existing (d408766cce on
origin/main) — blame-gated out.
The SINGLE-SWEEPER comment claimed a singleton, but runControlPlane wires
four producers over one queue client, so cron overrun can overlap sweeps;
a concurrent sweep cannot see the first sweep's grace set and could
claim-delete a row it just re-queued. Overlapping callers now piggyback on
the in-flight sweep; cross-process replicas remain documented as uncovered
(fleet deploys one control-plane).
The finally's unconditional eviction also fired on the malformed-input
refusal thrown BEFORE the release CAS, dropping the assumed-live lease the
indeterminate-renew containment depends on and killing the heartbeat of a
live job. The eviction is now scoped; the 'DONE either way' header claim is
corrected.
Attempt N can commit server-side then throw; a blind attempt N+1 re-seeded
result_processed:false, un-latching what the consumer aggregated in the
250ms pause window. The cross-call retry path already guarded exactly this;
the same read-before-write guard now applies inside the retry loop, and a
failed pre-read refuses the blind rewrite.
A family kept because countPendingForFamily failed (non-poisoned) was
indistinguishable from a clean zero-backlog read in the tick outcome —
the documented sweepFailed/enumerateFailed ambiguity class. Counted per
family and included in the tick-complete log meta.
A null/undefined/primitive enumerator element threw a TypeError out of the
tick body, rejecting the tick promise and violating the 'tick never
rejects' invariant. Non-object elements are now dropped loudly and counted
in the existing invalid-spec accounting.
The fail-closed gate matched a message substring duplicated in 3 places;
any rewording silently flipped it fail-closed -> fail-open. The refusal is
now a dedicated exported class, the producer gates via instanceof, the test
literal copies use the real class, and a drift test drives the REAL
queue-client refusal through the producer gate.
Jordan Ritter