Railway's GraphQL envelope can carry both a populated 'data' field
and a non-empty 'errors[]' (e.g. soft-deprecation warnings on a
nested field, or a permission gap on an optional sub-selection). The
previous gql() helper threw DiscoverySourceBackendError on any
non-empty errors[], discarding the populated 'data' payload — every
deprecation surfaced as a hard discovery failure.
New rules:
- data missing AND errors[] non-empty → throw (unchanged behaviour
for outright failures).
- data populated AND errors[] non-empty → log
discovery.railway-services.partial-errors at warn, return data;
schema validation downstream still catches shape rot.
Threads ctx.logger into makeGql so the helper can emit structured
warns in the same key namespace as the rest of the source.
Test additions for the R4 fix set:
- empty-string primitive is unresolvable (collision avoidance)
- `0` and `false` primitives still stringify verbatim
- `probe.driver-late-rejection` debug log on post-timeout rejection
- `signal.errName` populated from `err.name` on driver-error
- `signal.errName` populated from ZodError on input-rejected
- e2e_demos sort gate is bypassed for static-targets configs
- `probe.discovery-record-key-shadowed` warn fires on differing key
- key-shadow warn does NOT fire when record key matches interpolated
Hardens two pre-existing tests that were timing-coupled:
1. "respects max_concurrency" used a fixed 50ms setTimeout per worker
to assert peak === 3. Replaced with an explicit barrier: each
worker registers a release-promise on entry, the test awaits a
`barrierReached` promise resolved when inFlight first hits 3,
asserts peak === 3 deterministically, then drains via the
releasers. A half-broken pool that only spawns 2 workers fails by
vitest's per-test timeout — which is the failure mode we want.
2. "does not surface unhandledRejection" tagged the rejection with
a unique marker and filters captured rejections to only those
carrying the marker, so sibling tests' late rejections (if any
leak through process isolation) cannot pollute the count.
`process.off` cleanup migrated to `onTestFinished` for
guaranteed cleanup even if assertions throw.
The interpolation guard short-circuited `undefined` and `null` to the
unresolved-suffix path but let `""` (empty string), `0`, and `false`
flow through `String(value)`. Empty strings have no operator-
meaningful key value AND collide across records: every record with
`""` at the same slot collapses to the same writer key prefix, so
the writer dedupes one sibling onto another silently.
Routes empty-string primitives to the same `__unresolved_<idx>_<n>`
suffix path with `reason: "empty-string"`. `0` and `false`
intentionally still stringify — they're real primitive values and
have unique stringifications across records.
Also corrects the function-level docstring: it claimed empty-path
templates fall through the non-primitive branch, but the code has had
a dedicated `if (path.length === 0)` branch since R2.
The variables fan-out previously degraded EVERY non-fatal error to
'env: {}', including:
- AbortError from a per-tick timeout firing mid-loop (produces N
variables-failed warns instead of one keyed synthetic-error tick)
- DiscoverySourceAuthError from a token rotation race (every
remaining call also 401s; operators see green discovery while
blind to the auth break)
Rethrow both so the invoker surfaces them as one synthetic-error
ProbeResult; transport flakes / backend 5xx / single-service schema
drift keep the documented per-service-degraded behaviour.
Tests: rethrow on first 401 in loop; aborting mid-loop produces ≤1
variables-failed warn instead of one per remaining service.
Two correctness fixes for the probe-invoker fan-out:
1. The shortest-service-first sort for `kind: e2e_demos` was gated only
on `cfg.kind === "e2e_demos"`. A static-targets e2e_demos config
would have records that lack `demos`, so `demoCount(input)` returns
0 across the board and the tie-break on `key` would silently
re-order the authored YAML. Tightens the gate to also require
`"discovery" in cfg` so static configs preserve YAML order.
2. The discovery record `{ ...record, key }` spread silently overwrites
any pre-existing `key` field on the record. If a source's payload
happens to carry `key` (e.g. a Railway service env that exports
`key=...`), it disappears with no signal. Adds a
`probe.discovery-record-key-shadowed` warn-level log when the
record's `key` differs from the interpolated one. Override behavior
unchanged — the interpolated key remains authoritative for the
writer.
When diffProbeSchedules sees a probe removed from config, it must await the
scheduler.unregister() call. If unregister rejects, keep the entry in
probeConfigs so /api/probes still renders it with kind/timeout instead of
falling back to "unknown" — orphan stays visible with proper config.
Also propagate failures through reloadProbes so probeLoader.watch can emit
probes.reload.failed when a reload partially fails.
Three changes that make probe failures legible without parsing free-form
errorDesc strings:
1. Adds optional `errName` field to `ProbeInvokerSyntheticSignal`.
Operators can now distinguish a TypeError from a generic Error from
a ZodError on the dashboard without regex-matching errorDesc. Wired
through both the input-rejected path (`parsed.error.name`) and the
driver-error path (`err.name`).
2. Replaces the `void driverPromise.catch(() => {})` no-op with a
structured `logger.debug("probe.driver-late-rejection", ...)`
call. Late driver crashes (chromium SIGSEGV, post-timeout TypeError)
no longer land silently — the late signal flows into the
orchestrator's debug log stream as correlation fuel for post-mortem
while the primary timeout result still settles.
3. `logErrorWithStack` now ALWAYS emits a parallel
`logger.debug("<msg>.full-stack", ...)` carrying the untruncated
stack alongside the truncated 5-line warn/error. The previous
comment claimed the full stack "lives in the orchestrator's full
log stream if needed" — this commit makes that claim load-bearing
rather than aspirational.
- Reject non-object roots (null/number/string/array) with a distinct
registry-shape-invalid log key — previously crashed enumerate() with
TypeError on null.integrations access.
- Split registry-parse-failed from registry-read-failed so operators
can tell 'file is corrupt' from 'file isn't there' without parsing
err strings.
- Downgrade ENOENT (the steady-state for non-demos consumers like
image-drift, smoke, aimock-wiring) from warn to info; other read
failures (permission denied, abort, IO error) stay at warn.
- Tests cover all four invalid-root cases, the malformed-JSON path,
and the ENOENT info downgrade.
Bumps the upper bound on `timeout_ms` from 900_000ms (15 min) to
1_800_000ms (30 min). The previous cap was rejecting `e2e-demos.yml`
(20 min) at probe-loader parse time — the probe was dead-on-arrival
in production. Unit tests passed because probe-loader unit tests use
stub configs, not the real YAMLs.
The new ceiling gives ~10 min of headroom over today's slowest probe
(e2e-demos), which fans out a Playwright matrix across 32-avg /
38-largest demos × 17 frameworks. Adds a regression test pinned at
the e2e-demos production value (1_200_000ms) so a future cap squeeze
gets caught immediately.
CR-A2.1: pre-fix, `serve(...)` ran BEFORE `scheduler.start()`. If
`scheduler.start()` threw (a stopped-scheduler reentry, future
precondition check, etc.), boot rejected but the bound http.Server kept
listening — leaking one socket per restart loop until OOM. The fix
reorders so `scheduler.start()` runs first; if it throws, no socket is
ever bound. The HTTP server has no construction-time dependency on
scheduler state (the /health probes accept lazy callbacks), so the
reorder is safe and avoids needing a try/catch around start() to close
the server before rethrowing.
Covering test (red-then-green): doMock createScheduler so start()
throws, await boot rejection, then prove the port is silent via fetch.
Note: net.createServer().listen() can succeed for a still-bound orphan
because Node sets SO_REUSEADDR by default on macOS — the probe-bind
cannot detect the orphan. Only an actual fetch can prove the port is
silent.
per-test in /api/probes describe
CR-A2.3 (T-A1): two byte-identical `makeCronRule` declarations existed —
one inside `describe("orchestrator.diffCronSchedules per-rule isolation")`
and one at module top-level. The R28 describe blocks resolved to the
top-level one via function hoisting; the inner duplicate was dead code
that drifted easily during refactors.
Drop the inner declaration; the outer one services both R25 and R28
blocks unchanged.
T-A2 (bonus): the /api/probes describe captured `prevToken` once at
module-load time. If any earlier test mutated OPS_TRIGGER_TOKEN and
forgot to restore it, this describe's afterEach reset to the wrong
value. Move the capture into beforeEach so each test isolates from
cross-test pollution.
CR-A1.6: four related tightenings to the probe_runs collection migration.
- Down-migration's catch now wraps only findCollectionByNameOrId, not
deleteCollection. A real delete failure (FK constraint, permission,
etc.) must propagate so the migration framework can roll back.
Swallowing both calls would have left a half-deleted collection live
in PB while looking like a clean down-migration.
- triggered field is now required:true. The writer always sets it
explicitly (running rows pass true|false), so the schema should match
the contract and fail loud at insert time on a forgetful caller.
- duration_ms field gains options.min:0 to reject negative durations
from clock skew at the storage layer.
- summary maxSize tightened from 2MB to 64KB. The shape is
{total, passed, failed, services?} — well under 64KB. The 2MB
ceiling was an exfiltration sink given the public listRule below;
budget now matches realistic max.
CR-A1.4: trigger rate-limit now stamps lastTriggerAt IMMEDIATELY after
the check passes, BEFORE awaiting scheduler.trigger(). Previously the
stamp landed only on success after the await, so two concurrent requests
could both pass the check before either recorded a timestamp (TOCTOU).
On InflightConflictError 409 the stamp is rolled back so a 409 doesn't
lock the operator out of the 5-minute window — the failed trigger never
actually consumed a slot. Same rollback applies to JSON parse failures
and other unexpected errors so operators aren't stuck with stale lockouts.
CR-A1.7: /api/probes routes are now restricted to ids registered as
probes (i.e. ids returning a non-undefined ProbeConfig from
getProbeConfig). The scheduler hosts non-probe entries too — internal
maintenance jobs (internal:s3-backup) and rule-cron entries
(<ruleId>:cron:<idx>) — and an authenticated trigger token holder could
previously fire any of them via /api/probes/<id>/trigger. The filter
applies in BOTH the GET listing (those entries don't appear) and the
trigger lookup (404 on non-probe ids).
Tests added:
- two near-simultaneous triggers: exactly one 200 + one 429
- 409 inflight rolls back stamp; immediate retry succeeds
- internal:s3-backup not visible in GET /api/probes listing
- POST /api/probes/internal:s3-backup/trigger returns 404
- GET /api/probes/internal:s3-backup returns 404
Several related correctness fixes to the probe-invoker so trigger filtering,
discovery failure semantics, and missing key_template fields all behave
fail-loud rather than silently degrading.
CR-A1.1: invoke() now accepts InvokerTriggerOptions ({filter:{slugs:[...]}})
and filters discovered inputs to the requested slugs before fan-out.
Discovery itself runs against the full roster (so source logs/metrics see
the unfiltered request); only enqueue + driver.run + writer.write are
restricted to matching slugs. Empty filter list = "no slugs match" — kept
honest rather than degrading to "filter undefined → run everything."
CR-A1.2: missing key_template fields no longer collapse to empty strings.
A new strict interpolation walk returns a discriminated InterpResult and
the resolver stamps a synthetic-error preError on records whose template
paths don't resolve. Each bad record gets a unique sentinel key
(<probeId>:invalid-key-template:<n>) so two missing records don't collide
in the tracker.services Map or overwrite each other in the writer.
CR-A1.5: resolveInputs now returns a discriminated {ok, inputs} | {ok:false,
error} result. On enumerate failure the invoker emits a synthetic-error
ProbeResult keyed off the probe id, persists state="failed" in the run row,
and surfaces discoveryFailed:true in the RunSummary. Operators can now
distinguish "no targets configured" from "discovery broke" (the original
bug was both rendered as state=completed/total=0 — fake green).
CR-A1.8: enumerate() is now raced against an abort-driven timeout promise
when timeout_ms is set, so a source that ignores abortSignal cannot stall
the tick indefinitely. Mirrors the executeOne() race pattern. Sources that
do honor abortSignal still abort their underlying work; sources that don't
get bypassed by the race resolution and the invoker treats it as a
discovery failure (state="failed").
CR-A1.9: stabilized BASE_DEPS test clock so tracker.startedAt and
runWriter.start({startedAt}) are guaranteed numeric ms aligned with the
Date instant drivers see via ctx.now(). Drift between Date.now() and
the injected now() was making timing assertions flaky.
Tests added:
- filter.slugs threads through; only matching slugs run + write
- discovery enumerate runs against full roster (filter is post-discovery)
- missing key_template field emits per-record synthetic error, no key
collisions, distinct tracker entries
- enumerate() throw → state=failed + discoveryFailed=true in summary
- enumerate() timeout when source ignores abortSignal → state=failed
within timeout window
- tracker.startedAt and runWriter.start.startedAt are numeric ms
CR-A1.1: scheduler.trigger(id, opts) now passes TriggerOptions through to
the handler so probe-invoker filter.slugs reaches discovery. ScheduleEntry
handler signature widened to accept the optional opts arg; cron-tick path
calls handler() with no opts (default = full discovery).
CR-A1.3: when a handler throws, lastRunSummary is now cleared (set to null)
instead of preserved at the previous successful run's value. Fresh
lastRunStartedAt/lastRunFinishedAt with stale pass/fail counts misled
operators reading getEntry(). The (timestamps, summary) pair is now
internally consistent — successful runs that returned no summary still
preserve the prior one (legacy void-returning handlers rely on this).
Tests added:
- trigger() passes opts to handler; cron-tick passes undefined
- handler throw clears lastRunSummary while keeping fresh timestamps
Previously `schedulerJobCount` was optional and the /health handler
treated a missing callback as "OK by default" (jobCountOk = true). An
orchestrator that forgot to wire the callback would silently report
/health: 200 with zero cron jobs — exactly the misconfiguration the
signal exists to surface.
Promote the callback to a required type-level field and add a runtime
guard in buildServer() so dynamic call sites (compiled with looser TS
settings, or built from unknown/any) also fail loud at boot rather
than reach a misleading /health response.
- ServerDeps.schedulerJobCount: () => number (no longer optional)
- buildServer() throws if the callback is missing at runtime
- /health no longer carries the `jobCount === undefined` branch; the
field is always present in the response and zero is always 503
- Updated existing tests to supply schedulerJobCount: () => 1
- Added regression test asserting buildServer throws when the callback
is omitted (with @ts-expect-error to lock in the type contract)
Production already wires the callback in orchestrator.ts (boot()), so
no production behaviour change beyond the new fail-loud guard.
Sub-second deltas previously rendered as "in 0s" / "0s ago", which
flickered between sides of the boundary on every tick. Collapse to a
single "now" sentinel for |delta| < 1s and add a Number.isFinite guard
that falls back to "—" so non-finite inputs don't cascade into
unreadable output.
Sparkline previously divided unconditionally by (durations.length - 1)
inside a length===1 branch that the early return at length<2 made
unreachable. Removed the dead conditional. Added a small-viewport guard
that falls back to the flat-dash placeholder when innerW or innerH is
non-positive, preventing negative or NaN coords when callers pass
width <= PADDING * 2.
The tabs useMemo closed over handleTrigger but did not list it in its
dependency array. Today the handler is a no-op so the latent bug is
invisible, but once B4b wires this to the real ops API trigger
endpoint the memo would retain a stale closure. Wrap handleTrigger in
useCallback and add it to the tabs deps array so the memo refreshes
correctly.
The trigger menu's "Run specific..." entry was unusable for idle probes
because slugs are sourced from inflight.services, which is null in the
typical manual-trigger case. Hide the entry entirely when serviceSlugs
is empty (Option C); operators can still trigger all targets via "Run
all". A richer ProbeScheduleEntry contract that exposes discovered
targets for idle probes is tracked separately.
Also wraps onTrigger awaits in try/catch so async rejections surface
through a local error state instead of bubbling up as unhandled promise
rejections, and resets the selected slug set on completion so the next
trigger session starts clean.
runTone returned "gray" (unknown) when finishedAt was set but summary
was null, while runStateLabel returned "completed" for the same case —
producing a gray badge labeled "completed", exactly the misleading
combination the tone comment said it was avoiding. runStateLabel now
returns "unknown" so label and tone agree.
When all probes are idle, findNextScheduled previously returned the
entry with the smallest nextRunAt regardless of whether that timestamp
was already in the past, producing nonsensical "Next: smoke 5m ago"
banners. Filter out entries whose nextRunAt is < now and fall back to
the existing "No upcoming runs scheduled." sentinel.
Also typed STATE_ICON as Record<ServiceState, string> so missing service
states fail to compile rather than silently rendering undefined icons.
The local interface in status-tab.tsx was marked TODO and had drifted
from the canonical lib/ops-api definition: it lacked error? and
finishedAt? on service progress entries, and used a bare 'string' for
'kind' instead of the ProbeKind union.
Delete the local interface and re-export the canonical type from
lib/ops-api so existing 'import type { ProbeScheduleEntry } from
"./status-tab"' call sites continue to compile against the single
source of truth (CR-B1.1).
The previous useProbes/useProbeDetail hooks reset aliveRef.current to
true at the top of every effect run. A fetch from the prior cycle that
resolved between cleanup and re-init could pass the new effect's alive
check and clobber state with stale data.
- useProbes: per-effect 'cancelledRef' (mirrored so the imperative
refetch callback can honor it). Cleanup flips cancelledRef true.
- useProbeDetail: per-effect 'let cancelled = false' closure; cleanup
flips it. Each effect run gets its own closure, so a stale fetch from
the prior cycle cannot land setData (CR-B1.3).
- useProbeDetail also clears data immediately on id change so the
panel header (driven by prop) is never out of sync with the body
while the new fetch is still in flight (CR-B1.4).
- useTriggerProbe gains a per-call AbortController whose signal is
forwarded to triggerProbe; unmount aborts any in-flight POST so we
don't fire duplicate runs after the user navigates away (CR-B1.5).
Tests cover: rapid-deps stale-data path, rapid-id stale-data path,
data-clear on id change, and unmount-during-trigger abort.
- ProbeLastRun.summary is now nullable to match ProbeRun.summary; the
server can complete a run with no summary (showcase-ops CR-A1.5).
- ensureOk surfaces body-read failures with a 'body read failed: ...'
marker instead of swallowing them silently (CR-B1.6 / CR-C15).
- New parseJson helper attributes JSON parse failures to the requesting
URL so call sites can tell which endpoint produced bad bytes.
- triggerProbe gains a TriggerProbeOptions interface with an optional
AbortSignal; passed straight through to fetch() so callers can cancel
in-flight POSTs (CR-B1.5).
A6 (subject scope): probe-config-parity.test.ts resolved
`path.resolve(process.cwd(), "config/probes")`, which broke when
vitest was launched from any directory other than `showcase/ops`.
Switch to `import.meta.url`-based resolution: derive `__dirname` via
`fileURLToPath`, then `path.resolve(__dirname, "../../../config/probes")`
walks up from `src/probes/loader/` to the configs dir regardless of
cwd. Updated the comment to drop the "mirrors probe-loader.test.ts"
claim and explain why cwd was wrong.
B1 (test polish): the shortest-service-first integration test's
per-goto sleep was `demoCount × 1ms` (5/20/38ms total). With
concurrency=2 and `Date.now()` resolution at the millisecond, the
`tiny.lastGotoAt < huge.firstGotoAt` assertion could tie at the
millisecond boundary on a fast host. Bump the multiplier to 5×
(25/100/190ms per goto, total 125/2000/7220ms) so the assertion is
well-separated. Bumped the per-test timeout to 15s to accommodate
the wider wall-clock.
Test counts after this round: full-suite 845 passing (up from 842),
2 pre-existing failures unchanged (probe-loader watch + aimock-
fixture-coverage).
A5 (subject scope): the cost-budget paragraph said "17 framework
packages and ~32 demos each, total ~544 page loads" but referenced a
"38-demo service" elsewhere — internal contradiction. Rewrite to:
- explicit ~32-demo average vs ~38-demo largest service
- explicit absolute worst case ~38 × 30s ≈ ~19 min (against the
20-min outer cap)
- explicit "wall-clock bounded by slowest service under bounded
concurrency, NOT aggregate-divided-by-concurrency"
- typical 3-5s per goto stays as the realistic case
No semantic config changes — comment-only.
A2 (subject scope): `loadDemosMap` called `fs.readFile(path, "utf-8")`
which ignored `ctx.abortSignal`. A stalled volume mount could orphan
past the probe's `timeout_ms`. Pass `{ encoding: "utf-8", signal:
ctx.abortSignal }` so an AbortError lands in the same catch as other
fs failures and degrades to an empty demos map (sibling probes still
get the service list, just without demo enrichment).
Added regression test that pre-aborts the discovery context's signal
before calling `enumerate()`, then asserts `registry-read-failed`
warned exactly once and (when GraphQL didn't also abort) every record
has `demos: []`.
A1 (subject scope): the `ProbeInvokerErrorClass` union declared
`"discovery-error"` but the enumerate-throw path returned `[]` — operators
saw zero writes, indistinguishable from "no services matched the
filter". Mirror the misconfigured-discovery sentinel pattern: when
`source.enumerate()` throws, return a single sentinel `ResolvedInput`
keyed `${probeId}:enumerate-failed`. The invoker's existing top-level
sentinel branch writes a synthetic `state:"error"` ProbeResult with
`errorClass: "discovery-error"`. Sibling probes are unaffected.
A3 (subject scope): drop the unconditional `discoveryAbort.abort()` in
the discovery `finally`. Aborting after a successful enumerate signals
timeout-on-success to any source listener that snapshots
`signal.reason` — the inverse of what the signal represents. Listener
cleanup is not a real concern: the controller is local to the function
call and listeners die with it on GC. Updated comment to capture the
rationale.
A4 (subject scope): rewrite the timeout-bound paragraph in
`buildProbeInvoker` to remove the contradiction between "bounded by the
slowest single phase" and the formula `timeout_ms × (1 + ⌈n/c⌉)`. The
formula IS roughly the sum (discovery + ⌈n/c⌉ rounds of executors), so
the framing is now consistent.
A7 (subject scope): the late-rejection unhandled-handler test now waits
on a deterministic `whenRejected` promise resolved by the driver right
after scheduling its late reject(), then waits an additional 250ms (was
50ms) for any `unhandledRejection` to land. Removes the "passes
vacuously on slow CI" risk.
E1 test updated: hanging discovery source now emits ONE synthetic tick
with `errorClass: "discovery-error"` (was zero writes).
B2 polish: tighten `peak >= 2` to `peak === 3` in the max_concurrency
test so a half-broken pool can't pass.
New tests:
- enumerate-throw → one synthetic discovery-error tick
- sibling probes' invocations unaffected by a throwing source
Call-site enumeration:
- `discoveryAbort.signal.aborted` post-success: grep showed only smoke.ts
and e2e-demos.ts use `.signal.aborted` on their OWN per-driver
AbortController instances, not on `discoveryAbort` — A3 is safe.
- `MISCONFIGURED_DISCOVERY_INPUT` identity equality: only the top-level
sentinel branch checks identity; the new `ENUMERATE_FAILED_INPUT`
sentinel is checked in the same branch via `||` and discriminated
by the same identity check.
Threads runtime deps the prior slots flagged so the Status tab works end-to-end:
- buildProbeInvoker now receives `scheduler` + `runWriter` so B7's
ProbeRunTracker registration and probe_runs row start/finish actually fire
in production. Pre-fix both deps were optional and unset, leaving tracker
+ run-history dead code.
- Mounts the /api/probes router (B3) on the live server when
OPS_TRIGGER_TOKEN is set. Skipped with an info log otherwise so existing
tests / dev boots without the env var still come up.
- Threads `getProbeConfig(id)` from a per-probe config map populated in
`diffProbeSchedules`, so the router renders timeout_ms / max_concurrency /
discovery instead of nulls.
- Adds an integration test that boots the orchestrator with a minimal probe
YAML and verifies GET /api/probes returns the probe with non-null config
and the resolved kind.
- Patches FakeScheduler in http/probes.test.ts with a `setEntryTracker`
no-op (B7 widened the Scheduler interface; B3's test fake was integrated
before that change).
lastRun.state plumbing left for follow-up — the scheduler's lastRun fields
don't carry per-run state today, and B3's router hard-codes 'completed'.
Surfacing a real state requires extending EntryStatus + scheduler bookkeeping
through a path that touches several test surfaces; deferred.
The CatalogCell type gained three required fields (integration_name,
feature_name, category_name) before this branch landed; the existing
test fixtures predate that change and surface as 32 typecheck errors
in CI. Add plausible Title-Case names to each fixture so the type
checker is happy. Test logic and assertions are unchanged.
StatusTable gains an optional onSelect(probeId) prop. When provided, rows
become clickable (cursor-pointer) and the action cell stops propagation so
that clicking the trigger button does not also open the drilldown.
StatusTab owns the selection state (useState) and renders StatusDetailPanel
beneath the running panel. The panel returns null when no probe is
selected, so this is a zero-cost addition to the default render.
Adds three new components for the Status tab drilldown:
- StatusRunsList: recent-runs table with relative-time, duration, state badge
(running/completed/failed tones), manual chip when triggered=true, and
pass/total summary with em-dash fallback.
- StatusDurationSparkline: pure-SVG inline sparkline (no charting deps),
oldest-to-newest convention, dash placeholder for <2 points, configurable
width/height.
- StatusDetailPanel: renders nothing when probeId is null, shows loading,
error, and data states from useProbeDetail; composes the runs list and the
sparkline; close button calls onClose.
All three components ship with red-green vitest + RTL coverage (21 new tests).
B5. New test asserts smoke.yml / e2e-smoke.yml / e2e-demos.yml share
an identical `discovery.filter.nameExcludes` list. The three YAMLs'
header comments claim "kept in sync" but no automated check
enforced it — a new infra service added to one would silently flap
red on the others.
Reads the YAMLs directly (not through createProbeLoader) to avoid a
registry-mock bootstrap; the parity invariant is independent of
probe registration.
Compares as sorted arrays so list order doesn't trip the assertion;
only set membership matters for exclusion semantics.
Extends `resolveCell` to read the `d5:<slug>/<featureType>` and
`d6:<slug>/<featureType>` PB row keys produced by the e2e-deep (D5)
and e2e-parity (D6) drivers. The new badges are informational and do
NOT contribute to the rollup — same model as the post-Phase-3 smoke
slot — because the alert engine routes those dimensions independently.
`useLiveStatus` already pulls every dimension via the page-level
unfiltered subscription, so no subscription-shape change is required.
The `d5` / `d6` literals already live in the closed `DIMENSIONS` enum
in showcase/ops/src/types/index.ts.
Adds D5 / D6 chips to `CellStatus` next to the existing E2E badge —
two ~4-line additions that reuse the existing LiveBadge wrapper, no
structural changes to feature-grid / cells-view / parity-view.
B1. Framework count drift: 15 → 17 (matches MEMORY/platform reality).
Aggregate page loads per tick recomputed: 17 × ~32 = ~544.
B2. Wall-clock arithmetic: rewritten to express the actual bound. Under
bounded `max_concurrency`, wall-clock is dominated by the SLOWEST
single service's serial per-demo fan-out (~3 minutes for a 38-demo
service at ~3-5s/goto), NOT aggregate-divided-by-concurrency.
Smaller services interleave in parallel slots and don't extend
wall-clock past the slowest one.
B3. `key_template` comment block: separated from the preceding
`nameExcludes` list with a `#`-only blank comment line so the
visual hierarchy makes clear `key_template` is a sibling of
`source` / `filter` (not under `filter`). Schema confirms this
placement: `loader/schema.ts:DiscoveryBlockSchema` declares
`key_template` directly on `discovery`, not under `filter`.
B4. Outer-timeout side-emit claim: clarified that the per-demo abort
rows on outer timeout are the DRIVER's responsibility, not the
invoker's. Verified against `e2e-demos.ts` lines 452-473: the
driver's per-demo loop checks `abort.signal.aborted` at the top
of every iteration and writes a red side-row before continuing,
so the YAML's existing claim is accurate. Comment now explicitly
distinguishes driver vs invoker behaviour to prevent future
confusion.
Plus a TS typing fix on the new A1 regression test (the rejecting
driver.run now has an unreachable return-block to satisfy the
ProbeDriver signature).
D1. Concurrency-peak test: per-tick sleep bumped from 15ms → 50ms so
"peak >= 2" holds under heavy CI load. Wall-clock impact ≤ ~170ms.
D2. Invoker-timeout-while-driver-ignores-abort test: ceiling raised
from 70ms → 100ms. The assertion still catches the regression
where the invoker awaited the full driver.run() promise (would
cross the 80ms driver-sleep), but stops flaking on slow workers.
D3. driverResolved assertion tightened from `void` to
`expect(driverResolved).toBe(false)`. Given timeout_ms=20 <
driver-sleep=80, the assertion is deterministic and the previous
"non-deterministic" comment was overcautious.
C1. mkCtx() now stubs `fetchImpl` + `abortSignal` so ProbeContext is
fully populated. Both fields are optional today but a complete stub
catches future tightening without rewriting every test.
C2. starter-skip test: replaces `state.closed === false` (weak proxy —
could pass even on a launch+leak regression) with an explicit
`launched` flag captured inside the launcher closure. Asserts the
launcher was never invoked AND no contexts opened.
C3. failure-on-goto test: adds `state.contextsClosed === contextsOpened`
assertion mirroring the happy-path test. Catches browser-context
leaks where a throw inside runDemo() bypasses context.close().
C4. emits-green-without-goto test: documents the load-bearing coupling
between pageScripts.length and the driver's routeless-cell skip
behaviour. The gotoCalls assertion is the actual regression catch;
the array length is documentary.
Adds regression tests for the bucket (c) probe-invoker fixes landed in
the prior commit. Each test asserts a single behavioural contract so a
future refactor that breaks one item produces a localised failure.
E1. Discovery enumerate timeout — when source.enumerate() observes
abortSignal and the per-tick timeout_ms fires, the invoker returns
promptly with zero writes (discovery failed → nothing to fan out).
E2. Writer.write failure isolation — a writer hiccup on one key does
NOT short-circuit sibling targets; all 3 targets attempt their
write, only the failing key throws.
A1 regression test — late driver rejections (after the timeout has
already settled the race) must not surface as
process.on('unhandledRejection'). The driver promise is now caught
via a sacrificial .catch(() => {}) so a well-behaved driver that
rejects on abort doesn't leak.
A3 collision test — two records with missing template paths receive
distinct unique keys (`__unresolved_<idx>_<n>` suffix) so the writer
doesn't collapse them into one row.
A3 non-primitive test — interpolating an object-typed path emits the
unresolved-suffix instead of producing `[object Object]` keys.
A6 array-typed record — falls back to the `{ key }` branch with a
structured warning, rather than spreading the array shape.
A11 errorClass discriminator — four call sites all flow the correct
class through the synthetic ProbeResult signal:
- timeout
- input-rejected
- driver-error
- discovery-source-missing
Wires B10 (SSE interceptor), B11 (parity-compare), and B12 (reference-
capture) into an executable D6 e2e-parity driver. Each tick picks ONE
integration via the scoping helper (weekly-rotation by default; on-
demand via env), runs every D5 featureType for that integration,
captures a ParitySnapshot, and compares against the reference snapshot
on disk under fixtures/d6-reference/.
Files:
- src/probes/helpers/d6-scoping.ts/test.ts — rotation math (ISO week
modulo sorted slug list), on-demand env selector, and the reference-
snapshot loader (ENOENT->missing, JSON/shape errors->invalid).
- src/probes/drivers/e2e-parity.ts/test.ts — driver. Aggregate state
(red/degraded/green) over per-feature severity. Severity: 0 axis
failures->green, 1-2->amber (ProbeState "degraded"), 3-4->red. Skip
semantics: missing script or missing reference -> green note;
malformed reference -> red. Per-turn SSE-interceptor attach pattern,
starter short-circuit, scoping/launcher/goto error mapping.
- config/probes/e2e-parity.yml — Mondays 04:00 UTC, timeout 600_000,
max_concurrency 2, same discovery filter as e2e-deep. Documents on-
demand mode (D6_MODE/D6_TARGET_INTEGRATION).
- scripts/d6-capture-references.ts — CLI for refreshing reference
snapshots; defaults to langgraph-python; --feature targets one type;
exits 1 on any failure.
- src/types/index.ts — adds "e2e_parity" + "d6" to DIMENSIONS.
- fixtures/d6-reference/README.md — points operators at the CLI.
Tests: 37/37 D6-specific (20 scoping + 17 driver). Full suite 1024
passing; 2 pre-existing failures unrelated to D6 (aimock-fixture
coverage missing byoc-json-render fixtures, probe-loader chokidar
flake). YAML config validates against ProbeConfigSchema.
Bucket (c) sweep — pre-existing defects in probe-invoker.ts surfaced during
CR R1. Each item below is independently testable; existing tests already
cover the timeout/race contract (probe-invoker.test.ts) and discovery flow
(e2e-demos.test.ts integration), so no behavioural regressions.
A1. Late driver-rejection unhandled-rejection: attach a sacrificial
`.catch(() => {})` to the driver promise before racing it against the
timeout promise. A well-behaved driver that rejects on abort AFTER the
invoker has settled the race no longer surfaces as
`process.on('unhandledRejection')`.
A2. timeoutPromise dead-code path: resolve the synthetic timeout
ProbeResult directly inside the setTimeout callback rather than
routing through an abort-event listener. Anything else that aborts
the controller in the future cannot starve the timeout promise.
abortCtrl.abort() is still called separately so well-behaved drivers
receive the cancellation signal.
A3. interpolateTemplate empty-key/non-primitive/empty-path collisions:
widen the regex to `[^}]*`, route empty-path/missing-path/non-
primitive-value cases through a structured `probe.template-path-
unresolvable` warning AND a unique `__unresolved_<recordIndex>_<n>`
suffix so colliding paths surface as distinct keys instead of silently
overwriting each other.
A4. discovery-source-missing produces a synthetic dashboard tick: when
the YAML names a source the registry doesn't know, emit a synthetic
`state: "error"` ProbeResult with stable key
`${cfg.id}:misconfigured` and `errorClass: "discovery-source-missing"`.
Operators previously had to check the log stream to notice.
A5. Defensive `Math.max(concurrency, 1)` worker-pool floor. Schema
already requires `>= 1` but the floor protects against a future
schema relaxation silently dropping every input.
A6. Discovery records that are arrays no longer spread into bare `key`
objects. Arrays go through the `{ key }` fallback branch with a
`probe.discovery-record-non-object` warning.
A7. Discovery AbortController gets `.abort()` in the success-path
`finally` so listeners attached by the source clear out promptly.
AbortController is idempotent so the call is safe.
A8. Wall-clock comment updated to truthful bound:
`timeout_ms × (1 + ⌈n/c⌉)` instead of "roughly 2×timeout".
A9. logger.error call sites enriched with `errName` + first-5-line stack
via a new `logErrorWithStack` helper. Truncation matches Slack /
PocketBase render budgets.
A10. inputSchema rejections now log `parsed.error.issues` (structured
array) so nested zod errors are debuggable from the log stream
without re-running the probe.
A11. `errorClass` discriminator added to synthetic-error ProbeResults:
`"timeout" | "input-rejected" | "driver-error" | "discovery-error"
| "discovery-source-missing"`. Type exported as
`ProbeInvokerErrorClass` + `ProbeInvokerSyntheticSignal`. Alert
rules now have a stable taxonomy to branch on without parsing
`errorDesc` prose.
A12. `inputs[idx]!` non-null assertion replaced with explicit
`if (!item) break;` guard in the worker-pool loop.
A13. `writer` field removed from `ExecuteOneOpts` (was unused locally —
only consumed via `ctx.writer` which now reads `parentWriter`).
Call-site enumeration: `ExecuteOneOpts` is local to this file (no
external consumers); the synthetic-result type was previously inline so
no consumer was checking the old shape; `interpolateTemplate` has one
caller in this file.
The "Until PD1 lands the sort, this block is RED" paragraph predates
the merge of the invoker sort. The tests are green and have been since
PD1 landed. The surrounding context comment that describes what the
test actually catches (regressions where the sort is bypassed) stays.
Update the invoker's sort comment block to point to where demos
actually arrive on the input - the railway-services discovery source's
loadDemosMap, which reads registry.json once per enumerate() call.
Previously the comment was correct in shape but silent on the source
of input.demos, so a future maintainer reading the gate would not know
the sort depends on the discovery enrichment landing.
Also refresh demoCount's docstring to drop the misleading "zod-validated
by e2e-demos.ts:inputSchema" claim - the inputSchema runs inside
executeOne AFTER the sort has already used the count, so it cannot
gatekeep the sort path.
YAML: e2e-demos.yml's max_concurrency block previously said the sort
read from "the registry/discovery record's demos array, zod-validated
by the driver schema". Both halves were misleading: the registry vs.
discovery distinction mattered (now: registry.json joined onto the
discovery record at enumerate time), and the zod-validated claim was
wrong (zod runs in executeOne, which is post-sort). Rewritten to
describe the real plumbing.
The probe-invoker's shortest-service-first sort for e2e_demos reads
input.demos to compute demoCount(input). Until now, the railway-services
discovery source emitted records with no demos field, so demoCount was
always 0 and the sort degenerated to key-only ordering. The lazy
demosResolver in the e2e-demos driver runs INSIDE driver.run, AFTER the
invoker has dispatched, so it cannot feed the pre-dispatch sort.
Discovery now reads registry.json once per enumerate() call, parses
integrations[].slug -> demos[].id, derives the slug from the Railway
service name (showcase-<slug>), and attaches demos: string[] to every
emitted record. Read failures degrade gracefully: a structured warn on
discovery.railway-services.registry-read-failed and demos: [] for every
service. A missing/unreadable registry must not abort the tick - sibling
probes (smoke, image-drift) need their service list regardless.
Path resolution honours REGISTRY_JSON_PATH for tests/dev, falling back
to /app/data/registry.json (matches the e2e-demos driver's default and
the production Dockerfile mount).
Call-site enumeration of the RailwayServiceInfo interface change: the
new readonly demos: readonly string[] field is non-breaking. Existing
consumers (smoke, image-drift, aimock-wiring) read by named fields only
and ignore unknown keys, so adding demos has zero behavioural impact on
them. New consumer is the probe-invoker's e2e_demos sort, which now
reads a real count instead of always 0.
Tests: 4 new cases in railway-services.test.ts covering happy-path slug
join, missing-from-registry default, registry-read-failure warn path,
and REGISTRY_JSON_PATH override. All red before this commit, green
after. Full discovery suite: 42/42 passing.
Three Hono routes mounted at /api/probes:
- GET /api/probes : full schedule + execution state
- GET /api/probes/:id : probe + last 10 runs from probe_runs
- POST /api/probes/:id/trigger : bearer-auth gated, 5-min rate-limited
The list/detail handlers iterate scheduler.list() and synthesize the
EntryStatus + ProbeConfig + nextRunAt into the spec's JSON envelope,
including duck-typed tracker.snapshot() for inflight rendering. The
trigger handler enforces a per-probe 5-minute window held in-process
(sufficient for showcase-ops's single-instance deploy), passes the
optional filter.slugs through to scheduler.trigger() so when B7's
invoker honours it the wiring is already in place, and translates
InflightConflictError into 409 {error:"inflight"}.
Cache-Control: no-cache on every response per spec — the dashboard
client polls these endpoints and any proxy caching would mask
in-flight state transitions.
Auth construction is fail-loud at boot (MissingAuthTokenError when
neither triggerToken nor OPS_TRIGGER_TOKEN is set) so a misconfigured
deploy can't silently expose an unauthenticated trigger.
server.ts gains a single optional `probes` dep on ServerDeps; the
existing /health and /metrics paths are untouched. Older test setups
that only need /health continue to work without supplying scheduler
plumbing.
Closes B3 / blitz-status part 1 (API endpoints) + part 6 (auth).
Probe-invoker now registers a ProbeRunTracker on the scheduler entry for
the duration of each run, calls enqueue/start/complete/fail per service,
and clears the tracker on completion. Returns a RunSummary so the
scheduler populates lastRunSummary, and writes a row to the probe_runs
collection (start at run start, finish on completion). PB writer
failures are logged but never thrown — observability stays best-effort.
Also: scheduler now imports the real ProbeRunTracker class from
probes/run-tracker.ts (was a structural placeholder pre-B2) and exposes
setEntryTracker(id, tracker) so the invoker can mutate the slot's
tracker field through the public API instead of the read-only snapshot.
Provides the data layer for the dashboard's Status tab against the
showcase-ops HTTP contract being built in parallel (B3).
- lib/ops-api.ts: types + fetchProbes / fetchProbeDetail / triggerProbe.
baseUrl resolves explicit param → NEXT_PUBLIC_OPS_BASE_URL → /api/ops.
Trigger sends Bearer token from caller; URL-encodes ids; uniform
status-line errors on non-2xx.
- hooks/use-probes.ts: useProbes / useProbeDetail / useTriggerProbe.
10s default polling, AbortController cancels in-flight on unmount or
interval rollover, AbortError suppressed from user-visible state.
33 new tests (red→green), full shell-dashboard suite green (163/164,
1 pre-existing skip).
Jordan Ritter