- Drop the hardcoded '>6h' duration from the degraded tooltip; the
threshold lives in the producer config and lying about it in copy
misleads operators when the cadence changes.
- Replace the empty-string default branch with a loud unknown-state
fallback gated by a `never` exhaustiveness check, mirroring rowTone /
formatLabel.
- For red and degraded states, append a truncated (80-char) summary of
`row.signal` so the tooltip carries triage context without ballooning.
- When the SSE connection is in error AND the cached row is red or
degraded, append the row's last-known state so operators don't lose
triage context to the offline banner.
- keyFor now throws when slug or featureId contain ':' or '/', preventing
silent collisions in the live-status lookup map (e.g. `smoke:a/b` vs
`smoke:a` + `/b`).
- rowTone() and formatLabel() switch over the State union with explicit
`never` exhaustiveness checks so adding a new variant fails the build
instead of silently mapping to a misleading green/gray default.
- Unknown row states now surface a distinct 'error' tone (rowTone) and a
'?' glyph (formatLabel) rather than a silent green check.
- Tests cover the three throw paths and remain green for the existing
per-feature key shapes.
Adds two feature-parity.json fixtures for the langgraph-python
byoc-json-render E2E spec prompts that the
aimock-fixture-coverage probe flagged as uncovered:
- "Break down revenue by category as a pie chart"
- "Show me monthly expenses as a bar chart"
Responses mirror the worked examples baked into the
byoc_json_render_agent system prompt verbatim, so the fixture
output matches what the live LLM would emit (a single
@json-render/react flat-spec JSON object) and the spec
assertions on [data-testid="pie-chart"] /
[data-testid="bar-chart"] resolve deterministically when the
suite runs against the aimock sidecar.
Replace the real-chokidar watch test with a vi.mock that swaps chokidar
for an in-memory EventEmitter. The original test relied on macOS
fsevents firing within 500ms of fs.writeFile, which fails ~5/5 runs
locally. Same pattern as rule-loader.test.ts, which abandoned its real-
watcher path for the identical reason. Adds a companion test asserting
the 100ms debounce collapses bursty add/change events into one reload.
Probe-loader test suite is now deterministic across 5 consecutive runs.
The previous `readChildCount(page, selector)` body did an in-page
re-resolution cascade across multiple selectors, which obscured which
selector actually drove the count and made the function impossible to
test for selector-fidelity. Swap to a single-selector helper
`readChildCountForSelector` whose evaluate fn embeds the selector via
`JSON.stringify(selector)` (no string interpolation, no XSS surface),
then explicitly cascade at the call site. Test asserts the selector
is interpolated literally in the evaluate fn source so the wiring
stays visible.
Findings addressed: A11.
Three HITL hardenings:
1. Runtime page-shape guard (A9). The `page as HitlPage` cast was a
silent type widening; if a fake page passed by a test lacked
`click()`, the script would silently no-op rather than fail loudly.
Switch to `page as unknown as HitlPage` and add a typeof check
that throws "page is missing click() — cannot drive HITL dialog"
if the method is absent. Applied symmetrically to both HITL
scripts (approve-deny + text-input).
2. Anchored selectors (A10). Both `approveOrDeny` (button selectors)
and `pickTimeSlot` (slot selectors) previously queried the page
globally, which could match buttons on the surrounding chat or in
a tooltip layered above the dialog. Scope every candidate selector
under its dialog/card root: `${dialogSelector} ${sel}`. Test
coverage asserts the scoped form is what's actually used.
3. Post-stable settle (A17). `waitForNextAssistantMessage` previously
returned as soon as the assistant message count grew past
baseline, but follow-up tokens could still be streaming when the
next assertion ran — racing the SSE tail. Add a phase-2 wait for
`ASSISTANT_FOLLOWUP_SETTLE_MS = 1500` of count-stability after
growth detection so the response is fully committed before we
move on.
Findings addressed: A9 (page-shape guard), A10 (anchored selectors),
A17 (post-stable settle).
`readMessageCount` previously matched bare `[role="article"]` as a
fallback, which counted user-input bubbles as assistant messages on
composers that tag both. The poll could "settle" before the assistant
even started streaming because the user's just-sent message bumped
the count past the baseline.
Fix: prefer the canonical `[data-message-role="assistant"]` testid;
fall back to `[role="article"]:not([data-message-role="user"])` so we
never count user articles. Two new tests cover the tagged-assistant
path and the fallback narrowing path with a fake document patched in
via `evaluate`.
Findings addressed: A7.
The contract axis previously dumped both "field absent" and "field
present but wrong type" into one `missing_fields` array, conflating
two operationally distinct signals: a vanished field is a delete,
a typed-differently field is a shape regression — the dashboard
triagers handle them differently. Add `type_mismatched_fields` to the
report `details.contract` shape and route each failure mode to the
correct bucket. The verdict still fails on either bucket being
non-empty; only the breakdown is split. Test coverage updated with a
direct case asserting both arrays land their entries correctly.
Findings addressed: A14.
Three independent fixes in the D6 parity driver, all touching the same
control flow:
1. Stop caching fleet-resolver failures. The pre-fix code assigned
`cache = []` on any error, which permanently disabled D6: a
transient ENOENT during process startup would lock the resolver to
`[]` forever. Now we leave `cache` null on error so the next call
retries (matching what operators get from a process restart) and
log a warning with the registry path + error.
2. Stop appending the invocation slug to the fleet when missing.
Per-invocation appends broke rotation determinism: different
invocations with different slugs all produced different fleet sizes,
and `weekNumber % len` then picked different integrations within
the same tick. Registry drift now correctly results in "this slug
sits out the rotation" with a `self-not-in-fleet` warning the
operators can act on.
3. Emit per-feature side rows on launcher error. Previously, when
Chromium failed to launch we only emitted the aggregate red — every
feature's individual cell would render whatever state the previous
tick left behind, including "green" from an earlier successful run.
The launcher-error branch now writes a `d6:<slug>/<ft>` side row
for every runnable feature with `errorClass: "launcher-error"` and
`errorDesc` (truncated to 1200 utf8 bytes), so the dashboard's
per-feature drilldown stays truthful.
Test coverage updated to assert all three behaviours (fleet-error
retries on next call, self-not-in-fleet warning + scoping note,
launcher-error side rows for every runnable featureType).
Findings addressed: A4 (fleet caching), A12 (self-not-in-fleet),
A13 (launcher-error side rows). Wires in the script loader from the
prior commit via the new `scriptLoader` driver dep.
The script-loader was previously module-private, so the D6 parity
driver and the `d6-capture-references` CLI both ran with an empty
`D5_REGISTRY` whenever they were invoked standalone (without `e2e-deep`
running first in the same process). Symptoms: every featureType
classified as `skippedScript`, D6 silently no-op, D6 capture run
reports "no features captured" that looks like success.
Changes:
* Lift the filename matcher to an exported `D5_SCRIPT_FILE_MATCHER`
constant. The new regex explicitly rejects co-located `*.test.{js,ts}`
files (which would re-import the script under test and trigger
double-registration throws) and `*.d.ts` declaration files. Covered
by 4 new tests asserting accept/reject behaviour.
* Export `defaultScriptLoader` so callers can reuse one canonical
implementation. The D6 CLI now invokes it at startup and fails
fast if the registry is still empty afterwards (with a clear
"check that scripts/d5-*.ts call registerD5Script" hint).
* Replace the `feature as D5FeatureType` cast in the CLI with the
`isD5FeatureType` type guard from this commit's parent.
Findings addressed: A1 (matcher hardening), A2 (loader export +
fail-fast in CLI).
Two changes to the D5 registry:
1. Atomic registration. The pre-fix loop validated and inserted in a
single pass, so a script with `featureTypes: ["a", "b"]` whose `b`
collided would still set `a` before throwing — leaving the registry
in a partial state. Split into a pre-pass that validates every
featureType BEFORE inserting any. Covered by a new test that
asserts neither key is registered when the second collides.
2. Add `isD5FeatureType` runtime type guard backed by a closed-set
readonly mirror of the type union (kept honest by `satisfies`).
Replaces ad-hoc `value as D5FeatureType` casts that would let an
unknown CLI/env-var string slip past the type system. Covered by
tests for accept/reject/non-string/narrowing.
Findings addressed: A6 (atomic registration), A8 (type guard).
The `*/60 * * * *` form is non-canonical: the divisor equals the field
range (0-59), and some cron parsers reject it as invalid stepping.
Switch to `0 * * * *` ("every hour, on the hour"), which every parser
accepts. Also clarify in the e2e-parity comment header that the
`schedule` field IS the source of truth — the orchestrator reads it
directly via `cfg.schedule`.
Findings addressed: A3.
Three test improvements landed together because they share the same test
file and target the same R4 CR findings:
G/I — replace Date.now()+sleep timing in shortest-service-first
integration test with a monotonic ordinal counter. Removes the
millisecond-tie flake risk on fast hosts and drops ~7s of test
wall-clock for a strictly-ordered semantic assertion.
A — pin the resolution order for the driver-internal cap:
E2E_DEMOS_TIMEOUT_MS env > deps.timeoutMs > built-in default.
Two new tests: env override wins, invalid env value falls back.
L — verify driver fires the internal cap MID fan-out and side-emits
errorClass: 'abort' rows for every remaining demo (not just an
aggregate). Per-demo dot semantics depend on this — silent skip
would leave dashboard cells gray rather than red.
After B1 tightened ProbeLastRun.summary to ProbeRunSummary | null, three
sites in status-table.tsx accessed .summary fields without null-guards
(TS2533 errors at lines 121 and 165).
- lastRunTone: when summary is null but state === "completed", return
"gray" (matches runs-list "unknown" semantics — finished but no count
info means we can't claim coverage). state === "failed" still returns
"red" regardless of summary.
- Result cell: when summary is null, render "—" (matches existing
fallbacks for unknown values); otherwise render passed/total as before.
Adds two covering tests for the null-summary completed (gray + "—") and
null-summary failed (red + "—") cases.
Move runIdCounter and nextRunId from module scope into the createScheduler
closure so each Scheduler gets its own counter starting at 1. Previously two
schedulers in the same process (test fixtures, future multi-tenant cases)
shared a single counter and observed leaked, monotonically-increasing IDs
across instances — a real test-isolation defect even though production runs
a single scheduler today.
Adds a covering test that creates two Schedulers, triggers each, and asserts
both first-run IDs end with counter "1" rather than a shared sequence.
Three related fixes to the e2e-demos driver test file:
1. Hoist `const cleanups: Array<() => void> = []` + `afterEach(...)`
from the BOTTOM of the `describe("e2e-demos driver", ...)` block to
the TOP. Vitest's hoisting kept this working at runtime, but the
source-ordering smell was fragile — a future test that pushes
BEFORE the array initializer line would hit a ReferenceError. Top
placement eliminates the footgun.
2. Move `cleanups.push(() => fs.rmSync(tmp, ...))` to IMMEDIATELY
after `fs.mkdtempSync(...)` — before any subsequent `writeFileSync`
that could throw. Previously the push happened AFTER writeFileSync,
so a disk-full / permission throw would orphan the tmp dir for the
rest of the test process's lifetime.
3. Align an inline writer mock that returned `{}` with the surrounding
`mkWriter` helper which returns `undefined`. ProbeResultWriter's
`write` is typed `Promise<unknown>` so both work, but the
inconsistency was a needless code smell.
- makeCtx now provides a fresh AbortController().signal by default
rather than leaving abortSignal undefined; pass abortSignal:undefined
explicitly to opt into the legacy behaviour. Catches regressions
that drop signal forwarding.
- Add abort-propagation test that fires the controller mid-flight and
asserts the next gql() call sees aborted state. The legacy 'threads
ctx.abortSignal' test asserts identity equality but never aborts —
a regression forwarding a different never-aborted signal would
still pass.
- Add concurrent-noise isolation test that stages REGISTRY_JSON_PATH
to a tmp file so 5 parallel enumerate() calls don't fall through to
/app/data/registry.json and emit log noise into CI.
- Add starter slug contract test asserting that
showcase-starter-ag2 → starter-ag2 misses the registry (which uses
bare integration slugs like ag2) and produces demos: [] — matches
the e2e-demos driver's shape-gated fan-out (it doesn't dispatch
for starters anyway).
- Document the same starter-slug contract on
deriveSlugFromServiceName().
CR-C-page.1: `catalog as unknown as CatalogData` violated the project
policy against double casts. catalog.json is generated by pretest/
predev/prebuild from generate-registry.ts and conforms to CatalogData
by construction, so a single `as CatalogData` cast is safe and clearer.
Typecheck: no new errors (pre-existing __tests__ + status-table.tsx
errors are unrelated and unchanged). All 220 vitest tests pass.
The previous parity test only checked nameExcludes across the
canonical trio (smoke, e2e-smoke, e2e-demos). Two gaps closed:
1. namePrefix parity is also load-bearing — a typo on one would
silently scope it to a different service set. Mechanise it in
the same test so a drift in either field fails CI together.
2. Other railway-services-backed YAMLs (image-drift, qa, ...) need
the same infra-exclude floor or must be explicitly exempted.
Glob all YAMLs, find the railway-services-source ones, and
assert their nameExcludes is a SUPERSET of the smoke set. Add
RAILWAY_PROBES_EXEMPT_FROM_EXCLUDES_FLOOR for the two probes
that legitimately don't carry the floor (image-drift checks
ALL services including infra; qa is scoped to one service via
precise namePrefix).
Also add a vacuous-parity guard (smoke set must be non-empty) so
three empty lists can't accidentally pass.
Pick a single canonical framing for the demo-count math: 17 framework
packages averaging ~32 demos each with the largest service carrying
38, total ~544 typical page loads per tick. The earlier "~32 demos
each" + "38-demo service" were not internally consistent.
Also update the timeout_ms block to describe the new orchestrator-
threaded driver-internal cap (E2E_DEMOS_TIMEOUT_MS) so the YAML's
documented behaviour ("driver iterates remaining demos and side-
emits abort rows when the cap fires at 1200000ms") is now truthful
under the singleton-driver registration model.
nameExcludes already use the unquoted style — no quote changes
needed (the parity test is what guards drift now).
The driver was registered as a singleton at orchestrator boot with
no per-probe deps, so its internal hard-cap defaulted to 5 min while
the YAML's timeout_ms documented 20 min — the driver would abort
mid-fan-out 15 min before the invoker's outer race fired,
contradicting the YAML's "driver iterates remaining demos and
side-emits abort rows when the cap fires at 1200000ms" claim.
The driver now reads ctx.env.E2E_DEMOS_TIMEOUT_MS per-run() in front
of the deps default, and the orchestrator projects each probe's
cfg.timeout_ms onto process.env.E2E_DEMOS_TIMEOUT_MS at probe-load
time. Singleton registration stays intact; the env-thread carries
the YAML value into the driver without rebuilding it.
Invalid env values (non-numeric / zero) fall through to deps default
so a typo can't silently disable the cap.
- Remove path.resolve() wrap on the absolute fallback registry path —
path.resolve() is a no-op for absolute paths.
- Drop the unused 'node:path' import.
- Drop the redundant 'env = {}' reassignment inside the variables
schema-success branch; the variable is already initialised to {}
at declaration. Schema-failure path now relies on the initialiser
with an explicit comment explaining the contract.
Railway's GraphQL envelope can carry both a populated 'data' field
and a non-empty 'errors[]' (e.g. soft-deprecation warnings on a
nested field, or a permission gap on an optional sub-selection). The
previous gql() helper threw DiscoverySourceBackendError on any
non-empty errors[], discarding the populated 'data' payload — every
deprecation surfaced as a hard discovery failure.
New rules:
- data missing AND errors[] non-empty → throw (unchanged behaviour
for outright failures).
- data populated AND errors[] non-empty → log
discovery.railway-services.partial-errors at warn, return data;
schema validation downstream still catches shape rot.
Threads ctx.logger into makeGql so the helper can emit structured
warns in the same key namespace as the rest of the source.
Test additions for the R4 fix set:
- empty-string primitive is unresolvable (collision avoidance)
- `0` and `false` primitives still stringify verbatim
- `probe.driver-late-rejection` debug log on post-timeout rejection
- `signal.errName` populated from `err.name` on driver-error
- `signal.errName` populated from ZodError on input-rejected
- e2e_demos sort gate is bypassed for static-targets configs
- `probe.discovery-record-key-shadowed` warn fires on differing key
- key-shadow warn does NOT fire when record key matches interpolated
Hardens two pre-existing tests that were timing-coupled:
1. "respects max_concurrency" used a fixed 50ms setTimeout per worker
to assert peak === 3. Replaced with an explicit barrier: each
worker registers a release-promise on entry, the test awaits a
`barrierReached` promise resolved when inFlight first hits 3,
asserts peak === 3 deterministically, then drains via the
releasers. A half-broken pool that only spawns 2 workers fails by
vitest's per-test timeout — which is the failure mode we want.
2. "does not surface unhandledRejection" tagged the rejection with
a unique marker and filters captured rejections to only those
carrying the marker, so sibling tests' late rejections (if any
leak through process isolation) cannot pollute the count.
`process.off` cleanup migrated to `onTestFinished` for
guaranteed cleanup even if assertions throw.
The interpolation guard short-circuited `undefined` and `null` to the
unresolved-suffix path but let `""` (empty string), `0`, and `false`
flow through `String(value)`. Empty strings have no operator-
meaningful key value AND collide across records: every record with
`""` at the same slot collapses to the same writer key prefix, so
the writer dedupes one sibling onto another silently.
Routes empty-string primitives to the same `__unresolved_<idx>_<n>`
suffix path with `reason: "empty-string"`. `0` and `false`
intentionally still stringify — they're real primitive values and
have unique stringifications across records.
Also corrects the function-level docstring: it claimed empty-path
templates fall through the non-primitive branch, but the code has had
a dedicated `if (path.length === 0)` branch since R2.
The variables fan-out previously degraded EVERY non-fatal error to
'env: {}', including:
- AbortError from a per-tick timeout firing mid-loop (produces N
variables-failed warns instead of one keyed synthetic-error tick)
- DiscoverySourceAuthError from a token rotation race (every
remaining call also 401s; operators see green discovery while
blind to the auth break)
Rethrow both so the invoker surfaces them as one synthetic-error
ProbeResult; transport flakes / backend 5xx / single-service schema
drift keep the documented per-service-degraded behaviour.
Tests: rethrow on first 401 in loop; aborting mid-loop produces ≤1
variables-failed warn instead of one per remaining service.
Two correctness fixes for the probe-invoker fan-out:
1. The shortest-service-first sort for `kind: e2e_demos` was gated only
on `cfg.kind === "e2e_demos"`. A static-targets e2e_demos config
would have records that lack `demos`, so `demoCount(input)` returns
0 across the board and the tie-break on `key` would silently
re-order the authored YAML. Tightens the gate to also require
`"discovery" in cfg` so static configs preserve YAML order.
2. The discovery record `{ ...record, key }` spread silently overwrites
any pre-existing `key` field on the record. If a source's payload
happens to carry `key` (e.g. a Railway service env that exports
`key=...`), it disappears with no signal. Adds a
`probe.discovery-record-key-shadowed` warn-level log when the
record's `key` differs from the interpolated one. Override behavior
unchanged — the interpolated key remains authoritative for the
writer.
When diffProbeSchedules sees a probe removed from config, it must await the
scheduler.unregister() call. If unregister rejects, keep the entry in
probeConfigs so /api/probes still renders it with kind/timeout instead of
falling back to "unknown" — orphan stays visible with proper config.
Also propagate failures through reloadProbes so probeLoader.watch can emit
probes.reload.failed when a reload partially fails.
Three changes that make probe failures legible without parsing free-form
errorDesc strings:
1. Adds optional `errName` field to `ProbeInvokerSyntheticSignal`.
Operators can now distinguish a TypeError from a generic Error from
a ZodError on the dashboard without regex-matching errorDesc. Wired
through both the input-rejected path (`parsed.error.name`) and the
driver-error path (`err.name`).
2. Replaces the `void driverPromise.catch(() => {})` no-op with a
structured `logger.debug("probe.driver-late-rejection", ...)`
call. Late driver crashes (chromium SIGSEGV, post-timeout TypeError)
no longer land silently — the late signal flows into the
orchestrator's debug log stream as correlation fuel for post-mortem
while the primary timeout result still settles.
3. `logErrorWithStack` now ALWAYS emits a parallel
`logger.debug("<msg>.full-stack", ...)` carrying the untruncated
stack alongside the truncated 5-line warn/error. The previous
comment claimed the full stack "lives in the orchestrator's full
log stream if needed" — this commit makes that claim load-bearing
rather than aspirational.
- Reject non-object roots (null/number/string/array) with a distinct
registry-shape-invalid log key — previously crashed enumerate() with
TypeError on null.integrations access.
- Split registry-parse-failed from registry-read-failed so operators
can tell 'file is corrupt' from 'file isn't there' without parsing
err strings.
- Downgrade ENOENT (the steady-state for non-demos consumers like
image-drift, smoke, aimock-wiring) from warn to info; other read
failures (permission denied, abort, IO error) stay at warn.
- Tests cover all four invalid-root cases, the malformed-JSON path,
and the ENOENT info downgrade.
Bumps the upper bound on `timeout_ms` from 900_000ms (15 min) to
1_800_000ms (30 min). The previous cap was rejecting `e2e-demos.yml`
(20 min) at probe-loader parse time — the probe was dead-on-arrival
in production. Unit tests passed because probe-loader unit tests use
stub configs, not the real YAMLs.
The new ceiling gives ~10 min of headroom over today's slowest probe
(e2e-demos), which fans out a Playwright matrix across 32-avg /
38-largest demos × 17 frameworks. Adds a regression test pinned at
the e2e-demos production value (1_200_000ms) so a future cap squeeze
gets caught immediately.
CR-A2.1: pre-fix, `serve(...)` ran BEFORE `scheduler.start()`. If
`scheduler.start()` threw (a stopped-scheduler reentry, future
precondition check, etc.), boot rejected but the bound http.Server kept
listening — leaking one socket per restart loop until OOM. The fix
reorders so `scheduler.start()` runs first; if it throws, no socket is
ever bound. The HTTP server has no construction-time dependency on
scheduler state (the /health probes accept lazy callbacks), so the
reorder is safe and avoids needing a try/catch around start() to close
the server before rethrowing.
Covering test (red-then-green): doMock createScheduler so start()
throws, await boot rejection, then prove the port is silent via fetch.
Note: net.createServer().listen() can succeed for a still-bound orphan
because Node sets SO_REUSEADDR by default on macOS — the probe-bind
cannot detect the orphan. Only an actual fetch can prove the port is
silent.
per-test in /api/probes describe
CR-A2.3 (T-A1): two byte-identical `makeCronRule` declarations existed —
one inside `describe("orchestrator.diffCronSchedules per-rule isolation")`
and one at module top-level. The R28 describe blocks resolved to the
top-level one via function hoisting; the inner duplicate was dead code
that drifted easily during refactors.
Drop the inner declaration; the outer one services both R25 and R28
blocks unchanged.
T-A2 (bonus): the /api/probes describe captured `prevToken` once at
module-load time. If any earlier test mutated OPS_TRIGGER_TOKEN and
forgot to restore it, this describe's afterEach reset to the wrong
value. Move the capture into beforeEach so each test isolates from
cross-test pollution.
CR-A1.6: four related tightenings to the probe_runs collection migration.
- Down-migration's catch now wraps only findCollectionByNameOrId, not
deleteCollection. A real delete failure (FK constraint, permission,
etc.) must propagate so the migration framework can roll back.
Swallowing both calls would have left a half-deleted collection live
in PB while looking like a clean down-migration.
- triggered field is now required:true. The writer always sets it
explicitly (running rows pass true|false), so the schema should match
the contract and fail loud at insert time on a forgetful caller.
- duration_ms field gains options.min:0 to reject negative durations
from clock skew at the storage layer.
- summary maxSize tightened from 2MB to 64KB. The shape is
{total, passed, failed, services?} — well under 64KB. The 2MB
ceiling was an exfiltration sink given the public listRule below;
budget now matches realistic max.
CR-A1.4: trigger rate-limit now stamps lastTriggerAt IMMEDIATELY after
the check passes, BEFORE awaiting scheduler.trigger(). Previously the
stamp landed only on success after the await, so two concurrent requests
could both pass the check before either recorded a timestamp (TOCTOU).
On InflightConflictError 409 the stamp is rolled back so a 409 doesn't
lock the operator out of the 5-minute window — the failed trigger never
actually consumed a slot. Same rollback applies to JSON parse failures
and other unexpected errors so operators aren't stuck with stale lockouts.
CR-A1.7: /api/probes routes are now restricted to ids registered as
probes (i.e. ids returning a non-undefined ProbeConfig from
getProbeConfig). The scheduler hosts non-probe entries too — internal
maintenance jobs (internal:s3-backup) and rule-cron entries
(<ruleId>:cron:<idx>) — and an authenticated trigger token holder could
previously fire any of them via /api/probes/<id>/trigger. The filter
applies in BOTH the GET listing (those entries don't appear) and the
trigger lookup (404 on non-probe ids).
Tests added:
- two near-simultaneous triggers: exactly one 200 + one 429
- 409 inflight rolls back stamp; immediate retry succeeds
- internal:s3-backup not visible in GET /api/probes listing
- POST /api/probes/internal:s3-backup/trigger returns 404
- GET /api/probes/internal:s3-backup returns 404
Several related correctness fixes to the probe-invoker so trigger filtering,
discovery failure semantics, and missing key_template fields all behave
fail-loud rather than silently degrading.
CR-A1.1: invoke() now accepts InvokerTriggerOptions ({filter:{slugs:[...]}})
and filters discovered inputs to the requested slugs before fan-out.
Discovery itself runs against the full roster (so source logs/metrics see
the unfiltered request); only enqueue + driver.run + writer.write are
restricted to matching slugs. Empty filter list = "no slugs match" — kept
honest rather than degrading to "filter undefined → run everything."
CR-A1.2: missing key_template fields no longer collapse to empty strings.
A new strict interpolation walk returns a discriminated InterpResult and
the resolver stamps a synthetic-error preError on records whose template
paths don't resolve. Each bad record gets a unique sentinel key
(<probeId>:invalid-key-template:<n>) so two missing records don't collide
in the tracker.services Map or overwrite each other in the writer.
CR-A1.5: resolveInputs now returns a discriminated {ok, inputs} | {ok:false,
error} result. On enumerate failure the invoker emits a synthetic-error
ProbeResult keyed off the probe id, persists state="failed" in the run row,
and surfaces discoveryFailed:true in the RunSummary. Operators can now
distinguish "no targets configured" from "discovery broke" (the original
bug was both rendered as state=completed/total=0 — fake green).
CR-A1.8: enumerate() is now raced against an abort-driven timeout promise
when timeout_ms is set, so a source that ignores abortSignal cannot stall
the tick indefinitely. Mirrors the executeOne() race pattern. Sources that
do honor abortSignal still abort their underlying work; sources that don't
get bypassed by the race resolution and the invoker treats it as a
discovery failure (state="failed").
CR-A1.9: stabilized BASE_DEPS test clock so tracker.startedAt and
runWriter.start({startedAt}) are guaranteed numeric ms aligned with the
Date instant drivers see via ctx.now(). Drift between Date.now() and
the injected now() was making timing assertions flaky.
Tests added:
- filter.slugs threads through; only matching slugs run + write
- discovery enumerate runs against full roster (filter is post-discovery)
- missing key_template field emits per-record synthetic error, no key
collisions, distinct tracker entries
- enumerate() throw → state=failed + discoveryFailed=true in summary
- enumerate() timeout when source ignores abortSignal → state=failed
within timeout window
- tracker.startedAt and runWriter.start.startedAt are numeric ms
CR-A1.1: scheduler.trigger(id, opts) now passes TriggerOptions through to
the handler so probe-invoker filter.slugs reaches discovery. ScheduleEntry
handler signature widened to accept the optional opts arg; cron-tick path
calls handler() with no opts (default = full discovery).
CR-A1.3: when a handler throws, lastRunSummary is now cleared (set to null)
instead of preserved at the previous successful run's value. Fresh
lastRunStartedAt/lastRunFinishedAt with stale pass/fail counts misled
operators reading getEntry(). The (timestamps, summary) pair is now
internally consistent — successful runs that returned no summary still
preserve the prior one (legacy void-returning handlers rely on this).
Tests added:
- trigger() passes opts to handler; cron-tick passes undefined
- handler throw clears lastRunSummary while keeping fresh timestamps
Previously `schedulerJobCount` was optional and the /health handler
treated a missing callback as "OK by default" (jobCountOk = true). An
orchestrator that forgot to wire the callback would silently report
/health: 200 with zero cron jobs — exactly the misconfiguration the
signal exists to surface.
Promote the callback to a required type-level field and add a runtime
guard in buildServer() so dynamic call sites (compiled with looser TS
settings, or built from unknown/any) also fail loud at boot rather
than reach a misleading /health response.
- ServerDeps.schedulerJobCount: () => number (no longer optional)
- buildServer() throws if the callback is missing at runtime
- /health no longer carries the `jobCount === undefined` branch; the
field is always present in the response and zero is always 503
- Updated existing tests to supply schedulerJobCount: () => 1
- Added regression test asserting buildServer throws when the callback
is omitted (with @ts-expect-error to lock in the type contract)
Production already wires the callback in orchestrator.ts (boot()), so
no production behaviour change beyond the new fail-loud guard.
Sub-second deltas previously rendered as "in 0s" / "0s ago", which
flickered between sides of the boundary on every tick. Collapse to a
single "now" sentinel for |delta| < 1s and add a Number.isFinite guard
that falls back to "—" so non-finite inputs don't cascade into
unreadable output.
Sparkline previously divided unconditionally by (durations.length - 1)
inside a length===1 branch that the early return at length<2 made
unreachable. Removed the dead conditional. Added a small-viewport guard
that falls back to the flat-dash placeholder when innerW or innerH is
non-positive, preventing negative or NaN coords when callers pass
width <= PADDING * 2.
The tabs useMemo closed over handleTrigger but did not list it in its
dependency array. Today the handler is a no-op so the latent bug is
invisible, but once B4b wires this to the real ops API trigger
endpoint the memo would retain a stale closure. Wrap handleTrigger in
useCallback and add it to the tabs deps array so the memo refreshes
correctly.
Jordan Ritter