R2-A.11: two robustness improvements to 1777165230_create_probe_runs.js:
- CREATE INDEX statements now use IF NOT EXISTS so a partial-apply
doesn't trip the next migration run on the index DDL step. The
collection-presence gate above already covers saveCollection
idempotency; this covers the per-index path in case PB ever runs
index DDL separately from the schema commit.
- Document why the up-migration's findCollectionByNameOrId catch must
remain broad: PB JSVM does not expose typed error discrimination
(no ErrCollectionNotFound), so we cannot narrow without a runtime
feature change. The down-migration's catch is already narrowed
because it operates on a resolved-or-skip path.
R2-A.10: ProbeRunTracker.start() spread the existing entry, so a
defensive call on a slug that previously transitioned through complete
or fail carried finishedAt / result / error into the new running entry.
The snapshot then surfaced a "running" service with terminal-state
metadata, which is incoherent.
Replace the spread with an explicit reset to the running shape so a
re-started slug starts fresh.
Tests: assert finishedAt/result/error are undefined after start() on a
previously completed or failed service.
R2-A.7: when pb.getOne returns null (row deleted, lookup race, etc.)
the previous code fell through with NaN duration and called pb.update
on a non-existent id, either throwing on the underlying client or
silently writing junk. Add an explicit null check that logs a warning
and returns early, so finish() is a no-op when the row is gone instead
of producing a stale-row error.
Test: asserts pb.update is not called when getOne returns null.
Five hardening fixes on the /api/probes routes:
R2-A.4: validate filter.slugs is a string[] before forwarding to the
scheduler. The invoker constructs new Set(filterSlugs) on the value, so
a string sneaks through as per-character set membership. Reject malformed
shapes with 400 invalid_filter at the route boundary.
R2-A.5: stamp the rate-limit window AFTER body read + JSON parse +
filter validation, not before. A 400 from validation must not consume
the operator's 5-minute hold. New sequence: auth -> 404 -> body read ->
parse -> filter shape -> rate-limit check -> stamp -> trigger.
R2-A.6: rollbackRateLimit now compare-and-swaps on its own t value
before restoring the prior stamp. Without this, two racing triggers
where A errors after B successfully stamped would have A's rollback
delete B's stamp.
R2-A.8: cap POST body at 16 KiB. Reject oversized payloads with 413
both via Content-Length pre-check and post-read length check (chunked
clients can lie about Content-Length).
R2-A.9: GET /api/probes/:id wraps writer.recent in try/catch so a
transient PB outage degrades to runs:[] + runsError: history_unavailable
with a 200, instead of 500-ing the whole probe-detail page.
Tests: added 13 new tests across five describe blocks covering each
fix, including a concurrent-trigger CAS rollback scenario and chunked
body-size enforcement.
Three correctness fixes in the probe invoker:
R2-A.1: when discovery enumerate fails, the RunSummary previously returned
total=0 with failed=1, violating total === passed + failed. Treat the
discovery itself as the single failed unit so the invariant holds.
Also stop polluting tracker.snapshot().services with a fake entry keyed
by the probe id (cfg.id is not a service slug); the failure is conveyed
structurally via discoveryFailed: true.
R2-A.2: Promise.race against the driver promise left the driver's tail
rejection unhandled when the timeoutPromise won. Under
--unhandled-rejections=throw this can crash. Attach a no-op .catch() to
the driver promise before the race so late rejections are absorbed; the
race outcome is unchanged.
R2-A.3: same fix for the discovery enumerate Promise.race path.
Tests: updated CR-A1.5 invariant assertion plus added two unhandled-
rejection guard tests for driver and enumerate late-reject paths.
Bucket-(b) bonus polish:
- humanizeCron now renders "M H * * *" with a non-zero minute as
"Daily at HH:MM" instead of falling through to the raw expression.
- humanizeCron rejects */0 step values for both the every-Nh and every-Nm
shapes — that's not a valid cron step.
- StatusRunningPanel: rename `completed` to `finished`. The variable
counts services in either completed OR failed states, so "completed"
read as misleadingly success-only. The data-completed attribute name
stays for backwards compatibility with existing tests.
A run with failed=0 but passed<total represents some services that were
skipped/unknown — rendering this as green "completed" misleads operators
into thinking all targets passed (R2-D.4). Now badged amber with a
"partial (N skipped)" label so the partial-coverage case is visible.
Three related fixes to StatusTriggerButton:
- R2-D.2: render onTrigger errors inline via data-testid="status-trigger-error"
instead of discarding the value through a write-only setter.
- R2-D.3: when serviceSlugs prop changes, intersect selected with the new
list so stale slugs cannot be POSTed; also clear selection on
outside-click and Escape close so reopening doesn't show ghost checks.
- R2-D.5: drop unreachable serviceSlugs.length === 0 ternary inside the
picker — the outer guard already requires length > 0.
Replace the no-op handleTrigger stub and empty-list placeholder with real
hooks. useProbes feeds the schedule grid via 10s polling; useTriggerProbe
POSTs /trigger with the operator token. If NEXT_PUBLIC_OPS_TRIGGER_TOKEN
is unset, the trigger throws a clear "token required" error so operators
see fail-loud feedback instead of a silent no-op (R2-D.1).
POST to /trigger is non-idempotent — the server may have already received
the request and queued the run by the time the component unmounts.
Aborting the controller client-side hides the result, but does NOT
unfire the server-side action. Stop aborting on unmount; rely on
aliveRef to gate any setState after teardown.
Back-to-back trigger() calls intentionally abort the prior controller.
The first call's promise was rejecting with AbortError, surfacing as a
user-facing error on the Status tab. Detect AbortError (and the
cancelled-controller case) in the catch block and resolve silently
instead of setError + rethrow. The owning-controller guard on the
finally block prevents a superseded call from flipping pending off
while a newer call is still in flight.
Bonus consistency for useProbes: setLoading(true) on dep change so
consumers can render a "refreshing" indicator across baseUrl swaps.
Tests:
- unmount during in-flight trigger does NOT abort the signal
- post-unmount resolution does not call setState (no act warnings)
- back-to-back triggers: first promise resolves silently, error stays null
ensureOk and parseJson previously caught body-read failures and rewrapped
them in a generic Error, losing the AbortError name. Hooks filter abort
noise via err.name === "AbortError", so the wrap defeated cancellation
filtering and surfaced "AbortError" as a user-facing failure on the
Status tab.
Re-throw AbortError as-is from both helpers; non-Abort body errors keep
the descriptive wrap path. Also tighten ProbeKind to use (string & {})
so known-kind autocomplete is preserved alongside forward-compat strings.
Add covering tests:
- response.text() rejects with AbortError → ensureOk re-throws as-is
- response.json() rejects with AbortError → parseJson re-throws as-is
- non-Abort body errors still wrap with descriptive message
- triggerProbe with already-aborted signal surfaces AbortError unwrapped
Backstop the matrix fixes and address weak/no-op assertions:
- Tighten depth-chip count assertions from '>= N' to exact 'N'
where the fixture pins the count.
- Replace the no-op filter=wired test with a fixture containing a
feature whose every cell is unshipped, then assert that row IS
visible under filter=all and IS HIDDEN under filter=wired.
- Extend the collapse test with a re-expand click that asserts
chip count returns to baseline (regression guard for one-way
toggle state).
- Cover filter=regressions empty-state.
- Cover starter cells (feature === null) in both matrices: the
cell index must skip them without polluting the grid or leaking
a 'null' string into rendered output.
- Pin parity-matrix integration-header ordering across all five
parity tiers (reference, at_parity, partial, minimal, not_wired).
- Cover D2 cap for starter cells in deriveDepth (chat/tools cannot
lift past D2 because D3 is not evaluable without a feature id).
- Replace the locally-declared CatalogCell shape in depth-utils.test
with the source type so tests cannot drift from production.
- Add afterEach(vi.unstubAllGlobals) to both test files so the
localStorage stub does not leak across files in the same worker.
Three related cleanups in cell-matrix.tsx and parity-matrix.tsx:
1. Drop the 'as 0 | 1 | 2 | 3 | 4' typecasts on depth.achieved. The
newly tightened DepthResult.achieved literal-union type makes
them redundant.
2. Skip starter cells (feature === null) when building the
cell-by-key index. The previous code interpolated null into the
key string, producing '<slug>/null' entries that orphaned
starter rows in the index.
3. Render an explicit empty-state in CellMatrix when filter is
'regressions'. The placeholder logic returned false for every
row, silently hiding the entire matrix. Until regression
detection lands (DepthResult.isRegression always false), the
matrix now surfaces a 'Regression detection not yet implemented'
message via data-empty-reason='regressions-not-implemented'.
deriveDepth's return type carried the loose 'number' on achieved, forcing
every consumer to cast to '0 | 1 | 2 | 3 | 4' to satisfy DepthChip's
prop type. Expose AchievedDepth alias and narrow the source so the
casts at use sites (cell-matrix.tsx, parity-matrix.tsx) become
unnecessary.
Per-cell status row corrections + a focused unit-test file covering CP1,
CP2, CP5, CP7, CP8 behaviors:
- CP3: stop consuming `cell.smoke` in CellStatus. resolveCell still
computes the field for backwards-compat with other call-sites, but
per Phase 3 Decision #7 the smoke L1 signal is integration-scoped and
lives in the per-integration strip — there's no per-feature smoke
badge in the cell row. (The CellState type narrowing for the `smoke`
field is owned by the live-status.ts fix in a sibling worktree; this
patch only removes the consumer.)
- CP8: hide the D5/D6 LiveBadges when feature.kind === "testing".
The e2e-deep / e2e-parity producers only emit rows for primary
features per spec; testing features would otherwise render a
perpetual gray "?" that adds noise without information. Mirrors the
docs-row visibility rule (already gated on isTesting).
- CP9: document the missing per-cell drilldown URL convention for D5/D6
with a TODO comment. No drilldown route exists today; once one does,
wire it through keyFor here.
Tests cover red-green for CP1 (mouseleave handler wired), CP2 (first /
error / green_to_red transition formatting), CP5 (opt-out vs absent
tooltip copy), and CP8 (D5/D6 hidden for testing features). The
existing docs-link.test.tsx still passes.
- CP1: wrap Badge in a span that resets `tooltipOpen` on mouseleave/blur
so the lazy-fetch flag can flip back to false. Without this, the flag
stayed true forever after the first hover, defeating the §5.6
optimization and (once the hook upgrades to a subscription) leaking it.
- CP2: discriminate `first` and `error` transitions in the tooltip copy
so operators can tell "first observation (initial: <state>)" apart
from "producer error (error → <state>)" instead of seeing the same
fallback string. Logic moved into a named `formatTransitionLine`
helper for testability.
Four fixes to the per-cell docs row:
- CP4: extract the `unselected` shell-routing fallback into
SHELL_UNSELECTED_PATH so the coupling is documented and easy to find.
- CP5: split the `missing` tooltip into "framework opt-out" vs "no docs
URL declared" so operators can distinguish a framework that declined
docs from a registry entry that never declared one.
- CP6: add `default: never` exhaustiveness guard to DocsLink's switch so
any new DocState variant is a compile error until handled.
- CP7: render `<a>` (not `<span>`) for `state === "error"` when an href
is present so operators can manually verify a flaky probe — `notfound`
stays a span because the URL is positively known broken.
LS6: `mergeRowsToMap` now logs a `console.warn` when the same key is
seen twice across the merged groups. Each dimension is supposed to own
a disjoint slice of the keyspace, so a collision is an upstream bug.
Return semantics are unchanged (last-wins remains correct under
eventual consistency), but the warn surfaces the regression in dev mode.
LS7: `upsertByKey` returns the SAME array reference when the incoming
row is observably identical (key + state + observed_at +
transitioned_at). Producers re-emit unchanged rows on every poll tick;
allocating a fresh array for each one defeats React's reference-equality
short-circuit and re-renders every memoised consumer downstream.
Tests cover no-op identity, mutation paths (state change,
observed_at change), the disjoint-merge happy path, and the collision
warn.
LS1: `allGreen` now requires BOTH `healthRow` and `e2eRow` to be present
and green. Previously a missing e2e row was treated as green-eligible,
which let a brand-new cell read green before any e2e probe had ticked —
the same flavour of stale-green lie as the connection=error case.
LS11: smoke lookup now uses the integration-scoped key (`smoke:<slug>`)
to match the producer. Pre-fix, resolveCell looked up
`smoke:<slug>/<featureId>` which always missed, leaving every smoke badge
gray. Tests update fixtures accordingly and add a regression guard so a
stray feature-keyed smoke row cannot bleed into the resolver.
- Drop the hardcoded '>6h' duration from the degraded tooltip; the
threshold lives in the producer config and lying about it in copy
misleads operators when the cadence changes.
- Replace the empty-string default branch with a loud unknown-state
fallback gated by a `never` exhaustiveness check, mirroring rowTone /
formatLabel.
- For red and degraded states, append a truncated (80-char) summary of
`row.signal` so the tooltip carries triage context without ballooning.
- When the SSE connection is in error AND the cached row is red or
degraded, append the row's last-known state so operators don't lose
triage context to the offline banner.
- keyFor now throws when slug or featureId contain ':' or '/', preventing
silent collisions in the live-status lookup map (e.g. `smoke:a/b` vs
`smoke:a` + `/b`).
- rowTone() and formatLabel() switch over the State union with explicit
`never` exhaustiveness checks so adding a new variant fails the build
instead of silently mapping to a misleading green/gray default.
- Unknown row states now surface a distinct 'error' tone (rowTone) and a
'?' glyph (formatLabel) rather than a silent green check.
- Tests cover the three throw paths and remain green for the existing
per-feature key shapes.
Adds two feature-parity.json fixtures for the langgraph-python
byoc-json-render E2E spec prompts that the
aimock-fixture-coverage probe flagged as uncovered:
- "Break down revenue by category as a pie chart"
- "Show me monthly expenses as a bar chart"
Responses mirror the worked examples baked into the
byoc_json_render_agent system prompt verbatim, so the fixture
output matches what the live LLM would emit (a single
@json-render/react flat-spec JSON object) and the spec
assertions on [data-testid="pie-chart"] /
[data-testid="bar-chart"] resolve deterministically when the
suite runs against the aimock sidecar.
Replace the real-chokidar watch test with a vi.mock that swaps chokidar
for an in-memory EventEmitter. The original test relied on macOS
fsevents firing within 500ms of fs.writeFile, which fails ~5/5 runs
locally. Same pattern as rule-loader.test.ts, which abandoned its real-
watcher path for the identical reason. Adds a companion test asserting
the 100ms debounce collapses bursty add/change events into one reload.
Probe-loader test suite is now deterministic across 5 consecutive runs.
The previous `readChildCount(page, selector)` body did an in-page
re-resolution cascade across multiple selectors, which obscured which
selector actually drove the count and made the function impossible to
test for selector-fidelity. Swap to a single-selector helper
`readChildCountForSelector` whose evaluate fn embeds the selector via
`JSON.stringify(selector)` (no string interpolation, no XSS surface),
then explicitly cascade at the call site. Test asserts the selector
is interpolated literally in the evaluate fn source so the wiring
stays visible.
Findings addressed: A11.
Three HITL hardenings:
1. Runtime page-shape guard (A9). The `page as HitlPage` cast was a
silent type widening; if a fake page passed by a test lacked
`click()`, the script would silently no-op rather than fail loudly.
Switch to `page as unknown as HitlPage` and add a typeof check
that throws "page is missing click() — cannot drive HITL dialog"
if the method is absent. Applied symmetrically to both HITL
scripts (approve-deny + text-input).
2. Anchored selectors (A10). Both `approveOrDeny` (button selectors)
and `pickTimeSlot` (slot selectors) previously queried the page
globally, which could match buttons on the surrounding chat or in
a tooltip layered above the dialog. Scope every candidate selector
under its dialog/card root: `${dialogSelector} ${sel}`. Test
coverage asserts the scoped form is what's actually used.
3. Post-stable settle (A17). `waitForNextAssistantMessage` previously
returned as soon as the assistant message count grew past
baseline, but follow-up tokens could still be streaming when the
next assertion ran — racing the SSE tail. Add a phase-2 wait for
`ASSISTANT_FOLLOWUP_SETTLE_MS = 1500` of count-stability after
growth detection so the response is fully committed before we
move on.
Findings addressed: A9 (page-shape guard), A10 (anchored selectors),
A17 (post-stable settle).
`readMessageCount` previously matched bare `[role="article"]` as a
fallback, which counted user-input bubbles as assistant messages on
composers that tag both. The poll could "settle" before the assistant
even started streaming because the user's just-sent message bumped
the count past the baseline.
Fix: prefer the canonical `[data-message-role="assistant"]` testid;
fall back to `[role="article"]:not([data-message-role="user"])` so we
never count user articles. Two new tests cover the tagged-assistant
path and the fallback narrowing path with a fake document patched in
via `evaluate`.
Findings addressed: A7.
The contract axis previously dumped both "field absent" and "field
present but wrong type" into one `missing_fields` array, conflating
two operationally distinct signals: a vanished field is a delete,
a typed-differently field is a shape regression — the dashboard
triagers handle them differently. Add `type_mismatched_fields` to the
report `details.contract` shape and route each failure mode to the
correct bucket. The verdict still fails on either bucket being
non-empty; only the breakdown is split. Test coverage updated with a
direct case asserting both arrays land their entries correctly.
Findings addressed: A14.
Three independent fixes in the D6 parity driver, all touching the same
control flow:
1. Stop caching fleet-resolver failures. The pre-fix code assigned
`cache = []` on any error, which permanently disabled D6: a
transient ENOENT during process startup would lock the resolver to
`[]` forever. Now we leave `cache` null on error so the next call
retries (matching what operators get from a process restart) and
log a warning with the registry path + error.
2. Stop appending the invocation slug to the fleet when missing.
Per-invocation appends broke rotation determinism: different
invocations with different slugs all produced different fleet sizes,
and `weekNumber % len` then picked different integrations within
the same tick. Registry drift now correctly results in "this slug
sits out the rotation" with a `self-not-in-fleet` warning the
operators can act on.
3. Emit per-feature side rows on launcher error. Previously, when
Chromium failed to launch we only emitted the aggregate red — every
feature's individual cell would render whatever state the previous
tick left behind, including "green" from an earlier successful run.
The launcher-error branch now writes a `d6:<slug>/<ft>` side row
for every runnable feature with `errorClass: "launcher-error"` and
`errorDesc` (truncated to 1200 utf8 bytes), so the dashboard's
per-feature drilldown stays truthful.
Test coverage updated to assert all three behaviours (fleet-error
retries on next call, self-not-in-fleet warning + scoping note,
launcher-error side rows for every runnable featureType).
Findings addressed: A4 (fleet caching), A12 (self-not-in-fleet),
A13 (launcher-error side rows). Wires in the script loader from the
prior commit via the new `scriptLoader` driver dep.
The script-loader was previously module-private, so the D6 parity
driver and the `d6-capture-references` CLI both ran with an empty
`D5_REGISTRY` whenever they were invoked standalone (without `e2e-deep`
running first in the same process). Symptoms: every featureType
classified as `skippedScript`, D6 silently no-op, D6 capture run
reports "no features captured" that looks like success.
Changes:
* Lift the filename matcher to an exported `D5_SCRIPT_FILE_MATCHER`
constant. The new regex explicitly rejects co-located `*.test.{js,ts}`
files (which would re-import the script under test and trigger
double-registration throws) and `*.d.ts` declaration files. Covered
by 4 new tests asserting accept/reject behaviour.
* Export `defaultScriptLoader` so callers can reuse one canonical
implementation. The D6 CLI now invokes it at startup and fails
fast if the registry is still empty afterwards (with a clear
"check that scripts/d5-*.ts call registerD5Script" hint).
* Replace the `feature as D5FeatureType` cast in the CLI with the
`isD5FeatureType` type guard from this commit's parent.
Findings addressed: A1 (matcher hardening), A2 (loader export +
fail-fast in CLI).
Two changes to the D5 registry:
1. Atomic registration. The pre-fix loop validated and inserted in a
single pass, so a script with `featureTypes: ["a", "b"]` whose `b`
collided would still set `a` before throwing — leaving the registry
in a partial state. Split into a pre-pass that validates every
featureType BEFORE inserting any. Covered by a new test that
asserts neither key is registered when the second collides.
2. Add `isD5FeatureType` runtime type guard backed by a closed-set
readonly mirror of the type union (kept honest by `satisfies`).
Replaces ad-hoc `value as D5FeatureType` casts that would let an
unknown CLI/env-var string slip past the type system. Covered by
tests for accept/reject/non-string/narrowing.
Findings addressed: A6 (atomic registration), A8 (type guard).
The `*/60 * * * *` form is non-canonical: the divisor equals the field
range (0-59), and some cron parsers reject it as invalid stepping.
Switch to `0 * * * *` ("every hour, on the hour"), which every parser
accepts. Also clarify in the e2e-parity comment header that the
`schedule` field IS the source of truth — the orchestrator reads it
directly via `cfg.schedule`.
Findings addressed: A3.
Three test improvements landed together because they share the same test
file and target the same R4 CR findings:
G/I — replace Date.now()+sleep timing in shortest-service-first
integration test with a monotonic ordinal counter. Removes the
millisecond-tie flake risk on fast hosts and drops ~7s of test
wall-clock for a strictly-ordered semantic assertion.
A — pin the resolution order for the driver-internal cap:
E2E_DEMOS_TIMEOUT_MS env > deps.timeoutMs > built-in default.
Two new tests: env override wins, invalid env value falls back.
L — verify driver fires the internal cap MID fan-out and side-emits
errorClass: 'abort' rows for every remaining demo (not just an
aggregate). Per-demo dot semantics depend on this — silent skip
would leave dashboard cells gray rather than red.
After B1 tightened ProbeLastRun.summary to ProbeRunSummary | null, three
sites in status-table.tsx accessed .summary fields without null-guards
(TS2533 errors at lines 121 and 165).
- lastRunTone: when summary is null but state === "completed", return
"gray" (matches runs-list "unknown" semantics — finished but no count
info means we can't claim coverage). state === "failed" still returns
"red" regardless of summary.
- Result cell: when summary is null, render "—" (matches existing
fallbacks for unknown values); otherwise render passed/total as before.
Adds two covering tests for the null-summary completed (gray + "—") and
null-summary failed (red + "—") cases.
Move runIdCounter and nextRunId from module scope into the createScheduler
closure so each Scheduler gets its own counter starting at 1. Previously two
schedulers in the same process (test fixtures, future multi-tenant cases)
shared a single counter and observed leaked, monotonically-increasing IDs
across instances — a real test-isolation defect even though production runs
a single scheduler today.
Adds a covering test that creates two Schedulers, triggers each, and asserts
both first-run IDs end with counter "1" rather than a shared sequence.
Three related fixes to the e2e-demos driver test file:
1. Hoist `const cleanups: Array<() => void> = []` + `afterEach(...)`
from the BOTTOM of the `describe("e2e-demos driver", ...)` block to
the TOP. Vitest's hoisting kept this working at runtime, but the
source-ordering smell was fragile — a future test that pushes
BEFORE the array initializer line would hit a ReferenceError. Top
placement eliminates the footgun.
2. Move `cleanups.push(() => fs.rmSync(tmp, ...))` to IMMEDIATELY
after `fs.mkdtempSync(...)` — before any subsequent `writeFileSync`
that could throw. Previously the push happened AFTER writeFileSync,
so a disk-full / permission throw would orphan the tmp dir for the
rest of the test process's lifetime.
3. Align an inline writer mock that returned `{}` with the surrounding
`mkWriter` helper which returns `undefined`. ProbeResultWriter's
`write` is typed `Promise<unknown>` so both work, but the
inconsistency was a needless code smell.
- makeCtx now provides a fresh AbortController().signal by default
rather than leaving abortSignal undefined; pass abortSignal:undefined
explicitly to opt into the legacy behaviour. Catches regressions
that drop signal forwarding.
- Add abort-propagation test that fires the controller mid-flight and
asserts the next gql() call sees aborted state. The legacy 'threads
ctx.abortSignal' test asserts identity equality but never aborts —
a regression forwarding a different never-aborted signal would
still pass.
- Add concurrent-noise isolation test that stages REGISTRY_JSON_PATH
to a tmp file so 5 parallel enumerate() calls don't fall through to
/app/data/registry.json and emit log noise into CI.
- Add starter slug contract test asserting that
showcase-starter-ag2 → starter-ag2 misses the registry (which uses
bare integration slugs like ag2) and produces demos: [] — matches
the e2e-demos driver's shape-gated fan-out (it doesn't dispatch
for starters anyway).
- Document the same starter-slug contract on
deriveSlugFromServiceName().
CR-C-page.1: `catalog as unknown as CatalogData` violated the project
policy against double casts. catalog.json is generated by pretest/
predev/prebuild from generate-registry.ts and conforms to CatalogData
by construction, so a single `as CatalogData` cast is safe and clearer.
Typecheck: no new errors (pre-existing __tests__ + status-table.tsx
errors are unrelated and unchanged). All 220 vitest tests pass.
The previous parity test only checked nameExcludes across the
canonical trio (smoke, e2e-smoke, e2e-demos). Two gaps closed:
1. namePrefix parity is also load-bearing — a typo on one would
silently scope it to a different service set. Mechanise it in
the same test so a drift in either field fails CI together.
2. Other railway-services-backed YAMLs (image-drift, qa, ...) need
the same infra-exclude floor or must be explicitly exempted.
Glob all YAMLs, find the railway-services-source ones, and
assert their nameExcludes is a SUPERSET of the smoke set. Add
RAILWAY_PROBES_EXEMPT_FROM_EXCLUDES_FLOOR for the two probes
that legitimately don't carry the floor (image-drift checks
ALL services including infra; qa is scoped to one service via
precise namePrefix).
Also add a vacuous-parity guard (smoke set must be non-empty) so
three empty lists can't accidentally pass.
Pick a single canonical framing for the demo-count math: 17 framework
packages averaging ~32 demos each with the largest service carrying
38, total ~544 typical page loads per tick. The earlier "~32 demos
each" + "38-demo service" were not internally consistent.
Also update the timeout_ms block to describe the new orchestrator-
threaded driver-internal cap (E2E_DEMOS_TIMEOUT_MS) so the YAML's
documented behaviour ("driver iterates remaining demos and side-
emits abort rows when the cap fires at 1200000ms") is now truthful
under the singleton-driver registration model.
nameExcludes already use the unquoted style — no quote changes
needed (the parity test is what guards drift now).
The driver was registered as a singleton at orchestrator boot with
no per-probe deps, so its internal hard-cap defaulted to 5 min while
the YAML's timeout_ms documented 20 min — the driver would abort
mid-fan-out 15 min before the invoker's outer race fired,
contradicting the YAML's "driver iterates remaining demos and
side-emits abort rows when the cap fires at 1200000ms" claim.
The driver now reads ctx.env.E2E_DEMOS_TIMEOUT_MS per-run() in front
of the deps default, and the orchestrator projects each probe's
cfg.timeout_ms onto process.env.E2E_DEMOS_TIMEOUT_MS at probe-load
time. Singleton registration stays intact; the env-thread carries
the YAML value into the driver without rebuilding it.
Invalid env values (non-numeric / zero) fall through to deps default
so a typo can't silently disable the cap.
- Remove path.resolve() wrap on the absolute fallback registry path —
path.resolve() is a no-op for absolute paths.
- Drop the unused 'node:path' import.
- Drop the redundant 'env = {}' reassignment inside the variables
schema-success branch; the variable is already initialised to {}
at declaration. Schema-failure path now relies on the initialiser
with an explicit comment explaining the contract.
Jordan Ritter