R3-D.2: the toggle-close branch on the Trigger button forgot to reset
pickerOpen and selected, while outside-click and Escape both did. So
toggling the menu closed and reopening it showed ghost checks from a
prior session. Extract a closeMenu() helper used by all three close
paths so the invariant is structural, not duplicated.
Bonus: add aliveRef guard so async rejections from onTrigger that
resolve after unmount don't trigger setState-on-unmounted warnings.
R3-D.1: fetchProbes and fetchProbeDetail are polled every ~10s for live
status data. Without an explicit cache directive the browser and Next.js
App Router fetch wrapper may serve stale responses, defeating the poll.
Add cache: 'no-store' to both GET call sites and cover with tests.
E1: railway-services floor scan now matches both `.yml` and `.yaml`
extensions so a future probe authored as `<name>.yaml` cannot silently
bypass the floor invariant.
E2: every entry in `RAILWAY_PROBES_EXEMPT_FROM_EXCLUDES_FLOOR` must
either carry a `namePrefix` strictly narrower than the broad
`showcase-` floor, or its rationale string must include the grep-able
phrase `all services`. Today qa.yml passes via the narrow-prefix path
(`showcase-langgraph-python`); image-drift.yml passes via the
rationale-phrase path. If qa.yml is later widened to `showcase-`
without dropping the exemption (or restating an explicit
all-services rationale), this guard fires instead of letting infra
services flap red on a stale exemption.
E3: shorten the duplicated cwd-resolution rationale at the top of the
file; the load-bearing explanation now lives once on `configsDir()`.
The watch-callback test for probe-loader was flaking because chokidar's
native FSEvents backend on macOS does not reliably emit events for files
written into a freshly mkdtemp'd directory under Node 22+: the parent path
isn't pre-watched by the kernel, so the first add silently drops.
Add a test-only watcherOptionsOverride field on ProbeLoaderDeps so tests
can opt into chokidar's polling backend (the documented chokidar workaround
for this exact case), and switch the flaky test to use it. Also replace the
fixed setTimeout(500ms) wait with vi.waitFor polling so the assertion no
longer races chokidar's debounce on slow CI.
Production callers leave watcherOptionsOverride undefined, so prod still
gets FSEvents/inotify with no perf change.
Verified stable across 7 consecutive runs of the watch test.
The aimock-fixture-coverage probe scans byoc-json-render.spec.ts and was
flagging two prompts without matching feature-parity.json fixtures:
- "Break down revenue by category as a pie chart"
- "Show me monthly expenses as a bar chart"
Existing pie/bar fixtures used substrings ("revenue distribution by
category", "expenses by category") that don't appear in these prompts,
so the substring matcher returned no hits.
Added two new fixtures with substrings that DO appear in the prompts
("revenue by category as a pie chart", "monthly expenses as a bar
chart") so the coverage probe stays green and aimock returns
deterministic chart data when the BYOC json-render specs run against
the sidecar.
The degraded-state tooltip read "stale — last pass @ <observed_at>", but
observed_at is the most recent producer tick (when the row was last
*seen* in any state) — for a degraded row that is when degradation was
last observed, NOT when it last went green. Operators reading "last
pass @ ..." would mis-triage by treating that timestamp as the last
green tick.
Change copy to "stale — last seen @ <observed_at>" and add a covering
test asserting the new wording (and the absence of "last pass").
R11: The 'How to invoke' example referenced a nonexistent
defaultLaunchBrowser symbol. Reorganized to lead with the CLI wrapper
(scripts/d6-capture-references.ts) which already wires the default
deps; the programmatic example below is annotated as test/internal-
only with a comment pointing at the script for the canonical wiring.
R7 (e2e-parity half): Same per-turn slicing pattern as reference-
capture — runFeatureCapture's inner runConversation always reports
failure_turn === 1. Translated to outer 1-based index in the
'conversation-error' errorDesc so dashboards show which turn actually
failed. Added covering test asserting turn 3 (not turn 1) is reported
when the third turn fails.
R8: createDefaultFleetResolver cached successful reads forever, so
operators editing registry.json needed a process restart for D6
rotation to see the new fleet membership. Added mtime-based
invalidation plus a 60s wall-clock TTL safety rail. Resolver factory
now accepts injected now/stat for deterministic testing. Added two
covering tests: mtime-change invalidation and TTL elapsed even when
mtime is unchanged.
R10: When selectedThisTick === false (rotation skips a slug), the
driver returned aggregate green but emitted no per-feature side rows,
leaving stale red on the dashboard for the entire rotation cycle (up
to a week). Now emits a green per-feature side row with a 'not
selected this tick' note for every requested-known feature so cells
refresh while clearly signalling we didn't actually verify parity.
Updated the existing rotation-skip test to assert one side row per
requested feature and tightened slug expectations.
R9: pickToolCallName checked the bare 'name' field before the
structured 'tool_call.name' / 'tool_use.name' fields. Some transports
include a top-level 'name' that's a step/event label alongside a
structured tool descriptor — that ordering harvested the label and
poisoned the parity-compare engine's tool-call list.
Reordered to: toolCallName -> tool_call.name -> tool_use.name -> name.
Added 3 covering tests asserting structured fields win over a top-level
step-label and the canonical toolCallName still wins over everything.
R7 (reference-capture half): The capture helper runs each turn as
runConversation(page, [turn]) so the SSE interceptor only sees that
turn's request. The inner runConversation always reports
failure_turn === 1 (its slice-local index), which would mislabel
every per-turn capture failure as 'turn 1' regardless of which
outer turn actually failed. Translated to outer 1-based turn index
when constructing the failure reason. Updated the existing covering
test to assert turn 2 (not turn 1) when the second turn fails.
R6: Four sibling files duplicated the assistant-message selector
cascade with subtle drift; some used a bare '[role="article"]'
fallback that picked up user-input bubbles into the transcript and
caused turn-1 user-message substrings to spuriously match the user's
own input.
Hoisted ASSISTANT_MESSAGE_PRIMARY_SELECTOR (canonical testid) and
ASSISTANT_MESSAGE_FALLBACK_SELECTOR ([role="article"] excluding
data-message-role="user") to conversation-runner.ts. Updated
_hitl-shared, _gen-ui-shared, d5-agentic-chat, and d5-shared-state to
import the shared constants and apply them via a string-templated
new Function so closures don't have to cross the page.evaluate
boundary while DOM-lib remains excluded from the helper module.
R5: TIME_PICKER_SLOT_SELECTORS entries themselves embedded the
'[data-testid="time-picker-card"] ...' prefix, and the helper then
composed cardSelector + entry — producing nonsense like
'[data-testid="time-picker-card"] [data-testid="time-picker-card"] button >> nth=0'
that never matched. Simplified the cascade to its bare slot variants
('[data-testid="time-picker-slot"]', 'button >> nth=0') and rely on
the helper's prefixing for card anchoring.
Added _hitl-shared.test.ts with 3 covering tests: no doubled card
prefix anywhere in probed selectors, click composes card + slot, and
fallthrough cascade still anchors under the resolved card selector.
R4: The CLI script's package-root resolution was inlined and not unit
tested. Hoisted to src/probes/helpers/package-root.ts with dependency-
injected exists predicate so it picks up vitest's src/**/*.test.ts
glob. Added 5 covering tests (source-tree, dist-tree, startDir match,
missing throws, deepest stop).
R1: classesEqual was a strict length-equality check that rejected
captured-superset cases. Replaced with capturedClassesIncludeReference
(multiset superset) so a captured DOM with extra classes still passes
while a missing reference class fails the comparison.
R2-R3: When captured.total_chunks === 0 but reference.total_chunks > 0
the timing math degenerates and the stream axis silently passed. Added
an early-return guard with a clear failure reason so the parity verdict
reflects reality.
R2-B.1 — diffCronSchedules now awaits scheduler.unregister and emits
orchestrator.cron-unregister-failed on rejection; orphan rule stays
visible in scheduler.list() (mirrors CR-A2.2 probe-path fix). Watch
callback uses .then/.catch to surface reload failures via the bus.
R2-B.2 — boot wraps serve() in try/catch; on throw, awaits
scheduler.stop() and resets schedulerRunning before rethrowing, so a
late bind failure (e.g. EADDRINUSE) cannot leave cron jobs ticking
without an owner.
Tests: 2 new red-green specs added under orchestrator.test.ts
covering both rejection paths (vi.spyOn(logger.error) + vi.doMock for
the @hono/node-server serve() throw).
R2-B.3: distinguish unset from set-but-empty. An operator who mistypes
`OPS_TRIGGER_TOKEN=` ships an empty value AND sees the silent-skip
log instead of a clear error. Trim and reject whitespace-only too.
- unset → boot succeeds, info log (intentional disable)
- set-empty → boot throws (misconfig — refuse to mount probes router)
- set-value → boot succeeds, router mounted
Adds 4 covering tests under "OPS_TRIGGER_TOKEN empty-string handling".
R2-A.11: two robustness improvements to 1777165230_create_probe_runs.js:
- CREATE INDEX statements now use IF NOT EXISTS so a partial-apply
doesn't trip the next migration run on the index DDL step. The
collection-presence gate above already covers saveCollection
idempotency; this covers the per-index path in case PB ever runs
index DDL separately from the schema commit.
- Document why the up-migration's findCollectionByNameOrId catch must
remain broad: PB JSVM does not expose typed error discrimination
(no ErrCollectionNotFound), so we cannot narrow without a runtime
feature change. The down-migration's catch is already narrowed
because it operates on a resolved-or-skip path.
R2-A.10: ProbeRunTracker.start() spread the existing entry, so a
defensive call on a slug that previously transitioned through complete
or fail carried finishedAt / result / error into the new running entry.
The snapshot then surfaced a "running" service with terminal-state
metadata, which is incoherent.
Replace the spread with an explicit reset to the running shape so a
re-started slug starts fresh.
Tests: assert finishedAt/result/error are undefined after start() on a
previously completed or failed service.
R2-A.7: when pb.getOne returns null (row deleted, lookup race, etc.)
the previous code fell through with NaN duration and called pb.update
on a non-existent id, either throwing on the underlying client or
silently writing junk. Add an explicit null check that logs a warning
and returns early, so finish() is a no-op when the row is gone instead
of producing a stale-row error.
Test: asserts pb.update is not called when getOne returns null.
Five hardening fixes on the /api/probes routes:
R2-A.4: validate filter.slugs is a string[] before forwarding to the
scheduler. The invoker constructs new Set(filterSlugs) on the value, so
a string sneaks through as per-character set membership. Reject malformed
shapes with 400 invalid_filter at the route boundary.
R2-A.5: stamp the rate-limit window AFTER body read + JSON parse +
filter validation, not before. A 400 from validation must not consume
the operator's 5-minute hold. New sequence: auth -> 404 -> body read ->
parse -> filter shape -> rate-limit check -> stamp -> trigger.
R2-A.6: rollbackRateLimit now compare-and-swaps on its own t value
before restoring the prior stamp. Without this, two racing triggers
where A errors after B successfully stamped would have A's rollback
delete B's stamp.
R2-A.8: cap POST body at 16 KiB. Reject oversized payloads with 413
both via Content-Length pre-check and post-read length check (chunked
clients can lie about Content-Length).
R2-A.9: GET /api/probes/:id wraps writer.recent in try/catch so a
transient PB outage degrades to runs:[] + runsError: history_unavailable
with a 200, instead of 500-ing the whole probe-detail page.
Tests: added 13 new tests across five describe blocks covering each
fix, including a concurrent-trigger CAS rollback scenario and chunked
body-size enforcement.
Three correctness fixes in the probe invoker:
R2-A.1: when discovery enumerate fails, the RunSummary previously returned
total=0 with failed=1, violating total === passed + failed. Treat the
discovery itself as the single failed unit so the invariant holds.
Also stop polluting tracker.snapshot().services with a fake entry keyed
by the probe id (cfg.id is not a service slug); the failure is conveyed
structurally via discoveryFailed: true.
R2-A.2: Promise.race against the driver promise left the driver's tail
rejection unhandled when the timeoutPromise won. Under
--unhandled-rejections=throw this can crash. Attach a no-op .catch() to
the driver promise before the race so late rejections are absorbed; the
race outcome is unchanged.
R2-A.3: same fix for the discovery enumerate Promise.race path.
Tests: updated CR-A1.5 invariant assertion plus added two unhandled-
rejection guard tests for driver and enumerate late-reject paths.
Bucket-(b) bonus polish:
- humanizeCron now renders "M H * * *" with a non-zero minute as
"Daily at HH:MM" instead of falling through to the raw expression.
- humanizeCron rejects */0 step values for both the every-Nh and every-Nm
shapes — that's not a valid cron step.
- StatusRunningPanel: rename `completed` to `finished`. The variable
counts services in either completed OR failed states, so "completed"
read as misleadingly success-only. The data-completed attribute name
stays for backwards compatibility with existing tests.
A run with failed=0 but passed<total represents some services that were
skipped/unknown — rendering this as green "completed" misleads operators
into thinking all targets passed (R2-D.4). Now badged amber with a
"partial (N skipped)" label so the partial-coverage case is visible.
Three related fixes to StatusTriggerButton:
- R2-D.2: render onTrigger errors inline via data-testid="status-trigger-error"
instead of discarding the value through a write-only setter.
- R2-D.3: when serviceSlugs prop changes, intersect selected with the new
list so stale slugs cannot be POSTed; also clear selection on
outside-click and Escape close so reopening doesn't show ghost checks.
- R2-D.5: drop unreachable serviceSlugs.length === 0 ternary inside the
picker — the outer guard already requires length > 0.
Replace the no-op handleTrigger stub and empty-list placeholder with real
hooks. useProbes feeds the schedule grid via 10s polling; useTriggerProbe
POSTs /trigger with the operator token. If NEXT_PUBLIC_OPS_TRIGGER_TOKEN
is unset, the trigger throws a clear "token required" error so operators
see fail-loud feedback instead of a silent no-op (R2-D.1).
POST to /trigger is non-idempotent — the server may have already received
the request and queued the run by the time the component unmounts.
Aborting the controller client-side hides the result, but does NOT
unfire the server-side action. Stop aborting on unmount; rely on
aliveRef to gate any setState after teardown.
Back-to-back trigger() calls intentionally abort the prior controller.
The first call's promise was rejecting with AbortError, surfacing as a
user-facing error on the Status tab. Detect AbortError (and the
cancelled-controller case) in the catch block and resolve silently
instead of setError + rethrow. The owning-controller guard on the
finally block prevents a superseded call from flipping pending off
while a newer call is still in flight.
Bonus consistency for useProbes: setLoading(true) on dep change so
consumers can render a "refreshing" indicator across baseUrl swaps.
Tests:
- unmount during in-flight trigger does NOT abort the signal
- post-unmount resolution does not call setState (no act warnings)
- back-to-back triggers: first promise resolves silently, error stays null
ensureOk and parseJson previously caught body-read failures and rewrapped
them in a generic Error, losing the AbortError name. Hooks filter abort
noise via err.name === "AbortError", so the wrap defeated cancellation
filtering and surfaced "AbortError" as a user-facing failure on the
Status tab.
Re-throw AbortError as-is from both helpers; non-Abort body errors keep
the descriptive wrap path. Also tighten ProbeKind to use (string & {})
so known-kind autocomplete is preserved alongside forward-compat strings.
Add covering tests:
- response.text() rejects with AbortError → ensureOk re-throws as-is
- response.json() rejects with AbortError → parseJson re-throws as-is
- non-Abort body errors still wrap with descriptive message
- triggerProbe with already-aborted signal surfaces AbortError unwrapped
Backstop the matrix fixes and address weak/no-op assertions:
- Tighten depth-chip count assertions from '>= N' to exact 'N'
where the fixture pins the count.
- Replace the no-op filter=wired test with a fixture containing a
feature whose every cell is unshipped, then assert that row IS
visible under filter=all and IS HIDDEN under filter=wired.
- Extend the collapse test with a re-expand click that asserts
chip count returns to baseline (regression guard for one-way
toggle state).
- Cover filter=regressions empty-state.
- Cover starter cells (feature === null) in both matrices: the
cell index must skip them without polluting the grid or leaking
a 'null' string into rendered output.
- Pin parity-matrix integration-header ordering across all five
parity tiers (reference, at_parity, partial, minimal, not_wired).
- Cover D2 cap for starter cells in deriveDepth (chat/tools cannot
lift past D2 because D3 is not evaluable without a feature id).
- Replace the locally-declared CatalogCell shape in depth-utils.test
with the source type so tests cannot drift from production.
- Add afterEach(vi.unstubAllGlobals) to both test files so the
localStorage stub does not leak across files in the same worker.
Three related cleanups in cell-matrix.tsx and parity-matrix.tsx:
1. Drop the 'as 0 | 1 | 2 | 3 | 4' typecasts on depth.achieved. The
newly tightened DepthResult.achieved literal-union type makes
them redundant.
2. Skip starter cells (feature === null) when building the
cell-by-key index. The previous code interpolated null into the
key string, producing '<slug>/null' entries that orphaned
starter rows in the index.
3. Render an explicit empty-state in CellMatrix when filter is
'regressions'. The placeholder logic returned false for every
row, silently hiding the entire matrix. Until regression
detection lands (DepthResult.isRegression always false), the
matrix now surfaces a 'Regression detection not yet implemented'
message via data-empty-reason='regressions-not-implemented'.
deriveDepth's return type carried the loose 'number' on achieved, forcing
every consumer to cast to '0 | 1 | 2 | 3 | 4' to satisfy DepthChip's
prop type. Expose AchievedDepth alias and narrow the source so the
casts at use sites (cell-matrix.tsx, parity-matrix.tsx) become
unnecessary.
Per-cell status row corrections + a focused unit-test file covering CP1,
CP2, CP5, CP7, CP8 behaviors:
- CP3: stop consuming `cell.smoke` in CellStatus. resolveCell still
computes the field for backwards-compat with other call-sites, but
per Phase 3 Decision #7 the smoke L1 signal is integration-scoped and
lives in the per-integration strip — there's no per-feature smoke
badge in the cell row. (The CellState type narrowing for the `smoke`
field is owned by the live-status.ts fix in a sibling worktree; this
patch only removes the consumer.)
- CP8: hide the D5/D6 LiveBadges when feature.kind === "testing".
The e2e-deep / e2e-parity producers only emit rows for primary
features per spec; testing features would otherwise render a
perpetual gray "?" that adds noise without information. Mirrors the
docs-row visibility rule (already gated on isTesting).
- CP9: document the missing per-cell drilldown URL convention for D5/D6
with a TODO comment. No drilldown route exists today; once one does,
wire it through keyFor here.
Tests cover red-green for CP1 (mouseleave handler wired), CP2 (first /
error / green_to_red transition formatting), CP5 (opt-out vs absent
tooltip copy), and CP8 (D5/D6 hidden for testing features). The
existing docs-link.test.tsx still passes.
- CP1: wrap Badge in a span that resets `tooltipOpen` on mouseleave/blur
so the lazy-fetch flag can flip back to false. Without this, the flag
stayed true forever after the first hover, defeating the §5.6
optimization and (once the hook upgrades to a subscription) leaking it.
- CP2: discriminate `first` and `error` transitions in the tooltip copy
so operators can tell "first observation (initial: <state>)" apart
from "producer error (error → <state>)" instead of seeing the same
fallback string. Logic moved into a named `formatTransitionLine`
helper for testability.
Four fixes to the per-cell docs row:
- CP4: extract the `unselected` shell-routing fallback into
SHELL_UNSELECTED_PATH so the coupling is documented and easy to find.
- CP5: split the `missing` tooltip into "framework opt-out" vs "no docs
URL declared" so operators can distinguish a framework that declined
docs from a registry entry that never declared one.
- CP6: add `default: never` exhaustiveness guard to DocsLink's switch so
any new DocState variant is a compile error until handled.
- CP7: render `<a>` (not `<span>`) for `state === "error"` when an href
is present so operators can manually verify a flaky probe — `notfound`
stays a span because the URL is positively known broken.
LS6: `mergeRowsToMap` now logs a `console.warn` when the same key is
seen twice across the merged groups. Each dimension is supposed to own
a disjoint slice of the keyspace, so a collision is an upstream bug.
Return semantics are unchanged (last-wins remains correct under
eventual consistency), but the warn surfaces the regression in dev mode.
LS7: `upsertByKey` returns the SAME array reference when the incoming
row is observably identical (key + state + observed_at +
transitioned_at). Producers re-emit unchanged rows on every poll tick;
allocating a fresh array for each one defeats React's reference-equality
short-circuit and re-renders every memoised consumer downstream.
Tests cover no-op identity, mutation paths (state change,
observed_at change), the disjoint-merge happy path, and the collision
warn.
LS1: `allGreen` now requires BOTH `healthRow` and `e2eRow` to be present
and green. Previously a missing e2e row was treated as green-eligible,
which let a brand-new cell read green before any e2e probe had ticked —
the same flavour of stale-green lie as the connection=error case.
LS11: smoke lookup now uses the integration-scoped key (`smoke:<slug>`)
to match the producer. Pre-fix, resolveCell looked up
`smoke:<slug>/<featureId>` which always missed, leaving every smoke badge
gray. Tests update fixtures accordingly and add a regression guard so a
stray feature-keyed smoke row cannot bleed into the resolver.
- Drop the hardcoded '>6h' duration from the degraded tooltip; the
threshold lives in the producer config and lying about it in copy
misleads operators when the cadence changes.
- Replace the empty-string default branch with a loud unknown-state
fallback gated by a `never` exhaustiveness check, mirroring rowTone /
formatLabel.
- For red and degraded states, append a truncated (80-char) summary of
`row.signal` so the tooltip carries triage context without ballooning.
- When the SSE connection is in error AND the cached row is red or
degraded, append the row's last-known state so operators don't lose
triage context to the offline banner.
- keyFor now throws when slug or featureId contain ':' or '/', preventing
silent collisions in the live-status lookup map (e.g. `smoke:a/b` vs
`smoke:a` + `/b`).
- rowTone() and formatLabel() switch over the State union with explicit
`never` exhaustiveness checks so adding a new variant fails the build
instead of silently mapping to a misleading green/gray default.
- Unknown row states now surface a distinct 'error' tone (rowTone) and a
'?' glyph (formatLabel) rather than a silent green check.
- Tests cover the three throw paths and remain green for the existing
per-feature key shapes.
Adds two feature-parity.json fixtures for the langgraph-python
byoc-json-render E2E spec prompts that the
aimock-fixture-coverage probe flagged as uncovered:
- "Break down revenue by category as a pie chart"
- "Show me monthly expenses as a bar chart"
Responses mirror the worked examples baked into the
byoc_json_render_agent system prompt verbatim, so the fixture
output matches what the live LLM would emit (a single
@json-render/react flat-spec JSON object) and the spec
assertions on [data-testid="pie-chart"] /
[data-testid="bar-chart"] resolve deterministically when the
suite runs against the aimock sidecar.
Replace the real-chokidar watch test with a vi.mock that swaps chokidar
for an in-memory EventEmitter. The original test relied on macOS
fsevents firing within 500ms of fs.writeFile, which fails ~5/5 runs
locally. Same pattern as rule-loader.test.ts, which abandoned its real-
watcher path for the identical reason. Adds a companion test asserting
the 100ms debounce collapses bursty add/change events into one reload.
Probe-loader test suite is now deterministic across 5 consecutive runs.
The previous `readChildCount(page, selector)` body did an in-page
re-resolution cascade across multiple selectors, which obscured which
selector actually drove the count and made the function impossible to
test for selector-fidelity. Swap to a single-selector helper
`readChildCountForSelector` whose evaluate fn embeds the selector via
`JSON.stringify(selector)` (no string interpolation, no XSS surface),
then explicitly cascade at the call site. Test asserts the selector
is interpolated literally in the evaluate fn source so the wiring
stays visible.
Findings addressed: A11.
Three HITL hardenings:
1. Runtime page-shape guard (A9). The `page as HitlPage` cast was a
silent type widening; if a fake page passed by a test lacked
`click()`, the script would silently no-op rather than fail loudly.
Switch to `page as unknown as HitlPage` and add a typeof check
that throws "page is missing click() — cannot drive HITL dialog"
if the method is absent. Applied symmetrically to both HITL
scripts (approve-deny + text-input).
2. Anchored selectors (A10). Both `approveOrDeny` (button selectors)
and `pickTimeSlot` (slot selectors) previously queried the page
globally, which could match buttons on the surrounding chat or in
a tooltip layered above the dialog. Scope every candidate selector
under its dialog/card root: `${dialogSelector} ${sel}`. Test
coverage asserts the scoped form is what's actually used.
3. Post-stable settle (A17). `waitForNextAssistantMessage` previously
returned as soon as the assistant message count grew past
baseline, but follow-up tokens could still be streaming when the
next assertion ran — racing the SSE tail. Add a phase-2 wait for
`ASSISTANT_FOLLOWUP_SETTLE_MS = 1500` of count-stability after
growth detection so the response is fully committed before we
move on.
Findings addressed: A9 (page-shape guard), A10 (anchored selectors),
A17 (post-stable settle).
`readMessageCount` previously matched bare `[role="article"]` as a
fallback, which counted user-input bubbles as assistant messages on
composers that tag both. The poll could "settle" before the assistant
even started streaming because the user's just-sent message bumped
the count past the baseline.
Fix: prefer the canonical `[data-message-role="assistant"]` testid;
fall back to `[role="article"]:not([data-message-role="user"])` so we
never count user articles. Two new tests cover the tagged-assistant
path and the fallback narrowing path with a fake document patched in
via `evaluate`.
Findings addressed: A7.
The contract axis previously dumped both "field absent" and "field
present but wrong type" into one `missing_fields` array, conflating
two operationally distinct signals: a vanished field is a delete,
a typed-differently field is a shape regression — the dashboard
triagers handle them differently. Add `type_mismatched_fields` to the
report `details.contract` shape and route each failure mode to the
correct bucket. The verdict still fails on either bucket being
non-empty; only the breakdown is split. Test coverage updated with a
direct case asserting both arrays land their entries correctly.
Findings addressed: A14.
Three independent fixes in the D6 parity driver, all touching the same
control flow:
1. Stop caching fleet-resolver failures. The pre-fix code assigned
`cache = []` on any error, which permanently disabled D6: a
transient ENOENT during process startup would lock the resolver to
`[]` forever. Now we leave `cache` null on error so the next call
retries (matching what operators get from a process restart) and
log a warning with the registry path + error.
2. Stop appending the invocation slug to the fleet when missing.
Per-invocation appends broke rotation determinism: different
invocations with different slugs all produced different fleet sizes,
and `weekNumber % len` then picked different integrations within
the same tick. Registry drift now correctly results in "this slug
sits out the rotation" with a `self-not-in-fleet` warning the
operators can act on.
3. Emit per-feature side rows on launcher error. Previously, when
Chromium failed to launch we only emitted the aggregate red — every
feature's individual cell would render whatever state the previous
tick left behind, including "green" from an earlier successful run.
The launcher-error branch now writes a `d6:<slug>/<ft>` side row
for every runnable feature with `errorClass: "launcher-error"` and
`errorDesc` (truncated to 1200 utf8 bytes), so the dashboard's
per-feature drilldown stays truthful.
Test coverage updated to assert all three behaviours (fleet-error
retries on next call, self-not-in-fleet warning + scoping note,
launcher-error side rows for every runnable featureType).
Findings addressed: A4 (fleet caching), A12 (self-not-in-fleet),
A13 (launcher-error side rows). Wires in the script loader from the
prior commit via the new `scriptLoader` driver dep.
The script-loader was previously module-private, so the D6 parity
driver and the `d6-capture-references` CLI both ran with an empty
`D5_REGISTRY` whenever they were invoked standalone (without `e2e-deep`
running first in the same process). Symptoms: every featureType
classified as `skippedScript`, D6 silently no-op, D6 capture run
reports "no features captured" that looks like success.
Changes:
* Lift the filename matcher to an exported `D5_SCRIPT_FILE_MATCHER`
constant. The new regex explicitly rejects co-located `*.test.{js,ts}`
files (which would re-import the script under test and trigger
double-registration throws) and `*.d.ts` declaration files. Covered
by 4 new tests asserting accept/reject behaviour.
* Export `defaultScriptLoader` so callers can reuse one canonical
implementation. The D6 CLI now invokes it at startup and fails
fast if the registry is still empty afterwards (with a clear
"check that scripts/d5-*.ts call registerD5Script" hint).
* Replace the `feature as D5FeatureType` cast in the CLI with the
`isD5FeatureType` type guard from this commit's parent.
Findings addressed: A1 (matcher hardening), A2 (loader export +
fail-fast in CLI).
Jordan Ritter