The ms-agent-harness-dotnet slug was excluded from per-cell D6/BE/smoke
probe enumeration by a placeholder fence added 2026-06-07, before the
real column existed. The column shipped in PR #5569 and its d6/d4 aimock
fixtures landed on main today (e10df0b4), so the fence is now stale.
Remove the slug from all 8 exclude SSOT sites so the column populates.
Other dimension drivers under showcase/harness/src/probes/drivers/ follow
the dN-<purpose>.ts naming pattern (d4-chat-roundtrip.ts, d6-all-pills.ts).
D2 and D3 didn't — bring them in line.
drivers/liveness.ts -> drivers/d2-liveness.ts
drivers/liveness.test.ts -> drivers/d2-liveness.test.ts
drivers/e2e-readiness.ts -> drivers/d3-readiness.ts
drivers/e2e-readiness.test.ts -> drivers/d3-readiness.test.ts
The d3 driver loses the redundant 'e2e' prefix — the dimension is D3
and the corresponding dashboard badge is 'UI', so the file's job is best
described as 'is the demo frontend ready?'.
Imports updated in:
- showcase/harness/src/orchestrator.ts
- showcase/harness/src/orchestrator.test.ts
- showcase/harness/src/cli/runner.ts
- showcase/harness/src/probes/drivers/d2-liveness.test.ts (self-ref)
- showcase/harness/src/probes/drivers/d3-readiness.test.ts (self-ref)
Doc-comment references to the old file paths updated in:
- showcase/harness/src/cli/targets.ts
- showcase/harness/src/probes/liveness.ts (the helper, not the driver)
- showcase/harness/src/probes/loader/probe-invoker.ts
- showcase/harness/src/probes/discovery/railway-services.ts
- showcase/harness/src/probes/drivers/starter-smoke.ts
- showcase/harness/src/probes/drivers/starter-smoke.test.ts
- showcase/harness/src/types/dimensions.test.ts
- showcase/harness/config/probes/e2e-demos.yml
- showcase/harness/README.md (driver table)
Function/symbol exports unchanged (livenessDriver, e2eReadinessDriver,
createE2eDemosDriver, etc.) — file rename only. The parent helper module
showcase/harness/src/probes/liveness.ts is also untouched: it's a
separate, non-driver helper that exports deriveHealthUrl, livenessProbe,
and LIVENESS_SLACK_SAFE_FIELDS. Not renamed: drivers/e2e-parity.ts,
drivers/aimock-wiring.ts, drivers/pin-drift.ts, drivers/image-drift.ts
(non-dimensional probes).
Repoints the D5 probe at the unified D6 driver. The fleet D5
enumerator now stamps driverKind=e2e_d6 with representativeOnly + a
"d5" rowPrefix; the CLI's buildDeepInputs carries the same inputs;
config/probes/e2e-deep.yml declares kind=e2e_d6. Drops the e2e_deep
driver registration (orchestrator + worker registry + BROWSER_KINDS +
the worker-internal kind set), keeping the D5 producer schedule/cadence
intact. The worker now honors driverInputs.rowPrefix when filtering the
aggregate side-row out of captured cells so a "d5:<slug>" aggregate
doesn't leak into the D6 entry's "d6:<slug>" cell capture. This
eliminates the separate D5 launcher path whose own launcher instance +
cadence systematically dropped x-aimock-context against the shared fleet
pool (aimock strict 503 -> red).
Rewrite the starter-smoke probe to the v2 path-based multi-route runtime
protocol the deployed starters actually speak (proven via a local
build+curl gate against examples/integrations/crewai-crews at 1.59.5):
- agent rung: GET /api/copilotkit/info -> 200 + version (was: any non-404)
- chat rung: POST /api/copilotkit/agent/<id>/run (Accept: text/event-stream)
asserting >=1 TEXT_MESSAGE_CONTENT delta + terminal RUN_FINISHED + no
RUN_ERROR (was: non-empty body)
- health rung: repointed to GET /api/copilotkit/info (the starters serve
no /api/health route)
- interaction rung: GET / (unchanged)
- drop trailing slash on all runtime POSTs (the old /api/copilotkit/ 308s
and drops the POST -> 404, the core cause of the red rung)
Plus a unified abort/body-read error classification (one abortOutcome()
helper keyed on the local externallyAborted flag): a self-timeout or a
non-abort error is transport-error (soft), only a genuine external abort
is 'aborted'; a cut-short body read softens to transport-error instead of
hard smoke-failed; the level loop short-circuits on external abort.
NOTE: the content-level chat rung will correctly read RED against the
deployed starters until the Python-side ag-ui-crewai RUN_ERROR (a
real-LLM, in-process agent failure, NOT a probe bug) is fixed.
Deferred follow-ups: verify the alert/staleness consumer branches on
errorClass (not state) so the soft/hard split is actually consumed;
res.text() resolve-partial precision edge; redirect:follow POST edge;
health/interaction body-drain; health/agent shared /info; CLASS_RANK
aborted-lowest; unmapped-starter columnSlug/prototype-key; test-coverage
gaps (resolveTimeoutMs, deriveStarterSlug, real-timer abort).
Documentation the diff invalidated, swept to match reality:
- browser-pool.ts JSDoc/comments + browser-pool.test.ts comments claimed
hard-recovery "purges the stale /tmp/playwright_* profile dirs". The code is
purge-free — hardRecover is a PACED cold relaunch that gives the
thread-exhausted kernel time to relax. Rewrite to match: paced cold relaunch,
NO /tmp purge (the wedge is the cgroup pids ceiling, mitigated demand-side;
/tmp is a proven non-cause).
- probe-invoker.ts stagger comment said "8 simultaneous workers", "~400 thread
spawns", "10-minute D6 budget" → updated to the current 5 / 20-min reality.
- e2e-deep.yml: the global cap note (40, "D6 peak 32 + D5 peak 8") → 24, with
D6 peak now 5×4=20; 20+8=28 > 24 so a d6+d5 overlap serializes under the
global cap (intended demand-side back-pressure).
- d6-all-pills-e2e.yml: the worked example (max_concurrency 8, ceil(18/8)=3
rounds) → 5 / ceil(18/5)=4 rounds, cross-referencing the overlap block that
justifies the concurrency itself.
The live wedge peaked 998/1000 against the cgroup pids.max=1000 ceiling
specifically during a d6+d5 overlap — the simultaneous chromium renderers of
both probes summed past the ceiling. Fewer concurrent d6 feature-workers cut
the simultaneous-renderer thread peak, leaving headroom for a co-firing d5 (or
a recovery relaunch) without crossing 1000. Complementary to the
MAX_CONTEXTS 40->24 lever already landed. Comment updated to the pids-ceiling
reasoning; 5 x 4 = 20 concurrent contexts stays under the 24-context cap.
The d6-all-pills-e2e probe header was sized for a legacy "8 services x 4
features" single-round fleet (timeout_ms: 600000 / 10 min). Discovery now
enumerates ~18 showcase demo services, so at max_concurrency: 8 the fan-out
runs ceil(18/8) = 3 serialized rounds. Rounds 2-3 start late and execute under
CPU contention, stretching each service toward ~200s and blowing the 10-min
per-service budget — all 18 services went red with `driver timeout after
600000ms`.
Raise the outer cap to 1200000 (20 min), matching the sibling 18-service
e2e-demos probe (same fleet, same budget). 3 rounds x ~200s fits comfortably.
Timeout-only change: max_concurrency stays at 8, so D6 peak stays at 32
contexts (under BROWSER_POOL_MAX_CONTEXTS=40, leaving 8 for the offset D5
tick). Header comment rewritten to document the real fleet size and the
fan-out math.
The starter_smoke probe runs hourly (`schedule: "40 * * * *"`), but
STARTER_STALE_AFTER_MS was set to 13h with a comment claiming a 6h cadence
— ~13 missed hourly ticks before amber, defeating the intended two-miss
flip. Re-derive the window from the 1h probe period to 2.5h: strictly
> 2 periods (so two consecutive misses, last row ~3h old, flip amber) yet
< 3h (so a single missed/slow-wake tick, last row ~2h old, stays green,
absorbing a scale-to-zero cold-start). Reconcile both the staleness.ts and
starter_smoke.yml comments to the same hourly basis + 2.5h window in
lockstep (no more "6h" in the starter context). Extend the staleness tests
with explicit hourly-tick boundaries (1 miss → green, 2 misses → amber).
Also drop the inaccurate "hollow" from the not-supported ✗ comments — there
is no hollow render variant; the state renders as grey ✗ text.
Adds the `starter_smoke` probe driver + config/probes/starter_smoke.yml that
fans out per-starter HTTP health/agent/chat/interaction levels, registers it
in the orchestrator, and exempts it from probe-config parity (its matrix
shape differs from the depth-dimension probes). Includes 13 driver unit tests.
Probe slot (S2) of the starter-row-group spec (model B): harness HTTP-probes
the deployed (sleepable) Railway starter services.
Commit 7ac3e59a5 ("D6 all-pills probe driver") replaced the
`e2eParityDriver` registration (kind `e2e_parity`) with `e2eFullDriver`
(kind `e2e_d6`) in BOTH branches of `registerAllProbeDrivers`. After that
commit NO driver registers kind `e2e_parity`, yet
`config/probes/e2e-parity.yml` still shipped.
The probe-loader hard-rejects any YAML whose `kind` has no registered
driver (`no driver registered for kind 'e2e_parity'`), so `e2e-parity.yml`
failed to load on every boot and was never scheduled — silently dropping
that probe family. The D6/parity dashboard dimension is now produced by the
`e2e_d6` (all-pills) driver, which emits the identical
`d6:<slug>/<featureType>` rows the dashboard reads by key prefix (plus a
`d6:<slug>` aggregate), so `e2e_parity` is genuinely superseded, not just
temporarily unregistered.
Delete the orphaned config so the loader no longer rejects it and the full
probe-config set loads clean. Add a loader test that exercises the REAL
shipped `config/probes` set against the REAL driver registry and asserts
every YAML loads with zero `probes.reload.failed` errors — the existing
orchestrator guard only checks a hardcoded kind list and never read the
on-disk YAMLs, so it missed this orphan.
the integration was added to the railway-services SSOT and smoke.yml nameExcludes; this
adds it to the sibling probe configs the probe-config-parity invariant requires to stay
in lockstep.
- qa probe: manifest.yaml path corrected from showcase/packages/ to
showcase/integrations/ (Slice 3 moved files; probe code and test
fixture still referenced the old path).
- pin-drift probe: fail-baseline.json ENOENT in container — the 5-level
import.meta.url walk-up from dist/probes/drivers/ overshoots /app and
lands at /. Added PIN_DRIFT_REPO_ROOT=/app env var and COPY of the
baseline file into the runtime stage.
- aimock-wiring probe: showcase-ms-agent-harness-dotnet (deployed: false)
was not in the EXCLUDE_SERVICES set, causing it to be flagged as
unwired. Added to the exclude list in both aimock-wiring.ts and
smoke.yml.
Add d6-all-pills driver that exercises every pill on every integration
behind aimock context routing. Rename e2e-chat-tools/e2e-deep to
d4-chat-roundtrip/d5-single-pill to match depth convention. Wire
D5 representative map, D6 scoping helpers, dimension types, and
the orchestrator config for the new probe IDs.
PR #4718 added the LGP D5 coverage wave, taking langgraph-python's
declared D5 feature set to ~28. With FEATURE_CONCURRENCY=2 that's 14
features per worker × ~15s realistic wall-clock = ~210s — over the
prior 3-min cap before any slack. Post-merge cycles confirmed the math:
17 langgraph-python features ran cleanly and flipped green, but the
remaining ~10 hit `errorClass: "abort"` with `errorDesc: "aborted"`,
the abort-cascade signature where the global cap fires before the
per-feature loop reaches them. Per-feature timeout (5min) bounds wedged
features but doesn't bound the slug's total feature count against the
outer cap.
Bump to 10 min — gives ~5x headroom at current feature count. Lighter
integrations (5–10 features, ~60s) still exit early without sitting on
the cap, so cycle wall-clock barely changes (cron stays every 15 min,
concurrency=4 hides the slow integration's tail behind the others).
Revisit `FEATURE_CONCURRENCY` before raising further — Chromium context
count scales with it and the Railway pod's 2.4GB headroom budget assumes
the current value.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
With aimock backing all 18 integrations, there is zero LLM cost per
probe tick. Tightened schedules to catch regressions faster:
smoke: */15 → */5 min (L1/L2 HTTP health)
e2e-smoke: */30 → */15 min (L3/L4 Playwright chat round-trip)
e2e-deep: hourly → */15 min (D5 multi-turn conversations)
e2e-demos: 6-hourly → hourly (per-demo structural coverage)
Offsets preserved to avoid Chromium contention across probes.
The monitoring/alerting service is a test harness (probes, assertions,
alerting), not an operations service. Rename the directory, package
name (@copilotkit/showcase-ops → @copilotkit/showcase-harness), all
internal references (Dockerfile, Prometheus metric prefix
showcase_ops_ → showcase_harness_, orchestrator log messages, probe
YAML nameExcludes, test fixtures), and regenerate pnpm-lock.yaml.
Wire protocol names (X-Ops-* headers) and shell-dashboard internal
API naming (OPS_BASE_URL, ops-api.ts) are intentionally unchanged —
they are stable contracts between sender and receiver.
Jordan Ritter