30-minute schedule (half the smoke cadence — L3+L4 per tick pulls a
real chromium page so we trade freshness for memory + bandwidth
headroom). 180s outer timeout covers two 60s page timeouts plus
chromium launch overhead. max_concurrency 2 caps peak memory at ~600MB
(each chromium ~300MB RSS) to fit the orchestrator's Railway footprint.
Discovery uses the railway-services source with namePrefix + the new
nameExcludes filter to hit every user-facing showcase without the
7 infra services.
Runtime stage needs two things for e2e-smoke: chromium (headless) and
the shell registry so the driver can gate L4 on demos membership.
Build stage pulls the single registry.json file through rather than
fetching at probe-tick time, keeping the image hermetic. Runtime
stage is already on Debian-slim (needed by Playwright's glibc-linked
chromium); `playwright install --with-deps chromium` runs as root
before the USER node drop so apt-get succeeds. /ms-playwright is
chown'd to node so the runtime user can read the browser tree.
Wire the in-process Playwright driver so probe ticks resolving to
kind: e2e_smoke dispatch to it. Drops the 'deliberately NOT
registered yet' placeholder that flagged the driver as scaffold-only.
The e2e-smoke probe needs to discover every user-facing showcase
service without enumerating infra (aimock, ops, pocketbase, shell,
shell-{dashboard,docs,dojo}). Extend the discovery filter schema
with an optional nameExcludes array so a single namePrefix can pull
in all 34 services and explicitly drop the 7 that don't expose user
demos, rather than maintaining 27 separate namePrefix entries.
Replace the JSON-reporter scaffold with an in-process Playwright
driver. Each invocation launches one headless chromium, runs L3
against /demos/agentic-chat (any non-empty response) and, when the
service registry entry includes tool-rendering, runs L4 against
/demos/tool-rendering (weather vocabulary check). Primary return is
e2e-smoke:<slug>; chat:<slug> and tools:<slug> side-emit via
ctx.writer. AbortSignal is wired end-to-end so timeout teardown
kills the browser cleanly. Launcher + demos resolver are dep-injected
for unit testability.
Input schema accepts { key, backendUrl?, publicUrl?, name?, demos? }
with a zod refine requiring at least one URL. Slug derivation strips
the "showcase-" prefix so Railway service names align with registry
entries (bare slugs). The default demos resolver memoises a read of
/app/data/registry.json at first use, keeping probe-tick overhead at
a single JSON parse across the orchestrator lifetime.
Extends the smoke probe driver to emit a third side-emission per target:
`agent:<slug>` — POST `${backendUrl}/api/copilotkit/` with `{}` body,
green on non-404 response (runtime is mounted), red on 404 or transport
failure. Matches the `checkAgentEndpoint` contract in the e2e helpers
(integration-smoke.spec.ts L2 `@agent` assertion).
Switches `config/probes/smoke.yml` from a static 17-target list to the
`railway-services` discovery source with `namePrefix: "showcase-"` and
an explicit `nameExcludes` for infra services (aimock/ops/pocketbase/
shell*). New showcase services are picked up automatically on the next
tick without YAML edits.
- src/probes/discovery/railway-services.ts: add `nameExcludes: string[]`
filter field, applied AFTER the prefix check so excluded services skip
per-service env fetch entirely.
- src/probes/drivers/smoke.ts: accept both static (`{key,url}`) and
discovery (`{key,name,publicUrl,imageRef,env}`) input shapes. Derive
slug by stripping `showcase-` prefix from `name` (discovery) or
splitting key on `:` (static fallback). Rewrite primary result key to
`smoke:<slug>` in discovery mode so existing alerts keyed on
`smoke:ag2` / `smoke:starter-ag2` stay intact.
- Sequential probes kept for socket-count bound: at max_concurrency=6 *
34 services * 3 endpoints, parallel would push 612 inflight sockets.
- Coverage: smoke driver 98.55% lines, exceeds 95% gate.
Adds three new alert-rule YAMLs modeled on smoke-red-tick.yml so the
per-starter depth signals side-emitted by the smoke + e2e-smoke drivers
(agent:<slug>, chat:<slug>, tools:<slug>) produce Slack alerts on
green_to_red / sustained_red / red_to_green transitions with the same
!channel 1-hour escalation + 15-minute rate limit + 20-minute deploy
grace as the L1 smoke rule.
Also extends the DIMENSIONS enum with the three new closed values so
rule-side typos continue to fail at load with a listed-valid-members
Zod error instead of silently never matching any probe key.
No driver or discovery changes in this commit — pure rule YAML + the
single enum entry each rule's `signal.dimension` resolves against.
Multi-stage Dockerfile built on bookworm-slim with Playwright chromium
pre-installed (e2e-smoke runner) + pnpm-workspace.yaml + every
workspace package.json COPYd so the runtime stage resolves workspace:*
deps. README organised operate-first / build-second so an on-call
operator can configure + inspect the service before needing to rebuild
it. Rotation drill doc captures the weekly / monthly schedule for
pin-drift + redirect-decommission audits.
Jordan Ritter