Change 3 langgraph health routes from /ok to /health endpoint and
fix Record<string, any> to Record<string, unknown>. Rewrite mastra
health route from unconditional 200 to actual runtime reachability
check via self-call to /api/copilotkit.
Replace full SSE body drain (await res.text()) with a TTFB-based
pattern that reads only the first chunk via ReadableStream, then
cancels. This makes smoke probes apples-to-apples: every package
measures time-to-first-byte, not time-to-complete-LLM-response.
Also rewrites 3 langgraph smoke routes from upstream-only /ok ping
to full CopilotKit stack exercise, fixes mastra agentId
(weatherAgent to agentic_chat), normalizes all timeouts to 45s and
maxDuration to 60, and removes the crewai-crews UPSTREAM_TIMEOUT_MS
variable whose rationale is moot with TTFB.
D5 conversation-runner probes were failing across all 17 showcase packages with
`conversation-error` because the selector cascade resolved to a `<div
data-testid="copilot-chat-input">` wrapper. Playwright's `page.fill()` only
works on input/textarea/select/contenteditable elements, so the very first
`fill()` call threw on every probe.
CopilotKit V2 renders the wrapper div around an actual `<textarea
data-testid="copilot-chat-textarea">`. Reorder the cascade in
`conversation-runner.ts` and `e2e-demos.ts` to target the real textarea first,
then a scoped descendant under the wrapper, then the bare `textarea`, then the
input-element fallbacks. The bare wrapper-div testid is dropped from the
runner cascade entirely (it would always make `fill()` throw); the e2e-demos
cascade keeps parity since it only checks visibility.
Tests: update the cascade-walk assertions in both test suites to reflect the
new ordering, and add a `fill()`-on-textarea assertion in conversation-runner
that pins the wrapper-div selector OUT of the fill path.
## Summary
- The e2e-deep (D5) driver expected `features[]` in its input but
railway-services discovery only populates `demos[]` (from
registry.json). Every service short-circuited with "no D5 features
declared" green — D5 probes were structurally complete but functionally
inert.
- Adds `d5-feature-mapping.ts` — a many-to-one mapping from registry
feature IDs to D5 feature types, derived from what each `d5-*.ts` script
actually registers.
- Driver now falls back to `demosToFeatureTypes(input.demos)` when
`input.features` is empty. Explicit `features` (used by tests) still
wins.
## Test plan
- [x] New tests: demos-only path, explicit features override, both-empty
short-circuit
- [x] `nx run @copilotkit/showcase-ops:build` — green
- [x] `nx run @copilotkit/showcase-ops:test` — 1287/1287 passing
- [ ] After merge + deploy: trigger `e2e-deep` probe, verify per-feature
`d5:<slug>/<featureType>` side rows appear in PocketBase (currently
zero)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The D5 e2e-deep driver fans out over `input.features`, but the
`railway-services` discovery source populates `demos[]` (registry
feature IDs) on each emitted record — `features` is never set in
production. Result: every service short-circuited the
"no D5 features declared" green path even when full demo coverage
was available.
Add `helpers/d5-feature-mapping.ts` translating registry feature IDs
(`tool-rendering-default-catchall`, `hitl-in-app`,
`shared-state-read-write`, etc.) to the closed `D5FeatureType` enum
the driver fans out over. Many-to-one (multiple registry IDs roll
up to one D5 type) and one-to-many (`shared-state-read-write` →
both read and write) cases are both supported. Unmapped IDs are
silently dropped — D5 covers a closed set.
Wire the mapping into `e2e-deep.ts`:
- Add `demos: z.array(z.string()).optional()` to `inputSchema`.
- Before the `requestedFeatures` filter, derive features from
`input.demos` via `demosToFeatureTypes` when `input.features`
is empty/absent. Explicit `features` always wins so existing
test fixtures never interact with the demos mapping.
Tests: add three cases to `e2e-deep.test.ts` covering (a) the
demos→features derivation including many-to-one + one-to-many
mapping and unmapped-ID drop, (b) explicit `features` overriding
populated `demos`, and (c) both empty still hitting the
"no D5 features declared" green short-circuit.
The e2e-parity driver has the same `features`-only inputSchema and
will pick up the same helper in a follow-up — out of scope here to
keep the fix minimal.
## Summary
Restores the two probe YAMLs for the D5 (`e2e_deep`) and D6
(`e2e_parity`) drivers. Without these, showcase-ops boots cleanly but D5
and D6 never tick.
## Why
PR #4292 originally shipped these YAMLs alongside the D5/D6 drivers. The
drivers themselves merged but were never wired into `orchestrator.ts`'s
registration block, so the probe-loader emitted
`probe-loader.file-failed` on every boot.
To stop the noise, commit `8e7417354` (PR #4300) deleted the orphan
YAMLs with the note "re-adding the YAMLs in a future PR is one-line."
PR #4301 has now landed the missing registry calls. This PR completes
the chain by restoring the YAMLs so the drivers can actually be
scheduled.
## Changes
- Restored `showcase/ops/config/probes/e2e-deep.yml` verbatim from
`origin/blitz/d5-driver`
- Restored `showcase/ops/config/probes/e2e-parity.yml` verbatim from
`origin/blitz/d6-driver`, plus a one-line comment near the schedule
documenting that D6 is hard-gated by the `D6_ENABLED` env var (default
false; driver short-circuits with "D6 disabled" note when unset; cron
schedule is irrelevant until the gate is flipped)
No source changes — pure YAML restore. The existing
`orchestrator.test.ts` registry-list assertion (added by #4301) covers
the wiring; this PR adds the YAMLs that wiring depends on.
## Test plan
- [x] `pnpm typecheck` clean (no source changes)
- [x] `pnpm test` — all 1286 tests pass
- [x] Formatter check clean
- [ ] CI green
PR #4292 originally shipped these YAMLs to register the new D5 (e2e_deep)
and D6 (e2e_parity) drivers. The drivers themselves merged but were never
wired into orchestrator.ts's registration block, so the probe-loader
emitted probe-loader.file-failed on every boot.
To stop the noise, commit 8e7417354 (PR #4300) deleted the orphan YAMLs
with the note "re-adding the YAMLs in a future PR is one-line."
PR #4301 has now landed the missing registry calls. This PR completes the
chain by restoring the YAMLs so the drivers can actually be scheduled.
Without these, showcase-ops boots cleanly but D5 and D6 never tick.
The e2e-parity.yml gets a one-line comment near the schedule documenting
that D6 is hard-gated by D6_ENABLED env var (default false); the cron
schedule is irrelevant until the gate is flipped.
No source changes — pure YAML restore. Existing orchestrator.test.ts
registry-list assertion (added by #4301) covers the wiring; this PR adds
the YAMLs that wiring depends on.
## Summary
Surface the U/W/C/T tier mapping inline next to the L1-L4 Status column
header on the Packages tab, so operators can read the strip without
cross-referencing docs.
| Letter | Label | Probe dimension |
|--------|-------|-----------------|
| U | Up | `health:<slug>` |
| W | Wired | `agent:<slug>` |
| C | Chats | `chat:<slug>` |
| T | Tools | `tools:<slug>` |
## Test plan
- [x] New test asserts the legend renders with exact text `(U=Up,
W=Wired, C=Chats, T=Tools)` and `data-testid="packages-uwct-legend"`
- [x] All 6 packages-section tests pass
- [x] `npx tsc --noEmit` clean
Surface the U/W/C/T tier mapping inline next to the L1-L4 Status column
header so operators don't need to cross-reference docs to read the strip.
Maps to the underlying probe dimensions per level-strip.tsx:
U = Up (health:<slug>)
W = Wired (agent:<slug>)
C = Chats (chat:<slug>)
T = Tools (tools:<slug>)
CR R1 follow-ups on top of the /api/ops proxy fix.
Bucket (a) — must-fix:
- Dockerfile: declare ARG/ENV OPS_BASE_URL in the builder stage. next.config.ts
evaluates rewrites() at build time and throws if OPS_BASE_URL is unset, which
aborted `next build` in CI. Mirrors the existing NEXT_PUBLIC_SHELL_URL /
NEXT_PUBLIC_POCKETBASE_URL pattern.
- showcase_deploy.yml: pipe OPS_BASE_URL through to docker build for the
shell-dashboard matrix entry, defaulting to the production
showcase-ops-production.up.railway.app URL.
- next.config.ts: strip trailing slashes from OPS_BASE_URL before constructing
the rewrite destination, matching the same normalization in
src/lib/ops-api.ts:resolveBaseUrl so server-side rewrite and client-side
fetch agree on the URL shape.
- src/lib/ops-api.ts: treat empty / whitespace NEXT_PUBLIC_OPS_BASE_URL as
"no override". `??` only short-circuits on null/undefined, so an env var set
to "" silently produced baseUrl="" and URLs of the form "/probes" with no
/api/ops prefix.
- use-probes.integration.test.tsx: snapshot+restore process.env.NEXT_PUBLIC_OPS_BASE_URL
in beforeEach/afterEach so tests never leak env state. Strengthen the proxy
contract assertions: lock toHaveBeenCalledTimes(1), assert method=GET,
cache=no-store, accept JSON header, and signal is an AbortSignal. Tighten
the 404 regression to assert the canonical ensureOk message shape so a
refactor that changes the format trips the test.
Bucket (b) — applied since the diff stayed focused:
- triggerProbe: add cache:"no-store" for parity with the GET fetches.
- fetchProbeDetail / triggerProbe: throw early when id is empty so callers
get a clean error instead of a request to /probes//... .
- ensureOk: bump body-truncation cap from 200 to 500 chars and append a
`[truncated, N bytes total]` marker so operators can see they're missing
tail bytes when the server returns a long HTML/stack-trace body.
- ops-api.ts: drop dev-loop review-cycle tag prefixes (R2-C.3, R3-C, R3-D.1)
from comments. Keep the actual rationale.
- ops-api.ts header docstring: clarify that NEXT_PUBLIC_OPS_BASE_URL is read
live at runtime in this codebase (SSR + tests), not just statically inlined
into the client bundle.
Verified:
- Tests: vitest run — 26 files, 293 passed, 1 skipped (no test count change).
- Typecheck: tsc --noEmit clean.
- Lint + format: oxlint + oxfmt clean on changed files.
- Local Docker build: `docker build --build-arg OPS_BASE_URL=https://...` succeeds.
Without --build-arg the build fails with "OPS_BASE_URL must be set" as
expected, confirming the fix is load-bearing.
Status tab was rendering "All probes idle. No upcoming runs scheduled."
even though showcase-ops `/api/probes` returns 9 probes.
Root cause:
* `lib/ops-api.ts` falls back to `/api/ops` when no explicit baseUrl and no
`NEXT_PUBLIC_OPS_BASE_URL` is set. `NEXT_PUBLIC_OPS_BASE_URL` is not
set on Railway shell-dashboard.
* `next.config.ts` was empty — there was no rewrite forwarding
`/api/ops/*` to the showcase-ops service. So the browser hit
`dashboard.showcase.copilotkit.ai/api/ops/probes` and got a 404 from
the dashboard origin itself.
* Direct cross-origin fetch from the browser was not an option either:
showcase-ops has no CORS allowlist.
Fix:
* Add a Next.js rewrite that forwards `/api/ops/:path*` to
`${OPS_BASE_URL}/api/:path*` server-side. The browser only ever sees
same-origin calls, and `OPS_BASE_URL` stays out of the client bundle.
* `OPS_BASE_URL` is required at build/start — fail loudly if unset
rather than silently rendering an empty status table.
* Update `lib/ops-api.ts` doc comment to reflect that `/api/ops` is now
the deterministic same-origin proxy path (contract), not a "guess".
* Add `src/hooks/use-probes.integration.test.tsx` — exercises the hook
through the real `lib/ops-api` client to a mocked `fetch`. Asserts the
default URL is exactly `/api/ops/probes`, parses real probe data, and
surfaces a useful error when the proxy 404s. The pre-existing
`lib/ops-api.test.ts` and `use-probes.test.ts` each mock at a different
layer, so neither caught this contract gap.
Operational follow-up: `OPS_BASE_URL` set on Railway shell-dashboard service
points to `https://showcase-ops-production.up.railway.app`. Auto-update
will pick up the rewrite on the next image rebuild.
## Summary
- The probe-invoker's three discovery-failure paths (`!resolved.ok`,
`MISCONFIGURED_DISCOVERY_INPUT`, `ENUMERATE_FAILED_INPUT`) emitted
synthetic-error
ProbeResults under bare `cfg.id`-derived keys (`smoke`,
`image-drift:enumerate-failed`,
`image-drift:misconfigured`). Those keys collide with the dashboard's
per-cell
`keyFor("<dim>", slug)` lookup and render as phantom red service cells.
Route all three paths through `discovery:${cfg.id}` so failures stay
surfaced
via PB queries / metrics / alerts but no longer pollute the integration
grid.
- Remove orphan probe configs `e2e-deep.yml` and `e2e-parity.yml`. Their
drivers
exist (PR #4292) but were never wired into orchestrator's
`probeRegistry.register()`
block, so the loader emitted `probe-loader.file-failed` on every boot.
The
closed-enum `DIMENSIONS` slots (`e2e_deep`, `e2e_parity`, `d5`, `d6`)
stay
in place so re-adding the YAMLs once registration lands is one line.
## Why
Original goal was to clear false-red Cs on the dashboard. The
smoke-probe slowness
piece was already shipped as #4299 (parallel-fanout in
`railway-services.ts`). Two
remaining root causes: (1) discovery-failure synthetic rows polluting
the
integration grid as phantom service cells; (2) orphan YAMLs producing
`probe-loader.file-failed` on every boot.
## Test plan
- [x] `nx run @copilotkit/showcase-ops:test` — 1284 tests pass (65
files)
- [x] `nx run @copilotkit/showcase-ops:typecheck` — clean
- [x] `pnpm lint` — clean
- [x] Updated `probe-invoker.test.ts` assertions for new keys at
lines 1093, 1413, 1515, 1610 (verified by green test run)
- [ ] After merge: confirm dashboard no longer shows phantom red cells
under
smoke/image-drift columns and `probe-loader.file-failed` stops appearing
in showcase-ops boot logs
## Summary
- Replaces the racy `ForwardedPropsMiddleware` (a `BaseHTTPMiddleware`
subclass that called `await request.body()` and reinstalled
`request._receive`) with a raw ASGI middleware that owns the receive
stream from the outset.
- Extracts the `forwardedProps` -> `state.inputs` splice into a pure
`_splice_forwarded_props` helper so the agent-config demo's tone /
expertise / responseLength contract is preserved verbatim.
- Adds `tests/python/test_forwarded_props.py` — TDD red-green coverage
that reproduces the production failure at the ASGI layer (no LLM, no
crewai required) and pins five contracts.
## Why
Production logs from the crewai-crews Railway service showed the chat
regression as AG-UI clients receiving `RUN_STARTED` and then the SSE
stream aborting with `RUN_ERROR: INCOMPLETE_STREAM` — no
`TEXT_MESSAGE_*` / `RUN_FINISHED` ever emitted. Server-side root cause:
```
RuntimeError: Unexpected message received: http.request
at starlette/middleware/base.py:55
```
The previous implementation used `BaseHTTPMiddleware`, called
`await request.body()` to inspect / mutate the JSON body, then
reinstalled `request._receive` with a one-shot replay. Stacked under
`HealthMiddleware` (also `BaseHTTPMiddleware`), that body-replay raced
with Starlette's inner anyio TaskGroup `wrapped_receive`. The
middleware's `forwardedProps` bailout check ran **after**
`request.body()` had already consumed the receive queue, so the bug
fired on every POST regardless of whether `forwardedProps` was present.
Switching to a raw ASGI middleware sidesteps the `BaseHTTPMiddleware`
machinery entirely. We buffer the request body once, optionally
rewrite it in dict form, and replay the (possibly mutated) bytes via a
single fresh `receive` to the inner ASGI app. No `_receive`
reinstallation, no inner TaskGroup to race against.
## Test plan
- [x] RED test reproduces the exact production `RuntimeError` via
Starlette `TestClient` — no LLM / crewai needed.
- [x] GREEN: `tests/python/test_forwarded_props.py` — all 5 cases pass:
streaming POST completes without `RuntimeError`, the racy
`BaseHTTPMiddleware` subclass is gone from the source, the real
`agent_server` module imports + streams cleanly, `/health` still
short-circuits, and non-agent-config POSTs pass through with the
original body bytes.
- [x] Existing test suite (`tests/python/test_aimock_toggle.py`) still
green: 99 / 99 pass overall.
- [x] Ruff lint count unchanged from `main` (13 pre-existing E402s from
the order-critical `load_dotenv()` shim — unrelated to this PR).
- [ ] CI to confirm.
- [ ] Smoke test the chat probe against the crewai-crews showcase
service after merge.
Production showed `probe-loader.file-failed` on every showcase-ops boot
after PR #4292 merged:
err="probe-loader: e2e-deep.yml: no driver registered for kind 'e2e_deep'"
err="probe-loader: e2e-parity.yml: no driver registered for kind 'e2e_parity'"
The D5 (`e2e_deep`) and D6 (`e2e_parity`) drivers shipped as exports
from showcase/ops/src/probes/drivers/e2e-{deep,parity}.ts but the
orchestrator never called `probeRegistry.register(...)` on them, so
the probe-loader rejected the YAMLs at boot, the drivers never ran,
and no D5/D6 PB rows ever got written.
Fix: import both drivers and register them alongside the rest. While
here, extract the registration block into an exported
`registerAllProbeDrivers(registry)` helper so a unit test can lock
the canonical kind set without spinning up the full boot path —
guarding against the same drift recurring on the next driver landing.
Origin: PR #4292.
The probe-invoker emitted synthetic-error rows for discovery-side failures
(enumerate threw, source missing, !resolved.ok) under bare cfg.id-derived
keys like `smoke`, `smoke:enumerate-failed`, or `image-drift:misconfigured`.
Those keys collide with the dashboard's per-cell `keyFor("<dim>", slug)`
lookup space and render as phantom red service cells (e.g. an `enumerate-
failed` integration appearing in the smoke column). Route all three paths
through `discovery:${cfg.id}` so the failures stay surfaced via PB queries,
metrics, and alerts but no longer pollute the integration grid.
Also remove orphan probe configs `e2e-deep.yml` and `e2e-parity.yml`. The
drivers exist as code but were never wired into orchestrator's
`probeRegistry.register()` block, so the loader emitted
`probe-loader.file-failed` on every boot. Drop the YAMLs until the
drivers are properly registered (the closed-enum DIMENSIONS slots stay so
re-adding the YAMLs in a future PR is one-line.)
Note: the parallel-fanout fix in railway-services.ts already shipped on
main as #4299 (commit 0e3e742c1) and is not included here.
## Summary
Production hotfix. The serial `for-of` loop over per-service
`variables()` GraphQL round-trips in `railway-services.ts` enumerate
scaled linearly with service count and was hitting the discovery
`timeout_ms` cap (30s on aimock-wiring + image-drift) once the fleet
grew past ~30 services. With 41 services × ~1s/round-trip = >30s
wall-clock, the discoveryAbort fired mid-loop and produced ~50
`discovery.railway-services.variables-failed` warns per tick.
## Fix
Replace the for-loop with a bounded-concurrency worker pool (8 in
flight) mirroring probe-invoker's existing pattern. Wall-clock drops
from ~ceil(N) round-trips to ~ceil(N/8).
Per-service error semantics preserved exactly:
- `AbortError` and `DiscoverySourceAuthError` still escape and propagate
up
- Transport flakes / 5xx / schema drift still degrade to `env={}` with a
warn
## Verification
- Local: `pnpm exec vitest run
src/probes/discovery/railway-services.test.ts` → 64/64 passing
- TypeScript: `pnpm exec tsc --noEmit` → clean
- Production: rebuilt + redeployed showcase-ops with this patch ahead of
the PR per the in-prod incident; PR documents the change for review
## Context
Surfaced as the runtime-tier "Railway adapter consolidation" risk called
out in PR #4294's risk analysis. Pre-existing N+1 made visible by R5-G4
D2 consolidation that routed aimock-wiring through the same enumerate
path as image-drift/e2e-smoke. Section A entry in the follow-up backlog
at https://www.notion.so/copilotkit/34e3aa3818528149ad3bc20165268dfc.
## Test plan
- [ ] Unit tests pass (`vitest run
src/probes/discovery/railway-services.test.ts`)
- [ ] Production: aimock-wiring tick after deploy emits 0
`variables-failed` warns
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The serial `for-of` loop over per-service variables() round-trips scaled
linearly with service count and routinely hit the discovery timeout_ms
cap (30s on aimock-wiring + image-drift) once the fleet grew past ~30
services. With 41 services × ~1s/round-trip = >30s total wall-clock,
the discoveryAbort fired mid-loop and produced ~50 variables-failed
warns per tick.
Replaces the for-loop with a bounded-concurrency worker pool (8 in
flight) that mirrors probe-invoker's pattern. Per-service degrade-on-
error semantics preserved exactly: AbortError + AuthError still escape
the catch and propagate; transport flakes / 5xx still degrade to env={}
with a warn.
Production verification: rebuilt + redeployed showcase-ops; expect the
discovery.railway-services.variables-failed warn cascade to drop from
~50/tick to 0/tick on the next aimock-wiring/image-drift cron fire.
CR Round 2 bucket-(a) fixes for PR #4298:
- agent_server.py: import sys; pass file=sys.stderr to the two observability
print() calls in ForwardedPropsASGIMiddleware (JSON parse failure +
replay_receive() exception path). The accompanying comment claimed stderr
was used "directly" but print() defaults to stdout, so log lines were
interleaving with the SSE response stream on stdout.
- test_forwarded_props.py: the fixture comment claimed "Module-scoped +
autouse" but the decorator is @pytest.fixture(autouse=True) without a
scope= argument (i.e. function-scoped, the default). Function scope is
REQUIRED here — the teardown's sys.modules.pop("agent_server", None) must
run between tests so each one re-binds against fresh stubs. Updated the
comment to match reality and explain why function scope matters; left the
decorator as-is.
Bucket-(b) hygiene applied while in the area (tests still pass):
- Replaced the inline `assert isinstance(forwarded, dict)` in the test-side
reference splice with the `if not isinstance: return body` pattern that
mirrors the production-side narrowing in agent_server.py
(assert is stripped under `python -O`).
- Tightened test_real_agent_server_streams_post_root_without_runtimeerror's
SSE assertions: parse the STATE event JSON and assert
state.inputs.tone == "casual" + the agent_config_guidance prefix shape,
rather than substring-checking the whole response body (which would pass
even if props landed in the wrong key).
Verification: 102/102 pytest passing. Standalone runs of
test_post_without_forwarded_props_passes_through_unchanged and
test_health_endpoint_short_circuits both pass. Ruff clean on edited lines
(pre-existing E402 in the load_dotenv ordering block intentionally left
alone). Pyright clean on edited surface.
Address bucket-(a) review findings on the raw ASGI middleware:
* replay_receive: explicitly catch and re-raise asyncio.CancelledError so
task cancellation propagates correctly (don't rely on the BaseException
hierarchy). Narrow the bare `except Exception` to log unexpected
transport errors before treating them as disconnect — silent swallow
was masking real bugs.
* Mid-buffer http.disconnect: replay buffered chunks as a single
http.request to the inner ASGI app BEFORE forwarding the disconnect.
Previously the inner app saw a clean disconnect with no body even
when partial chunks had arrived, hiding what the client actually sent.
* JSON parse failure: log a stderr breadcrumb with method/path context.
Behavior unchanged (still falls through to verbatim replay), but
malformed bodies are now observable instead of silently invisible.
Bucket-(b) cleanups picked up while editing:
* _splice_forwarded_props docstring updated to note in-place mutation.
* Replace the strippable `assert isinstance(forwarded, dict)` with a
defensive `if not isinstance(...)` so `python -O` doesn't lose the
guard.
* Use `k.lower() == b"content-type"` for parity with the content-length
filter and to be defensive against non-spec ASGI servers.
* Drop the empty-string defensive case from the path tuple — ASGI's
`path` is always at least "/".
Tests:
* test_replay_receive_propagates_cancellederror — drives the middleware
with a receive() that raises CancelledError and asserts it propagates.
* test_replay_receive_explicitly_handles_cancellederror — source-level
pin that `except asyncio.CancelledError` is present.
* test_replay_disconnect_delivers_buffered_chunks_before_disconnect —
asserts the inner ASGI app sees buffered chunks BEFORE the
http.disconnect when disconnect arrives mid-stream.
Full python suite: 102 passed (was 99 + 3 new). pyright clean on
agent_server.py modulo the pre-existing missing ag_ui_crewai stub.
Two correctness issues with the bucket-(a) tests CR called out:
1. The streaming regression test had a no-op pre-fix path: it built
`_OldForwardedPropsMiddleware` via `_build_app_with_real_middleware` but
then never called `app.add_middleware(OldMW)` — `if OldMW is not None:
pass` was dead. The "RED on master" claim was false; the regression
pin lives entirely in the structural assertion that grep-checks
`class ForwardedPropsMiddleware(BaseHTTPMiddleware` is gone from
agent_server.py. Verified the pin trips RED if the racy shape
reappears.
Deleted: `_build_app_with_real_middleware`, `_make_streaming_app`,
`_import_agent_server_helpers` (raised NotImplementedError, never
called), and the dead `if OldMW is not None: pass` block. Rewrote
the streaming test docstring to describe the actual ASGI middleware
fix (not the false "splice moved into route handler" narrative).
2. `test_health_endpoint_short_circuits` and
`test_post_without_forwarded_props_passes_through_unchanged`
depended on `test_real_agent_server_*` running first to install
`ag_ui_crewai` + `agents.*` stubs in `sys.modules`. Running either
alone (e.g. via `pytest -k <name>`, pytest-xdist, pytest-randomly)
failed with `ModuleNotFoundError: No module named 'ag_ui_crewai'`.
Moved stub setup into a module-scoped autouse fixture
(`_stub_agent_server_deps`). The fixture also drops the cached
`agent_server` from `sys.modules` so each test re-imports against
our stubs (not whatever sibling test installed first).
Verified: 99/99 pytest pass, standalone `pytest -k
test_post_without_forwarded_props_passes_through_unchanged` passes,
ruff clean, pyright 0 errors, structural pin trips on simulated
pre-fix source.
Add a hard env-var gate at the top of e2e-parity.ts::run. When
D6_ENABLED is anything other than the literal string "true"
(case-insensitive) the driver short-circuits with an aggregate green
"D6 disabled" signal and emits no per-feature side rows.
Why: D5 ships immediately (auto-fires after merge) but D6 is deferred
until calibration is ready. The previous soft gate (throw on unset
D6_MODE) flapped red on every tick — wrong signal for "intentionally
deferred". Green-with-note matches existing skipped semantics.
The gate supersedes the cron schedule: even with the schedule firing,
the driver no-ops unless an operator flips the flag on the
showcase-ops Railway service. Once enabled, all existing behaviour
applies (D6_MODE / D6_TARGET_INTEGRATION still required for
on-demand).
Tests cover unset / "true" / "TRUE" / "false" / "" cases with
mock call-count assertions proving collaborators are not invoked
when disabled. README and YAML config updated to document the flag.
Pyright was unable to narrow `inputs` after the
`state.get("inputs") if isinstance(state.get("inputs"), dict) else {}` pattern
(two distinct expressions, no narrowing across them) — it inferred
`Unknown | dict | None` and then mismatched `dict()` overloads, which cascaded
into bogus bytes-key errors on `inputs[...] = ...` assignments. Pull the
narrowing into a single named binding and add explicit `dict[str, Any]`
annotations on `state` and `inputs` so the dict-mutation calls type-check.
Also add an `assert isinstance(forwarded, dict)` after `_has_agent_config_props`
returns true (the helper already guarantees it, but Pyright can't see the
contract through the bool return).
In tests, replace direct attribute assignment on `types.ModuleType` instances
with `setattr()` — the dynamic stubbing is the test's whole purpose, and
Pyright correctly flags the static attribute writes as
`reportAttributeAccessIssue`. `setattr()` is the standard escape hatch for
this pattern.
No behavior change. 99/99 tests still green.
Small, collapsible disclosure panel above StatsBar on the Cells tab with
plain-English definitions of the D0–D6 chip layers. Default collapsed;
open/closed state persists across navigations via
localStorage["chip-explainer-open"]. Footer link points to the Notion
plain-English page for deeper detail.
The Cells tab is consulted dozens of times a day by operators who already
know what the chips mean; the explainer is for first-time viewers and
rare refreshers, so the trigger is a small text button and the default
is collapsed to keep the layout clean.
The chat regression on crewai-crews surfaced as AG-UI clients receiving
RUN_STARTED and then the SSE stream aborting (RUN_ERROR: INCOMPLETE_STREAM,
no TEXT_MESSAGE_* / RUN_FINISHED ever emitted). Production logs from the
Railway crewai-crews service confirmed the abort was triggered by:
RuntimeError: Unexpected message received: http.request
raised from `starlette/middleware/base.py:55` mid-stream.
Root cause: ForwardedPropsMiddleware (introduced as part of the agent-config
demo plumbing) was a BaseHTTPMiddleware subclass that called
`await request.body()` to inspect / mutate the JSON body, then reinstalled
`request._receive` with a one-shot replay. Stacked under HealthMiddleware
(also a BaseHTTPMiddleware), that body-replay raced with Starlette's inner
anyio TaskGroup `wrapped_receive`. The middleware fired its bailout check
(`_has_agent_config_props`) only AFTER `request.body()` had already consumed
the receive queue — so the bug fired on every POST, regardless of whether
forwardedProps was present.
Fix: drop ForwardedPropsMiddleware and replace it with a raw ASGI middleware
(`ForwardedPropsASGIMiddleware`). The raw form owns the receive stream from
the outset and replays buffered bytes via a single fresh receive — no
`_receive` reinstallation, no BaseHTTPMiddleware machinery to race against.
The forwardedProps -> state.inputs splice logic is unchanged (extracted into
a pure dict-in / dict-out `_splice_forwarded_props` helper) so the
agent-config demo's tone / expertise / responseLength contract is preserved.
TDD: tests/python/test_forwarded_props.py reproduces the bug at the ASGI
layer (no LLM, no crewai required) and pins five contracts:
- streaming POST through the middleware stack completes without RuntimeError
- the racy `class ForwardedPropsMiddleware(BaseHTTPMiddleware` is gone
- end-to-end import of agent_server + POST to / streams cleanly
- /health continues to short-circuit
- non-agent-config POSTs pass through unchanged
Pre-fix RED: 3 failures, including the exact RuntimeError from production
logs surfaced via Starlette TestClient. Post-fix GREEN: 99 / 99 pass.
R4-A.2: ProbeInvokerDeps.schedulerId is now required (no `?? cfg.id` fallback).
The earlier optional+default path silently re-introduced the R3-A.2 bug for any
caller that forgot to supply the prefixed scheduler id — tracker registration
became a silent no-op against the live scheduler. Required-at-typecheck
surfaces missing-prefix mistakes immediately. All 30 test invoker
constructions updated to pass schedulerId: cfg.id.
R4-A.7: outer fan-out catch now synthesizes a failed-tile + bumps `failed`
+ flips runState to "failed" instead of just logging. Pre-fix, if an
invariant inside the inner pool ever broke, the run would persist as
state="completed" with failed:0 while a real defect occurred — fake-green.
Post-fix, the synthetic tile is keyed `<probeId>:__internal_invariant__`,
written through the writer like any other failure, and the RunSummary
preserves total === passed + failed.
Adds covering tests for both: the typecheck contract via @ts-expect-error
on a fenced call, and the runtime fail-loud behavior via a fake scheduler
whose tracker.start throws (the only path that escapes Promise.all into
the outer catch).
- R4-A.1: response `scope` is now null when no filter was provided in the
request body, vs the actual array (possibly empty) when filter.slugs was
sent. Operators can distinguish "no filter" from "empty scope" in audit
logs.
- R4-A.4: post-read body cap now compares Buffer.byteLength(raw, "utf8")
against TRIGGER_BODY_LIMIT_BYTES instead of raw.length. UTF-16 char
counts let multibyte payloads (emoji = 4 bytes / 2 code units) slip
past a byte-denominated cap.
- R4-A.5: replaced the post-read length check with Hono's bodyLimit
middleware. Chunked uploads that omit (or lie about) Content-Length
now abort the body read at limit+small via a streaming reader, rather
than buffering the full body before checking. BodyLimitError surfaces
as 413 on the read; the post-read byte check stays as defense in depth.
- R4-A.6: rate-limit rollback now fires on ALL non-success paths from
scheduler.trigger, not just InflightConflictError. A transient 5xx no
longer locks the operator out of the 5-min window. Added structured
console.error("probes.trigger-unexpected", ...) before the rethrow so
500s are diagnosable from logs alone.
Tests: 6 new red-green tests covering scope null vs [], UTF-8 byte cap,
streaming early bail, and generic-error rollback. Updated the existing
"queued envelope" test to assert scope: null (was scope: []).
R4-B.1: useProbes effect now calls setData(null) on dep change to mirror
useProbeDetail's CR-B1.4 behavior. Without this, the dashboard renders
stale list data from the previous baseUrl until the new fetch resolves.
R4-B.6: useTriggerProbe checks controller.signal.aborted after the await
resolves. If a back-to-back trigger aborted the controller after the
fetch already returned, honor the R3-C.1 supersession contract and
resolve to null instead of returning the stale TriggerResponse.
serve() from @hono/node-server returns the http.Server synchronously, but
the bind happens via server.listen() which emits 'error' asynchronously
for conditions like EADDRINUSE. The R2-B.2 try/catch only caught
synchronous throws, so a real bind failure resolved boot() successfully
with an orphaned scheduler still ticking.
Race 'listening' vs 'error' on the returned server. On 'error', flip
schedulerRunning false, fire scheduler.stop() as a separate cleanup
chain (so the rejection isn't gated on stop), and reject boot() with
the original error. Guard the 'already listening' edge case so a sync
bind that beats listener attachment doesn't hang.
Adds covering test that mocks serve() to return an EventEmitter-like
stub which emits 'error' on next tick — boot() must reject and
scheduler.stop() must be called.
Two related test-isolation fixes for `src/orchestrator.test.ts`:
1. Bump global `hookTimeout` from 10s default to 60s. Several integration
tests boot a real Hono server, chokidar watcher, scheduler, and PB
client in beforeEach and tear them down in afterEach. On macOS + CI
runners the teardown can spike well past the default 10s ceiling
(chokidar.close on a deleted tempdir + http.Server.close + scheduler
settle), producing cascading "Hook timed out in 10000ms" failures on
/health-wiring, OPS_TRIGGER_TOKEN, and probe-unregister blocks. The
R21-a SIGHUP-reload test deletes its configDir mid-test and chokidar's
watcher.close() on a vanished path can hang for ~30s. 60s absorbs the
variance without masking real hangs (a stuck stop() blocks for minutes,
not seconds).
2. R21-a SIGHUP test: replace the fixed `setTimeout(100)` wait with a
polling loop that checks for the synthetic `<sighup>` emission with a
15s ceiling. Empirically the SIGHUP reload chain
(loader.load → loadDefaults readFile ENOENT → fs.readdir ENOENT →
reloadRules .catch → bus.emit) settles 2-5s later on macOS under load
— the prior 100ms budget produced a deterministic false-fail even
though the production path was correct.
Result: full showcase-ops suite — 941/941 tests pass; orchestrator.test.ts
run twice consecutively without flake. Typecheck clean.
Test names and section banners carry CR-round bookkeeping markers
((A1), (A6), (A11), (E1), (E2), F1.B/C/D/E/F) that leaked into the
suite during the R4 fix cycle. Those markers are useful while the
round is open but become noise once the work lands — operators
running `vitest -t` against the master suite shouldn't have to
remember which bucket a fix came from.
Strip all CR-bucket prefixes from `it()` names, describe-block names,
and section-divider comments. Test contents and assertions are
unchanged; this is a pure rename pass for long-term hygiene.
If `probeLoader.watch(...)` threw synchronously (e.g. an underlying
fs.watch ENOENT path), `unwatchProbes` was never bound, so a
subsequent `stop()` invocation hit `ReferenceError: unwatchProbes is
not defined` and orphaned the engine + scheduler.
Initialize `unwatchProbes` as a no-op BEFORE the watch() call, then
overwrite if registration succeeds. A failed init is logged via
`logErrorWithStack` rather than crashing the boot path.
R5-G4 D7.
Pre-fix, if both SHARED_SECRET and SHARED_SECRET_PREV were unset/empty,
webhookSecrets resolved to `[]` and webhook auth on the deploy.result
endpoint was silently disabled — anyone who knew the URL could POST.
Mirror the POCKETBASE_URL fail-loud pattern: in production, missing
webhook secret is a deploy bug that must surface immediately, so throw
FATAL-CONFIG. In dev/test emit an info-level log so developers see the
bypass without crashing local boot.
R5-G4 D5.
Comment-only cleanups (no behaviour change):
A3: rephrase the logErrorWithStack JSDoc so the "every time" wording
no longer contradicts the implementation (string-thrown values have
no stack to preserve, so the full-stack debug emission is conditional).
A4: collapse two duplicate sort-gate paragraphs into one, mention
"head-of-line blocking under bounded concurrency" so the reasoning
for why the sort exists is in one place rather than scattered.
A5: replace the dismissive "listener-cleanup is not a real concern"
finally-block comment with an honest acknowledgement that the listener-
leak window is bounded by GC of the AbortController after resolveInputs
returns. Documents the tradeoff rather than pretending it doesn't
exist.
A10 (schema.ts): drop the "the 845/847 unit-suite passed" reference
that pinned the comment to one specific test count. Replace with a
generic statement that the loader unit tests use stubs not real YAMLs,
so the gap only surfaces in integration.
Jordan Ritter