CR batch for packages/runtime managed-channels activation:
- RC11: getOrCreateChannelManager now passes a `log` adapter bridging the
ChannelManager diagnostic sink to the shared logger
(`log: (msg, meta) => logger.warn({ meta }, msg)`). Previously every
breadcrumb (setup_required, failed-to-activate, dropped-session,
teardown-stop failure) was a no-op, so a channel that failed to activate
was permanently dead with zero output.
- f2: stopEntry logs a swallowed handle.stop() error via the sink instead of
discarding it; teardown stays resilient (never rethrows).
- sync-throw guard: stopEntry wraps handle.stop() in
`Promise.resolve().then(...)` so a foreign/injected handle that throws
SYNCHRONOUSLY is caught by the same `.catch` — otherwise the throw escaped,
skipped resolveSettled(), and hung `settled` forever.
- f3: ready() short-circuits and RESOLVES when the manager is stopped. A
channel that settled to `error` before stop() had already rejected its
`settled` promise, so a later ready() threw an AggregateError even though
status().overall was "stopped" — now consistent with the after-stop case.
- RC12: parseProjectIdFromApiKey no longer slices a fixed 8 chars off an
arbitrary key (which echoed secret bytes for a `cpk-_...`-shaped key). The
failure message now echoes NONE of the key value, only the expected
`cpk-{projectId}_` format hint.
- RC13: deriveChannelActivationConfig enforces the lowercase-kebab-case
channel-name rule (/^[a-z][a-z0-9]*(?:-[a-z0-9]+)*$/, length 3-64) up front
and throws a clear ChannelConfigError, instead of passing a bad name to the
launcher where assertValidChannelRealtimeScope throws deep and the channel
is silently degraded to `error`. Regex/bounds are a literal copy of
channels-intelligence's assertValidChannelRealtimeScope (the source of
truth; not statically imported — it's an optional pure-ESM peer).
- ready() docblock reworded: async ready() REJECTS (not throws) the
ChannelConfigError.
Tests (red-green verified): RC11 (handler logger spy + manager log sink),
f3 (ready resolves post-stop after pre-stop error), sync-throw guard, RC12
(no secret-tail leak), RC13 (Slack/support_bot/cs/65-char reject; support
passes).
Call-site enumeration for changed signatures/behaviors:
- getOrCreateChannelManager (new internal `log` arg): single caller
fetch-handler.ts:402; public signature unchanged, no caller impact.
- deriveChannelActivationConfig (RC13 now throws on bad name): single caller
channel-manager.ts:323 inside activate()'s per-channel loop, already
wrapped in try/catch that converts a throw to a rejected activation ->
recorded as `error` status and surfaced via ready()'s AggregateError.
- parseProjectIdFromApiKey (RC12 message-only change): single caller
channel-activation-config.ts:131; error type/behavior identical.
- ready() (f3 early-return): only affects a stopped manager (now resolves
instead of throwing) — strictly more lenient; sole public reference is a
doc example in endpoints/node.ts:42.
Skipped: a test exercising defaultActivateChannel's module-not-found friendly
error — the fn is unexported and the dynamic import specifier is not
injectable, and channels-intelligence IS installed in the workspace so a real
MODULE_NOT_FOUND can't be forced without contorting the code. isModuleNotFound
remains unit-tested.
Centralize the stop-vs-settle race class in ChannelManager behind one guarded,
idempotent teardown path instead of per-branch patches:
- ChannelEntry gains a private `handleStopped` flag; new private `stopEntry()`
sets status="stopped" and stops the handle AT MOST once. Both settle handlers
and stop() route through it.
- RC5: a rejection arriving AFTER stop() now keeps the entry "stopped" and
resolves settled (no error/setup_required, no rejectSettled), so a late
connect failure can't resurrect a stopped channel or reject a later ready().
- RC7: stop() runs `Promise.allSettled` over per-entry stopEntry() calls; the
handleStopped guard means a handle assigned in the same tick as stop() is
stopped exactly once even when both stop() and the success handler reach it.
- RC9 (fetch-handler): getOrCreateChannelManager now calls activate() BEFORE
inserting into the WeakMap, so a synchronous throw (duplicate/missing names)
caches nothing and every retry re-throws instead of returning an inert
manager that falsely reports "online".
- RC8: reconcile class + ready() docstrings — activation throws synchronously
(ChannelConfigError) only on up-front misconfiguration; all other failures
are recorded as channel status.
- assertUniqueChannelNames checks missing/empty name FIRST so two nameless
channels get the accurate "missing name" error, not a spurious "undefined"
duplicate.
- Remove the dead ChannelEntry.promise field (unread residue of the removed
reconnect path).
- RC4 (packaging): move @copilotkit/channels-intelligence from
optionalDependencies (auto-installed, force-pulls the pure-ESM package into
every OSS consumer) to an optional peerDependency, mirroring the other
optional integrations.
- Test nits: clear the dangling stop()-hang setTimeout; drop the redundant
not.toBe("reconnecting") assertion.
Call sites of changed symbols:
- stopEntry (new private): channel-manager.ts only — success handler, reject
handler, and stop(); no external callers.
- ChannelEntry.promise (removed): grep confirms no reads anywhere in the repo
(the only .promise reads are unrelated test signals).
- getOrCreateChannelManager (reordered, no signature change): single caller at
fetch-handler.ts createCopilotRuntimeHandler.
TDD: RC5 and RC9 red-green verified against prior code (RC5 reported "error"
not "stopped"; RC9 retry returned an inert healthy manager). RC7 pins the
single-stop guarantee for the new idempotent design.
RC1 — remove manager-level re-activation reconnect (delegate to Phoenix). The
ChannelManager's supervised reconnect re-invoked the activation engine on the
SAME already-started Channel, which throws in channel.addAdapter (started=true)
— so it could never succeed on the real launcher. It was also redundant:
Phoenix's Socket auto-reconnects and auto-rejoins, re-sending the join
declaration; the gateway's join/3 re-runs record_heartbeat (re-registers the
listener) and terminate/2 releases the dead socket's leases (verified against
Intelligence #511 sdk_channel.ex). Removed: runReconnect, reconnectLoops,
onChannelClosed, the RECONNECT_BASE_DELAY_MS / RECONNECT_MAX_DELAY_MS /
RECONNECT_MAX_ATTEMPTS constants, the injectable sleep arg + defaultSleep, and
stoppedSignal/resolveStopped. onClose is now a log-only breadcrumb (no state
mutation, no re-activation). Once a channel activates it stays online; a
transient drop is invisible (Phoenix self-heals). ChannelStatus keeps
"reconnecting" in the union marked reserved to avoid churning the public type;
computeOverall no longer assigns it.
RC2 — stop() no longer aborts teardown on a throwing handle.stop(). The real
launcher's stop() rethrows after session.disconnect(), so Promise.all rejected
and skipped the status loop; with stopped already set, a retry no-oped, leaving
the manager permanently un-torn-down. Switched to Promise.allSettled so every
handle attempts teardown and every entry is marked "stopped". Red-green test
added.
RC3 — parseProjectIdFromApiKey no longer echoes the full cpk-… secret in
ChannelConfigError (it is logged and surfaced via ready()'s AggregateError).
Message now includes only a short non-sensitive prefix; test asserts the format
hint is present but the full key is not.
Reconnect unit test rewritten to the new contract (a drop makes no further
engine call, does not throw, manager stays usable/coherent); obsolete
backoff-growth / give-up-to-error / cancel-pending-backoff cases removed.
Integration test step 5 updated: a drop stays online with no extra engine call.
Call sites cleared: grep over packages/runtime/src for runReconnect,
reconnectLoops, RECONNECT_BASE_DELAY_MS, RECONNECT_MAX_DELAY_MS,
RECONNECT_MAX_ATTEMPTS, onChannelClosed, stoppedSignal, resolveStopped,
defaultSleep, and the injectable ChannelManager sleep arg returns no matches;
fetch-handler exposes no sleep/reconnect seam. Nothing external referenced the
removed symbols.
A1: ChannelManager.activate() now asserts unique channel names before any
engine call (entries keyed by name silently leaked the first session on a
duplicate). Throws ChannelConfigError naming the dup. Reworded the stale
runtime.ts comment that claimed startChannels validates uniqueness — the
managed path activates one Channel per launcher call, so uniqueness is
enforced by ChannelManager.activate().
A4: stop() no longer awaits pending activations (a hung connect that
ready({timeoutMs}) tolerates would hang teardown/SIGTERM forever). It stops
only handles that already exist; a post-settle guard on the initial-activation
path tears down any handle arriving after stop(), mirroring the reconnect
loop's guard. Idempotent.
A3: reconnect success clears the reconnectLoops marker BEFORE re-arming
onClose, so a synchronous onClose re-fire on the fresh handle starts a new loop
instead of leaving the Channel stuck reconnecting with no loop.
B1/B3: doc fixes — onClose seam is present-tense (launcher delegates to
session.onClose); parseProjectIdFromApiKey @throws no longer describes an
unreachable empty-segment case.
Call sites reviewed (behavior holds at each):
- activate() throws on dup: fetch-handler.ts getOrCreateChannelManager (l.187),
reached from createCopilotRuntimeHandler at handler-creation time → now fails
loud at startup instead of leaking; channel-manager ready() (l.424) surfaces
the throw as a rejected promise.
- stop() prompt-resolve: examples/slack/app/managed.ts:174 SIGTERM shutdown
await listener.channels?.stop() — the exact hang this fixes. Endpoint
adapters (node/express/hono) only attach .channels; no direct stop callers.
Tests: channel-manager.test.ts + channel-manager-reconnect.test.ts 17 passed
(2 new + 1 new, red→green); channel-activation-config + fetch-handler green;
@copilotkit/runtime:check-types clean.
Addresses review on #5969:
- Add getMessages tests: string content, content-part array (text parts joined,
non-text contributes ""), and role→isBot/user derivation; plus the no-getHistory
transport case returning [].
- Log on the unexpected getMessages catch (matches conversationStore's seeding
path) instead of degrading to [] silently — HttpDeliverySource already swallows
its own fetch failures, so this outer catch only fires on a real throw.
- Fold the duplicated role boolean (isBot computed once, drives both fields).
## Summary
Companion structural PR to #5971. It fixes the **root cause** behind the
a2ui divergence #5971 patched: the showcase's single-source symlinks
eroded to real, drifting copies.
`showcase/integrations/*/tools`, `*/shared-tools`, `*/_shared` are meant
to be **symlinks into `showcase/shared/...`** — `stage_shared()`
dereferences them for the Docker build, `restore_symlinks()` restores
them. An **accidental `stage_shared()` leak** (commit `534cd1efa7`, PR
#4449 "D5 all-green") committed the dereferenced real files instead of
restoring the symlinks. Once they were real files, they drifted — which
is exactly how the a2ui `render_a2ui` vs `_design_a2ui_surface` split
(fixed in #5971) arose.
## Changes
- **Restore 12 Python `tools/` dirs to symlinks** →
`../../shared/python/tools` (ag2, agno, claude-sdk-python, crewai-crews,
google-adk, langgraph-fastapi, langgraph-python, langroid, llamaindex,
ms-agent-python, pydantic-ai, strands). Content is byte-adopted from
shared — verified no load-bearing per-integration code is lost (only
`render_a2ui` naming + shared `roll_dice`/sanitize additions).
Integrations' intentional internal-planner names (llamaindex,
ms-agent-python) live in `src/`, not `tools/`, and are untouched.
- **`showcase/AGENTS.md` (+ `CLAUDE.md`, root pointers,
INTEGRATION-CHECKLIST section)** — canonical statement of the 4 iron
rules (identical tests, near-identical frontends, minimal backends,
per-integration fixtures) + the single-source symlink mechanism ("edit
the shared source only; a real file there is a bug"). These were
previously written down nowhere.
- **`validate-shared-symlinks` CI guard** — fails on any NEW erosion
(real dir where a symlink belongs), with a shrink-only baseline that
tightens to fully-enforcing as symlinks are restored. Mirrors the
existing `validate-*` ratchet pattern.
## Scope / independence
- **No overlap with #5971** — this PR touches nothing under
`showcase/shared/typescript/` and does not modify the 3 TS integration
`shared-tools/` dirs (verified: empty file-set intersection). Mergeable
independently.
- Build-safe: `stage_shared()` correctly dereferences the restored
symlinks (targets resolve within the build context);
`restore_symlinks()` recreates them post-build.
## Verified
- `validate-shared-symlinks` test suite: 7/7 pass; validator EXIT 0 (no
new erosion).
- Reviewed by a full panel (correctness, content-integrity, build/CI,
docs, scope, silent-failure, simplicity) — zero mandatory findings.
## Follow-ups (deliberately out of scope)
1. **3 TS `shared-tools` dirs** (mastra, claude-sdk-typescript,
langgraph-typescript) remain real (baselined) — symlink them in a
follow-up **after #5971 merges**, to avoid overlapping its TS edits.
2. **Guard hardening**: validate the symlink *target* (not just that
it's a symlink), fail-loud on a malformed baseline, and code-enforce the
shrink-only ratchet. (This PR's guard catches the real-file erosion —
the actual failure mode; these are robustness extras.)
3. Pre-existing `shared/python` a2ui test failures (#5971-adjacent) and
a couple of stale doc line-refs, noted during review.
Companion: #5971.
## Summary
The TypeScript A2UI operation builders
(`buildA2uiOperationsFromToolCall`) emitted the **legacy flat**
operation shape (`{ type: "create_surface", surfaceId }`). A2UI
consumers process operations by their **nested** `createSurface` /
`updateComponents` / `updateDataModel` keys — a flat op is never
processed as a valid nested operation, so the surface's schema and
components are never applied and the UI renders nothing (the
`generate_a2ui` / `render_a2ui` path).
Python was fixed to the v0.9 nested shape long ago (#4792, #5832); the
TypeScript builders were **born flat and never fixed** — a Python/TS
parity gap with no guard. This aligns the TS side and adds a guard so
the two can't silently drift again.
## Changes
- **4 TS builders → v0.9 nested** (byte-identical):
`shared/typescript/tools` + integrations `mastra`,
`claude-sdk-typescript`, `langgraph-typescript`.
- **Empty-data parity fix**: TS `if (data)` treated `{}` as truthy and
emitted a spurious `updateDataModel` op; Python `if data:` does not. Now
guarded to match Python (empty object → no `updateDataModel`). Our
mastra fixture records `"data": {}`, so this is exercised directly.
- **v0.9 parity guard test** in all 4 test files (asserts nested keys,
no flat `type`).
- **12 `gen-ui-a2ui-fixed` aimock fixtures** for the fixed-schema a2ui
demo.
## Red–green evidence
- Empty-data: pre-fix builder emits 3 ops on `data:{}` →
`toHaveLength(2)` **FAILS (red)**; fixed builder emits 2 → **passes
(green)**.
- Parity guard: flat shape → `.type` present / nested keys absent →
**red**; nested → **green**.
## Validation (please read — what CI does and doesn't cover)
CI **does** run `check-types`, `format`/`oxlint`, `Validate Showcase`,
and `build-check` on the changed integrations (mastra,
langgraph-typescript, claude-sdk-typescript) — these catch TS/build
breaks. But the unit-test workflow has `paths-ignore: showcase/**`, so
the **showcase vitest suites where the parity guard and aimock-fixtures
tests live are NOT run in CI**. Those were validated **locally**:
- `aimock-fixtures`: **837 passed** (all 12 new fixtures valid).
- Parity guard + empty-data red–green: **verified**
(`showcase/shared/typescript` vitest).
- `tsc --noEmit --strict`: **clean** on all 4 builders.
- **mastra Playwright screenshot**: the `render_a2ui` flow renders the
flight card (SFO→JFK, Flight Details, $289) with the nested ops.
## Real-surface confirmation (bin/showcase test --direct)
Proven on the live probe, not just unit tests:
- **RED** (old flat builder): `d6:mastra a2ui-fixed-schema` → `1 failed
— [data-testid="a2ui-fixed-card"] failed to mount within 60000ms`.
- **GREEN** (this branch, rebuilt image): `d6:mastra a2ui-fixed-schema`
→ `✓ green, 1 passed` — the card mounts and renders (SFO→JFK, UNITED,
$289, "Book flight").
- **≥3 cells `--direct` GREEN**: `mastra` ✓, `langgraph-typescript` ✓,
`pydantic-ai` ✓.
- **Frontend parity**: mastra ≡ langgraph-python render the same flight
card (byte-identical frontend; only expected agent-specific
tool-row/ordering differences).
## Iron-rule adherence
| Rule | Evidence |
|------|----------|
| Identical tests | One shared harness probe (`d5-gen-ui-a2ui-fixed`)
measures the feature across all integrations; no per-integration test
copies added. |
| Near-identical frontends | mastra ≡ langgraph-python frontend
(byte-identical); visual parity confirmed on the rendered card. |
| Minimal backends | Change is the minimal flat→nested + empty-data
guard; 4 TS builder copies byte-identical. |
| Per-integration fixtures | 12 `gen-ui-a2ui-fixed.json`, one per slug,
context-keyed. |
Note: the *single-source symlink* restoration (the
`shared-tools/`/`tools/` symlinks that eroded to real files repo-wide)
is handled in a **companion structural PR**, plus a `showcase/AGENTS.md`
documenting the iron rules and a CI check that fails on future erosion.
## Known follow-ups (out of scope here)
1. **`render_a2ui` naming drift (pre-existing):** the *shared* builder
exports `_design_a2ui_surface` while the 3 integrations + Python use
`render_a2ui`; the shared test asserts `render_a2ui` and is red on
`main` today. Not run by CI. A one-line rename aligns shared to the
source of truth and greens the file — happy to fold it in if wanted.
2. **Integration test copies aren't executed** by their vitest `include`
globs (only `shared/typescript` runs its copy) — add a cross-copy
byte-identity CI check.
3. **Full `_sanitize_a2ui_components` parity** — TS forwards components
raw; Python sanitizes (drops entries missing id/component, unstringifies
Gemini JSON-string arrays).
4. **Showcase tests are CI-excluded** (`paths-ignore: showcase/**`) — no
automated gate for showcase unit tests.
Restore the 12 Python integration tools/ dirs to symlinks into
shared/python/tools. They had eroded to real, drifting copies via an
accidental stage_shared() leak (commit 534cd1efa7) — the structural root
cause of showcase divergence bugs. Symlinking re-establishes the single
source of truth; content is identical to shared (only render_a2ui naming
and the shared roll_dice/sanitize additions are adopted).
Add showcase/AGENTS.md documenting the 4 iron rules and the single-source
symlink mechanism, plus a validate-shared-symlinks CI guard (shrink-only
baseline) that fails on any NEW erosion.
The shared TS builder exported `_design_a2ui_surface` / `DESIGN_A2UI_SURFACE_TOOL_SCHEMA`
while the Python builder, all integration copies, and agent_server.ts use `render_a2ui`
/ `RENDER_A2UI_TOOL_SCHEMA`. Align the shared source to that source-of-truth name; this
also unbreaks the re-export in shared/typescript/tools/index.ts.
Flip the 4 TS a2ui builders (shared/typescript + mastra,
claude-sdk-typescript, langgraph-typescript) from the legacy flat
operation shape to v0.9 nested (createSurface / updateComponents /
updateDataModel), matching the Python builder and what A2UI consumers
process. Flat ops were never processed as valid nested operations, so
the surface schema and components were never applied.
Also align the empty-data guard to Python's `if data:` semantics (empty
object -> no updateDataModel), add a v0.9 parity guard test to all 4
test files, and add 12 gen-ui-a2ui-fixed aimock fixtures.
getHistory was Slack-only: it keyed off threadTs and returned [] for any route
without one, so a Teams route ({adapter:'teams', tenantId, conversationId})
never hit /api/channels/history. That starved BOTH history paths on Teams --
agent.messages seeding (conversationStore.getOrCreate) and the read_thread tool
-- so the agent always reported "no earlier messages" / "no image in thread".
- getHistory is now adapter-aware (mirrors conversationKeyFromReplyTarget's
per-adapter switch): Slack keys off teamId/channel/threadTs; Teams sends
adapter=teams + tenantId + conversationId, matching app-api's
teams:{tenantId}:{conversationId} thread_key. The /api/channels/history route
already accepts this shape. Slack query order is unchanged.
- Add getMessages to the adapter so thread.getMessages() (the read_thread tool)
reads reconstructed history via the transport and maps it to ThreadMessage[].
Without it Thread.getMessages() returns [] and thread-reading tools see
nothing even when history exists.
Adds tests for the Teams getHistory query and the missing-id short-circuit.
## Summary
Pressing **Stop** while an assistant message is streaming
(CopilotRuntime + `HttpAgent` proxy) crashed the chat with:
```
Cannot send event type 'TEXT_MESSAGE_END': The run has already errored with 'RUN_ERROR'. No further events can be sent.
```
Root cause: `finalizeRunEvents` appended a trailing `TEXT_MESSAGE_END`
**after** the `RUN_ERROR` that the aborted agent had already emitted.
Fixes#5812.
## Root cause
When the upstream agent (e.g. pydantic-ai's `AGUIAdapter`) is aborted
mid-stream it emits a live `RUN_ERROR` while a text message is still
open — it does **not** close the message first. All runners
(`in-memory`, `intelligence`, `sqlite`) stream `finalizeRunEvents`'
output *after* everything the agent already emitted, so the appended
closer landed past the terminal:
| | outgoing event order |
|---|---|
| **Before** | `… TEXT_MESSAGE_CONTENT → RUN_ERROR → TEXT_MESSAGE_END` ❌
verifier throws |
| **After** | `… TEXT_MESSAGE_CONTENT → RUN_ERROR` ✅ terminal closes the
message client-side |
Per the AG-UI invariant: at most one terminal event per run, and no
sub-events after it. I confirmed against the real `@ag-ui/client`
`verifyEvents` (the verifier the browser runs) that a terminal arriving
with a message still open is valid — the terminal implicitly closes it.
## Fix
`finalizeRunEvents` (in `@copilotkit/shared`) now returns early and
appends **nothing** when the stream already contains a terminal event
(`RUN_FINISHED` or `RUN_ERROR`). The abrupt-end path (no terminal →
close open streams + synthesize a terminal, in the correct order) is
unchanged. No API/signature change; the in-memory, intelligence, and
sqlite runners all inherit the fix.
## Testing
RED→GREEN verified — each new/updated assertion was confirmed to fail
against the pre-fix code:
- **`finalize-events.test.ts`** — terminal-present appends nothing
(parametrized over `RUN_FINISHED` and `RUN_ERROR`) + a named #5812 case.
- **`in-memory-runner.test.ts`** — end-to-end mid-stream-stop
regression: a fake `HttpAgent`-style agent is stopped between
`TEXT_MESSAGE_START` and `TEXT_MESSAGE_END`; asserts no events follow
`RUN_ERROR` **and** that the collected stream passes `verifyEvents`
(before the fix this threw the exact browser error).
- **`intelligence-runner.test.ts`** — corrected a pre-existing assertion
that had encoded the buggy post-terminal `TEXT_MESSAGE_END`.
Green: full `@copilotkit/runtime` suite, `@copilotkit/sqlite-runner`,
`@copilotkit/shared`, `check-types`, `oxlint` (0 errors), and build.
## Reviewer notes
- The behavior change is a single early-return in `finalize-events.ts`;
the `terminalEventMissing` guards simplify away because they're only
reachable when no terminal exists.
- Diff is +204/−55 across 4 files, the bulk of it tests.
## Problem
`pnpm dev` on the shell-dashboard fails to start the fold-importing
routes:
```
Module not found: Can't resolve './live-status.js'
Module not found: Can't resolve './format-ts.js'
Module not found: Can't resolve './staleness.js'
GET / 500
```
The dashboard re-exports the shared **cell-model fold** from the harness
(`showcase/harness/src/shared/cell-model/*.ts`). Those fold files are
authored for the harness's pure-Node-ESM runtime, so their internal
relative imports carry explicit `.js` extensions (e.g. `import {
formatTs } from "./format-ts.js"`) even though they exist on disk as
`.ts`. `export *` does not rewrite those internal edges.
`next build` (webpack) already resolves this via the
`resolve.extensionAlias` in `next.config.ts` (added in #5955), which
tells webpack to try the TS sources for a `.js` specifier. But the `dev`
script forced Turbopack (`next dev --turbopack`), and **Turbopack has no
`resolve.extensionAlias` parity** ([Next
#82945](https://github.com/vercel/next.js/issues/82945)) — so dev
couldn't resolve the fold.
## Fix
Drop `--turbopack` from the `dev` script. On Next 15.5.x, `--turbopack`
is an explicit opt-in flag (there is no `--webpack` opt-out); plain
`next dev` runs **webpack**, which honours the existing
`extensionAlias`.
```diff
- "dev": "next dev --turbopack --port 3002",
+ "dev": "next dev --port 3002",
```
`build` is unchanged (`next build` = webpack). A note in
`next.config.ts` explains why dev uses webpack.
**Tradeoff:** dev loses Turbopack's faster HMR and falls back to
webpack-speed dev until Turbopack ships `extensionAlias` parity
(#82945), at which point dev can switch back.
## Red / Green (empirical, this branch)
**Empirical proof of which bundler each command runs** (Next 15.5.15
startup banner):
- `next dev --turbopack` → `▲ Next.js 15.5.15 (Turbopack)`
- `next dev` → `▲ Next.js 15.5.15` (no "(Turbopack)" = webpack)
**RED** — `next dev --turbopack`, `GET /`:
```
▲ Next.js 15.5.15 (Turbopack)
Module not found: Can't resolve './format-ts.js'
Module not found: Can't resolve './live-status.js'
Module not found: Can't resolve './staleness.js'
GET / 500 in 3691ms
```
**GREEN** — `next dev` (webpack), `GET /`:
```
▲ Next.js 15.5.15
✓ Ready in 1124ms
✓ Compiled / in 1948ms (704 modules)
GET / 200 in 2844ms
```
No `Can't resolve`.
**No regression** — `next build` (webpack) still resolves the fold:
```
Creating an optimized production build ...
✓ Compiled successfully in 3.1s
```
(The fold compiles clean under webpack via the unchanged
`extensionAlias`.)
## Deploy path is UNAFFECTED
The `dev` script is never in the build or runtime path.
`showcase/shell-dashboard/Dockerfile` builds with `npx next build`
(webpack) and serves with `npx next start`. This change touches only
local `pnpm dev` — the built/deployed dashboard image is byte-for-byte
identical.
Turbopack has no resolve.extensionAlias parity (Next #82945), so
'next dev --turbopack' can't resolve the shared cell-model fold's
.js->.ts specifiers and fails with Can't resolve './live-status.js'.
The extensionAlias in next.config.ts (added in #5955) is honoured by
webpack, which next build already uses. Drop --turbopack from the dev
script so dev runs on webpack too and resolves the fold.
Deploy path unaffected: the Dockerfile builds with 'next build' (webpack)
and serves with 'next start' -- the dev script is never in the build or
runtime path.
## Two silent-failure gaps in the showcase build/deploy/notify pipeline
These are **pre-existing** silent-failure holes surfaced in code review
(not
caused by any recent PR). This PR fixes the two load-bearing ones.
### 1. Green-but-zero-redeploy (silent "we thought we shipped but
didn't")
The `redeploy-staging` job computes the redeploy set as the intersection
of the
build matrix and the build-success set. This job **only runs when
`aggregate-build-results.outputs.any_success == 'true'`** (job-level
`if:`
guard). So if that intersection comes back **EMPTY**, it does NOT mean
"nothing
to deploy" — it means at least one slot built successfully yet none of
those
successes maps back to a matrix `dispatch_name`. That's a
`dispatch_name`↔
`service` contract skew (the aggregator's `service` values and the
matrix's
`dispatch_name` values drifted apart).
The old code emitted `services=` (empty) and exited 0 → the build went
**GREEN
while redeploying NOTHING**, silently.
**Fix:** on an empty intersection in this any_success-guaranteed step,
fail loud
(`::error::` + `exit 1`) with a diagnostic naming both sides of the
skew.
The legitimate "nothing changed / nothing succeeded" no-op paths are
guarded at
the **job level** (`has_changes=='true' && any_success=='true'`), so the
fixed
step never runs there — no false-red.
### 2. Starter build failures had no alert surface (invisible failures)
The `notify` job's `needs` (and its `if: failure()`) omitted
`detect-starter-changes` and `build-starters`, and `build-starters`
wrote no
per-slot build-result artifact. So a **failed starter image build
produced NO
Slack alert and NO PR comment** — it shipped silently.
**Fix:**
- Added `detect-starter-changes` + `build-starters` to `notify.needs` so
`if: failure()` sees a starter build failure → Slack alert + PR comment.
- Gave `build-starters` a per-slot build-result artifact **mirroring the
main
`build` matrix** (same `{service,status}` shape, `cancelled→skipped`
normalization, `if: always()`, `if-no-files-found: error`), using a
**distinct `starter-build-result-*` prefix** so it never matches the
aggregator's `build-result-*` download pattern (starters must not
pollute the
showcase redeploy set keyed by `dispatch_name`).
### Red / Green
**Finding #1** — extracted the step's shell/jq logic and drove it with
synthetic
inputs:
RED (pre-fix), any_success=true + empty intersection:
```
No services in matrix ∩ success-set — skipping redeploy.
Computed services CSV (matrix ∩ build-success):
EXIT=0 # $GITHUB_OUTPUT: services= -> silent pass, redeploys NOTHING
```
GREEN (post-fix), same inputs:
```
::error::Build succeeded (any_success=true) but matrix ∩ success-set is EMPTY — dispatch_name/service contract skew; nothing would be redeployed.
Successful build service values: ["shell-RENAMED","mastra-RENAMED"]
Scheduled matrix dispatch_name values: ["shell","mastra"]
EXIT=1 # fails loud
```
No-regression: non-empty intersection → `EXIT=0 ; services=shell`. The
nothing-changed/nothing-succeeded paths are skipped at the job level
(never
reach the step) → no false-red.
**Finding #2** — modeled `if: failure()` (fires iff any `needs` job
result is
`failure`):
```
BEFORE (starters NOT in needs), starter=failure -> notify fires = False (INVISIBLE, the bug)
AFTER (starters IN needs), starter=failure -> notify fires = True (FIXED)
AFTER no-regression, starters=skipped, all green -> notify fires = False (quiet)
```
### Validation
- `python3 yaml.safe_load` parses OK.
- `actionlint`: only pre-existing findings remain (matrix jq SC2086 +
the known
`depot-ubuntu-24.04-4` runner-label warning); no new errors in edited
regions.
- `yamllint`: only pre-existing line-length/document-start/truthy
warnings.
### Scope
Touches **only** `.github/workflows/showcase_build.yml`, and only these
two
concerns. Does NOT touch the `shell_dashboard` paths-filter region (PR
#5955's
domain), nor the other backlog debt (false-root-cause comment,
double-alert,
check-lockfile guard). Self-contained; not stacked on #5955.
Two pre-existing silent-failure gaps in the showcase build/deploy/notify
pipeline (surfaced in code review):
1. Green-but-zero-redeploy: the redeploy-staging job computes the redeploy
set as (build matrix ∩ build-success). This job only runs when
any_success=='true', so an EMPTY intersection means builds succeeded but
none maps to a matrix dispatch_name — a dispatch_name/service contract
skew. The old code emitted an empty services= and exited 0, going GREEN
while redeploying nothing. Now it fails loud with a diagnostic naming both
sides of the skew. The legitimate nothing-changed/nothing-succeeded no-ops
stay guarded at the job level, so they are unaffected.
2. Starter-failure-invisible: the notify job's needs omitted build-starters,
so a failed starter image build produced no Slack alert and no PR comment.
Added detect-starter-changes + build-starters to notify.needs, and gave
build-starters a per-slot build-result artifact mirroring the main build
matrix (distinct starter-build-result-* prefix so it never pollutes the
showcase aggregator's build-result-* set).
## What broke
PR #5952 (`9a8cf615`) added explicit `.js` extensions to the relative
imports inside the harness's shared cell-model fold
(`showcase/harness/src/shared/cell-model/{cell-model,live-status,staleness}.ts`).
Those `.js` extensions are **REQUIRED** for the harness's pure-Node-ESM
runtime and are **correct** — this PR does not revert them.
The problem: the dashboard re-exports that fold via re-export shims
(`showcase/shell-dashboard/src/lib/{cell-model,live-status,staleness,format-ts}.ts`,
each `export * from "../../../harness/src/shared/cell-model/*"`), which
pulls the fold **into the dashboard's `next build`**. `export *` does
not rewrite the fold's *internal* `.js` edges, and the dashboard's
`next.config.ts` was empty (no `extensionAlias`), so webpack resolved
`./live-status.js` **literally**, found only the `.ts` source, and
failed:
```
../harness/src/shared/cell-model/cell-model.ts
Module not found: Can't resolve './live-status.js'
Module not found: Can't resolve './staleness.js'
../harness/src/shared/cell-model/live-status.ts
Module not found: Can't resolve './format-ts.js'
> Build failed because of webpack errors
```
First-red at `9a8cf615`; reproduced in CI run `29306712559`
(shell-dashboard build job).
## The fix (two parts, one coherent subject)
**1. Resolution** — `showcase/shell-dashboard/next.config.ts`: add a
webpack `resolve.extensionAlias` so `.js`/`.mjs` specifiers resolve to
`.ts`/`.tsx`/`.mts` sources. This is the standard bundler complement to
TypeScript NodeNext's `.js`-import convention, and it applies to the
`next build` (webpack) path CI uses. A shim-only fix does **not** work —
`export *` doesn't intercept the fold's internal `.js` edges; the alias
in the dashboard build is the correct layer. The harness fold `.js`
imports are **left untouched**.
**2. CI gap** — `.github/workflows/showcase_build.yml`: the dashboard
build didn't run on #5952 because the build matrix is path-filtered and
#5952 only touched `showcase/harness/**`, which selects
`showcase_harness` but **not** `shell_dashboard`. Added
`showcase/harness/src/shared/**` to the `shell_dashboard`
`dorny/paths-filter` set. Now any change to the shared fold the
dashboard compiles in also selects the dashboard build — a fold change
can never again ship an unbuilt dashboard. (Verified only
`shell-dashboard` consumes this fold, so the gate is scoped precisely.)
## Local red-green proof
**RED** (latest main, before the `next.config.ts` fix) — from
`showcase/shell-dashboard`, `next build`:
```
../harness/src/shared/cell-model/cell-model.ts
Module not found: Can't resolve './live-status.js'
Module not found: Can't resolve './staleness.js'
../harness/src/shared/cell-model/live-status.ts
Module not found: Can't resolve './format-ts.js'
Module not found: Can't resolve './staleness.js'
> Build failed because of webpack errors
```
(4 fold-resolve errors.)
**GREEN** (after the `extensionAlias` fix) — same `next build`:
```
(0 fold-resolve errors — the fold resolves)
```
The only remaining `Module not found` errors are
`@/data/{catalog,registry,docs-status}.json`, which are generated by the
dashboard's `prebuild` scripts (`generate-registry.ts` /
`probe-docs.ts`) that were skipped in the local repro. CI's Docker build
runs `prebuild` first, so those files exist there — unrelated to this
fix.
## CI-gap trace
`dorny/paths-filter` emits `changes` as the JSON array of filter keys
whose patterns matched. A change to
`showcase/harness/src/shared/cell-model/live-status.ts` now matches both
`showcase_harness` (via `showcase/harness/**`) **and** `shell_dashboard`
(via the new `showcase/harness/src/shared/**`), so the matrix `select`
(`$dispatch == "" and ($changes | index($fk) != null)`) includes the
`shell-dashboard` slot. Gap closed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
PR #5952 (9a8cf615) added explicit `.js` extensions to the relative imports
inside the harness's shared cell-model fold
(showcase/harness/src/shared/cell-model/{cell-model,live-status,staleness}.ts)
— REQUIRED for the harness's pure-Node-ESM runtime and correct as-is.
But the dashboard re-exports that fold via shims
(showcase/shell-dashboard/src/lib/{cell-model,live-status,staleness,format-ts}.ts
`export * from "../../../harness/src/shared/cell-model/*"`), pulling the fold
into the dashboard's `next build`. `export *` does not rewrite the fold's
INTERNAL `.js` edges, and the dashboard's empty next.config.ts had no
extensionAlias, so webpack resolved `./live-status.js` literally, found only
the `.ts` source, and failed:
Module not found: Can't resolve './live-status.js'
Module not found: Can't resolve './staleness.js'
Module not found: Can't resolve './format-ts.js'
> Build failed because of webpack errors
Two-part fix (one coherent subject):
1. Resolution: add `webpack.resolve.extensionAlias` to
showcase/shell-dashboard/next.config.ts so `.js`/`.mjs` specifiers resolve
to `.ts`/`.tsx`/`.mts` sources — the bundler complement to TS NodeNext's
`.js`-import convention. Covers the `next build` (webpack) path CI uses.
The harness fold's `.js` imports are left untouched (they are correct).
2. CI gap: the dashboard build did not run on #5952 because the build matrix
is path-filtered and #5952 only touched `showcase/harness/**`, which
selects `showcase_harness` but not `shell_dashboard`. Add
`showcase/harness/src/shared/**` to the `shell_dashboard` paths-filter so
any change to the shared fold the dashboard compiles in also selects the
dashboard build — a fold change can never again ship an unbuilt dashboard.
Local red-green proof:
- RED (main, before fix): `next build` in showcase/shell-dashboard emitted the
4 fold-resolve errors above.
- GREEN (after extensionAlias): same build → 0 fold-resolve errors; the fold
resolves. Remaining `@/data/*.json` errors are the prebuild-generated files
(generate-registry/probe-docs) skipped in the local repro, produced in CI's
Docker build — unrelated to this fix.
## Summary
Routes the ~20 showcase demo backends to **aimock** (the record/replay
LLM proxy) over Railway **private networking** (`*.railway.internal`)
instead of aimock's **public** `*.up.railway.app` host.
Railway bills traffic to a public domain as **egress even
intra-project**, while `*.railway.internal` private networking is
**free** and **env-scoped**. The 240-concurrent-browser harness fleet
drives every demo continuously, so every LLM SSE stream from aimock back
to a demo backend is currently billed egress.
- aimock ≈ **89% of showcase egress**, ≈ **92% of the 13TB→78TB/mo
increase**.
- Estimated impact: avoids the ≈ **$602/mo → $3,856/mo** growth on the
aimock path.
## Change (config-only, reversible; SSOT-driven)
1. **SSOT** (`showcase/scripts/railway-envs.ts`): add an env-scoped
`internalDomain: "showcase-aimock.railway.internal"` to the aimock entry
in **both** envs. The public `domain` is **kept** (health probes /
external reachability).
2. **Emitter** (`showcase/scripts/emit-railway-envs-json.ts`): emit
`internalDomains` (additive, after `domains`) into the generated JSON.
Every non-aimock service keeps its frozen shape.
3. **Generated JSON** regenerated (oxfmt-canonical; 4-line additive
diff, only the aimock entry).
4. **Promote preflight** (`showcase/bin/railway`): `ssot_target_host`
now **prefers** the private `internalDomains[env]` over the public
`domains[env]`, so the Stage-2 (U5) serviceRef assertion requires demo
backends' `OPENAI_BASE_URL`/etc. to point at the private host.
Non-aimock targets (no `internalDomains`) fall back to their public host
unchanged.
5. **Harness** wiring probe needs **no code change** (it matches on
hostname); added a discriminating test pair + updated the drift-alert
Fix text to the private host.
**Target:** aimock binds `0.0.0.0:4010` (per
`showcase/aimock/RAILWAY.md`); demo backends resolve to
`http://showcase-aimock.railway.internal:4010`.
Deployed env vars, both envs (before → after):
| key | before (public, billed egress) | after (private, free) |
|---|---|---|
| `OPENAI_BASE_URL` | `https://<aimock>.up.railway.app/v1` |
`http://showcase-aimock.railway.internal:4010/v1` |
| `ANTHROPIC_BASE_URL` | `https://<aimock>.up.railway.app` |
`http://showcase-aimock.railway.internal:4010` |
| `GOOGLE_GEMINI_BASE_URL` | `https://<aimock>.up.railway.app` |
`http://showcase-aimock.railway.internal:4010` |
| `AIMOCK_URL` | `https://<aimock>.up.railway.app` |
`http://showcase-aimock.railway.internal:4010` |
`<aimock>` = `aimock-staging` (staging) / `showcase-aimock-production`
(prod). `railway.internal` is env-scoped, so staging demos reach the
staging aimock and prod demos reach prod aimock automatically — the same
private DNS name in both envs.
---
## Red-green proof (verbatim)
### RED — live staging today (billed public egress)
Deployed `showcase-langgraph-fastapi` (staging, service `06cccb5c-…`)
via Railway `variables(...)` GraphQL:
```
OPENAI_BASE_URL = https://aimock-staging.up.railway.app/v1
ANTHROPIC_BASE_URL = https://aimock-staging.up.railway.app
GOOGLE_GEMINI_BASE_URL = https://aimock-staging.up.railway.app
AIMOCK_URL = https://aimock-staging.up.railway.app
```
Pre-fix generated JSON aimock entry — **no** `internalDomains`:
```json
{ "domains": { "staging": "aimock-staging.up.railway.app",
"prod": "showcase-aimock-production.up.railway.app" },
"internalDomains": "ABSENT" }
```
### RED — Ruby U5 serviceref resolver, with the resolver reverted to
public-only
The three new U5 tests FAIL when `ssot_target_host` returns the public
host:
```
7 runs, 15 assertions, 3 failures
1) test_serviceref_prod_pointing_at_public_aimock_host_refuses:
expected REFUSE for prod serviceRef on the public egress host, got []
2) test_serviceref_prod_pointing_at_private_aimock_passes:
prod private aimock ref must not REFUSE, got ["REFUSE: §5.2 (showcase-ag2): prod
OPENAI_BASE_URL="http://showcase-aimock.railway.internal:4010/v1" does NOT point at
aimock's env-LOCAL prod host "showcase-aimock-production.up.railway.app" ..."]
3) test_ssot_target_host_prefers_internal_over_public:
expected "showcase-aimock.railway.internal",
actual "showcase-aimock-production.up.railway.app"
```
### GREEN — after the fix
Post-fix generated JSON aimock entry:
```json
{ "domains": { "staging": "aimock-staging.up.railway.app",
"prod": "showcase-aimock-production.up.railway.app" },
"internalDomains": { "staging": "showcase-aimock.railway.internal",
"prod": "showcase-aimock.railway.internal" } }
```
Ruby U5 serviceref tests (fixed resolver — prefers `internalDomains`):
```
7 runs, 20 assertions, 0 failures, 0 errors, 0 skips
```
Full Ruby spec suite:
```
184 runs, 715 assertions, 0 failures, 0 errors, 0 skips
```
Harness aimock-wiring probe (hostname-match; internal host with `:4010`
+ `/v1` → green, demo still on public host while harness on private →
red):
```
src/probes/aimock-wiring.test.ts 29 passed (was 27; +2 new: internal-host green, public-host drift red)
src/probes/drivers/aimock-wiring.test.ts 16 passed
src/rules/rule-loader.test.ts 61 passed (aimock-wiring-drift.yml parses after Fix-text update)
renderer + render-red-tick + orchestrator 157 passed (no alert-text snapshot broke)
```
Scripts test suite (emitter golden + everything): `2147 passed, 7
skipped` (one pre-existing `/tmp` lockfile flake in
`integration-smoke-registry.test.ts`, green on rerun after clearing the
stale lock). `emit --check` idempotent + oxfmt-canonical. Harness `tsc
--noEmit`: clean.
### GREEN — live infra confirmation
- aimock **staging** deployment status = `SUCCESS` (running), binds
`0.0.0.0:4010` — so `showcase-aimock.railway.internal:4010` resolves to
a live listener for any peer in the staging env.
- aimock serving LLM-shaped responses on `:4010`: `GET /health` → `200`;
`GET /v1/models` → `200` `{gpt-4o, gpt-4o-mini}`.
## What was vs wasn't live-validated
**Validated live:** the RED (deployed staging vars still on the public
egress host); aimock staging is deployed/running and serving on `:4010`;
the full unit/wiring/promote-preflight test surface passes with the new
internal-host values.
**NOT live-validated in-session:** the in-Railway-network DNS resolution
of `showcase-aimock.railway.internal:4010` from a peer service, and a
full staging deploy that flips the four keys + redeploys a demo backend.
Reason: the in-network vantage needs `railway ssh` (requires registering
a persistent account SSH key — a stateful, human-gated change I declined
to make unsupervised) or a staging deploy (the local Railway access
token was expired; the CLI refreshed it for read/GraphQL but a deploy is
a separate gated action). Railway private networking
(`*.railway.internal`) is a standard platform feature; the local
`docker-compose.local.yml` already runs the identical
`http://aimock:4010` internal-host pattern, and the wiring probe's
hostname match is exercised by the new tests. The staging deploy +
in-network curl is the first step of the rollout plan below and must be
run before prod.
## Irreducible egress remains
This does **not** zero showcase egress. Still billed: real browse users
hitting the public demo/shell domains; and aimock in **record mode**
proxying to real providers (the outbound prompt to
OpenAI/Anthropic/Google still bills).
## Rollout plan (reversible config change, staging-first, user-gated)
1. Land this branch (SSOT + generated JSON + assertions).
2. **Staging first:** set the four keys on staging demo backends +
`AIMOCK_URL` on the harness to
`http://showcase-aimock.railway.internal:4010` (`/v1` on
`OPENAI_BASE_URL`); redeploy one demo backend + aimock; from inside a
staging service curl
`http://showcase-aimock.railway.internal:4010/health` (expect 200) and
run a real demo LLM turn / aimock-wiring probe (expect green); confirm
the aimock egress path stops accruing
(`usage(measurements:[NETWORK_TX_GB])`).
3. **User-gated** promote to prod (staging→prod), same key flip.
4. **Rollback** = flip the keys back to the public host (no code revert
needed).
## Follow-ups (out of scope — do NOT bundle)
- Fleet right-sizing (240-concurrent-browser harness).
- `OPENAI_API_KEY` consolidation.
---
Draft — do not merge. Do not deploy to prod.
## What broke
The showcase control-plane harness crash-looped at boot on `origin/main`
HEAD, breaking the staging auto-deploy:
```
BOOT_ERR ERR_MODULE_NOT_FOUND Cannot find module '.../dist/shared/cell-model/live-status' imported from .../dist/shared/cell-model/cell-model.js
```
## Root cause
The harness ships as **pure Node ESM**:
- `showcase/harness/package.json` has `"type": "module"`
- build is `tsc -p tsconfig.build.json` with `moduleResolution:
"bundler"`, which **preserves extensionless import specifiers at emit**
(it does not rewrite `./live-status` → `./live-status.js`)
- the Docker `CMD` and the `start` script both run `node
dist/orchestrator.js`
Under pure Node ESM, relative import specifiers **must** carry the `.js`
extension. The harness already honors this convention everywhere —
`orchestrator.ts` has 79 relative imports, 79/79 ending in `.js`.
The relocated `showcase/harness/src/shared/cell-model/` fold broke the
convention: `cell-model.ts`, `live-status.ts`, `staleness.ts`, and the
equivalence fixtures/test imported their siblings extensionless
(`"./live-status"`, `"./staleness"`, etc). `tsc`, `vitest`, and `tsx`
all resolve extensionless specifiers fine, so it built green and passed
every test — then `node dist/orchestrator.js` threw
`ERR_MODULE_NOT_FOUND` at boot and crash-looped.
## The fix
Add the `.js` extension to every offending relative import in the
cell-model fold, matching the harness convention. **No tsconfig change**
(deliberately not switching to `nodenext`) — the minimal,
convention-matching fix is the extensions.
Files fixed (7 relative import specifiers across 5 files):
- `cell-model.ts` — 4 specifiers (`./live-status` ×3, `./staleness` ×2
counting the re-export)
- `live-status.ts` — `./format-ts`, `./staleness`
- `staleness.ts` — `./live-status`
- `cell-model.equivalence-fixtures.ts` — `./live-status` ×2,
`./cell-model`, `./staleness`
- `cell-model.equivalence.test.ts` — `./cell-model`,
`./cell-model.equivalence-fixtures`
The shell-dashboard re-export shims
(`showcase/shell-dashboard/src/lib/{cell-model,live-status,staleness,format-ts}.ts`)
were intentionally **left unchanged**: that package is a Next.js build
(not `node dist`) and uses extensionless relative imports as its own
convention, including reaching into harness `src`. Adding `.js` there
would break it.
## Local RED / GREEN proof
Same `node dist/orchestrator.js` boot probe, before and after the fix.
**RED** (unmodified branch code, after `pnpm --filter
@copilotkit/showcase-harness build`):
```
BOOT_ERR ERR_MODULE_NOT_FOUND Cannot find module '/…/showcase/harness/dist/shared/cell-model/live-status' imported from /…/showcase/harness/dist/shared/cell-model/cell-model.js
```
**GREEN** (after the fix, rebuilt):
```
BOOT_OK
```
The orchestrator module graph now loads fully — `ERR_MODULE_NOT_FOUND`
is gone.
## Docker boot result
Built the harness image locally from repo root (`docker build -f
showcase/harness/Dockerfile -t harness-esm-test .`) — build succeeded
(exit 0). Ran the container (`docker run harness-esm-test`, no env
supplied). It did **not** crash-loop on module resolution: the full ESM
module graph loaded and boot reached `bootFleet` in `orchestrator.js`,
then exited cleanly on the expected missing-env application error — **no
`ERR_MODULE_NOT_FOUND`**:
```
{"level":"error","msg":"showcase-harness.boot-failed","err":"HARNESS_ROLE must be set to one of: control-plane, worker (got: <unset>). ...","stack":"Error: HARNESS_ROLE must be set ...
at resolveFleetRoleConfig (file:///app/dist/fleet/role-config.js:72:15)
at bootFleet (file:///app/dist/orchestrator.js:3680:20)
at file:///app/dist/orchestrator.js:3762:5
at ModuleJob.run (node:internal/modules/esm/module_job:343:25)"}
```
Every `dist/` module resolved (note
`file:///app/dist/fleet/role-config.js` and
`file:///app/dist/orchestrator.js`); the crash was the intended
env-validation guard, exactly the "later runtime error is acceptable"
case. On pre-fix code this same container would have thrown
`ERR_MODULE_NOT_FOUND` at import time and crash-looped.
## CI regression guard
CI missed this because `tsc` (bundler resolution), `vitest`, and `tsx`
all resolve extensionless specifiers — **no existing CI step ever ran
the real `node dist` module graph** that the container boots.
Added a **harness ESM boot-smoke** step to the `Showcase: Validate` job
(already gated on `showcase/harness/**`, a required PR check): after
building the harness dist, it loads `dist/orchestrator.js` via `node
import()` and fails hard on `ERR_MODULE_NOT_FOUND`. A later runtime
error from missing env/PocketBase is expected and passes — only a
module-resolution failure reddens the build.
**Guard red-green** (the guard's own proof, run locally against both
code states):
- **pre-fix** (extensionless imports): `BOOT_FAIL: ERR_MODULE_NOT_FOUND
…/dist/shared/cell-model/live-status` → exit **1** ✅ caught
- **post-fix** (with `.js`): `BOOT_OK: orchestrator module graph loaded`
→ exit **0** ✅ passes
## Verification
- `pnpm typecheck` / `tsc --noEmit`: clean
- `cell-model.equivalence.test.ts`: 7/7 pass
- lefthook lint + commitlint: pass on both commits
Do not merge — merge is user-gated after code review.
The boot-smoke previously failed only when the thrown error carried a
module-RESOLUTION code (ERR_MODULE_NOT_FOUND + siblings, walked through
the cause chain / AggregateError members) and passed everything else.
That defaults-to-pass on module-EVALUATION crashes — a top-level throw,
an await-rejection, a bad named binding, or a SyntaxError — none of which
carry a resolution code, so a real boot-crashing regression of that class
would ship green.
The smoke runs `node -e "import('./dist/orchestrator.js')"` with
process.argv[1] UNSET, so bootFleet() (the env/PocketBase validation that
legitimately throws) never runs — only the module graph is linked and
evaluated. A clean build therefore loads with no thrown error, so ANY
error thrown by import() here is a boot regression and must fail the gate.
Now: any rejection -> BOOT_FAIL / exit 1 (resolution AND
evaluation/link/binding/syntax/top-level-throw). Successful load ->
BOOT_OK / exit 0. The collectErrorCodes cause/AggregateError walk is
retained ONLY to label the failure ("module-resolution failure" vs
"boot/evaluation failure") — both exit 1, richer diagnostics preserved.
Kept process.exit(0) on success, the timeout 120s wrapper, and
timeout-minutes: 5 (a hang still fails).
No-false-red proof: built the real harness dist and ran the strict guard
against the real dist/orchestrator.js under node -e (argv[1] unset) —
BOOT_OK, exit 0, ~0.28s, no hang, confirming a clean graph loads without
throwing and the strict guard does not false-red real CI.
The boot-smoke gate classified pass/fail using only the top-level `e.code`.
A module-resolution error that arrives WRAPPED — nested in `e.cause`
(possibly a chain), bundled inside an `AggregateError` (`e.errors[]`), or
rethrown without preserving `.code` at the top — showed no code to the
`MODULE_RESOLUTION_CODES.has(e.code)` check and was misclassified as
BOOT_OK, defeating the gate.
Add a `collectErrorCodes` helper that gathers every code reachable from
the thrown error: the error itself, its cause chain (recursively), and any
AggregateError members (recursively), with a depth cap to bound cause
cycles. If ANY collected code is a module-resolution code -> BOOT_FAIL /
exit 1. Purely additive to the FAIL set: direct top-level codes still
redden, and a benign non-resolution runtime error (e.g. the
`HARNESS_ROLE must be set` guard, which carries no such code anywhere)
still passes as BOOT_OK / exit 0. The `timeout 120s` wrapper,
`timeout-minutes: 5`, and success/expected-error `process.exit(0)` are
unchanged.
Local red-green (classifier extracted to a temp file, driven against
synthetic errors):
- RED (top-level-only): wrapped cause -> BOOT_OK exit 0 (swallowed);
AggregateError member -> BOOT_OK exit 0 (swallowed).
- GREEN (hardened): wrapped -> exit 1; aggregate -> exit 1; direct
ERR_MODULE_NOT_FOUND -> still exit 1; benign ERR_INVALID_ARG_TYPE and
HARNESS_ROLE error -> BOOT_OK exit 0; real built dist/orchestrator.js ->
BOOT_OK exit 0 in <200ms (prompt exit, no hang).
Benjamin Taylor