Producer-side foundation for fleet item 3b — two behavior-preserving
generalizations, byte-identical for the d6 case:
1. Generalize the d6 service enumerator into a parameterized
`createServiceEnumerator(params)` carrying the service-set filter, the
driverKind, and the probeKey prefix builder. `createD6ServiceEnumerator`
is now a thin wrapper passing the d6 params (e2e_d6 kind, d6:<slug> keys,
D6_DISCOVERY_FILTER), so d6 behavior is unchanged.
2. Generalize createControlPlane to accept an array of
{ scheduleId, cron, producer } entries and register each on the scheduler.
The single-d6 case degenerates to a one-element array on
fleet-job-producer @ 40 * * * *, preserving current behavior.
Out of scope (gated on in-flight PRs): orchestrator runControlPlane wiring
to pass multiple schedules (conflicts with #5284), the e2e_smoke/demos/deep
families (Phase 2), and worker-loop/payload-mapper/probe-loader.
CR fixes for the fleet worker driverKind→driver registry (PR #5283):
- Fix default (self-contained) worker boot: build the default d6 as a
registry entry { driver, payloadToInput, aggregateSlugKey } instead of a
bare driver with no mapper, so startWorkerLoop's construction guard no
longer throws "Fleet worker has no drivers".
- Thread aggregate-key derivation through DriverRegistryEntry
(aggregateSlugKey?), defaulting to d6:<slug> so the d6 cell-capture filter
stays byte-identical while non-d6 kinds can supply their own scheme.
- Add construction-time fail-loud assert that each registry entry's factory
kind matches its key; raise unknown-driver-kind log to error to match the
sibling protocol-violation logs.
- Extract shared buildPooledBrowserDrivers consumed by both
registerAllProbeDrivers and the worker registry; collapse no-op per-kind
payload-mapper aliases to the single createPayloadToInput.
- Introduce typed DriverKind union (contained to worker/payload-mapper);
keep contracts.driverKind a string wire boundary.
- Tests: new fleet/orchestrator.test.ts (default-boot equivalence), driver
construction-guard + custom aggregateSlugKey coverage, and lock-step +
registry-wiring pins (factory kind == constant).
Generalize the fleet worker from a single hardwired d6 driver into a
driver REGISTRY keyed by payload.driverKind, so one worker can host all
four browser driver families (e2e_d6, e2e_deep, e2e_demos, e2e_smoke).
- worker-loop: accept a `drivers: Map<kind, { driver, payloadToInput }>`
and dispatch each claimed job by `payload.driverKind`. Unknown kind →
terminal `worker-protocol-violation` (same shape as an unmappable
payload), never a worker crash. Legacy single `driver`+`payloadToInput`
pair retained as a fallback for back-compat. Fail-loud at construction
when neither a registry nor the legacy pair is supplied.
- payload-mapper: generalize createD6PayloadToInput into a shared
createPayloadToInput plus per-kind aliases and the four driver-kind
constants (the input re-hydration is identical across families; each
driver's own zod schema is the validation gate).
- orchestrator runWorker: build all four pooled drivers on the shared
BrowserPool and register them by kind, lifted from the legacy
registerAllProbeDrivers pooled construction.
d6 routing is unchanged (equivalence gate). Red-green tests cover
routing e2e_smoke/e2e_deep/e2e_d6 to their drivers and unknown-kind →
protocol-violation. Full harness suite green (2085 tests); tsc clean
except the known pre-existing toReversed error.
The pool-fleet cutover manually created the staging-only
`showcase-harness-worker` Railway service (HARNESS_ROLE=worker, 2
replicas) and flipped the existing `harness` service to
HARNESS_ROLE=control-plane. The new service was untracked in the SSOT,
so verify-railway-image-refs.ts failed the "Showcase: Build & Push"
workflow on every push to main (1 untracked Railway service), which
skipped the harness `build` job and blocked harness image rebuilds.
Add the worker to SERVICES as a staging-only, domain-less queue worker:
- ciBuilt:false — it runs the SAME `showcase-harness` image the existing
harness build slot produces; there is no separate worker build.
- gateIgnore:true / gateValidated:false — no prod instance and no public
domain, so it does not fit the symmetric dual-env shape the image-ref
gate validates. gateIgnore clears the "untracked Railway service"
failure (any SSOT entry counts as known) without tripping a false
"missing from prod" failure.
- repoNameOverride → showcase-harness so the image-ref shape resolves.
- probe disabled in both envs (no externally-reachable health endpoint).
Regenerate railway-envs.generated.json and update the SSOT-count and
gate-coverage test invariants (27→28 services; worker is the sole
intentional gateIgnore/gateValidated:false entry).
The harness starter-smoke CHAT rung 503'd on staging because its raw fetch()
POST sent only Content-Type — omitting X-AIMock-Context. The scoped
per-integration "Hello" fixtures in showcase/aimock/d4/<integration>/chat.json
only match (under aimock strict mode) when the request carries
X-AIMock-Context:<context>, and that context token IS the dashboard column
slug (verified against each integration's
showcase/integrations/<col>/playwright.config.ts extraHTTPHeaders and the
fixture's match.context). The local browser e2e passes only because Playwright
injects the header, forwarded by HeaderForwardingMiddleware to aimock.
Send X-AIMock-Context: <columnSlug> on the chat POST only (matching the
browser; the GET rungs hit the runtime /info route, not aimock). columnSlug is
already resolved via starterToColumnSlug before the chat rung runs, so the skew
cases (langgraph-js->langgraph-typescript, adk->google-adk,
strands-python->strands, ms-agent-framework-*->ms-agent-*) get the correct
context for free.
Proven RED->GREEN against live staging aimock
(https://aimock-staging.up.railway.app): without the header -> 503
no_fixture_match; with the mapped column-slug context -> 200 matching the
scoped fixture.
Framework landing heroes (e.g. /langgraph-typescript) now show the same
two StartCommandCards as the home hero, and the "new project" command
pre-fills the framework via the CLI's --framework flag (e.g.
--framework langgraph-js). The mapping from docs slug to CLI framework is
verified against the CLI's AGENT_FRAMEWORKS enum; slugs with no 1:1
template fall back to a bare `npx copilotkit@latest create`. Frameworks
with bespoke setup (a2a git clone, ms-agent-dotnet) keep their own single
copy-command chip. Drops the old "Start the quickstart" button and the
legacy `npx copilotkit@latest init` chip.
Replace the home landing hero's Quickstart dropdown + command-copy chip
with a two-card <HeroStartCommands> block, framed by situation:
- Start a new project -> npx copilotkit@latest create
- Add to an existing project -> npx copilotkit@latest skills onboard
Both cards are equal weight with no pre-selected default. Each command
row copies on click with an aria-live announcement and a clipboard-blocked
fallback. Commands pin @latest so npx never reuses a stale cached CLI.
Removes the now-unused hero-quickstart-dropdown and hero-command-copy
components.
## Summary
Rewrites the `starter-smoke` probe to the **v2 path-based multi-route**
runtime protocol the deployed starters actually speak — proven via a
local build+curl gate against `examples/integrations/crewai-crews` at
1.59.5 (the runtime mounts `createCopilotEndpoint` in default
`mode:"multi-route"`, identical at 1.59.3 and 1.59.5).
- **agent** → `GET /api/copilotkit/info` requiring 200 + `version` (was:
"any non-404", which a health-JSON/HTML error page wrongly passed)
- **chat** → `POST /api/copilotkit/agent/<id>/run` (`Accept:
text/event-stream`), asserting ≥1 `TEXT_MESSAGE_CONTENT` delta **+**
terminal `RUN_FINISHED` **+** no `RUN_ERROR` (was: "non-empty body")
- **health** → repointed to `GET /api/copilotkit/info` (the starters
serve no `/api/health` route)
- **interaction** → `GET /` (unchanged)
- **drop the trailing slash** on runtime POSTs — `POST /api/copilotkit/`
308-redirects and drops the POST → 404, the core cause of the red rungs
Plus a **unified abort/body-read error classification** (one
`abortOutcome()` helper keyed on the local `externallyAborted` flag): a
self-timeout or a non-abort error → `transport-error` (soft), only a
genuine external abort → `aborted`; a cut-short body read softens to
`transport-error` instead of hard `smoke-failed`; the level loop
short-circuits on external abort. Reviewed across 3 CR rounds
(path-based rewrite + 2 abort-classification fix rounds + confirmation).
> **NOTE:** the content-level chat rung will correctly read **RED**
against the deployed starters until the Python-side `ag-ui-crewai`
`RUN_ERROR` (a real-LLM, in-process agent failure — *not* a probe bug)
is fixed. That is the truthful signal.
## Deferred follow-ups (tracked, not in scope)
- **Verify the alert/staleness consumer branches on `errorClass`, not
`state`** — else the soft/hard (`transport-error` vs `smoke-failed`)
split is cosmetic at the dashboard layer
- `res.text()` resolve-partial (truncated-but-resolved body) precision
edge → could mis-class as `smoke-failed`
- `redirect:"follow"` POST→GET edge; health/interaction success-path
body drain; health/agent share `/info` (correlated signals)
- `CLASS_RANK` ranks `aborted` lowest (design call); unmapped-starter
`columnSlug` invariant + prototype-key collision
- test-coverage gaps: `resolveTimeoutMs` env parsing,
`deriveStarterSlug` fallback, real-timer abort path
## Test plan
- [x] Harness unit suite — 1829 tests pass (105 files); starter-smoke
35/35; red-green verified for protocol + all abort/body-read fixes
- [x] Typecheck (no new errors), oxlint (0 errors), oxfmt, harness build
- [ ] Post-merge: redeploy harness, re-probe; chat rung greens once the
Python-side RUN_ERROR is fixed (separate workstream)
The chat rung hardcoded `/api/copilotkit/agent/default/run`, which 404s for
mastra — it registers dynamic non-`default` agent keys via
`MastraAgent.getLocalAgents` rather than `agents:{default}`. Resolve the
agent id per-starter from the first key of the `/info` `agents` map (the same
info response the health + agent rungs already fetch), falling back to
`default` only when that map is empty/unreadable, and surface the resolved id
on the chat row signal for drilldown.
Rewrite the starter-smoke probe to the v2 path-based multi-route runtime
protocol the deployed starters actually speak (proven via a local
build+curl gate against examples/integrations/crewai-crews at 1.59.5):
- agent rung: GET /api/copilotkit/info -> 200 + version (was: any non-404)
- chat rung: POST /api/copilotkit/agent/<id>/run (Accept: text/event-stream)
asserting >=1 TEXT_MESSAGE_CONTENT delta + terminal RUN_FINISHED + no
RUN_ERROR (was: non-empty body)
- health rung: repointed to GET /api/copilotkit/info (the starters serve
no /api/health route)
- interaction rung: GET / (unchanged)
- drop trailing slash on all runtime POSTs (the old /api/copilotkit/ 308s
and drops the POST -> 404, the core cause of the red rung)
Plus a unified abort/body-read error classification (one abortOutcome()
helper keyed on the local externallyAborted flag): a self-timeout or a
non-abort error is transport-error (soft), only a genuine external abort
is 'aborted'; a cut-short body read softens to transport-error instead of
hard smoke-failed; the level loop short-circuits on external abort.
NOTE: the content-level chat rung will correctly read RED against the
deployed starters until the Python-side ag-ui-crewai RUN_ERROR (a
real-LLM, in-process agent failure, NOT a probe bug) is fixed.
Deferred follow-ups: verify the alert/staleness consumer branches on
errorClass (not state) so the soft/hard split is actually consumed;
res.text() resolve-partial precision edge; redirect:follow POST edge;
health/interaction body-drain; health/agent shared /info; CLASS_RANK
aborted-lowest; unmapped-starter columnSlug/prototype-key; test-coverage
gaps (resolveTimeoutMs, deriveStarterSlug, real-timer abort).
## Summary
Follow-up to #5256 (the concurrent-paged initial-fetch perf fix). Trims
the live-status payload and hardens the realtime/SSE path of the
showcase Coverage dashboard.
- **Trim initial projection** — drop the heavy `signal` field from the
initial PocketBase fetch via a compiler-guarded `STATUS_LIST_FIELDS`
projection + `skipTotal` + length-based bounded-concurrency pagination.
Consumers adapted: `CellDrilldown` lazy-loads `signal` on demand
(genuine failures only), `DiscoveryAuthBanner` self-fetches detail
(identity-tagged cache), legacy `LiveBadge` degrades gracefully when
`signal` is absent.
- **Harden the realtime path** — `flushPending` teardown guards,
keyless-delete `id→key` buffer collapse (per-row last-write-wins, pure
ref+effect mirror), SSE-delta signal-presence upsert, and a `degraded`
flapping flag (clears on heartbeat success, resets on terminal error).
- **Grid/UX** — a `stale` (muted, non-authoritative) tally state
distinct from the no-rows loading state, an amber-chip/green-D5
dimension-classification fix, and `degraded` surfaced in the header
`LiveIndicator` (offline outranks degraded).
## Test plan
- [x] `tsc --noEmit` clean
- [x] `vitest run` — 879 passed / 1 skipped (56 files); +63 new
behavioral tests across the changed areas (real-SDK socket tests for
fetch/autocancel/lazy-signal, fake-timer tests for the flapping detector
and reconnect chain, exhaustive resolveCell truth-tables)
- [x] `next build` — compiled successfully, 5/5 static pages
- [x] oxfmt + oxlint clean
- [ ] CI green
- [ ] Visual verification on staging after deploy
## Known follow-ups (deferred — pre-existing / out-of-subject, not
introduced by this PR)
These were surfaced during review, verified pre-existing on `main` (or
unreachable in the shipping config), and intentionally not fixed here to
keep the PR scoped:
- Column-header tally counts deprecated features hidden from the grid
body (`computeColumnTally` iterates the full `features` list while the
body renders `visibleFeatures`) — pre-existing, header/body over-count
identical before and after. Strong candidate for a dedicated follow-up.
- `resolveCell` rollup does not suppress amber/degraded cells when
`connection === "error"` (only green is suppressed) — pre-existing
precedence.
- `formatTooltip` offline branch renders "degraded since
<transitioned_at>" for a staleness-downgraded green row.
- `LiveBadge` arms the `useLastTransition` fetch for
staleness-downgraded amber badges (no `isGenuineFailure` filter, unlike
the drilldown).
- `CellStatus` passes the raw `featureId` as the CV-badge transition
key, so mapped D5 families query a non-existent history row.
- Cosmetic: `extractSignalFields` renders `{}` for an empty-object error
field; `degraded` lacks an effect-cleanup reset on the
(production-unused) `dimension` param path; minor comment-accuracy nits
(reconnect backoff tiers, rollup-precedence JSDoc).
New users were still discovering cloud.copilotkit.ai through docs pages,
the README, example READMEs, and in-app banners/console messages. Replace
all user-facing web links with dashboard.operations.copilotkit.ai (the
destination the marketing-site CTAs already use). Functional API endpoints
(api.cloud.copilotkit.ai) are deliberately untouched since existing cloud
customers depend on them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Our previous doc site had a fumadocs/shadcn theme that was a bit
hodge-podge. This brings all of it into one clean and visually appealing
design. The border radii are standardized, colors are standardized and
bunch of passes for mobile/tablet have been done. An added bonus is that
the header nav has been redone to feel more natural to the eyes.
<img width="1850" height="1256" alt="Screenshot 2026-06-05 at 7 48
00 AM"
src="https://github.com/user-attachments/assets/0c6032f2-e406-4e51-9c31-d69a9d258d61"
/>
Applies the updated shell-docs theme treatment to the overview cards, sample tabs, framework selectors, hero command controls, and Copilot Cloud CTAs.
Call-site enumeration:
- FrameworkSelector/FrameworkTabs/IntegrationGrid changes stay on the docs landing and framework overview surfaces that already own those controls.
- OpsPlatformCTA and LinkToCopilotCloud keep their existing call sites while switching to shared chrome variables.
Refreshes the shell docs chrome around the shared theme tokens, announcement banner, desktop and mobile navigation, search trigger, theme toggle, and sidebar footer actions.
Call-site enumeration:
- PrimaryDocsTabs: rendered by MobileTopNav for tablet header tabs and ShellDocsLayout for mobile sidebar tabs.
- MobileSidebarFooterTalk: rendered by ShellDocsLayout sidebar.footer and hidden outside mobile/tablet sidebar usage via responsive classes.
- INTELLIGENCE_CTA_HREF and TALK_TO_ENGINEER_HREF: exported from BrandNav and reused by BrandNav/MobileSidebarFooterTalk so CTA destinations stay centralized.
It was left at false while fastapi/typescript are true. Under the opt-in
A2UI model false means no tool is injected, so the python demo rendered
no surfaces (and the docs code-tab showed no injectA2UITool). Set true to
match the other langgraph integrations.
Treat `signal` as optional on legacy LiveBadge rows (neutral 'unknown' marker
when absent), so the trimmed initial projection cannot misrender or throw.
Fetch `signal` on demand for genuinely-failing badges (excluding staleness-
downgraded greens via fail_count), with per-badge loading/error resolution that
distinguishes in-flight, partial-failure (requested-but-absent), and resolved
states. Injection-safe pb.filter OR query; object-valued fields render as JSON.
Banner visibility still derives from projection-safe key+state on the live rows;
detail copy self-fetches `signal` (via pb.filter placeholders, page size = key
count) only when omitted by the trimmed projection, with an identity-tagged cache
(id + transitioned_at) that won't serve a stale signal across content changes.
Thread a `stale` flag (muted, non-authoritative counts during reconnect-with-rows,
distinct from the no-rows loading state), fix the amber-chip/green-D5 cell to
classify into the health dimension, and surface the hook's `degraded` flag in the
header LiveIndicator (offline outranks degraded). Wire `degraded` from
useLiveStatus through to FeatureGrid in dashboard-page.
Drop the heavy `signal` field from the initial PocketBase fetch via a
STATUS_LIST_FIELDS projection + skipTotal + length-based bounded-concurrency
pagination, restoring fast first paint. Harden the realtime path: flushPending
teardown guards, keyless-delete id->key buffer collapse (last-write-wins),
SSE-delta signal-presence upsert, and a `degraded` flapping flag that clears on
heartbeat-success and resets on terminal error. ConnectionStatus deduped.
Restore stubbed globals (EventSource, localStorage, window config) in
afterEach via vi.stubGlobal/unstubAllGlobals + snapshot, and unmount the
hook in a finally so the 30s heartbeat interval and its pb singleton don't
leak past the test onto the about-to-close in-process server.
Also normalize the fetchInitial page merge to
[first.items, ...rest.map((r) => r.items)].flat() for clarity (behavior
identical to the prior heterogeneous flatMap).
The PocketBase JS SDK derives a request key from method+path and
auto-cancels any in-flight request that shares it. useLiveStatus's
fetchInitial fans pages 2..N out concurrently at the same path
(/api/collections/status/records), so every page after the first
cancelled its predecessor — the cancelled promises rejected, Promise.all
rejected, and the hook dropped to OFFLINE. Pass requestKey:null on the
page-1 read, every fan-out page, and the heartbeat ping so all
concurrent same-path reads complete.
Add a real-SDK regression test (the mocked suite never exercised the SDK
so it missed this): an in-process http server replays paged status
records with per-page latency, the production getPb() client drives the
real hook, and it asserts "live" with all rows. RED without requestKey:null.
A connecting status with an empty status map now renders a loading
affordance instead of authoritative checkmark-0 / D0 cells and
zeroed column tallies. Make loading required on the tally type so the
contract is explicit, and default the fail-safe undefined-tally
fallback to loading rather than fake zeros. Adds coverage for the
loading-state quadrants across computeColumnTally /
computeColumnTallyDetail and the OverlayColumnHeader affordance.
Replace the single serial-paginating getFullList with a concurrent
page-1 fetch + Promise.all fan-out over the remaining pages, merged by
array index so results are deterministic regardless of resolution
order. Forward a stable sort:"id" on every page so concurrent requests
share one ordering. Wrap the initial setRows in startTransition to
avoid a synchronous full-matrix re-render. The snapshot is best-effort;
the live SSE subscription reconciles anything created in the brief
fetch->subscribe window.
A STARTER-row column for an integration that has NO starter (the 7 columns
absent from STARTER_TO_COLUMN) was rendering a grey "✗" with a "no starter
for this integration" tooltip. That mis-communicates the truth: a grey/red
✗ reads as "we expected data and got none" or "a probe ran and failed",
when the reality is "this integration is architecturally unsupported in the
starter row".
buildStarterBadge's mapping-derived !isSupported branch now returns the same
🚫 "Not supported by this framework" treatment depth-chip/unified-cell
already use for unsupported cells. Columns that DO have a starter keep
surfacing their real probe status — a genuinely-red starter still renders
red ✗ and is never masked as 🚫.
Close the recurring "mutation result not validated" defect class in the
starter-fleet provisioner and make the existing-services snapshot fail loud
instead of silently feeding erroneous create decisions.
- Add a uniform assertMutationOk guard and route EVERY mutation through it:
serviceCreate (assert .id), serviceInstanceUpdate (was DISCARDED — a false
Boolean! return meant sleep/healthcheck/image/creds were never applied while
the script reported success; now asserted and "configured" is logged only
after verification), serviceDomainCreate (assert .domain on the create path),
and serviceInstanceRedeploy (routed through the same guard for consistency).
- Absorb a serviceCreate "already exists" rejection: on a snapshot-miss the
create path now re-fetches the service id by name and falls through to UPDATE
instead of aborting the whole fleet. Predicate renamed ALREADY_EXISTS_RE and
reused by the domain-create path.
- fetchExistingServices fails loud on page-drain truncation (hasNextPage still
true at the defensive bound) rather than returning a partial byName map.
- fetchExistingServices coalesces null serviceInstances/.edges (transitional
service nodes) so an unguarded .find can't TypeError and abort the fetch;
interface fields marked optional/nullable.
- TRANSIENT_ERROR_RE made single-line ([^\n]*? not [\s\S]*?) so a newline-joined
multi-error blob can't bridge "Service" and "not found" across lines.
- withRetry wraps the schedule-exhaustion rethrow with context and { cause }.
Three functional fixes to the starter-fleet provisioner found in CR:
- fetchExistingServices now drains the Relay ServiceConnection via
pageInfo.hasNextPage/endCursor. A single un-paginated query truncated the
snapshot (~27 SSOT + 12 starter services span >1 page), making an existing
starter look absent → CREATE path → serviceCreate "already exists" →
non-transient abort of the whole run.
- TRANSIENT_ERROR_RE now matches Railway's INTERPOLATED "Service <id> not
found" (id embedded), not just the contiguous "Service not found", so the
post-create eventual-consistency retry actually fires.
- serviceInstanceRedeploy result check: documented the verified Boolean!
contract (sources: redeploy-env.ts, bin/railway RestoreCommand) and now
gates on truthiness (rejects false/null, accepts truthy defensively).
Hardening: ABORT on the live path when GITHUB_TOKEN is unset (private GHCR
images would image-pull-backoff while reporting success); warn-and-continue
only under --dry-run. Benign domain "already exists" no-op now logs the actual
matched Railway message for a forensic trail.
Harden the committed starter-fleet Railway provisioner against the
partial-failure / mistyped-flag / un-deployed-image failure modes
surfaced in CR:
- Domain idempotency: a serviceDomainCreate that Railway rejects with an
"already exists" error (start-of-run snapshot missed the domain due to
eventual consistency, or a prior run died mid-fleet) is now caught as a
benign no-op (marked "existing", logged) so a re-run converges instead
of aborting the entire remaining fleet. A genuine non-transient error
still aborts.
- Explicit redeploy: serviceCreate + serviceInstanceUpdate(source.image)
only PINS the image; it does not start a deployment, and Railway's image
auto-updates fire only on a NEW digest push. Added serviceInstanceRedeploy
after the instance update on BOTH the create and update paths so the
pinned image actually runs (and starter_smoke can find the service up).
Mirrors the documented update+redeploy pattern in bin/railway and the
explicit redeploy showcase_deploy.yml issues after each GHCR push.
- argv validation: parseArgs() now rejects any unrecognized argument
(e.g. a mistyped --dry-rn) with a usage hint before any provisioning,
instead of silently ignoring it and proceeding to REAL live provisioning.
- Fail-fast safety: validate the Railway token AND registry credentials
up front in main() (token resolution no longer process.exit()s deep in
the GraphQL boundary; main().catch owns the exit). Broadened the
withRetry transient predicate to the domain/instance eventual-consistency
class via an overridable per-call predicate. Dry-run now reports a new
service's domain as "would-create" for a faithful preview.
Adds showcase/scripts/provision-starter-fleet.ts — a committed, idempotent
provisioner for the SSOT-decoupled "starter container fleet". It creates (or
updates) one sleepable Railway service per starter template in the STAGING
environment, deriving the 12 targets from STARTER_TO_COLUMN (the smoke-matrix
SSOT) so the fleet can never drift from the build matrix.
Per service: serviceCreate scoped to the STAGING env (environmentId on
ServiceCreateInput, so NO production instance is ever materialized) with
source.image=ghcr.io/copilotkit/starter-<slug>:latest (RAW starter slug) and
GHCR registryCredentials; then serviceInstanceUpdate against staging with
sleepApplication:true + healthcheckPath="/" + region=us-west1; then
serviceDomainCreate for a generated staging domain. A bounded retry absorbs
Railway's eventual-consistency "ServiceInstance not found" right after create.
Healthcheck is "/" not "/api/health": the starters' single deployable image
EXPOSEs 3000 running the Next.js frontend, which serves "/" and
"/api/copilotkit" but has no "/api/health" route; the agent's "/health" is on
the internal 8123 port Railway does not expose. region read-back is null on the
serviceInstance for ALL existing showcase services too — that is normal Railway
behavior, so the fleet matches the existing services.
The fleet is decoupled from the 27-service railway-envs SSOT (starter-* services
are auto-discovered by the starter_smoke probe). #5254 already made
verify-railway-image-refs.ts tolerate starter-* names, so provisioning does not
trip the image-ref gate / skip the showcase build.
Red-green tested against an injected Railway GraphQL mock: target derivation
(raw vs remapped slug), GHCR credential resolution, sleepApplication:true,
staging-env scoping on BOTH create and update (never prod), idempotent
update-vs-create, domain de-duplication, and transient-error retry.
The starter container fleet (starter-<slug>) is decoupled from the
27-service railway-envs SSOT: each starter-* service is auto-discovered
at runtime by the starter_smoke probe (railway-services discovery,
namePrefix "starter-") and is never read from railway-envs.ts.
verify-railway-image-refs.ts is a hard needs: of the build job and runs a
bidirectional live-Railway drift check. Before this change, provisioning
a starter-* service made it untracked in the SSOT, so findUntrackedServices
failed and the showcase build was SKIPPED (the canary regression).
Scope both drift checks (findUntrackedServices and, defensively,
findMissingServices) to exclude services matched by a single, well-named
predicate isStarterFleetService(name) => name.startsWith("starter-"),
mirroring the harness discovery filter convention. Real showcase-*/infra
services are still drift-checked exactly as before.
This is the prerequisite for Phase-3 starter provisioning — provisioning
must happen AFTER this merges.
The starter_smoke probe registers its driver under kind `starter_smoke`
(starterSmokeDriver, orchestrator.ts) and config/probes/starter_smoke.yml
declares `kind: starter_smoke`, but the probe-config Zod `kind` enum is
built from DIMENSIONS, which only listed the `starter` emit-prefix and not
the `starter_smoke` probe kind. The enum therefore rejected the YAML at
parse time; because `kind` is shared across all three union variants the
failure surfaced as the misleading "Unrecognized key(s) in object:
'discovery'" union error, the probe never loaded, AND probe-loader.test.ts's
shipped-config assertion failed (the lone recurring failure).
Add `starter_smoke` alongside `starter`, mirroring the existing kind/emit
pairs (e2e_d6/d6, e2e_deep/d5, e2e_demos/e2e). `discovery` was a red herring
— it is a valid key the DiscoveryBlockSchema already accepts; the working
d6-all-pills-e2e.yml uses the identical discovery shape.
probe_runs.triggered was `required: true`, and PocketBase rejects the
boolean `false` as empty (validation_required). The fleet aggregator opens
every run-history row with `triggered: false` (scheduled, not ad-hoc), so
run-history start() failed on every non-triggered run. Add an idempotent
migration that ALTERs the existing collection's field to optional (the
create migration already ran on staging/prod volumes, so we alter rather
than recreate).
github-actions[bot]