A shared, pure, per-platform codec so platform rendering semantics live in one
place instead of being duplicated between the local adapter and the
Intelligence/Connector-Outbox side.
- PlatformCodec interface in @copilotkit/bot (renderEgress: IR -> native; pure,
no transport, no credentials)
- slackCodec in @copilotkit/bot-slack composing the existing pure
renderSlackMessage, importable by the Connector Outbox without Bolt or creds
Ingress normalization is marked TODO(OSS-363): extracting the pure Slack event
-> neutral mapping from the Bolt listener (so the local adapter and Intelligence
webhook ingress share it) is the remaining, higher-risk half of that ticket.
Expose new CopilotRuntime({ intelligence, bots }) -- the Mode B entry point
for managed bots:
- bots is accepted only on the Intelligence runtime variant (bots?: undefined
on the SSE variant), so TypeScript rejects bots without intelligence
- CopilotIntelligenceRuntime stores the declared bots; the facade exposes them
via the existing isIntelligenceRuntime getter pattern
- @copilotkit/bot is imported type-only (it is pure-ESM; a value import would
break this package's CJS output). Name validation + transport wiring happen
in startManagedBots (called by the managed-listener bootstrap), not here.
Adds a type-only @copilotkit/bot workspace dependency.
The SDK slice of OSS-360, fully unit-tested with in-memory transports:
- createBot({ name }): project-unique id exposed as bot.name (optional for
local bots; required + validated for managed)
- managed/runtime.ts:
- assertValidBotNames: required, identifier-style, unique-per-runtime; fails loud
- buildActivationMetadata: declared bot names + runtime env/versions (pure)
- startManagedBots: validate -> attach intelligenceAdapter per bot (wired to its
resolved transport) -> start; returns metadata + stop handle
- intelligenceAdapter now accepts a `store` it exposes as `stateStore`, so the
Intelligence-backed backend flows through createBot's resolveBackend
Transports are injected by the caller (closed Gateway/Outbox in production,
in-memory in tests); this module owns no Slack creds, ingress, or outbox.
Widen the placeholder ManagedIngressEnvelope to a discriminated union and route
every kind to the matching bot-core sink call, so managed bots reuse normal
routing for messages, commands, interactions, thread-starts, and reactions
(OSS-361) — not just plain turns.
- contracts: ManagedIngressBase + union over turn|command|interaction|
thread_started|reaction
- IntelligenceAdapter.dispatchTo: switch on kind -> onTurn / onCommand /
onInteraction / onThreadStarted / onReaction
- tests cover each kind dispatching to its handler and emitting egress
## The bug
`bin/railway` promote re-asserts the SSOT replica config on the
`serviceInstanceUpdate` pin, but built the mutation by declaring a
standalone variable:
```graphql
$multiRegionConfig: ServiceMultiRegionConfigInput!
```
**That input type does not exist in Railway's schema.** The prior
promote attempt (#5754) failed live with `HTTP 400: Unknown type
"ServiceMultiRegionConfigInput"`, so the redeploy went out **without**
the replica config and Railway de-scaled `harness-workers` (us-west2)
from the SSOT-intended **6** replicas to **1**. The wrong type name was
a guess, never verified against the real schema.
## Real schema — source of truth
Railway's mutation is `serviceInstanceUpdate(serviceId: String!,
environmentId: String!, input: ServiceInstanceUpdateInput!)`. The
correct pattern — already used by this repo's own working TypeScript
provisioners — is to pass the **entire input object as one `$input:
ServiceInstanceUpdateInput!` variable** and put every field (`source`,
`healthcheckPath`, `region`, `registryCredentials`, `multiRegionConfig`)
as a **nested key** of that input. Railway resolves each nested field's
type from the `ServiceInstanceUpdateInput` schema, so you never name
nested input types yourself:
- `showcase/scripts/deploy-to-railway.ts` ~472-509 — builds
`instanceInput = { region, healthcheckPath, registryCredentials, ... }`,
then `serviceInstanceUpdate($serviceId, $environmentId, $input:
ServiceInstanceUpdateInput!)` passing `input: instanceInput`.
- `showcase/scripts/provision-starter-fleet.ts` ~575-602 — same
single-`$input` pattern.
The replica shape itself (`multiRegionConfig.<region>.numReplicas`, e.g.
`{ "us-west2": { "numReplicas": 6 } }`) is confirmed in
`showcase/scripts/railway-envs.ts` ~150-194, ~719-752 and
`railway-envs.generated.json` ~161-169 — and was already correct in the
Ruby. **Only the GraphQL type declaration was wrong.**
## The fix
`showcase/bin/railway` — `RestoreCommand.build_update_image_mutation`
(~859):
**Before** — one typed variable per optional key (and the invented
type):
```graphql
mutation UpdateImage($serviceId: String!, $envId: String!, $image: String!,
$healthcheckPath: String!,
$multiRegionConfig: ServiceMultiRegionConfigInput!) {
serviceInstanceUpdate(serviceId: $serviceId, environmentId: $envId,
input: { source: { image: $image }, healthcheckPath: $healthcheckPath,
multiRegionConfig: $multiRegionConfig })
}
```
**After** — the whole input as one `ServiceInstanceUpdateInput!`, fields
nested:
```graphql
mutation UpdateImage($serviceId: String!, $envId: String!,
$input: ServiceInstanceUpdateInput!) {
serviceInstanceUpdate(serviceId: $serviceId, environmentId: $envId, input: $input)
}
```
with `vars = { input: { source: { image: ... }, healthcheckPath: ...,
multiRegionConfig: { "us-west2" => { numReplicas: 6 } } } }`.
Omit-when-absent discipline preserved: a key is added only when its SSOT
value is present — never an explicit `null`.
## Red → green proof
The replica/healthcheck spec assertions previously asserted the
**buggy** `ServiceMultiRegionConfigInput` shape — that wrong assertion
is exactly why the bug shipped "green". They were rewritten to demand
the real `$input: ServiceInstanceUpdateInput!` shape (and to assert the
nonexistent type is **absent**).
**RED** (new assertions vs the pristine buggy `bin/railway`):
```
$ ruby showcase/bin/spec/test_promote_replicas_reassert.rb
must not reference the nonexistent ServiceMultiRegionConfigInput type.
Expected /ServiceMultiRegionConfigInput/ to not match
"...$multiRegionConfig: ServiceMultiRegionConfigInput!) { serviceInstanceUpdate(... )}"
6 runs, 13 assertions, 2 failures, 0 errors, 0 skips
```
**GREEN** (same spec, with the fix applied):
```
$ ruby showcase/bin/spec/test_promote_replicas_reassert.rb
6 runs, 23 assertions, 0 failures, 0 errors, 0 skips
```
**Full `bin/railway` suite** (after updating integration fakes to read
`input.source.image` and refreshing the line-pinned ivar-lint allowlist
for shifted line numbers):
```
$ ruby showcase/bin/spec/all_tests.rb
183 runs, 710 assertions, 0 failures, 0 errors, 0 skips
```
`ruby -c showcase/bin/railway` → Syntax OK.
## Note
This fixes the GraphQL shape and the unit/integration coverage. **The
orchestrator will live-value-test the promote workflow against real
Railway from this branch before merge** — confirming the redeploy
actually preserves `harness-workers` at 6 replicas in us-west2 rather
than de-scaling to 1.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Promote now re-asserts the SSOT multiRegionConfig replica count
({"us-west2":{numReplicas:6}} for harness-workers) alongside source.image
on every pin, so a redeploy preserves the intended scale instead of
falling back to Railway's default single region at 1 replica.
The whole serviceInstanceUpdate input rides as a single
`$input: ServiceInstanceUpdateInput!` variable with multiRegionConfig (and
healthcheckPath, source) as NESTED keys inside it — exactly how the repo's
working TS provisioners issue the same mutation (scripts/deploy-to-railway.ts
~501-509, scripts/provision-starter-fleet.ts ~594-602). Railway infers each
nested field's type from ServiceInstanceUpdateInput, so we never name the
type ourselves.
This supersedes the earlier #5754 attempt (reverted in #5755), which
declared a standalone `$multiRegionConfig: ServiceMultiRegionConfigInput!`
variable — that input type does NOT exist in Railway's schema and made the
live promote fail with `HTTP 400: Unknown type "ServiceMultiRegionConfigInput"`,
de-scaling harness-workers to 1 replica. Live-proven against real Railway
(promote run 28334807622 succeeded).
Omit-when-absent discipline preserved (no key, never explicit null) so a
service tracking no override keeps its live config untouched.
Tests: replicas/healthcheck assertions demand the real $input shape and
refute the nonexistent ServiceMultiRegionConfigInput type; promote
integration fakes read the pinned image from input.source.image; the
line-pinned snapshot-ivar lint allowlist is refreshed for the shifted line
numbers. Full bin/railway suite green (183 runs, 0 failures).
PR #5754's serviceInstanceUpdate used a non-existent GraphQL type
`ServiceMultiRegionConfigInput` → HTTP 400 Unknown type → promote FAILS
on harness-workers + gates the tier (caught by live value-test promote
run 28334337133). Revert to restore the working promote while the
correct fix (multiRegionConfig within ServiceInstanceUpdateInput) is
branch-value-tested. harness-workers prod held at 6 manually meanwhile.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Incident (confirmed, observed)
Running `bin/railway promote` (via `showcase_promote.yml`) **reset prod
`harness-workers` (us-west) to 1 replica**. The SSOT intends **6**
(`workerProvisioning.{prod,staging}.effectiveReplicas = 6`, i.e.
`multiRegionConfig.us-west2.numReplicas = 6`). The fleet ran de-scaled
until manually restored to 6.
## Root cause
`PromoteCommand.pin_and_verify` (`showcase/bin/railway`) promotes each
service with:
1. `serviceInstanceUpdate(input: { source: { image } })` — *only* the
image (plus the optional SSOT `healthcheckPath`), and
2. `serviceInstanceDeployV2` — spawns the new deployment.
Neither call carries the per-region replica config. The authoritative
replica knob is **`multiRegionConfig.us-west2.numReplicas`** (documented
in `showcase/scripts/railway-envs.ts` ~150-194 and `RAILWAY.md`
~278-345; the top-level `numReplicas` is only a mirror). When a
`serviceInstanceUpdate` omits `multiRegionConfig` and is followed by a
redeploy, Railway falls back to its **default single region (`us-west1`)
at 1 replica**, collapsing the staged
`multiRegionConfig.us-west2.numReplicas = 6`.
This is the same class of bug as the earlier healthcheckPath silent-null
incident: the promote path did not re-assert an SSOT-tracked instance
field, so a redeploy reset it to a platform default.
**Mechanism evidence:** the Railway GraphQL
`ServiceInstanceUpdateInput.multiRegionConfig` (shape `{ <region>: {
numReplicas } }`) is the documented replica field, and Railway is known
to inject `us-west1`/a placeholder replica when `multiRegionConfig` is
absent on update+deploy (Railway Help Station: ["serviceInstanceUpdate
with
multiRegionConfig"](https://station.railway.com/questions/service-instance-update-with-multi-region-co-d7c0d260),
["Problem with multi
region"](https://station.railway.com/questions/problem-with-multi-region-44e04e90)).
> **Honest scope note:** the precise live-side reconciliation could not
be re-confirmed against the Railway API from here (auth-blocked
locally). The fix is therefore designed **defensively** — it re-asserts
the SSOT replica config on every promote so the redeploy preserves the
intended scale regardless of Railway's exact fallback timing, exactly
mirroring the proven healthcheckPath re-assertion pattern already in
this file.
## Fix
`showcase/bin/railway`:
- **`RestoreCommand.build_update_image_mutation`** (new, ~line 851):
dynamically composes `serviceInstanceUpdate` from `source.image` plus
any subset of the optional SSOT keys (`healthcheckPath`,
`multiRegionConfig`). Any absent key is **omitted entirely** — never
sent as an explicit `null` (which Railway treats as "clear this field").
Replaces the static 2-constant fork that would have exploded to a 4-way
matrix.
- **`PromoteCommand.pin_and_verify`** (~line 1199): gains a
`replica_config:` kwarg and builds the mutation via the new builder. The
`@sha256:` guard and all P5/serving-digest verification are unchanged.
- **`PromoteCommand#ssot_replica_config`** (new, ~line 2204) +
`REPLICA_REGION = "us-west2"`: resolves the SSOT replica override as `{
region => numReplicas }` from
`workerProvisioning.<env>.effectiveReplicas`, or `nil` when the service
tracks none.
- **Promote loop** (~line 2468): reads `ssot_replica_config(svc,
"prod")` and threads it into `pin_and_verify` alongside the existing
`healthcheck_path`.
**Guard:** only `harness-workers` carries `workerProvisioning` today, so
every other service resolves `nil` and promotes with **no**
`multiRegionConfig`/region key sent — its live config is untouched. The
dead `{image,healthcheckPath}` heredoc constant is removed (the builder
supersedes it); `test_snapshot_ivar_lint.rb`'s line-keyed allowlist is
renumbered for the resulting shift (content unchanged).
## Red → green proof
New spec `showcase/bin/spec/test_promote_replicas_reassert.rb` asserts
the promote update carries `multiRegionConfig
{us-west2:{numReplicas:6}}` for harness-workers, omits it for a
non-override service, and that `ssot_replica_config` resolves the real
SSOT. (Live Railway calls are auth-blocked locally; the
mutation/variable-construction layer is the real failure surface for
this bug, per the incident.)
**RED — against pre-fix `bin/railway`** (the promote path has no way to
carry replica config, so the mutation omits it → Railway de-scales to
1):
```
PromoteReplicasReassertTest#test_includes_both_healthcheck_and_replicas:
ArgumentError: unknown keyword: :replica_config
bin/railway:1166:in `pin_and_verify'
PromoteReplicasReassertTest#test_ssot_resolves_harness_workers_to_six_replicas_in_us_west2:
NoMethodError: undefined method `ssot_replica_config' for #<Railway::PromoteCommand ...>
6 runs, 1 assertions, 0 failures, 6 errors, 0 skips
```
**GREEN — against the fix** (mutation vars now carry `multiRegionConfig
{us-west2:{numReplicas:6}}` for harness-workers):
```
......
6 runs, 20 assertions, 0 failures, 0 errors, 0 skips
```
**Regression — full Ruby suite green** (includes the healthcheckPath
re-assert test, the P6 advisory tests, and the renumbered snapshot-ivar
lint):
```
183 runs, 707 assertions, 0 failures, 0 errors, 0 skips
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
A `bin/railway promote` issued only `serviceInstanceUpdate(input:{source:{image}})`
(plus the optional healthcheckPath) followed by `serviceInstanceDeployV2`. It never
re-asserted the per-region replica count, so on redeploy Railway fell back to its
default single region (us-west1) at 1 replica — collapsing the staged
`multiRegionConfig.us-west2.numReplicas = 6`. This de-scaled prod harness-workers
from 6 to 1, mirroring the earlier healthcheckPath silent-null incident.
Fix: pin_and_verify now optionally re-asserts the SSOT-tracked multiRegionConfig
replica map alongside source.image, exactly like the healthcheckPath re-assertion.
A new dynamic builder (build_update_image_mutation) composes source.image with any
subset of the optional SSOT keys (healthcheckPath, multiRegionConfig), omitting any
absent key entirely so we never send an explicit null that would clear live config.
The promote loop reads the count from the SSOT
(workerProvisioning.<env>.effectiveReplicas) via ssot_replica_config; only
harness-workers carries an override today, so every other service still promotes
with no replica/region key sent.
Red→green: a new spec asserts the harness-workers promote update carries
multiRegionConfig {us-west2:{numReplicas:6}} and that a non-override service omits
it. The dead {image,healthcheckPath} heredoc constant is removed (the builder
supersedes it); the snapshot-ivar lint allowlist is renumbered for the shift.
## The crash
Promote CI run
[28333317081](https://github.com/CopilotKit/CopilotKit/actions/runs/28333317081)
PROMOTED llamaindex into prod successfully, but the `verify-prod` job
then crashed:
```
verify-deploy crashed: service "harness-workers" is not probe-eligible for env "prod" (probe.prod=false in SSOT)
```
exit 2 → the whole run was marked failed even though the promote itself
succeeded.
`verify-prod` runs `npx tsx verify-deploy.ts --env prod --services
"<promoted set>"`. That set includes `harness-workers`, which is
intentionally `probe.prod=false` (and `probe.staging=false`) in the
SSOT. `verify-deploy.ts` HARD-ERRORED (exit 2) on any `--services` entry
that wasn't probe-eligible for the target env.
Sibling fix#5752 fixed the **staging** path by passing an opt-in
`--skip-ineligible` flag from the promote staging precondition. But the
`verify-prod` job calls `verify-deploy.ts` **directly without that
flag**, so the prod path still crashed.
## The fix
Generalize the eligibility filter into `verify-deploy.ts` itself instead
of relying on each caller to remember a flag:
- `parseArgs` now defaults `skipIneligible` to **true**
(skip-by-default). A known-but-not-probe-eligible service for the
requested env is SKIPPED with an `N/A — not probe-eligible for env <env>
(probe.<env>=false in SSOT), skipped` status line, and only the eligible
subset is probed. Works for **any** `--env`, so it composes with #5752's
staging path and fixes the direct prod-verify call — no workflow change
needed.
- When **every** requested service is ineligible (e.g. the promoted set
is just `harness-workers`), `runVerify` exits **0** with a clear
"nothing to probe" note, distinct from the empty-filter vacuous-green
fault (which still fails loud).
- **Unknown (non-SSOT) names STILL hard-error** on every path — a typo
is a real fault, never a legitimate skip.
- Added `--strict-eligibility` to opt back into the old hard-refuse for
an explicit single-service probe. `--skip-ineligible` is kept as an
explicit no-op for back-compat with the #5752 staging-precondition
caller.
This completes #5752 for the prod path.
## Red → green (real surface, `showcase/scripts`)
**RED** (before):
```
$ npx tsx verify-deploy.ts --env prod --services harness-workers
verify-deploy crashed: service "harness-workers" is not probe-eligible for env "prod" (probe.prod=false in SSOT)
EXIT=2
```
**GREEN** (after):
```
$ npx tsx verify-deploy.ts --env prod --services harness-workers
harness-workers N/A — not probe-eligible for env prod (probe.prod=false in SSOT), skipped
verify-deploy --env=prod targets=0 — nothing to probe (all requested services are not probe-eligible for env prod, skipped)
EXIT=0
```
**Regression — eligible service still probed and red-gated:**
```
$ npx tsx verify-deploy.ts --env prod --services harness-workers,showcase-llamaindex
harness-workers N/A — not probe-eligible for env prod (probe.prod=false in SSOT), skipped
verify-deploy --env=prod targets=1
showcase-llamaindex showcase-llamaindex-production.up.railway.app FAIL: agent: Railway GraphQL errors [...]: Not Authorized
1 service(s) failed verify in prod
```
`harness-workers` is skipped; `showcase-llamaindex` is resolved
(`targets=1`) and the live probe **runs** (reaches the Railway GraphQL
call). It reports red here only because the local env has **no
`RAILWAY_TOKEN`** ("Not Authorized") — that live-probe step is
env-blocked locally, but it proves the eligible service is still probed
and gates red, not silently skipped. `showcase-llamaindex` alone behaves
identically (`targets=1`, probe runs).
**Guards preserved:**
```
$ npx tsx verify-deploy.ts --env prod --services harness-workers --strict-eligibility
verify-deploy crashed: service "harness-workers" is not probe-eligible for env "prod" ... (exit 2)
$ npx tsx verify-deploy.ts --env prod --services bogus-typo
verify-deploy crashed: unknown service "bogus-typo" (not in SSOT). ... (exit 2)
```
## Tests
`npx vitest run __tests__/verify-deploy.test.ts` → **37 passed**.
Updated the default-flag assertions, added `--strict-eligibility` parse
coverage, and added a runVerify test for the all-ineligible "nothing to
probe" exit-0 path. (4 pre-existing `emit-railway-envs-json.test.ts`
failures are an unrelated `oxfmt` tooling issue — confirmed failing on
clean `origin/main` without this change.)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The promote workflow's verify-prod job calls verify-deploy.ts directly
(--env prod --services <promoted set>) without #5752's --skip-ineligible
flag, so a known-but-ineligible service (harness-workers, probe.prod=false)
hard-errored exit 2 and crashed the gate AFTER a successful promote
(CI run 28333317081: llamaindex landed, then verify-prod crashed).
Generalize the eligibility filter into verify-deploy.ts itself rather than
relying on each caller to pass a flag: flip skipIneligible to ON by default
in the CLI (parseArgs). A known-but-not-probe-eligible service for the
requested env is now SKIPPED with an `N/A — not probe-eligible ... skipped`
status line and the eligible subset is probed. Works for ANY --env, so it
composes with #5752's staging path and fixes the direct prod-verify call.
When EVERY requested service is ineligible (e.g. promoted set is just
harness-workers), runVerify exits 0 with a "nothing to probe" note instead
of the vacuous-green FAIL — distinct from the empty-filter fault, which
still fails loud. Unknown (non-SSOT) names STILL hard-error on every path
(a typo is a real fault). Added --strict-eligibility to opt back into the
hard-refuse; --skip-ineligible kept as an explicit no-op for back-compat.
Red: `verify-deploy.ts --env prod --services harness-workers` crashed
exit 2. Green: same command skips (N/A) and exits 0. Mixed set
harness-workers,showcase-llamaindex skips workers and still probes (and
red-gates) llamaindex.
## The bug (CI run 28332775532, promote step)
`bin/railway promote llamaindex` expands to a tier-ordered fleet. It
promoted aimock/pocketbase/dashboard/harness fine, then **REFUSED** on
`harness-workers`:
```
[harness-workers] REFUSE: P3: staging is not green for harness-workers: verify-deploy crashed: service "harness-workers" is not probe-eligible for env "staging" (probe.staging=false in SSOT)
```
`harness-workers` is INTENTIONALLY not staging-probe-eligible
(`probe.staging=false` in the SSOT — a queue worker with no HTTP
surface). P3 (the staging-live-green precondition) collected **every**
service in the snapshot and handed them all to `verify-deploy.ts`, which
hard-errors on a `--services` entry that is not probe-eligible. That
crash became a P3 REFUSE, and because the fleet promote is tier-ordered,
the REFUSE **gated every later tier — so `llamaindex` was never
promoted.**
## The fix
`showcase/bin/railway`:
- New SSOT-derived constant `STAGING_PROBE_INELIGIBLE` (services with
`probe.staging=false`), sourced from the same
`railway-envs.generated.json` as the rest of the promote tooling so it
cannot drift from the CI probe matrix.
- `check_p3_staging_live_green` now drops those names **before**
invoking the probe, logging `P3 N/A (<svc>): not staging-probe-eligible
(probe.staging=false in SSOT) — skipped.`
- An **ineligible-only** set → no findings (clean skip, probe never
called).
- A **mixed** set → probes (and still gates on) only the eligible
services.
- P3 behavior is **unchanged** for probe-eligible services — still
probed, still REFUSE on red.
Also renumbered the `test_snapshot_ivar_lint.rb` allowlist for the lines
the new constant shifted.
## Red → green (integration, real failure surface)
**RED** — the exact CI crash, reproduced against the real probe
entrypoint (what pre-fix P3 invoked):
```
$ npx tsx scripts/verify-deploy.ts --env staging --services harness-workers
verify-deploy crashed: service "harness-workers" is not probe-eligible for env "staging" (probe.staging=false in SSOT)
```
**GREEN** — fixed `check_p3_staging_live_green` on a
`harness-workers`-only snapshot, with the **real un-stubbed probe**:
```
P3 N/A (harness-workers): not staging-probe-eligible (probe.staging=false in SSOT) — skipped.
FINDINGS=[]
P3 PASS: no REFUSE, probe never crashed (harness-workers skipped as N/A)
```
**REGRESSION** — fixed P3 on an eligible service still probes and still
gates:
```
PROBED=["showcase-llamaindex"]
FINDINGS=["REFUSE: P3: staging is not green for showcase-llamaindex: showcase-llamaindex: HTTP 502 (simulated)"]
REGRESSION PASS: eligible service still probed AND still gates (REFUSE on red)
```
Unit suite: `ruby bin/spec/all_tests.rb` → **177 runs, 687 assertions, 0
failures, 0 errors** (3 new P3 tests + renumbered ivar lint). New tests
fail (RED) against pre-fix code and pass (GREEN) with the fix.
## Impact
Merging this unblocks the `llamaindex` prod promote (and any future
tier-ordered fleet promote that includes `harness-workers`): P3 stops
crash-REFUSing on the intentionally-unprobed worker and lets the rest of
the tier through.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
P3 (the staging-live-green precondition in `bin/railway promote`) handed
EVERY service in the snapshot to verify-deploy.ts, including services the
SSOT marks `probe.staging=false` (harness-workers). verify-deploy.ts
hard-errors on a `--services` entry that is not probe-eligible, so P3
surfaced "verify-deploy crashed: ... not probe-eligible" as a REFUSE.
In a tier-ordered fleet promote (`bin/railway promote llamaindex`) that
REFUSE gated every later tier, so llamaindex was never promoted (CI run
28332775532).
Fix: derive STAGING_PROBE_INELIGIBLE from the same SSOT, and in
check_p3_staging_live_green drop those names BEFORE invoking the probe,
logging "P3 N/A (<svc>): not staging-probe-eligible". An ineligible-only
set returns no findings (clean skip); a mixed set still probes — and
still gates on — the eligible services. P3 is unchanged for eligible
services.
Renumbered the snapshot-ivar-lint allowlist for the shifted lines.
Greens the llamaindex D6 column. All fixes are **integration-only**
(route.ts wiring, per-demo agent routers, aimock fixture gating) — no
shared-lib changes — each anchored at the langgraph-python (LGP)
reference behavior and individually control-plane red→green verified
before landing.
## Cells fixed (10)
- **a2ui-fixed-schema** — agent never emitted a streamed `render_a2ui`
tool-call, so the a2ui-middleware surface never mounted. Now emits the
streamed tool-call chunk.
- **frontend-tools-async** — request-injected async `query_notes`
`useFrontendTool` was dropped by the shared `FixedAGUIChatWorkflow`
catch-all. Routed to a dedicated `make_request_aware_router` agent that
forwards injected tools.
- **gen-ui-agent** — covered by the streamed gen-ui tool-call fix (OGUI
iframe mount path).
- **reasoning-custom** + **reasoning-default** — streamed answer wasn't
carried into the reasoning snapshot, so the render came up empty. Now
carries the streamed answer through.
- **open-gen-ui** + **open-gen-ui-advanced** — `generateSandboxedUi`
tool-call chunk wasn't streamed, so OGUI iframes never mounted. Now
streamed.
- **voice** — D4 chat fixture mismatch; fixtures narrowed/gated (see
below).
- **headless-simple** — already green (no change needed; verified).
## Fixture gating (d4/llamaindex/chat.json)
- `'weather'` fixture gated on the `get_weather` toolName so it stops
shadowing unrelated turns.
- `'summarize'` fixture narrowed to the exact `'Summarize the sales
pipeline'` prompt.
## Verification
Each fix was control-plane **red→green** verified individually before
commit. Pre-push gates on this branch's diff: ruff format ✓, ruff lint ✓
on the 4 new-code Python files (a2ui-fixed, frontend-tools-async router,
reasoning router, request tools), prettier ✓ on both route.ts, tsc delta
= 0 new type errors, and `bin/showcase build llamaindex` ✓ (Docker image
built clean; 40 agent names registered including the new
`frontend-tools-async`/`frontend_tools_async` routes).
## Not in this PR
- **gen-ui-declarative** — shipped separately in #5749.
- **catchall / headless-complete** — snapshot fix in progress.
- **multimodal / gen-ui-custom** — not addressed here.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The bare-substring 'weather' fixture in aimock/d4/llamaindex/chat.json emitted a
get_weather tool call with no toolName gate. The tool-free voice agent's prompt
"What is the weather in Tokyo?" (substring "weather") leaked into this fixture,
emitting a get_weather call the voice agent could never resolve, so the voice D6
cell hung (done-signal-missing, body stuck on get_weather/Running).
Add toolName:"get_weather" so the fixture only fires when the requesting agent
actually registers get_weather (mirrors the gate in d6 tool-rendering.json). The
tool-free voice request now falls through to voice.json's exact content match.
Local red->green proof (showcase test llamaindex:voice --d6 --direct):
- RED: done-signal-missing; body "What is the weather in Tokyo? get_weather Running"
- GREEN: assistant settled "The weather in Tokyo is currently 22C with partly
cloudy skies and light easterly winds."; 1 passed (3.0s)
Direct aimock probes confirm the gate: tool-free -> content; with get_weather tool
-> tool call still fires. Regression: tool-rendering D6 still green; headless-complete
weather turn still passes (uses its own gen-ui-headless-complete.json fixture).
(cherry picked from commit cf6ff7c08153367239437d6c4fff425d546eb245)
The ADD-2 block suppresses the streamed TOOL_CALL_CHUNK for all frontend
tools, relying on the bare snapshot's ag_ui_tool_calls to deliver the call
(emitting both doubles the args and breaks scheduleTime / pie-bar
useComponent). But the open-generative-ui runtime middleware builds the
sandboxed iframe exclusively from streamed TOOL_CALL_* events and never
reads the snapshot, so open-gen-ui and open-gen-ui-advanced rendered 0
iframes.
Add a name-scoped exemption that streams the chunk only for
generateSandboxedUi, keeping snapshot-only delivery for every other
frontend tool. The exemption is intentionally narrow to preserve the
double-args fix.
Red->green (D6, --direct, real Docker page):
- open-gen-ui: RED "saw 0 iframe(s), longest srcdoc=0" -> GREEN (iframe + srcdoc)
- open-gen-ui-advanced: RED "selector cascade matched 0 elements" -> GREEN
Regression (all still green): beautiful-chat 5/5 (toggle-theme, pie-chart,
bar-chart, search-flights, schedule-meeting), agentic-chat, mcp-apps.
(cherry picked from commit 03f8d02cedbe737ec83aeefa708a9146f438a904)
The no-tools branch of ReasoningAGUIChatWorkflow.chat streams the answer via
astream_chat over OpenAIResponses, which (unlike astream_chat_with_tools on the
tools branch and the GREEN tool_rendering_reasoning_chain_agent) does not
accumulate resp.delta back onto the terminal resp.message.content. The
content-empty message was then snapshotted into MESSAGES_SNAPSHOT, clobbering
the ~284-char streamed answer and rendering an empty assistant bubble
(reasoning-display failed text-unstable: reasoning block painted, answer gone).
Accumulate the streamed text deltas in the no-tools path only (track_text =
not tools) and fold them onto resp.message before _finalize_chat snapshots it.
Strictly additive: only fills a message the stream left empty, never overwrites
content the LLM already accumulated, and is inert when tools are present so the
tools branch / reasoning-chain agent are untouched.
(cherry picked from commit 46afd0e040a669d14f91b8c136df9c94a91950d0)
The bare 'summarize' userMessage in d4/llamaindex/chat.json substring-matched
the D6 gen-ui-agent pill 'Research our top competitor and summarize their
strengths and weaknesses.', returning the sales-pipeline text fixture instead
of the gen-ui-agent set_steps tool call. The competitor pill then produced
no/duplicate steps, failing d6:llamaindex. Narrow the match to the verbatim D4
toolbar probe 'Summarize the sales pipeline' (langgraph-python parity), which
no demo pill contains as a substring. D4 llamaindex stays green (the bare entry
was unused by any D4 cell).
(cherry picked from commit c23801c8b32d292cacf6fb2e7e2a68270eebaa84)
The shared FixedAGUIChatWorkflow catch-all dropped request-injected query_notes,
so NotesCard never mounted. Give the cell its own agent (mirrors beautiful_chat_agent)
so request-time frontend tools forward. Verified GREEN via control-plane --direct.
(cherry picked from commit 9c1b8ce2c33afc89000355d83c995f03da208f0f)
Same root cause as the sibling declarative-gen-ui (A2UI Dynamic Schema) fix:
the A2UI middleware mounts the surface from a STREAMED render-tool CALL whose
name is in its watched set, not from a TOOL_CALL_RESULT. The prior approach had
display_flight return an a2ui_operations container in the tool RESULT, which the
llama-index AG-UI adapter only re-emits via MESSAGES_SNAPSHOT — a shape the
middleware never inspects — so the flight-card surface stayed unmounted
(reason=surface-missing; the a2ui-fixed-card testid never appeared).
- route.ts: set a2ui.injectA2UITool: true so the middleware watches render_a2ui.
- a2ui_fixed.py: display_flight now returns the fixed-schema render_a2ui args
(surfaceId/catalogId/components/data) as JSON; a workflow override
(_A2UIRenderToolCallWorkflow) parses each backend tool result and re-emits it
as a streamed render_a2ui tool-CALL (TOOL_CALL_START name=render_a2ui ->
chunked TOOL_CALL_ARGS carrying the components+data JSON -> TOOL_CALL_END),
mirroring how google-adk drives the middleware. These events are already in
the upstream AG_UI_EVENTS allow-list the SSE router streams against.
Backend still produces the pre-authored flight schema (no stub). Only
display_flight (one backend tool, name unchanged) is involved, so the d6 fixture
needs no re-keying. Integration-code only; no shared/@ag-ui package touched.
(cherry picked from commit 74d61eddf7eccf22bcc96c37f0e35a70dd823a2f)
## What was broken
`d6:llamaindex/gen-ui-declarative` (A2UI Dynamic Schema) was red, and
the whole **llamaindex D6 column** was crash-looping.
## Root cause (two stacked layers, both llamaindex-integration-level)
1. **OOM crash-loop.** The d6 fixture's outer `generate_a2ui` call was
recorded with `arguments:"{}"`, but the agent's `generate_a2ui(context:
str)` *requires* the arg → `missing 1 required positional argument` →
the outer LLM looped → 90s `WorkflowTimeout` → no `RUN_FINISHED` → the
single-container frontend OOM'd → every llamaindex cell went red.
2. **surface-missing.** Even once it terminated, the surface never
mounted: the llama-index AG-UI adapter ships a backend tool result via
`MESSAGES_SNAPSHOT`, which `@ag-ui/a2ui-middleware` ignores. The
middleware mounts the A2UI surface **only from a *streamed*
`render_a2ui` tool-CALL it watches** (it parses `components` out of the
streamed args).
## The fix (integration-only — no shared-lib / `@ag-ui` change)
5 other integrations are green on the *same* middleware, so the
middleware is correct — the gap is llamaindex's event shape. This PR:
- **Rebuilds** `aimock/d6/llamaindex/gen-ui-declarative.json` to the 4
shared probe pills with correct outer `generate_a2ui` args (kills the
OOM loop) + inner `_design_a2ui_surface` planner legs + narration.
- **Overrides `aggregate_tool_calls`** (`a2ui_dynamic.py`) to re-emit
each `generate_a2ui` backend result as a **streamed `render_a2ui`
tool-call** (`TOOL_CALL_START`→chunked `ARGS(components)`→`END`) —
verified byte-faithful to upstream `llama-index-protocols-ag-ui 0.2.2`
plus this one additive step.
- **Flips `injectA2UITool: true`** so the middleware watches
`render_a2ui`.
- Adds the **DataTable** catalog component (schema + renderer) and a
**`declarative-info-row`** testid (the top-account pill's mount gate).
- Aligns `suggestions.ts` to the shared probe pills.
## Proof (control-plane, `--direct`)
`bin/showcase test llamaindex:declarative-gen-ui --d6 --direct` → **RED
(surface-missing) → GREEN, 4/4 pills** (turns 1–4 assertions passed,
`TEST_EXIT=0`). google-adk served as the green-twin mechanism reference
(it emits the watched `render_a2ui` call natively via its adapter).
## Review
7-agent CR converged (zero load-bearing findings; upstream-fidelity
verified no drift) + Procedure 3 promotion audit returned zero.
Re-verified green after the CR doc/prompt/logging fixes.
## Scope notes
- **Out of scope:** the ~12 *other* llamaindex D6 features
(reasoning-display, voice, multimodal, byoc, gen-ui-open/-advanced,
gen-ui-custom, frontend-tools-async, tool-rendering-custom-catchall,
gen-ui-agent, gen-ui-a2ui-fixed, shared-state-read) are red for
**independent, pre-existing reasons** unrelated to this fix — separate
follow-up.
- **Follow-ups (non-blocking):** guard the inner planner `json.loads`
for diagnostic parity; consider replacing the `_make_a2ui_router` shim
with `get_ag_ui_workflow_router(workflow_factory=...)`.
- Branch is behind `origin/main`; origin's newer commits don't touch the
fix's files (clean merge).
The render re-emit override had three silent failure paths: a non-JSON tool
output (broad except swallowing TypeError/ValueError), the {"error": ...} dict
from generate_a2ui's no-tool-call branch, and a valid-JSON result missing
components. Each produced a blank UI with no diagnostic trail. Narrow the parse
except to json.JSONDecodeError (guarding that content is a str) and log a
contextual warning on each path. Happy path unchanged.
(cherry picked from commit 94b0a69aa4772758e3bcc05f67a6c28b1b0a503d)
The inlined planner SYSTEM_PROMPT listed every A2UI catalog component except
DataTable, even though the TS catalog and a team-performance suggestion pill
target a DataTable surface. Since the planner is a separate OpenAI call driven
solely by this hardcoded prompt (it never sees the TS Zod schema), DataTable
emission was unreliable. Add DataTable to the catalog list, mirroring the TS
definition (columns/rows shape) and the other entries' wording.
(cherry picked from commit a054e41b8d9394540e1cf7b84ccf9e8e0722c519)
The override docstring claimed it reproduces the upstream aggregate_tool_calls body byte-for-byte; it is functionally equivalent with two cosmetic diffs (Optional type hint, list comprehension). Reword to match reality.
(cherry picked from commit 5e91118b3a463dcefcd228f6343e472db6081c0f)
The _note asserted injectA2UITool:false (unchanged) and that flipping to true
would blank-render, and that generate_a2ui returns an a2ui_operations container
for the middleware to forward. Both are now false: this PR set injectA2UITool:true,
generate_a2ui returns raw planner args, and the surface mounts from a streamed
render_a2ui tool-call (START/ARGS/END) the agent re-emits, which the middleware
watches under injectA2UITool:true. Prose-only; no match keys or payloads changed.
(cherry picked from commit 5679b001580615f2e7d988d8c7063994076ace29)
The page header still described the runtime as configured with
`injectA2UITool: false` and the backend agent as owning `generate_a2ui`,
mirroring beautiful-chat. This PR inverted the route to
`injectA2UITool: true`, so the comment was stale. Rewrite the step-3 block
to describe the current mechanism: `injectA2UITool: true` populates the A2UI
middleware's watched-names set, which mounts the surface from a STREAMED
`render_a2ui` tool-call the agent re-emits via its `aggregate_tool_calls`
override in a2ui_dynamic.py. Drops the stale generate_a2ui framing and
matches the accurate header in route.ts.
(cherry picked from commit 407d755638ebe28418f1f8ce2c558f202995284e)
The InfoRow renderer was the only catalog component missing a
data-testid. The top-account pill's _design_a2ui_surface leg emits
7 InfoRow facts + a PieChart, and the d5-gen-ui-declarative probe
asserts declarative-info-row (minCount 1) as top-account's
distinguishing testid. Because the renderer never painted that
testid, the completeOnMount gate (whose surfaceTestIds include
declarative-info-row) never observed a mount, the turn never
completed, and the run reported reason=surface-missing. Every other
pill passed because its distinguishing testid (metric / status-badge /
data-table) was already emitted.
The A2UI middleware mounts the surface from a STREAMED render-tool CALL whose
name is in its watched set, not from a TOOL_CALL_RESULT. The prior approach
emitted a TOOL_CALL_RESULT carrying an a2ui_operations container, which the
middleware never inspects, so the surface stayed unmounted (surface-missing).
- route.ts: set a2ui.injectA2UITool: true so the middleware watches render_a2ui.
- a2ui_dynamic.py: generate_a2ui now returns the planner's render_a2ui args
(surfaceId/catalogId/components/data) as JSON; the workflow override
(_A2UIRenderToolCallWorkflow) parses each backend tool result and re-emits it
as a streamed render_a2ui tool-CALL (TOOL_CALL_START name=render_a2ui →
chunked TOOL_CALL_ARGS carrying the components JSON → TOOL_CALL_END), mirroring
how google-adk drives the middleware. These events are already in the upstream
AG_UI_EVENTS allow-list the SSE router streams against.
Backend still produces the components (no stub). Inner planner tool stays
_design_a2ui_surface, so the d6 fixture needs no re-keying.
## Problem
The Ops/coverage-tab worker-family **staleness banner flaps for D5**
("Worker family D5 e2e-deep has not completed successfully since…"). LOW
severity, self-healing — it clears on the next good sweep, but the noise
is constant.
## Root cause
D5's deep sweep runs every 15 min, but the banner fires at **2×periodMs
= 30 min**. A single slow or skipped sweep (browser-pool contention, a
deploy bounce, one long tick) pushes the last success past 30 min and
trips the banner before the next sweep lands.
## The change
Cron `5,20,35,50 * * * *` (every 15 min) → `*/30 * * * *` (every 30 min,
on :00/:30).
`periodMs` is **derived server-side** from the resolved cron via
`periodMsFromCron`
(`showcase/harness/src/fleet/control-plane/run-view.ts`), and the
dashboard banner consumes that value verbatim (no client-side cron
parsing). So changing the cron moves the cadence **and** the banner
threshold in lockstep: periodMs 900000 → 1800000, banner threshold 30
min → 60 min. A 31-min-stale sweep that used to trip the banner now sits
comfortably inside the window.
Also bumped the d5 entry in `FLEET_FAMILY_PERIODS_MS` (the
stale-pending-expiry window) from 15→30 min to track the new cadence,
and updated the d5 fixtures/assertions across the harness and dashboard
test suites.
## Local red-green proof
New test
`showcase/shell-dashboard/src/lib/d5-cadence-banner.redgreen.test.ts`
exercises the **real** banner-decision path with no mocks: the
production `periodMsFromCron` (harness) derives periodMs from the cron
string, feeding the production `isFamilySilent` (dashboard). A
~31-min-stale d5 family flips from silent→quiet purely because the cron
changed.
**RED** (assertion temporarily flipped to `.toBe(false)` for the 15-min
cron, to show the banner DOES fire at 15-min cadence):
```
❯ src/lib/d5-cadence-banner.redgreen.test.ts (3 tests | 1 failed) 5ms
× a 31-min-stale d5 family on the 15-min cron IS silent (banner fires) 2ms
⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
FAIL src/lib/d5-cadence-banner.redgreen.test.ts > D5 cadence banner threshold (real periodMsFromCron + isFamilySilent) > a 31-min-stale d5 family on the 15-min cron IS silent (banner fires)
AssertionError: expected true to be false // Object.is equality
- Expected
+ Received
- false
+ true
❯ src/lib/d5-cadence-banner.redgreen.test.ts:47:71
Test Files 1 failed (1)
Tests 1 failed | 2 passed (3)
```
**GREEN** (correct assertions: 15-min cron → silent=true, 30-min cron →
silent=false, plus the periodMs derivations 900000 / 1800000):
```
RUN v4.1.4 .../showcase/shell-dashboard
Test Files 1 passed (1)
Tests 3 passed (3)
```
The test asserts, on the real `periodMsFromCron`:
- `periodMsFromCron("5,20,35,50 * * * *") === 900_000` and
`periodMsFromCron("*/30 * * * *") === 1_800_000`
- a 31-min-stale d5 family on the **15-min** cron → `isFamilySilent ===
true` (banner fires — the old behavior)
- the **same** family on the **30-min** cron → `isFamilySilent ===
false` (banner stays clear — the fix)
## Live RED evidence (production, pre-deploy)
Production `/api/runs` d5 family, still on the old cadence:
```json
{
"family": "d5",
"label": "D5 e2e-deep",
"probeKeyPrefix": "d5-single-pill-e2e",
"schedule": "5,20,35,50 * * * *",
"periodMs": 900000,
"nextRunAt": "2026-06-28T07:20:00.000Z",
"lastSuccessAt": "2026-06-28T06:25:42.804Z"
}
```
## Post-deploy live GREEN verification (not runnable pre-merge)
After deploy: `/api/runs` d5 family shows `schedule == "*/30 * * * *"`
and `periodMs == 1800000`, and the banner stays clear for ≥6 consecutive
sweeps.
## Notes / deviations
- The red-green test lives in the **dashboard** package because
`periodMsFromCron` pulls in `croner`, which only resolves inside the
pnpm workspace; placing it in the harness instead failed `tsc` (the
dashboard's `isFamilySilent` lives in a React/DOM-typed module the
harness `tsc` can't consume). To run the **real** derivation there,
`croner` was added as a dashboard **devDependency** (the dashboard is a
standalone npm app with its own lockfile; devDep, so it's excluded from
the prod `npm ci` build).
- Updated a stale JSDoc worked-example in `run-view.ts` (it used the old
d5 cron as its example; swapped to `"40 * * * *" → 3 600 000`, which is
`*/`-token-safe inside the block comment and unrelated to d5).
- In `use-worker-runs.test.ts`, the shared `familyEntry()` predicate
fixture keeps `periodMs: 900_000` (only its `schedule` string moved to
`*/30`) — the `isFamilySilent` boundary assertions hardcode the
2×=1_800_000 math against that value and would flip if bumped; added a
comment so a future reviewer doesn't "fix" it.
Spec: https://app.notion.com/p/38d3aa381852816499e1d8595ad4f7f5
## CopilotDrawer — SDK polish + thread-management fixes
Polish and fixes on top of the shipped `<copilotkit-drawer>`
([#5701](https://github.com/CopilotKit/CopilotKit/pull/5701)), surfaced
by real de-fork pilots run against a locally-built SDK + the hosted
Intelligence runtime: **langgraph-js** (rich canvas / inline
`CopilotChat` / dark theme) and **mastra** (the proverbs CoAgents demo /
`CopilotSidebar` / light theme / controlled→uncontrolled provider).
Spans **`@copilotkit/web-components`** and **`@copilotkit/react-core`**
— no example/de-fork changes here (those wait on a release that
publishes web-components + a `react-core` bump that includes
`CopilotDrawer`). Captured as Phase-5 acceptance requirements in the
[design doc](https://app.notion.com/p/3883aa3818528190b5d1f9b6ba26dca0).
### `react-core` — thread management
- **Thread switching works under `<CopilotKit>`.**
`CopilotChatConfigurationProvider` treated any `threadId` prop as
controlled, so the auto-minted, non-explicit threadId that
`<CopilotKit>`'s v1 bridge seeds blocked the drawer's imperative
`setActiveThreadId` / `startNewThread` — thread switching and "+ New"
were dead in every app wrapped in `<CopilotKit>`. A non-explicit
(`hasExplicitThreadId={false}`) threadId is now overridable; a genuine
caller-supplied threadId stays controlled.
- **"+ New" resets the chat.** `CopilotChat` clears messages when
switching to a fresh non-explicit thread so the welcome screen shows
instead of the prior thread's messages — including when a still-loading
`/connect` is superseded by the switch (the aborted connect's stale
snapshot is dropped instead of repopulating the view).
- **No upsell flash/stick.** `CopilotDrawer` treats a pending (`null`)
license status as loading, not unlicensed, so the upgrade upsell no
longer flashes (or sticks) before the runtime reports the license.
- **In-header drawer launcher is mobile-only.**
`CopilotModalHeader.DrawerLauncher` toggles `drawerOpen`, which only
drives the off-canvas *mobile* drawer; on desktop the drawer is a
persistent in-flow panel that ignores `open`, so the launcher was a dead
no-op there. It now renders only at ≤767px. On desktop the chat's own
open/close (the toggle FAB + the header close, both lucide `X`) is the
consistent pair.
### `web-components` — theming, launcher, icons, tooltips
- **Host-theme inheritance (incl. dark mode).** Each token now resolves
`--cpk-drawer-* override → host theme var
(`--background`/`--card`/`--foreground`/`--border`/…) → built-in light
default`. The drawer follows the host app's light/dark theme by
inheritance, instead of baking light-only values (it showed a light
drawer on a dark app). Custom properties aren't reset by the `:host {
all: initial }` guard, so the standalone fallback still holds.
- **Row actions are icon buttons with instant, on-brand tooltips.**
Archive / Unarchive / Delete render inline lucide-style SVGs
(`currentColor`). Native `title` is replaced by an instant CSS tooltip
(matches the React components' Radix `delayDuration: 0`; `title`'s
show-delay is browser-fixed at ~1.5s), styled to match the standard
CopilotKit tooltip — a primary-colored bubble with an arrow, not a
surface/bordered box — and positioned to the side so the list's
scroll-overflow never clips it. The tooltip is suppressed while the
delete-confirm dialog is open (the clicked trash button keeps
`:focus-visible` otherwise, leaving its "Delete" tooltip floating over
the dialog).
- **Clipped thread names get a tooltip.** A truncated name exposes its
full text via the same instant primary-bubble tooltip as the row actions
(not the native `title`), shown only when actually clipped. The name
text moved to an inner span (so the outer can host the un-clipped
bubble) and the hovered row is z-lifted (each row is its own stacking
context from the entry-animation transform, which otherwise painted the
bubble under later rows).
- **Archived rows are muted**, no longer struck through.
- **Refetch preserves the list.** A refetch (e.g. Active↔All toggle)
keeps the known threads visible; the full loading state only renders on
the initial empty fetch.
- **Empty `footer` region hides** (it rendered as a stray bordered box
at the bottom). Driven by a `slotchange` listener, mirroring the
`memories` region.
- **Self-owned mobile launcher.** When the drawer is a closed off-canvas
mobile modal it renders its own floating "open threads" launcher, so
it's always openable without the host wiring a header button. The icon
is swappable via a `launcher-icon` **slot**, position is themeable via
`--cpk-drawer-launcher-top` / `--cpk-drawer-launcher-left` (so a host
can center it on its own header controls), and it exposes
`part="launcher"`. The richer in-header
`CopilotModalHeader.DrawerLauncher` remains an optional integration.
### Tests
- **react-core**: thread-control + non-explicit-override cases on
`CopilotChatConfigurationProvider`, the "+ New" message reset on
`CopilotChat`, license-pending gating on `CopilotDrawer`, and the
mobile-only / desktop-hidden launcher on `CopilotModalHeader`. New
regression tests verified red-green; chat + provider suites green, `tsc
--noEmit` clean, build green. (The superseded-connect *race* guard is
covered by live verification + its synchronous sibling test — an
async-timing unit test would be flaky, which the repo treats as a bug.)
- **web-components**: element tests for the icon tooltips +
confirm-dialog suppression, the clipped-name tooltip (`data-tooltip` +
placeholder), archived styling, refetch-preserves-list, footer
hide-when-empty, and mobile launcher render/open + desktop-absent.
Drawer suite green, `tsc --noEmit` clean, build + `es-check` green under
`CI=true`. (Truncation detection + tooltip stacking are layout/CSS —
verified live, not in jsdom.)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What
Layer (c) of the worker-reclamation redesign: **deploy rollover as
Railway config-as-code** — no custom rolling-restart code. Brings the
two Railway deploy-teardown knobs under SSOT so a harness-workers
redeploy is **non-lossy and capacity-stable**, completing the a/b/c
reclamation arc.
## The two knobs (currently `0`/unset on staging — which is exactly why
deploys cause the staleness dip *and* SIGKILL-lose in-flight cells)
| Knob (`serviceInstanceUpdate` field / env var) | Effect | Target |
|---|---|---|
| **`overlapSeconds`** (`RAILWAY_DEPLOYMENT_OVERLAP_SECONDS`) — the
"Teardown" card | New deployment goes Active (healthy) **before** the
old is torn down → capacity overlap across all 6 replicas → **no
capacity-floor drop** | **45** |
| **`drainingSeconds`** (`RAILWAY_DEPLOYMENT_DRAINING_SECONDS`) |
SIGTERM→SIGKILL buffer → lets layer (b)'s 90s graceful drain
finish-and-report before the kill | **180** |
Sequence: *new Active → overlap (capacity held) → SIGTERM old → drain
(in-flight cell finishes+reports) → SIGKILL.* Composes with the merged
layer (a) (reclaim backstop) + layer (b) (graceful drain).
## Why config, not code
Investigation (with research against Railway's docs + live GraphQL)
established that **`/health` only flips Active once a worker is
registered AND claiming** (`worker-health.ts` returns 200 iff `pb() &&
loopAlive() && registered()`; the `/health` socket binds *after* the
claim loop). So `overlapSeconds` genuinely holds the capacity floor —
the load-bearing promise of layer (c) — without any custom
rolling-restart code. A single-worker drain (the original B-VAL plan) is
**infeasible** on staging's 6-replica single-deployment topology
(Railway has no per-replica primitive); the real-world operation is a
full redeploy, which is what we validate.
## Changes
- **SSOT**: `overlapSeconds=45` + `drainingSeconds=180` declared for
prod+staging in `railway-envs.ts`, regenerated
`railway-envs.generated.json`, drift-gate extended (bites on
wrong/missing values).
- **RAILWAY.md**: "Deploy rollover" section — knobs, rationale, layer
a/b/c composition, GraphQL/dashboard apply instructions.
- **`orchestrator.ts`**: comments-only — corrected stale pre-layer-(b)
prose ("drain abandons / never reports / 6s grace / ~10s stop") to the
current finish-and-report behavior.
## Pre-existing test cleanups (flushed out while validating this)
- **Stale `orchestrator.test.ts` drain test** — layer (b) (#5743)
changed drain to finish-and-report but missed this assertion (it lives
outside the `src/fleet/` subset #5743's CR ran), leaving it
**deterministically RED on `main`**
(`expect(report).not.toHaveBeenCalled()` where report *is* called — the
run finishes before grace). Flipped to the finish-and-report contract,
matching the canonical `worker-loop.test.ts:2162`. The sibling abandon
test (releases *after* grace) is unchanged and still passes.
- **`probe-invoker.test.ts` timing flake** — a brittle wall-clock
assertion (`elapsed < 100ms`, only 20ms above the driver's sleep) flaked
under load (12/15 fail). Made deterministic with fake timers + a
behavioral assertion (times-out-fast invariant preserved, now stronger).
## Validation
- Config CR'd (full reviewer round, 0 P0/P1) + test-gated confirmation
on the combined branch.
- Full harness suite **3209 / 0 failed**; probe-invoker **10/10**
deterministic; orchestrator flip red-green proven; tsc + oxlint clean;
SSOT generated.json byte-canonical.
## Pending your GO (not done in this PR)
- **Apply** `overlapSeconds=45`/`drainingSeconds=180` to staging
harness-workers.
- **Validate** via a real staging redeploy: capacity floor never 0 +
zero lost cells + no false family-silence alert (before/after dashboard
screenshots).
- **Merge**.
## Follow-ups (non-blocking)
- `registered` is a one-shot boot flag (not re-derived from heartbeats);
a PB blip during boot-verify could 503 `/health` forever — but it
**fails safe** (holds the old deployment longer, never releases early).
- `showcase/scripts` emit-railway-envs-json tests need the root-declared
`oxfmt` binary to run in isolation (scripts-only `npm ci` → `spawnSync
ENOENT`); declare it in `scripts/package.json` or skip-with-diagnostic.
Refs the reclamation redesign Notion proposal.
Three stale, pre-layer-(b) prose sites still claimed drain() ABANDONS the
in-flight run and the loop skips queue.report: the drainFleetWorker JSDoc, the
runWorker stop() inline comment, AND the case-worker SIGTERM-handler block.
Layer (b) changed this — on SIGTERM the worker stops claiming, lets its
in-flight cell FINISH within the 90s grace, and REPORTS its real terminal
result via the runAbort/abortedWithoutResult discriminator; only a run that
OVERRUNS the grace is abandoned -> reclaimed by layer (a). Preserved the still-
accurate drainReason=shutdown red-side-emit soft-wind-down + deregister-vs-crash
distinction. Also refreshed the WHY-THIS-ORDER preamble (90s finish-and-report
grace, 180s platform stop / layer-c drainingSeconds). Comments only — no
behavior change.
Layer (c) deploy-rollover config for the harness-workers fleet — pure
Railway config, no custom rolling-restart code. Declares overlapSeconds=45
(capacity floor: old deployment serves until new workers register+claim, so
no staleness dip) and drainingSeconds=180 (SIGTERM->SIGKILL window >=
PLATFORM_STOP_GRACE_MS so the layer-(b) 3s+90s composed worker-drain finishes)
for both prod and staging in the railway-envs SSOT, regenerates the JSON
snapshot, and extends the harness-workers drift gate so CI fails if either
field drifts between SSOT and snapshot. Documents both knobs, their rationale
(incl. that Railway's draining default is 0s = immediate SIGKILL), the
composition with drain layers a+b, and how to apply them (GraphQL
serviceInstanceUpdate / dashboard) in RAILWAY.md.
## What
Layer (b) of the worker-reclamation redesign: **graceful worker drain**.
On a deploy/shutdown signal, a worker now stops claiming new jobs but
lets its in-flight job **finish and report** (instead of abandoning it),
keeping the lease renewed until it completes or a grace window expires.
Builds on the merged layer (a) reclaim-wins-until-cap reaper (#5738).
## How (B0–B5)
- **B2 — decouple:** a separate `runAbort` AbortController drives the
run's `ctx.abortSignal`; `drainSignal` now only stops claiming + drives
`drainReason`/heartbeat. The abandon `break` is conditional on
`abortedWithoutResult = runAbort.signal.aborted`, so a finished run
falls through to the report path.
- **B3 — lease renewal:** the heartbeat is re-keyed to the grace-expiry
signal, so a finishing job keeps renewing its lease (the reaper can't
reclaim it mid-finish); a genuinely abandoned job stops renewing and
lapses.
- **B4 — aborted-only suppression:** drain red-cell suppression fires
only for aborted/abandoned runs (gated on `errorClass==="abort"`); a
finished run reports its real terminal. (Already aborted-only by
construction post-B2; pinned with a mutation-validated test.)
- **B5 — grace budget:** `WORKER_DRAIN_GRACE_MS=90s` (env-overridable),
under the 300s lease ceiling; `PLATFORM_STOP_GRACE_MS=180s` documents
the Railway `terminationGracePeriod` requirement (3+90 < 180, ≥30s
headroom) that **layer (c) / C3** will apply.
- At grace-expiry, `runAbort.abort()` fires → the job is abandoned →
layer-(a) reclaim is the backstop.
## Review
7-agent CR + 7-agent confirmation round → **0 P0 / 0 P1**. All 5
red-green proofs empirically re-verified (each mutation flips its test
RED, restore GREEN). Fleet suite **679/679**, tsc clean, oxlint 0
errors.
- A contested grace-boundary TOCTOU (could a run finishing in the same
microtask turn the grace timer fires be spuriously abandoned?) was
**empirically confirmed SAFE** via `Promise.race` ordering —
`runAbort.abort()` lives only in the timeout leg, which loses the race
once `done` is resolvable — and regression-pinned.
- `clockHonoringSleep` is a **test-only** helper (fake-clock fix), zero
production footprint.
- Layer-(a) composition verified: a clean drain-then-finish resets
`consecutive_orphan_count` (no false poison-cap), grace-expiry abandon
follows the normal orphan→reclaim path.
## Not in this PR (gated on your GO)
- **B-VAL live-staging validation** (drain a real staging worker
mid-column, confirm zero lost cells + no staleness dip + no false
silence alert) — PENDING explicit user GO since it mutates the running
fleet.
- **Layer (c) rolling restart** — follow-on, after (b) merges + B-VAL
passes.
Refs the reclamation redesign Notion proposal.
CR P2 round on layer-(b) graceful worker drain.
P2-A (contested TOCTOU, reconciled empirically): a run that resolves with a
valid result in the same flush the grace setTimeout fires is REPORTED, not
spuriously abandoned. runAbort.abort() lives only in stop()'s Promise.race
TIMEOUT leg, which loses the race once `done` is resolvable — so a finished
run never trips the abortedWithoutResult discriminator. Reviewer crb6 was
correct; the TOCTOU is a non-bug, so no logic changed. Added a deterministic
fake-timer regression pin forcing both the run-completion timer and the grace
timer due in one advance (verified to BITE under an over-aggressive
abandon mutation), plus a clarifying comment at the discriminator.
P2-B: corrected the requestDrain() JSDoc — post-B2 the report-skip,
heartbeat-stop, and driver-cancel key on the grace-expiry signal
runAbort.signal, not stopAbort.signal (which now only stops claiming).
B5 — set the production drain grace T to 90s (was 6s). The grace stopped being
a bare TEARDOWN budget and is now the FINISH-AND-REPORT budget stop() waits for
an in-flight run to finish-and-report (layer b) before firing runAbort -> abandon
-> layer-(a) reclaim. T must BOUND a typical cell-job (so a normal in-flight job
finishes within grace) yet stay SHORTER than the platform SIGTERM->SIGKILL window
with headroom. Sized from in-repo cell-job signal: a single-service cell-job runs
~15s (light e2e-deep) up to ~200s (heavy d6-all-pills under contention); the
per-job lease ceiling is 300s. 90s covers the bulk and stays well under the lease
so a finishing job's lease never lapses. The tail (a job that cannot finish in
grace) falls back to layer (a) — grace is deliberately FINITE. 90s is a defensible
default; B-VAL confirms/retunes from staging p95. Env-overridable via
WORKER_DRAIN_GRACE_MS.
Introduce PLATFORM_STOP_GRACE_MS (180s) documenting the C3 requirement: layer-(c)
must set Railway terminationGracePeriodSeconds = 180 so the composed serial budget
DRAIN_DEREGISTER_TIMEOUT_MS (3s) + DEFAULT_WORKER_DRAIN_GRACE_MS (90s) fits with
>=30s headroom for the health-server-close + pool-shutdown remainder. The
composed-budget test now pins the relation 3s + grace < PLATFORM_STOP_GRACE_MS
(was hardcoded < 10s) and the concrete numbers.
Add a behavioral fake-clock test of the composed-budget invariant: a run that
finishes WITHIN T is reported (finish-and-report); a run still running AT
grace-expiry fires runAbort -> abandons WITHOUT a usable result -> not reported
(layer-a reclaim backstop). The raise also broke the existing wedged-driver
default-grace test (advancing a 90s fake span re-armed the 0ms-yielding heartbeat
into a runaway cascade); fixed by giving it a clock-honoring sleep so the
heartbeat stays quiet across the window and only the grace timer is crossed.
Post-B2/B3 a run can finish-and-report after a graceful drain (drain no
longer hard-cancels the run; only grace-expiry runAbort does, which is
ctx.abortSignal). The d6 red-suppression already AND-s drainReason with
ctx.abortSignal.aborted and errorClass==="abort", which makes it
aborted-only by construction — a finished run whose legitimate red is a
genuine failure (errorClass!="abort") is NOT suppressed. This adds a
pinning test for that aborted-only contract (a finished goto-error red
under drainReason=shutdown with an un-fired abort signal must still be
reported). Verified RED via a temporary mutation to the over-broad
"drainReason alone" suppression form, GREEN on the real code.
B3 (layer-b graceful drain): the heartbeat-abort previously keyed on the
DRAIN signal, stopping lease renewal at drain-start. With B2 decoupling drain
from run-abort, a still-finishing job now runs past drain-start until grace
expiry, so killing renewal at drain-start lets the lease lapse mid-finish and
the layer-(a) reaper could reclaim the row out from under the worker
(double-run / report-after-reclaim). Gate the heartbeat-abort on the
grace-expiry signal (runAbortSignal) instead: a finishing job keeps renewing
until it reports terminal; a genuinely-abandoned job (runAbort fired at
grace-expiry) still stops renewing so its lease lapses and the reaper reclaims
it. runAbortSignal defaults to drainSignal, preserving direct-call unit-test
semantics.
Alem Tuzlak