Three stale, pre-layer-(b) prose sites still claimed drain() ABANDONS the
in-flight run and the loop skips queue.report: the drainFleetWorker JSDoc, the
runWorker stop() inline comment, AND the case-worker SIGTERM-handler block.
Layer (b) changed this — on SIGTERM the worker stops claiming, lets its
in-flight cell FINISH within the 90s grace, and REPORTS its real terminal
result via the runAbort/abortedWithoutResult discriminator; only a run that
OVERRUNS the grace is abandoned -> reclaimed by layer (a). Preserved the still-
accurate drainReason=shutdown red-side-emit soft-wind-down + deregister-vs-crash
distinction. Also refreshed the WHY-THIS-ORDER preamble (90s finish-and-report
grace, 180s platform stop / layer-c drainingSeconds). Comments only — no
behavior change.
Layer (c) deploy-rollover config for the harness-workers fleet — pure
Railway config, no custom rolling-restart code. Declares overlapSeconds=45
(capacity floor: old deployment serves until new workers register+claim, so
no staleness dip) and drainingSeconds=180 (SIGTERM->SIGKILL window >=
PLATFORM_STOP_GRACE_MS so the layer-(b) 3s+90s composed worker-drain finishes)
for both prod and staging in the railway-envs SSOT, regenerates the JSON
snapshot, and extends the harness-workers drift gate so CI fails if either
field drifts between SSOT and snapshot. Documents both knobs, their rationale
(incl. that Railway's draining default is 0s = immediate SIGKILL), the
composition with drain layers a+b, and how to apply them (GraphQL
serviceInstanceUpdate / dashboard) in RAILWAY.md.
## What
Layer (b) of the worker-reclamation redesign: **graceful worker drain**.
On a deploy/shutdown signal, a worker now stops claiming new jobs but
lets its in-flight job **finish and report** (instead of abandoning it),
keeping the lease renewed until it completes or a grace window expires.
Builds on the merged layer (a) reclaim-wins-until-cap reaper (#5738).
## How (B0–B5)
- **B2 — decouple:** a separate `runAbort` AbortController drives the
run's `ctx.abortSignal`; `drainSignal` now only stops claiming + drives
`drainReason`/heartbeat. The abandon `break` is conditional on
`abortedWithoutResult = runAbort.signal.aborted`, so a finished run
falls through to the report path.
- **B3 — lease renewal:** the heartbeat is re-keyed to the grace-expiry
signal, so a finishing job keeps renewing its lease (the reaper can't
reclaim it mid-finish); a genuinely abandoned job stops renewing and
lapses.
- **B4 — aborted-only suppression:** drain red-cell suppression fires
only for aborted/abandoned runs (gated on `errorClass==="abort"`); a
finished run reports its real terminal. (Already aborted-only by
construction post-B2; pinned with a mutation-validated test.)
- **B5 — grace budget:** `WORKER_DRAIN_GRACE_MS=90s` (env-overridable),
under the 300s lease ceiling; `PLATFORM_STOP_GRACE_MS=180s` documents
the Railway `terminationGracePeriod` requirement (3+90 < 180, ≥30s
headroom) that **layer (c) / C3** will apply.
- At grace-expiry, `runAbort.abort()` fires → the job is abandoned →
layer-(a) reclaim is the backstop.
## Review
7-agent CR + 7-agent confirmation round → **0 P0 / 0 P1**. All 5
red-green proofs empirically re-verified (each mutation flips its test
RED, restore GREEN). Fleet suite **679/679**, tsc clean, oxlint 0
errors.
- A contested grace-boundary TOCTOU (could a run finishing in the same
microtask turn the grace timer fires be spuriously abandoned?) was
**empirically confirmed SAFE** via `Promise.race` ordering —
`runAbort.abort()` lives only in the timeout leg, which loses the race
once `done` is resolvable — and regression-pinned.
- `clockHonoringSleep` is a **test-only** helper (fake-clock fix), zero
production footprint.
- Layer-(a) composition verified: a clean drain-then-finish resets
`consecutive_orphan_count` (no false poison-cap), grace-expiry abandon
follows the normal orphan→reclaim path.
## Not in this PR (gated on your GO)
- **B-VAL live-staging validation** (drain a real staging worker
mid-column, confirm zero lost cells + no staleness dip + no false
silence alert) — PENDING explicit user GO since it mutates the running
fleet.
- **Layer (c) rolling restart** — follow-on, after (b) merges + B-VAL
passes.
Refs the reclamation redesign Notion proposal.
CR P2 round on layer-(b) graceful worker drain.
P2-A (contested TOCTOU, reconciled empirically): a run that resolves with a
valid result in the same flush the grace setTimeout fires is REPORTED, not
spuriously abandoned. runAbort.abort() lives only in stop()'s Promise.race
TIMEOUT leg, which loses the race once `done` is resolvable — so a finished
run never trips the abortedWithoutResult discriminator. Reviewer crb6 was
correct; the TOCTOU is a non-bug, so no logic changed. Added a deterministic
fake-timer regression pin forcing both the run-completion timer and the grace
timer due in one advance (verified to BITE under an over-aggressive
abandon mutation), plus a clarifying comment at the discriminator.
P2-B: corrected the requestDrain() JSDoc — post-B2 the report-skip,
heartbeat-stop, and driver-cancel key on the grace-expiry signal
runAbort.signal, not stopAbort.signal (which now only stops claiming).
B5 — set the production drain grace T to 90s (was 6s). The grace stopped being
a bare TEARDOWN budget and is now the FINISH-AND-REPORT budget stop() waits for
an in-flight run to finish-and-report (layer b) before firing runAbort -> abandon
-> layer-(a) reclaim. T must BOUND a typical cell-job (so a normal in-flight job
finishes within grace) yet stay SHORTER than the platform SIGTERM->SIGKILL window
with headroom. Sized from in-repo cell-job signal: a single-service cell-job runs
~15s (light e2e-deep) up to ~200s (heavy d6-all-pills under contention); the
per-job lease ceiling is 300s. 90s covers the bulk and stays well under the lease
so a finishing job's lease never lapses. The tail (a job that cannot finish in
grace) falls back to layer (a) — grace is deliberately FINITE. 90s is a defensible
default; B-VAL confirms/retunes from staging p95. Env-overridable via
WORKER_DRAIN_GRACE_MS.
Introduce PLATFORM_STOP_GRACE_MS (180s) documenting the C3 requirement: layer-(c)
must set Railway terminationGracePeriodSeconds = 180 so the composed serial budget
DRAIN_DEREGISTER_TIMEOUT_MS (3s) + DEFAULT_WORKER_DRAIN_GRACE_MS (90s) fits with
>=30s headroom for the health-server-close + pool-shutdown remainder. The
composed-budget test now pins the relation 3s + grace < PLATFORM_STOP_GRACE_MS
(was hardcoded < 10s) and the concrete numbers.
Add a behavioral fake-clock test of the composed-budget invariant: a run that
finishes WITHIN T is reported (finish-and-report); a run still running AT
grace-expiry fires runAbort -> abandons WITHOUT a usable result -> not reported
(layer-a reclaim backstop). The raise also broke the existing wedged-driver
default-grace test (advancing a 90s fake span re-armed the 0ms-yielding heartbeat
into a runaway cascade); fixed by giving it a clock-honoring sleep so the
heartbeat stays quiet across the window and only the grace timer is crossed.
Post-B2/B3 a run can finish-and-report after a graceful drain (drain no
longer hard-cancels the run; only grace-expiry runAbort does, which is
ctx.abortSignal). The d6 red-suppression already AND-s drainReason with
ctx.abortSignal.aborted and errorClass==="abort", which makes it
aborted-only by construction — a finished run whose legitimate red is a
genuine failure (errorClass!="abort") is NOT suppressed. This adds a
pinning test for that aborted-only contract (a finished goto-error red
under drainReason=shutdown with an un-fired abort signal must still be
reported). Verified RED via a temporary mutation to the over-broad
"drainReason alone" suppression form, GREEN on the real code.
B3 (layer-b graceful drain): the heartbeat-abort previously keyed on the
DRAIN signal, stopping lease renewal at drain-start. With B2 decoupling drain
from run-abort, a still-finishing job now runs past drain-start until grace
expiry, so killing renewal at drain-start lets the lease lapse mid-finish and
the layer-(a) reaper could reclaim the row out from under the worker
(double-run / report-after-reclaim). Gate the heartbeat-abort on the
grace-expiry signal (runAbortSignal) instead: a finishing job keeps renewing
until it reports terminal; a genuinely-abandoned job (runAbort fired at
grace-expiry) still stops renewing so its lease lapses and the reaper reclaims
it. runAbortSignal defaults to drainSignal, preserving direct-call unit-test
semantics.
Decouple 'stop claiming new jobs' from 'abort the in-flight run': drain()
no longer fires the run's abortSignal. The run's hard cancel is now a
separate grace-expiry signal (runAbort), so a run seconds from done
finishes within grace and is reported instead of being abandoned to the
sweeper. The abandon break is now conditional on abortedWithoutResult
(runAbort fired at grace-expiry → no usable result); a finished run falls
through to the report path. A wedged run that overruns grace is still cut
(runAbort.abort() in stop()'s timeout leg) and abandons → layer (a)
reclaim is the backstop.
Reconcile the drain tests that encoded the old abort-on-drain coupling to
the new contract (drain no longer fires ctx.abortSignal; the grace-expiry
abort fires at grace, exercised via a short WORKER_DRAIN_GRACE_MS).
## What
Removes the `@copilotkit/bot-store-redis` and
`@copilotkit/bot-store-postgres` adapters, added in #5613. The pluggable
`StateStore` interface and the in-memory `MemoryStore` default stay;
durable backends can be reintroduced as a follow-up when there's a
concrete need.
## Why
Both adapter packages were merged but **never published to npm**, so
there are no consumers — removal is a clean delete with no migration or
deprecation cycle. Trimming the surface keeps the bot persistence story
to one well-tested in-memory default plus a documented "bring your own
`StateStore`" path.
## Changes
- Delete `packages/bot-store-redis` and `packages/bot-store-postgres`.
- Revert the `bot` release scope and the release drift-guard test to
`bot + bot-ui` (count 2).
- Strip the Redis dependency, `demo:restart` script, restart demo,
`docker-compose.yml`, and `REDIS_URL` env from `examples/slack`.
- Rewrite the bot persistence/transcripts docs around "MemoryStore
default + implement the `StateStore` interface yourself for durability"
(unrelated enterprise/Helm Redis/Postgres docs untouched).
## Verification
- `nx run @copilotkit/bot:build` — pass
- `nx run @copilotkit/bot:test` — 121/121 pass
- Release drift guard — pass at count 2
- Repo-wide grep for
`bot-store-redis|bot-store-postgres|createRedisStore|createPostgresStore`
— zero matches
## What
Replaces the queue reaper's delete-wins behavior for stale long-expired
orphaned in-flight rows with **reclaim-wins-until-cap**: an orphaned row
is re-queued (not deleted) up to a bounded number of CONSECUTIVE
re-orphans before final claim-delete. This is layer (a) of the
worker-reclamation + graceful-rollover redesign — it makes a worker
bounce mid-column non-lossy (the column's in-flight work is reclaimed by
surviving workers instead of dropped).
## How
- **Reclaim-wins-until-cap** in `runSweepExpired`
(`showcase/harness/src/fleet/queue-client.ts`): stale long-expired
orphaned in-flight rows are re-queued; the prior delete-wins carve-out
is inverted.
- **Consecutive-orphan budget** (`consecutive_orphan_count`, migration
`1779990400`): the reclaim cap (`MAX_RECLAIM_ATTEMPTS=3`) is scoped to
*consecutive* sweeper re-orphans — incremented only on the sweeper
re-queue path, **reset to 0 on terminal `done`/`failed`**. Peer-worker
expired-lease *steals* bump the lifetime `reclaim_count` (retained as a
dashboard diagnostic) but do NOT consume the reclaim budget.
- **Stale-age re-anchoring** (`requeued_at ?? created`, migration
`1779990300`): a reclaimed row's staleness clock restarts so it isn't
immediately re-expired.
## Review
7-agent CR + 7-agent confirmation round → **0 P0 / 0 P1**. Highlights:
- **Red-green GENUINE (empirically re-verified):** P1-A — a long-lived
job with lifetime `reclaim_count=3` but `consecutive_orphan_count=0` is
RE-QUEUED on a fresh orphan (pre-fix: wrongly deleted); P1-B — low
boundary at `MAX-1=2` re-queues, mutation-killed. Both non-tautological
(the test fake bumps `consecutive_orphan_count` only on reclaim, never
on steal; a dedicated pin test confirms steals don't consume budget).
- **No regression / terminates / fails-safe:** 248/248 tests, tsc +
oxlint clean; reclaim loop terminates at 3 consecutive orphans →
claim-delete; queue cannot grow unbounded; SweepResult metrics +
`reclaim_count` dashboard consumers (run-view `jobs.reclaimed`,
family-silence) unbroken.
- **Migration `1779990400`** additive, nullable, default-0;
pre-migration rows evaluate to 0 → safe re-queue path.
## Follow-on (not in this PR)
- Layer (b) graceful worker drain and layer (c) rolling restart are the
remaining phases of the redesign — see the Notion proposal:
https://app.notion.com/p/38b3aa381852817bacf5c9cda1f11cc0🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What
Tightens the **D4/BE chat-roundtrip probe gate** from `text.length > 0`
to a **provenance check**. A turn now only counts green if the assistant
text came from the `[data-testid="copilot-assistant-message"]` container
— not from the `<body>` fallback scrape (which picks up static page
chrome).
## Why
This is the gate weakness that **masked the BIA prod outage**: a dead
agent (RUN_STARTED→RUN_FINISHED with zero `TEXT_MESSAGE`) left the page
chrome on screen, the `<body>` scrape returned non-empty text, and the
old `text.length > 0` gate reported **green** while the agent was
actually broken. The new gate threads a `fromAssistantContainer` flag
(set true only on the testid-container read path) so a body-scrape-only
result fails (`!fromAssistantContainer` → red). The same guard is
applied at L4 (tool-rendering).
## Review
**7-agent CR → 0 P0 / 0 P1.** Highlights:
- **Red-green genuine** (empirical): old gate false-passes BIA (1/52
RED); post-fix 52/52. 3-way guard (green /
empty-via-unchanged-length-guard / BIA-via-provenance).
- **Blast radius: no false-fail risk** — all 20 integrations'
`agentic-chat` + `tool-rendering` demos use the shared v2 `CopilotChat`
which mounts the testid unconditionally; aimock `d4/<slug>/chat.json`
fixtures are text-bearing for all 20. Verdict: **ship as a blocking
gate, no soak.**
- **No lost coverage** — the `<body>`-scrape fallback *code* is retained
(diagnostics); only the gate *semantics* narrowed. No D4-probed cell
relied on the fallback to pass.
- **No cross-family regression** — change is confined to
`d4-chat-roundtrip.ts`; other probe drivers untouched; 169/0 tests pass;
tsc clean.
## Notes / follow-ups (non-blocking, P2)
- A *future* probed demo using a `CopilotChatAssistantMessage`
children-render override (which doesn't emit the testid) would
false-fail — worth a one-line note near the fallback comment if that
pattern is ever added to a D4 route.
Part of the 2026-06-26 incident remediation (companion to #5733); this
gate would have surfaced the BIA failure that the dashboard masked.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What
Bundles the four code fixes from the 2026-06-26 showcase prod incident
remediation. (Prod config reconciles — fleet image promotes, worker
scaling, CVDIAG keys — were applied directly to Railway and are not
code; the post-incident debugging lesson shipped separately in #5727.)
## Incident context (why these exist)
Prod's coverage dashboard cascaded to a wall of red. Root cause was
**staleness, not a feature break**: the harness worker pool was starved
(a deploy-bounce + under-provisioning), so probe sweeps couldn't
complete within the staleness windows → cells aged out → rendered
red/BE✗ even though the apps were fine. Compounded by a `llamaindex`
container crash (per-request log flood tripping Railway's log cap) and a
2-day-stale prod image fleet. See #5727 for the debugging lesson, and
the Notion proposal below for the durable worker-reclamation redesign.
## The four fixes
1. **Dashboard stale-while-revalidate**
(`shell-dashboard/depth-chip.tsx`) — a passing (green) cell keeps its
color + a non-destructive `⟳` refreshing affordance during a re-probe
instead of flapping to grey; never-run stays grey; failure/regression
keeps its color with no spinner; staleness bound preserved. + a11y
(`role=status`, flag-gated regression label, `motion-reduce`).
2. **Family-silence grace window** (`harness/fleet/control-plane/*` +
dashboard banner) — a normal harness deploy's post-bounce worker drain
no longer fires a false "worker family silent" banner/alert (suppressed
within a `2×period` bounce grace keyed on the freshest worker
registration), while genuine silence beyond the window still fires on
both the Slack-alert and banner paths. No silent failure (PB-down →
grace disabled → real outages still alert).
3. **llamaindex per-request log gate**
(`integrations/llamaindex/.../route.ts`) — gates the chatty
`[copilotkit/route]` per-request logs behind `SHOWCASE_ROUTE_DEBUG`
(default off) so they can't flood Railway's log cap and kill the
container. Error logging untouched.
4. **SSOT worker-provisioning** (`scripts/railway-envs.ts` + drift gate)
— brings harness-workers replica provisioning under SSOT so prod/staging
can't silently drift. Models the **effective** field
(`multiRegionConfig.<region>.numReplicas`) +
`BROWSER_POOL_MAX_CONTEXTS`, declares current reconciled reality
(prod=staging=6 replicas, 40 contexts), with a CI drift-detection test.
## Review
7-agent CR round + 7-agent confirmation round → converged to **0 P0 / 0
P1**. Two post-CR fixes (SSOT effective-field model after discovering
`multiRegionConfig` is the live knob; re-anchoring a grace-edge test to
genuinely pin the 2× boundary) each re-confirmed clean. Suites:
shell-dashboard 92 + harness 63 + scripts 12 = **167 passed, 0 failed**.
No new tsc errors (pre-existing only).
## Notes / follow-ups (non-blocking)
- llamaindex log-gate is logging-only and untested (acceptable); the
chatty pattern exists in ~16 integrations incl. the gold standard — a
**fleet-wide log-flood gate** is a worthwhile follow-up.
- `freshestBounceMs` has two equivalent impls (harness `parseIso` /
dashboard `Date.parse`) — flagged for future lockstep.
- The D4/BE probe gate weakness (asserts `text.length>0`, which masked
the BIA outage) is being addressed in a **separate** PR
(`fix/d4-gate-tighten`).
## Refs
- Post-incident debugging lesson: #5727
- Worker reclamation + graceful-rollover redesign (Notion proposal):
https://app.notion.com/p/38b3aa381852817bacf5c9cda1f11cc0🤖 Generated with [Claude Code](https://claude.com/claude-code)
P1-A (cap semantics): the MAX_RECLAIM_ATTEMPTS cap was keyed on
`reclaim_count`, a LIFETIME tally bumped by BOTH the sweeper re-queue
path AND the peer-worker expired-lease steal (claim CAS). A long-lived
job that accrues benign peer steals could exhaust its 3-budget and then
get claim-DELETED on its first real orphan rather than re-queued.
Fix: introduce a dedicated `consecutive_orphan_count` column (migration
1779990400) that is bumped ONLY by the sweeper re-queue path in the
fleet-claim release CAS, and reset to 0 on every terminal done|failed
release. The peer-worker steal (claim CAS wasExpiredSteal branch) does
NOT touch this counter. The reaper's cap check now uses
`consecutive_orphan_count` instead of `reclaim_count`. `reclaim_count`
is left intact as the lifetime dashboard diagnostic (jobs.reclaimed).
P1-B (low boundary): adds a test at consecutive_orphan_count = MAX-1
(= 2) asserting the row is RE-QUEUED, not deleted. The off-by-one
mutation `>= MAX` -> `>= MAX-1` causes this test to go RED.
Test-fake honesty: `makeReclaimClaim`'s claimJob now explicitly models
the steal-bump on `reclaim_count` (matching the real hook) while
intentionally NOT bumping `consecutive_orphan_count`, and adds an
explicit pin test confirming steals do not consume the reclaim budget.
JSDoc on MAX_RECLAIM_ATTEMPTS updated to describe the correct semantics:
consecutive re-orphans scoped by sweeper re-queue, reset on terminal.
Red-green proof:
- P1-A RED: revert cap to reclaim_count → "P1-A CAP SCOPE" fails with
`expect(undefined).toBeDefined()` (job deleted instead of re-queued)
- P1-A GREEN: consecutive_orphan_count cap → test passes (re-queued)
- P1-B RED: mutate `>= MAX` to `>= MAX-1` → low-boundary test fails
- P1-B GREEN: revert mutation → low-boundary test passes
Suite: 145 queue-client + 103 producer = 248 total, all green.
_syncNameClipping toggled name-clipped on .row-name, but the stacking fix in
styles.ts targets .row.name-clipped:hover — a different element — so the z-index
lift never matched and clipped-name tooltips still painted under later rows
(each row is its own transform stacking context). Stamp the flag on the owning
.row too; the tooltip bubble stays scoped to .row-name:hover so a row-action
hover never surfaces it. Adds a red-green contract test asserting the class
lands on both the row-name and the row.
Addresses MikeRyanDev review on #5707.
Layer (a) of the worker reclamation+rollover redesign: make a worker bounce
non-lossy. The reaper's long-expired carve-out (G1d) used to claim-DELETE an
orphaned in-flight (claimed/running) row whose lease expired beyond its
family's stale window AND whose created-age was past that window —
`reclaimed=0, expiredPending++` — silently dropping work an abrupt bounce
(SIGKILL past grace / OOM / crash) left mid-flight.
Invert it to RECLAIM-WINS-UNTIL-CAP: re-queue the orphan to pending (it
re-runs; idempotent probes make at-least-once safe) until its durable
`reclaim_count` reaches MAX_RECLAIM_ATTEMPTS (3), only then claim-deleting a
row that keeps re-orphaning so a poison job cannot loop forever.
The carve-out existed to dodge an honesty bind: re-queueing a `created`-stale
row emitted a "back in flight" gray the next sweep falsified by claim-deleting
it off the renewal-immune `created` age. Dissolve the bind with a new
`requeued_at` column (migration 1779990300) the release CAS stamps on every
pending re-queue; both stale phases now age off `staleAgeAnchorMs`
(`requeued_at ?? created`), so a reclaimed row is genuinely young again and the
next sweep does not delete it. `reclaim_count` (migration 1779990200) is reused
as the attempt counter — no second tally to drift.
Red-green proven on the real reaper (queue-client.test.ts): a stale-aged
long-expired orphan below the cap goes RED (deleted, reclaimed=0,
expiredPending=1) on delete-wins and GREEN (re-queued, reclaimed=1,
expiredPending=0, requeued_at stamped) on reclaim-wins; plus an attempt-cap
deletion test and a next-sweep no-falsification test.
The L3 chat-roundtrip gate only asserted `text.length > 0`, which the
`<body>` fallback scrape can satisfy with static page text (nav links,
footer copy, demo blurb) trailing the sent message. When the agent run
finished with ZERO assistant content (RUN_STARTED -> RUN_FINISHED, no
TEXT_MESSAGE) the assistant-message bubble never rendered, so the probe
fell back to scraping <body> and false-PASSED on incidental page chrome —
the mechanism that masked the BIA outage (dead agent reported green).
Track the provenance of the captured response and require it to come from
the [data-testid="copilot-assistant-message"] container (a genuine
assistant turn — the DOM-layer equivalent of RUN_FINISHED + a non-empty
TEXT_MESSAGE), not the body fallback. Apply the same guard to L4 so weather
content must also come from a real assistant turn. The instrumented probe
sends the aimock fixture headers and renders real content into the
container, so this does not affect a working agent; it only rejects the
empty-turn false-pass.
Adds a red-green regression test reproducing the BIA false-pass.
A long thread name is clipped with an ellipsis; expose the full text via the
same instant primary-bubble tooltip as the row actions (not the native `title`),
shown only when the name is actually truncated. The name text moved to an inner
span so the outer `.row-name` can host the un-clipped bubble, and the hovered
row is z-lifted so the bubble isn't painted under later rows (each row is its
own stacking context from the entry-animation transform).
The in-header thread-drawer launcher toggles `drawerOpen`, which only drives the
off-canvas MOBILE drawer; on desktop the drawer is a persistent in-flow panel
that ignores `open`, so the launcher was a dead no-op there (it appeared to do
nothing on click). Gate it on a reactive mobile viewport check so it renders
only at ≤767px. On desktop the chat's own open/close — the toggle FAB + the
header close (both lucide X) — is the consistent pair; nothing masquerades as a
chat control.
Row-action tooltips used a surface background + border, which read as a button.
Restyle to the standard CopilotKit tooltip: primary bubble, primary-foreground
text, an arrow, no border. Also suppress the tooltip while the delete-confirm
dialog is open — the clicked trash button keeps :focus-visible, which otherwise
left its "Delete" tooltip floating over the dialog.
CopilotChatConfigurationProvider treated any `threadId` prop as controlled, so
the auto-minted non-explicit threadId that <CopilotKit>'s v1 bridge seeds
blocked the drawer's imperative setActiveThreadId/startNewThread — thread
switching and "+ New" were dead in every app wrapped in <CopilotKit>. A
non-explicit (`hasExplicitThreadId={false}`) threadId is now overridable; a
genuine caller-supplied threadId stays controlled.
Also, surfaced during the langgraph-js de-fork validation:
- CopilotChat clears messages when switching to a fresh non-explicit thread
("+ New") so the welcome screen shows instead of the prior thread — including
when a still-loading /connect is superseded by the switch.
- CopilotDrawer treats a pending (null) license status as loading, not
unlicensed, so the upgrade upsell no longer flashes or sticks before the
runtime reports the license.
Polish on top of the shipped <copilotkit-drawer> (#5701), surfaced by the first
real de-fork pilot (langgraph-js):
- Theming follows the host app's light/dark theme: each token resolves
override -> host theme var (--background/--card/...) -> built-in light default,
instead of baking light-only values (drawer was light on a dark app).
- Row actions are icon buttons (inline lucide-style SVG, currentColor) with
instant tooltips. Native title is replaced by a CSS tooltip (matches the
react components' delayDuration:0; title's show-delay is browser-fixed) and is
positioned to the side so the list's scroll-overflow never clips it.
- Archived rows are muted, no longer struck through.
- A refetch (e.g. Active<->All filter toggle) keeps the known list visible;
the full loading state shows only on the initial empty fetch.
- The reserved footer region hides when nothing is slotted into it (it rendered
as an empty box at the bottom); mirrors the memories region.
- Mobile gets a self-owned floating launcher so the drawer is always openable
with no host header wiring. Its icon is swappable via a launcher-icon slot and
its position via --cpk-drawer-launcher-top/left (host can center it on its own
header controls); exposes part="launcher".
Adds element tests for icons+tooltips, archived styling, refetch-preserves-list,
footer hide-when-empty, and the mobile launcher (render + open + desktop-absent).
## Summary
Two pre-existing isolation-CLI bugs that #5730 (slot-reaping) surfaced
and correctly deferred as out-of-subject — both real, now fixed with
real-surface red-green.
1. **Port-conflict guard silent-failure** — `_slot_ports_free` consumed
`_slot_offset_ports` via process substitution (`done < <(...)`), so a
`die` on an out-of-range/non-numeric slot exited only the subshell; the
loop read zero ports and the function returned 0 ("all free"), silently
defeating the port-conflict guard for a bad slot. Now captures into a
variable with `|| die` so the failure propagates to the caller.
2. **`--isolate` arg-parsing** — `--isolate <name> <slug>` (name before
slug) mis-parsed the name as the slug then died "Unexpected argument"; a
bare `--isolate=` silently auto-picked. Now order-independent (deferred
`pending_iso_name` resolution after the parse loop), rejects an empty
value loudly, and `--isolate=<name>` (non-numeric) binds an explicit
name. `--isolate=<N>` numeric pin unchanged. **Intended behavior
change:** `--isolate=<name>` previously died in the picker; it now binds
a name.
## Tests
`bats showcase/scripts/__tests__/` = 170/0 — real slot dirs, real
bad-slot input, real parser end-to-end (no mocks). CR converged round 1
(zero bucket-a), Procedure 3 promotion audit clean.
## Deferred to a potential follow-up (pre-existing "port-enumeration
completeness")
CR surfaced, and correctly kept out of scope: `_slot_offset_ports`
hardcoded/incomplete infra-port list (phantom 8081, misses other
published ports); apply_isolation port-offset regex matches quoted
short-form only; `.last-test-ts` not repointed per-isolate (concurrent
clobber); `--repeat` treats a following flag as its count.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The StateStore interface and the in-memory MemoryStore default remain;
durable backends can be reintroduced as a follow-up. Both adapter packages
were merged in #5613 but never published to npm, so removal is a clean
delete with no consumer impact.
- Delete packages/bot-store-redis and packages/bot-store-postgres.
- Revert the bot release scope and drift guard to bot + bot-ui.
- Strip the Redis dep, demo:restart script, restart demo, docker-compose,
and REDIS_URL env from examples/slack.
- Rewrite the bot persistence/transcripts docs around "MemoryStore default
+ implement the StateStore interface yourself for durability".
## Release bot v0.1.0
**Scope:** `bot` | **Bump:** `minor`
---
### How this release process works
1. **This PR was created automatically** by the "release / create-pr"
workflow.
It bumped the `bot` packages to `0.1.0`
and generated AI-enhanced release notes.
2. **CI runs on this PR** — the full test suite (unit tests, lint, type
checks, build)
must pass before merging. This is the review gate.
3. **Review the release notes** in `release-notes.md` in this PR.
If a Notion draft was created, you can edit the release notes there
before merging.
4. **When this PR is merged**, the `release / publish` workflow
automatically:
- Builds all packages
- Publishes the `bot` packages to npm at version `0.1.0`
- Creates git tag `bot/v0.1.0`
- Creates a GitHub Release with the final release notes
### Before merging
- [ ] CI is green (tests, lint, types, build)
- [ ] Version bumps look correct
- [ ] Release notes are accurate (edit in Notion if a draft was created)
---
> **Do not merge until CI is fully green.** The full test suite runs
automatically on this PR.
## What
Adds anonymous **`oss.bot.*`** usage telemetry to the `@copilotkit/bot`
SDK — a **configured → started → agent_run** adoption funnel that
answers "what are people doing with the bot packages?" (which adapters,
which store backends, how they iterate on config, where they fail) —
without knowing *who* the customer is.
Design spec: [Bot SDK Telemetry — oss.bot.* Adoption
Funnel](https://app.notion.com/p/38b3aa3818528177a3abfda9ce35bdbc) ·
Plan: [Implementation
Plan](https://app.notion.com/p/38b3aa38185281d5a09cfea114df8897)
## Events (all 100%, anonymous, metadata-only)
| Event | When | Notable props |
|-------|------|---------------|
| `oss.bot.configured` | `createBot()` returns (repeats per restart →
iteration signal) | `platforms`, `store`, `toolsCount`, `hasComponents`,
… |
| `oss.bot.started` | `bot.start()` connected ≥1 adapter | `platforms`,
`startedCount`, `failedCount`, handler flags |
| `oss.bot.start_failed` | an adapter threw during start | `platform`,
`errorClass` |
| `oss.bot.agent_run` | a successful agent run | `platform`,
`durationMs`, `toolCallCount`, `iterations`, `interrupted` |
| `oss.bot.agent_run_failed` | an agent run errored | `platform`,
`errorClass`, `stage` |
Catalog + property reference: `packages/bot/telemetry-events.json`.
## Design
- **Anonymous.** Bot deployments carry no `telemetry_id`/license/API
key, so events are stitched by a persisted `anonymous_id` (durable
`StateStore` → project-local cache file → per-process UUID) plus a
per-`createBot` `bot_session_id`. Reuses `lambdaClient.send` from
`@copilotkit/shared` (no new deps).
- **Zero-config.** Works out of the box; **no new env vars**. Reads only
the pre-existing optional `COPILOTKIT_TELEMETRY_URL` (override) and
`COPILOTKIT_TELEMETRY_DISABLED` / `DO_NOT_TRACK` (opt-out), and
`NODE_ENV`/`VITEST` for the `environment` tag + test suppression.
- **Fire-and-forget.** `capture()` never throws into or blocks the host
app; all dispatch/fetch/fs failures are swallowed.
- **PII guardrails.** Snapshots are scalars/counts only (never
adapter/store option objects → no token/connection-string leakage);
errors are mapped to a bounded `errorClass` category
(`auth`/`network`/`timeout`/`validation`/`unknown`) — never a raw
message or stack; no message text, user ids, or channel names anywhere.
**Sink side** (`oss.bot.` allow-list prefix + `anonymous_id` → PostHog
`distinct_id`) shipped separately in oss-path-to-production PR #175
(already merged).
## Testing
**Unit (TDD, red→green per module):**
- `sanitize-error.test.ts` — category mapping + a "never leaks the
message" secret-redaction assertion (2)
- `install-id.test.ts` — durable-store persistence, file persistence,
unwritable-dir fallback (3)
- `bot-telemetry.test.ts` — global-props shape, disabled no-op,
never-throws-on-send-reject, event-name set (4)
- `events-catalog.test.ts` — drift guard: catalog keys == emitted event
names (1)
- `run-loop.test.ts` — new `{iterations, interrupted}` return value
(interrupt + normal paths)
- `create-bot-telemetry.test.ts` — mocked-telemetry wiring: configured
snapshot, started/start_failed (with `xoxb-SECRET` redaction assertion),
agent_run
**End-to-end:**
- `e2e-telemetry.test.ts` — drives the **real `BotTelemetry`** through
`createBot → start → runAgent` (only the network boundary is stubbed),
asserts the `configured → started → agent_run` payloads carry
`anonymous_id`/`bot_session_id` and **runs with an empty telemetry env**
(proves zero-config; asserts no license token attached).
**Full suite (authoritative, on a real `pnpm install` in the integration
worktree):**
- `tsc --noEmit` → 0 errors · `@copilotkit/bot` vitest → **26 files /
141 tests pass**
**In-session manual smoke (real wire bytes):** stubbed
`globalThis.fetch`, drove a real bot through one turn, captured the
actual serialized POSTs:
```
sink URL (no env var set → default): https://telemetry.copilotkit.ai/ingest
oss.bot.configured {"properties":{"platforms":["fake"],"adapterCount":1,"store":"memory",...},
"global_properties":{"anonymous_id":"74b3c598-…","bot_session_id":"7cddd757-…","environment":"production"},
"package":{"name":"@copilotkit/bot","version":"0.0.3"},"ts":1782502088}
oss.bot.started {"properties":{"platforms":["fake"],"startedCount":1,"failedCount":0,"hasMentionHandler":true,...}, "global_properties":{ same anonymous_id + bot_session_id }}
oss.bot.agent_run {"properties":{"platform":"fake","durationMs":0,"toolCallCount":0,"iterations":1,"interrupted":false}, "global_properties":{ same anonymous_id + bot_session_id }}
```
All three share one `anonymous_id` + `bot_session_id` (funnel
stitching), default sink URL with no env var set, no license/key
anywhere.
**No new env vars (verified):** `git diff origin/main --
packages/bot/src/create-bot.ts packages/bot/src/thread.ts | grep
process.env` → none; wiring code reads no env var directly.
**Code review:** one focused reviewer pass on the full diff — no
blocker/high findings; PII, fire-and-forget, zero-config, and
no-regression axes all confirmed clean. Two low/nit findings applied
(dropped `TypeError`→`validation` mis-categorization; documented the
interrupt→resume `agent_run` double-count in the catalog).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The harness-workers SSOT modeled only the top-level numReplicas, whose drift
gate watched a field that does not drive the live replica count. harness-workers
is single-region (us-west2); Railway derives the live count from
multiRegionConfig.us-west2.numReplicas.
- WorkerProvisioning gains effectiveReplicas (= multiRegionConfig.us-west2.
numReplicas), the authoritative field the drift gate now asserts. Top-level
numReplicas is retained as a documented mirror.
- Declared values reflect current reconciled reality (verified live via the
Railway GraphQL environment.config staged-config read): prod and staging both
at effectiveReplicas=6 (parity achieved by B-reconcile scaling prod 3 -> 6 in
both the top-level field and multiRegionConfig). BROWSER_POOL_MAX_CONTEXTS is
40 on both envs (verified live).
- Regenerated railway-envs.generated.json; drift-gate test asserts
effectiveReplicas (RED-GREEN proven). RAILWAY.md documents multiRegionConfig
as the effective knob and the achieved parity.
Adds a monorepo-invariant test that scans sibling bot-* adapter packages for their declared `platform` literal and fails if any is missing from normalizePlatform's allow-list. It immediately caught the new bot-teams adapter (platform "teams") that landed on main — now added to the allow-list.
Add WorkerProvisioning interface and workerProvisioning field to the
harness-workers ServiceEntry in railway-envs.ts. Declares current live
reality: prod=3 replicas, staging=6 replicas, BROWSER_POOL_MAX_CONTEXTS=40
per worker (both envs).
Worker model: 1-worker-per-replica (Railway runs one process per
container, keyed on HOSTNAME). HARNESS_POOL_COUNT is informational
only — not a fork factor. Authoritative concurrency knob per worker is
BROWSER_POOL_MAX_CONTEXTS.
Comments flag: staging config-field drift (Railway field=2, live=6) as
a follow-up item; and the prod/staging parity decision (prod=3 vs
staging=6) as deliberately deferred.
Extends emit-railway-envs-json.ts to emit workerProvisioning into the
generated JSON snapshot. Adds drift-gate test
(harness-workers-provisioning.test.ts) that fails if SSOT numReplicas
diverges from the committed JSON snapshot — no live Railway API calls.
Red-green-red-green verified locally.
Updates RAILWAY.md with the 1-worker-per-replica model, declared
values, manual apply procedure, and drift-gate reference.
The existing tooling is verify-only for numReplicas; applying a replica
count change to Railway remains a manual operation (Railway Dashboard or
GraphQL API).
The llamaindex copilotkit API route logged '[copilotkit/route] POST ...'
and '[copilotkit/route] Response status: 200' unconditionally on EVERY
request. Under d6 probe fan-out this exceeded Railway's 500-logs/sec cap
('Messages dropped' -> 'Stopping Container'), killing the replica.
Gate both per-request console.log lines behind a SHOWCASE_ROUTE_DEBUG env
flag (default off). Module-load logs and error logging are unchanged.
This chatty pattern is shared/copied across ~16 integrations (including
the langgraph-python gold standard); this commit scopes the fix to
llamaindex. The others are flagged as follow-up.
A normal harness deploy rebuilds the shared showcase-harness image and
bounces the pool workers (PR #5715). Immediately after the bounce the
workers re-register, the producers re-arm, and every family is mid-sweep:
lastSuccessAt still points at the pre-bounce success, so it reads stale
against the silence thresholds (banner 2x period, Slack alert 3x period +
3 consecutive ticks). The result was a FALSE "worker family X has not
completed successfully" banner AND Slack family-silence alert during the
expected post-bounce drain window.
Fix: a bounce-keyed grace window. The freshest worker registered_at across
the /api/runs workers strip is the fleet's most-recent bounce instant
(independent of CP boot — a worker can bounce while the CP stays up). While
now - bounce < 2 x period, a family with no success yet is DRAINING, not
silent, so neither the §7.4 banner, the §7.3 cell glyph, nor the §9 Slack
alert flags it. Beyond the window with still-no-success, genuine silence
fires exactly as before.
The determination lives in two surfaces (server monitor for the Slack
alert; client isFamilySilent for the banner + glyph), so both now consume
the same new SSOT field (WorkerView.registeredAt) and the same 2x-period
grace constant, keeping them consistent.
- run-view.ts: project registered_at -> WorkerView.registeredAt (server)
- family-silence-monitor.ts: BOUNCE_GRACE_PERIOD_MULTIPLIER + freshest-bounce
grace gate, keyed off body.workers
- worker-runs-context.tsx: freshestBounceMs + bounceAtMs grace arg on
isFamilySilent; banner + cell glyph pass it
- ops-api.ts: WorkerView.registeredAt on the client DTO
A GREEN coverage cell flapped green -> grey -> green every time its probe
job's worker lease lapsed and the control-plane sweeper re-queued the job
(worker-reclaimed-pending). The data layer already preserves the
last-known-good colour in chipColor while surfaceState flips to "pending",
but depth-chip.tsx's `if (pending)` branch did a destructive grey
early-return that never read chipColor.
Split the pending branch:
- prior-good (chipColor is a real colour, or depth > 0): render the normal
coloured chip via the default branch's exact ternary (threading regression
through) PLUS a non-destructive refreshing affordance -- a corner ⟳ glyph
and a subtle pulsing ring in the chip's own colour, with
data-refreshing="true" / data-has-prior="true". Colour preserved.
- no-prior (never-run / first load: gray + depth 0): keep today's honest grey
⟳ chip, now tagged data-has-prior="false".
The unreachable red ⚡ overlay and the "failure never masked" gate are
untouched; red/regression cells pass through with no spinner. The refreshing
cue is conveyed by shape (⟳) + motion (ring), never by colour alone, and the
pulse/spin respect prefers-reduced-motion (motion-reduce:animate-none), so the
static ⟳ carries the meaning for colour-blind and reduced-motion operators.
Renderer-only change; unified-cell.tsx is the sole caller passing `pending`.
Address PR review: bound the free-form adapter platform label to slack|discord|telegram|whatsapp|custom (no tenant/project leakage, capped cardinality); emit oss.bot.agent_run only after the transcript-append + renderer.finish() steps succeed, with a finalize stage on oss.bot.agent_run_failed for late failures.
--isolate <name> <slug> (name before slug) mis-parsed the name as the
slug, then died "Unexpected argument"; a bare --isolate= silently
fell through to auto-pick. Defer the ambiguous post---isolate token to
pending_iso_name and resolve it after the parse loop (slug present =>
token was the name; no slug => token was the slug); reject an empty
--isolate= loudly; --isolate=<name> (non-numeric) now binds an explicit
isolate name. The --isolate=<N> numeric pin is unchanged.
_slot_ports_free consumed _slot_offset_ports via process substitution
(done < <(...)), so a die on an out-of-range/non-numeric slot exited only
the subshell — the loop read zero ports, any_held stayed 0, and the
function returned 0 ("all free"), silently defeating the port-conflict
guard for a bad slot. Capture into a variable with || die so the failure
propagates to the caller. Real-surface bats prove a bad slot now fails
loudly while valid-slot free/held behavior is unchanged.
Jordan Ritter