## What this is now
Opt-in instrumentation for the channel egress path, plus a
replay-stability guard. **Measurement and safety only — no behaviour
change to how frames are sent.**
It started as a fix too. That fix is reverted here; see below.
## The problem it measures
A streaming reply pushes one render frame per AG-UI token and awaits
each durable acceptance receipt before sending the next, so reply
latency scales with token count. Nothing recorded that, so the cost was
invisible and got attributed to the Connector Outbox's 400ms poll
instead.
```bash
COPILOTKIT_CHANNELS_RENDER_METRICS=summary # one summary per turn
COPILOTKIT_CHANNELS_RENDER_METRICS=frames # also one line per frame
```
Unset means off and no collector is allocated. Output goes to the
transport `log` callback when configured, else the console.
## What it found, measured against production
Hosted api + realtime gateway, local runtime, fixture agent emitting
exactly 120 deltas 15ms apart:
```
pushMsMean: 478.43 pushMsP50: 463 pushMsP95: 589 pushBlockedPct: 99.67
```
**478ms per round trip against an ~85ms network RTT**, so roughly 390ms
of every push is server-side work — the `lookupClerkOrgId` query, the
off-cluster ops-api entitlement call, the read-model build, and the
Redis publish. Intelligence#627 removes those for `text_delta` frames.
At 478ms a push, a 120-token reply costs ~123 round trips ≈ **59s**.
That matches the original "sometimes many seconds" report.
## Why the fix was reverted
Coalescing adjacent text deltas worked — 123 frames to 4, a production
turn from ~59s to 3.4s — and broke replay.
A turn's id is a pure function of its delivery id (`turn_<deliveryId>`,
enforced by app-api) and the adapter resets the per-turn seq counter on
every dispatch, so a redelivery replays into the **same seq space**.
app-api treats that space as an idempotency key: same payload at a seq
is `duplicate_accepted`, a different payload is
`CHANNEL_RENDER_FRAME_CONFLICT`, which nacks the delivery. Merge
boundaries depend on wall-clock arrival, so a replay cuts them
differently:
| | attempt 1 (slow, merged) | attempt 2 (redelivery, faster) |
|---|---|---|
| seq 1–3 | *merged away, never pushed* | `"a"`, `"b"`, `"c"` → INSERT ✓
|
| seq 4 | `text_delta "abcd"` | `text_delta "d"` → **CONFLICT** ✗ |
Before coalescing, deltas were 1:1 with AG-UI events and a replay was
byte-identical. I verified both directions: the new test passes on the
pre-coalescing adapter and fails with coalescing. The regression was
mine.
`render-frame-replay.test.ts` stays as a standing guard — it passes with
frames 1:1 and fails for any change that makes a frame's payload at a
given seq depend on timing.
## The replacement
Batch the **transport**, don't merge frame **content**: send the
buffered frames as an array in one request. Same collapse in round
trips, `seq k` still carries `delta k` so replay stays byte-identical,
and seqs stay dense so the outbox can later require contiguity
(OSS-653).
That needs an app-api batch accept and a gateway envelope, so it ships
after launch with its own deploy ordering (app-api → gateway → SDK)
rather than two days before.
## Also found, filed separately
OSS-653 — the outbox advances an in-memory cursor to the highest seq it
read (`WHERE seq > cursor ORDER BY seq ASC`), so a frame accepted out of
order is skipped and never rendered. Pre-existing and independent of
this PR. Not durable corruption: thread history is rebuilt from all
acceptance rows ordered by seq, independent of that cursor, so threads
read back complete.
## Validation
```
nx run-many -t test check-types build --projects=@copilotkit/channels-intelligence --skip-nx-cache
```
216 tests pass (12 files), check-types and build clean, `oxlint` clean
on all changed files.
refs OSS-648
Coalescing adjacent text deltas cut a 120-token reply from 123 frames to 4 and
took a production turn from an extrapolated ~59s to 3.4s, but it broke replay.
A turn's id is a pure function of its delivery id (turn_<deliveryId>, which
app-api enforces) and the adapter resets the per-turn seq counter on every
dispatch, so a redelivery replays into the same seq space. app-api treats that
space as an idempotency key: the same payload at a given seq is
duplicate_accepted, a different payload is CHANNEL_RENDER_FRAME_CONFLICT, which
nacks the delivery. Merge boundaries depend on wall-clock arrival, so a replay
cuts them differently and collides — attempt 1 stores "abcd" at seq 4, attempt 2
pushes "d" there. Before coalescing, deltas were 1:1 with AG-UI events and a
replay was byte-identical, so this worked; the regression is mine.
Keep the instrumentation, which is what found the problem and is what will
measure the replacement. Add the replay-stability test as a standing guard: it
passes with frames 1:1 and fails for any change that makes a frame's payload at a
given seq depend on timing.
The cost problem is real and unfixed. The replacement is to batch the TRANSPORT
(many frames per request) rather than merge frame CONTENT: it collapses the same
number of round trips, keeps seq k carrying delta k so replay stays
byte-identical, and keeps seqs dense so the outbox can later require contiguity
(OSS-653). That needs an app-api batch accept and a gateway envelope, so it ships
after launch with its own deploy ordering rather than two days before.
A streaming reply emitted one render frame per AG-UI token and awaited each
durable acceptance receipt before sending the next, so reply latency scaled
with token count: a 100-token reply cost 100 serialized round trips. Against a
hosted API that is tens of seconds for a long reply, and it dominated every
other cost in the egress path.
Frames now land in a buffer that a single pump drains, folding adjacent
text deltas for the same message into one frame. The buffer only ever holds
what arrived while a push was already in flight, which is exactly the set the
old serial chain kept queued, so this waits on no timer and adds no latency.
Batch size self-tunes: the slower the round trip, the more deltas merge.
Ordering is preserved by keeping seq allocation at enqueue time, so a discrete
post or update interleaved mid-run still orders correctly against the run's own
frames. A merged frame carries the seq of the last delta it absorbed; skipped
seqs are never pushed, and the accepted high-water mark still advances. Only
adjacent same-message deltas merge, so a tool call or a second message ends the
run. Merged text is capped at 12k chars, under the platform's 40k delta limit
and within one Slack appendStream call.
Measured with the instrumentation from the previous commit, on a 40-token reply
with a 100ms round trip and tokens arriving every 10ms: 42 frames and 4242ms
before, 7 frames and 706ms after, with 40 deltas folded into 5 pushes and no
text lost.
Channel replies are slow in proportion to how many tokens they contain. The
run renderer pushes one render frame per AG-UI event and awaits its durable
acceptance receipt before sending the next, so a streaming reply costs one
round trip per token. Nothing recorded that, so the cost was invisible: no
frame counts, no per-push latency, no measure of how much of a turn was spent
waiting.
Add opt-in instrumentation at the renderer's serial push chain, the one point
every frame passes through, so it covers the HTTP render-accept transport and
the realtime-gateway transport with one seam. Set
COPILOTKIT_CHANNELS_RENDER_METRICS=summary for a per-turn summary, or =frames
to also log a line per frame. Unset means off and no collector is built.
The summary reports frame counts by kind, characters carried per text_delta
frame, total/mean/p50/p95/max push latency, and the share of the turn spent
blocked on receipts. charsPerTextFrame and pushBlockedPct are the two numbers
that show whether a batching change helped: one round trip per token reads as
a single-digit charsPerTextFrame and a pushBlockedPct near 100.
Measurement only. No change to frame ordering, sequencing, or delivery.
## Problem
All five `OSS-599` references in
`packages/runtime/src/v2/runtime/core/channel-manager.ts` describe the
missing gateway/canonical/reliability wiring for **direct** Channels as
*"deferred"*:
> it is NOT wired into the Intelligence gateway/canonical/reliability
layer (deferred, OSS-599)
That reads as pending work — as though a direct Channel eventually
reaches managed parity.
**OSS-599 says the opposite.** Its boundary discipline places
run-correctness (canonical cross-surface history, fenced
outer-run/single-terminal, durable HITL-resume-across-restart, selection
pinning) and the reliability layer **Intelligence-side only**, and
states plainly that shipping an SDK-side equivalent *"collapses the
build-vs-buy moat"*. A direct Channel's ceiling is the SDK's in-process
run loop, permanently.
So the comments point the next reader at implementing precisely the
thing the ticket forbids.
## Change
Reword all five sites to say the boundary is by design, not pending:
| Site | Was | Now |
|---|---|---|
| `ChannelStatus` doc | "(deferred, OSS-599)" | "BY DESIGN, not a
deferral" + why, + "do not 'finish' this by pulling the layer into the
SDK" |
| `ChannelManager` class doc | "wiring … is deferred" | "stay below the
canonical/reliability layer by design" |
| `activate()` inline | "is deferred (OSS-599)" | "by design, not
pending work (OSS-599)" |
| `startDirectChannel` doc | "(deferred, OSS-599)" | "that boundary is
permanent, not a deferral" |
| direct-start log string | "wiring deferred (OSS-599)" | "stay below
the canonical/reliability layer by design (OSS-599)" |
Comments and one log string only. **No behavior change.**
## Testing
- **Log-string assertion preserved.** `channel-manager.test.ts:611-619`
asserts the direct-start breadcrumb contains `"direct adapter"` and
`"ɵruntime.start()"`. Both substrings survive the reword — only the
parenthetical changed.
- **Test run + control.** Ran `channel-manager.test.ts`,
`channel-manager-reconnect.test.ts`, and
`channel-activation-config.test.ts` in the worktree: `4 failed | 56
passed`. Ran the same suite on an **unmodified `origin/main`** worktree
as a control: `4 failed | 34 passed` — the *identical* four failures.
The four are a worktree artifact, not a regression:
`@copilotkit/channels` resolves to the outer checkout's pre-#6145 build,
which has no `ɵruntime`, so every "real direct transport" test fails
there. Same failures before and after the change; this diff adds none.
CI (with a correct install) is the real gate.
- **Formatted** with `oxfmt`.
## Notes
Pre-commit hooks were bypassed: `test-and-check-packages` runs `nx` in
the worktree, where `@copilotkit/core:build` fails for unrelated
environment reasons. The change is comments-only.
Follow-up, not in this PR: OSS-599 was written the day before Plan C
shipped, so its own framing ("a DIY runner is ~15 lines over
`ɵruntime.start()`", "a DIY runner gets this") is stale now that the DIY
path is removed. Its §2 response-policy and four-mode-binding scope is
unaffected. I'm leaving a reconcile note on the ticket.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Closes OSS-646. Split out of OSS-641 as the unambiguous half. This PR
does **not** change when activation happens — whether the long-running
wrappers should auto-connect stays open on OSS-641.
## Why
`createCopilotRuntimeHandler` builds the `ChannelManager` but opens no
connection; activation is lazy, triggered by the first
`channels.ready()`. That is deliberate (`fbf35ac59`, OSS-473) —
Cloudflare/Next isolates freeze and recycle per request, so cold starts
would mint conflicting listeners. Two things were left inconsistent with
it:
1. `endpoints/node.ts` still documented the pre-`fbf35ac59` world — "the
same `ChannelsControl` surface the underlying fetch handler **activates
at creation time**" — and labelled the one required call as `//
Optional:`. That's the TSDoc developers and coding agents see in-editor,
and it contradicted every channel-package README. Same failure class as
OSS-634.
2. `68349bc1f` gave the fetch handler a branded overload so
`handler.channels.ready()` type-checks without `?.`, but the node
wrapper never got it — so every call site, including our own example and
all nine showcase docs pages, was written defensively.
### A live consequence, found en route
`examples/slack/app/managed.ts` never called `ready()`. It built the
runtime, mounted the listener, logged `[channel] started managed Channel
"…"`, and only ever called `stop()` — so since activation went lazy it
has connected nothing while reporting success. It was written against
exactly the creation-time model the TSDoc described. Fixed here, with a
regression assertion.
## What changed
- **Types** — `createCopilotNodeListener` gets the branded overload pair
mirroring `createCopilotRuntimeHandler`: a runtime with at least one
declared Channel yields non-optional `.channels`; `activateChannels:
false` and channel-less runtimes keep the optional shape. Adds
`NodeCopilotListenerWithChannels`; both listener types are now exported
from `@copilotkit/runtime/v2/node`.
- **Docs** — node/express/hono TSDoc corrected: creation opens no
connection, `ready()` is what activates, and it is required on a
long-running host. Same stale claim fixed in the three example comments
and `examples/slack/README.md` that repeated it.
- **Call sites** — `?.` dropped from `examples/slack`, `examples/teams`,
both READMEs, and the nine `showcase/shell-docs` channel pages.
## Deliberate scope choices, called out
- **Express/Hono keep an optional `.channels`.** Only their TSDoc is
corrected here. Their own type docs name Node as the lifecycle-owning
surface and attach `.channels` best-effort, so the branded overload is
Node-only for now; `endpoints-channels.test.ts` still uses `!` for those
two. Say the word if the overload should extend to them.
- **The non-optional shape requires a literal `channels` tuple**
(`readonly [Channel, ...Channel[]]`). A runtime built from a
dynamically-assembled `Channel[]` is unbranded and still needs `?.`. Now
stated in the node TSDoc.
- **`examples/slack/app/managed.ts` now exits nonzero if activation
fails**, where before it stayed up serving HTTP with nothing connected.
Intentional — fail loud, and it matches `index.ts`. Note that `ready()`
resolves for `setup_required`, so a declared-but-unprovisioned channel
still logs as started.
- **Signal handlers are registered before awaiting activation** in
`managed.ts`, so a Ctrl-C inside the 30s activation window still tears
the Channel down instead of hitting Node's default handler.
## Verification
- **Type contract, red → green:** the new `KeyIsRequired<typeof
listener, "channels">` assertion in `handler-channels-types.test.ts`
failed to compile before the overload (`error TS2344: Type 'false' does
not satisfy the constraint 'true'`) and passes after.
- **Example bug, red → green:** stashing only `managed.ts` fails the new
guard with `expected "vi.fn()" to be called once, but got 0 times`.
- **Strict-null proof:** `slack-example` and `teams-example` both `tsc
--noEmit` clean under `strict: true` with the `?.` removed. This matters
because the runtime package compiles with `strict: false`, so its own
type test can only probe the optionality modifier structurally.
- Runtime channel suites 54/54; slack example 63/63.
- **Coverage limit:** the `managed.ts` guard is mocked — it proves the
example *calls* `ready()` with a bound, not that a Channel connects.
Nothing in CI exercises a real gateway connect for these examples.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Makes `new CopilotKitIntelligence({ apiKey })` the whole managed-service
setup. `apiUrl` and `wsUrl` were required on every construction, so the
two correct hosts had to be found and copied by hand — which is how an
agent came to invent them. Linear:
[OSS-638](https://linear.app/copilotkit/issue/OSS-638/add-default-rest-and-ws-api-urls-to-intelligence-sdk).
## The change
Both fields are now optional and default to CopilotKit's managed
platform (`https://api.intelligence.copilotkit.ai` /
`wss://realtime.intelligence.copilotkit.ai`). Overrides are unchanged
for self-hosted and non-production deployments, with two guards that the
previous required-field signature made unnecessary:
**A blank value counts as unset.** These URLs are almost always wired
from env vars, and a declared-but-empty variable
(`COPILOTKIT_INTELLIGENCE_URL=`, common in generated `.env` files and
container configs) arrives as `""`. Left as-is it slips past a `??`
default and produces host-relative requests — the failing test showed a
request to `/api/threads?...` with no host at all.
**Setting only one of the pair warns.** The API and realtime planes are
separate hosts, so a lone override silently pairs (say) a self-hosted
API with the managed gateway. That failure surfaces as a 30s hang rather
than an error, because the socket layer treats an unreachable host as a
retryable reconnect — so it is worth a log line at construction.
## Docs
Swept the copy-paste surfaces to the short form so nobody is handed URLs
to get wrong: channels quickstart/index/mcp + the 5 platform pages, the
7 channels package READMEs, the runtime and react-core skill references
(package and root copies), and the slack/teams examples with their
READMEs and `.env.example`s. The "separate hosts, override both or
neither" lesson now lives in the quickstart's override note and the SDK
JSDoc instead of in every snippet.
Two things beyond the mechanical sweep:
- `dev-docs/architecture/setup-intelligence.md` documented `apiUrl:
"https://your-intelligence-host/api"` and `wsUrl:
"wss://your-intelligence-host/socket"` — the exact double-`/api` and
fake-`/socket` suffixes the runtime skill reference warns against. Fixed
while here.
- The `CopilotKitIntelligence` class JSDoc sat above
`ThreadDeletedPayload`, so it never appeared on hover of the class.
Reattached — that is the discoverability half of the original complaint.
The integration examples (langgraph, mastra, adk, …) are deliberately
untouched: they pass `process.env.INTELLIGENCE_API_URL ??
"http://localhost:4201"` to target a local stack, where the managed
defaults would be wrong.
## Verification
Tests written first and watched fail — 5 on undefined URLs, then the
blank-URL case on the host-relative request.
- `client.test.ts` 54/54; full `runtime/src/v2` suite 1128/1128
- slack example 63/63, teams example 2/2, channels-intelligence 195/195
- `tsc --noEmit` clean on `packages/runtime`; both edited examples
typecheck clean against the new signature (verified by resolving
`@copilotkit/runtime` at the worktree source, since the symlinked
`node_modules` otherwise reports stale required-field types)
- oxlint 0 warnings / 0 errors on touched files, oxfmt clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What
Two SDK-side delivery / terminal-signal hardening fixes in the managed
channels
adapter (`packages/channels-intelligence/src/intelligence-adapter.ts`),
from the
pre-merge adversarial CR of #5972. Closes OSS-491.
## Scope correction (verified end-to-end during review)
The ticket originally framed Finding 1 as a live "self-hosted /
no-Outbox"
silent-failure. That premise is **wrong**, and I traced the full render
path to
confirm it:
- Every real deployment wires a render sink: the realtime gateway
launcher passes
one, and the config-free `intelligenceAdapter()` auto-wires
`HttpRenderEventSink`
in `start()`. On that path a `run_error` is POSTed to app-api's
`/api/channels/deliveries/:id/render-events/accept` route, durably
persisted,
projected into canonical history as a `RUN_ERROR` terminal, **and** the
Connector
Outbox posts a user-visible `⚠️ Agent error: …` to Slack.
- The Connector Outbox is the **same app-api binary** and runs in
**self-hosted / composite** deployments too (gated only by the channels
feature
flag, not deployment mode). Self-hosted handles `run_error` identically
— there
is no server-side gap.
- The ad-hoc fallback sink in `renderSinkFor` — the code Finding 1
patches — is
reached **only** when a consumer injects both `source` and `egress` but
no
`renderSink` (in-memory tests, and hypothetical BYO custom-transport
integrations). It is not on any default or self-hosted path.
So Finding 1 is **defensive hardening of a fallback plus a lifecycle
fix**, not a
HIGH-severity production silent-failure (severity retriaged to Low on
the ticket).
Finding 2 is on the shared `dispatch()` path and does affect real
deployments.
## Finding 1 — `run_error` on the ad-hoc HTTP-fallback sink
`renderSinkFor`'s ad-hoc plain-text sink (used only when no `renderSink`
is wired)
had no `run_error` branch, so on that path a `run_error` was dropped,
the run
resolved, and `dispatch()` acked a text-less turn as success — silent,
nothing
logged. Now that sink logs the raw error via `config.log`, flushes any
buffered
partial text, and posts a generic user-visible error. Posted + logged +
**acked, not nacked**: a `run_error` is frequently a deterministic agent
failure
and the Channel path reprocesses redeliveries (`skipIngressDedup`), so
nacking
would livelock. A genuine egress failure of the error post itself still
nacks
(`emit` throws → `drain()` rejects).
Also — and this part **does** apply to the realtime path —
`onRunErrorEvent` now
calls `ensureRunStarted()`, so a `run_error` that precedes any text/tool
event
emits a preceding `run_started` (parity with the text/tool handlers; a
well-formed
run lifecycle on both paths). This is the one intentional realtime-path
change: a
text-less error now yields `[run_started, run_error, finalize]` instead
of
`[run_error, finalize]`. `run_error` **rendering** is unchanged — still
a frame to
the sink, never converted to an egress error post.
## Finding 2 — a failed `ack()` no longer nacks a completed turn
`ack()` sat inside the same `try` as `dispatchTo`, so an `ack()` network
failure
fell into the catch and nacked an already-processed turn with a
misleading reason;
with `skipIngressDedup`, the whole turn reran (duplicate agent run —
re-calls the
model, re-runs tools — even though egress op-id dedupe catches duplicate
posts).
Ack is now a distinct phase: a failed ack is logged, never nacked. This
kills the
**spurious** rerun — notably the common "ack landed server-side but the
response
was lost" case. A genuinely-failed ack still redelivers via lease expiry
(inherent
to at-least-once, unavoidable); the turn already posted its reply, so
only
delivery bookkeeping is delayed.
## Tests
TDD; the behavior tests were written failing first.
- Fallback text-less `run_error` ⇒ emits a user-visible error post
**and** logs it.
- `run_error` before any text/tool event ⇒ `run_started` precedes it.
- Realtime path ⇒ `run_error` still forwarded to the render sink,
**not** converted
to an egress post (regression guard).
- Failed `ack()` on a successful turn ⇒ not nacked, no rerun, ack
failure logged.
- Failed error-post emit on the fallback path ⇒ `drain()` rejects
(nacks/retries).
- Buffered partial text flushed before the error post (order preserved).
## Verification
- `nx run @copilotkit/channels-intelligence:check-types` ✅
- `nx run @copilotkit/channels-intelligence:test` ✅ (all green)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
`CopilotKitIntelligence` required `apiUrl` and `wsUrl` on every construction,
so the two correct hosts had to be found and copied by hand — which is how an
agent came to invent them. Both now default to CopilotKit's managed platform,
making `new CopilotKitIntelligence({ apiKey })` the whole managed-service setup.
Overrides are unchanged for self-hosted and non-production deployments, with two
guards that the previous required-field signature made unnecessary:
- A blank value counts as unset. These URLs are usually wired from env vars, and
a declared-but-empty variable arrives as `""`, which would otherwise produce
host-relative requests instead of falling back to the managed platform.
- Setting only one of the pair warns. The API and realtime planes are separate
hosts, so a lone override silently splits the client across two deployments —
and that failure surfaces as a hang, not an error.
Sweeps the doc, skill, README, and example surfaces to the short form so the
copy-paste path no longer hands anyone URLs to get wrong, and reattaches the
`CopilotKitIntelligence` class JSDoc, which was orphaned above an interface and
so never appeared on hover.
Linear: OSS-638
`createCopilotNodeListener` now mirrors `createCopilotRuntimeHandler`'s branded
overload pair, so a runtime with at least one declared Channel yields a listener
whose `.channels` is non-optional and the documented `listener.channels.ready()`
call type-checks with no `!` and no `?.`. `activateChannels: false` and
channel-less runtimes keep the optional shape. Both listener types are exported
from `@copilotkit/runtime/v2/node`.
Corrects TSDoc on the node, express, and hono wrappers that still claimed
activation happens "at creation time" and labelled `ready()` as optional — stale
since activation was deferred to make the Fetch handler serverless-safe. On a
long-running host that call is required, not optional.
Fixes a live consequence of that stale model: `examples/slack/app/managed.ts`
never called `ready()`, so it mounted a listener, logged "started managed
Channel", and connected nothing. Covered by a regression assertion.
Drops the now-unnecessary `?.` from the examples, READMEs, and channel docs, and
adds compile-time contracts for the listener shape alongside the existing
handler ones. The examples compile with `strict: true`, so they prove the
`?.`-free call under strict null checks, which the runtime package (strict:
false) cannot.
Close two assert-without-a-test gaps found in review of the OSS-491 fix:
a failed error-post emit on the fallback run_error path propagates (drain
rejects → the delivery nacks/retries rather than acking a dropped error),
and buffered partial text is flushed before the error post (order preserved,
no lost or duplicated output).
Two pre-existing SDK delivery / terminal-signal defects in the managed
channels adapter, surfaced by the pre-merge adversarial CR of #5972 and
deliberately left out of that PR's scope. Both bite the HTTP-fallback /
self-hosted path (no Connector Outbox); the realtime + Outbox path is
unchanged.
Finding 1 (HIGH) — run_error swallowed on the HTTP-fallback render path,
delivery acked as SUCCESS. `renderSinkFor`'s ad-hoc plain-text sink had no
`run_error` branch, so an agent-side failure with no accompanying text (model
error, tool crash, upstream 5xx) resolved the run, drained, and acked the
delivery as success — no user-visible message and nothing logged ("bot goes
silent on every error"). Now the fallback sink logs the raw error via
`config.log`, flushes any buffered partial text, and posts a generic
user-visible error. Posted + logged + acked (not nacked) on purpose: a
run_error is frequently a deterministic agent failure and the Channel path
reprocesses redeliveries (skipIngressDedup), so nacking would livelock. A
genuine egress failure of the error post itself still nacks (emit throws →
drain rejects). Also: `onRunErrorEvent` now calls `ensureRunStarted()` so a
run_error that precedes any text/tool event still emits a preceding
run_started.
Finding 2 (LOW) — a failed ack() nacked an already-processed turn. `ack()`
sat inside the same try as `dispatchTo`, so an ack network failure fell into
the catch and nacked a completed turn with a misleading reason; because the
Channel path reprocesses redeliveries, the whole turn reran (duplicate agent
run / posts). Ack is now a distinct phase: a failed ack is logged, never
conflated with a turn failure and never nacked.
A runtime that declares a managed channel the Intelligence project does not have
crashed startup: ready() rejected and the channel settled to error. Creating the
channel and attaching credentials is the setup step every new channel goes
through, so this was the first run of every managed channel, before the happy
path.
app-api had no declaration-response token for "the project has no channel by that
name", so it sent runtime_not_declared. That token means the inverse on a join —
a channel exists but no runtime declared it, while on a join the runtime is the
one declaring. classifyJoinError read it literally, which was correct, and so
treated a benign setup gap as a hard failure.
app-api now sends channel_not_declared for that case (Intelligence#615). Adding it
to SETUP_REQUIRED_CHANNEL_STATES lets the channel settle to setup_required and
ready() resolve. runtime_not_declared keeps its literal reading and stays a hard
error, as do runtime_conflict, platform_setup_failed, delivery_failed,
egress_failed and runtime_offline.
This also answers the TODO(gateway-owner) the classifier carried, which asked
whether runtime_not_declared was ever benign. It was, but not for the reason the
comment guessed: the server was overloading the token, not racing. Replaced the
TODO with what the answer turned out to be.
npx vitest run src/realtime-gateway.test.ts: 21 pass
33 failures elsewhere in this package are pre-existing on origin/main (a stale
@copilotkit/channels build missing the runtime symbol) and unrelated.
Closes OSS-623 (split out of OSS-621's acceptance item 3).
## The bug
A managed Channel whose `wsUrl` points at a wrong or unreachable host
sat in `connecting` for the caller's full settle budget and then
rejected with `did not settle in time` — no indication that the *host*
was the problem. A wrong **API** host fails fast with a clean HTTP
error; a wrong **WS** host hung.
Three failure modes collapsed into that one opaque timeout: DNS that
does not resolve, a refused/failed TCP or TLS connect, and a host that
serves HTTP with no Phoenix socket mounted at the endpoint (the prod
split-plane trap — `api.intelligence` and `realtime.intelligence` are
separate hosts).
## Why it hung
Two Phoenix behaviors compound, and the second is why the existing join
timeout did not save us:
1. `Socket` retries a failed **first** connect forever and reports
transport errors as reconnect churn, so nothing distinguishes "still
handshaking" from "this host will never work."
2. The join push's own `timeoutMs` is silently defused.
`Channel.onError` calls `joinPush.reset()` while the channel is joining
(phoenix 1.8.4 `channel.js`), which nulls `refEvent` and `off`s the
reply binding — the armed timeout then fires into a dead event name and
no `"timeout"` recHook ever runs.
Net: the connect await hung *indefinitely*; the 30s in the ticket was
purely `ChannelManager.ready({ timeoutMs })`'s deadline, which is why
the message was a stopwatch.
## The fix
**Watch the initial connect separately from reconnects**, keyed off
whether the socket has ever opened (phoenix passes exactly that as
`socket.onError`'s third argument, `establishedConnections`):
- **Never opened** → reject with a new `RealtimeGatewayUnreachableError`
naming the endpoint and the cause, and tear the socket down so it stops
retrying a host that will not answer.
- **Opened, then dropped** → unchanged; the existing `reconnecting` /
`gave_up` machine owns it.
**Get the cause from HTTP, because the socket does not have it.** This
is the part an adversarial review of the first commit corrected. Node's
global WebSocket (undici — the production default) dispatches the *same*
detail-free `ErrorEvent` for all three modes: `"Received network error
or non-101 status code."`, no OS code, an inner `Error` with nothing
more. Browsers are deliberately worse. So on the failure path only, we
probe the same origin **and path** over http(s) with `fetch`, which does
discriminate — verified against the real network:
| target | WebSocket says | `fetch` says |
|---|---|---|
| nonexistent host | network error or non-101 | `cause.code = ENOTFOUND`
|
| closed port | network error or non-101 | `cause.code = ECONNREFUSED` |
| `https://example.com/runner` | network error or non-101 | **HTTP 404**
|
A transport that *does* expose an OS code (the `ws` package, custom
transports) is trusted directly and skips the probe; a successful
connect issues no request at all.
**Failing fast vs. tolerating a boot race** is split by what the
diagnosis found:
- **The host does not resolve** → fail immediately. Measured against the
real network: **25ms**, message `the host does not resolve (getaddrinfo
ENOTFOUND …)`. This is the OSS-621 case (`realtime.copilotkit.ai` has no
DNS record at all).
- **Anything else** → wait out `connectTimeoutMs` (default **30s**,
plumbed through the launcher). A `ChannelManager` marks an activation
failure `error` **permanently, with no retry**, so a shorter window
would convert a rolling restart of the gateway into a dead Channel where
the old forever-retry would have recovered.
Per acceptance item 4, the error deliberately does **not** carry the
`SETUP_REQUIRED` marker. An unreachable host is a config problem, but
`setup_required` is a benign settled state that lets `ready()` resolve —
a typo'd `wsUrl` must reject loudly, not go quiet-green.
## What the messages look like now
```
realtime gateway unreachable: the socket never connected to wss://oss623-does-not-exist.copilotkit.ai/runner
(the host does not resolve (getaddrinfo ENOTFOUND oss623-does-not-exist.copilotkit.ai))
— check that the Intelligence wsUrl points at the gateway host [25ms]
realtime gateway unreachable: the socket never connected to ws://127.0.0.1:45999/runner
(the host refused the connection (connect ECONNREFUSED 127.0.0.1:45999))
— check that the Intelligence wsUrl points at the gateway host [full window]
realtime gateway unreachable: the socket never connected to wss://example.com/runner
(the host answered HTTP 404 for https://example.com/runner, so it resolves and accepts
connections — only the WebSocket handshake failed, so the wsUrl path may not be the
gateway's socket endpoint) [full window]
```
## Tests
Rebuilt around the shapes the real implementations emit, with the undici
shape as the default case (the first version's fake fed a `ws`-shaped
coded error undici never produces, which is what hid the dead code):
- The host does not resolve → fails in <1s despite a 5s window, probes
host **and path**, names the cause, carries `code ===
"GATEWAY_UNREACHABLE"` and the underlying `cause`, is not
`SETUP_REQUIRED`, socket closed.
- Refused → waits out the window, then names `ECONNREFUSED`.
- Host answers HTTP 404 → reported as a handshake/path problem, not a
missing host.
- A transport that exposes an OS code → fails fast with **no** probe
issued.
- A connect that hangs with no transport event at all → bounded by the
connect window (not a 60s join timeout), and the deadline itself starts
the probe so a cause is still reported.
- Neither transport nor probe yields a cause → still names the endpoint.
- **The shipped probe against a real closed loopback port** — no seam,
no DNS, no egress.
- A gateway that fails once and accepts on retry still connects (the
boot race).
- A healthy connect issues no diagnostic request.
- The connect deadline is disarmed after a successful connect — a later
drop is still a reconnect episode.
Before the fix, four of these hang until vitest's own 5s timeout — the
bug reproduced directly.
## Verification
`check-types` clean; full package suite 195/195; oxlint + oxfmt clean;
lefthook pre-commit (whole package suite + `check:packages`) green. All
three failure modes additionally verified end-to-end against the real
network and the real `fetch`, not just fakes.
## Known adjacent case, not in scope
A socket that opens and *then* drops before the initial join reply hits
the same defused-join-timeout behavior (2 above) and still relies on the
caller's settle deadline. That path can genuinely self-heal via
Phoenix's rejoin, so it keeps the retry; bounding it is a separate call
from OSS-623's "never connected once."
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Fixes the Channels docs/examples pointing at an Intelligence host that
does not serve the API, and the websocket URL guidance that can never
produce a working prod value. Linear:
[OSS-621](https://linear.app/copilotkit/issue/OSS-621).
## The two bugs
**1. The documented host does not serve the API.** Probed every
plausible path, not just `/`:
| URL | Result |
| --- | --- |
| `api.copilotkit.ai` — `/`, `/api`, `/api/health`, `/health`,
`/api/threads`, `/api/v1/threads` | **404 on all**, `server:
awselb/2.0`, `content-length: 0` — an ALB with no target-group rule
behind it |
| `realtime.copilotkit.ai` | **no DNS record at all** |
| `api.intelligence.copilotkit.ai/` and `/api/health` | 200,
`x-powered-by: Express` |
| `api.intelligence.copilotkit.ai/api/threads` | **401** — a real,
auth-gated endpoint |
| `realtime.intelligence.copilotkit.ai/runner/websocket` | **403** —
mounted and auth-gated (a 404 would mean unmounted) |
So the documented host is not merely returning 404 at the root — nothing
is routed there on any path, and it is not the app (no `x-powered-by`).
The working pair, matching the CLI's baked-in prod defaults and
`gitops/environments/prod/values.yaml`, is
`https://api.intelligence.copilotkit.ai` +
`wss://realtime.intelligence.copilotkit.ai`.
**2. `wsUrl` was documented as derivable from `apiUrl`.** Prod splits
the API and realtime planes across *different hosts*, so a scheme-only
swap yields `wss://api.intelligence.copilotkit.ai` — wrong host. That
failure is silent: a wrong `apiUrl` returns a clean HTTP error, but a
wrong `wsUrl` sits in `connecting` until the settle timeout and reports
only "did not settle in time". The derive is not even correct locally,
where the API and gateway are on different ports (4201 vs 4401) and the
swap preserves the port. It is essentially never right, so it is deleted
rather than relabelled.
Demonstrated end to end rather than asserted — running `examples/teams`
with the WS URL unset:
```
# on main: derive(https://api.intelligence.copilotkit.ai) -> wss://api.intelligence.copilotkit.ai (wrong host, 30s hang)
# on this branch:
exit code: 1
Missing COPILOTKIT_INTELLIGENCE_WS_URL.
export COPILOTKIT_INTELLIGENCE_URL=https://api.intelligence.copilotkit.ai
export COPILOTKIT_INTELLIGENCE_WS_URL=wss://realtime.intelligence.copilotkit.ai
The API and websocket URLs are DIFFERENT hosts (api.… vs realtime.…), so
the websocket URL cannot be derived from the API URL — set both.
```
## Scope note
The ticket listed 8 sites; the actual blast radius was 30 across 22
files. Beyond the ticket's list:
- **Five more channels package READMEs** — `channels`, `channels-core`,
`channels-discord`, `channels-slack`, `channels-teams` (the ticket named
only telegram + whatsapp).
- **The live product docs** —
`showcase/shell-docs/src/content/docs/channels/` taught the broken
derive in 8 files. These escaped the ticket's grep because their host
was already a `your-intelligence-url` placeholder; only bug 2 was
present. This is the surface developers actually read.
- **A generated skills mirror** — `skills/runtime/` is produced from
`packages/runtime/skills/runtime/` by `pnpm sync:plugin-skills`; fixing
one without the other leaves the bug live and fails the
`check-plugin-skills` gate.
- `skills/copilotkit-debug/references/runtime-debugging.md` and a third
occurrence in `client.ts`.
## Acceptance item 4 — resolved, chain verified
The ticket flagged a contradiction: `realtime-gateway.ts` documented
`wss://gateway.example/socket` while the runtime skill listed `/socket`
as a mistake. **The skill is right**, confirmed by tracing the real
runtime path rather than inferring it:
1. `channel-activation-config.ts:145` — `const wsUrl =
intelligence.ɵgetRunnerWsUrl()`, i.e. base + `/runner`.
2. → `channel-manager.ts:237` `wsUrl: config.wsUrl` →
`startChannelsOverRealtimeGateway` → `connectRealtimeGateway`.
3. `realtime-gateway.ts:253` hands that to Phoenix's `Socket`, which
appends `/websocket`.
4. The gateway mounts exactly `/runner` and `/client`
(`realtime_gateway/endpoint.ex:10,17`). There is no `/socket`.
The two docs survived contradicting each other because they describe
**different layers**: the public `wsUrl` is a bare base, while
`connectRealtimeGateway` receives the already-derived runner URL. Its
doc comment and the test fixtures move to `/runner` and now name the
layer. Independently, `get-runtime-info.ts:93` uses `ɵgetClientWsUrl()`,
which confirms the `/info` sample in the debug skill correctly keeps its
`/client` suffix — only the host there was wrong.
## Acceptance item 3 — split out
Failing loudly instead of hanging is a behavior change in the launcher's
connect path that overlaps OSS-622's error classification, so it is
[OSS-623](https://linear.app/copilotkit/issue/OSS-623) rather than
smuggled into a docs fix.
## Verification
- 44/44 gateway tests in `packages/channels-intelligence`;
`examples/slack/app/managed.test.ts` passes; `examples/teams` typechecks
clean.
- `nx run-many -t test,publint,attw` across the 15 affected projects
passed via the pre-commit hook; full CI green (38 checks, including
`build-check (shell-docs)`).
- Behavior proven by running the example, not by mocks (above).
- 45 commits behind `main` at time of review with **zero overlap** on
changed files, and the diff covers every dead-host and derive site
present on *current* `main`.
- No logic changes in either published package — `client.ts` and
`realtime-gateway.ts` are JSDoc/field-comment only. Behavior changes are
confined to the three example apps.
## Self-review pass
An adversarial pass over this PR tried to falsify its central claim
(that `api.copilotkit.ai` does not serve the API) by probing non-root
paths — the evidence got stronger, not weaker. It also cleared a
suspected regression: three `channels-intelligence` files read
`COPILOTKIT_INTELLIGENCE_URL` without the WS var, but they are the
HTTP-only transport path and need no socket URL. Two genuine gaps it
*did* find are fixed in the last commit: the five platform pages had
lost deployment neutrality (managed hosts now carry a self-hosted note),
and `index.mdx`/`mcp.mdx` referenced the new variable without telling
the reader where it comes from.
**One product decision for the reviewer:** `api.copilotkit.ai` is a live
ALB that routes nothing. If it is meant to become the public API alias,
this PR is documenting the wrong long-term string and the ALB wants a
listener rule instead. I used the host the product actually hands users
today (CLI prod defaults + gitops).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Adversarial review of the previous commit against the real transport found the
cause-carrying half of it was dead code. Node's global WebSocket (undici) — the
production default — dispatches the SAME detail-free ErrorEvent for a host that
does not resolve, a refused port, and a host answering HTTP with no socket
mounted: "Received network error or non-101 status code.", no OS code, an inner
Error with nothing more. So the non-retryable code set never matched (measured:
NXDOMAIN burned the full 8s window instead of failing fast) and the reported
transport error was the same useless string for every cause. The tests passed
only because their fake fed a ws-package-shaped error undici never produces.
Get the cause from a source that has it: on the failure path only, probe the
same origin AND path over http(s) with fetch, which does distinguish the modes
(verified against the real network: ENOTFOUND / ECONNREFUSED / HTTP 404). A
transport that DOES expose an OS code is still trusted directly and skips the
probe; a successful connect issues no request at all. NXDOMAIN now fails in
~25ms rather than after the whole window, and the HTTP-answered case reports
that the host is up and only the handshake failed — the prod split-plane trap.
Also raise the default connect window 10s -> 30s. Because a supervising
ChannelManager marks an activation failure `error` permanently with no retry,
the old default could convert a rolling restart of the gateway into a
permanently dead Channel, where the previous forever-retry would have recovered.
Tests are rebuilt around the shapes the real implementations emit, including one
that exercises the SHIPPED probe against a closed loopback port (no DNS, no
egress), one asserting a healthy connect probes nothing, and coverage for the
GATEWAY_UNREACHABLE code and preserved cause.
Refs OSS-623
A managed Channel whose wsUrl points at a wrong or unreachable host sat in
connecting for the caller's full settle budget and then reported only a
timeout. Phoenix retries a failed FIRST connect forever and reports transport
errors as reconnect churn, and the join push's own timeout never fires in that
state (Channel.onError calls joinPush.reset() while joining, detaching the
reply binding the armed timeout would have triggered), so the connect await
hung indefinitely with no indication that the host was the problem.
Watch the initial connect separately from reconnects, keyed off whether the
socket has EVER opened: a socket that never opened rejects with a new
RealtimeGatewayUnreachableError naming the endpoint and the underlying
transport error, while one that opened and later dropped stays with the
existing reconnect/give-up state machine. NXDOMAIN fails immediately (a host
that does not exist cannot start existing); every other cause is given the new
connectTimeoutMs window so a still-booting gateway is not turned into a
permanently errored Channel — the supervising ChannelManager does not retry
activation.
The error deliberately does NOT carry the SETUP_REQUIRED marker: a
misconfigured host must reject ready(), not resolve as setup_required.
Refs OSS-623
ConnectRealtimeGatewayOptions.wsUrl was documented as
`wss://gateway.example/socket`, while the runtime skill lists `/socket` as a
mistake because it is not a real path. The skill is right, and the gateway
mounts exactly `/runner` and `/client` — there is no `/socket`.
The two docs were describing different layers, which is how the contradiction
survived: the public wsUrl a developer configures is a bare base, and the
Intelligence client appends `/runner` before it reaches connectRealtimeGateway,
after which Phoenix appends `/websocket`. So this option is the fully-resolved
runner URL, and the doc comment now says so instead of implying developers
should write `/socket` anywhere. Test fixtures follow.
Refs OSS-621
Every documented Intelligence endpoint used api.copilotkit.ai for both the
REST and websocket planes. That host returns 404, and the realtime host it
implied (realtime.copilotkit.ai) has no DNS record at all, so a developer
copying any of these snippets could not connect.
The real managed endpoints are split across two hosts, matching what the CLI
bakes in as its prod defaults and what gitops deploys:
apiUrl: https://api.intelligence.copilotkit.ai
wsUrl: wss://realtime.intelligence.copilotkit.ai
Because the planes are separate hosts, wsUrl cannot be produced from apiUrl by
swapping the scheme, so each site now says so explicitly. The runtime skill's
"common mistakes" section gains that as a third failure mode, and the
CopilotKitIntelligenceConfig JSDoc documents it on the field itself — a wrong
apiUrl fails fast with an HTTP error, but a wrong wsUrl only hangs until the
settle timeout, which is what made this expensive to diagnose.
skills/runtime is the generated mirror of packages/runtime/skills/runtime,
regenerated with pnpm sync:plugin-skills. skills/copilotkit-debug is
standalone; its /info sample keeps the /client suffix, which is correct there
because that response carries the already-derived client URL.
Refs OSS-621
## Summary
- lowers CopilotSidebar-scoped CSS selector specificity with
`:where(.copilotKitSidebar)`
- keeps the existing sidebar styles and layout behavior unchanged
- adds a regression test to keep sidebar selectors easy for app CSS to
override
## Root cause
Sidebar-specific selectors such as `.copilotKitSidebar
.copilotKitWindow` were more specific than ordinary user overrides like
`.copilotKitWindow`, so user CSS could lose precedence in Chrome.
Fixes#263.
## Validation
- `pnpm -C packages/react-ui test`
- `pnpm exec oxfmt --check
packages/react-ui/src/css/sidebar-specificity.test.ts
packages/react-ui/src/css/window.css
packages/react-ui/src/css/header.css
packages/react-ui/src/css/input.css`
- `pnpm exec oxlint
packages/react-ui/src/css/sidebar-specificity.test.ts`
## What does this PR do?
Fixes a bug where resizing a `CopilotPopup` (v2) rescrolls the message
list from top to bottom on every resize, regardless of the user's scroll
position.
**Root cause.** `CopilotPopup` built its `chatView` override component
inside a `useMemo` keyed on `width`/`height`. When those props change —
which happens continuously for consumers driving them from a
drag-to-resize handle that commits new dimensions on `mouseup` —
`useMemo` returns a **new component function**. That function is passed
to `CopilotChat` as the `chatView` slot and rendered via
`React.createElement(newType, …)`. Because React sees a new element
**type** at that position, it unmounts and remounts the entire chat
subtree rather than re-rendering it. The remount resets the scroll
container's `scrollTop` to `0`, after which the default
`initial="smooth"` auto-scroll animates the list from the top back down
to the bottom — the visible "rescroll" on every resize.
**Fix.** Give the override a stable module-scope identity so its type
never changes, and pass the popup shell props (`header`, `toggleButton`,
`width`, `height`, `clickOutsideToClose`, `defaultOpen`) through React
context instead of baking them into the override via `useMemo` deps.
Resizing is now a plain style update on `CopilotPopupView` — no remount
— so scroll position is preserved. `Object.assign` still copies the
`CopilotChatView` static slot members onto the override so callers
reaching them through the namespace continue to work.
**Test.** Adds a regression test (`CopilotPopup.resizeRemount.test.tsx`)
that injects a mount-counting component into the chat subtree and
asserts its mount count stays at `1` across several `width`/`height`
changes. It fails on the pre-fix code (one extra mount per resize —
observed `4` for 3 resizes) and passes with the fix.
Scope is limited to `packages/react-core`:
```
packages/react-core/src/v2/components/chat/CopilotPopup.tsx | 124 +++++++++------
packages/react-core/src/v2/components/chat/__tests__/CopilotPopup.resizeRemount.test.tsx | 98 +++++++++++
2 files changed, 185 insertions(+), 37 deletions(-)
```
Verification: full `@copilotkit/react-core` unit suite green (1443
tests), `check-types` green.
## Related PRs and Issues
- Closes#6172
## Checklist
- [x] I have read the [Contribution
Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md)
- [ ] If the PR changes or adds functionality, I have updated the
relevant documentation — _N/A: internal behavior fix, no public API or
documented behavior change_
- [x] "Allow edits by maintainers" is checked
## Summary
Fixes the web inspector crash when historical tool result messages have
empty `content`.
Some stored thread tool-result messages can have `content: ""`. The
inspector previously parsed tool result content with
`JSON.parse(msg.content ?? "{}")`, which still passes the empty string
into `JSON.parse` and throws `Unexpected end of JSON input`.
## Changes
-> Treat empty and whitespace-only tool result content as an empty
result object
-> Keep malformed non-empty result content flowing through the existing
parse-error fallback
-> Add regression coverage for empty and whitespace-only tool result
messages in the thread conversation mapper
## Testing
-> `pnpm exec oxlint packages/web-inspector/src/index.ts
packages/web-inspector/src/__tests__/web-inspector.spec.ts`
-> `pnpm -C packages/web-inspector exec vitest run
src/__tests__/web-inspector.spec.ts --pool=threads --testNamePattern
"maps empty tool result content" --reporter=verbose`
## Notes
-> `pnpm nx show project @copilotkit/web-inspector --json` timed out
locally
-> `pnpm -C packages/web-inspector run check-types` is currently blocked
by existing package-level TypeScript errors around explicit `.js` import
extensions and one unrelated existing error
## Related Issue
Closes#5598
CopilotPopup built its `chatView` override inside a `useMemo` keyed on
`width`/`height`. Consumers driving those props from a drag-to-resize
handle (committing new dimensions on mouseup) minted a new component
function per resize; rendering a new element type at that slot makes
React unmount and remount the whole chat subtree. The remount resets
scrollTop to 0, then `initial="smooth"` re-animates the message list
top-to-bottom on every resize, from any scroll position.
Give the override a stable module-scope identity and pass the popup
shell props (header, toggle, width, height, clickOutsideToClose,
defaultOpen) through React context. Resizing is now a plain style
update on CopilotPopupView with no remount, so scroll position holds.
Add a regression test asserting the chat subtree stays mounted (mount
count stays at 1) across width/height changes: it fails on the prior
code (one extra mount per resize) and passes with the fix.
## What does this PR do?
Provisional agents returned by `useAgent` now receive the provider's
`credentials` setting when they are created. Cached provisional agents
also refresh that setting when the provider configuration changes, while
keeping the same agent instance.
The regression test covers both the initial value and a later `include`
to `omit` update on the cached provisional agent.
### Verification
- `pnpm nx run @copilotkit/react-core:test`
- `pnpm nx run @copilotkit/react-core:check-types`
- `pnpm nx run @copilotkit/react-core:build`
- `pnpm check-format`
## Related PRs and Issues
- Closes https://github.com/CopilotKit/CopilotKit/issues/6116
## Checklist
- [x] I have read the [Contribution
Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md)
- [ ] If the PR changes or adds functionality, I have updated the
relevant documentation
- [x] "Allow edits by maintainers" is checked (lets us help iterate on
your PR directly — faster turnaround for everyone)
Per BenTaylorDev's review on #5805, dropping the specificity of the
sidebar variant created a tie at desktop widths with the
@media (min-width:640px) base rule (both (0,1,0)); the later-authored
base won and reverted the sidebar header's top corners from square
(border-radius:0) back to rounded 8px. Re-assert the media-scoped
sidebar override inside the @media block (mirroring window.css) so the
sidebar header stays square at all widths.
## Summary
In multi-turn conversations using the Anthropic adapter, a model turn
that contains both assistant text and a tool call can be replayed as two
consecutive `{role: "assistant"}` entries in the Anthropic payload.
Anthropic expects one message object per turn with alternating roles, so
that split payload can blur turn boundaries on the next request. This PR
coalesces same-role Anthropic messages before dispatch so one assistant
turn stays one assistant message.
## Root cause
[`convertMessageToAnthropicMessage`](https://github.com/CopilotKit/CopilotKit/blob/005aebbededbfdd7978b3c1ee221580b74dd088d/packages/runtime/src/service-adapters/anthropic/utils.ts#L142-L219)
maps each CopilotKit message independently. A mixed assistant turn, a
`TextMessage(role=assistant)` followed by an `ActionExecutionMessage`,
therefore becomes two separate assistant entries. The Anthropic Messages
API requires all content blocks for one assistant turn to be sent in a
single message object: https://docs.anthropic.com/en/api/messages
[`AnthropicAdapter.process()`](https://github.com/CopilotKit/CopilotKit/blob/005aebbededbfdd7978b3c1ee221580b74dd088d/packages/runtime/src/service-adapters/anthropic/anthropic-adapter.ts#L282-L366)
already deduplicates `tool_result` blocks on the user side, but it
previously forwarded the mapped assistant-side payload without a
coalescing pass. Rebasing onto current `main` also exposed test-fixture
drift in the two new regression tests, so the final branch updates those
fixtures to the current object-form `TextMessage` constructor and
removes one stale `ActionInput` field while keeping the production fix
unchanged.
## Changes
- `packages/runtime/src/service-adapters/anthropic/utils.ts`: add
`coalesceConsecutiveSameRoleMessages(messages)` to merge adjacent
equal-role
Anthropic messages by concatenating their `content` arrays.
-
`packages/runtime/src/service-adapters/anthropic/anthropic-adapter.ts`:
call
`coalesceConsecutiveSameRoleMessages` before
`limitMessagesToTokenCount`.
-
`packages/runtime/tests/service-adapters/anthropic/anthropic-adapter.test.ts`:
add same-role regression coverage, align the new fixtures with the
current
object-form `TextMessage` constructor, and remove the stale `parameters`
field
from the action fixture.
- `.changeset/coalesce-anthropic-same-role-messages.md`: patch changeset
for
`@copilotkit/runtime`.
## Scope
The `tool_result` allowlist deduplication stays unchanged. Token
trimming and the orphan-removal post-processor still run after
coalescing, so `tool_use` and `tool_result` pairing remains intact. The
OpenAI, Google, LangChain, and Groq adapters are unaffected.
The reported regression and the explicit regression coverage are
assistant-side. The coalescing helper itself is role-agnostic for
adjacent equal-role array content, but this PR does not add a separate
user-side regression case.
## Related PRs and Issues
Prior attempted fix: #2864, which addressed unrelated message callbacks
rather than the adapter payload.
## Test plan
- [x] `pnpm -C packages/runtime exec vitest run
tests/service-adapters/anthropic/anthropic-adapter.test.ts`
10/10 tests pass, including the two same-role coalescing cases.
- [x] `pnpm -C packages/runtime exec vitest run
tests/service-adapters/anthropic/utils-token-trimming.test.ts`
9/9 tests pass.
- [ ] CI green (`static / quality`, `test / unit` on Node 20/22/24)
## Upstream
Closes#2910.
Reported by @alonronin.
## Summary
When an assistant message contains multiple parallel tool calls
targeting frontend tools (`useFrontendTool`), the tool result messages
are spliced into `agent.messages` in reverse order. This violates the
ordering contract that some LLM providers (OpenAI via ADK + LiteLLM)
enforce: tool results must appear in the same order as their
corresponding `tool_call_id`s. The mismatch causes the provider to
silently discard unmatched results during the follow-up run, making it
appear as though only the first frontend tool executed.
## Root cause
`executeSpecificTool` and `executeWildcardTool` both used the fixed
insertion point `messageIndex + 1`, where `messageIndex` is the
assistant message's position. Because the assistant message's index
doesn't shift when a tool result is inserted after it, each successive
splice pushed previously inserted results to the right, reversing the
final order.
## Changes
- `packages/core/src/core/run-handler.ts`: In both `executeSpecificTool`
and `executeWildcardTool`, advance the insertion index past any
tool-result messages already present after the parent assistant message
before splicing.
- `packages/core/src/__tests__/core-parallel-tool-order.test.ts`: New
regression coverage for parallel tool-call ordering, including the
follow-up path, with type-safe assistant and tool-message assertions on
current `main`.
- `.changeset/fix-parallel-tool-order.md`: Patch changeset for
`@copilotkit/core`.
## What is NOT changed
- Sequential execution of tool handlers is preserved; only the splice
insertion position changes.
- `runTool()` (programmatic API) is unaffected; it creates a single tool
call per invocation.
- The `newMessages.findIndex` guard that skips already-answered tool
calls is unchanged.
- Follow-up recursion logic is unchanged.
## Related PRs and Issues
Closes#2809.
## Test plan
- [x] `npx nx run @copilotkit/core:build`
- [x] `pnpm -C packages/core exec vitest run
src/__tests__/core-parallel-tool-order.test.ts` — 3/3 pass
- [x] `pnpm -C packages/core exec vitest run` — 586/586 pass
- [x] `pnpm -C packages/core exec tsc --noEmit`
- [ ] CI green
tylerslaton