## What changed
- replaces the Channel live-session SDK with `channel_delivery_v1`
invitation, claim, one-use join, and exact packet retry
- runs Channel turns through the normal canonical Thread lock and
AgentRunner path
- removes Channel-specific runner tokens, run open/close calls, lease
heartbeats, and compatibility behavior
- updates the public Channel delivery contracts, files, provider
effects, tests, and docs
## Why
The SDK must match the new Realtime Gateway boundary: Runtime owns agent
execution, Gateway owns provider effects, and Redis holds one exact
unacknowledged packet.
## Impact
Channel Runtime connections move from the old live-session flow to
`/channels`. This is a hard cut with no protocol fallback or feature
flag.
## Companion PR
- Intelligence platform:
https://github.com/CopilotKit/Intelligence/pull/663
## Validation
- `NX_TUI=false pnpm nx run-many -t test,check-types,build,publint,attw
--projects=@copilotkit/channels-intelligence,@copilotkit/runtime
--skip-nx-cache`
- commit-hook package checks, including 1,809 Runtime tests
- shell docs lint, typecheck, and production build
- `git diff --check`
## Live boundary proof
- connected the actual SDK delivery transport to the actual Realtime
Gateway release
- drove a prepared delivery from App API through Redis wake, SDK
handling, Gateway provider execution, and PostgreSQL terminal storage
- verified one provider call and the exact
`complete:complete:provider_delivery_complete` result
- repeated with two Gateway nodes sharing Redis/PostgreSQL; both
received the wake, exactly one Runtime handler ran, and exactly one
provider call occurred
Not run: a real external Slack or Teams provider smoke. The live proof
used a local fake Slack endpoint so the full internal boundary and exact
provider-call count stayed observable.
Rethrow all session.effect failures from postFile; soft-return only
upload/config errors. Prevents claimAndHandle false complete terminals
for join/claim/TypeError protocol failures that the rethrow allowlist
previously missed.
CR r5 bucket (a): always stop native Slack streams on failure (thread
finish + NativeMessageStream queue drain), advance append/replace text
only after apply, rethrow permanent postFile gateway errors, exclude
stream.stop from provider-output tracking, classify errors by message,
validate prepared turn fields per kind, and stop unit tests from hitting
live lock cleanup HTTP.
Keep provider stream cleanup sendable after non-terminal failures; only
seal effects for terminal provider statuses; join then leave on reconnect;
leave after invalid prepared join reply; advance stream text only after ack.
Allow a failed/uncertain terminal after effect or complete-terminal push
failures; seal only after a successful terminal apply. Leave Phoenix child
channels on failed join, re-arm delivery handlers on restart, replay
onStateChange health, skip empty Teams stream deltas, and align docs/tests.
Close packet path after permanent push/ack failures so a later effect
cannot mint a new effectId on the same seq. Refresh owner generation on
join_token reconnect, add reconnect backoff, require claimed on claim
assert, reject unknown turn kinds, skip empty Slack stream deltas, and
surface missing file-client attachments instead of dropping them.
Always release the product thread lock after a Channel canonical run.
Align connectTimeoutMs docs, projectId validation, ops error guidance,
and test fixtures with the delivery ID contract.
Note: local lefthook skipped (no node_modules in this worktree); CI will
validate. CR findings addressed from PR #6249 review.
## Problem
`workspace:^` dependencies in a canary pack to caret ranges. A consumer
can therefore resolve a different semver-higher canary (for example
`canary.perfall1` over `canary.1785375021`) and combine incompatible
artifacts from separate runs.
## Why
The `scope=all` prerelease flow bumps every package together, but `pnpm
pack` preserves the range operator. Sharing a canary identifier
therefore did not guarantee that npm would install that same package
set.
## Fix
Immediately before the existing prerelease pack/publish loop, replace
internal dependency ranges with the exact versions of packages included
in that canary run. Dependencies outside the selected publish set remain
unchanged.
This is confined to `prerelease.ts`, which `publish-release.yml` invokes
only for prerelease mode. Stable publishing continues through
`publish-release.ts` unchanged.
Validation:
- 153 release tests passed.
- Targeted TypeScript, oxlint, and oxfmt checks passed.
Addresses maxkorp's third pass, including a regression in the maxMessages cap
added in the previous commit.
1. `truncate()` is now terminal and idempotent. It set `curPosted =
buffer.length` and appended the marker but left `curTs` set and recorded no
terminal state, so text still arriving grew the buffer, the next flush saw
undelivered text on an already-full message, and re-entered — one marker per
flush for the rest of the turn. Reproduced at production defaults on a 400k
reply: markerCount=36, last message 12,908 bytes against an 11,000 budget.
Against a transport enforcing a real 12k ceiling, 18 markers land and then 28
appends are rejected, spending exactly the error the incident was about. This
was the spam the cap exists to prevent, relocated inside one message.
A `truncated` flag short-circuits `appendPending` and both `finish()` paths,
and the marker is now charged to `curMessageBytes` — it was the one append
path that neither checked nor updated the budget.
2. Table boundaries prefer a row break. Inside a table the space fallback left
the continuation starting mid-row, so the re-emitted header was followed by a
malformed row. Fixes single-append boundaries completely; under incremental
cadence a fill append can still land mid-row before the room runs out, which
this does not address (see the PR thread).
3. `renderContextCloser`'s fence check now mirrors `hasFenceCodeContent`
exactly: non-whitespace after the language line, not merely a newline, so
```py\n no longer gets a closer. The comment claimed the mirror; now it holds.
Also trimmed the rationale at the boundary-closer catch site. It repeated the
"first append a wrong headroom assumption rejects, two things had to be right"
argument, which the reviewer measured and retracted: under byte counting the
fill append dies first and rollOver is never reached, so the two modes are
sequential rather than compounding, and the trigger window is only a few bytes
wide. The guard stays — an unguarded append that wedges rollover, directly above
a guarded one that doesn't, is worth three lines on the asymmetry alone — but the
codebase should not preserve reasoning that has been disproven.
## What changes
This replaces the managed Channels adapter and transport stack with live
sessions over Realtime Gateway.
- Channels use the standard AgentRunner and canonical AG-UI history for
each turn.
- One admitted delivery runs one prompt; multiple agent calls run in
order; concurrent calls fail with a bounded protocol error.
- Slack and Teams reuse their native renderers to emit destination-free
provider effects.
- Files, rich controls, interaction handlers, and provider cursors cross
the live-session protocol with bounded payloads.
- The old claim mapping, HTTP fallback, render batches, listener
election, in-memory transport, and legacy adapter code are removed.
- The public `@copilotkit/channels` umbrella remains limited to public
provider adapters; the managed launcher stays in
`@copilotkit/channels-intelligence`.
Companion service PR and kind proof:
https://github.com/CopilotKit/Intelligence/pull/638
## Why
Managed delivery must use the same AgentRunner path as other
Intelligence runs. SDK code emits provider-neutral effects; the trusted
Gateway owns credentials, destinations, admission, retries, and terminal
outcomes.
## Validation
- `pnpm nx run-many -t build,check-types,test -p
@copilotkit/channels-core,@copilotkit/channels-intelligence,@copilotkit/channels-slack,@copilotkit/channels-teams,@copilotkit/channels,@copilotkit/runtime`
— 31 tasks passed
- `pnpm nx test @copilotkit/channels-teams` — 89 passed
- `pnpm vitest run scripts/release/lib/channels-umbrella.test.ts` — 8
passed
- `pnpm verify:channels-umbrella` — packed snapshot, dependency
resolution, and TSX consumer passed
- affected package pre-commit tests, publint, and API type checks passed
- changed-file Prettier and `git diff --check` passed
## Known unrelated check
`pnpm nx build demo` now compiles past the prior Channels telemetry
dependency leak, then fails on the existing AG-UI 0.0.51 versus 0.0.57
private `_debug` type mismatch.
## Why
A `scope=all` canary took ~20 minutes. Profiling run
[30473483745](https://github.com/CopilotKit/CopilotKit/actions/runs/30473483745)
step-by-step showed almost none of that was real work.
The dominant cost: `npx --yes npm@11.15.0 publish` ran **once per
package**, and npx re-resolves the spec against the registry on every
invocation — **~16s of each package's ~21s**, leaving ~5s of actual pack
+ upload. That is why `all` hurt worst: 26 packages paid ~7 minutes of
pure npx overhead. The same loop exists in the stable path, so a
16-package monorepo release paid over 4 minutes of it too.
## What changed
1. **`scripts/release/lib/npm-cli.ts` (new)** — installs the pinned npm
**once** into a throwaway prefix and memoizes it; `prerelease.ts` and
`publish-release.ts` both reuse the binary. The version pin now has a
single source of truth (it was duplicated as a literal in both scripts).
A throwaway prefix rather than `npm i -g` keeps it hermetic, so running
these scripts locally doesn't rewrite a developer's global npm.
2. **Shallow checkout for canaries only** — prereleases push no tag, cut
no GH Release, read no commit range, and build with `nx run-many` (not
`affected`), so there is no merge base to resolve. Stable **keeps
`fetch-depth: 0`**, because its publish job resolves and pushes tags out
of that checkout's `.git` (shipped via the workspace artifact).
3. **Artifact slimming** — `compression-level: 0`, since the payload was
already gzipped by "Pack workspace" and was then re-deflated into the
artifact zip for ~no size win.
4. **`notify` job skipped for canaries** — the builder already returns
`should_post=false` for `mode=prerelease`, success *and* failure
(`build-release-notification.ts:196`), and the self-watchdog's Slack
post is already gated off since `npm_intended` is false for a
prerelease. So the job could only ever checkout + install + compute
"post nothing", on the critical path because canary.yml waits for the
whole run. `python_publish=true` still reaches it.
5. **pnpm store cache**, restore-only on canary refs (see the note
below).
6. Orchestrator polled for the delegated run *after* a 6s sleep; now
polls first.
## Results
Real `scope=all` publishes of all 26 packages across all three scopes:
**~20 min -> ~4m40s**.
| Step | Before | After |
|---|---|---|
| **Publish loop** (26 pkgs) | ~546s (26 x 21s serial) | **38s** (4 at a
time) |
| publish: Install deps | 41s | **27s** (root + packages only) |
| build: Checkout | 48s | **6s** |
| build: Pack workspace | 33s | **12s** |
| build: Upload workspace | 27s | **2s** |
| publish: Download / Unpack | 31s / 6s | **1s / 2s** |
| notify job | 76s | **skipped** |
| **build job total** | 261s | **164s** |
| **publish job total** | 107s + publish | **81s incl. publish** |
Measured runs:
[30504035212](https://github.com/CopilotKit/CopilotKit/actions/runs/30504035212)
(4m44s) and
[30504493051](https://github.com/CopilotKit/CopilotKit/actions/runs/30504493051)
(4m52s).
Honest note on those two totals: the second is 8s *slower* end-to-end
even though it added the narrow install, because its build job drifted
+17s on unrelated runner variance (Setup pnpm 4s->8s, build 91s->95s).
The narrow install did exactly what it should at step level —
publish-job install 41s -> 27s. Treat ~4m40s +/- 15s as the real figure;
per-step numbers are the controlled measurement, end-to-end totals carry
runner noise.
The publish log now shows one `Installing npm@11.15.0`, then `Publishing
26 package(s), 4 at a time...`, then 26 `Published` lines in 34s.
The build job (164s: 95s `nx run-many` + 46s full install) is now the
dominant remaining cost and is deliberately untouched — see Further
options below.
## On the pnpm cache — deliberately restore-only
Worth calling out because my first attempt here was wrong.
`publish-release.yml` was the only workflow in the repo with no pnpm
store cache, and one observed canary spent **9m08s** installing on
tarballs arriving at 2–49 KiB/s. But adding a plain read/write cache
turned out to be a *net negative* for canaries:
canary.yml mirrors the source branch to a unique
`canary/<slug>-<run_id>-<attempt>` ref, and Actions scopes cache entries
to the writing branch (plus the default branch). A cache saved from a
canary ref is **unreachable by every later canary** — measured at 874
MiB of orphan per run, competing for the repo's shared 10 GB budget and
evicting entries other workflows depend on. It also cost more than it
bought: 20s + 26s of post-job saves against a 25s faster install.
So canaries **restore read-only** (free win whenever main has an entry,
zero cost and zero pollution when it doesn't), and stable releases —
which run on main, where a write is durable and reused — do the writing.
Verified in CI that the read-only arm runs, the read/write arm is
skipped, and no `Post Cache pnpm store` save step is planned at all.
Honest framing on the mean: the cache is roughly break-even on typical
time (18s restore + 21s install vs 46s cold). Its value is bounding the
registry-throttle tail, not shaving the average.
## Testing
- **Real `scope=all` canary publish**
([30503117837](https://github.com/CopilotKit/CopilotKit/actions/runs/30503117837),
success): all 26 packages published across monorepo + angular + channels
under one shared canary id, confirming cross-scope `workspace:` deps
still resolve in-run. Verified live on the registry:
`@copilotkit/react-core@1.64.2-canary.perfall1`,
`@copilotkit/channels@0.4.1-canary.perfall1`,
`@copilotkit/angular@0.3.1-canary.perfall1`.
- **Dry-run canary**
([30502850575](https://github.com/CopilotKit/CopilotKit/actions/runs/30502850575),
success): confirmed the shallow checkout, artifact slimming, and that
the `notify` job is genuinely skipped.
- **Cache-fix validation run**: step conclusions confirm `Restore pnpm
store (prerelease — read-only)` → success, `Cache pnpm store (stable —
read/write)` → skipped, and no post-job save step exists.
- `143` release-script tests pass, including 6 new ones for the helper
(install-once, memoization, the OIDC `>= 11.5.1` version floor, and that
a failed install throws rather than silently falling back to the too-old
ambient npm).
- `actionlint` clean on all four release workflows; `shellcheck
--severity=warning` clean on `scripts/release/**/*.sh`;
`verify-release-scope-dropdowns.sh` clean; `oxlint` 0 warnings/errors;
`oxfmt` applied.
- Live probe of the helper: installs npm 11.15.0 in 3.2s, second call
0ms, same path, resolved binary reports `11.15.0`.
- `prerelease.ts --dry-run` still enumerates all 9 channels packages.
## Notes
- `@copilotkit/*@canary` now points at `…-canary.perfall1` from the
verification publish. Content-wise that is main's package code (this
diff only touches release scripts/workflows). Happy to republish from
main if you'd prefer the tag moved.
- No npm OIDC surface changed: the workflow filename, the `npm`/`pypi`
environments, and the trusted-publisher bindings are all untouched.
`npm` stays >= 11.5.1 (pinned at 11.15.0, now enforced by a test).
## Further options (want your call before I touch these)
The build job is now the long pole. Each remaining lever trades
something I don't think is mine to trade:
- **Bigger Depot runner for the build job** (~95s of CPU-bound `nx
run-many`). Depot is already used elsewhere in this repo, but it
requires `id-token: write` — on the one job deliberately kept
credential-free, and the job that runs arbitrary package build scripts.
npm's trusted-publisher record also demands the `npm` environment claim
which the build job lacks, so a token minted there almost certainly
cannot publish. Still your call, not mine.
- **Merge build + publish** (~60s: second install, artifact round-trip,
job spin-up). Directly erodes the build/publish credential boundary this
file's comments defend.
- **Scope-aware build.** An `angular` canary builds all ~70 projects to
publish 1 package. Zero help for `scope=all`, large win for single-scope
canaries.
- **Narrow the build job's install too.** `nx run-many -t build
--projects=packages/**` shouldn't need examples/ or showcase/ deps.
Lower confidence than the publish-job version, since nx builds a full
project graph.
Nx caching is a dead end here: Nx Cloud stays off per team decision, and
a local `.nx` cache hits the same ephemeral-canary-ref scoping problem
documented above.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Addresses maxkorp's second pass. All three reproduced locally first.
1. The boundary closer append was unguarded, and its failure disabled rollover
entirely (the worst of the set). It threw out of `rollOver` with `curTs` still
set, so the next flush recomputed a full message, re-entered `rollOver`, and
failed on the same closer forever. Probe: a transport rejecting closer-only
deltas delivered 11,000 of 37,410 chars across ONE message with zero
rollovers. The control confirms the asymmetry — the same input with
`stopStream` failing instead delivers everything, because that call was
already guarded.
This compounds with the unit question: the closer is appended exactly when the
message has just filled, so it is the FIRST append a wrong headroom
assumption rejects — and its rejection switched the preventive fix off before
it could fire. Now guarded with the reasoning already written for
`stopStream`: degraded markdown on one message beats losing the remainder.
2. Implausibly long fenced-code "languages" are clamped. `detectOpenContext`
reports everything up to the first newline after ``` as the language, so a
minified blob became a multi-kilobyte preamble re-injected into every
continuation: 5,004 chars of duplicated preamble atop six consecutive
messages, 54% useful, and a final message 1% useful. Past
MAX_FENCE_LANG_CHARS we re-open with a bare fence, which still preserves code
formatting where dropping the opener would not. Re-injected preamble drops
from 5,004 chars to 4.
3. Continuation count is bounded. A 500k-char reply became 46 Slack messages,
uncapped. Not a regression (the legacy chunker is uncapped too), but this PR
exists because a runaway became repeated copies in a channel, and silently
turning one over-long reply into dozens of real messages is that failure in a
different hat. DEFAULT_MAX_MESSAGES with a visible truncation marker, so the
cut is stated rather than silent.
Also removed the write-only `finished` field the review spotted.
Deferred, tracked in the PR thread: the emphasis closer's flanking position
(needs a real-workspace check, since Slack is not strictly CommonMark) and
`flushChunk`'s permanent chunk-disable after a transient continuation failure.
The publish job reinstalled all 4608 projects' dependencies (~41s) to run
two things: `pnpm tsx` and `pnpm pack`.
A root-only install is NOT sufficient — verified locally that pnpm pack
then dies with ERR_PNPM_CANNOT_RESOLVE_WORKSPACE_PROTOCOL, because pack
resolves each package's `workspace:` deps into real version ranges and
needs its workspace siblings installed to do it. That resolution is what
makes cross-scope canary deps pin the same-run version, so it is
load-bearing.
The true minimum is root + packages/**: measured 17s vs 46s locally, and
the packed tarball still carries resolved ranges (^0.4.0, ^1.64.1) with no
leftover workspace: refs. Everything trimmed is examples/ and showcase/ app
dependencies that no publish step touches.
Safe because no publishable package declares prepack or prepare, so pnpm
pack runs no lifecycle scripts; voice's prepublishOnly fires for neither
pnpm pack nor `npm publish <tarball>`.
Stable keeps the full install — its dependency surface is wider
(lib/notion.js, the umbrella verifier's root script) and it is the
highest-stakes path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
After the npx fix each publish is ~4.7s and almost entirely a registry
round-trip, so a 26-package scope=all canary still spent ~125s waiting
serially. Publish 4 at a time (CANARY_PUBLISH_CONCURRENCY=1 restores
serial for debugging).
This weakens no ordering invariant. prerelease.ts's own header already
documents that the cross-scope graph has cycles (runtime ->
channels-intelligence, channels-core -> core), so no serial order avoided
publishing a package before the same-run version it pins.
Per-package output is captured and replayed as one block rather than
inherited, since a pool would otherwise interleave several npm publishes
line-by-line — and that log is the only forensic record when a canary
half-publishes. Every package is attempted even if others fail, so one
report names all of them; main() now exits non-zero on failure rather
than letting an unhandled rejection pass the step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
canary.yml mirrors the source branch to a unique
canary/<slug>-<run_id>-<attempt> ref, and Actions scopes cache entries to
the writing branch (plus the default branch). A cache saved from a canary
ref is therefore unreachable by every later canary — measured at 874 MiB
of orphan per run, competing for the repo's shared 10 GB budget and
evicting entries other workflows depend on.
It also cost more than it bought: 20s + 26s of post-job saves against a
25s faster install.
Canaries now restore read-only (free win when main has an entry, zero
cost and zero pollution when it doesn't); stable releases, which run on
main where a write is durable and reused, do the writing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Addresses maxkorp's review. All three findings reproduced locally before fixing.
1. Budget is now UTF-8 bytes, not chars. The "safe whichever way Slack counts"
claim covered char-cap vs per-call-cap but not chars vs BYTES, and the
incident datapoint was English, where the two are 1:1. Under a byte-denominated
cap a char budget never fires: 漢字 x 10,000 delivered 4,000 of 20,000 chars
with ZERO rollovers and 33 swallowed msg_too_long — the original bug, intact,
for every non-Latin-script user. Cyrillic/Greek/Hebrew/Arabic break above ~6k
chars, CJK above ~4k. UTF-8 byte length is always >= char count, so an 11k-byte
budget is under a 12k ceiling in either unit. Costs extra messages for
non-Latin replies if the ceiling turns out to be chars; correct side to err on.
`spanWithinBudget` walks whole code points, which subsumes the previous
`avoidSurrogateSplit` guard for every boundary rather than patching each one.
2. `finish()` no longer drops the tail. It enqueues exactly one flush, so a
continuation `startStream` that throws there had no later flush to retry it:
probed at 11,000 of 25,000 chars delivered, no fallback, first message
finalized cleanly, so the turn reported success and the platform never
replayed it — a fresh silent-truncation path in a file whose purpose is to
stop silently truncating. Now drains with bounded retries, then hands the
remainder to the legacy transport seeded with ONLY the undelivered tail (not
the whole buffer, which is what makes failing over mid-reply duplicate text).
`onStartFailure` is deliberately not fired: a transient continuation failure
should not mark the whole workspace legacy.
3. Tables are re-opened across a boundary. `detectOpenContext` models fences,
inline code, and emphasis but not tables, so continuations started mid-row with
no delimiter above them and Slack rendered literal pipes — a regression in the
native-table rendering that motivates this transport, and long generated tables
are a common way to exceed the cap. `tableHeaderToReopen` re-emits the header
and delimiter rows.
Also from the review: the fence closer now mirrors `hasFenceCodeContent` so a
boundary just after ```lang no longer emits an empty code block; the
fits-in-one-message guard uses 10,900 rather than 5,000 so it actually pins the
boundary; and the config-only per-call branch has a test.
Deferred: the emphasis closer can land after whitespace and so not right-flank
under CommonMark. Slack's renderer is not strictly CommonMark, so this wants
confirming against a real workspace in the same manual pass as the cap unit.
The canary flow took ~11.5 min steady-state (and 20 min in an observed
run). Measured from run 30473499191, the time went to five avoidable
places rather than to real work.
1. `npx --yes npm@11.15.0 publish` ran per package, and npx re-resolves
the spec against the registry on EVERY invocation: ~16s of each
package's ~21s. A 9-package channels canary paid ~2.4 min of pure npx
overhead; a 16-package monorepo release paid over 4 min. Hoist the
pinned npm into lib/npm-cli.ts, install it once into a throwaway
prefix, and reuse the binary.
2. publish-release.yml was the only workflow in the repo with no pnpm
store cache, so all three jobs installed 4608 packages cold every
time. Usually ~45s each, but registry-bandwidth bound and heavy
tailed: the observed run spent 9m08s here on tarballs arriving at
2-49 KiB/s. Add the same node-version-keyed cache the rest of CI uses.
3. The notify job ran for canaries only to compute "post nothing" — the
builder already returns should_post=false for mode=prerelease and the
self-watchdog is already gated off. ~85s of dead work on the critical
path, since canary.yml waits for the whole run. Skip the job, keeping
it reachable for a python_publish dispatch.
4. The build job fetched full history for canaries, which need none (no
tag, no GH Release, no release-note commit range, and `nx run-many`
resolves no merge base). That rode along in the 837 MiB workspace
artifact too. Shallow-fetch prereleases; stable keeps depth 0 because
its publish job pushes tags out of that artifact's .git.
5. Two smaller ones: the artifact was gzipped and then re-deflated into
the artifact zip (compression-level: 0), and the orchestrator's
run-discovery loop slept 6s before its first poll.
Verified: 143 release-script tests pass (6 new for the npm-cli helper),
actionlint + shellcheck + the scope-dropdown guard are clean, the
prerelease dry-run path still enumerates all 9 channels packages, and a
live probe confirms the helper installs npm 11.15.0 once (3.2s) and
memoizes thereafter (0ms).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects found by adversarially probing the rollover added in the previous
commit. Both were reachable at the default 11k cap.
1. Unbounded message creation. The continuation re-opener is synthetic text
charged against the new message's budget, so a context whose opener is itself
larger than the cap filled each fresh message with nothing but its own
preamble and rolled over again — forever, never advancing `curPosted`.
`detectOpenContext` reports everything up to the first newline after ``` as
the fence "language", so an agent emitting a >11k whitespace-free blob
(minified JSON, base64, a long log line) straight after a fence triggers it.
Probed at the default cap: `posted` frozen at 11,000 while `startStream` was
called 41+ times, bounded only by the probe's own budget. In production that
is an unbounded stream of Slack messages — worse than the truncation it
replaced. The opener is now dropped when it cannot leave
MIN_MESSAGE_PROGRESS_CHARS of room, so every message carries text and the
loop always terminates.
2. Surrogate pairs split across boundaries. Whitespace-free text reaches
`breakPoint`'s hard-cut fallback routinely (CJK, emoji runs, base64), and
cutting at an arbitrary UTF-16 offset left a lone high surrogate ending one
message and an orphaned low surrogate starting the next — a broken glyph per
boundary. Concatenating the messages still compares equal, which is why the
existing preservation assertions passed straight through it. All boundaries
now step back off a pair via `avoidSurrogateSplit`, including the per-call
12k slices, which had the same latent exposure before rollover ever existed.
Both are pinned by tests that fail against the previous commit: a runaway guard
that turns a rollover loop into a failed test rather than a hung one, and a
per-message surrogate-edge assertion (the join-and-compare check cannot see it).
Slack enforces a cumulative text cap per streamed message, not just the
documented 12k-per-`markdown_text`-call cap. `NativeMessageStream` only honored
the per-call cap and kept appending to a single message, so once a reply passed
the cumulative ceiling every further `chat.appendStream` was rejected with
`msg_too_long` — permanently, for the rest of the run.
Observed in dev: a ~23k-char reply streamed 11,607 chars into one message and
then failed every subsequent append. The remaining text was silently dropped
(the append error is swallowed by design), the turn never reached finalize, and
the egress operation replayed from its uncheckpointed cursor — posting a fresh
truncated copy per attempt until it hit max_attempts.
`flushText` now rolls over: when a message fills and text remains, the boundary
is frozen on a line (else word) break, any open markdown construct is closed,
the message is finalized, and the reply continues in a new message that
re-opens that construct. Both append loops share one rollover-aware path, and a
continuation `startStream` failure no longer falls back to the legacy transport
— that sink replays the whole buffer, which would duplicate everything already
streamed.
The soft per-message limit is 11k, below the smallest total Slack was observed
to accept, leaving headroom for the closers a boundary appends.
The test transport did not model the cumulative cap at all, which is why this
survived the suite; it does now, and the case that asserted "keeps a long reply
in ONE message" was asserting the bug.
## Problem
Main CI cannot compile the channels-intelligence test suite because
ReplyTarget and Sub are referenced without declarations.
## Why
The merged Slack tool-status regression coverage introduced the
references but omitted the corresponding type declarations. Every
dependent Nx workflow fails during the shared package build.
## Fix
Restore the ReplyTarget type import and the local Sub subscriber helper
type in render-events.test.ts.
Validation: pnpm exec nx run @copilotkit/channels-intelligence:build;
pnpm exec nx run @copilotkit/channels-intelligence:test; pnpm exec nx
run @copilotkit/channels-intelligence:check-types.
## Problem
Slack threads render every tool invocation as visible progress, leaving
noisy `Used …` rows ahead of the final answer.
## Why
The direct Slack renderer treated tool status as enabled when
`showToolStatus` was omitted, while the managed Intelligence renderer
always emitted tool lifecycle frames. The managed renderer's opt-in was
not reachable through the supported `createChannel()` and
`CopilotRuntime` API.
## Fix
- Hide tool-call progress by default for direct and managed Slack
routes.
- Expose `createChannel({ showToolStatus: true })` as the managed
Channels opt-in and carry it through runtime activation to the
Intelligence launcher.
- Preserve `slack({ showToolStatus: true })` as the direct Slack opt-in.
- Keep existing managed behavior for non-Slack routes.
- Continue capturing and executing tool calls while their status frames
are hidden.
- Add regression coverage for the public managed API, legacy and
native-streaming Slack, and Realtime Gateway paths.
Paired with https://github.com/CopilotKit/Intelligence/pull/632.
Three fixes from the OSS-670 investigation, where a dev gateway restart
landed mid-turn: the agent's answer was produced and then thrown away,
the end user got a generic error, and the bot went silent for 19 minutes
with nothing in its logs.
## A revoked lease is a fenced outcome, not a turn failure
`CHANNEL_DELIVERY_LEASE_INVALID` means another owner holds the delivery
— app-api fences every intent on `leased_until > now()` AND the
lease-token hash, so nothing we send can be accepted and redelivery is
app-api's job.
- The guard lives in `IntelligenceAdapter.dispatch`, the site that
actually runs. An earlier revision guarded only the realtime transport,
which is unreachable in production because `dispatch` catches every turn
error and nacks first — so a fenced delivery still put a doomed
`channel.delivery.fail.v1` on the wire, and that second 409 is what
surfaced as `nack after turn failure failed` followed by `no delivery
state`.
- The code is no longer retried through the 3-attempt render-batch
ladder.
- A `fail` intent that is itself fenced is logged and dropped rather
than rejected.
- The predicate lives in `transports.ts` because both transports surface
the same code (the HTTP path as a 409 body).
The adapter-dispatch test is a real regression test: reverting the guard
makes it fail on `channel.delivery.fail.v1` being pushed.
### Abandoning is a third outcome, not a silent one
Swallowing the coded error means the `DeliverySource` observes a
**successful** `onDelivery` — so neither its own revoked-lease branch
nor `ack` runs, and those are the only two places the per-delivery lease
record is dropped. Without more, every fence leaked a lease token,
delivery scope and accepted-seq high-water map for a delivery this
runtime can never speak for again (app-api may already have handed it to
another runtime, so the id may never be seen here again).
`DeliverySource.abandon(deliveryId, reason)` is that third outcome
alongside ack/nack: retire local state, send nothing. It is synchronous
and wire-free by design, optional on the interface so existing
implementations stay valid, and implemented on
`RealtimeGatewayTransport` (`deliveries`), `HttpDeliverySource`
(`leases`) and `InMemoryDeliverySource`. The transport's internal
revoked-lease catch routes through it too, so there is one retirement
path rather than two.
Both wiring tests fail with the `abandon` call reverted — the adapter
test on an empty `abandoned[]`, and the composed adapter+transport test
because its probe `nack` pushes a second, doomed
`channel.delivery.fail.v1` off the retained state.
## A refused reconnect is diagnosed instead of retried in silence
`socket.onError` returned early once the socket had connected, so a
refused *reconnect* recorded nothing and Phoenix retried forever with no
output. The cause is now recorded, and when the transport exposes no OS
code an HTTP probe of the endpoint runs once per outage.
The probe is unauthenticated, so it deliberately does **not** claim your
key is revoked — its 401/403 would look identical with a perfectly good
credential. It establishes reachability: 5xx reads as an
unhealthy/restarting gateway, anything else as "reachable, handshake
refused — verify the project API key".
## Honest status, loud while down
`gave_up` maps to status `error`, but Phoenix keeps retrying underneath
and a successful rejoin restores `online` — the log now says so instead
of reading as terminal. The drop cause travels with each state
transition (it was previously `{ meta: undefined }`), and a "still down"
line repeats while the session is out, including after `gave_up`.
Channel status semantics and the "manager never re-activates on a drop"
contract are unchanged.
## Not included
Delivery-lease renewal has no owner across a gateway restart — draining
stops the channel process that renews it, so a long turn loses its lease
to expiry regardless. That, and bounding an in-flight turn whose lease
is already dead, are follow-ups on the ticket. Retiring the local record
(above) stops the leak; it does not keep the lease alive.
Tests: `channels-intelligence` 13 files / 244 tests; `runtime` 127 files
/ 1799 tests; `check-types` clean on both.
Tyler Slaton