## Problem
Managed Channel deliveries can request a canonical thread lock with the
inner agent ID, or `default` when that ID is unset. Intelligence owns
the thread under the declared Channel name, so the lock fails with
`THREAD_AGENT_MISMATCH`.
## Why
The SDK treated the agent object identity as the managed Channel
identity. Those values are independent: `createChannel({ name })`
declares the managed thread owner, while an agent ID is optional and may
differ.
## Fix
Pass the declared Channel name into the managed delivery adapter and use
it for canonical runs. Add a public `thread.runAgent()` regression test
that proves an agent with a different ID still runs under the Channel
name.
Validated with the Channels Intelligence test suite, type check, and
build; the runtime Channel manager tests, type check, and build; repo
lint; and the affected-package pre-commit checks.
> [!IMPORTANT]
> **Re-grounded 2026-07-30.** This PR was opened against the old
queued-egress architecture and originally referenced OSS-677.
Intelligence#638 ("replace queued delivery with live sessions") has
since landed, moving managed provider egress into the realtime gateway
and deleting the egress-lease, outbox, and render-frame machinery
entirely. OSS-677 was canceled as a result.
>
> **The SDK defect this PR fixes is unaffected and still present on
`main`** — `native-stream.ts` is live, still owns Slack streaming
cadence, and still has no continuation rollover. The work is now tracked
as **OSS-685**.
>
> What *did* change is the blast radius (see "What the symptom looks
like now") and the platform-side half, which no longer exists to fix.
## What
A managed Slack bot asked to write a novella posted **five truncated
copies** of it. The proximate cause is in this file.
`NativeMessageStream` honored Slack's documented
12k-chars-per-`markdown_text`-call cap but never the *cumulative* cap on
what one streamed message can hold. Once a reply crossed that ceiling,
every further `chat.appendStream` was rejected `msg_too_long` —
deterministically, for the rest of the run.
The assumption was stated explicitly in the file's own header, and it is
wrong:
> A single streamed message holds the whole reply: Slack documents no
cumulative per-message cap, only a **12k char limit per `markdown_text`
call** … (no multi-message splitting — that was a `chat.update`-era
workaround).
### Evidence (dev, 2026-07-29 23:24–23:27 UTC)
A ~23,279-char reply streamed 11,607 chars into one message, then failed
every subsequent append. Egress op `019fb031-97eb-…`:
| field | value |
| --- | --- |
| `attempts` | 5 / 5 |
| `lease_generation` | 5 |
| `accepted_high_water` | 386 |
| `applied_high_water` | **-1** |
| `payload.posts` | **5 distinct Slack ts** |
| `status` | wedged in `sending` |
The dropped text is invisible by design (per-append failures are
swallowed and logged), so the turn never reached `finalize`, never
checkpointed, and the platform replayed it from frame 0 — a fresh
`chat.startStream` and a fresh truncated copy per attempt, five times,
then wedged. Worker logs show 3,512 `msg_too_long` rejections in 25
minutes; the retry burn also starved other projects' lanes.
> [!NOTE]
> That table describes the **pre-cutover** schema.
`channel_egress_operations`, `attempts` / `max_attempts`,
`lease_generation`, and the render-frame high-water columns were all
dropped by Intelligence#638 (migration `000008.sql`). The evidence
stands as the historical record of how the defect was found; the columns
no longer exist to query.
### What the symptom looks like now
The root cause is identical — the cumulative cap is still unhandled —
but the failure mode has changed:
- **Then:** per-append failures were swallowed, the turn never
finalized, and the platform replayed from frame 0, producing N truncated
copies and burning the attempt budget.
- **Now:** `f1c4b6ca79` ("keep managed Slack replies atomic") added a
`strict` flag that the managed live-session path sets. In strict mode an
`appendText` / `stopStream` failure re-throws instead of being
swallowed, so `msg_too_long` surfaces as **one failed turn** rather than
five truncated copies plus a wedge.
That is strictly better, and it removes the retry-storm half of the
incident. It does not deliver the reply. The customer still loses a long
answer — which is what this PR fixes.
The direct/self-hosted path (SDK holds the token, `strict` unset) still
swallows the failure and silently truncates.
## How
`flushText` now rolls over. When a message fills and text remains:
freeze the boundary on the last line break (else last space, so a word
is never torn), close any markdown construct left open, `stopStream` it,
and continue in a fresh message that re-opens that construct. This is
the same shape as the legacy `ChunkedMessageStream`, whose splitting
logic was removed as an unnecessary `chat.update`-era workaround.
Three decisions worth review:
- **Soft limit 11k UTF-8 bytes** (`messageByteLimit`, configurable;
changed from chars after review). Slack's cumulative cap is
undocumented; 11,607 chars were *observed accepted*, so 11k sits
provably below the smallest known-good total with headroom for the
closers a boundary appends. Crossing the cap is unrecoverable for the
whole reply, not merely a truncated append, so the asymmetry favors
staying under.
- **A continuation `startStream` failure no longer falls back to
legacy.** The legacy sink is seeded with the entire accumulated buffer,
so failing over mid-reply would re-post everything already streamed —
trading one duplication bug for another. First-message failures still
fall back, preserving "opting in can never break a bot"; a continuation
failure just propagates and the next flush retries the boundary.
- **`renderContextCloser` is new rather than reusing
`autoCloseOpenMarkdown`.** The latter inserts closers *before* trailing
whitespace, which an append-only transport cannot express —
`appendStream` sends deltas and Slack has no "un-append". The new helper
is symmetric with the existing `renderContextOpener` and only ever
appends.
Both append loops (`flushText` and the pre-chunk `flushTextInline`) now
share one rollover-aware path instead of carrying duplicate cap logic,
and `flushChunk` re-checks its target after a text flush, since a
rollover retargets the stream.
### Interaction with `strict` mode
Worth a reviewer's eye now that both exist: a rollover must not be
defeated by the strict re-throw firing before the continuation is
attempted. The rollover path is entered on the *budget check*, not on a
caught provider error, so it should run ahead of any throw — but this
branch predates `strict` and the two have not been exercised together.
Needs a test that runs the rollover with `strict: true`.
## Why the suite didn't catch it
The fake transport modelled no cumulative cap at all, so the failure was
unreachable in tests. It models one now. The existing case asserting
*"keeps a long reply in ONE message, chunking appends under the 12k
per-call cap"* was asserting the bug, and is replaced by the rollover
expectations plus a fits-in-one-message regression guard.
## Related
- **OSS-685** — the tracking ticket (successor to the canceled OSS-677).
Also carries the open question of whether the gateway needs its own
guard:
`apps/realtime-gateway/lib/realtime_gateway/channels/provider_executor.ex`
now makes the managed Slack call and has no `msg_too_long`, length, or
splitting handling of its own.
- **The platform-side half is gone, not deferred.** The original plan
was app-api work — classify `msg_too_long` terminal instead of burning 5
attempts, and stop the re-claim storm. Intelligence#638 deleted the
attempt/lease machinery that produced both, so there is nothing left to
fix there. `9a6ee861` and the OSS-677 branch fixes were merged into code
that has since been removed.
- **#6238** (`fix(channels): key render lanes per delivery attempt…`)
was **closed on 2026-07-30** — it was written against render lanes and
per-delivery attempts, both retired by the cutover. The trade-off the
two PRs used to share (that PR knowingly accepting duplicate provider
output on redelivery) no longer has a referent, so nothing in this PR
depends on it.
- `packages/channels-slack` is the only live site; `bot-slack` ships
`dist` only.
## Testing
Package suite, build, and both typecheck projects, on this branch off
`main` **as of the original run** — this branch predates the
Intelligence#638 cutover and has not been re-based since:
```
$ pnpm build # tsc -p tsconfig.json
(no output)
$ pnpm check-types # tsconfig.json + tsconfig.check.json
(no output)
$ npx vitest run
Test Files 23 passed (23)
Tests 286 passed (286)
```
New coverage: rollover past the cumulative cap (no char lost or
reordered, every message finalized, per-call cap still respected,
`firstTs` still the first message), line-boundary splitting, code-fence
re-opening across a boundary, resume-after-transient-append-failure, and
the fits-in-one-message guard.
> [!WARNING]
> **Before merge:** re-base on current `main` and re-run.
`native-stream.ts` gained `strict` mode (`f1c4b6ca79`) and other changes
after this branch was cut, so the numbers above are stale and the
rollover has not been tested against `strict: true`.
> [!NOTE]
> **Updated after review** (see the two review comments for full
detail). The soft limit is now denominated in **UTF-8 bytes**, not
chars: the incident reply was English, where the two are 1:1, so it
could not tell us which unit Slack's ceiling uses — and under a byte
ceiling a char budget never fires, silently truncating any CJK/Cyrillic
reply exactly as before. Bytes are >= chars, so an 11k-byte budget is
safe in either unit.
>
> Also fixed after review: `finish()` no longer drops the undelivered
tail when a continuation `startStream` fails, and markdown tables are
re-opened across a boundary.
>
> A manual >12k reply against a real workspace is still worth doing
before ship — and it should be **mostly CJK or Cyrillic**, since that is
where the byte and char budgets disagree most.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Coverage on native-stream.ts was 87.5% stmts / 80.45% branches; now 92.7% /
88.5%. The gaps were concentrated in code this branch added, and two of them
were places I had claimed coverage that did not exist.
Genuinely uncovered logic, now tested:
- `renderContextCloser`'s inline-code and emphasis closers, and the
`hasFenceCodeContent` mirror added in the previous commit — the closer is
appended at every boundary and none of its three branches were exercised.
- `tableHeaderToReopen`'s "no longer inside the rows" guard, so a boundary landing
after a table has ended does not inject a stray header.
- The 2-byte UTF-8 width class (Cyrillic). CJK covered 3-byte and emoji 4-byte;
the class between them was untested.
- The `callEnd < byteEnd` branch. The test that claimed it never reached it: with
a 40k budget and 30k of ASCII the whole reply fits, so the first branch won and
the per-call path was never entered. Retargeted with multi-byte text.
- Rollover `stopStream` failure, non-strict (proceeds) and strict-with-a-failed
closer (reports the closer, not the consequence).
- Truncation marker failure, non-strict (logged) and strict (reported).
- A legacy tail fallback that itself fails.
- Chunk behaviour around a rollover: delivered to the current message, degraded
when the continuation cannot be opened, and skipped entirely when a strict text
failure rejects the shared flush queue — the last of which corrected my
assumption that `flushChunk` still runs in that case. It does not: `.then()` on
a rejected promise skips its callback.
Four guards are unreachable through the public API and are now commented as
deliberately defensive rather than left looking like missing tests: the
`roomBytes <= 0` and "not even one code point fits" rollover checks (every
filling branch rolls over in the same iteration), `appendSlice`'s empty-delta
stall guard (both boundary helpers return an index strictly greater than
`curPosted`), and `truncate`'s own re-entry check (`appendPending` short-circuits
first). Kept as loop-safety invariants.
The residue is pre-existing or upstream: `append()`'s legacy forwarding, the
zero-interval `scheduleFlush` path, `flushChunk`'s start-failure degradation, and
`flushTextInline`'s strict rethrow.
## What
Bumps `ag-ui-adk` from `0.6.3` to `0.7.0` in the ADK starter templates
(`examples/integrations/adk` and `examples/integrations/adk-angular`),
and
regenerates both `uv.lock` files.
## Why
`npx create-ag-ui-app@latest` → ADK scaffolds from
`examples/integrations/adk` (via `copilotkit create -f adk`, which
resolves
`-f adk` to `copilotKitStarter("examples/integrations/adk")`).
That starter pins `ag-ui-adk==0.6.3`. A2UI generative-UI rendering for
ADK
landed in `ag-ui-adk` **0.7.0** (OSS-158, ag-ui#1955), so every ADK
project
scaffolded today ships a backend with no A2UI support at all.
## Compatibility
`ag-ui-adk` 0.7.0 requires `ag-ui-protocol>=0.1.15` (starter pins
`0.1.18` ✓),
`google-adk>=1.28.1,<3.0.0` (unpinned in the starter ✓), and pulls in
two new
transitives: `ag-ui-a2ui-toolkit>=0.0.3` and
`a2ui-agent-sdk>=0.2.4,<0.3.0`.
No manifest changes beyond the `ag-ui-adk` pin were needed.
Note on the large `uv.lock` diff: regenerating the lockfiles re-resolved
`google-adk` from `1.26.0` → `2.5.0`. The previous lock pinned
`google-adk`
below 0.7.0's new `>=1.28.1` floor, so it *had* to move; `2.5.0` is the
latest
release inside the `<3.0.0` ceiling. That major re-resolve (and its
leaner 2.x
dependency tree) accounts for the bulk of the lockfile churn. Both
starters
`uv sync` and boot (`main.py` imports cleanly) against the new tree.
## Verified
- `uv lock --check` clean on both starters
- `uv sync` resolves `ag-ui-adk 0.7.0`
- `from ag_ui_adk import get_a2ui_tool` imports (symbol does not exist
in 0.6.3)
## ⚠️ Follow-up required — this PR alone does not reach users
The `copilotkit` CLI pins the template ref at **build time**:
```js
function getTemplateRef() {
return true ? "a1c9b3147829ac358bae82df651f45a1aea2a437" : "main";
}
```
`copilotkit@4.5.0` is currently pinned to `a1c9b31`, which predates this
change. Merging this PR does **not** change what `npx
create-ag-ui-app@latest`
produces — the CLI will keep serving `ag-ui-adk==0.6.3` until a new
`copilotkit` CLI release is cut whose `getTemplateRef()` points at a
commit
containing this fix.
**A CLI release is required to ship this.**
## Out of scope
Bumping the pin gives the starter the A2UI *capability*. Whether the
scaffolded frontend registers an A2UI catalog (required for anything to
actually render) was not audited here and is left to a follow-up.
Resolves against three main commits that touched this streamer after the branch
point: `strict` mode (f1c4b6ca79), the always-reach-stopStream drain
(9250f22d88), and the zero-interval flush path (5ddcb14233).
`native-stream.ts` auto-merged textually but not semantically — my rollover,
truncation and finish-drain paths were all added after `strict` existed and did
not honour it, so the merge as-generated would have quietly defeated it:
- `rollOver` swallowed both its closer append and its `stopStream`. Now records
the closer failure, still reaches `stopStream` (main's guarantee: never leave a
native stream open), then rethrows under strict with the same earliest-error
precedence `finish()` uses.
- `truncate` swallowed the marker append; now rethrows under strict, with its
terminal state settled *before* the append so a throw cannot re-enter it.
- `finish()`'s drain retries awaited `this.queue` unguarded, which under strict
threw before `stopStream` — reintroducing exactly the bug 9250f22d88 fixed.
Errors are now captured into `queueError` like the initial drain. The retry
also resets `this.queue` first: a strict rejection settles it rejected, and
`.then()` on a rejected promise skips its callback, so the retry would have
re-raised the settled error instead of running a flush.
- The legacy tail fallback is now gated on `!strict`. Strict disables the legacy
fallback by design, so an undelivered tail is surfaced as an error instead of
being routed around it.
Test file conflicted on the fake transport's `appendText`, where main added
`failAppend` and this branch added the cumulative byte cap; kept both.
Two tests added for the strict/rollover interaction, which main's strict tests
predate and so do not cover: a rejected boundary closer surfaces under strict
while the message is still finalized, and an undelivered tail surfaces rather
than engaging the legacy sink.
## What changed
- replaces the Channel live-session SDK with `channel_delivery_v1`
invitation, claim, one-use join, and exact packet retry
- runs Channel turns through the normal canonical Thread lock and
AgentRunner path
- removes Channel-specific runner tokens, run open/close calls, lease
heartbeats, and compatibility behavior
- updates the public Channel delivery contracts, files, provider
effects, tests, and docs
## Why
The SDK must match the new Realtime Gateway boundary: Runtime owns agent
execution, Gateway owns provider effects, and Redis holds one exact
unacknowledged packet.
## Impact
Channel Runtime connections move from the old live-session flow to
`/channels`. This is a hard cut with no protocol fallback or feature
flag.
## Companion PR
- Intelligence platform:
https://github.com/CopilotKit/Intelligence/pull/663
## Validation
- `NX_TUI=false pnpm nx run-many -t test,check-types,build,publint,attw
--projects=@copilotkit/channels-intelligence,@copilotkit/runtime
--skip-nx-cache`
- commit-hook package checks, including 1,809 Runtime tests
- shell docs lint, typecheck, and production build
- `git diff --check`
## Live boundary proof
- connected the actual SDK delivery transport to the actual Realtime
Gateway release
- drove a prepared delivery from App API through Redis wake, SDK
handling, Gateway provider execution, and PostgreSQL terminal storage
- verified one provider call and the exact
`complete:complete:provider_delivery_complete` result
- repeated with two Gateway nodes sharing Redis/PostgreSQL; both
received the wake, exactly one Runtime handler ran, and exactly one
provider call occurred
Not run: a real external Slack or Teams provider smoke. The live proof
used a local fake Slack endpoint so the full internal boundary and exact
provider-call count stayed observable.
Rethrow all session.effect failures from postFile; soft-return only
upload/config errors. Prevents claimAndHandle false complete terminals
for join/claim/TypeError protocol failures that the rethrow allowlist
previously missed.
CR r5 bucket (a): always stop native Slack streams on failure (thread
finish + NativeMessageStream queue drain), advance append/replace text
only after apply, rethrow permanent postFile gateway errors, exclude
stream.stop from provider-output tracking, classify errors by message,
validate prepared turn fields per kind, and stop unit tests from hitting
live lock cleanup HTTP.
Keep provider stream cleanup sendable after non-terminal failures; only
seal effects for terminal provider statuses; join then leave on reconnect;
leave after invalid prepared join reply; advance stream text only after ack.
Allow a failed/uncertain terminal after effect or complete-terminal push
failures; seal only after a successful terminal apply. Leave Phoenix child
channels on failed join, re-arm delivery handlers on restart, replay
onStateChange health, skip empty Teams stream deltas, and align docs/tests.
Close packet path after permanent push/ack failures so a later effect
cannot mint a new effectId on the same seq. Refresh owner generation on
join_token reconnect, add reconnect backoff, require claimed on claim
assert, reject unknown turn kinds, skip empty Slack stream deltas, and
surface missing file-client attachments instead of dropping them.
Always release the product thread lock after a Channel canonical run.
Align connectTimeoutMs docs, projectId validation, ops error guidance,
and test fixtures with the delivery ID contract.
Note: local lefthook skipped (no node_modules in this worktree); CI will
validate. CR findings addressed from PR #6249 review.
## Problem
`workspace:^` dependencies in a canary pack to caret ranges. A consumer
can therefore resolve a different semver-higher canary (for example
`canary.perfall1` over `canary.1785375021`) and combine incompatible
artifacts from separate runs.
## Why
The `scope=all` prerelease flow bumps every package together, but `pnpm
pack` preserves the range operator. Sharing a canary identifier
therefore did not guarantee that npm would install that same package
set.
## Fix
Immediately before the existing prerelease pack/publish loop, replace
internal dependency ranges with the exact versions of packages included
in that canary run. Dependencies outside the selected publish set remain
unchanged.
This is confined to `prerelease.ts`, which `publish-release.yml` invokes
only for prerelease mode. Stable publishing continues through
`publish-release.ts` unchanged.
Validation:
- 153 release tests passed.
- Targeted TypeScript, oxlint, and oxfmt checks passed.
Addresses maxkorp's third pass, including a regression in the maxMessages cap
added in the previous commit.
1. `truncate()` is now terminal and idempotent. It set `curPosted =
buffer.length` and appended the marker but left `curTs` set and recorded no
terminal state, so text still arriving grew the buffer, the next flush saw
undelivered text on an already-full message, and re-entered — one marker per
flush for the rest of the turn. Reproduced at production defaults on a 400k
reply: markerCount=36, last message 12,908 bytes against an 11,000 budget.
Against a transport enforcing a real 12k ceiling, 18 markers land and then 28
appends are rejected, spending exactly the error the incident was about. This
was the spam the cap exists to prevent, relocated inside one message.
A `truncated` flag short-circuits `appendPending` and both `finish()` paths,
and the marker is now charged to `curMessageBytes` — it was the one append
path that neither checked nor updated the budget.
2. Table boundaries prefer a row break. Inside a table the space fallback left
the continuation starting mid-row, so the re-emitted header was followed by a
malformed row. Fixes single-append boundaries completely; under incremental
cadence a fill append can still land mid-row before the room runs out, which
this does not address (see the PR thread).
3. `renderContextCloser`'s fence check now mirrors `hasFenceCodeContent`
exactly: non-whitespace after the language line, not merely a newline, so
```py\n no longer gets a closer. The comment claimed the mirror; now it holds.
Also trimmed the rationale at the boundary-closer catch site. It repeated the
"first append a wrong headroom assumption rejects, two things had to be right"
argument, which the reviewer measured and retracted: under byte counting the
fill append dies first and rollOver is never reached, so the two modes are
sequential rather than compounding, and the trigger window is only a few bytes
wide. The guard stays — an unguarded append that wedges rollover, directly above
a guarded one that doesn't, is worth three lines on the asymmetry alone — but the
codebase should not preserve reasoning that has been disproven.
## What changes
This replaces the managed Channels adapter and transport stack with live
sessions over Realtime Gateway.
- Channels use the standard AgentRunner and canonical AG-UI history for
each turn.
- One admitted delivery runs one prompt; multiple agent calls run in
order; concurrent calls fail with a bounded protocol error.
- Slack and Teams reuse their native renderers to emit destination-free
provider effects.
- Files, rich controls, interaction handlers, and provider cursors cross
the live-session protocol with bounded payloads.
- The old claim mapping, HTTP fallback, render batches, listener
election, in-memory transport, and legacy adapter code are removed.
- The public `@copilotkit/channels` umbrella remains limited to public
provider adapters; the managed launcher stays in
`@copilotkit/channels-intelligence`.
Companion service PR and kind proof:
https://github.com/CopilotKit/Intelligence/pull/638
## Why
Managed delivery must use the same AgentRunner path as other
Intelligence runs. SDK code emits provider-neutral effects; the trusted
Gateway owns credentials, destinations, admission, retries, and terminal
outcomes.
## Validation
- `pnpm nx run-many -t build,check-types,test -p
@copilotkit/channels-core,@copilotkit/channels-intelligence,@copilotkit/channels-slack,@copilotkit/channels-teams,@copilotkit/channels,@copilotkit/runtime`
— 31 tasks passed
- `pnpm nx test @copilotkit/channels-teams` — 89 passed
- `pnpm vitest run scripts/release/lib/channels-umbrella.test.ts` — 8
passed
- `pnpm verify:channels-umbrella` — packed snapshot, dependency
resolution, and TSX consumer passed
- affected package pre-commit tests, publint, and API type checks passed
- changed-file Prettier and `git diff --check` passed
## Known unrelated check
`pnpm nx build demo` now compiles past the prior Channels telemetry
dependency leak, then fails on the existing AG-UI 0.0.51 versus 0.0.57
private `_debug` type mismatch.
## Why
A `scope=all` canary took ~20 minutes. Profiling run
[30473483745](https://github.com/CopilotKit/CopilotKit/actions/runs/30473483745)
step-by-step showed almost none of that was real work.
The dominant cost: `npx --yes npm@11.15.0 publish` ran **once per
package**, and npx re-resolves the spec against the registry on every
invocation — **~16s of each package's ~21s**, leaving ~5s of actual pack
+ upload. That is why `all` hurt worst: 26 packages paid ~7 minutes of
pure npx overhead. The same loop exists in the stable path, so a
16-package monorepo release paid over 4 minutes of it too.
## What changed
1. **`scripts/release/lib/npm-cli.ts` (new)** — installs the pinned npm
**once** into a throwaway prefix and memoizes it; `prerelease.ts` and
`publish-release.ts` both reuse the binary. The version pin now has a
single source of truth (it was duplicated as a literal in both scripts).
A throwaway prefix rather than `npm i -g` keeps it hermetic, so running
these scripts locally doesn't rewrite a developer's global npm.
2. **Shallow checkout for canaries only** — prereleases push no tag, cut
no GH Release, read no commit range, and build with `nx run-many` (not
`affected`), so there is no merge base to resolve. Stable **keeps
`fetch-depth: 0`**, because its publish job resolves and pushes tags out
of that checkout's `.git` (shipped via the workspace artifact).
3. **Artifact slimming** — `compression-level: 0`, since the payload was
already gzipped by "Pack workspace" and was then re-deflated into the
artifact zip for ~no size win.
4. **`notify` job skipped for canaries** — the builder already returns
`should_post=false` for `mode=prerelease`, success *and* failure
(`build-release-notification.ts:196`), and the self-watchdog's Slack
post is already gated off since `npm_intended` is false for a
prerelease. So the job could only ever checkout + install + compute
"post nothing", on the critical path because canary.yml waits for the
whole run. `python_publish=true` still reaches it.
5. **pnpm store cache**, restore-only on canary refs (see the note
below).
6. Orchestrator polled for the delegated run *after* a 6s sleep; now
polls first.
## Results
Real `scope=all` publishes of all 26 packages across all three scopes:
**~20 min -> ~4m40s**.
| Step | Before | After |
|---|---|---|
| **Publish loop** (26 pkgs) | ~546s (26 x 21s serial) | **38s** (4 at a
time) |
| publish: Install deps | 41s | **27s** (root + packages only) |
| build: Checkout | 48s | **6s** |
| build: Pack workspace | 33s | **12s** |
| build: Upload workspace | 27s | **2s** |
| publish: Download / Unpack | 31s / 6s | **1s / 2s** |
| notify job | 76s | **skipped** |
| **build job total** | 261s | **164s** |
| **publish job total** | 107s + publish | **81s incl. publish** |
Measured runs:
[30504035212](https://github.com/CopilotKit/CopilotKit/actions/runs/30504035212)
(4m44s) and
[30504493051](https://github.com/CopilotKit/CopilotKit/actions/runs/30504493051)
(4m52s).
Honest note on those two totals: the second is 8s *slower* end-to-end
even though it added the narrow install, because its build job drifted
+17s on unrelated runner variance (Setup pnpm 4s->8s, build 91s->95s).
The narrow install did exactly what it should at step level —
publish-job install 41s -> 27s. Treat ~4m40s +/- 15s as the real figure;
per-step numbers are the controlled measurement, end-to-end totals carry
runner noise.
The publish log now shows one `Installing npm@11.15.0`, then `Publishing
26 package(s), 4 at a time...`, then 26 `Published` lines in 34s.
The build job (164s: 95s `nx run-many` + 46s full install) is now the
dominant remaining cost and is deliberately untouched — see Further
options below.
## On the pnpm cache — deliberately restore-only
Worth calling out because my first attempt here was wrong.
`publish-release.yml` was the only workflow in the repo with no pnpm
store cache, and one observed canary spent **9m08s** installing on
tarballs arriving at 2–49 KiB/s. But adding a plain read/write cache
turned out to be a *net negative* for canaries:
canary.yml mirrors the source branch to a unique
`canary/<slug>-<run_id>-<attempt>` ref, and Actions scopes cache entries
to the writing branch (plus the default branch). A cache saved from a
canary ref is **unreachable by every later canary** — measured at 874
MiB of orphan per run, competing for the repo's shared 10 GB budget and
evicting entries other workflows depend on. It also cost more than it
bought: 20s + 26s of post-job saves against a 25s faster install.
So canaries **restore read-only** (free win whenever main has an entry,
zero cost and zero pollution when it doesn't), and stable releases —
which run on main, where a write is durable and reused — do the writing.
Verified in CI that the read-only arm runs, the read/write arm is
skipped, and no `Post Cache pnpm store` save step is planned at all.
Honest framing on the mean: the cache is roughly break-even on typical
time (18s restore + 21s install vs 46s cold). Its value is bounding the
registry-throttle tail, not shaving the average.
## Testing
- **Real `scope=all` canary publish**
([30503117837](https://github.com/CopilotKit/CopilotKit/actions/runs/30503117837),
success): all 26 packages published across monorepo + angular + channels
under one shared canary id, confirming cross-scope `workspace:` deps
still resolve in-run. Verified live on the registry:
`@copilotkit/react-core@1.64.2-canary.perfall1`,
`@copilotkit/channels@0.4.1-canary.perfall1`,
`@copilotkit/angular@0.3.1-canary.perfall1`.
- **Dry-run canary**
([30502850575](https://github.com/CopilotKit/CopilotKit/actions/runs/30502850575),
success): confirmed the shallow checkout, artifact slimming, and that
the `notify` job is genuinely skipped.
- **Cache-fix validation run**: step conclusions confirm `Restore pnpm
store (prerelease — read-only)` → success, `Cache pnpm store (stable —
read/write)` → skipped, and no post-job save step exists.
- `143` release-script tests pass, including 6 new ones for the helper
(install-once, memoization, the OIDC `>= 11.5.1` version floor, and that
a failed install throws rather than silently falling back to the too-old
ambient npm).
- `actionlint` clean on all four release workflows; `shellcheck
--severity=warning` clean on `scripts/release/**/*.sh`;
`verify-release-scope-dropdowns.sh` clean; `oxlint` 0 warnings/errors;
`oxfmt` applied.
- Live probe of the helper: installs npm 11.15.0 in 3.2s, second call
0ms, same path, resolved binary reports `11.15.0`.
- `prerelease.ts --dry-run` still enumerates all 9 channels packages.
## Notes
- `@copilotkit/*@canary` now points at `…-canary.perfall1` from the
verification publish. Content-wise that is main's package code (this
diff only touches release scripts/workflows). Happy to republish from
main if you'd prefer the tag moved.
- No npm OIDC surface changed: the workflow filename, the `npm`/`pypi`
environments, and the trusted-publisher bindings are all untouched.
`npm` stays >= 11.5.1 (pinned at 11.15.0, now enforced by a test).
## Further options (want your call before I touch these)
The build job is now the long pole. Each remaining lever trades
something I don't think is mine to trade:
- **Bigger Depot runner for the build job** (~95s of CPU-bound `nx
run-many`). Depot is already used elsewhere in this repo, but it
requires `id-token: write` — on the one job deliberately kept
credential-free, and the job that runs arbitrary package build scripts.
npm's trusted-publisher record also demands the `npm` environment claim
which the build job lacks, so a token minted there almost certainly
cannot publish. Still your call, not mine.
- **Merge build + publish** (~60s: second install, artifact round-trip,
job spin-up). Directly erodes the build/publish credential boundary this
file's comments defend.
- **Scope-aware build.** An `angular` canary builds all ~70 projects to
publish 1 package. Zero help for `scope=all`, large win for single-scope
canaries.
- **Narrow the build job's install too.** `nx run-many -t build
--projects=packages/**` shouldn't need examples/ or showcase/ deps.
Lower confidence than the publish-job version, since nx builds a full
project graph.
Nx caching is a dead end here: Nx Cloud stays off per team decision, and
a local `.nx` cache hits the same ephemeral-canary-ref scoping problem
documented above.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Addresses maxkorp's second pass. All three reproduced locally first.
1. The boundary closer append was unguarded, and its failure disabled rollover
entirely (the worst of the set). It threw out of `rollOver` with `curTs` still
set, so the next flush recomputed a full message, re-entered `rollOver`, and
failed on the same closer forever. Probe: a transport rejecting closer-only
deltas delivered 11,000 of 37,410 chars across ONE message with zero
rollovers. The control confirms the asymmetry — the same input with
`stopStream` failing instead delivers everything, because that call was
already guarded.
This compounds with the unit question: the closer is appended exactly when the
message has just filled, so it is the FIRST append a wrong headroom
assumption rejects — and its rejection switched the preventive fix off before
it could fire. Now guarded with the reasoning already written for
`stopStream`: degraded markdown on one message beats losing the remainder.
2. Implausibly long fenced-code "languages" are clamped. `detectOpenContext`
reports everything up to the first newline after ``` as the language, so a
minified blob became a multi-kilobyte preamble re-injected into every
continuation: 5,004 chars of duplicated preamble atop six consecutive
messages, 54% useful, and a final message 1% useful. Past
MAX_FENCE_LANG_CHARS we re-open with a bare fence, which still preserves code
formatting where dropping the opener would not. Re-injected preamble drops
from 5,004 chars to 4.
3. Continuation count is bounded. A 500k-char reply became 46 Slack messages,
uncapped. Not a regression (the legacy chunker is uncapped too), but this PR
exists because a runaway became repeated copies in a channel, and silently
turning one over-long reply into dozens of real messages is that failure in a
different hat. DEFAULT_MAX_MESSAGES with a visible truncation marker, so the
cut is stated rather than silent.
Also removed the write-only `finished` field the review spotted.
Deferred, tracked in the PR thread: the emphasis closer's flanking position
(needs a real-workspace check, since Slack is not strictly CommonMark) and
`flushChunk`'s permanent chunk-disable after a transient continuation failure.
The publish job reinstalled all 4608 projects' dependencies (~41s) to run
two things: `pnpm tsx` and `pnpm pack`.
A root-only install is NOT sufficient — verified locally that pnpm pack
then dies with ERR_PNPM_CANNOT_RESOLVE_WORKSPACE_PROTOCOL, because pack
resolves each package's `workspace:` deps into real version ranges and
needs its workspace siblings installed to do it. That resolution is what
makes cross-scope canary deps pin the same-run version, so it is
load-bearing.
The true minimum is root + packages/**: measured 17s vs 46s locally, and
the packed tarball still carries resolved ranges (^0.4.0, ^1.64.1) with no
leftover workspace: refs. Everything trimmed is examples/ and showcase/ app
dependencies that no publish step touches.
Safe because no publishable package declares prepack or prepare, so pnpm
pack runs no lifecycle scripts; voice's prepublishOnly fires for neither
pnpm pack nor `npm publish <tarball>`.
Stable keeps the full install — its dependency surface is wider
(lib/notion.js, the umbrella verifier's root script) and it is the
highest-stakes path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
After the npx fix each publish is ~4.7s and almost entirely a registry
round-trip, so a 26-package scope=all canary still spent ~125s waiting
serially. Publish 4 at a time (CANARY_PUBLISH_CONCURRENCY=1 restores
serial for debugging).
This weakens no ordering invariant. prerelease.ts's own header already
documents that the cross-scope graph has cycles (runtime ->
channels-intelligence, channels-core -> core), so no serial order avoided
publishing a package before the same-run version it pins.
Per-package output is captured and replayed as one block rather than
inherited, since a pool would otherwise interleave several npm publishes
line-by-line — and that log is the only forensic record when a canary
half-publishes. Every package is attempted even if others fail, so one
report names all of them; main() now exits non-zero on failure rather
than letting an unhandled rejection pass the step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
canary.yml mirrors the source branch to a unique
canary/<slug>-<run_id>-<attempt> ref, and Actions scopes cache entries to
the writing branch (plus the default branch). A cache saved from a canary
ref is therefore unreachable by every later canary — measured at 874 MiB
of orphan per run, competing for the repo's shared 10 GB budget and
evicting entries other workflows depend on.
It also cost more than it bought: 20s + 26s of post-job saves against a
25s faster install.
Canaries now restore read-only (free win when main has an entry, zero
cost and zero pollution when it doesn't); stable releases, which run on
main where a write is durable and reused, do the writing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Addresses maxkorp's review. All three findings reproduced locally before fixing.
1. Budget is now UTF-8 bytes, not chars. The "safe whichever way Slack counts"
claim covered char-cap vs per-call-cap but not chars vs BYTES, and the
incident datapoint was English, where the two are 1:1. Under a byte-denominated
cap a char budget never fires: 漢字 x 10,000 delivered 4,000 of 20,000 chars
with ZERO rollovers and 33 swallowed msg_too_long — the original bug, intact,
for every non-Latin-script user. Cyrillic/Greek/Hebrew/Arabic break above ~6k
chars, CJK above ~4k. UTF-8 byte length is always >= char count, so an 11k-byte
budget is under a 12k ceiling in either unit. Costs extra messages for
non-Latin replies if the ceiling turns out to be chars; correct side to err on.
`spanWithinBudget` walks whole code points, which subsumes the previous
`avoidSurrogateSplit` guard for every boundary rather than patching each one.
2. `finish()` no longer drops the tail. It enqueues exactly one flush, so a
continuation `startStream` that throws there had no later flush to retry it:
probed at 11,000 of 25,000 chars delivered, no fallback, first message
finalized cleanly, so the turn reported success and the platform never
replayed it — a fresh silent-truncation path in a file whose purpose is to
stop silently truncating. Now drains with bounded retries, then hands the
remainder to the legacy transport seeded with ONLY the undelivered tail (not
the whole buffer, which is what makes failing over mid-reply duplicate text).
`onStartFailure` is deliberately not fired: a transient continuation failure
should not mark the whole workspace legacy.
3. Tables are re-opened across a boundary. `detectOpenContext` models fences,
inline code, and emphasis but not tables, so continuations started mid-row with
no delimiter above them and Slack rendered literal pipes — a regression in the
native-table rendering that motivates this transport, and long generated tables
are a common way to exceed the cap. `tableHeaderToReopen` re-emits the header
and delimiter rows.
Also from the review: the fence closer now mirrors `hasFenceCodeContent` so a
boundary just after ```lang no longer emits an empty code block; the
fits-in-one-message guard uses 10,900 rather than 5,000 so it actually pins the
boundary; and the config-only per-call branch has a test.
Deferred: the emphasis closer can land after whitespace and so not right-flank
under CommonMark. Slack's renderer is not strictly CommonMark, so this wants
confirming against a real workspace in the same manual pass as the cap unit.
The canary flow took ~11.5 min steady-state (and 20 min in an observed
run). Measured from run 30473499191, the time went to five avoidable
places rather than to real work.
1. `npx --yes npm@11.15.0 publish` ran per package, and npx re-resolves
the spec against the registry on EVERY invocation: ~16s of each
package's ~21s. A 9-package channels canary paid ~2.4 min of pure npx
overhead; a 16-package monorepo release paid over 4 min. Hoist the
pinned npm into lib/npm-cli.ts, install it once into a throwaway
prefix, and reuse the binary.
2. publish-release.yml was the only workflow in the repo with no pnpm
store cache, so all three jobs installed 4608 packages cold every
time. Usually ~45s each, but registry-bandwidth bound and heavy
tailed: the observed run spent 9m08s here on tarballs arriving at
2-49 KiB/s. Add the same node-version-keyed cache the rest of CI uses.
3. The notify job ran for canaries only to compute "post nothing" — the
builder already returns should_post=false for mode=prerelease and the
self-watchdog is already gated off. ~85s of dead work on the critical
path, since canary.yml waits for the whole run. Skip the job, keeping
it reachable for a python_publish dispatch.
4. The build job fetched full history for canaries, which need none (no
tag, no GH Release, no release-note commit range, and `nx run-many`
resolves no merge base). That rode along in the 837 MiB workspace
artifact too. Shallow-fetch prereleases; stable keeps depth 0 because
its publish job pushes tags out of that artifact's .git.
5. Two smaller ones: the artifact was gzipped and then re-deflated into
the artifact zip (compression-level: 0), and the orchestrator's
run-discovery loop slept 6s before its first poll.
Verified: 143 release-script tests pass (6 new for the npm-cli helper),
actionlint + shellcheck + the scope-dropdown guard are clean, the
prerelease dry-run path still enumerates all 9 channels packages, and a
live probe confirms the helper installs npm 11.15.0 once (3.2s) and
memoizes thereafter (0ms).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects found by adversarially probing the rollover added in the previous
commit. Both were reachable at the default 11k cap.
1. Unbounded message creation. The continuation re-opener is synthetic text
charged against the new message's budget, so a context whose opener is itself
larger than the cap filled each fresh message with nothing but its own
preamble and rolled over again — forever, never advancing `curPosted`.
`detectOpenContext` reports everything up to the first newline after ``` as
the fence "language", so an agent emitting a >11k whitespace-free blob
(minified JSON, base64, a long log line) straight after a fence triggers it.
Probed at the default cap: `posted` frozen at 11,000 while `startStream` was
called 41+ times, bounded only by the probe's own budget. In production that
is an unbounded stream of Slack messages — worse than the truncation it
replaced. The opener is now dropped when it cannot leave
MIN_MESSAGE_PROGRESS_CHARS of room, so every message carries text and the
loop always terminates.
2. Surrogate pairs split across boundaries. Whitespace-free text reaches
`breakPoint`'s hard-cut fallback routinely (CJK, emoji runs, base64), and
cutting at an arbitrary UTF-16 offset left a lone high surrogate ending one
message and an orphaned low surrogate starting the next — a broken glyph per
boundary. Concatenating the messages still compares equal, which is why the
existing preservation assertions passed straight through it. All boundaries
now step back off a pair via `avoidSurrogateSplit`, including the per-call
12k slices, which had the same latent exposure before rollover ever existed.
Both are pinned by tests that fail against the previous commit: a runaway guard
that turns a rollover loop into a failed test rather than a hung one, and a
per-message surrogate-edge assertion (the join-and-compare check cannot see it).
Slack enforces a cumulative text cap per streamed message, not just the
documented 12k-per-`markdown_text`-call cap. `NativeMessageStream` only honored
the per-call cap and kept appending to a single message, so once a reply passed
the cumulative ceiling every further `chat.appendStream` was rejected with
`msg_too_long` — permanently, for the rest of the run.
Observed in dev: a ~23k-char reply streamed 11,607 chars into one message and
then failed every subsequent append. The remaining text was silently dropped
(the append error is swallowed by design), the turn never reached finalize, and
the egress operation replayed from its uncheckpointed cursor — posting a fresh
truncated copy per attempt until it hit max_attempts.
`flushText` now rolls over: when a message fills and text remains, the boundary
is frozen on a line (else word) break, any open markdown construct is closed,
the message is finalized, and the reply continues in a new message that
re-opens that construct. Both append loops share one rollover-aware path, and a
continuation `startStream` failure no longer falls back to the legacy transport
— that sink replays the whole buffer, which would duplicate everything already
streamed.
The soft per-message limit is 11k, below the smallest total Slack was observed
to accept, leaving headroom for the closers a boundary appends.
The test transport did not model the cumulative cap at all, which is why this
survived the suite; it does now, and the case that asserted "keeps a long reply
in ONE message" was asserting the bug.
Austin Merrick