mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
Ben Taylor
a76c2bcce2
## What this is now Opt-in instrumentation for the channel egress path, plus a replay-stability guard. **Measurement and safety only — no behaviour change to how frames are sent.** It started as a fix too. That fix is reverted here; see below. ## The problem it measures A streaming reply pushes one render frame per AG-UI token and awaits each durable acceptance receipt before sending the next, so reply latency scales with token count. Nothing recorded that, so the cost was invisible and got attributed to the Connector Outbox's 400ms poll instead. ```bash COPILOTKIT_CHANNELS_RENDER_METRICS=summary # one summary per turn COPILOTKIT_CHANNELS_RENDER_METRICS=frames # also one line per frame ``` Unset means off and no collector is allocated. Output goes to the transport `log` callback when configured, else the console. ## What it found, measured against production Hosted api + realtime gateway, local runtime, fixture agent emitting exactly 120 deltas 15ms apart: ``` pushMsMean: 478.43 pushMsP50: 463 pushMsP95: 589 pushBlockedPct: 99.67 ``` **478ms per round trip against an ~85ms network RTT**, so roughly 390ms of every push is server-side work — the `lookupClerkOrgId` query, the off-cluster ops-api entitlement call, the read-model build, and the Redis publish. Intelligence#627 removes those for `text_delta` frames. At 478ms a push, a 120-token reply costs ~123 round trips ≈ **59s**. That matches the original "sometimes many seconds" report. ## Why the fix was reverted Coalescing adjacent text deltas worked — 123 frames to 4, a production turn from ~59s to 3.4s — and broke replay. A turn's id is a pure function of its delivery id (`turn_<deliveryId>`, enforced by app-api) and the adapter resets the per-turn seq counter on every dispatch, so a redelivery replays into the **same seq space**. app-api treats that space as an idempotency key: same payload at a seq is `duplicate_accepted`, a different payload is `CHANNEL_RENDER_FRAME_CONFLICT`, which nacks the delivery. Merge boundaries depend on wall-clock arrival, so a replay cuts them differently: | | attempt 1 (slow, merged) | attempt 2 (redelivery, faster) | |---|---|---| | seq 1–3 | *merged away, never pushed* | `"a"`, `"b"`, `"c"` → INSERT ✓ | | seq 4 | `text_delta "abcd"` | `text_delta "d"` → **CONFLICT** ✗ | Before coalescing, deltas were 1:1 with AG-UI events and a replay was byte-identical. I verified both directions: the new test passes on the pre-coalescing adapter and fails with coalescing. The regression was mine. `render-frame-replay.test.ts` stays as a standing guard — it passes with frames 1:1 and fails for any change that makes a frame's payload at a given seq depend on timing. ## The replacement Batch the **transport**, don't merge frame **content**: send the buffered frames as an array in one request. Same collapse in round trips, `seq k` still carries `delta k` so replay stays byte-identical, and seqs stay dense so the outbox can later require contiguity (OSS-653). That needs an app-api batch accept and a gateway envelope, so it ships after launch with its own deploy ordering (app-api → gateway → SDK) rather than two days before. ## Also found, filed separately OSS-653 — the outbox advances an in-memory cursor to the highest seq it read (`WHERE seq > cursor ORDER BY seq ASC`), so a frame accepted out of order is skipped and never rendered. Pre-existing and independent of this PR. Not durable corruption: thread history is rebuilt from all acceptance rows ordered by seq, independent of that cursor, so threads read back complete. ## Validation ``` nx run-many -t test check-types build --projects=@copilotkit/channels-intelligence --skip-nx-cache ``` 216 tests pass (12 files), check-types and build clean, `oxlint` clean on all changed files. refs OSS-648