Add coverage (slot-6 F1) that `writeUnrecoverable` NAMES the proven wedge signal
in the Slack/alarm message when the give-up snapshot measured the cgroup PID
ceiling (cgroupPidsCurrent >= 0) — pids.current/pids.max + threads — and OMITS
the clause when the gauges are the off-Linux `-1` sentinel.
Documentation the diff invalidated, swept to match reality:
- browser-pool.ts JSDoc/comments + browser-pool.test.ts comments claimed
hard-recovery "purges the stale /tmp/playwright_* profile dirs". The code is
purge-free — hardRecover is a PACED cold relaunch that gives the
thread-exhausted kernel time to relax. Rewrite to match: paced cold relaunch,
NO /tmp purge (the wedge is the cgroup pids ceiling, mitigated demand-side;
/tmp is a proven non-cause).
- probe-invoker.ts stagger comment said "8 simultaneous workers", "~400 thread
spawns", "10-minute D6 budget" → updated to the current 5 / 20-min reality.
- e2e-deep.yml: the global cap note (40, "D6 peak 32 + D5 peak 8") → 24, with
D6 peak now 5×4=20; 20+8=28 > 24 so a d6+d5 overlap serializes under the
global cap (intended demand-side back-pressure).
- d6-all-pills-e2e.yml: the worked example (max_concurrency 8, ceil(18/8)=3
rounds) → 5 / ceil(18/5)=4 rounds, cross-referencing the overlap block that
justifies the concurrency itself.
Robustness/bounding nits a 7-agent CR flagged:
- browser-pool: the hot acquire/release path emitted a SECOND `hot-gauges` info
log duplicating inUse/available already on the acquire/release line. Fold the
two cheap cgroup-pid fields INTO the existing acquire/release log line (one
line, not two); keep the cheap two-file-read sampling (now `readHotGauges`).
- probe-invoker: `logTickGauges` double-encoded the sample as both a formatted
`gauges:` string AND the structured `...g` spread. Drop the redundant string;
keep the queryable spread.
- orchestrator: the best-effort Slack ping in `writeUnrecoverable` had no
timeout, so a hung webhook could stall the serialized health-signal write
chain. Add an AbortController timeout on the fetch.
Normalize the migration schema comments to reality: the heartbeat is 45s (not
the stale "~30-60s"), retention prunes by stable row id (robust to
same-millisecond observed_at ties) rather than a bare timestamp cutoff, and
document the null-vs-unavailable convention — the nullable number fields store
`null` for the `-1` "unavailable" sentinel so post-wedge queries cleanly
separate a measured reading from an unavailable one.
The snapshot writer had several robustness gaps a 7-agent CR flagged:
- PRUNE TIES: the ring prune deleted `observed_at < cutoff` (strict), but
observed_at is ms-resolution and several snapshots can fire in the same
millisecond (degraded+heartbeat+crash). If surplus rows tied the boundary
timestamp, strict `<` deleted nothing → knownOverCap stayed true → every
insert re-ran a full no-op sweep → unbounded growth + per-insert cost. Now
the prune deletes the oldest `surplus` rows by stable row id, which converges
regardless of ties, with a deleted==0 safety valve that can't spin.
- RATE-LIMIT CATCH-UP: a local row-count estimate (re-anchored to the real
total every sweep) forces a catch-up prune the moment we cross the cap, so the
rate-limit only suppresses sweeps while UNDER cap and never strands us over.
- FAILURE ESCALATION: swallow+warn alone hid a systematic PB rejection (unrun
migration / outage / schema drift). Now consecutive failures are counted, the
per-failure warn is throttled, and after a threshold the writer escalates ONCE
to logger.error (latched, reset on first success).
- BOUNDING: a best-effort in-flight cap drops snapshots past N concurrent
unawaited creates (launch-crash-loop burst), and a Promise.race write timeout
frees the in-flight slot if a PB write hangs.
- NULL SENTINELS: the `-1` "unavailable" gauge sentinel is written as `null`
(PB number fields are nullable) so post-wedge queries separate measured-vs-
unavailable.
- DOCS: cadence prose + JSDoc updated to the real 45s heartbeat, and a
single-writer note that the fleet case needs an elected pruner.
Red-green tests added: identical-boundary-timestamp prune converges to maxRows;
rate-limit keeps prune off the steady-state path but knownOverCap catches up
over cap; N consecutive failures escalate to error once and reset on success;
`-1` gauge written as null.
browser-pool.ts imported `./resource-gauges` without a `.js` extension while
sibling files use `.js`. The harness builds with tsc (module: esnext) and runs
the emitted ESM directly (`node dist/orchestrator.js`); moduleResolution:
bundler lets it typecheck but tsc emits the specifier verbatim and Node ESM
does NO extension resolution, so the dist would fail at boot with
ERR_MODULE_NOT_FOUND. Append `.js` to the extensionless relative specifiers in
browser-pool.ts and resource-gauges.test.ts. Verified by building the dist and
boot-importing dist/orchestrator.js (BOOT-IMPORT-OK, no module-not-found).
The live wedge peaked 998/1000 against the cgroup pids.max=1000 ceiling
specifically during a d6+d5 overlap — the simultaneous chromium renderers of
both probes summed past the ceiling. Fewer concurrent d6 feature-workers cut
the simultaneous-renderer thread peak, leaving headroom for a co-firing d5 (or
a recovery relaunch) without crossing 1000. Complementary to the
MAX_CONTEXTS 40->24 lever already landed. Comment updated to the pids-ceiling
reasoning; 5 x 4 = 20 concurrent contexts stays under the 24-context cap.
Previously full gauges were logged on only 3 sites (launch, self-heal-launch-
failed, probe-tick). Extend forensic gauge coverage across the whole pool
lifecycle so a recurring wedge has copious history to look back at.
- Full forensic snapshot (sample + log + durable onSnapshot hook) on every
meaningful transition: init, recycle (crash/hygiene), crash/disconnect,
set-empty degraded, self-heal-launch-failed, recovered, unrecoverable, and
shutdown.
- Periodic heartbeat (~45s, env BROWSER_POOL_HEARTBEAT_MS) via a
shutdown-signal-gated self-rescheduling loop — NOT a raw setInterval that
leaks past shutdown — for a baseline trend between events.
- Cheap cgroup-PID-only subset (logHotGauges) on the hot acquire/release path:
high-resolution view of the headline pids.current/pids.max signal without
the full-sample /proc-walk + df cost, and no durable write on the hot path.
- onSnapshot is best-effort: a throwing hook is caught + logged, never crashes
the pool.
The browser-pool wedge ends in a container restart that clears in-memory
state, and Railway's stdout window rolls off — so stdout/in-memory gauges
are not retrievable post-wedge. Persist the OS resource gauges to a new
`resource_snapshots` PB collection so the forensic history survives the
restart and the PID/thread-ceiling exhaustion is reconstructable after.
- New PB migration `1779989300_create_resource_snapshots.js` (mirrors the
probe_runs idempotent create + public-read pattern).
- `resource-snapshot-writer.ts`: best-effort writer (swallows + logs every PB
error so a missing migration / PB hiccup never breaks the pool) with
ring-style retention (cap last N rows, default 5000, env-overridable).
- Orchestrator wires the writer to the pool's onSnapshot hook, reusing the
shared pb client.
The Dockerfile FIX#3 comment claimed the in-image `ulimit -u` raised the cgroup
pids.max (target 16384). It cannot: `ulimit -u` only lifts this process's
RLIMIT_NPROC; the cgroup pids.max is set by the container runtime
(--pids-limit), is platform-fixed at 1000 on staging, and is not raisable from
the image. Rewrite the comment to state this accurately and point at the real
demand-side mitigation (fewer concurrent contexts + gauge/alarm). The harmless
`ulimit -u` line itself is retained. Also update the env test to assert the new
MAX_CONTEXTS default of 24.
The wedge was reproduced as cgroup PID/thread-ceiling exhaustion (pthread EAGAIN
-> "has been closed" -> crash-loop with pids.current pegged at pids.max=1000).
Every /tmp candidate was refuted by measurement, so this removes the /tmp
machinery that targeted a non-cause and replaces it with the demand-side fix +
observability:
- Remove the proactive stale-profile sweep (init + recycle), defaultPurgeProfileDirs,
the PurgeProfileDirs type, the staleProfileDirTtlMs / PURGE_MIN_AGE_MS options
and env, and the breaker's /tmp purge — the hard recovery is now just the paced
cold relaunch. Corresponding purge/sweep tests removed.
- KEEP the circuit-breaker failure-counting -> give-up -> onUnrecoverable alarm
(root-cause-agnostic backstop) and the footgun clamp; the breaker tests are
adapted to drive the wedge via the launcher instead of the purge.
- Log the resource gauges on every launchBrowser(), on the self-heal-launch-failed
path, and at probe-tick start/end so a burst approaching pids.max is observable
and an EAGAIN correlates to a measured pids.current.
- Include pids.current/pids.max + thread count in the onUnrecoverable payload and
the orchestrator's terminal alarm message so the alert names the real signal.
- Lower the BROWSER_POOL_MAX_CONTEXTS default 40 -> 24 to cap thread demand under
the platform-fixed ceiling (still env-overridable).
Standalone, unit-tested gauge sampler that reads the cgroup PID controller
(pids.current / pids.max, v2 with v1 fallback) plus the process-tree thread/proc
counts, FDs, RSS, /dev/shm, and /tmp inode/space from /proc + /sys/fs/cgroup.
The cgroup PID counters are the headline signal for the proven browser-pool
wedge (PID/thread-ceiling exhaustion). Degrades gracefully to -1 off-Linux so it
is safe to import anywhere. Consumed by the browser pool and probe invoker in
follow-up commits.
The recurring browser-pool wedge's ROOT CAUSE: every chromium.launch() leaves
a /tmp/playwright_* profile dir + playwright-artifacts-* dir whose Playwright
close-time cleanup is best-effort (swallowed on failure). In a long-lived
harness container under sustained d6 cron load these orphans accumulate
monotonically until /tmp exhausts inodes/space — then every launch throws
"...has been closed" and self-heal relaunches into the same exhausted /tmp
forever (only a restart, which gives a fresh /tmp, clears it).
PREVENTION: sweep the age-gated stale dirs PROACTIVELY on a cadence — once at
init() and on EVERY recycle relaunch (crash + hygiene) — so /tmp never reaches
exhaustion. This is the durable complement to the circuit-breaker ESCAPE: the
sweep prevents the wedge; the breaker escapes it if it ever still happens.
Unifies the purge into ONE canonical helper — defaultPurgeProfileDirs now takes
an age threshold (minAgeMs) as its first arg. The proactive sweep passes the
conservative staleProfileDirTtlMs (5min default, env
BROWSER_POOL_STALE_PROFILE_DIR_TTL_MS); the breaker's hard-recovery keeps its
aggressive PURGE_MIN_AGE_MS (60s). Same hardened (symlink-safe, realpath-checked)
implementation, different age — no duplicate purge logic.
Adds 4 prevention tests (init-sweep, every-recycle-sweep, bounded-accumulation
over 30 cycles, real-FS age gate) and adapts the existing breaker tests to gate
their purge-count assertions on the breaker's age so the shared cadence does not
inflate them.
defaultPurgeProfileDirs (the self-heal hard-recovery profile-dir purge) ran a
blind readdir + rm(recursive, force) against the world-writable OS temp dir:
- A planted playwright_* SYMLINK had its TARGET recursively deleted. Now read
dirents (withFileTypes), SKIP symlinks + non-directories, and verify the
realpath stays under the scan dir before removing.
- A browser launching CONCURRENTLY with the purge had its FRESH profile dir
nuked, re-wedging the very launch being revived. Now only dirs older than
PURGE_MIN_AGE_MS (60s) are eligible (mtime via lstat).
The real purge previously had ZERO coverage (the pool tests inject a no-op).
Parameterize the scan dir so it is unit-testable against a temp fixture, export
it, and add tests: removes old playwright_*/playwright-artifacts-* dirs, leaves
non-matching dirs, SKIPS symlinks (target untouched), leaves too-fresh dirs, and
swallows ENOENT. (Also pulls in the resolvePositive helper used by the breaker
clamp, which lives adjacent to this function.)
Two circuit-breaker robustness fixes:
Latch second-episode silence: this.unrecoverable was cleared ONLY on a
successful launch, so a permanently-wedged container that re-empties after the
first give-up hit `if (this.unrecoverable) return` and went silent on every
subsequent degraded episode — the exact silent-spin the breaker exists to kill.
Remove the instance latch entirely; the loop-local consecutiveHardRecoveries
guard already gives once-per-episode (the loop returns immediately after firing),
and each fresh degraded episode spawns its own self-heal loop that can alarm.
Stale JSDoc updated to match. The onUnrecoverable hook now carries breaker
counters so the wired alarm can report how hard the pool tried.
Disable footgun: a selfHealHardRecoveryThreshold / selfHealMaxHardRecoveries of
0 silently disabled both the hard-recovery escape AND the give-up (guards are
`> 0`), reverting to the infinite silent spin. Clamp resolved breaker thresholds
to >= 1 (resolvePositive) so a config typo can't disable the safety net.
Tests: a second degraded episode ALSO fires onUnrecoverable (red→green: prior
latch silenced it); a 0 threshold stays armed (red→green: prior it spun
forever); plus a browsers:2 multi-browser test locking in the partial-revive
counter-reset / breaker pacing that was previously unverified.
The self-heal circuit-breaker's terminal give-up shipped a mechanism with no
operator signal — the pool's onUnrecoverable hook was never wired, so a
breaker give-up silently stopped spinning without telling anyone a redeploy is
required. Add createBrowserPoolHealthSignals.writeUnrecoverable and wire it to
the pool's onUnrecoverable hook:
- Unconditional health-signal writes: a DISTINCT system:browser-pool-unrecoverable
key (red, terminal, critical, redeployRequired + breaker counters) AND an
escalation of the shared degraded key to severity=critical/terminal, so a
give-up is distinguishable from a transient self-heal-degraded.
- Best-effort, env-guarded Slack ping on SLACK_WEBHOOK_BROWSER_POOL_UNRECOVERABLE
(unset by default per alerting discipline — deploy the code, wire the URL
separately); a failing/absent webhook never blocks the health-signal write.
The pool's onUnrecoverable hook now passes breaker counters so the alarm can
report how hard the pool tried before giving up.
The recurring staging BrowserPool collapse (#5185/#5221/#5225 each chipped at it but it kept recurring): after the long-lived harness container runs ~hours under sustained d6 cron load, chromium enters a launch crash-loop where every chromium.launch() throws 'Target page, context or browser has been closed'. The set empties, startSelfHeal() kicks in, and its loop just relaunches into the SAME wedged state forever (self-heal-launch-failed repeating) without ever escaping — the stale /tmp/playwright_* profile dirs / FD-shm pressure persist across every relaunch. acquire() therefore has no contexts forever and blocks to timeout fleet-wide; only a container restart cleared it (reactive).
Adds a circuit-breaker to the self-heal loop: after selfHealHardRecoveryThreshold (default 4) consecutive launch failures, instead of looping another identical relaunch the pool performs a HARD recovery — purges the stale ${tmpdir()}/playwright_* profile/temp dirs the crash-looping processes left behind, then cold-launches fresh. Any successful launch resets the counter. If selfHealMaxHardRecoveries (default 3) consecutive hard recoveries also fail, it fires a loud browser-pool.pool-unrecoverable alarm (onUnrecoverable hook) and stops the heal loop rather than spinning silently — the operator signal that a redeploy is genuinely required. All tunable via BROWSER_POOL_SELF_HEAL_* env vars; the purge is injectable for tests and defaults to a hermetic no-op when a fake launcher is injected.
## Summary
Adds a Slack notification to **#team-showcase** whenever a showcase
staging→prod promotion **succeeds**, matching our existing routing
convention (releases → #engr, PRs/promote-failures → #oss-alerts). Today
the promote workflow only posts to #oss-alerts on failure; the `success`
state was computed but unused.
## Changes (all in the `notify` job of `showcase_promote.yml`)
- New **success-only** "Post to #team-showcase" step, gated on `state ==
'success'`, mirroring the existing #oss-alerts step exactly (same pinned
`slackapi/slack-github-action@b0fa283 # v2.1.0`, `incoming-webhook`,
`toJSON(format(...))` payload).
- Job-level `SLACK_WEBHOOK_TS: ${{ secrets.SLACK_WEBHOOK_TEAM_SHOWCASE
}}` so the step `if:` can gate on `env.SLACK_WEBHOOK_TS != ''` (step
`if:` can't read `secrets.*` directly — same pattern as the existing
`SLACK_WEBHOOK`).
- Symmetric "Log (no Slack — team-showcase webhook unset)" `::notice::`
fallback, so a successful promote with the webhook unset still leaves a
trace (mirrors the failure path's fallback).
- Failures still route **only** to #oss-alerts — unchanged.
## Activation (required before notifications fire)
The new step **no-ops until** an org secret
`SLACK_WEBHOOK_TEAM_SHOWCASE` exists, so this is safe to merge now. To
activate: create the `#team-showcase` Slack incoming webhook, then set
it as a CopilotKit **org** secret (visibility=all), matching the
convention for `SLACK_WEBHOOK_OSS_ALERTS` / `SLACK_WEBHOOK_ENGR`.
## Test plan
- [x] `actionlint` exit 0
- [x] YAML parses
- [x] Step is mutually exclusive with the webhook-unset log fallback
(one fires per successful run)
- [ ] CI green
- [ ] After the org secret is set: a real promote posts to
#team-showcase
## Out of scope (separate follow-up)
Review surfaced several pre-existing `notify`/`verify-prod` robustness
items not touched here (verify-prod empty-CSV vacuous-green hardening,
`npx tsx` working-dir in the verify jobs, env-vs-secrets
webhook-reference fragility, literal `\n` rendering shared with the
existing #oss-alerts step). To be addressed in a separate
showcase-promote hardening PR.
## Summary
`showcase bin/railway promote docs` has been failing in CI on every run
with `REFUSE: P1 (docs): GHCR manifest HEAD 403 for
ghcr.io/copilotkit/showcase-shell-docs:latest`. Root cause:
`GHCR#bearer_for` sent the GitHub/Actions token **raw** as
`Authorization: Bearer <token>` to the GHCR OCI manifest endpoint, which
rejects it with `403`. GHCR requires the token to first be exchanged for
a short-lived registry bearer via its `/token` endpoint.
Verified live against the real public `showcase-shell-docs` package: raw
bearer → `403`, `/token`-minted bearer → `200`, anonymous → `401`.
## Changes
- `bearer_for` now **always** performs the GHCR `/token` exchange. When
a token is present it authenticates the exchange with Basic auth
(`base64("x-access-token:<token>")`) and uses the minted token as the
manifest bearer; public packages still mint anonymously.
- Fail-loud hardening: a failed exchange (non-2xx with a token present,
or an unparseable `/token` body) now raises `GHCR::Error` instead of
silently swallowing to `nil` and degrading to an anonymous request —
restoring `manifest_exists`'s documented "raises on transport/5xx"
contract.
- Tests: new `test_ghcr_bearer.rb` pins the on-the-wire behavior
(Basic-auth exchange, minted-bearer-not-raw-token, anonymous path,
raise-on-failed-exchange, never-anonymous-after-supplied-token-failure).
Existing GHCR specs updated to model the now-mandatory exchange.
## Test plan
- [x] `showcase/bin/spec` suite green: 117 runs, 410 assertions, 0
failures
- [x] `ruby -c showcase/bin/railway` syntax OK
- [x] Red-green verified on the new fail-loud paths
- [ ] CI green
## Out of scope (separate follow-up)
Review surfaced three pre-existing, unrelated `bin/railway` bugs not
touched here: `rollback`'s `find_previous_deployment` selects from an
unsorted deployment list, and `env-diff`'s `--ignore-env-scoped` flag /
advertised custom-domain comparison are no-ops. To be addressed in a
separate PR.
The A2UI middleware (@ag-ui/a2ui-middleware) forwards injectA2UITool on
forwardedProps; ag-ui-langgraph surfaces it into agent state at
state["ag-ui"]["inject_a2ui_tool"]. The CopilotKit LangGraph middleware
(py + js) now reads that flag and only injects generate_a2ui when it is
truthy (opt-in), drops the runtime's render_a2ui so the model sees one
A2UI tool, and skips if the agent already defines generate_a2ui. The
catalog only binds surfaces; it is no longer the gate.
Reverts the earlier runtime-forward + context-channel approach.
Deps: ag-ui-langgraph>=0.0.38 (py), @ag-ui/langgraph 0.0.37 (sdk-js),
@ag-ui/a2ui-middleware 0.0.6 + @ag-ui/langgraph 0.0.37 (runtime);
.npmrc min-release-age exclude for @ag-ui/a2ui-middleware. Showcase
langgraph pins bumped to copilotkit==0.1.94a3 / sdk-js 1.59.3-alpha.3 /
@ag-ui/langgraph 0.0.37.
declarative-gen-ui moved to the CopilotKitMiddleware auto-A2UI path across
the 3 langgraph integrations. The middleware's inner forced tool is
render_a2ui, so each integration's gen-ui-declarative.json gained 4
render_a2ui fixtures that share match keys with the pre-existing render_a2ui
entries in that integration's render-a2ui.json (the a2ui_fixed demo). 4
pills x 3 integrations = 12. Same-context cross-demo overlap, disambiguated
at runtime by the probe fixtureFile.
langgraph-fastapi bumped ag-ui-langgraph[fastapi] 0.0.35->0.0.37 (auto-A2UI
needs get_a2ui_tools). Both non-exact, so the FAIL count stays 63 but the
FAIL-set hash shifted. Update validatePinsFailHash to the new set.
The declarative-gen-ui demo across the three langgraph integrations now
relies on the middleware to inject and execute generate_a2ui — the agents
collapse to create_agent + CopilotKitMiddleware with no hand-rolled tool.
Adds render_a2ui fixtures for the new tool path and pins the integrations
to the A2UI alpha SDKs (copilotkit 0.1.94a1, @copilotkit/sdk-js 1.59.3-alpha.1).
The auto-injected generate_a2ui tool now resolves the A2UI catalog from
both delivery paths: the CopilotKit runtime proxy (a copilotkit.context
entry) and the AG-UI native endpoint (state["ag-ui"].a2ui_schema). The
registered catalog id is extracted and bound to generated surfaces so
BYOC custom catalogs render their own components instead of the basic
catalog. Covers @copilotkit/sdk-js and the copilotkit Python SDK.
Prebuilt agents get dynamic A2UI with no extra wiring — adding the
middleware is enough. When the frontend registers an A2UI catalog
(surfaced by the runtime into state["ag-ui"].a2ui_schema), the
middleware infers the agent's own model, advertises the generate_a2ui
tool in the model-call hook, and executes it in the tool-call hook.
No catalog → the tool is never advertised.
Covers both @copilotkit/sdk-js and the copilotkit Python SDK. Bumps
the A2UI tool-factory dependency to where get_a2ui_tools ships
(@ag-ui/langgraph 0.0.35, ag-ui-langgraph >=0.0.37).
Mirror the failure-path 'no Slack' observability fallback for the new
#team-showcase success post. When SLACK_WEBHOOK_TEAM_SHOWCASE is unset
(current default), a successful prod promote now emits a ::notice:: log
line instead of a silent green.
bearer_for ended with a blanket `rescue StandardError => nil` plus
`return nil if status >= 400`. Now that bearer_for ALWAYS performs the
/token exchange (even when a token is present), that swallow masked real
failures: a non-2xx /token response, a malformed JSON body, or a transport
error all collapsed to nil, after which manifest_exists issued the manifest
HEAD anonymously (no Authorization). That silently violated manifest_exists's
documented raise-on-transport/5xx contract and conflated "no token supplied"
with "supplied token failed to exchange".
bearer_for now:
- raises GHCR::Error on a >=400 /token response WHEN a token was supplied
(token absent still returns nil — the legitimate anonymous-public fallback);
- raises GHCR::Error on JSON::ParserError for an unparseable 200 body;
- no longer swallows StandardError, so transport exceptions
(Errno::ECONNREFUSED, Net::ReadTimeout, ...) propagate.
Call-site enumeration (bearer_for is called only by manifest_exists and
resolve_digest):
- manifest_exists: documents "Raises GHCR::Error on 5xx or transport failure",
so a raised exchange error is consistent with — and strengthens — its own
contract. Assumption holds.
- resolve_digest: already raises GHCR::Error on its own manifest HEAD >=400 and
does not rescue bearer_for, so a raised exchange error propagates exactly as
its other failures do. Assumption holds.
Neither caller is broken by bearer_for raising; both already propagate
GHCR::Error to their callers.
Tests: add 3 red-green cases in test_ghcr_bearer.rb (token-present 401 raises;
token-present malformed body raises; token-present exchange failure issues NO
anonymous manifest HEAD). The existing manifest_exists/digest fakes were only
green because the old swallow masked the fake's "no fake response" error — they
never modeled the mandatory /token exchange; added a successful /token fake to
each so they exercise the real path. Renumbered the snapshot-ivar-lint
allowlist (+13 lines) to track the bin/railway line drift.
bearer_for returned a raw GitHub/Actions token, which GHCR's OCI manifest
endpoint rejects with HTTP 403. Always perform the /token exchange instead:
authenticate with Basic base64("x-access-token:<token>") when a token is
present, fall back to the anonymous exchange for public packages, and use
the minted token as the manifest-read bearer. This fixes the docs promote
that has deterministically 403'd in CI.
Route the exchange through the injectable @http seam (matching resolve_digest
/ manifest_exists) so it is testable. Renumber the snapshot-ivar lint
allowlist for the +21 line shift.
Add a success-only Slack post to #team-showcase in the showcase_promote
notify job, mirroring the existing failure→#oss-alerts step. Guarded by
env.SLACK_WEBHOOK_TS so it safely no-ops until the SLACK_WEBHOOK_TEAM_SHOWCASE
org secret exists. Failures continue to route only to #oss-alerts.
Render a per-turn persistent intelligence indicator that stays stable
across multi-step turns and settles into a "finished" tag. Splits the
component into IntelligenceIndicator (logic) + IntelligenceIndicatorView
(presentation), wires it into CopilotChatView / CopilotChatMessageView,
adds the slot styles to globals.css, and a Storybook story plus
timer-free logic tests.
Add `useLearningContainers` / `...InCurrentThread`, which emit
`set_learning_containers` annotations (defaulting to the `project`
container) for the active thread, and export the new hooks from the
react-core hooks barrel.
Add the `recordAnnotation` client (posts to the runtime `/annotate`
endpoint; no client-side userId) and the `useLearnFromUserAction` /
`...InCurrentThread` hooks that record `user_action` annotations for the
self-learning loop.
Move enterprise-learning MCP attachment out of the BuiltInAgent-specific
path and the intelligence run handler into a single request-scoped hook:
- `attachIntelligenceEnterpriseLearning` (agent-utils) attaches
`@ag-ui/mcp-middleware` via `configureAgentForRequest`, gated on
`ɵisEnterpriseLearningEnabled()`, resolving the user via `identifyUser`
and the project apiKey.
- Called from `handleRunAgent`; the old `forwardedProps.auth` MCP plumbing
in `intelligence/run.ts` and the BuiltInAgent attach in `agent/index.ts`
are removed.
- Add released `@ag-ui/mcp-middleware@0.0.1` dependency (lockfile +
`@ag-ui/client` override). Drops the obsolete intelligence-mcp-helper test.
Replace the user-actions-specific path with a general thread-event
annotation endpoint:
- Route `PUT /connector/annotate/:clientEventId` via fetch-router /
fetch-handler / core hooks RouteInfo.
- `handleAnnotate` forwards `{ type, data, learningContainer }` to the
Intelligence platform.
- `CopilotKitIntelligence.annotate()` (idempotent PUT) supersedes the
prior `recordUserAction`; `AnnotateParams`/`AnnotateResponse` carry the
event `type` (`user_action`, `set_learning_containers`, ...).
Also renames the `mcpServer` config flag to `enableEnterpriseLearning`
(`ɵisEnterpriseLearningEnabled()`), consumed by the enterprise-learning
attach in the following commit.
## Summary
- Roll Intelligence/Threads support into
`examples/integrations/crewai-crews` using the #5151/#5189 migration
pattern.
- Add the env-gated `CopilotKitIntelligence` runtime path, REST thread
transport, Threads drawer UI, and local Intelligence env docs.
- Add a focused migration contract test covering the crewai-crews wiring
and stale v2 sidebar prop regression.
Linear:
https://linear.app/copilotkit/issue/ENT-734/roll-intelligencethreads-into-the-remaining-11-framework-examples-ent
## Verification
- `pnpm exec vitest run
scripts/__tests__/integration-intelligence-migration.test.ts`
- `pnpm exec oxfmt --check
scripts/__tests__/integration-intelligence-migration.test.ts
'examples/integrations/crewai-crews/src/app/api/copilotkit/[[...slug]]/route.ts'
examples/integrations/crewai-crews/src/app/page.tsx`
- `git diff --check`
- `./node_modules/.bin/tsc --noEmit` in
`examples/integrations/crewai-crews`
- `npm run build` in `examples/integrations/crewai-crews`
- `pnpm install --frozen-lockfile --ignore-scripts`
- pre-commit hook: `pnpm run lint --fix && pnpm run format`, `pnpm run
test`, `pnpm run check:packages`
- `pnpm run build`
## Manual smoke
- Copied local test env from `/Users/mothra/Projects/test-signups4/.env`
into the ignored example `.env`.
- Replaced missing `OPENAI_API_KEY` with a validated local
company/project key.
- Started the local Intelligence stack from `test-signups4` with `docker
compose up -d --wait`.
- Seeded `1_demo-user` / `demo-user` for organization `casa-de-erlang`.
- Ran the CrewAI agent on `:8000` and the UI on `:3010`, with `.env`
exported into both processes.
- Verified `/`, `/api/copilotkit/info`, and
`/api/copilotkit/threads?agentId=starterAgent&limit=20` returned
successfully; runtime info reported `mode: intelligence` and
`licenseStatus: valid`.
- Verified the licensed Threads UI renders in the browser and the drawer
opens with persisted threads.
- Created/selected a thread, sent `What are the proverbs?`, verified the
agent response rendered, refreshed, selected the saved thread, and
verified the user message + assistant response restored.
Note: this `CopilotSidebar` type surface does not expose visible
suggestion chips in `@copilotkit/react-core@1.59.1`, so the send step
was verified by typing the suggested prompt rather than clicking a
suggestion chip.
## Summary
- rolls the north-star Intelligence/Threads shell into
`examples/integrations/strands-python`
- wires the Strands route to use CopilotKit Intelligence when
`COPILOTKIT_LICENSE_TOKEN` is present, with in-memory fallback when
unlicensed
- documents the Intelligence env vars and removes the temporary
`strands-python` parity allowances
- adds a contract test for the migration pattern and stabilizes the
existing A2UI renderer test that blocked pre-commit
## Verification
- `pnpm exec vitest run
scripts/__tests__/integration-intelligence-migration.test.ts`
- `pnpm parity:verify --target=strands-python`
- `pnpm parity:check`
- `npm run build` in `examples/integrations/strands-python`
- `pnpm --dir packages/react-core exec vitest run
src/v2/__tests__/A2UIMessageRenderer.test.tsx`
- `NX_TUI=false pnpm nx run @copilotkit/react-core:test`
- `git diff --check`
- pre-commit hooks on both commits: `pnpm run test` and `pnpm run
check:packages`
## Manual smoke
- ran the local Intelligence stack from `test-signups4`
- started the Strands agent and Next UI on `localhost:3001`
- confirmed `/api/copilotkit/info` reported `mode: "intelligence"` and
`licenseStatus: "valid"`
- created a thread, asked the agent to enable app mode and add three
todos, then reloaded and selected the saved thread
- verified the restored thread showed the user prompt, tool calls,
assistant response, and the three persisted todos
Conflict resolutions:
- contract test: keep the rollout's parameterized version and add
strands-python to migratedIntegrations/appRoots (60/60 passing) in
place of her bespoke MIGRATED_INSTANCES file
- parity manifest: rollout's version with strands' three threads-shield
allowances removed (mirrors the langgraph-fastapi migration); parity
verify green — strands now 88 tracked files, zero drift
- package-lock: regenerated at 1.59.3 (a2ui-renderer stays 1.56.5,
the family-wide pin shared with the north-star)
Also rides: her react-core A2UIMessageRenderer test flake fix
(act -> waitFor), kept intentionally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
- add env-gated CopilotKit Intelligence wiring to the shared AgentCore
Hono/Lambda runtime bridge
- wire the Vite frontend to REST Threads transport with the Threads
drawer, locked gate, and shared `threadId` chat/canvas context
- add local Docker env propagation for Intelligence vars and the Vite
client-safe Threads gate
- pin AgentCore frontend/runtime packages to the threads-capable
CopilotKit/AG-UI versions and add missing frontend peer deps needed for
a clean build
- extend the integration migration regression suite with
AgentCore-specific coverage
## Verification
- `pnpm exec vitest run
scripts/__tests__/integration-intelligence-migration.test.ts`
- `pnpm exec oxfmt --check
scripts/__tests__/integration-intelligence-migration.test.ts
examples/integrations/agentcore/frontend/src/components/chat/CopilotKit/index.tsx
examples/integrations/agentcore/frontend/src/components/threads-drawer/locked-state.tsx
examples/integrations/agentcore/frontend/src/components/threads-drawer/threads-drawer.tsx
examples/integrations/agentcore/frontend/vite.config.ts
examples/integrations/agentcore/infra-cdk/lambdas/copilotkit-runtime/src/runtime.ts`
- `npm run build` from `examples/integrations/agentcore/frontend`
- `npm run build` from
`examples/integrations/agentcore/infra-cdk/lambdas/copilotkit-runtime`
- pre-commit hook: `check-binaries`, `sync-lockfile`, `lint-fix`,
`test-and-check-packages`, `commitlint`
- local Intelligence stack: copied
`/Users/mothra/Projects/test-signups4/.env` into AgentCore's ignored
`docker/.env`, started `docker compose up -d --wait`, seeded
`demo-user`/`1_demo-user`
- local bridge smoke: `GET
http://localhost:3101/copilotkit/threads?agentId=default` returned `200`
- Playwright smoke with local generated `aws-exports.json` + seeded OIDC
storage: authenticated Vite app rendered, licensed Threads drawer
visible on desktop, mobile floating pill visible, no horizontal
overflow, reload preserved authenticated render
## Notes
- Full live AgentCore response verification was not completed because
this path requires a deployed AWS AgentCore/Cognito stack and real
AgentCore agent runtime. The browser smoke used fake OIDC storage and a
dummy agent URL, so it verified frontend/runtime Threads wiring but not
a real agent response.
- In the fake-auth local smoke, the Intelligence realtime websocket
returned `403`, while the REST Threads list rendered successfully. I did
not treat that as proof of a production websocket issue because the
smoke bypassed real Cognito/auth, but it is worth keeping an eye on in a
deployed AgentCore environment.
Conflict resolution: take the rollout's contract test and append the
agentcore describe block (CDK lambda runtime gate, Vite frontend with
import.meta.env gate, docker env wiring). Bump both agentcore
package.jsons 1.59.1 -> 1.59.3 + regen lockfiles. 54/54.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
- add Intelligence Threads wiring for `examples/integrations/a2a-a2ui`
- add the reusable threads drawer UI and env-gated Intelligence runtime
config
- add a local A2UI v0.8 renderer so the demo renders the current A2A
operation payloads
- extend the migration verifier coverage for a2a-a2ui
## Verification
- `pnpm exec vitest run
scripts/__tests__/integration-intelligence-migration.test.ts`
- `npm run build` from `examples/integrations/a2a-a2ui`
- pre-commit hook: `pnpm run test && pnpm run check:packages` via
lefthook
Refs ENT-734.
Conflict resolution: take the rollout's parameterized contract test and
append the a2a-a2ui bespoke tests (namespaced helper, 1.59.3 pins).
Bump a2a-a2ui @copilotkit/* 1.59.1 -> 1.59.3 + regen lockfile. 49/49.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Audit follow-up to #5232, which fixed the gen-ui-interrupt d6 fixture
leg mis-order for langgraph-python + langgraph-typescript. aimock's
matchFixture is first-match-wins in array order and the loader dedups
on userMessage, so a toolCallId resume-leg placed BEFORE its
toolName:schedule_meeting first-leg shadows the tool-emitting leg on
turn-2 — the interrupt never fires and time-picker-card never mounts.
Fleet audit of every showcase/aimock/d6/*/gen-ui-interrupt.json found
ag2 as the only remaining mis-ordered fixture (both toolCallId
resume-legs preceded their toolName first-legs). Reorder ag2 so each
pill's toolName first-leg precedes its toolCallId resume-leg, matching
the proven-correct mastra/pydantic-ai/langgraph pattern. Reorder only;
response payloads and match keys are unchanged.
All other gen-ui-interrupt-supporting integrations (built-in-agent,
claude-sdk-*, langgraph-fastapi, langroid, ms-agent-dotnet/python,
pydantic-ai, spring-ai, strands) were already correctly ordered.
- bump a2a-middleware, mcp-apps, agent-spec from 1.59.1 (their verified
pre-revert state) to 1.59.3 to match the starters
- regenerate package-lock.json for the 11 examples whose package.json
changed (drawer deps re-added on starters, version bumps on the three)
- update the migration contract test's version assertions to 1.59.3
(43/43 passing); oxfmt pass
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Restores #5151 (north-star + batch 1 + crewai-flows + llamaindex),
#5196 (pydantic-ai), #5205 (a2a-middleware), #5211 (mcp-apps), reconciled
onto the current main baseline rather than the pre-revert tree:
- keep main's 1.59.3 pins, AGENT_URL normalization, default agent keys,
useConfigureSuggestions, available:false, useRenderTool
status/parameters API, call-time agent.state reads, and crewai-crews'
rebuilt page (not yet threads-migrated)
- graft the threads layer (drawer/gate/provider, env-gated route
intelligence block, next.config gate, env docs, drawer deps) on top
- drop threads-era sidebar suggestions props where main now registers
suggestions via useConfigureSuggestions (or omits them)
- fix the stale pydantic-ai doc link main reintroduced in ms-af-dotnet
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jordan Ritter