find_previous_deployment returned the second-newest SUCCESS regardless of the
head deploy's status. When the head deploy FAILED/CRASHED (precisely when
rollback is invoked) the newest SUCCESS is the last-known-good target, but the
old code skipped it and rolled back one good deploy too far. Now select the
newest SUCCESS strictly older than the current head (drop head, first SUCCESS
in the remainder), and fail loud when the fetched window is saturated and no
target is found — directing the operator to pass --to explicitly rather than
silently returning nil.
Also guard the one unguarded accessor in EnvDiffCommand#diff_services with
`|| []` so a snapshot lacking "services" hits the documented exit-2 path
instead of a raw NoMethodError.
Renumber the ivar-lint allowlist to track the added lines in bin/railway.
verify-prod commonly runs seconds after a promote pins a new image digest,
while Railway is still rolling the container out. checkDeploymentSuccess (in
verify-deploy.drivers.baseline.ts — the SSOT for the deployment-SUCCESS gate
shared by every driver) treated a transient in-progress status as a hard FAIL:
promote run 26966193624's predecessor pinned the docs digest, then verify-prod
fired ~17s later and FAILED with status="DEPLOYING" — a race, not a failure.
Now the deployment-status check polls when the latest deployment sits in any
non-terminal Railway status (QUEUED/BUILDING/INITIALIZING/DEPLOYING/WAITING/
NEEDS_APPROVAL), re-querying every 5s up to a 150s budget until it reaches a
terminal state, then asserts SUCCESS. Terminal-failure statuses (FAILED/
CRASHED/REMOVED/any non-SUCCESS-non-in-progress) still fail FAST with the
original error-string shape — no waiting. Infra/contract errors (network,
GraphQL errors[], missing edge) also fail fast.
The poll loop is fully seamed for tests (injectable sleep/now/budget); the
signature stays backward-compatible (trailing optional pollOpts). probeBaseline
forwards an optional deployPoll through, and both --env staging and --env prod
go through the identical code path, so staging behavior is unchanged.
Red-green tests: in-progress-then-SUCCESS passes (and polls), in-progress-until-
timeout fails with "still in progress", terminal FAILED fails fast with zero
sleeps.
Three pre-existing showcase_promote.yml issues surfaced by code review:
1. verify-prod's empty-SERVICES_CSV branch exited 0 vacuously, relying on the
unenforced assumption "promote already failed if the succeeded set is
empty". Pass PROMOTE_RESULT into the step env and fail loud on contract
violation: if promote reported success yet emitted no CSV, emit
"::error::promote reported success but succeeded_csv is empty — contract
violation" and exit 1. The genuine all-failed case keeps the ::notice:: +
exit 0.
2. Both verify-deploy.ts invocations (verify-staging-precondition, verify-prod)
ran `npm ci` in showcase/scripts but then `npx tsx showcase/scripts/...`
from the repo root, so npx could network-fetch tsx instead of using the
local install. Give those steps working-directory: showcase/scripts and the
relative path verify-deploy.ts, matching how resolve-targets/promote run
emit-railway-envs-json.ts. verify-deploy.ts takes no cwd-relative file args
(only --env/--services) and imports ./railway-envs relative to its own file,
so the cwd change is safe.
3. The notify job's Slack payloads embedded literal '\n' inside
toJSON(format('...\n...')). GitHub Actions expression string literals do not
interpret backslash escapes, so the '\n' survived toJSON as the two chars
\\n and Slack rendered it verbatim as "\n" instead of a line break.
EMPIRICALLY CONFIRMED against the live #team-showcase webhook: an existing
production post from a real promote run showed literal "\n", and a corrected
payload (real LF) rendered proper line breaks. Fixed BOTH posting steps by
injecting newlines via fromJSON('"\n"') (a real LF char) as a format()
argument instead of a literal '\n' in the template.
Two pre-existing bin/railway bugs surfaced by code review:
1. RollbackCommand#find_previous_deployment selected "the second SUCCESS in
reverse-chronological order" but never sorted the deployments — it trusted
Railway's arbitrary GraphQL connection order, so the wrong deploy could be
chosen as the rollback target. Now sort by createdAt descending before
selecting, mirroring the sibling fetch_latest_staging_deployments which
documents and guards against this exact hazard.
2. EnvDiffCommand advertised behavior it never implemented:
- The banner claimed it compares "custom domains", but run() only diffed
digest/startCommand/env_keys. custom_domains was captured in the snapshot
yet never compared. FIX: extract a pure diff_services helper and add a
custom-domain set comparison (both directions).
- --ignore-env-scoped was parsed into options[:ignore_env_scoped] but never
read, and the env_scoped? helper (+ ENV_SCOPED_URL_MARKERS) it relied on
was defined but never called. The flag had nothing meaningful to act on:
env-diff only compares env-var KEY names (snapshots never carry values),
and custom domains are compared as exact sets (silently dropping
"env-scoped" domains would defeat the comparison). DECISION: remove the
dead flag, the unused env_scoped? helper, and the now-orphan
ENV_SCOPED_URL_MARKERS constant rather than leave false advertising.
Tests: TDD red->green for both. test_rollback_sort feeds deployments in
non-chronological edge order (with a newer FAILED deploy as a decoy) and
asserts the true second-newest SUCCESS is returned. test_env_diff exercises
diff_services directly for identical and differing custom_domains. The
snapshot-ivar lint allowlist is renumbered (numbers only) to track the line
shifts from removing the dead code and adding the rollback comment.
## What does this PR do?
Bumps the default Gemini model across the Google ADK showcase from
`gemini-2.5-flash` to `gemini-3.1-flash-lite` — the closest GA model in
the Gemini 3.x line. The Gemini 2.5 family is slated for deprecation in
June 2026, so this gets ahead of the cutoff.
**Files touched (14):**
- Agent factories — `DEFAULT_MODEL`, `_DEFAULT_A2UI_MODEL`, `_SUB_MODEL`
in `shared_chat.py`, `main.py`, `beautiful_chat_agent.py`,
`subagents_agent.py`
- Prose references — docstrings in `shared_chat.py`,
`multimodal_agent.py`, `tool_rendering_reasoning_chain_agent.py`,
`registry.py`; comment in `entrypoint.sh`
- Test fixtures — `test_generate_a2ui.py` default-model assertion,
`test_stop_on_terminal_text.py` docstring
- Docs/config — `manifest.yaml` reasoning-demo description, QA prereqs
in `multimodal.md`, `headless-complete.md`, `headless-simple.md`
### Why Flash-Lite and not Flash?
There's no standalone `gemini-3.1-flash` identifier. The 3.1 Flash
family only ships as Flash-Lite (GA), Flash Live (preview), Flash Image
(preview), or Flash TTS (preview). Flash-Lite is the only stable choice;
the non-Lite `gemini-3-flash-preview` is still preview-only.
### Worth verifying empirically
A few behaviors that may differ from 2.5-flash and warrant a smoke test
before merging:
- **Thinking mode** (`build_thinking_chat_agent`, used by
`reasoning-custom` and `tool-rendering-reasoning-chain`) — Gemini 3.x
uses `thinkingLevel` instead of `thinkingBudget`, and Lite variants
historically have reduced reasoning vs full Flash/Pro
- **Forced tool-calling**
(`ToolConfig.function_calling_config.mode="ANY"` in A2UI planner)
- **Multimodal** image + PDF (Flash-Lite docs explicitly support
text/image/video/audio/PDF, so this should work)
- **`stop_on_terminal_text` loop-stopper** — the quirk was originally
observed on 2.5-flash; whether Flash-Lite still needs the guard is open,
but leaving it in is harmless
## Related PRs and Issues
- N/A — proactive model bump ahead of Gemini 2.5 deprecation
## Checklist
- [x] I have read the [Contribution
Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md)
- [x] If the PR changes or adds functionality, I have updated the
relevant documentation
- [x] "Allow edits by maintainers" is checked
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Red-green coverage for the writer-hardening fixes:
- in-flight cap DROP emits dropped-over-inflight and the counter does not leak
- a hung create times out, counts as a failure, and escalates after the threshold
- once writes are timing out, concurrent drops contribute to the escalation
- a drop while a write succeeds (healthy backpressure) does NOT escalate
- a no-progress deleteByFilter over cap backs off (gated by interval, not per insert)
- a cold start far over cap prunes in bounded batches (<= 200), never a thousands-row list/filter
- a hung Slack webhook is aborted at the timeout so the serialized health-signal chain isn't stalled
Three durable-logging robustness fixes in the resource-snapshot writer:
- PRUNE BATCH CAP + NO-PROGRESS BACKOFF: a cold start over a large backlog
(or lowering RESOURCE_SNAPSHOT_MAX_ROWS on a full collection) made
`surplus` thousands large, used directly as a `perPage` and a
one-clause-per-row id-delete filter — a giant request/URL that PB
rejects (400/414) so the prune never converged. Cap each sweep to
PRUNE_BATCH_MAX (200) oldest rows and let knownOverCap drain the rest
across inserts. When a sweep makes NO structural progress while still
over cap (PB delete-rule 403, filter 400, or 0 matched), back off via a
suppression window so the retry is gated to once-per-interval instead of
re-arming knownOverCap and re-running the failing sweep on every insert.
- DROP→ESCALATION SEAM: an in-flight-cap DROP did not count toward the
consecutive-failure escalation, so a 100%-broken PB whose writes hang
could keep new snapshots on the DROP path and delay/dodge the loud
`write-failing-systematically` error. A drop now contributes to the
failure signal when writes are already failing (consecutiveFailures > 0),
with no false positive on healthy backpressure (a drop while writes
succeed does not escalate).
- ORPHAN-CREATE REJECTION: on a write timeout the race rejects but the
orphaned `pb.create` keeps running; a later rejection from a client that
ignores the AbortSignal would be unhandled. Attach a no-op catch.
formatGauges had no production callers after the fix round removed the
gauge-string logging; only its own test referenced it. Delete the dead
function and the describe("formatGauges") block.
Add coverage (slot-6 F1) that `writeUnrecoverable` NAMES the proven wedge signal
in the Slack/alarm message when the give-up snapshot measured the cgroup PID
ceiling (cgroupPidsCurrent >= 0) — pids.current/pids.max + threads — and OMITS
the clause when the gauges are the off-Linux `-1` sentinel.
Documentation the diff invalidated, swept to match reality:
- browser-pool.ts JSDoc/comments + browser-pool.test.ts comments claimed
hard-recovery "purges the stale /tmp/playwright_* profile dirs". The code is
purge-free — hardRecover is a PACED cold relaunch that gives the
thread-exhausted kernel time to relax. Rewrite to match: paced cold relaunch,
NO /tmp purge (the wedge is the cgroup pids ceiling, mitigated demand-side;
/tmp is a proven non-cause).
- probe-invoker.ts stagger comment said "8 simultaneous workers", "~400 thread
spawns", "10-minute D6 budget" → updated to the current 5 / 20-min reality.
- e2e-deep.yml: the global cap note (40, "D6 peak 32 + D5 peak 8") → 24, with
D6 peak now 5×4=20; 20+8=28 > 24 so a d6+d5 overlap serializes under the
global cap (intended demand-side back-pressure).
- d6-all-pills-e2e.yml: the worked example (max_concurrency 8, ceil(18/8)=3
rounds) → 5 / ceil(18/5)=4 rounds, cross-referencing the overlap block that
justifies the concurrency itself.
Robustness/bounding nits a 7-agent CR flagged:
- browser-pool: the hot acquire/release path emitted a SECOND `hot-gauges` info
log duplicating inUse/available already on the acquire/release line. Fold the
two cheap cgroup-pid fields INTO the existing acquire/release log line (one
line, not two); keep the cheap two-file-read sampling (now `readHotGauges`).
- probe-invoker: `logTickGauges` double-encoded the sample as both a formatted
`gauges:` string AND the structured `...g` spread. Drop the redundant string;
keep the queryable spread.
- orchestrator: the best-effort Slack ping in `writeUnrecoverable` had no
timeout, so a hung webhook could stall the serialized health-signal write
chain. Add an AbortController timeout on the fetch.
Normalize the migration schema comments to reality: the heartbeat is 45s (not
the stale "~30-60s"), retention prunes by stable row id (robust to
same-millisecond observed_at ties) rather than a bare timestamp cutoff, and
document the null-vs-unavailable convention — the nullable number fields store
`null` for the `-1` "unavailable" sentinel so post-wedge queries cleanly
separate a measured reading from an unavailable one.
The snapshot writer had several robustness gaps a 7-agent CR flagged:
- PRUNE TIES: the ring prune deleted `observed_at < cutoff` (strict), but
observed_at is ms-resolution and several snapshots can fire in the same
millisecond (degraded+heartbeat+crash). If surplus rows tied the boundary
timestamp, strict `<` deleted nothing → knownOverCap stayed true → every
insert re-ran a full no-op sweep → unbounded growth + per-insert cost. Now
the prune deletes the oldest `surplus` rows by stable row id, which converges
regardless of ties, with a deleted==0 safety valve that can't spin.
- RATE-LIMIT CATCH-UP: a local row-count estimate (re-anchored to the real
total every sweep) forces a catch-up prune the moment we cross the cap, so the
rate-limit only suppresses sweeps while UNDER cap and never strands us over.
- FAILURE ESCALATION: swallow+warn alone hid a systematic PB rejection (unrun
migration / outage / schema drift). Now consecutive failures are counted, the
per-failure warn is throttled, and after a threshold the writer escalates ONCE
to logger.error (latched, reset on first success).
- BOUNDING: a best-effort in-flight cap drops snapshots past N concurrent
unawaited creates (launch-crash-loop burst), and a Promise.race write timeout
frees the in-flight slot if a PB write hangs.
- NULL SENTINELS: the `-1` "unavailable" gauge sentinel is written as `null`
(PB number fields are nullable) so post-wedge queries separate measured-vs-
unavailable.
- DOCS: cadence prose + JSDoc updated to the real 45s heartbeat, and a
single-writer note that the fleet case needs an elected pruner.
Red-green tests added: identical-boundary-timestamp prune converges to maxRows;
rate-limit keeps prune off the steady-state path but knownOverCap catches up
over cap; N consecutive failures escalate to error once and reset on success;
`-1` gauge written as null.
browser-pool.ts imported `./resource-gauges` without a `.js` extension while
sibling files use `.js`. The harness builds with tsc (module: esnext) and runs
the emitted ESM directly (`node dist/orchestrator.js`); moduleResolution:
bundler lets it typecheck but tsc emits the specifier verbatim and Node ESM
does NO extension resolution, so the dist would fail at boot with
ERR_MODULE_NOT_FOUND. Append `.js` to the extensionless relative specifiers in
browser-pool.ts and resource-gauges.test.ts. Verified by building the dist and
boot-importing dist/orchestrator.js (BOOT-IMPORT-OK, no module-not-found).
The live wedge peaked 998/1000 against the cgroup pids.max=1000 ceiling
specifically during a d6+d5 overlap — the simultaneous chromium renderers of
both probes summed past the ceiling. Fewer concurrent d6 feature-workers cut
the simultaneous-renderer thread peak, leaving headroom for a co-firing d5 (or
a recovery relaunch) without crossing 1000. Complementary to the
MAX_CONTEXTS 40->24 lever already landed. Comment updated to the pids-ceiling
reasoning; 5 x 4 = 20 concurrent contexts stays under the 24-context cap.
Previously full gauges were logged on only 3 sites (launch, self-heal-launch-
failed, probe-tick). Extend forensic gauge coverage across the whole pool
lifecycle so a recurring wedge has copious history to look back at.
- Full forensic snapshot (sample + log + durable onSnapshot hook) on every
meaningful transition: init, recycle (crash/hygiene), crash/disconnect,
set-empty degraded, self-heal-launch-failed, recovered, unrecoverable, and
shutdown.
- Periodic heartbeat (~45s, env BROWSER_POOL_HEARTBEAT_MS) via a
shutdown-signal-gated self-rescheduling loop — NOT a raw setInterval that
leaks past shutdown — for a baseline trend between events.
- Cheap cgroup-PID-only subset (logHotGauges) on the hot acquire/release path:
high-resolution view of the headline pids.current/pids.max signal without
the full-sample /proc-walk + df cost, and no durable write on the hot path.
- onSnapshot is best-effort: a throwing hook is caught + logged, never crashes
the pool.
The browser-pool wedge ends in a container restart that clears in-memory
state, and Railway's stdout window rolls off — so stdout/in-memory gauges
are not retrievable post-wedge. Persist the OS resource gauges to a new
`resource_snapshots` PB collection so the forensic history survives the
restart and the PID/thread-ceiling exhaustion is reconstructable after.
- New PB migration `1779989300_create_resource_snapshots.js` (mirrors the
probe_runs idempotent create + public-read pattern).
- `resource-snapshot-writer.ts`: best-effort writer (swallows + logs every PB
error so a missing migration / PB hiccup never breaks the pool) with
ring-style retention (cap last N rows, default 5000, env-overridable).
- Orchestrator wires the writer to the pool's onSnapshot hook, reusing the
shared pb client.
The Dockerfile FIX#3 comment claimed the in-image `ulimit -u` raised the cgroup
pids.max (target 16384). It cannot: `ulimit -u` only lifts this process's
RLIMIT_NPROC; the cgroup pids.max is set by the container runtime
(--pids-limit), is platform-fixed at 1000 on staging, and is not raisable from
the image. Rewrite the comment to state this accurately and point at the real
demand-side mitigation (fewer concurrent contexts + gauge/alarm). The harmless
`ulimit -u` line itself is retained. Also update the env test to assert the new
MAX_CONTEXTS default of 24.
The wedge was reproduced as cgroup PID/thread-ceiling exhaustion (pthread EAGAIN
-> "has been closed" -> crash-loop with pids.current pegged at pids.max=1000).
Every /tmp candidate was refuted by measurement, so this removes the /tmp
machinery that targeted a non-cause and replaces it with the demand-side fix +
observability:
- Remove the proactive stale-profile sweep (init + recycle), defaultPurgeProfileDirs,
the PurgeProfileDirs type, the staleProfileDirTtlMs / PURGE_MIN_AGE_MS options
and env, and the breaker's /tmp purge — the hard recovery is now just the paced
cold relaunch. Corresponding purge/sweep tests removed.
- KEEP the circuit-breaker failure-counting -> give-up -> onUnrecoverable alarm
(root-cause-agnostic backstop) and the footgun clamp; the breaker tests are
adapted to drive the wedge via the launcher instead of the purge.
- Log the resource gauges on every launchBrowser(), on the self-heal-launch-failed
path, and at probe-tick start/end so a burst approaching pids.max is observable
and an EAGAIN correlates to a measured pids.current.
- Include pids.current/pids.max + thread count in the onUnrecoverable payload and
the orchestrator's terminal alarm message so the alert names the real signal.
- Lower the BROWSER_POOL_MAX_CONTEXTS default 40 -> 24 to cap thread demand under
the platform-fixed ceiling (still env-overridable).
Standalone, unit-tested gauge sampler that reads the cgroup PID controller
(pids.current / pids.max, v2 with v1 fallback) plus the process-tree thread/proc
counts, FDs, RSS, /dev/shm, and /tmp inode/space from /proc + /sys/fs/cgroup.
The cgroup PID counters are the headline signal for the proven browser-pool
wedge (PID/thread-ceiling exhaustion). Degrades gracefully to -1 off-Linux so it
is safe to import anywhere. Consumed by the browser pool and probe invoker in
follow-up commits.
The recurring browser-pool wedge's ROOT CAUSE: every chromium.launch() leaves
a /tmp/playwright_* profile dir + playwright-artifacts-* dir whose Playwright
close-time cleanup is best-effort (swallowed on failure). In a long-lived
harness container under sustained d6 cron load these orphans accumulate
monotonically until /tmp exhausts inodes/space — then every launch throws
"...has been closed" and self-heal relaunches into the same exhausted /tmp
forever (only a restart, which gives a fresh /tmp, clears it).
PREVENTION: sweep the age-gated stale dirs PROACTIVELY on a cadence — once at
init() and on EVERY recycle relaunch (crash + hygiene) — so /tmp never reaches
exhaustion. This is the durable complement to the circuit-breaker ESCAPE: the
sweep prevents the wedge; the breaker escapes it if it ever still happens.
Unifies the purge into ONE canonical helper — defaultPurgeProfileDirs now takes
an age threshold (minAgeMs) as its first arg. The proactive sweep passes the
conservative staleProfileDirTtlMs (5min default, env
BROWSER_POOL_STALE_PROFILE_DIR_TTL_MS); the breaker's hard-recovery keeps its
aggressive PURGE_MIN_AGE_MS (60s). Same hardened (symlink-safe, realpath-checked)
implementation, different age — no duplicate purge logic.
Adds 4 prevention tests (init-sweep, every-recycle-sweep, bounded-accumulation
over 30 cycles, real-FS age gate) and adapts the existing breaker tests to gate
their purge-count assertions on the breaker's age so the shared cadence does not
inflate them.
defaultPurgeProfileDirs (the self-heal hard-recovery profile-dir purge) ran a
blind readdir + rm(recursive, force) against the world-writable OS temp dir:
- A planted playwright_* SYMLINK had its TARGET recursively deleted. Now read
dirents (withFileTypes), SKIP symlinks + non-directories, and verify the
realpath stays under the scan dir before removing.
- A browser launching CONCURRENTLY with the purge had its FRESH profile dir
nuked, re-wedging the very launch being revived. Now only dirs older than
PURGE_MIN_AGE_MS (60s) are eligible (mtime via lstat).
The real purge previously had ZERO coverage (the pool tests inject a no-op).
Parameterize the scan dir so it is unit-testable against a temp fixture, export
it, and add tests: removes old playwright_*/playwright-artifacts-* dirs, leaves
non-matching dirs, SKIPS symlinks (target untouched), leaves too-fresh dirs, and
swallows ENOENT. (Also pulls in the resolvePositive helper used by the breaker
clamp, which lives adjacent to this function.)
Two circuit-breaker robustness fixes:
Latch second-episode silence: this.unrecoverable was cleared ONLY on a
successful launch, so a permanently-wedged container that re-empties after the
first give-up hit `if (this.unrecoverable) return` and went silent on every
subsequent degraded episode — the exact silent-spin the breaker exists to kill.
Remove the instance latch entirely; the loop-local consecutiveHardRecoveries
guard already gives once-per-episode (the loop returns immediately after firing),
and each fresh degraded episode spawns its own self-heal loop that can alarm.
Stale JSDoc updated to match. The onUnrecoverable hook now carries breaker
counters so the wired alarm can report how hard the pool tried.
Disable footgun: a selfHealHardRecoveryThreshold / selfHealMaxHardRecoveries of
0 silently disabled both the hard-recovery escape AND the give-up (guards are
`> 0`), reverting to the infinite silent spin. Clamp resolved breaker thresholds
to >= 1 (resolvePositive) so a config typo can't disable the safety net.
Tests: a second degraded episode ALSO fires onUnrecoverable (red→green: prior
latch silenced it); a 0 threshold stays armed (red→green: prior it spun
forever); plus a browsers:2 multi-browser test locking in the partial-revive
counter-reset / breaker pacing that was previously unverified.
The self-heal circuit-breaker's terminal give-up shipped a mechanism with no
operator signal — the pool's onUnrecoverable hook was never wired, so a
breaker give-up silently stopped spinning without telling anyone a redeploy is
required. Add createBrowserPoolHealthSignals.writeUnrecoverable and wire it to
the pool's onUnrecoverable hook:
- Unconditional health-signal writes: a DISTINCT system:browser-pool-unrecoverable
key (red, terminal, critical, redeployRequired + breaker counters) AND an
escalation of the shared degraded key to severity=critical/terminal, so a
give-up is distinguishable from a transient self-heal-degraded.
- Best-effort, env-guarded Slack ping on SLACK_WEBHOOK_BROWSER_POOL_UNRECOVERABLE
(unset by default per alerting discipline — deploy the code, wire the URL
separately); a failing/absent webhook never blocks the health-signal write.
The pool's onUnrecoverable hook now passes breaker counters so the alarm can
report how hard the pool tried before giving up.
The recurring staging BrowserPool collapse (#5185/#5221/#5225 each chipped at it but it kept recurring): after the long-lived harness container runs ~hours under sustained d6 cron load, chromium enters a launch crash-loop where every chromium.launch() throws 'Target page, context or browser has been closed'. The set empties, startSelfHeal() kicks in, and its loop just relaunches into the SAME wedged state forever (self-heal-launch-failed repeating) without ever escaping — the stale /tmp/playwright_* profile dirs / FD-shm pressure persist across every relaunch. acquire() therefore has no contexts forever and blocks to timeout fleet-wide; only a container restart cleared it (reactive).
Adds a circuit-breaker to the self-heal loop: after selfHealHardRecoveryThreshold (default 4) consecutive launch failures, instead of looping another identical relaunch the pool performs a HARD recovery — purges the stale ${tmpdir()}/playwright_* profile/temp dirs the crash-looping processes left behind, then cold-launches fresh. Any successful launch resets the counter. If selfHealMaxHardRecoveries (default 3) consecutive hard recoveries also fail, it fires a loud browser-pool.pool-unrecoverable alarm (onUnrecoverable hook) and stops the heal loop rather than spinning silently — the operator signal that a redeploy is genuinely required. All tunable via BROWSER_POOL_SELF_HEAL_* env vars; the purge is injectable for tests and defaults to a hermetic no-op when a fake launcher is injected.
## Summary
Adds a Slack notification to **#team-showcase** whenever a showcase
staging→prod promotion **succeeds**, matching our existing routing
convention (releases → #engr, PRs/promote-failures → #oss-alerts). Today
the promote workflow only posts to #oss-alerts on failure; the `success`
state was computed but unused.
## Changes (all in the `notify` job of `showcase_promote.yml`)
- New **success-only** "Post to #team-showcase" step, gated on `state ==
'success'`, mirroring the existing #oss-alerts step exactly (same pinned
`slackapi/slack-github-action@b0fa283 # v2.1.0`, `incoming-webhook`,
`toJSON(format(...))` payload).
- Job-level `SLACK_WEBHOOK_TS: ${{ secrets.SLACK_WEBHOOK_TEAM_SHOWCASE
}}` so the step `if:` can gate on `env.SLACK_WEBHOOK_TS != ''` (step
`if:` can't read `secrets.*` directly — same pattern as the existing
`SLACK_WEBHOOK`).
- Symmetric "Log (no Slack — team-showcase webhook unset)" `::notice::`
fallback, so a successful promote with the webhook unset still leaves a
trace (mirrors the failure path's fallback).
- Failures still route **only** to #oss-alerts — unchanged.
## Activation (required before notifications fire)
The new step **no-ops until** an org secret
`SLACK_WEBHOOK_TEAM_SHOWCASE` exists, so this is safe to merge now. To
activate: create the `#team-showcase` Slack incoming webhook, then set
it as a CopilotKit **org** secret (visibility=all), matching the
convention for `SLACK_WEBHOOK_OSS_ALERTS` / `SLACK_WEBHOOK_ENGR`.
## Test plan
- [x] `actionlint` exit 0
- [x] YAML parses
- [x] Step is mutually exclusive with the webhook-unset log fallback
(one fires per successful run)
- [ ] CI green
- [ ] After the org secret is set: a real promote posts to
#team-showcase
## Out of scope (separate follow-up)
Review surfaced several pre-existing `notify`/`verify-prod` robustness
items not touched here (verify-prod empty-CSV vacuous-green hardening,
`npx tsx` working-dir in the verify jobs, env-vs-secrets
webhook-reference fragility, literal `\n` rendering shared with the
existing #oss-alerts step). To be addressed in a separate
showcase-promote hardening PR.
## Summary
`showcase bin/railway promote docs` has been failing in CI on every run
with `REFUSE: P1 (docs): GHCR manifest HEAD 403 for
ghcr.io/copilotkit/showcase-shell-docs:latest`. Root cause:
`GHCR#bearer_for` sent the GitHub/Actions token **raw** as
`Authorization: Bearer <token>` to the GHCR OCI manifest endpoint, which
rejects it with `403`. GHCR requires the token to first be exchanged for
a short-lived registry bearer via its `/token` endpoint.
Verified live against the real public `showcase-shell-docs` package: raw
bearer → `403`, `/token`-minted bearer → `200`, anonymous → `401`.
## Changes
- `bearer_for` now **always** performs the GHCR `/token` exchange. When
a token is present it authenticates the exchange with Basic auth
(`base64("x-access-token:<token>")`) and uses the minted token as the
manifest bearer; public packages still mint anonymously.
- Fail-loud hardening: a failed exchange (non-2xx with a token present,
or an unparseable `/token` body) now raises `GHCR::Error` instead of
silently swallowing to `nil` and degrading to an anonymous request —
restoring `manifest_exists`'s documented "raises on transport/5xx"
contract.
- Tests: new `test_ghcr_bearer.rb` pins the on-the-wire behavior
(Basic-auth exchange, minted-bearer-not-raw-token, anonymous path,
raise-on-failed-exchange, never-anonymous-after-supplied-token-failure).
Existing GHCR specs updated to model the now-mandatory exchange.
## Test plan
- [x] `showcase/bin/spec` suite green: 117 runs, 410 assertions, 0
failures
- [x] `ruby -c showcase/bin/railway` syntax OK
- [x] Red-green verified on the new fail-loud paths
- [ ] CI green
## Out of scope (separate follow-up)
Review surfaced three pre-existing, unrelated `bin/railway` bugs not
touched here: `rollback`'s `find_previous_deployment` selects from an
unsorted deployment list, and `env-diff`'s `--ignore-env-scoped` flag /
advertised custom-domain comparison are no-ops. To be addressed in a
separate PR.
The A2UI middleware (@ag-ui/a2ui-middleware) forwards injectA2UITool on
forwardedProps; ag-ui-langgraph surfaces it into agent state at
state["ag-ui"]["inject_a2ui_tool"]. The CopilotKit LangGraph middleware
(py + js) now reads that flag and only injects generate_a2ui when it is
truthy (opt-in), drops the runtime's render_a2ui so the model sees one
A2UI tool, and skips if the agent already defines generate_a2ui. The
catalog only binds surfaces; it is no longer the gate.
Reverts the earlier runtime-forward + context-channel approach.
Deps: ag-ui-langgraph>=0.0.38 (py), @ag-ui/langgraph 0.0.37 (sdk-js),
@ag-ui/a2ui-middleware 0.0.6 + @ag-ui/langgraph 0.0.37 (runtime);
.npmrc min-release-age exclude for @ag-ui/a2ui-middleware. Showcase
langgraph pins bumped to copilotkit==0.1.94a3 / sdk-js 1.59.3-alpha.3 /
@ag-ui/langgraph 0.0.37.
declarative-gen-ui moved to the CopilotKitMiddleware auto-A2UI path across
the 3 langgraph integrations. The middleware's inner forced tool is
render_a2ui, so each integration's gen-ui-declarative.json gained 4
render_a2ui fixtures that share match keys with the pre-existing render_a2ui
entries in that integration's render-a2ui.json (the a2ui_fixed demo). 4
pills x 3 integrations = 12. Same-context cross-demo overlap, disambiguated
at runtime by the probe fixtureFile.
langgraph-fastapi bumped ag-ui-langgraph[fastapi] 0.0.35->0.0.37 (auto-A2UI
needs get_a2ui_tools). Both non-exact, so the FAIL count stays 63 but the
FAIL-set hash shifted. Update validatePinsFailHash to the new set.
The declarative-gen-ui demo across the three langgraph integrations now
relies on the middleware to inject and execute generate_a2ui — the agents
collapse to create_agent + CopilotKitMiddleware with no hand-rolled tool.
Adds render_a2ui fixtures for the new tool path and pins the integrations
to the A2UI alpha SDKs (copilotkit 0.1.94a1, @copilotkit/sdk-js 1.59.3-alpha.1).
The auto-injected generate_a2ui tool now resolves the A2UI catalog from
both delivery paths: the CopilotKit runtime proxy (a copilotkit.context
entry) and the AG-UI native endpoint (state["ag-ui"].a2ui_schema). The
registered catalog id is extracted and bound to generated surfaces so
BYOC custom catalogs render their own components instead of the basic
catalog. Covers @copilotkit/sdk-js and the copilotkit Python SDK.
Prebuilt agents get dynamic A2UI with no extra wiring — adding the
middleware is enough. When the frontend registers an A2UI catalog
(surfaced by the runtime into state["ag-ui"].a2ui_schema), the
middleware infers the agent's own model, advertises the generate_a2ui
tool in the model-call hook, and executes it in the tool-call hook.
No catalog → the tool is never advertised.
Covers both @copilotkit/sdk-js and the copilotkit Python SDK. Bumps
the A2UI tool-factory dependency to where get_a2ui_tools ships
(@ag-ui/langgraph 0.0.35, ag-ui-langgraph >=0.0.37).
Mirror the failure-path 'no Slack' observability fallback for the new
#team-showcase success post. When SLACK_WEBHOOK_TEAM_SHOWCASE is unset
(current default), a successful prod promote now emits a ::notice:: log
line instead of a silent green.
bearer_for ended with a blanket `rescue StandardError => nil` plus
`return nil if status >= 400`. Now that bearer_for ALWAYS performs the
/token exchange (even when a token is present), that swallow masked real
failures: a non-2xx /token response, a malformed JSON body, or a transport
error all collapsed to nil, after which manifest_exists issued the manifest
HEAD anonymously (no Authorization). That silently violated manifest_exists's
documented raise-on-transport/5xx contract and conflated "no token supplied"
with "supplied token failed to exchange".
bearer_for now:
- raises GHCR::Error on a >=400 /token response WHEN a token was supplied
(token absent still returns nil — the legitimate anonymous-public fallback);
- raises GHCR::Error on JSON::ParserError for an unparseable 200 body;
- no longer swallows StandardError, so transport exceptions
(Errno::ECONNREFUSED, Net::ReadTimeout, ...) propagate.
Call-site enumeration (bearer_for is called only by manifest_exists and
resolve_digest):
- manifest_exists: documents "Raises GHCR::Error on 5xx or transport failure",
so a raised exchange error is consistent with — and strengthens — its own
contract. Assumption holds.
- resolve_digest: already raises GHCR::Error on its own manifest HEAD >=400 and
does not rescue bearer_for, so a raised exchange error propagates exactly as
its other failures do. Assumption holds.
Neither caller is broken by bearer_for raising; both already propagate
GHCR::Error to their callers.
Tests: add 3 red-green cases in test_ghcr_bearer.rb (token-present 401 raises;
token-present malformed body raises; token-present exchange failure issues NO
anonymous manifest HEAD). The existing manifest_exists/digest fakes were only
green because the old swallow masked the fake's "no fake response" error — they
never modeled the mandatory /token exchange; added a successful /token fake to
each so they exercise the real path. Renumbered the snapshot-ivar-lint
allowlist (+13 lines) to track the bin/railway line drift.
bearer_for returned a raw GitHub/Actions token, which GHCR's OCI manifest
endpoint rejects with HTTP 403. Always perform the /token exchange instead:
authenticate with Basic base64("x-access-token:<token>") when a token is
present, fall back to the anonymous exchange for public packages, and use
the minted token as the manifest-read bearer. This fixes the docs promote
that has deterministically 403'd in CI.
Route the exchange through the injectable @http seam (matching resolve_digest
/ manifest_exists) so it is testable. Renumber the snapshot-ivar lint
allowlist for the +21 line shift.
Add a success-only Slack post to #team-showcase in the showcase_promote
notify job, mirroring the existing failure→#oss-alerts step. Guarded by
env.SLACK_WEBHOOK_TS so it safely no-ops until the SLACK_WEBHOOK_TEAM_SHOWCASE
org secret exists. Failures continue to route only to #oss-alerts.
Render a per-turn persistent intelligence indicator that stays stable
across multi-step turns and settles into a "finished" tag. Splits the
component into IntelligenceIndicator (logic) + IntelligenceIndicatorView
(presentation), wires it into CopilotChatView / CopilotChatMessageView,
adds the slot styles to globals.css, and a Storybook story plus
timer-free logic tests.
Add `useLearningContainers` / `...InCurrentThread`, which emit
`set_learning_containers` annotations (defaulting to the `project`
container) for the active thread, and export the new hooks from the
react-core hooks barrel.
Add the `recordAnnotation` client (posts to the runtime `/annotate`
endpoint; no client-side userId) and the `useLearnFromUserAction` /
`...InCurrentThread` hooks that record `user_action` annotations for the
self-learning loop.
Move enterprise-learning MCP attachment out of the BuiltInAgent-specific
path and the intelligence run handler into a single request-scoped hook:
- `attachIntelligenceEnterpriseLearning` (agent-utils) attaches
`@ag-ui/mcp-middleware` via `configureAgentForRequest`, gated on
`ɵisEnterpriseLearningEnabled()`, resolving the user via `identifyUser`
and the project apiKey.
- Called from `handleRunAgent`; the old `forwardedProps.auth` MCP plumbing
in `intelligence/run.ts` and the BuiltInAgent attach in `agent/index.ts`
are removed.
- Add released `@ag-ui/mcp-middleware@0.0.1` dependency (lockfile +
`@ag-ui/client` override). Drops the obsolete intelligence-mcp-helper test.
Replace the user-actions-specific path with a general thread-event
annotation endpoint:
- Route `PUT /connector/annotate/:clientEventId` via fetch-router /
fetch-handler / core hooks RouteInfo.
- `handleAnnotate` forwards `{ type, data, learningContainer }` to the
Intelligence platform.
- `CopilotKitIntelligence.annotate()` (idempotent PUT) supersedes the
prior `recordUserAction`; `AnnotateParams`/`AnnotateResponse` carry the
event `type` (`user_action`, `set_learning_containers`, ...).
Also renames the `mcpServer` config flag to `enableEnterpriseLearning`
(`ɵisEnterpriseLearningEnabled()`), consumed by the enterprise-learning
attach in the following commit.
## Summary
- Roll Intelligence/Threads support into
`examples/integrations/crewai-crews` using the #5151/#5189 migration
pattern.
- Add the env-gated `CopilotKitIntelligence` runtime path, REST thread
transport, Threads drawer UI, and local Intelligence env docs.
- Add a focused migration contract test covering the crewai-crews wiring
and stale v2 sidebar prop regression.
Linear:
https://linear.app/copilotkit/issue/ENT-734/roll-intelligencethreads-into-the-remaining-11-framework-examples-ent
## Verification
- `pnpm exec vitest run
scripts/__tests__/integration-intelligence-migration.test.ts`
- `pnpm exec oxfmt --check
scripts/__tests__/integration-intelligence-migration.test.ts
'examples/integrations/crewai-crews/src/app/api/copilotkit/[[...slug]]/route.ts'
examples/integrations/crewai-crews/src/app/page.tsx`
- `git diff --check`
- `./node_modules/.bin/tsc --noEmit` in
`examples/integrations/crewai-crews`
- `npm run build` in `examples/integrations/crewai-crews`
- `pnpm install --frozen-lockfile --ignore-scripts`
- pre-commit hook: `pnpm run lint --fix && pnpm run format`, `pnpm run
test`, `pnpm run check:packages`
- `pnpm run build`
## Manual smoke
- Copied local test env from `/Users/mothra/Projects/test-signups4/.env`
into the ignored example `.env`.
- Replaced missing `OPENAI_API_KEY` with a validated local
company/project key.
- Started the local Intelligence stack from `test-signups4` with `docker
compose up -d --wait`.
- Seeded `1_demo-user` / `demo-user` for organization `casa-de-erlang`.
- Ran the CrewAI agent on `:8000` and the UI on `:3010`, with `.env`
exported into both processes.
- Verified `/`, `/api/copilotkit/info`, and
`/api/copilotkit/threads?agentId=starterAgent&limit=20` returned
successfully; runtime info reported `mode: intelligence` and
`licenseStatus: valid`.
- Verified the licensed Threads UI renders in the browser and the drawer
opens with persisted threads.
- Created/selected a thread, sent `What are the proverbs?`, verified the
agent response rendered, refreshed, selected the saved thread, and
verified the user message + assistant response restored.
Note: this `CopilotSidebar` type surface does not expose visible
suggestion chips in `@copilotkit/react-core@1.59.1`, so the send step
was verified by typing the suggested prompt rather than clicking a
suggestion chip.
Jordan Ritter