Three functional fixes to the starter-fleet provisioner found in CR:
- fetchExistingServices now drains the Relay ServiceConnection via
pageInfo.hasNextPage/endCursor. A single un-paginated query truncated the
snapshot (~27 SSOT + 12 starter services span >1 page), making an existing
starter look absent → CREATE path → serviceCreate "already exists" →
non-transient abort of the whole run.
- TRANSIENT_ERROR_RE now matches Railway's INTERPOLATED "Service <id> not
found" (id embedded), not just the contiguous "Service not found", so the
post-create eventual-consistency retry actually fires.
- serviceInstanceRedeploy result check: documented the verified Boolean!
contract (sources: redeploy-env.ts, bin/railway RestoreCommand) and now
gates on truthiness (rejects false/null, accepts truthy defensively).
Hardening: ABORT on the live path when GITHUB_TOKEN is unset (private GHCR
images would image-pull-backoff while reporting success); warn-and-continue
only under --dry-run. Benign domain "already exists" no-op now logs the actual
matched Railway message for a forensic trail.
Harden the committed starter-fleet Railway provisioner against the
partial-failure / mistyped-flag / un-deployed-image failure modes
surfaced in CR:
- Domain idempotency: a serviceDomainCreate that Railway rejects with an
"already exists" error (start-of-run snapshot missed the domain due to
eventual consistency, or a prior run died mid-fleet) is now caught as a
benign no-op (marked "existing", logged) so a re-run converges instead
of aborting the entire remaining fleet. A genuine non-transient error
still aborts.
- Explicit redeploy: serviceCreate + serviceInstanceUpdate(source.image)
only PINS the image; it does not start a deployment, and Railway's image
auto-updates fire only on a NEW digest push. Added serviceInstanceRedeploy
after the instance update on BOTH the create and update paths so the
pinned image actually runs (and starter_smoke can find the service up).
Mirrors the documented update+redeploy pattern in bin/railway and the
explicit redeploy showcase_deploy.yml issues after each GHCR push.
- argv validation: parseArgs() now rejects any unrecognized argument
(e.g. a mistyped --dry-rn) with a usage hint before any provisioning,
instead of silently ignoring it and proceeding to REAL live provisioning.
- Fail-fast safety: validate the Railway token AND registry credentials
up front in main() (token resolution no longer process.exit()s deep in
the GraphQL boundary; main().catch owns the exit). Broadened the
withRetry transient predicate to the domain/instance eventual-consistency
class via an overridable per-call predicate. Dry-run now reports a new
service's domain as "would-create" for a faithful preview.
Adds showcase/scripts/provision-starter-fleet.ts — a committed, idempotent
provisioner for the SSOT-decoupled "starter container fleet". It creates (or
updates) one sleepable Railway service per starter template in the STAGING
environment, deriving the 12 targets from STARTER_TO_COLUMN (the smoke-matrix
SSOT) so the fleet can never drift from the build matrix.
Per service: serviceCreate scoped to the STAGING env (environmentId on
ServiceCreateInput, so NO production instance is ever materialized) with
source.image=ghcr.io/copilotkit/starter-<slug>:latest (RAW starter slug) and
GHCR registryCredentials; then serviceInstanceUpdate against staging with
sleepApplication:true + healthcheckPath="/" + region=us-west1; then
serviceDomainCreate for a generated staging domain. A bounded retry absorbs
Railway's eventual-consistency "ServiceInstance not found" right after create.
Healthcheck is "/" not "/api/health": the starters' single deployable image
EXPOSEs 3000 running the Next.js frontend, which serves "/" and
"/api/copilotkit" but has no "/api/health" route; the agent's "/health" is on
the internal 8123 port Railway does not expose. region read-back is null on the
serviceInstance for ALL existing showcase services too — that is normal Railway
behavior, so the fleet matches the existing services.
The fleet is decoupled from the 27-service railway-envs SSOT (starter-* services
are auto-discovered by the starter_smoke probe). #5254 already made
verify-railway-image-refs.ts tolerate starter-* names, so provisioning does not
trip the image-ref gate / skip the showcase build.
Red-green tested against an injected Railway GraphQL mock: target derivation
(raw vs remapped slug), GHCR credential resolution, sleepApplication:true,
staging-env scoping on BOTH create and update (never prod), idempotent
update-vs-create, domain de-duplication, and transient-error retry.
The prior cross-over returned a different tree shape once ready (top-level
`surfaces` vs the held wrapper), so React unmounted+remounted ReactSurfaceHost at
the swap — throwing away the offscreen-painted surface and reintroducing the gap
at the swap point. Keep ReactSurfaceHost in ONE stable position and only toggle
its wrapper styling (offscreen→in-flow) + the loader overlay, so the painted
surface is preserved. Also track the last pre-paint snapshot from content (not the
lagging operations state) so the paint snapshot can't clobber it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The skeleton unmounted the instant a2ui_operations arrived, but the A2UIProvider
needs a couple ticks to process the ops and paint — leaving a visible empty gap
between skeleton and first card. Hold the loader in-flow while the surface mounts
and paints OFFSCREEN (absolute, opacity 0), then swap, so the first card truly
replaces the skeleton. Keep showing the last pre-paint snapshot during the
hand-off so the building count / retry status carries through without a flicker.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The starter container fleet (starter-<slug>) is decoupled from the
27-service railway-envs SSOT: each starter-* service is auto-discovered
at runtime by the starter_smoke probe (railway-services discovery,
namePrefix "starter-") and is never read from railway-envs.ts.
verify-railway-image-refs.ts is a hard needs: of the build job and runs a
bidirectional live-Railway drift check. Before this change, provisioning
a starter-* service made it untracked in the SSOT, so findUntrackedServices
failed and the showcase build was SKIPPED (the canary regression).
Scope both drift checks (findUntrackedServices and, defensively,
findMissingServices) to exclude services matched by a single, well-named
predicate isStarterFleetService(name) => name.startsWith("starter-"),
mirroring the harness discovery filter convention. Real showcase-*/infra
services are still drift-checked exactly as before.
This is the prerequisite for Phase-3 starter provisioning — provisioning
must happen AFTER this merges.
The starter_smoke probe registers its driver under kind `starter_smoke`
(starterSmokeDriver, orchestrator.ts) and config/probes/starter_smoke.yml
declares `kind: starter_smoke`, but the probe-config Zod `kind` enum is
built from DIMENSIONS, which only listed the `starter` emit-prefix and not
the `starter_smoke` probe kind. The enum therefore rejected the YAML at
parse time; because `kind` is shared across all three union variants the
failure surfaced as the misleading "Unrecognized key(s) in object:
'discovery'" union error, the probe never loaded, AND probe-loader.test.ts's
shipped-config assertion failed (the lone recurring failure).
Add `starter_smoke` alongside `starter`, mirroring the existing kind/emit
pairs (e2e_d6/d6, e2e_deep/d5, e2e_demos/e2e). `discovery` was a red herring
— it is a valid key the DiscoveryBlockSchema already accepts; the working
d6-all-pills-e2e.yml uses the identical discovery shape.
probe_runs.triggered was `required: true`, and PocketBase rejects the
boolean `false` as empty (validation_required). The fleet aggregator opens
every run-history row with `triggered: false` (scheduled, not ad-hoc), so
run-history start() failed on every non-triggered run. Add an idempotent
migration that ALTERs the existing collection's field to optional (the
create migration already ran on staging/prod volumes, so we alter rather
than recreate).
The pre-paint A2UI experience was split across three uncoordinated render
paths: a per-tool-call skeleton (render_a2ui), the a2ui-surface activity, and a
separate a2ui_recovery activity. That caused a duplicate "Building interface…"
skeleton (one per tool call / retry) and a skeleton that lingered after the
surface painted, and left retry/failure UI floating beside the loader.
Collapse the whole lifecycle onto the a2ui-surface activity as the single
owner, swapped in place on one stable messageId:
building -> retrying -> failed -> (painted card)
- New A2UIRecoveryStates.tsx: shared building/retrying/failed subcomponents +
the animated skeleton (ported from the retired tool-call renderer) + the
threshold-gated retry label + debugExposure resolution.
- A2UIMessageRenderer now branches: a2ui_operations present -> paint surfaces;
else status failed/retrying/building. Recovery options (showAfterMs,
showAfterAttempts, debugExposure) flow in via `a2ui.recovery`. Server-stamped
content.debugExposure still wins. Lifecycle metadata lives on the AG-UI
activity-content wrapper, never inside an A2UI envelope (op elements stay
{version, <one op>}, per v0.9).
- Retire the render_a2ui tool-call skeleton: it now renders nothing (still
registered, so raw args are suppressed). Kills the duplication at the root.
- Drop createA2UIRecoveryRenderer + the separate a2ui_recovery registration;
the surface renderer owns it. Tests moved to A2UIRecoveryStates.test.tsx.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- Removes three project-level Claude Code config files from the
checked-out PR code immediately after checkout, before any other step
runs
- Closes three pre-model-load code execution vectors that a malicious PR
could exploit when a maintainer triggers the social copy generator
## What's removed and why
| File | Vector |
|------|--------|
| `.claude/settings.json` | `SessionStart` hooks execute as shell
commands before the model loads |
| `.claude/settings.local.json` | Same hook vector — gitignored but can
be force-committed to a branch |
| `.mcp.json` | MCP servers are launched as subprocesses before the
model loads in non-`--bare` mode |
All three fire before the model processes any prompt, so
`--allowedTools` restrictions don't protect against them.
## Context
The social copy generator checks out the PR's HEAD SHA and then runs
`claude -p` with `ANTHROPIC_API_KEY` in scope. An existing write-access
permission gate (a maintainer must manually check a checkbox to trigger
generation) limits who can pull the trigger — but doesn't prevent a
social-engineering attack where a seemingly legitimate PR contains a
malicious config file.
The step is placed immediately after `actions/checkout` so nothing in
the checked-out workspace can influence Claude Code before the
sanitization runs.
## Test plan
- [ ] Trigger the social copy generator on a normal PR and confirm it
still produces output
The v2 `agent/run` handler
(`packages/runtime/src/v2/runtime/handlers/handle-run.ts`) logged:
```
[CopilotKit Runtime] Warning: "agents" feature is not licensed. Visit copilotkit.ai/pricing
```
on **every** agent run whenever a `licenseChecker` was present.
## Why remove it
- **It can never pass.** `checkFeature("agents")` resolves against
`@copilotkit/license-verifier`, whose license catalog defines no
`"agents"` feature id. `isFeatureEnabled` returns `false` for any
unknown feature before it ever consults the license, so the warning
fires for every customer regardless of plan (free, dev, or enterprise).
## Scope
- Removes only the warning block.
- `runtime.licenseChecker` remains wired (`createLicenseChecker` in
`core/runtime.ts`) and is still consumed by `get-runtime-info` for
license status reporting — so this does not touch any actual
licensing/status behavior.
- No tests asserted this warning.
The agent/run handler logged `[CopilotKit Runtime] Warning: "agents"
feature is not licensed.` on every run whenever a licenseChecker was
present. The check is purely cosmetic (it never gated execution) and
fires for every customer because no license catalog defines an "agents"
feature id, so checkFeature("agents") can never return true. This
removes the misleading warning. The licenseChecker remains wired and is
still consumed by get-runtime-info for status reporting.
The smoke Dockerfile previously relied on a floating `npm install
next@latest` to upgrade crewai-crews from its pinned Next ^15.5.15 to
Next 16. PR #5250 removed that floating line for all starters, which was
correct for the other five but broke crewai-crews: Next 15 cannot
compile @copilotkit/react-core's `export *` re-exports, so the app build
failed.
Pin `next` to 16.2.7 (current latest, the patched build that compiles
react-core and is free of the 16.0.x security advisory) instead of
restoring the floating dependency. Locally, a clean no-cache
docker-compose smoke build compiles, boots, and passes all 4 starter
smoke tests.
The smoke-test app Dockerfiles ran `RUN npm install next@latest` right after
installing the pinned deps from package.json. This overrode the deliberate
`next` pin with a floating version (non-deterministic builds) and added a
flaky network round-trip that broke the agno smoke build with ECONNRESET on
2026-06-04 (run 26972381969). Every affected starter already pins `next` in
package.json (agno 16.0.7, adk 16.1.1, crewai ^15.5.15, llamaindex 16.0.8,
ms-agent-framework-python 16.0.8, pydantic-ai 16.0.7), so the extra install
is pure harm. Removing it makes builds deterministic and removes the network
fragility.
GitHub Actions expression string literals don't interpret `\n`, so
`toJSON(format('...\n...'))` emits literal `\\n` and Slack renders the
two characters backslash-n instead of a line break. Inject real newlines
via a `fromJSON('"\n"')` placeholder, matching the starter-smoke fix.
Fixes the "all builds failed" and "Showcase Build Failed" alerts in
showcase_build.yml and the multi-line "showcase_validate failed" alert
in showcase_validate.yml. showcase_promote.yml already used the
fromJSON placeholder; the single-line validate alert has no newlines
and was left untouched.
The starter smoke failure alert built its message with literal `\n`
inside a GitHub Actions `format()` call. GHA expression string literals
do not interpret `\n` as an escape, so `format()` emitted the two
characters backslash+n, which `toJSON()` then encoded as `\\n` — Slack
rendered a literal "\n" and the triple-backtick fence as plain text
instead of a line break and a code block.
Inject real newlines via `fromJSON('"\n"')` so `toJSON` encodes them as
a single `\n` Slack honors, and place the code-fence delimiters on their
own lines so the failure summary renders as a proper code block.
PocketBase had no CI build path: `ghcr.io/copilotkit/showcase-pocketbase`
was a stale April `:latest`, and there was no way to ship pb_migrations /
pb_hooks changes without an ad-hoc manual build. Add a `pocketbase` slot to
showcase_build.yml's build matrix, mirroring the harness/aimock entries:
- dispatch_name `showcase-pocketbase`, context `showcase/pocketbase`, its
own Dockerfile, health `/api/health`, railway_id from the SSOT.
- a paths-filter key gated to `showcase/pocketbase/**` so the slot only
rebuilds when PB's own files change (the image is self-contained — no
shared-module copy), not on every showcase push.
- the workflow_dispatch service choice so PB is human-targetable.
Flip the SSOT entry (railway-envs.ts) to `ciBuilt: true` with
`dispatchName: "showcase-pocketbase"` so it is built+pushed (`:sha` +
`:latest`) and joins the default staging-redeploy scope; the build's
redeploy step only touches the matrix-intersect-success set, so PB still
only redeploys when its own files change. Regenerate
railway-envs.generated.json and the showcase_promote.yml service dropdown,
and update the SSOT/redeploy tests that pinned PB as out-of-band
(CI_BUILT_SERVICES 25 -> 26; webhooks stays the only non-CI-built service).
A PB image freshly built from main crash-loops staging PocketBase (502s)
because of two latent defects, both verified by booting the built image
against a real PB 0.22.21 binary on a volume that already has the
collections but has NOT recorded their migrations in `_migrations`.
1. Hook API. `pb_hooks/main.pb.js` registered its CORS middleware via the
bare global `onBeforeServe(...)`, which is undefined in PB 0.22.x JSVM
(only the `$app.onBeforeServe()` Go method exists) — it throws
`ReferenceError: onBeforeServe is not defined` at hook load and crashes
the server. Switch to the documented global `routerUse((next) => (c) =>
…)` entry point. Separately, the per-request closure runs in PB's pooled
goja runtime where top-level helpers/consts are out of scope, so calling
them throws per request and the router returns HTTP 400 on EVERY route;
inline the entire allowlist/env/match logic into the closure to fix that
second regression. Verified: health 200, collection reads 200, the
allowlisted origin is echoed on `Access-Control-Allow-Origin`, a
non-allowlisted origin is not, and OPTIONS preflight returns 204.
2. Migration idempotency. `1777700000_create_baseline.js` and the three
original `1745193*` creators (status, status_history, alert_state) called
`saveCollection(new Collection(...))` unconditionally, so on a volume
where the collection already exists they throw
`UNIQUE constraint failed: _collections.name`, aborting the ENTIRE
migration chain before later migrations (resource_snapshots, future fleet
collections) can run. Guard each with the proven find-or-skip pattern
already used by probe_runs / resource_snapshots, and harden their down
arms to tolerate an already-absent collection. Verified end to end:
deleting those migrations' `_migrations` rows while leaving the
collections in place (the exact staging state), then rebooting the built
image — boots healthy, re-records the migrations cleanly with no UNIQUE
abort and no duplicate collections, and a brand-new collection migration
still applies through the now-clean chain (the pool-fleet path).
Bump canonicalCopilotKitVersion 1.59.2 -> 1.59.4 and pin every
integration's @copilotkit/* to 1.59.4 (locks regenerated). Keeps the
whole showcase on one version instead of letting the langgraph A2UI
demos deviate. Existing per-slug overrides (built-in-agent pkg.pr.new,
ms-agent-harness-dotnet 1.57.2) unchanged.
The A2UI middleware now stamps recovery.debugExposure onto every a2ui_recovery
activity, so the server (covering Python and TS agents alike, since the middleware
is the single emitter) can drive how much retry/error detail the renderer surfaces.
Resolve debugExposure per-activity inside render() with precedence
content > client option > "collapsed" default, and declare it on
RecoveryContentSchema. Adds a test asserting the server value wins.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the hand-rolled render_a2ui/generate_a2ui + a2ui_prompt framing.
Document the two real paths: prebuilt agent (add CopilotKitMiddleware)
and graph agent (wire get_a2ui_tools). injectA2UITool stays the single
on/off switch. Fixes langgraph + deepagents + the generic page;
corrects the streamed op names (createSurface/updateComponents/
updateDataModel).
## Summary
Hardens a set of pre-existing `showcase` promote/`bin/railway` bugs
surfaced during review of the GHCR bearer fix (#5239) and the
#team-showcase notification (#5240). Each is an independent, real
defect; all changes are covered by tests (Ruby `bin/spec` **127 runs / 0
failures**, TS `verify-deploy` drivers **82 passing**, `actionlint`
clean).
## Fixes
1. **`rollback` could roll back to the wrong deployment.**
`find_previous_deployment` selected the second-newest SUCCESS from an
*unsorted* GraphQL result, and assumed the head deploy was always
SUCCESS — so when the latest deploy FAILED/CRASHED (the exact case
rollback is for) it rolled back one good deploy too far. Now sorts by
`createdAt` desc, selects the newest SUCCESS strictly older than the
current head (`sorted.drop(1).find { SUCCESS }`), and **fails loud**
(rather than silently mis-rolling) when the `first: N` window is
saturated with no valid target — telling the operator to pass `--to`.
2. **`env-diff` was dishonest.** It advertised custom-domain comparison
it never performed, and exposed a `--ignore-env-scoped` flag that was
parsed but never read. Now actually diffs `custom_domains` (the snapshot
already carried them), removes the dead flag/helper, and nil-guards
**every** accessor in `diff_services` (`services`, `env_keys`,
`custom_domains`) consistent with the rest of the file.
3. **`verify-prod` could vacuously pass.** Its empty-`succeeded_csv`
branch `exit 0`'d unconditionally. Now fails loud if `promote` reported
success but produced no succeeded set (contract violation), while still
skipping cleanly when promote genuinely failed.
4. **`verify-prod` raced the prod rollout.** It failed instantly when
the just-promoted deploy was still `DEPLOYING` (observed live: promote
succeeded, verify-prod failed ~17s later mid-rollout). `verify-deploy`
now polls in-progress statuses
(`DEPLOYING`/`BUILDING`/`INITIALIZING`/`WAITING`/`QUEUED`/`NEEDS_APPROVAL`)
until terminal (~150s budget), still failing fast on
`FAILED`/`CRASHED`/`REMOVED`.
5. **`npx tsx` ran from the wrong cwd** in the verify jobs (deps
installed in `showcase/scripts`, invoked from repo root → could fetch
`tsx` from the network). Now runs with `working-directory:
showcase/scripts`, matching the resolve/promote jobs.
6. **Slack payloads rendered literal `\n`.** Both promote Slack posts
used `toJSON(format('...\n...'))`, where the literal `\n` survives as
backslash-n (verified live in #team-showcase). Now uses the
`fromJSON('"\n"')` idiom for real line breaks.
## Test plan
- [x] `showcase/bin/spec` — 127 runs, 0 failures (new: rollback
head-FAILED + saturated-window, env-diff custom-domain + nil-guard
cases)
- [x] `showcase/scripts` `tsc --noEmit` clean; `verify-deploy` driver
tests 82 passing (in-progress→SUCCESS, →timeout, fast-fail-on-terminal)
- [x] `actionlint` clean
- [ ] CI green
## Out of scope (separate follow-up)
Review surfaced further pre-existing items intentionally NOT fixed here
(no diff overlap): `run_staging_probe`'s `IO.popen` nests its options
hash inside the argv array (stderr redirect verified working; the
`child_env` hash isn't applied but the probe inherits the parent env in
CI); `image_shape`/`parse_image_ref`/`PinCommand` colon-splitting for
registry-port refs (latent — portless `ghcr.io` only); the deeper
`DEPLOYMENTS_QUERY first:10` truncation beyond the new fail-loud guard;
`succeeded_csv` integrity under a 20-min promote-job timeout; and a few
comment/test-coverage nits.
## Release monorepo v1.59.4
**Scope:** `monorepo` | **Bump:** `patch`
---
### How this release process works
1. **This PR was created automatically** by the "release / create-pr"
workflow.
It bumped the `monorepo` packages to `1.59.4`
and generated AI-enhanced release notes.
2. **CI runs on this PR** — the full test suite (unit tests, lint, type
checks, build)
must pass before merging. This is the review gate.
3. **Review the release notes** in `release-notes.md` in this PR.
If a Notion draft was created, you can edit the release notes there
before merging.
4. **When this PR is merged**, the `release / publish` workflow
automatically:
- Builds all packages
- Publishes the `monorepo` packages to npm at version `1.59.4`
- Creates git tag `monorepo/v1.59.4`
- Creates a GitHub Release with the final release notes
### Before merging
- [ ] CI is green (tests, lint, types, build)
- [ ] Version bumps look correct
- [ ] Release notes are accurate (edit in Notion if a draft was created)
---
> **Do not merge until CI is fully green.** The full test suite runs
automatically on this PR.
## Summary
Three-tier wiring on the CopilotKit side, mirroring `useThreads`, to
surface user UI signals into CopilotKit Intelligence's self-learning
loop. Companion change in `CopilotKit/Intelligence` (PR #192) lands the
connector + schema.
- **Runtime client** — `CopilotKitIntelligence.recordUserAction(...)`
hits the idempotent platform endpoint
`${apiUrl}/connector/user-actions/record/:clientEventId`. Auth via the
deployment-level Intel API key (Bearer); the Intel key never reaches the
browser.
- **Runtime handler** — `handleRecordUserAction` resolves the Intel user
via `resolveIntelligenceUser`, forwards to the platform client, returns
`{ id, duplicate }`.
- **Fetch router** — `POST /user-actions` wired in
(`user-actions/record` `RouteInfo` variant + dispatch case).
- **React hook** — `useRecordUserAction()` and
`useRecordUserActionInCurrentThread()` in `@copilotkit/react-core/v2`.
Auto-generates a UUID `clientEventId` per call so retries are idempotent
by default. Throws when `runtimeUrl` is absent.
Linear: CPK-7587
## Test plan
- [ ] CI green on this PR
- [ ] Companion Intelligence PR #192 merged or coordinated
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Prebuilt agents get dynamic A2UI with no extra wiring — adding the
middleware is enough. When the frontend registers an A2UI catalog
(surfaced by the runtime into state["ag-ui"].a2ui_schema), the
middleware infers the agent's own model, advertises the generate_a2ui
tool in the model-call hook, and executes it in the tool-call hook. No
catalog → the tool is never advertised.
Covers both @copilotkit/sdk-js and the copilotkit Python SDK. Bumps the
A2UI tool-factory dependency to where get_a2ui_tools ships
(@ag-ui/langgraph 0.0.35, ag-ui-langgraph >=0.0.37).
find_previous_deployment returned the second-newest SUCCESS regardless of the
head deploy's status. When the head deploy FAILED/CRASHED (precisely when
rollback is invoked) the newest SUCCESS is the last-known-good target, but the
old code skipped it and rolled back one good deploy too far. Now select the
newest SUCCESS strictly older than the current head (drop head, first SUCCESS
in the remainder), and fail loud when the fetched window is saturated and no
target is found — directing the operator to pass --to explicitly rather than
silently returning nil.
Also guard the one unguarded accessor in EnvDiffCommand#diff_services with
`|| []` so a snapshot lacking "services" hits the documented exit-2 path
instead of a raw NoMethodError.
Renumber the ivar-lint allowlist to track the added lines in bin/railway.
verify-prod commonly runs seconds after a promote pins a new image digest,
while Railway is still rolling the container out. checkDeploymentSuccess (in
verify-deploy.drivers.baseline.ts — the SSOT for the deployment-SUCCESS gate
shared by every driver) treated a transient in-progress status as a hard FAIL:
promote run 26966193624's predecessor pinned the docs digest, then verify-prod
fired ~17s later and FAILED with status="DEPLOYING" — a race, not a failure.
Now the deployment-status check polls when the latest deployment sits in any
non-terminal Railway status (QUEUED/BUILDING/INITIALIZING/DEPLOYING/WAITING/
NEEDS_APPROVAL), re-querying every 5s up to a 150s budget until it reaches a
terminal state, then asserts SUCCESS. Terminal-failure statuses (FAILED/
CRASHED/REMOVED/any non-SUCCESS-non-in-progress) still fail FAST with the
original error-string shape — no waiting. Infra/contract errors (network,
GraphQL errors[], missing edge) also fail fast.
The poll loop is fully seamed for tests (injectable sleep/now/budget); the
signature stays backward-compatible (trailing optional pollOpts). probeBaseline
forwards an optional deployPoll through, and both --env staging and --env prod
go through the identical code path, so staging behavior is unchanged.
Red-green tests: in-progress-then-SUCCESS passes (and polls), in-progress-until-
timeout fails with "still in progress", terminal FAILED fails fast with zero
sleeps.
Three pre-existing showcase_promote.yml issues surfaced by code review:
1. verify-prod's empty-SERVICES_CSV branch exited 0 vacuously, relying on the
unenforced assumption "promote already failed if the succeeded set is
empty". Pass PROMOTE_RESULT into the step env and fail loud on contract
violation: if promote reported success yet emitted no CSV, emit
"::error::promote reported success but succeeded_csv is empty — contract
violation" and exit 1. The genuine all-failed case keeps the ::notice:: +
exit 0.
2. Both verify-deploy.ts invocations (verify-staging-precondition, verify-prod)
ran `npm ci` in showcase/scripts but then `npx tsx showcase/scripts/...`
from the repo root, so npx could network-fetch tsx instead of using the
local install. Give those steps working-directory: showcase/scripts and the
relative path verify-deploy.ts, matching how resolve-targets/promote run
emit-railway-envs-json.ts. verify-deploy.ts takes no cwd-relative file args
(only --env/--services) and imports ./railway-envs relative to its own file,
so the cwd change is safe.
3. The notify job's Slack payloads embedded literal '\n' inside
toJSON(format('...\n...')). GitHub Actions expression string literals do not
interpret backslash escapes, so the '\n' survived toJSON as the two chars
\\n and Slack rendered it verbatim as "\n" instead of a line break.
EMPIRICALLY CONFIRMED against the live #team-showcase webhook: an existing
production post from a real promote run showed literal "\n", and a corrected
payload (real LF) rendered proper line breaks. Fixed BOTH posting steps by
injecting newlines via fromJSON('"\n"') (a real LF char) as a format()
argument instead of a literal '\n' in the template.
Two pre-existing bin/railway bugs surfaced by code review:
1. RollbackCommand#find_previous_deployment selected "the second SUCCESS in
reverse-chronological order" but never sorted the deployments — it trusted
Railway's arbitrary GraphQL connection order, so the wrong deploy could be
chosen as the rollback target. Now sort by createdAt descending before
selecting, mirroring the sibling fetch_latest_staging_deployments which
documents and guards against this exact hazard.
2. EnvDiffCommand advertised behavior it never implemented:
- The banner claimed it compares "custom domains", but run() only diffed
digest/startCommand/env_keys. custom_domains was captured in the snapshot
yet never compared. FIX: extract a pure diff_services helper and add a
custom-domain set comparison (both directions).
- --ignore-env-scoped was parsed into options[:ignore_env_scoped] but never
read, and the env_scoped? helper (+ ENV_SCOPED_URL_MARKERS) it relied on
was defined but never called. The flag had nothing meaningful to act on:
env-diff only compares env-var KEY names (snapshots never carry values),
and custom domains are compared as exact sets (silently dropping
"env-scoped" domains would defeat the comparison). DECISION: remove the
dead flag, the unused env_scoped? helper, and the now-orphan
ENV_SCOPED_URL_MARKERS constant rather than leave false advertising.
Tests: TDD red->green for both. test_rollback_sort feeds deployments in
non-chronological edge order (with a newer FAILED deploy as a decoy) and
asserts the true second-newest SUCCESS is returned. test_env_diff exercises
diff_services directly for identical and differing custom_domains. The
snapshot-ivar lint allowlist is renumbered (numbers only) to track the line
shifts from removing the dead code and adding the rollback comment.
## What does this PR do?
Bumps the default Gemini model across the Google ADK showcase from
`gemini-2.5-flash` to `gemini-3.1-flash-lite` — the closest GA model in
the Gemini 3.x line. The Gemini 2.5 family is slated for deprecation in
June 2026, so this gets ahead of the cutoff.
**Files touched (14):**
- Agent factories — `DEFAULT_MODEL`, `_DEFAULT_A2UI_MODEL`, `_SUB_MODEL`
in `shared_chat.py`, `main.py`, `beautiful_chat_agent.py`,
`subagents_agent.py`
- Prose references — docstrings in `shared_chat.py`,
`multimodal_agent.py`, `tool_rendering_reasoning_chain_agent.py`,
`registry.py`; comment in `entrypoint.sh`
- Test fixtures — `test_generate_a2ui.py` default-model assertion,
`test_stop_on_terminal_text.py` docstring
- Docs/config — `manifest.yaml` reasoning-demo description, QA prereqs
in `multimodal.md`, `headless-complete.md`, `headless-simple.md`
### Why Flash-Lite and not Flash?
There's no standalone `gemini-3.1-flash` identifier. The 3.1 Flash
family only ships as Flash-Lite (GA), Flash Live (preview), Flash Image
(preview), or Flash TTS (preview). Flash-Lite is the only stable choice;
the non-Lite `gemini-3-flash-preview` is still preview-only.
### Worth verifying empirically
A few behaviors that may differ from 2.5-flash and warrant a smoke test
before merging:
- **Thinking mode** (`build_thinking_chat_agent`, used by
`reasoning-custom` and `tool-rendering-reasoning-chain`) — Gemini 3.x
uses `thinkingLevel` instead of `thinkingBudget`, and Lite variants
historically have reduced reasoning vs full Flash/Pro
- **Forced tool-calling**
(`ToolConfig.function_calling_config.mode="ANY"` in A2UI planner)
- **Multimodal** image + PDF (Flash-Lite docs explicitly support
text/image/video/audio/PDF, so this should work)
- **`stop_on_terminal_text` loop-stopper** — the quirk was originally
observed on 2.5-flash; whether Flash-Lite still needs the guard is open,
but leaving it in is harmless
## Related PRs and Issues
- N/A — proactive model bump ahead of Gemini 2.5 deprecation
## Checklist
- [x] I have read the [Contribution
Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md)
- [x] If the PR changes or adds functionality, I have updated the
relevant documentation
- [x] "Allow edits by maintainers" is checked
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Red-green coverage for the writer-hardening fixes:
- in-flight cap DROP emits dropped-over-inflight and the counter does not leak
- a hung create times out, counts as a failure, and escalates after the threshold
- once writes are timing out, concurrent drops contribute to the escalation
- a drop while a write succeeds (healthy backpressure) does NOT escalate
- a no-progress deleteByFilter over cap backs off (gated by interval, not per insert)
- a cold start far over cap prunes in bounded batches (<= 200), never a thousands-row list/filter
- a hung Slack webhook is aborted at the timeout so the serialized health-signal chain isn't stalled
Three durable-logging robustness fixes in the resource-snapshot writer:
- PRUNE BATCH CAP + NO-PROGRESS BACKOFF: a cold start over a large backlog
(or lowering RESOURCE_SNAPSHOT_MAX_ROWS on a full collection) made
`surplus` thousands large, used directly as a `perPage` and a
one-clause-per-row id-delete filter — a giant request/URL that PB
rejects (400/414) so the prune never converged. Cap each sweep to
PRUNE_BATCH_MAX (200) oldest rows and let knownOverCap drain the rest
across inserts. When a sweep makes NO structural progress while still
over cap (PB delete-rule 403, filter 400, or 0 matched), back off via a
suppression window so the retry is gated to once-per-interval instead of
re-arming knownOverCap and re-running the failing sweep on every insert.
- DROP→ESCALATION SEAM: an in-flight-cap DROP did not count toward the
consecutive-failure escalation, so a 100%-broken PB whose writes hang
could keep new snapshots on the DROP path and delay/dodge the loud
`write-failing-systematically` error. A drop now contributes to the
failure signal when writes are already failing (consecutiveFailures > 0),
with no false positive on healthy backpressure (a drop while writes
succeed does not escalate).
- ORPHAN-CREATE REJECTION: on a write timeout the race rejects but the
orphaned `pb.create` keeps running; a later rejection from a client that
ignores the AbortSignal would be unhandled. Attach a no-op catch.
formatGauges had no production callers after the fix round removed the
gauge-string logging; only its own test referenced it. Delete the dead
function and the describe("formatGauges") block.
Jordan Ritter