When the PR-changed file collection happens to be all files oxfmt
rejects internally (it does so for tsconfig-style JSONC, certain meta
configs, and other heuristic-filtered shapes — observed locally with
docs/**/meta.json and docs/lib/*.ts), oxfmt exits with "Expected at
least one target file" and fails the check job even though there is
nothing to format. The flag turns that no-op case into a clean exit,
matching the existing carve-out the workflow already documents for
lockfiles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move untrusted PR event data (title, head ref, base ref, SHAs) from
inline ${{ }} interpolation in shell run: blocks to env: blocks,
referencing them as shell variables instead. This prevents arbitrary
code execution via crafted PR titles or branch names.
Affected workflows:
- test_smoke-starter.yml: PR title, user login, head SHA
- publish-release.yml: PR head ref (branch name)
- static_quality.yml: PR base ref, base SHA, head SHA
Introduce machinery for keeping examples/integrations/* demos aligned to a
single north-star (langgraph-python). Built first so the upcoming
langgraph-js and langgraph-fastapi alignment PRs have a mechanical baseline
to work against instead of manual copy-paste.
- examples/integrations/_parity/manifest.json declares verbatim files,
tracked package.json keys, and expected agent surface (tool names,
state keys) per instance plus allowed-divergence lists.
- _parity/sync.ts copies verbatim files + rewrites tracked package.json
keys from north-star to a target instance. Dry-run supported.
- _parity/verify.ts diffs each instance vs north-star and exits non-zero
on unexpected drift. Checks verbatim content, tracked keys, canonical
prompt equality, and agent-surface grep-level presence.
- Canonical prompt at _parity/canonical/PROMPT.md — synced into each
instance's agent/PROMPT.md on parity:sync.
- Root package.json: pnpm parity:sync, parity:verify, parity:check.
- CI: .github/workflows/integrations_parity.yml runs parity:check on PRs
touching examples/integrations/**.
- Skill: .claude/skills/copilotkit-demo-parity/SKILL.md teaches agents
how to drive sync/verify and handle manual-merge zones (agent code,
api route, Dockerfile).
Does NOT touch the existing instance demos yet. Those alignment commits
follow in the same PR.
The eval workflow was missing --ci, causing it to try Docker Compose
lifecycle in CI (which fails because there's no .env file). The --ci
flag skips Docker and assumes services are already running or uses
native execution.
Add GET /trigger/eval route to eval-webhook with HMAC-signed URLs.
The showcase_eval_check.yml workflow now posts a bot comment with a
clickable "Run Evaluation" link. Clicking triggers the eval and
redirects back to the PR. The link is signed so it can't be forged.
Add --ci flag to eval orchestrator that skips Docker lifecycle and
assumes services are already running. Add ci-native-eval.sh helper
that installs deps, starts next dev + agent servers natively, health-
waits, then runs showcase eval --ci. Fix on-demand E2E workflow with
langgraph-python support and agent-type detection.
New showcase_eval_check.yml creates a Check Run with "Run Showcase
Eval" action button on every PR. Modified showcase_eval.yml adds
workflow_dispatch trigger with dispatch-gate job, Check Run update
in post-result, and devops bot token for Checks API calls.
Hono web server that receives check_run.requested_action webhooks from
GitHub, authenticates as the devops bot, updates the Check Run to
in_progress, and dispatches showcase_eval.yml via workflow_dispatch.
Includes GHCR build workflow and pnpm workspace registration.
Picks up the router fix from CopilotKit/aimock#148 — `toolCallId` matchers
now only fire when the tool message is the *last* message in the request,
preventing stale tool_call_ids from history shadowing `userMessage`
matchers on new user turns.
Surfaced as: in beautiful-chat, clicking a second suggestion replayed the
prior chart's "Pie chart rendered above…" content fixture instead of
producing a new tool call. Once Railway rebuilds `ghcr.io/copilotkit/aimock:latest`
and restarts the service, demos will pick up the fix automatically.
- Refresh `pnpm-lock.yaml` resolutions (workspace `@copilotkit/runtime` devDep)
- Refresh `showcase/scripts/package-lock.json` to 1.16.4
- Bump the floor in `test_e2e-showcase-on-demand.yml` from `^1.14.3` → `^1.16.4`
so the `/test-aimock` PR-comment workflow always installs a build that
contains the fix
The decoupled build workflow (PR #4471) removed the Railway deploy
trigger, assuming environmentPatchCommit auto-update would handle
deploys. Not all services have auto-update configured, so GHCR images
were pushed but Railway never pulled them. Restore the explicit
serviceInstanceRedeploy call after each GHCR push.
The old "Showcase: Build & Deploy" workflow used a single concurrency group
that cancelled in-flight builds on every push to main. When multiple PRs
merged in quick succession, most service builds got cancelled and never
deployed.
Split into two workflows:
1. showcase_build.yml ("Showcase: Build & Push") - triggered on push to main,
builds Docker images and pushes to GHCR. Has NO concurrency group so every
run completes. Railway auto-update picks up the new :latest tag.
2. showcase_deploy.yml ("Showcase: Verify Deploy") - triggered by workflow_run
from the build workflow. Polls Railway to verify each service picked up the
new image and is healthy. Uses cancel-in-progress since verification is
idempotent. Posts results to showcase-harness via webhook.
Also updates showcase_capture-previews.yml to trigger from the renamed build
workflow.
## Summary
- **Restores the eval system** accidentally deleted by PR #4449 ("D5
all-green" session removed it as "superseded by D5 depth probes" while
unaware #4448 had just merged)
- **Per-test JSON granularity** — eval runner now injects
`--reporter=list,json` + `PLAYWRIGHT_JSON_OUTPUT_NAME` when spawning
test subprocesses, reads JSON file after exit for per-test results,
falls back to exit-code when absent
- **Slug-keyed baselines** — `transformHarnessResponse` restructured to
key by integration slug (via `summary.services[]`) instead of probe ID,
matching `collectResults` format so `computeRegressions` works against
harness-prod baselines
## Context
PR #4448 added the eval system (5 CR rounds, 8 bugs fixed, 35+ agent
reviews). PR #4449 (a concurrent session) deleted it. This PR restores
the CR-converged code and adds two follow-up improvements from the eval
plan.
## Test plan
- [x] `npx nx run @copilotkit/showcase-harness:test` — 1375 tests pass
(70 files)
- [x] All 8 CR fixes from #4448 verified present in restored code
- [ ] CI green
Production showcase-aimock was running a week-old image because fixture
file changes in showcase/aimock/ did not trigger a CI rebuild. This adds
showcase-aimock to the Build & Deploy workflow matrix so it auto-deploys
on merge, creates a thin Dockerfile that bakes fixture files into the
image, documents the local-vs-production parity requirement in
docker-compose.local.yml, and adds an aimock fixture deployment section
to the RUNBOOK.
PR #4449 removed the eval tier system while unaware #4448 had just
merged. Restoring orchestrator, matrix, scope, config, GHA workflow,
and CLI wiring with all 8 CR fixes intact.
The format job in static_quality.yml globs *.json (and *.yaml) to feed
oxfmt --write on PR-changed files. Lockfiles match those globs but oxfmt
rejects them internally (likely a size threshold or filename heuristic),
so lockfile-only PRs failed with 'Expected at least one target file' and
exit 123 even though the count check thought there were 18 files to format.
Exclude package-lock.json, pnpm-lock.yaml, and yarn.lock from the glob.
Lockfiles are auto-generated and should never be hand-formatted regardless,
so this is corrective — small JSON/YAML config files (tsconfig.json,
package.json, *.yml workflow files) still get formatted as before.
Caused PR #4438's format check to fail; surfaced again on this PR which
only touches lockfiles.
The audit checks repository == null on the Packages API to detect
unlinked packages. But some packages (showcase-pocketbase) have Actions
access configured manually via the UI, so pushes work fine despite
repository being null. There is no API to detect manual Actions access,
so this adds an explicit VERIFIED_ACCESS allowlist that excludes
known-good packages from the alert.
Single-version 3.12 job hides regressions on floor (3.10) and newly
supported ceiling (3.14). Run all five versions with fail-fast off
so partial breakage is visible.
## Summary
Adds a checklist item to the PR template reminding contributors to keep
"Allow edits by maintainers" checked. This lets us push fixes directly
to contributor PRs instead of going back and forth in review comments.
## Test plan
- [x] Template renders correctly on new PR creation
Remove all showcase-starter-* services from the CI build matrix in
showcase_deploy.yml. These starter demos were decommissioned in PR
#4378 (code removed) and confirmed dormant with zero traffic. Removes
entries from workflow_dispatch options, paths-filter definitions, and
the ALL_SERVICES JSON array.
- Update collision-avoidance comment in showcase_deploy.yml to remove
starter-specific examples (service/starter collision no longer possible)
- Fix showcase_keep-alive.yml description: pings showcase services, not starters
- Remove "starter" from catalog-types.ts manifestation union type
- Remove dead starter cell skip logic from cell-matrix.tsx cellIndex builder
- Update depth-utils.ts comment (defensive null guard, not starter-specific)
- Remove starter-specific test cases from depth-utils and cell-matrix tests
The showcase-starter-* Railway services are being deprovisioned after
the packages/starters integration merge. Delete the dedicated smoke
workflow (test_smoke-starter-deployed.yml) and the "Deployed Starters"
describe block from integration-smoke.spec.ts to prevent 17 false-red
alerts per 6-hour cron cycle.
The extension is a leaf node in the monorepo dependency graph (nothing
depends on it) and has its own independent release cycle. The standalone
repo at CopilotKit/vscode-extension has full git history, working CI,
verified OIDC publishing, and branch protection.
## Summary
Adds `.github/workflows/ghcr_unlinked_packages.yml` — a scheduled
GitHub Actions workflow that audits all CopilotKit org container
packages daily and Slack-alerts when any are unlinked from a source
repository (`repository: null` on the GHCR API).
## Why
When a GHCR container package is unlinked from a repo, workflow
builds in `CopilotKit/CopilotKit` get `403 Forbidden` on push to GHCR
— the workflow `GITHUB_TOKEN` only has package-write permissions
when the package is linked to the actor's repo. We hit this twice in
quick succession:
- `showcase-ops` — caught manually after a failed deploy
- `showcase-pocketbase` — caught by a preemptive scan
There is **no GitHub API to programmatically link a package to a
repo** — it is UI-only. So the only way to prevent future surprises
is to detect drift early via a scheduled audit + Slack alert.
## Behavior
- Runs daily at 14:00 UTC, plus `workflow_dispatch` for ad-hoc runs.
- Lists every container package in the `CopilotKit` org via
`gh api /orgs/CopilotKit/packages?package_type=container`.
- Filters for `repository == null`.
- If any are unlinked, posts a Slack message that includes the count,
a bulleted list with deep links to each package's settings page,
the exact UI fix steps, and a footer noting the failure mode.
- Exits 0 in all non-error cases. The Slack message IS the alert;
failing the workflow on drift would create noisy red CI checks.
## Required setup before this workflow can fire
Two new repo secrets must be added:
1. **`ORG_READ_PACKAGES_PAT`** — a fine-grained PAT with
`read:packages` scope, org-scoped to `CopilotKit`. The default
`secrets.GITHUB_TOKEN` does NOT have org-package-list scope. The
workflow fails loudly if this is missing.
2. **`SLACK_WEBHOOK_GHCR_DRIFT`** — a CopilotKit-internal Slack
incoming-webhook URL for an alerts channel. If missing, the audit
still runs and logs a warning; only the Slack post is skipped.
Without these secrets the workflow will either fail loudly (PAT) or
log-only (webhook). It will not silently mask drift.
## Test plan
- [ ] Add `ORG_READ_PACKAGES_PAT` and `SLACK_WEBHOOK_GHCR_DRIFT` repo
secrets.
- [ ] Trigger via `workflow_dispatch` on `main` post-merge.
- [ ] Confirm the audit lists packages and reports the unlinked count.
- [ ] If drift exists, confirm Slack receives the alert with working
deep links.
- [ ] If no drift, confirm workflow exits 0 with a "no drift" log line
and skips the Slack post.
Adds a scheduled GitHub Actions workflow that detects when CopilotKit
org container packages drift into an unlinked state (`repository: null`
on the GHCR API) and posts a Slack alert with deep links to the UI fix.
This drift breaks future workflow builds with `403 Forbidden` on push
to GHCR — the workflow `GITHUB_TOKEN` only has package-write
permissions when the package is linked to the actor's repo. We hit
this twice in quick succession: `showcase-ops` (caught manually after
a failed deploy) and `showcase-pocketbase` (caught by a preemptive
scan). There is no GitHub API to programmatically link a package to
a repo — it is UI-only — so the only way to prevent future surprises
is to detect drift early.
Schedule: daily at 14:00 UTC, plus `workflow_dispatch` for ad-hoc
runs. Exits 0 on drift (the Slack message IS the alert; failing the
workflow on a schedule would create noisy red CI checks).
Requires two new repo secrets:
- `ORG_READ_PACKAGES_PAT` (read:packages, org-scoped to CopilotKit)
- `SLACK_WEBHOOK_GHCR_DRIFT` (CopilotKit-internal alerts webhook)
Workflow fails loudly if the PAT is missing; logs a warning and
continues if the webhook is missing.
CR R1 follow-ups on top of the /api/ops proxy fix.
Bucket (a) — must-fix:
- Dockerfile: declare ARG/ENV OPS_BASE_URL in the builder stage. next.config.ts
evaluates rewrites() at build time and throws if OPS_BASE_URL is unset, which
aborted `next build` in CI. Mirrors the existing NEXT_PUBLIC_SHELL_URL /
NEXT_PUBLIC_POCKETBASE_URL pattern.
- showcase_deploy.yml: pipe OPS_BASE_URL through to docker build for the
shell-dashboard matrix entry, defaulting to the production
showcase-ops-production.up.railway.app URL.
- next.config.ts: strip trailing slashes from OPS_BASE_URL before constructing
the rewrite destination, matching the same normalization in
src/lib/ops-api.ts:resolveBaseUrl so server-side rewrite and client-side
fetch agree on the URL shape.
- src/lib/ops-api.ts: treat empty / whitespace NEXT_PUBLIC_OPS_BASE_URL as
"no override". `??` only short-circuits on null/undefined, so an env var set
to "" silently produced baseUrl="" and URLs of the form "/probes" with no
/api/ops prefix.
- use-probes.integration.test.tsx: snapshot+restore process.env.NEXT_PUBLIC_OPS_BASE_URL
in beforeEach/afterEach so tests never leak env state. Strengthen the proxy
contract assertions: lock toHaveBeenCalledTimes(1), assert method=GET,
cache=no-store, accept JSON header, and signal is an AbortSignal. Tighten
the 404 regression to assert the canonical ensureOk message shape so a
refactor that changes the format trips the test.
Bucket (b) — applied since the diff stayed focused:
- triggerProbe: add cache:"no-store" for parity with the GET fetches.
- fetchProbeDetail / triggerProbe: throw early when id is empty so callers
get a clean error instead of a request to /probes//... .
- ensureOk: bump body-truncation cap from 200 to 500 chars and append a
`[truncated, N bytes total]` marker so operators can see they're missing
tail bytes when the server returns a long HTML/stack-trace body.
- ops-api.ts: drop dev-loop review-cycle tag prefixes (R2-C.3, R3-C, R3-D.1)
from comments. Keep the actual rationale.
- ops-api.ts header docstring: clarify that NEXT_PUBLIC_OPS_BASE_URL is read
live at runtime in this codebase (SSR + tests), not just statically inlined
into the client bundle.
Verified:
- Tests: vitest run — 26 files, 293 passed, 1 skipped (no test count change).
- Typecheck: tsc --noEmit clean.
- Lint + format: oxlint + oxfmt clean on changed files.
- Local Docker build: `docker build --build-arg OPS_BASE_URL=https://...` succeeds.
Without --build-arg the build fails with "OPS_BASE_URL must be set" as
expected, confirming the fix is load-bearing.
showcase-ops was excluded from .github/workflows/showcase_deploy.yml, so
commits touching showcase/ops/** never produced a fresh GHCR image. PR
#4293 (Status tab + /api/probes route) merged to main on 2026-04-26 but
no rebuild fired — the deployed Railway image is stale and /api/probes
404s in production.
Adding showcase-ops as a first-class matrix entry:
- dispatch_name: showcase-ops (workflow_dispatch option + filter_key)
- paths-filter: showcase/ops/**, plus shared/scripts/manifests
(showcase-ops's Dockerfile bundles all four into the runtime image
via build-stage COPY + generate-registry.ts)
- context: '.' (repo root) so the Dockerfile can COPY from
pnpm-workspace.yaml + packages/ + showcase/{ops,shared,packages,scripts}
- dockerfile: showcase/ops/Dockerfile
- image: showcase-ops -> ghcr.io/copilotkit/showcase-ops:latest
- railway_id: 3a14bfed-0537-4d71-897b-7c593dca161d
- health_path: /health (matches Dockerfile HEALTHCHECK + Hono route)
- timeout: 20 (heavier build than shells: pnpm deploy + chromium
install via playwright --with-deps)
- lfs: false (no Git LFS assets in showcase/ops)
- linux/amd64 platform inherited from existing build step (Depot)
Resulting matrix: 39 services (was 38). dispatch_name uniqueness +
JSON validity verified locally; actionlint/yamllint surface only
pre-existing findings on the workflow.
oxfmt 0.36 supports .md but not .mdx — including .mdx in the file-list
glob causes MDX-only PRs to fail with "Expected at least one target
file" because oxfmt drops every input as an unknown target and then
errors on the empty target set. Mixed PRs (.mdx + .tsx/.json/etc) pass
because the non-MDX files keep the target set non-empty, which is why
this has only surfaced now on a shell-docs-only sync PR.
Removing .mdx from the glob lets MDX-only PRs hit the existing count=0
skip path and pass cleanly. Add .mdx back when oxfmt ships MDX support.
The test at integration-smoke.spec.ts:21 imports registry.json, which
is gitignored (generated at build time). After PR #4236 removed it
from tracking, every CI run fails with "Cannot find module
'../../shell/src/data/registry.json'" — producing the repeating
"Starter Deployed Smoke Test Failed — 0 failure(s) — job-level error"
Slack alerts in #oss-alerts.
Adds a generate-registry step before the Playwright test run.
Three fixes batched to minimize PR churn:
1. Dockerfile: add ARG NEXT_PUBLIC_POCKETBASE_URL so the PB URL gets
baked into the Next.js bundle at build time. Without this, pb.ts
resolves to the sentinel URL and the dashboard shows "unavailable"
on every tab. Pre-existing bug exposed by fresh deploys.
2. showcase_deploy.yml: pass NEXT_PUBLIC_POCKETBASE_URL and
NEXT_PUBLIC_SHELL_URL as build args for the shell-dashboard service
in the CI matrix. Neither was ever passed before.
3. cell-matrix.tsx + parity-matrix.tsx: flatten nested table pattern
that caused column misalignment. Category rows used colSpan with
an inner <table> whose columns floated independently of the header.
Replaced with useCollapsible hook + flat sibling <tr> rows.
Also regenerates package-lock.json for the plugin-react downgrade
from PR #4241 (npm ci was failing in Docker).
Local Docker build verified with --build-arg for both NEXT_PUBLIC vars.
- deploy workflow: add shared/scripts/manifest paths to shell-dashboard
and shell-docs filters (previously triggered implicitly by committed
JSON diffs in those directories)
- capture-previews: add generate-registry step before capture; use
git add -f for the gitignored registry.json
- e2e smoke test: document generator dependency in import comment
Two bugs in the earlier version, both surfaced once main advanced and
was merged into the branch:
- `fetch-depth: ${{ ... && 0 || 1 }}` evaluated to `1` on PRs because
the short-circuit treats `0` as falsy, so the base SHA was missing
locally and `git diff` exited 128.
- Diffing against the PR's stored `base.sha` includes every file main
touched since the PR opened once main is merged into the branch, which
defeats the whole point of this change. Diff against the current tip
of the base branch instead.
Jordan Ritter