Replaces full pnpm install with standalone binary installs. Adds ruff
format for Python files in both PR and push-to-main paths. Reduces
format job from ~8min to ~30s.
The build/publish split passes the workspace via artifact, but the
credential helper from actions/checkout doesn't survive the transfer.
Add url.insteadOf to inject GITHUB_TOKEN for git push without
collapsing the security boundary.
## Summary
Client-side telemetry on `docs.showcase.copilotkit.ai` was silent. The
shell-docs Dockerfile and `showcase_build.yml` workflow never plumbed
the `NEXT_PUBLIC_*` analytics keys through to `next build`, so the
client JS chunks shipped with empty strings (verified by grepping the
live bundle: `let l = i(95704).env.NEXT_PUBLIC_POSTHOG_KEY` — a runtime
lookup with no inlined value).
Railway runtime env doesn't reach the Docker build phase, so server-side
reads (middleware `POSTHOG_KEY`, server-component canonical URLs) worked
but client-side reads (posthog-js init, RB2B, Scarf, Reo, GA) silently
no-op'd in the browser.
## Changes
- **`showcase/shell-docs/Dockerfile`** — declare `ARG` + `ENV` for
`NEXT_PUBLIC_POSTHOG_KEY`, `NEXT_PUBLIC_RB2B_ID`,
`NEXT_PUBLIC_SCARF_PIXEL_ID`, `NEXT_PUBLIC_REO_KEY`,
`NEXT_PUBLIC_GOOGLE_ANALYTICS_TRACKING_ID` in the builder stage so they
reach `next build`.
- **`.github/workflows/showcase_build.yml`** — add
`build_args_analytics: "yes"` flag to the shell-docs matrix entry;
extend the `Prepare build args` step to emit the five `NEXT_PUBLIC_*`
`--build-arg`s when the flag is set, sourcing values from repo secrets.
Mirrors the existing shell-dashboard pattern (`build_args_pb_url` /
`build_args_shell_url` / `build_args_ops_url`).
## Secrets
Existing repo secret reused: `POSTHOG_PROJECT_KEY`.
New repo secrets required (configured separately in repo settings before
this lands):
- `RB2B_ID`
- `SCARF_PIXEL_ID`
- `REO_PROJECT_KEY`
- `GOOGLE_ANALYTICS_TRACKING_ID`
## Out of scope (intentionally)
- `NEXT_PUBLIC_BASE_URL` is already correctly working via Railway
runtime env (canonical links render with `https://docs.copilotkit.ai`) —
left alone.
- Server-side `POSTHOG_KEY` (no `NEXT_PUBLIC_` prefix) stays on Railway
runtime env; middleware reads it at Edge Runtime.
## Test plan
- [ ] Next build of shell-docs succeeds with new ARGs in scope
- [ ] After deploy, search the live bundle on
`docs.showcase.copilotkit.ai` for the literal `phc_` prefix — must be
present (not `process.env.NEXT_PUBLIC_POSTHOG_KEY` runtime lookup)
- [ ] PostHog Live Events shows `$pageview` (client) and `$autocapture`
arriving from staging
- [ ] RB2B / Scarf / Reo / GA dashboards show events from staging
- [ ] Server-side `seo_redirect` + `docs_pageview` continue firing (no
regression)
The previous commit used `NEXT_PUBLIC_RB2B_ID` based on a stale entry
in the cutover plan doc, but `app/layout.tsx:86` reads
`NEXT_PUBLIC_REB2B_KEY`. Without this fix the build-arg would be
piped under the wrong name and the REB2B Script tag would still not
render.
Client-side telemetry on docs.showcase.copilotkit.ai was silent: the
shell-docs Dockerfile and Showcase Build & Push workflow never plumbed
NEXT_PUBLIC_POSTHOG_KEY / RB2B_ID / SCARF_PIXEL_ID / REO_KEY /
GOOGLE_ANALYTICS_TRACKING_ID through to `next build`. Railway runtime
env doesn't reach the Docker build phase, so the client JS chunks
shipped with empty strings — posthog-js.init etc. silently no-op'd in
the browser.
Mirrors the shell-dashboard pattern: matrix flag triggers the args
block; values come from repo secrets (POSTHOG_PROJECT_KEY already
existed; RB2B_ID, SCARF_PIXEL_ID, REO_PROJECT_KEY,
GOOGLE_ANALYTICS_TRACKING_ID added separately in repo settings).
Server-side telemetry (middleware seo_redirect, docs_pageview) was
unaffected — it reads POSTHOG_KEY at Edge Runtime, which Railway
runtime env satisfies.
Runs the same Depot Docker build as the post-merge pipeline but with
push: false, catching Dockerfile-specific failures (missing deps,
broken layers) before they land on main.
Pre-build lint step that fails if any build config file exists that
is not on the checked-in allowlist. Catches TanStack-style attacks
where a malicious vite_setup.mjs is auto-discovered during build.
Runs before pnpm install — zero npm dependencies.
Build and publish now run in isolated GitHub Actions jobs. NPM_TOKEN
is only available in the publish job, preventing /proc/mem token
extraction from build-time code.
When the PR-changed file collection happens to be all files oxfmt
rejects internally (it does so for tsconfig-style JSONC, certain meta
configs, and other heuristic-filtered shapes — observed locally with
docs/**/meta.json and docs/lib/*.ts), oxfmt exits with "Expected at
least one target file" and fails the check job even though there is
nothing to format. The flag turns that no-op case into a clean exit,
matching the existing carve-out the workflow already documents for
lockfiles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move untrusted PR event data (title, head ref, base ref, SHAs) from
inline ${{ }} interpolation in shell run: blocks to env: blocks,
referencing them as shell variables instead. This prevents arbitrary
code execution via crafted PR titles or branch names.
Affected workflows:
- test_smoke-starter.yml: PR title, user login, head SHA
- publish-release.yml: PR head ref (branch name)
- static_quality.yml: PR base ref, base SHA, head SHA
Introduce machinery for keeping examples/integrations/* demos aligned to a
single north-star (langgraph-python). Built first so the upcoming
langgraph-js and langgraph-fastapi alignment PRs have a mechanical baseline
to work against instead of manual copy-paste.
- examples/integrations/_parity/manifest.json declares verbatim files,
tracked package.json keys, and expected agent surface (tool names,
state keys) per instance plus allowed-divergence lists.
- _parity/sync.ts copies verbatim files + rewrites tracked package.json
keys from north-star to a target instance. Dry-run supported.
- _parity/verify.ts diffs each instance vs north-star and exits non-zero
on unexpected drift. Checks verbatim content, tracked keys, canonical
prompt equality, and agent-surface grep-level presence.
- Canonical prompt at _parity/canonical/PROMPT.md — synced into each
instance's agent/PROMPT.md on parity:sync.
- Root package.json: pnpm parity:sync, parity:verify, parity:check.
- CI: .github/workflows/integrations_parity.yml runs parity:check on PRs
touching examples/integrations/**.
- Skill: .claude/skills/copilotkit-demo-parity/SKILL.md teaches agents
how to drive sync/verify and handle manual-merge zones (agent code,
api route, Dockerfile).
Does NOT touch the existing instance demos yet. Those alignment commits
follow in the same PR.
The eval workflow was missing --ci, causing it to try Docker Compose
lifecycle in CI (which fails because there's no .env file). The --ci
flag skips Docker and assumes services are already running or uses
native execution.
Add GET /trigger/eval route to eval-webhook with HMAC-signed URLs.
The showcase_eval_check.yml workflow now posts a bot comment with a
clickable "Run Evaluation" link. Clicking triggers the eval and
redirects back to the PR. The link is signed so it can't be forged.
Add --ci flag to eval orchestrator that skips Docker lifecycle and
assumes services are already running. Add ci-native-eval.sh helper
that installs deps, starts next dev + agent servers natively, health-
waits, then runs showcase eval --ci. Fix on-demand E2E workflow with
langgraph-python support and agent-type detection.
New showcase_eval_check.yml creates a Check Run with "Run Showcase
Eval" action button on every PR. Modified showcase_eval.yml adds
workflow_dispatch trigger with dispatch-gate job, Check Run update
in post-result, and devops bot token for Checks API calls.
Hono web server that receives check_run.requested_action webhooks from
GitHub, authenticates as the devops bot, updates the Check Run to
in_progress, and dispatches showcase_eval.yml via workflow_dispatch.
Includes GHCR build workflow and pnpm workspace registration.
Picks up the router fix from CopilotKit/aimock#148 — `toolCallId` matchers
now only fire when the tool message is the *last* message in the request,
preventing stale tool_call_ids from history shadowing `userMessage`
matchers on new user turns.
Surfaced as: in beautiful-chat, clicking a second suggestion replayed the
prior chart's "Pie chart rendered above…" content fixture instead of
producing a new tool call. Once Railway rebuilds `ghcr.io/copilotkit/aimock:latest`
and restarts the service, demos will pick up the fix automatically.
- Refresh `pnpm-lock.yaml` resolutions (workspace `@copilotkit/runtime` devDep)
- Refresh `showcase/scripts/package-lock.json` to 1.16.4
- Bump the floor in `test_e2e-showcase-on-demand.yml` from `^1.14.3` → `^1.16.4`
so the `/test-aimock` PR-comment workflow always installs a build that
contains the fix
The decoupled build workflow (PR #4471) removed the Railway deploy
trigger, assuming environmentPatchCommit auto-update would handle
deploys. Not all services have auto-update configured, so GHCR images
were pushed but Railway never pulled them. Restore the explicit
serviceInstanceRedeploy call after each GHCR push.
The old "Showcase: Build & Deploy" workflow used a single concurrency group
that cancelled in-flight builds on every push to main. When multiple PRs
merged in quick succession, most service builds got cancelled and never
deployed.
Split into two workflows:
1. showcase_build.yml ("Showcase: Build & Push") - triggered on push to main,
builds Docker images and pushes to GHCR. Has NO concurrency group so every
run completes. Railway auto-update picks up the new :latest tag.
2. showcase_deploy.yml ("Showcase: Verify Deploy") - triggered by workflow_run
from the build workflow. Polls Railway to verify each service picked up the
new image and is healthy. Uses cancel-in-progress since verification is
idempotent. Posts results to showcase-harness via webhook.
Also updates showcase_capture-previews.yml to trigger from the renamed build
workflow.
## Summary
- **Restores the eval system** accidentally deleted by PR #4449 ("D5
all-green" session removed it as "superseded by D5 depth probes" while
unaware #4448 had just merged)
- **Per-test JSON granularity** — eval runner now injects
`--reporter=list,json` + `PLAYWRIGHT_JSON_OUTPUT_NAME` when spawning
test subprocesses, reads JSON file after exit for per-test results,
falls back to exit-code when absent
- **Slug-keyed baselines** — `transformHarnessResponse` restructured to
key by integration slug (via `summary.services[]`) instead of probe ID,
matching `collectResults` format so `computeRegressions` works against
harness-prod baselines
## Context
PR #4448 added the eval system (5 CR rounds, 8 bugs fixed, 35+ agent
reviews). PR #4449 (a concurrent session) deleted it. This PR restores
the CR-converged code and adds two follow-up improvements from the eval
plan.
## Test plan
- [x] `npx nx run @copilotkit/showcase-harness:test` — 1375 tests pass
(70 files)
- [x] All 8 CR fixes from #4448 verified present in restored code
- [ ] CI green
Production showcase-aimock was running a week-old image because fixture
file changes in showcase/aimock/ did not trigger a CI rebuild. This adds
showcase-aimock to the Build & Deploy workflow matrix so it auto-deploys
on merge, creates a thin Dockerfile that bakes fixture files into the
image, documents the local-vs-production parity requirement in
docker-compose.local.yml, and adds an aimock fixture deployment section
to the RUNBOOK.
PR #4449 removed the eval tier system while unaware #4448 had just
merged. Restoring orchestrator, matrix, scope, config, GHA workflow,
and CLI wiring with all 8 CR fixes intact.
The format job in static_quality.yml globs *.json (and *.yaml) to feed
oxfmt --write on PR-changed files. Lockfiles match those globs but oxfmt
rejects them internally (likely a size threshold or filename heuristic),
so lockfile-only PRs failed with 'Expected at least one target file' and
exit 123 even though the count check thought there were 18 files to format.
Exclude package-lock.json, pnpm-lock.yaml, and yarn.lock from the glob.
Lockfiles are auto-generated and should never be hand-formatted regardless,
so this is corrective — small JSON/YAML config files (tsconfig.json,
package.json, *.yml workflow files) still get formatted as before.
Caused PR #4438's format check to fail; surfaced again on this PR which
only touches lockfiles.
The audit checks repository == null on the Packages API to detect
unlinked packages. But some packages (showcase-pocketbase) have Actions
access configured manually via the UI, so pushes work fine despite
repository being null. There is no API to detect manual Actions access,
so this adds an explicit VERIFIED_ACCESS allowlist that excludes
known-good packages from the alert.
Single-version 3.12 job hides regressions on floor (3.10) and newly
supported ceiling (3.14). Run all five versions with fail-fast off
so partial breakage is visible.
Remove all showcase-starter-* services from the CI build matrix in
showcase_deploy.yml. These starter demos were decommissioned in PR
#4378 (code removed) and confirmed dormant with zero traffic. Removes
entries from workflow_dispatch options, paths-filter definitions, and
the ALL_SERVICES JSON array.
- Update collision-avoidance comment in showcase_deploy.yml to remove
starter-specific examples (service/starter collision no longer possible)
- Fix showcase_keep-alive.yml description: pings showcase services, not starters
- Remove "starter" from catalog-types.ts manifestation union type
- Remove dead starter cell skip logic from cell-matrix.tsx cellIndex builder
- Update depth-utils.ts comment (defensive null guard, not starter-specific)
- Remove starter-specific test cases from depth-utils and cell-matrix tests
The showcase-starter-* Railway services are being deprovisioned after
the packages/starters integration merge. Delete the dedicated smoke
workflow (test_smoke-starter-deployed.yml) and the "Deployed Starters"
describe block from integration-smoke.spec.ts to prevent 17 false-red
alerts per 6-hour cron cycle.
The extension is a leaf node in the monorepo dependency graph (nothing
depends on it) and has its own independent release cycle. The standalone
repo at CopilotKit/vscode-extension has full git history, working CI,
verified OIDC publishing, and branch protection.
## Summary
Adds `.github/workflows/ghcr_unlinked_packages.yml` — a scheduled
GitHub Actions workflow that audits all CopilotKit org container
packages daily and Slack-alerts when any are unlinked from a source
repository (`repository: null` on the GHCR API).
## Why
When a GHCR container package is unlinked from a repo, workflow
builds in `CopilotKit/CopilotKit` get `403 Forbidden` on push to GHCR
— the workflow `GITHUB_TOKEN` only has package-write permissions
when the package is linked to the actor's repo. We hit this twice in
quick succession:
- `showcase-ops` — caught manually after a failed deploy
- `showcase-pocketbase` — caught by a preemptive scan
There is **no GitHub API to programmatically link a package to a
repo** — it is UI-only. So the only way to prevent future surprises
is to detect drift early via a scheduled audit + Slack alert.
## Behavior
- Runs daily at 14:00 UTC, plus `workflow_dispatch` for ad-hoc runs.
- Lists every container package in the `CopilotKit` org via
`gh api /orgs/CopilotKit/packages?package_type=container`.
- Filters for `repository == null`.
- If any are unlinked, posts a Slack message that includes the count,
a bulleted list with deep links to each package's settings page,
the exact UI fix steps, and a footer noting the failure mode.
- Exits 0 in all non-error cases. The Slack message IS the alert;
failing the workflow on drift would create noisy red CI checks.
## Required setup before this workflow can fire
Two new repo secrets must be added:
1. **`ORG_READ_PACKAGES_PAT`** — a fine-grained PAT with
`read:packages` scope, org-scoped to `CopilotKit`. The default
`secrets.GITHUB_TOKEN` does NOT have org-package-list scope. The
workflow fails loudly if this is missing.
2. **`SLACK_WEBHOOK_GHCR_DRIFT`** — a CopilotKit-internal Slack
incoming-webhook URL for an alerts channel. If missing, the audit
still runs and logs a warning; only the Slack post is skipped.
Without these secrets the workflow will either fail loudly (PAT) or
log-only (webhook). It will not silently mask drift.
## Test plan
- [ ] Add `ORG_READ_PACKAGES_PAT` and `SLACK_WEBHOOK_GHCR_DRIFT` repo
secrets.
- [ ] Trigger via `workflow_dispatch` on `main` post-merge.
- [ ] Confirm the audit lists packages and reports the unlinked count.
- [ ] If drift exists, confirm Slack receives the alert with working
deep links.
- [ ] If no drift, confirm workflow exits 0 with a "no drift" log line
and skips the Slack post.
Adds a scheduled GitHub Actions workflow that detects when CopilotKit
org container packages drift into an unlinked state (`repository: null`
on the GHCR API) and posts a Slack alert with deep links to the UI fix.
This drift breaks future workflow builds with `403 Forbidden` on push
to GHCR — the workflow `GITHUB_TOKEN` only has package-write
permissions when the package is linked to the actor's repo. We hit
this twice in quick succession: `showcase-ops` (caught manually after
a failed deploy) and `showcase-pocketbase` (caught by a preemptive
scan). There is no GitHub API to programmatically link a package to
a repo — it is UI-only — so the only way to prevent future surprises
is to detect drift early.
Schedule: daily at 14:00 UTC, plus `workflow_dispatch` for ad-hoc
runs. Exits 0 on drift (the Slack message IS the alert; failing the
workflow on a schedule would create noisy red CI checks).
Requires two new repo secrets:
- `ORG_READ_PACKAGES_PAT` (read:packages, org-scoped to CopilotKit)
- `SLACK_WEBHOOK_GHCR_DRIFT` (CopilotKit-internal alerts webhook)
Workflow fails loudly if the PAT is missing; logs a warning and
continues if the webhook is missing.
Jordan Ritter