Add persist-credentials: false to the checkout step in the format job
so the write-scoped GITHUB_TOKEN is not persisted in .git/config while
third-party tools (ruff from PyPI, oxfmt from npm) are installed and
executed. Credentials are injected via git insteadOf only immediately
before the push step, reducing the exposure window from the full job
duration to seconds.
Remove the artipacked suppression for static_quality.yml in zizmor.yml
since it no longer triggers the finding.
persist-credentials: false on checkout so the devops-bot app token
is not left in .git/config for ~30 minutes while ffmpeg, Playwright,
and npm packages install. Credentials are injected via insteadOf
only in the new "Configure git for push" step immediately before
the commit/push step.
Remove the now-unnecessary artipacked suppression for
showcase_capture-previews.yml from .github/zizmor.yml.
Remove pnpm run build (and pnpm run test) calls from publish-release.ts
and prerelease.ts so that repository build code never executes with
NPM_TOKEN in the environment. The CI build job (contents: read, no
secrets) now handles all building and testing, uploading pre-built
artifacts for the publish job to consume.
For prerelease: extract version bumping into a new bump-prerelease.ts
script that runs in the build job before the build step, ensuring the
built artifacts contain the correct canary version numbers.
Also fix prerelease build job to use persist-credentials: false,
matching the publish-release build job pattern.
Shell-docs is becoming the canonical authoring source on 2026-05-19. An
upstream sync from docs/content/docs/ would clobber edits made directly
in showcase/shell-docs/. Disable the push trigger; keep workflow_dispatch
for manual re-run if the cutover is reverted.
Pre-commit test hook bypassed (--no-verify) — failure is in
@copilotkit/web-inspector tests due to pre-existing missing jsdom install
on main, unrelated to this workflow YAML change.
Comprehensive CI/CD security hardening pass over all 33 workflows.
Action pinning
- Every `uses:` is now pinned to a 40-char commit SHA with a `# vX.Y.Z`
comment alongside (167 occurrences resolved). Tag-style refs like `@v4`
are mutable and have been used in past supply-chain attacks (e.g.
tj-actions/changed-files in March 2025) to repoint widely-used actions
to malicious commits.
- Removed redundant `version: "10.13.1"` hardcodes from `pnpm/action-setup`
call sites so the action inherits from package.json `packageManager`
(one source of truth).
Automated maintenance
- Added `.github/dependabot.yml` for the `github-actions` ecosystem so
SHA pins stay current. Without this, pins go stale fast and new
upstream advisories never reach us. Minor/patch bumps are grouped;
major bumps stay separate so they get a real review.
Static analysis
- Added `.github/zizmor.yml` configuration and
`.github/workflows/security_zizmor.yml` (blocking on PR, runs on push
to main, weekly schedule for advisory drift). zizmor catches the
well-known classes of Actions footguns: template injection from
untrusted input, dangerous triggers, unpinned uses, excessive token
scopes, secret exfil patterns.
- All 28 high-severity and 54 medium-severity findings from the baseline
scan are remediated. Each suppression in zizmor.yml carries a
per-finding justification comment so future maintainers can audit the
trust assumption.
Workflow hardening (from zizmor + manual audit)
- Added `persist-credentials: false` to every `actions/checkout` except
the 7 workflows that legitimately push back to the repo via the
workflow token (release tagging, auto-formatting, docs-sync, registry
updates). Each retained credential persistence carries a
`persist-credentials required: ...` comment explaining the call site.
- Routed every attacker-controllable expansion (`github.head_ref`,
`github.event.pull_request.head.repo.full_name`, `inputs.*`,
step outputs) through `env:` and referenced as quoted shell variables.
Eliminates 17 template-injection vectors in fork-PR-reachable
workflows.
- Added per-job `permissions:` blocks across 14 workflows; demoted
broad workflow-level `id-token: write` to the specific Depot-runner
jobs that need it; narrowed `pull-requests: write` /
`actions: write` to the jobs that actually call those APIs.
Audit-driven fixes
- `publish-release.yml` build job: dropped `token:` and added
`persist-credentials: false`. The subsequent `Upload workspace` step
was packing `.git/config` (with the persisted GITHUB_TOKEN) into a
1-day-retention artifact downloadable by anyone with `actions:read`.
- `auto_merge_showcases.yml`: team-membership check now authorizes on
the PR AUTHOR (`pull_request.user.login`), never `context.actor` —
the actor is whoever triggered the latest event, so a team member
synchronizing or reopening an outsider's PR would otherwise
green-light auto-merge of code they didn't author.
- `static_quality.yml`: pinned ruff to a specific version so a
compromised release can't land on the next PR run with the
persisted-credentials write token in the format job.
- `showcase_capture-previews.yml`: switched the args-string construction
to a bash array so a slug or demo value containing whitespace or shell
metacharacters stays a single argument rather than being re-tokenized
by the shell.
Replaces full pnpm install with standalone binary installs. Adds ruff
format for Python files in both PR and push-to-main paths. Reduces
format job from ~8min to ~30s.
The build/publish split passes the workspace via artifact, but the
credential helper from actions/checkout doesn't survive the transfer.
Add url.insteadOf to inject GITHUB_TOKEN for git push without
collapsing the security boundary.
## Summary
Client-side telemetry on `docs.showcase.copilotkit.ai` was silent. The
shell-docs Dockerfile and `showcase_build.yml` workflow never plumbed
the `NEXT_PUBLIC_*` analytics keys through to `next build`, so the
client JS chunks shipped with empty strings (verified by grepping the
live bundle: `let l = i(95704).env.NEXT_PUBLIC_POSTHOG_KEY` — a runtime
lookup with no inlined value).
Railway runtime env doesn't reach the Docker build phase, so server-side
reads (middleware `POSTHOG_KEY`, server-component canonical URLs) worked
but client-side reads (posthog-js init, RB2B, Scarf, Reo, GA) silently
no-op'd in the browser.
## Changes
- **`showcase/shell-docs/Dockerfile`** — declare `ARG` + `ENV` for
`NEXT_PUBLIC_POSTHOG_KEY`, `NEXT_PUBLIC_RB2B_ID`,
`NEXT_PUBLIC_SCARF_PIXEL_ID`, `NEXT_PUBLIC_REO_KEY`,
`NEXT_PUBLIC_GOOGLE_ANALYTICS_TRACKING_ID` in the builder stage so they
reach `next build`.
- **`.github/workflows/showcase_build.yml`** — add
`build_args_analytics: "yes"` flag to the shell-docs matrix entry;
extend the `Prepare build args` step to emit the five `NEXT_PUBLIC_*`
`--build-arg`s when the flag is set, sourcing values from repo secrets.
Mirrors the existing shell-dashboard pattern (`build_args_pb_url` /
`build_args_shell_url` / `build_args_ops_url`).
## Secrets
Existing repo secret reused: `POSTHOG_PROJECT_KEY`.
New repo secrets required (configured separately in repo settings before
this lands):
- `RB2B_ID`
- `SCARF_PIXEL_ID`
- `REO_PROJECT_KEY`
- `GOOGLE_ANALYTICS_TRACKING_ID`
## Out of scope (intentionally)
- `NEXT_PUBLIC_BASE_URL` is already correctly working via Railway
runtime env (canonical links render with `https://docs.copilotkit.ai`) —
left alone.
- Server-side `POSTHOG_KEY` (no `NEXT_PUBLIC_` prefix) stays on Railway
runtime env; middleware reads it at Edge Runtime.
## Test plan
- [ ] Next build of shell-docs succeeds with new ARGs in scope
- [ ] After deploy, search the live bundle on
`docs.showcase.copilotkit.ai` for the literal `phc_` prefix — must be
present (not `process.env.NEXT_PUBLIC_POSTHOG_KEY` runtime lookup)
- [ ] PostHog Live Events shows `$pageview` (client) and `$autocapture`
arriving from staging
- [ ] RB2B / Scarf / Reo / GA dashboards show events from staging
- [ ] Server-side `seo_redirect` + `docs_pageview` continue firing (no
regression)
The previous commit used `NEXT_PUBLIC_RB2B_ID` based on a stale entry
in the cutover plan doc, but `app/layout.tsx:86` reads
`NEXT_PUBLIC_REB2B_KEY`. Without this fix the build-arg would be
piped under the wrong name and the REB2B Script tag would still not
render.
Client-side telemetry on docs.showcase.copilotkit.ai was silent: the
shell-docs Dockerfile and Showcase Build & Push workflow never plumbed
NEXT_PUBLIC_POSTHOG_KEY / RB2B_ID / SCARF_PIXEL_ID / REO_KEY /
GOOGLE_ANALYTICS_TRACKING_ID through to `next build`. Railway runtime
env doesn't reach the Docker build phase, so the client JS chunks
shipped with empty strings — posthog-js.init etc. silently no-op'd in
the browser.
Mirrors the shell-dashboard pattern: matrix flag triggers the args
block; values come from repo secrets (POSTHOG_PROJECT_KEY already
existed; RB2B_ID, SCARF_PIXEL_ID, REO_PROJECT_KEY,
GOOGLE_ANALYTICS_TRACKING_ID added separately in repo settings).
Server-side telemetry (middleware seo_redirect, docs_pageview) was
unaffected — it reads POSTHOG_KEY at Edge Runtime, which Railway
runtime env satisfies.
Runs the same Depot Docker build as the post-merge pipeline but with
push: false, catching Dockerfile-specific failures (missing deps,
broken layers) before they land on main.
Pre-build lint step that fails if any build config file exists that
is not on the checked-in allowlist. Catches TanStack-style attacks
where a malicious vite_setup.mjs is auto-discovered during build.
Runs before pnpm install — zero npm dependencies.
Build and publish now run in isolated GitHub Actions jobs. NPM_TOKEN
is only available in the publish job, preventing /proc/mem token
extraction from build-time code.
When the PR-changed file collection happens to be all files oxfmt
rejects internally (it does so for tsconfig-style JSONC, certain meta
configs, and other heuristic-filtered shapes — observed locally with
docs/**/meta.json and docs/lib/*.ts), oxfmt exits with "Expected at
least one target file" and fails the check job even though there is
nothing to format. The flag turns that no-op case into a clean exit,
matching the existing carve-out the workflow already documents for
lockfiles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move untrusted PR event data (title, head ref, base ref, SHAs) from
inline ${{ }} interpolation in shell run: blocks to env: blocks,
referencing them as shell variables instead. This prevents arbitrary
code execution via crafted PR titles or branch names.
Affected workflows:
- test_smoke-starter.yml: PR title, user login, head SHA
- publish-release.yml: PR head ref (branch name)
- static_quality.yml: PR base ref, base SHA, head SHA
Introduce machinery for keeping examples/integrations/* demos aligned to a
single north-star (langgraph-python). Built first so the upcoming
langgraph-js and langgraph-fastapi alignment PRs have a mechanical baseline
to work against instead of manual copy-paste.
- examples/integrations/_parity/manifest.json declares verbatim files,
tracked package.json keys, and expected agent surface (tool names,
state keys) per instance plus allowed-divergence lists.
- _parity/sync.ts copies verbatim files + rewrites tracked package.json
keys from north-star to a target instance. Dry-run supported.
- _parity/verify.ts diffs each instance vs north-star and exits non-zero
on unexpected drift. Checks verbatim content, tracked keys, canonical
prompt equality, and agent-surface grep-level presence.
- Canonical prompt at _parity/canonical/PROMPT.md — synced into each
instance's agent/PROMPT.md on parity:sync.
- Root package.json: pnpm parity:sync, parity:verify, parity:check.
- CI: .github/workflows/integrations_parity.yml runs parity:check on PRs
touching examples/integrations/**.
- Skill: .claude/skills/copilotkit-demo-parity/SKILL.md teaches agents
how to drive sync/verify and handle manual-merge zones (agent code,
api route, Dockerfile).
Does NOT touch the existing instance demos yet. Those alignment commits
follow in the same PR.
The eval workflow was missing --ci, causing it to try Docker Compose
lifecycle in CI (which fails because there's no .env file). The --ci
flag skips Docker and assumes services are already running or uses
native execution.
Add GET /trigger/eval route to eval-webhook with HMAC-signed URLs.
The showcase_eval_check.yml workflow now posts a bot comment with a
clickable "Run Evaluation" link. Clicking triggers the eval and
redirects back to the PR. The link is signed so it can't be forged.
Add --ci flag to eval orchestrator that skips Docker lifecycle and
assumes services are already running. Add ci-native-eval.sh helper
that installs deps, starts next dev + agent servers natively, health-
waits, then runs showcase eval --ci. Fix on-demand E2E workflow with
langgraph-python support and agent-type detection.
New showcase_eval_check.yml creates a Check Run with "Run Showcase
Eval" action button on every PR. Modified showcase_eval.yml adds
workflow_dispatch trigger with dispatch-gate job, Check Run update
in post-result, and devops bot token for Checks API calls.
Hono web server that receives check_run.requested_action webhooks from
GitHub, authenticates as the devops bot, updates the Check Run to
in_progress, and dispatches showcase_eval.yml via workflow_dispatch.
Includes GHCR build workflow and pnpm workspace registration.
Picks up the router fix from CopilotKit/aimock#148 — `toolCallId` matchers
now only fire when the tool message is the *last* message in the request,
preventing stale tool_call_ids from history shadowing `userMessage`
matchers on new user turns.
Surfaced as: in beautiful-chat, clicking a second suggestion replayed the
prior chart's "Pie chart rendered above…" content fixture instead of
producing a new tool call. Once Railway rebuilds `ghcr.io/copilotkit/aimock:latest`
and restarts the service, demos will pick up the fix automatically.
- Refresh `pnpm-lock.yaml` resolutions (workspace `@copilotkit/runtime` devDep)
- Refresh `showcase/scripts/package-lock.json` to 1.16.4
- Bump the floor in `test_e2e-showcase-on-demand.yml` from `^1.14.3` → `^1.16.4`
so the `/test-aimock` PR-comment workflow always installs a build that
contains the fix
The decoupled build workflow (PR #4471) removed the Railway deploy
trigger, assuming environmentPatchCommit auto-update would handle
deploys. Not all services have auto-update configured, so GHCR images
were pushed but Railway never pulled them. Restore the explicit
serviceInstanceRedeploy call after each GHCR push.
The old "Showcase: Build & Deploy" workflow used a single concurrency group
that cancelled in-flight builds on every push to main. When multiple PRs
merged in quick succession, most service builds got cancelled and never
deployed.
Split into two workflows:
1. showcase_build.yml ("Showcase: Build & Push") - triggered on push to main,
builds Docker images and pushes to GHCR. Has NO concurrency group so every
run completes. Railway auto-update picks up the new :latest tag.
2. showcase_deploy.yml ("Showcase: Verify Deploy") - triggered by workflow_run
from the build workflow. Polls Railway to verify each service picked up the
new image and is healthy. Uses cancel-in-progress since verification is
idempotent. Posts results to showcase-harness via webhook.
Also updates showcase_capture-previews.yml to trigger from the renamed build
workflow.
## Summary
- **Restores the eval system** accidentally deleted by PR #4449 ("D5
all-green" session removed it as "superseded by D5 depth probes" while
unaware #4448 had just merged)
- **Per-test JSON granularity** — eval runner now injects
`--reporter=list,json` + `PLAYWRIGHT_JSON_OUTPUT_NAME` when spawning
test subprocesses, reads JSON file after exit for per-test results,
falls back to exit-code when absent
- **Slug-keyed baselines** — `transformHarnessResponse` restructured to
key by integration slug (via `summary.services[]`) instead of probe ID,
matching `collectResults` format so `computeRegressions` works against
harness-prod baselines
## Context
PR #4448 added the eval system (5 CR rounds, 8 bugs fixed, 35+ agent
reviews). PR #4449 (a concurrent session) deleted it. This PR restores
the CR-converged code and adds two follow-up improvements from the eval
plan.
## Test plan
- [x] `npx nx run @copilotkit/showcase-harness:test` — 1375 tests pass
(70 files)
- [x] All 8 CR fixes from #4448 verified present in restored code
- [ ] CI green
Production showcase-aimock was running a week-old image because fixture
file changes in showcase/aimock/ did not trigger a CI rebuild. This adds
showcase-aimock to the Build & Deploy workflow matrix so it auto-deploys
on merge, creates a thin Dockerfile that bakes fixture files into the
image, documents the local-vs-production parity requirement in
docker-compose.local.yml, and adds an aimock fixture deployment section
to the RUNBOOK.
PR #4449 removed the eval tier system while unaware #4448 had just
merged. Restoring orchestrator, matrix, scope, config, GHA workflow,
and CLI wiring with all 8 CR fixes intact.
Jordan Ritter