## Summary
Unblocks the `Validate Showcase` CI check which has been failing on
`main` for all 17 packages (last 3 runs: 5b1a137a, eb13a8dc, bc35c2e8)
with:
```
[FAIL] <pkg>: demo 'hitl-in-chat' declared in manifest but no src/app/demos/hitl-in-chat/ directory
```
Every package manifest declares `id: hitl-in-chat` with `route:
/demos/hitl`, and every package has its demo files in
`src/app/demos/hitl/`. The validator was using `demo.id` as the expected
directory name, which is incorrect when `demo.route` deliberately points
elsewhere.
## Fix
- Add optional `route` field to `ManifestDemo` (validated — must start
with `/demos/` with a non-empty tail)
- Add `routeToDirName()` helper that extracts the tail segment from
`demo.route`
- In `validate-parity.ts`, resolve `expectedDir =
routeToDirName(demo.route) ?? demo.id` before probing
`src/app/demos/<expectedDir>/`
- Expand the `missing-demo-dir` `PackageIssue` shape to carry `{ demoId,
expectedDir }` for clearer error output
- Red-green regression tests in `validate-parity.test.ts` +
`manifest.test.ts`
## Scope
Pure validator code — **no manifest changes, no demo directory renames,
no package content changes**. 4 files changed (+314/−18) all under
`showcase/scripts/`.
## Verification
- `pnpm test` in `showcase/scripts/`: 1027/1027 pass
- `pnpm tsx validate-parity.ts` against all 17 packages: **17 pass, 0
fail** (remaining 128 warnings are pre-existing and unrelated — missing
QA/e2e files)
## Test plan
- [ ] CI `Validate Showcase` transitions to green
- [ ] No unrelated CI checks regress
## Not included
This PR is intentionally scoped to the validator fix only. It is
cherry-picked off PR #4064 (showcase bundle-0/1 cleanup) so that `main`
can unblock without waiting on that larger branch.
parseManifest (lib/manifest.ts):
- Shape errors for demos[i].route: number / null / object / empty string
- Shape error for route not starting with /demos/
- Happy path persists frozen demo.route on the parsed entry
- Backward-compat: demos[i] without route is accepted with undefined route
validate-parity:
- Negative-case regression: missing-demo-dir's expectedDir is derived from
demo.route (not demo.id) when route is present; mismatched id + route
was silently hiding drift.
- Fallback-case: demo with no route resolves expectedDir from demo.id
- routeToDirName unit tests: undefined / bare /demos/ / normal tail segment
TDD verified: mutating the /demos/ prefix guard failed the relevant test
(RED), restoring the guard passed it (GREEN). Mutating expectedDir to
demo.id-only failed the negative-case test (RED), restoring passed
(GREEN).
demo.id is the CATALOG identifier (matched to qa/spec filenames and
shell registry entries). demo.route is the URL + filesystem path
(/demos/<dir> → src/app/demos/<dir>/). They are deliberately separate
— a manifest with id: hitl-in-chat and route: /demos/hitl lives at
src/app/demos/hitl/.
validate-parity.ts previously resolved the demo directory from
demo.id, producing a spurious missing-demo-dir MUST for every such
split. Fix:
- lib/manifest.ts: add optional route field to ManifestDemo; if present,
parser requires it to be a non-empty string beginning with "/demos/".
- validate-parity.ts: introduce routeToDirName helper (matches
bundle-demo-content.ts idiom); loop over demos resolving expected
dir from route and falling back to id. missing-demo-dir PackageIssue
now carries both demoId and expectedDir so deriveMessage can flag
route-resolved paths distinctly.
- __tests__/validate-parity.test.ts: red-green regression test — a
package with id: hitl-in-chat, route: /demos/hitl, and dir
src/app/demos/hitl/ must PASS (no missing-demo-dir error).
## Summary
- Adds an env-controlled aimock toggle to
`showcase/packages/crewai-crews` (and every starter whose package ships
`.env.example`) so the Python agent can be redirected to a local
[aimock](https://github.com/CopilotKit/aimock) server for dev / CI /
smoke. **Production default is unchanged** — when `AIMOCK_URL` is unset,
behavior is identical to today (real OpenAI via `OPENAI_API_KEY`). A
production guard additionally refuses to apply the toggle when any of 8
prod-ish env vars (`NODE_ENV`, `ENV`, `ENVIRONMENT`, `APP_ENV`,
`DEPLOY_ENV`, `PYTHON_ENV`, `RAILWAY_ENVIRONMENT`,
`RAILWAY_ENVIRONMENT_NAME`) equals `production`/`prod`, so a
misconfigured deploy that leaks `AIMOCK_URL` cannot silently redirect
real traffic through a mock.
- Extends `showcase/scripts/generate-starters.ts` to ship
`aimock_toggle.py` alongside `agent_server.py` AND to propagate
`.env.example` into every starter whose package ships one. The copy step
rewrites `AGENT_URL=http://localhost:8000` → `:8123` during propagation
(starter dev scripts bind on 8123 while package dev scripts bind on
8000). The port-rewrite regex is scoped to `localhost` / `127.0.0.1` AND
is exported so `starter-consistency.test.ts` imports the same pattern —
generator + test cannot drift.
- Hardens `.github/workflows/showcase_aimock-e2e.yml` against script
injection via PR comment bodies (attacker-controlled text passes through
`env:` instead of expression interpolation; slug is validated against a
strict whitelist before reaching `cd` / `pip install`; job is gated on
`author_association` so third-party commenters cannot trigger runs at
all). R4 additions: aimock pinned to `@copilotkit/aimock@^1.14.3` (was
`@latest`); `pnpm install --ignore-scripts` (defense against postinstall
on untrusted PR content); `workflow_dispatch` slug is a `choice` enum
restricted to packages that ship `aimock_toggle.py`; package-type step
short-circuits with a clear `::error::` for non-toggle slugs; slug
parsing uses POSIX-safe `grep -oE` + `sed` (no PCRE); Python agent
health-check extended to 90s for CrewAI cold imports; pip cached via
`actions/setup-python`; `OPENAI_BASE_URL` is no longer pre-set for
python agent start (toggle must do it); dead `AIMOCK_URL` exports
removed from Next.js dev-server + Playwright steps; `Post result to PR`
passes slug + job status via `env:` (not `${{ }}` interpolation). R7
additions: `npm bin -g` replaced with `npm prefix -g`/`bin/aimock` (the
former was removed in npm 9 and hard-failed on the node 22 runner); `pip
install --only-binary :all:` loosened to `--prefer-binary` to tolerate
sdist-only transitive deps (author_association gate bounds the residual
risk); aimock PID captured + `kill -0` in the readiness loop so a crash
fails fast; `APP_MODULE` fallback dropped (unreachable); comment-trigger
TOCTOU documented in the residual-trust-model header.
- Splits pytest into its own `python-unit-tests` job in
`showcase_validate.yml`. Now a **matrix on Python 3.10 + 3.12**
(fail-fast disabled) — 3.10 exercises the `typing_extensions`
`NotRequired` fallback path that was previously uncovered by CI. `pip
install` per package is a HARD FAIL (was `::warning::` + continue) so a
broken `requirements.txt` cannot ship green.
- **94 pytest cases** covering the toggle contract (red-green verified;
+6 in R7 for `RAILWAY_ENVIRONMENT_NAME` parametrize entries across both
prod-guard "refuses" + "silent no-op" variants). 1013 showcase-scripts
vitest cases pass.
## Context
Tonight's outage traced to defects in `ag-ui-crewai` triggered by
upstream LLM behavior. A parallel PR in `ag-ui-protocol/ag-ui` fixes
those defects. This PR is the complementary dev/CI/smoke-side
improvement: routing non-prod traffic through aimock removes one class
of trigger and cuts dev API costs + external dependency.
## Interface contract
| Scenario | Result |
|---|---|
| `AIMOCK_URL` set, `USE_AIMOCK` unset | **enabled** (default when URL
is set) |
| `AIMOCK_URL` set, `USE_AIMOCK=1\|true\|yes\|on` | **enabled** |
| `AIMOCK_URL` set, `USE_AIMOCK=0\|false\|no\|off` | **disabled**
(explicit opt-out) |
| `AIMOCK_URL` set, `USE_AIMOCK=<unknown value>` | **disabled** +
WARNING (fail safe) |
| `AIMOCK_URL` set, pre-existing `OPENAI_BASE_URL` OR `LITELLM_API_BASE`
differs (raw) | overwrite + WARNING on each |
| `AIMOCK_URL` set, `USE_AIMOCK=""` | **enabled** (empty fall-through) |
| `AIMOCK_URL` set, `OPENAI_API_KEY` unset/empty/whitespace | dummy key
injected (`sk-aimock-dev-ci-only`) |
| `AIMOCK_URL` set, any prod-guard var set to `production`/`prod` |
**refused** + WARNING (production guard) |
| `USE_AIMOCK=1` alone, no `AIMOCK_URL` | no-op + WARNING (reason:
`AIMOCK_URL unset`) |
| `AIMOCK_URL=" "` (whitespace-only) | no-op (reason: `... empty or
whitespace-only`) |
| Both unset | **no-op**, production path preserved |
When enabled, the toggle populates `OPENAI_BASE_URL` +
`LITELLM_API_BASE` and injects a dummy `OPENAI_API_KEY` (with `sk-`
prefix, self-documenting value) if absent/empty/whitespace.
## Import ordering (load-bearing)
`load_dotenv()` + `configure_aimock()` sit at the **very top** of
`agent_server.py`, above the `from crewai.cli import crew_chat` import.
crewai / litellm can construct clients at import time that latch onto
`OPENAI_BASE_URL` / `OPENAI_API_KEY` as they were at import — mutating
env after the fact is invisible to those clients.
## Python version compatibility
`aimock_toggle.py` imports `NotRequired` from `typing` (Python 3.11+)
with a fallback to `typing_extensions` (listed in `requirements.txt` as
`typing_extensions>=4.6; python_version < "3.11"`, so the backport only
installs on older Pythons). The 3.10 matrix entry in `python-unit-tests`
exercises the real-stdlib-absent fallback end-to-end; an additional
pytest case (`test_typing_extensions_fallback_path_importable`) forces
the branch via `sys.modules` manipulation for defense-in-depth even on
3.11+.
## CI integration
- **`showcase_aimock-e2e.yml`** (manual + `/test-aimock` comment
trigger): slug arrives from an attacker-controlled comment body, so we
route it through an `env:` block (not `${{ ... }}` expression
interpolation) and validate it against `^[a-z0-9-]+$` AND confirm
`showcase/packages/<slug>/src/aimock_toggle.py` exists before any `cd` /
`pip install`. Job is additionally gated on `author_association` ∈
`{OWNER, MEMBER, COLLABORATOR}`. No silent-fallback slug — bare
`/test-aimock` with no slug is rejected (R6). Auto-detects
`agent_server:app`; hard-fails if the agent doesn't bind. R7: uses `npm
prefix -g` (not the removed-in-npm-9 `npm bin -g`) and `pip install
--prefer-binary` (not `--only-binary :all:`, which broke on sdist-only
transitive deps).
- **`showcase_validate.yml`** (every PR): dedicated `python-unit-tests`
matrix job (3.10 + 3.12) discovers `showcase/packages/*/tests/python/`,
installs each package's `requirements.txt` as a hard-fail, then runs
pytest.
## Security hardening (cumulative through R7)
- **Script injection via PR comment body**: blocked — comment body
passes through `env:` (not expression interpolation), and the derived
slug is validated against a strict regex + existing-directory check
before any downstream shell consumes it.
- **Path traversal via `cd showcase/packages/$SLUG`**: blocked — slug
regex rejects `../`, `/`, and anything else that could escape the
packages dir.
- **Third-party comment triggers**: blocked by `author_association` gate
on the job.
- **Postinstall on untrusted PR content**: blocked — `pnpm install
--ignore-scripts` and `npm install --ignore-scripts` for the aimock
install.
- **Unpinned `@latest` aimock**: blocked — pinned to `^1.14.3`.
- **Production toggle leakage**: blocked — 8 prod-ish env vars
(NODE_ENV, ENV, ENVIRONMENT, APP_ENV, DEPLOY_ENV, PYTHON_ENV,
RAILWAY_ENVIRONMENT, RAILWAY_ENVIRONMENT_NAME) refuse the toggle with a
WARNING when set to `production`/`prod`.
- **Copy-pasteable `sk-...` placeholder in `.env.example`**: replaced
with `replace-with-your-key` across every package + starter so users who
skip editing get a loud 401 from real OpenAI instead of a silent aimock
success masking misconfig.
- **Known TOCTOU**: the gap between trusted `/test-aimock` comment and
`pulls.get` HEAD resolution is an architectural limitation of GitHub
Actions' comment-trigger model (no native comment-time SHA pin).
Documented in the workflow header; mitigated by the author_association
gate + code-review social contract.
## Verification
```bash
# Unit tests — red-green verified
cd showcase/packages/crewai-crews
python3 -m pytest tests/python/ -v # 94 passed
# Starter consistency + generator guards
pnpm nx run @copilotkit/showcase-scripts:test # 1013 passed
cd showcase/scripts && pnpm exec tsx generate-starters.ts --check # OK
# Docker build (local verification per docker-ci-safety)
cd showcase/packages/crewai-crews
cp -R ../../shared/python shared_python # CI copies this in
docker build -f Dockerfile -t test-crewai-crews . # SUCCESS
cd ../../starters/crewai-crews
docker build -t test-crewai-crews-starter . # SUCCESS
# Confirm AGENT_URL port rewrite in all starters
grep AGENT_URL showcase/starters/*/.env.example
# every hit reads `AGENT_URL=http://localhost:8123`
```
## Test plan
- [x] `pytest tests/python/` — 94/94 passed on Python 3.10
(typing_extensions fallback) and 3.12 (stdlib)
- [x] Red-green: R7 RAILWAY_ENVIRONMENT_NAME parametrize entries (6 new
cases) fail pre-fix, pass after
- [x] `pnpm nx run @copilotkit/showcase-scripts:test` — 1013 passed (20
starter-consistency + shared regex)
- [x] `generate-starters.ts --check` — all 17 starters in sync
- [x] `docker build` both package and starter — succeeds
- [x] All 17 starter `.env.example` files:
`AGENT_URL=http://localhost:8123` and
`OPENAI_API_KEY=replace-with-your-key`
- [ ] CI green on the PR (verify post-force-push)
Review pass across all 18 langgraph-python demo columns. For each
demo, verified strict minimality (no cross-contamination from other
features) and extracted any inline view JSX >= 15 lines into dedicated
sibling files. All 18 demos validated end-to-end via Playwright
screenshot + visual review.
View components extracted to dedicated files:
- `a2ui-fixed-schema/flight-card.tsx` + `catalog.ts`
- `agentic-chat-reasoning/reasoning-block.tsx`
- `chat-slots/custom-welcome-screen.tsx`, `custom-assistant-message.tsx`,
`custom-disclaimer.tsx`
- `gen-ui-agent/InlineAgentStateCard.tsx`
- `gen-ui-interrupt/InterruptCard.tsx`
- `headless-complete/message-list.tsx`, `user-bubble.tsx`,
`assistant-bubble.tsx`, `input-bar.tsx`, `typing-indicator.tsx`
- `hitl-in-chat/approval-card.tsx`
- `open-gen-ui/sandbox-functions.ts`, `suggestions.ts`
- `tool-rendering/weather-card.tsx`
Backend fixes:
- `mcp_apps_agent.py`: tighten system prompt to prevent the LLM from
hallucinating unrelated framework names in its reply.
- `open_gen_ui_agent.py`: short-circuit subsequent runs when a
`ToolMessage` is already in state. `generateSandboxedUi` is
registered with `followUp: true` by the provider, which caused the
agent to re-emit the tool call in a loop after each sandbox handler
response.
`agentic-chat/README.md`: removed stale references to tools that no
longer exist in the minimized demo.
Regenerated `shell/src/data/demo-content.json`.
Each demo's page.tsx is now pure composition; presentational code lives
in dedicated sibling files. Zero cross-contamination between demos
(audited via grep for forbidden-feature imports).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Parallel review subagents each need a fresh
showcase-langgraph-python build to screenshot. Running
`dev-local.sh up langgraph-python` N times in parallel serializes on
Docker and wastes time rebuilding identical source.
`scripts/agent-notes/rebuild-coord.sh` aligns rebuild attempts to a
30-second clock tick with 0-5s jitter. First agent on a tick acquires
a `mkdir`-based lock, rebuilds, and writes a per-tick done marker at
`/tmp/showcase-rebuild.done-<tick>`. Peers on the same tick skip their
own build and wait up to 30s for that marker. If no marker appears,
they retry the next tick (up to 3 ticks).
Agents call `bash scripts/agent-notes/rebuild-coord.sh` instead of
`./scripts/dev-local.sh up langgraph-python` when they need the live
container refreshed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Move `agentic-chat-reasoning` from `chat-ui` category to
`generative-ui`, positioned after `tool-rendering`.
- Rename display "Agentic Chat (Reasoning)" -> "Reasoning".
- Update langgraph-python manifest demo entry's name, description,
and tag to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These 7 feature ids had zero manifest references, zero demo dirs, and
zero matching packages backing them in the showcase. They added noise
to the feature matrix without carrying substance:
- mobile-react-native
- mobile-swiftui
- mobile-android
- web-svelte
- web-vue
- web-tanstack — conceptually misplaced anyway; TanStack Start is a
React meta-framework, not a React alternative
- web-angular — `packages/angular/` exists as a published SDK, but the
showcase has zero integration for it. Angular support is really its
own matrix/story; removing this placeholder row until there's a real
showcase integration.
Also drops the `mobile` and `web-frameworks` categories now that they
have no members. Regenerates registry.json + constraints.json.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Generated via `generate-registry.ts` and `bundle-demo-content.ts` from
the new manifest + feature-registry + demo sources.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Minimal demo pages covering the Chat & UI category:
- `agentic-chat/page.tsx` — stripped to pure chat (CopilotKit +
CopilotChat, single "Write a sonnet" suggestion). No tool calls.
- `chat-slots/page.tsx` — three slot overrides: custom welcome screen,
custom disclaimer (input sub-slot), and a custom assistant message
bubble so the slot is visible during active chat.
- `headless-simple/page.tsx` — bare-bones custom chat built on
`useAgent` (no CopilotChat) + a single `useComponent` for inline
card rendering.
- `headless-complete/page.tsx` — fuller custom chat: scrollable
message list, user/assistant bubbles, auto-scroll, disabled state
while running, inline tool-call rendering, cancel button.
- `agentic-chat-reasoning/page.tsx` — custom `messageView.reasoningMessage`
slot renders an amber "Reasoning" block when the agent emits
`role: "reasoning"` messages.
- `prebuilt-chat/page.tsx` — minimal `<CopilotChat />`.
- `prebuilt-sidebar/page.tsx` — dummy main content + `<CopilotSidebar />`
(defaultOpen).
- `prebuilt-popup/page.tsx` — dummy main content + `<CopilotPopup />`
(defaultOpen).
Each outer function wraps provider + layout; inner `Chat` holds only
the registrations for that demo's feature. No cross-contamination
(no a2ui/HITL/interrupt/tool-rendering in demos that don't demonstrate
them).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- manifest.yaml: add 12 new demo entries (chat-slots, headless-simple,
headless-complete, agentic-chat-reasoning, hitl-in-chat,
gen-ui-interrupt, declarative-gen-ui, a2ui-fixed-schema, mcp-apps,
open-gen-ui, prebuilt-chat/sidebar/popup).
- langgraph.json: register dedicated graphs for demos that need a
non-default backend (tool_rendering, interrupt_agent, reasoning_agent,
a2ui_dynamic, a2ui_fixed, mcp_apps, open_gen_ui).
- route.ts: register each new agent name; demos with dedicated graphs
are mapped explicitly outside the shared-graph loop. A2UI runtime
middleware scoped to `declarative-gen-ui` + `a2ui-fixed-schema` so
non-A2UI demos keep their `useComponent` renderers intact. MCP Apps
middleware wraps only the `mcp-apps` agent and synthesizes an
`ACTIVITY_SNAPSHOT` event when the backend `show_mcp_app` tool fires.
- New `/api/copilotkit-ogui/route.ts`: dedicated isolated runtime for
the `open-gen-ui` demo. `openGenerativeUI: {...}` advertises the
`openGenerativeUIEnabled` flag globally on the probe response, which
causes the CopilotKit provider's setTools effect to wipe per-demo
`useFrontendTool`/`useComponent` registrations; isolating to its own
endpoint keeps the default runtime clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds rows for the demos being built in this branch:
- Split `chat-prebuilt` into `prebuilt-chat`, `prebuilt-sidebar`,
`prebuilt-popup` (CopilotChat / CopilotSidebar / CopilotPopup).
- Reorder `generative-ui` rows and add: `hitl-in-chat` (In-Chat HITL),
`gen-ui-interrupt`, `declarative-gen-ui` (Dynamic Schema),
`a2ui-fixed-schema` (new), `mcp-apps`, `open-gen-ui`, `gen-ui-agent`,
`tool-rendering`.
- `constraints.yaml`: expand `constrained-explicit` allowlist so the
langgraph-python manifest validates against the new feature ids.
- Bump expected langgraph-python feature/demo count in
`generate-registry.test.ts` from 10 -> 22 to match the manifest.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Run 24602985507 (singleFork: true) was strictly worse than the prior
fork-per-file run (24602657301):
singleFork (24602985507): 1/14 files completed, 64.64s, then timeout
fork-per-file (24602657301): 14/14 files passed, 169.56s, then teardown timeout
The RPC timeout under Node 20 is a hardcoded 60 s DEFAULT_TIMEOUT in
birpc (DEFAULT_TIMEOUT = 6e4 in vitest's index.B521nVV-.js, not tunable
via config). validate-pins.test.ts alone takes ~60 s because of its
134 `npx tsx` subprocess invocations, so singleFork blows its RPC
budget before a single file finishes. Fork-per-file gives each file
its own fresh 60 s RPC channel, which is why the prior run got all
14/14 files to complete before the final teardown hiccup.
Revert `poolOptions.forks.singleFork: true`. Keep the teardownTimeout
/ hookTimeout bumps to 30 s — they don't affect the RPC timeout but
do protect against slow per-hook teardowns under the same load. Note
in comments that the RPC timeout is upstream-hardcoded and tracked at
https://github.com/vitest-dev/vitest/issues/6129.
`sorts integrations by sort_order` read registry.json without invoking
the generator, making it dependent on whichever run last left disk
state in place. afterEach(dataRestorer.restore()) restores the data
files to HEAD between tests, so test 2 was reading the committed
baseline registry — not the live generator output — and would have
silently agreed with whatever state happened to be on main.
Add a `runGenerator()` call at the top of test 2 mirroring the
`runBundlerAndRead()` pattern that tests 2-5 of bundle-demo-content
already use. Test now exercises the actual generator's sort
behavior under live conditions.
The PR description advertised a drifted-baseline guard on CI for
restoreFromGitHead, but the implementation never actually ran a
post-heal `git diff --quiet HEAD -- <tracked>` check — the only
diff was the off-CI pre-checkout guard against clobbering developer
edits. A racing external mutator (or a parallel test suite we haven't
accounted for) could rewrite a tracked file between our `git checkout
HEAD --` and our subsequent snapshot, and we'd silently bake the drift
into the baseline and the afterEach restore loop would maintain it
forever.
This commit adds the missing post-heal check immediately after the
`git checkout HEAD --`. On CI it throws with `drifted-baseline guard:
post-heal diff failed` citing every offending path; off-CI it warns so
developers iterating on a dirty tree aren't blocked.
Also:
- Rephrases the `isBenignPathspec` catch comment to describe the
realistic case (belt-and-braces against a rm race, not the normal
flow) now that partitionTrackedPaths pre-filters untracked paths.
The branch is intentionally kept — cheap to tolerate, and it
guards against a race that's hard to rule out in a shared worktree.
- Red-green unit coverage: new tests use a counter-based git shim
that fails the N-th `diff --quiet` invocation, so we can target
the post-heal diff independently of the off-CI pre-checkout diff.
- CI throws on drift with the advertised message
- off-CI warns (does not throw) with the same message
- no false-positive on a clean tracked path
Verified red without the guard, green with it.
Run 24602657301 showed unit (20.x) still emitting
`Timeout calling "onTaskUpdate"` during @copilotkit/showcase-scripts:test
teardown — 1008/1008 tests passing, then ELIFECYCLE. The previous fix
switched the pool from threads to forks but left the default fork-per-file
behavior, so every file teardown tore down and reestablished the
parent↔child RPC channel. Under Node 20 the channel occasionally fails
to reestablish within the 10s default teardownTimeout — a known vitest
issue (https://github.com/vitest-dev/vitest/issues/6129).
Two defenses:
- poolOptions.forks.singleFork: true — one long-lived fork across all
test files (combined with the existing fileParallelism: false, files
still run sequentially). The RPC channel stays warm for the whole
run instead of being torn down and reestablished between every file,
which eliminates the per-file teardown as a surface for the race.
- teardownTimeout / hookTimeout bumped from the 10s default to 30s,
matching testTimeout. A slow teardown can't be the bottleneck that
kills the run.
This is the most conservative setting short of dropping to a single-
thread pool entirely. Node 22 / 24 unaffected.
Three small CR8 fixes bundled by area (crewai-crews Python):
* aimock_toggle.py: prod-guard WARNING now uses a grep-friendly prefix
`aimock_toggle: DISABLED (prod-guard: VAR=value)`. Rationale in-file:
`ENV=production` is a common shell-profile default that can silently
disable the toggle in dev/CI without the operator realizing; a
consistent greppable prefix lets them find unexpected refusals without
reading every WARNING line. Strengthened the refusal test to assert
exactly one WARNING is emitted with that prefix (regression guard
against future iteration of _PROD_ENV_VARS firing past the first match).
* agent_server.py: added a `CORSMiddleware` comment documenting why
`allow_origins=["*"]` is intentional for this LOCAL DEMO STARTER — the
agent binds to localhost:8000 (or :8123 inside a generated starter
container) and is only reached by the Next.js frontend during
development. Pointed at `.env.example` / a future `CORS_ORIGIN` env var
as the path for real deployments. No behavior change.
* test_aimock_toggle.py: `test_default_arg_path_mutates_os_environ` now
`monkeypatch.delenv`s every entry in `_PROD_ENV_VARS` before asserting
`enabled is True`. A CI runner or dev shell that exports e.g.
`ENV=production` or `RAILWAY_ENVIRONMENT=production` would otherwise
silently flip this test from "toggle enabled" to "prod-guard refused"
and the enabled-True assertion would fail for the wrong reason.
monkeypatch restores on teardown so other tests are unaffected.
Regenerated crewai-crews starter to mirror the package changes.
The R6 "scope NODE_ENV to Next.js invocation only" fix landed for the
Python Dockerfile template but missed the .NET / Java / TypeScript
templates — they still set `ENV NODE_ENV=production` at the image level,
which leaks into every child process. That's surprising for the TS
templates specifically (mastra dev, langgraph-cli dev, tsx) because those
frameworks gate HMR / source maps / telemetry on `NODE_ENV` and should
NOT see "production" when they're running as the agent backend behind
Next.js.
Fix: remove image-level `ENV NODE_ENV=production` from all three non-Python
Dockerfile templates. The entrypoint template already binds
`env NODE_ENV=production npx next start` so Next.js still sees the value —
only the child agent processes get their environment back. Added a lang-
specific comment to each template explaining the intent and calling out
the affected child processes.
Regenerated the 5 affected starter Dockerfiles
(claude-sdk-typescript, langgraph-typescript, mastra, ms-agent-dotnet,
spring-ai) via `tsx generate-starters.ts`; diffs are byte-identical to the
template change.
Blocking fix: aimock serves `/__aimock/health` + `/v1/*` only — probing `/`
returns HTTP 404 which fails `curl -sf`, so the readiness loop never broke
and every run hard-failed at "Start aimock". Switched all three probes
(startup loop, startup verify, pre-Playwright re-probe) to
`/__aimock/health`.
Also tightened the workflow in four smaller ways:
* Added `--max-time 2 --connect-timeout 1` to every curl probe (aimock
readiness + verify, agent readiness + verify, dev-server readiness +
verify, pre-Playwright re-probe). A hung socket can't blow the loop's
iteration budget now.
* Shell slug extractor (`grep -oE`) and job-level `startsWith` gate now
agree on "first token only": the extractor anchors at `^` instead of
allowing a leading-whitespace alternative. Header comment updated so the
two-layer defense-in-depth rationale reflects the now-identical behavior.
* Captured aimock PID into `$GITHUB_ENV` and added an `if: always()`
"Re-check aimock liveness after Playwright" step that `kill -0`s the PID
after tests. If aimock OOM'd mid-Playwright, the job now fails loudly
instead of trusting cached / fall-through results.
* Removed dead `OPENAI_BASE_URL` / `OPENAI_API_KEY` env on the
`Run Playwright tests` step (Next.js was already running from the earlier
step with these inline on that process; env on `npx playwright test`
didn't flow anywhere). Added comment explaining why setting them here
would be misleading.
* Expanded comment on `workflow_dispatch.inputs.slug.type: choice` to
document that single-choice UI preselect IS intentional (refused the
sentinel-value footgun).
Propagates the crewai-crews aimock_toggle.py (adds RAILWAY_ENVIRONMENT_NAME
to `_PROD_ENV_VARS` + updated header comment) and .env.example (full
prod-guard var enumeration) from packages/ to starters/ via
generate-starters.ts. Only crewai-crews ships aimock_toggle.py today so
only that starter changes; other starters re-generate byte-identical.
Verified: `tsx generate-starters.ts --check` OK across all 17 starters.
- Add RAILWAY_ENVIRONMENT_NAME to `_PROD_ENV_VARS`. Railway exposes BOTH
the legacy `RAILWAY_ENVIRONMENT` and the newer canonical
`RAILWAY_ENVIRONMENT_NAME`; both can hold "production" on a prod service,
so both must be guarded. Missing this one was a production-aimock-leak
footgun for anyone hosting on newer Railway service templates.
- Update the parametrized prod-guard tests (both "refuses" + "silent
no-op" variants) to cover RAILWAY_ENVIRONMENT_NAME so new vars added to
`_PROD_ENV_VARS` can't drift from the test matrix.
- Sync .env.example comment block to enumerate ALL 8 prod-guard vars
(NODE_ENV, ENV, ENVIRONMENT, APP_ENV, DEPLOY_ENV, PYTHON_ENV,
RAILWAY_ENVIRONMENT, RAILWAY_ENVIRONMENT_NAME). The old comment said
"NODE_ENV or ENV" which was misleading — operators reading the .env
scaffold had no way to discover the full guard surface.
- Update outdated "Local dev on macOS ships Python 3.9" header comment
in aimock_toggle.py. CI matrix is 3.10/3.12 and requirements.txt
enforces 3.10+ via marker-gated typing_extensions.
Tests: 94 pass (was 88, +6 from RAILWAY_ENVIRONMENT_NAME parametrize entries
across both refuse + silent-no-op variants).
- Replace `npm bin -g` with `npm prefix -g` + /bin/aimock. `npm bin` was
removed in npm 9.0.0 and setup-node@v4 with node 22.x ships npm 10+, so
the old form exited with "Unknown command: 'bin'" and the existence
check below fired on every run, hard-failing the workflow.
- Replace `pip install --only-binary :all:` with `--prefer-binary`. CrewAI's
transitive graph (tiktoken / chromadb / litellm cadence releases) ships
sdist-only revisions often enough that `--only-binary :all:` made the
workflow fail-loud with "Could not find a version that satisfies the
requirement" on otherwise-valid requirements.txt. `--prefer-binary`
keeps the wheel-first preference while letting sdist-only deps install;
residual source-build-hook risk is bounded by the author_association gate.
- Document the comment-trigger TOCTOU in the residual-trust-model header:
there is a window between `/test-aimock` (reviewing diff D1) and
`pulls.get` (resolves whatever HEAD is current), so a force-push of
malicious content in between wins. GitHub Actions has no native
comment-time SHA pin, so the mitigation is the author_association gate
+ social contract — documenting accepts the known residual risk.
- Drop dead `else APP_MODULE=agent:app` branch. All currently-dispatchable
slugs ship src/agent_server.py (enforced by the ships_toggle check
above), so the legacy path was unreachable. Fail loud with `::error::`
instead of silently falling through to a guessed module name.
- Capture aimock PID and `kill -0` inside the readiness loop so an aimock
that crashes on startup fails in ~1s instead of burning the full 20s
polling a dead process.
- Rewrite restoreFromGitHead JSDoc to match actual behavior for tracked
vs untracked paths (including the on-CI throw / off-CI warn for an
entirely-untracked input list).
- Rephrase internal review-round markers ("CR4/CR5 HIGH/MEDIUM") in
test-cleanup.ts and test-cleanup.test.ts to describe the behavior or
invariant being guarded instead of the review history.
- Point the WINDOWS comment at the sibling test files that actually
invoke npx (this module itself doesn't).
- Drop the "belt-and-braces" afterAll disclaimer from
bundle-demo-content.test.ts and generate-registry.test.ts — the
afterAll(restore) pattern is self-explanatory.
unit (20.x) CI was failing with 'Timeout calling onTaskUpdate' during
the showcase-scripts test run — an unhandled vitest error that causes
ELIFECYCLE after otherwise-green tests. Reproducible on Node 20 only;
22.x and 24.x are green on the same code.
Root cause: vitest's default thread-based worker pool times out on the
parent-worker RPC channel when a test file spawns many subprocesses
(validate-pins.test.ts runs 134 tests each invoking a subprocess;
create-integration / generate-registry / bundle-demo-content each
spawn npx tsx via execFileSync). Under Node 20 the stdio / signal
traffic from these children contends with the worker-thread RPC
channel and surfaces as an unhandled timeout mid-suite.
Switch to pool: 'forks' — the fork pool uses node IPC for the RPC
rather than worker-thread messageports and is robust under the same
load. fileParallelism: false keeps files sequential so shared-env /
tmp-dir mutations don't race, but each file now gets its own fresh
fork so one file's subprocess churn can't stall the RPC for
subsequent files. Node 22/24 unaffected either way.
Also pipes stdio explicitly on every git subprocess in
test-cleanup.test.ts — inherited stdio on a fork vitest worker
interleaves with the worker's own stdout/stderr and is another input
to the RPC contention under Node 20.
generate-registry.ts writes to shell/src/data/registry.json and
shell/src/data/constraints.json. bundle-demo-content.ts writes to
shell/src/data/demo-content.json. All three files are tracked, and
without restoration every run leaks regenerated JSON into the working
tree.
This commit:
- replaces execSync-with-path-interpolation invocations with
execFileSync argv form via runGenerator() / runBundler() helpers;
eliminates any shell-parser involvement (clean hygiene even when
the interpolated constant happens to be safe today).
- snapshots the three data files in beforeAll, restores them in
afterEach + afterAll, adds a regression-guard test that proves the
hooks actually heal drift (sentinel append + bit-exact in-memory
comparison), and a terminal safety-net bit-for-bit check.
- drops a redundant bundler pre-run in beforeAll (test 1 exercises
the bundler itself), and replaces byte-length sentinel checks with
content-level Buffer.concat assertions so the regression guard
survives any hypothetical fs shim that updates stat but not bytes.
create-integration generates an integration package in
showcase/test-integration-tmp and mutates existing workflow YAMLs in
.github/workflows/ (showcase_deploy.yml, showcase_drift-detection.yml,
starter-smoke.yml) to register the new slug. Without restoration, both
the generated tmp package and the workflow-YAML drift leak into the
working tree on every run, and the workflow-YAML drift in particular
breaks pnpm run check on subsequent test invocations.
This commit wires FileSnapshotRestorer + restoreFromGitHead into the
test so:
- tracked workflow files are restored to HEAD before each test run
- the tmp output dir is deleted eagerly in afterEach (independent of
the generator — no reliance on its internal cleanup)
- a regression-guard test proves the snapshot/restore hooks actually
heal drift via a sentinel append + bit-exact assertion against
the in-memory snapshot (not a re-read of disk, which would
silently agree with a buggy restore()).
- a terminal safety net re-asserts every snapshotted file is
byte-identical to its baseline at the end of the suite.
Introduces FileSnapshotRestorer + restoreFromGitHead helpers used by the
showcase test suites to snapshot tracked files in beforeAll and restore
them in afterEach / afterAll. Several of our test scripts invoke real
generators (create-integration, generate-registry, bundle-demo-content)
that write to tracked files outside any tmp dir: .github/workflows/ and
showcase/shell/src/data/*.json. Without explicit restoration these writes
leak into the working tree on every nx run-many -t test and, under Node
20 + vitest worker pools, the accumulated drift races the worker-RPC
channel surfacing as 'Timeout calling onTaskUpdate' -> ELIFECYCLE on CI.
Highlights:
- FileSnapshotRestorer captures bytes at snapshot time, rewrites only
drifted files via atomic temp+rename, and sweeps leftover
.<basename>.<hex>.tmp stragglers scoped to the snapshotted basenames
(no more whole-directory unlink).
- restoreFromGitHead uses execFileSync with a frozen env (GIT_*
scrubbed, PATH/HOME preserved) to heal a working tree left dirty by
a crashed prior run before we snapshot.
- On CI, a baseline that drifts after the pre-snapshot heal is a hard
error (git binary missing, sandbox, etc.); off-CI it warns instead
of blocking local iteration.
- Narrow catch in the git path partitioner: only genuine 'not in
index' pathspec errors are treated as untracked; ENOENT / EACCES /
non-exit-1 failures re-raise so sandbox and missing-binary cases
don't get silently swallowed and lock in a drifted baseline.
- test-cleanup.test.ts itself strips GIT_* from child env when it
creates tmp repos — pre-commit hooks (lefthook) run with GIT_DIR /
GIT_INDEX_FILE set, which would cause tmp-repo 'git commit' calls
to ignore cwd and write to the HOST working-tree HEAD. Without the
scrub, running 'git commit' itself silently accumulates 'initial' /
'init' commits on the real repo.
Also pulls scripts-dir / repo-root / data-dir constants into a shared
paths.ts so future layout changes flip in one place.
Downstream of the template refactor: Dockerfile + entrypoint.sh across
all 17 starters regenerated to pick up the generic NODE_ENV comment,
the `env NODE_ENV=production` invocation form, and the langgraph-only
gating of `/app/.langgraph_api`. Non-langgraph starters lose the
spurious mkdir layer they never used.
- Reword the NODE_ENV=production comment in Dockerfile + entrypoint
templates to generic language. The previous comment mentioned
`aimock_toggle.py refuses to apply` — that text got rendered
verbatim into .NET / TypeScript / Java starters via the shared
template, misleading readers who don't have a Python toggle. New
comment describes the general behavior (image-level NODE_ENV leaks
into every child process, most of which don't interpret it like
Next.js does).
- Use `env NODE_ENV=production npx next start` instead of bare
`NODE_ENV=production npx next start` in entrypoint.sh. `env` prefix
is the syntactically robust form across shells.
- Gate the `RUN mkdir -p /app/.langgraph_api` layer to langgraph-*
starters only via a new `{{LANGGRAPH_MKDIR}}` template variable.
Creating that directory in crewai / agno / pydantic-ai / claude-sdk
/ etc. was copy-paste residue — langgraph_cli is the only thing
that ever writes there.
- Apply the same NODE_ENV rewording to the crewai-crews SOURCE
Dockerfile + entrypoint (the templates' downstream consumers
regenerate via generate-starters, but the source package is hand-
maintained).
- Extend `_PROD_ENV_VARS` from {NODE_ENV, ENV} to also include
ENVIRONMENT, APP_ENV, DEPLOY_ENV, PYTHON_ENV, RAILWAY_ENVIRONMENT.
Railway is this project's primary hosting target (per CLAUDE.md:
"Always Railway for hosting") so its env var was a prominent gap.
The other additions cover common PaaS / framework conventions —
keep the guard broad (cheap) vs narrow (production-aimock-leak
footgun if someone deploys via a convention we don't recognize).
- Parametrize the prod-guard tests symmetrically across all prod vars.
Both the "refuses with warning" and "silent when URL unset" contracts
now run on NODE_ENV / ENV / ENVIRONMENT / APP_ENV / DEPLOY_ENV /
PYTHON_ENV / RAILWAY_ENVIRONMENT so a future var add that forgets
the silent-path fix would fail here.
- Collapse `_classify`'s "empty" + "whitespace" buckets into a single
"blank" label. The two never produced different behavior at any
use site; keeping them distinct was noise in the state machine.
Diagnostic strings still use "empty or whitespace-only" for the
operator-facing message (the input distinction doesn't matter; the
fix-your-env-value message is identical for both).
Test run: 88 passed (was 78 before; +10 from new prod-var parametrize).
- Add `persist-credentials: false` to the PR-HEAD checkout. Prevents the
workflow's GITHUB_TOKEN from being left in .git/config where attacker-
controlled PR-HEAD code (pip build hooks, Playwright fixtures) could
read it. Defense-in-depth on top of `permissions: contents: read`.
- Guard the Resolve PR HEAD step with `pr.state === 'open'`. A trusted
commenter typing `/test-aimock` on a stale closed PR would otherwise
re-exec the old HEAD — wasting CI or re-running known-bad code.
- Add a residual-trust-model header comment to the workflow documenting
the social contract (trusted commenter reviews PR diff before typing
the command; not a security boundary against a rogue OWNER/MEMBER).
- Tighten the Actions-level gate from `contains(body, '/test-aimock ')`
to `startsWith(body, '/test-aimock ')`. Substring matches fire on
mentions inside code blocks / quoted replies; startsWith requires
`/test-aimock ` at the very start of the comment.
- Replace `npx @copilotkit/aimock@...` with a direct invocation of the
globally-installed binary via `$(npm bin -g)/aimock`. npx may re-resolve
the spec against the registry, defeating both `--ignore-scripts` (which
doesn't propagate to npx's transient install) and the caret pin if a
new patch published between global install and npx.
- Add a second aimock liveness probe right before Playwright runs. If
aimock died between initial readiness and test execution, Playwright
would silently fall through to real OpenAI (since OPENAI_BASE_URL
points at a dead port). Fail loud instead.
- Remove the dead `/` fallback in the Python agent health check. FastAPI's
root is typically a POST endpoint that always fails `curl -sf` — the
fallback was copy-paste residue that added no signal.
- Pin `pnpm/action-setup` to `@v4.4.0` matching showcase_validate.yml
(was floating `@v4`).
- Make workflow_dispatch `slug` input required (no default). A hidden
default to crewai-crews contradicted the "no silent fallback" claim
enforced by the comment-path extractor.
- Reword the Next.js OPENAI_BASE_URL comment to acknowledge it's
defensive-only in the CrewAI showcase (Next proxies via runtime, not
direct OpenAI); the value still prevents a future route-level leak.
- Add `--only-binary :all:` to the pip install in showcase_validate.yml's
python-unit-tests job, mirroring the hardening already in the aimock-e2e
sibling workflow. This job runs on EVERY PR automatically without a
trusted-commenter gate, so the broader exposure deserves the same
wheel-only install defense against PR-controlled setup.py hooks.
- Drop the unconditional per-package requirements.txt install. Today's
tests are stdlib-only (+ typing_extensions fallback path); installing
the full CrewAI graph on every PR run was a ~30-60s tax with no benefit.
Install only pytest + typing_extensions. Comment documents the upgrade
path for future tests that need package runtime deps.
- Drop the now-unused pip cache on setup-python (keyed on a never-used
requirements.txt would be pure overhead).
Regenerate all 17 starters via `tsx generate-starters.ts` to propagate:
- aimock_toggle.py: prod-guard only fires when AIMOCK_URL is set;
dummy key renamed to `sk-aimock-dev-ci-only` (from previous file).
- agent_server.py: removed dead `main()`/RELOAD entrypoint (non-langgraph
Python starters).
- Dockerfile + entrypoint.sh: `NODE_ENV=production` moved from global
image ENV onto the `npx next start` invocation so the Python agent
doesn't inherit the prod-guard that would silently disable aimock.
- langgraph-fastapi/.env.example: source-package port fix already propagated
(port was already correct at :8123 for this starter; no content change
beyond reflowing the comment from the source package).
The starter regen is a single atomic commit (separate from the source
edits) so reviewers can inspect which input edit produced which starter
output without wading through unrelated edits in the same diff.
LOW CR5-2 L4: Drop the `test:python` npm script. CI runs pytest directly
(.github/workflows/showcase_validate.yml invokes `python -m pytest
tests/python/ -v` in each package dir) and the script isn't wired into
any Nx target — it's an orphan knob that would only cause drift between
"what CI runs" and "what a contributor gets from `npm test:python`".
LOW CR5-6 L1 (docs): Document the AIMOCK_URL + NODE_ENV=production
interaction in .env.example so operators know the prod-guard refuses to
apply the toggle even when AIMOCK_URL is set, and understand why.
M2: The imported `AGENT_URL_LOCALHOST_8000_RE` carries the /g flag, so
`.test()` / `.exec()` advance lastIndex — sharing the exact instance
across the describe-loop iterations coupled any two iterations that
happened to touch it. Clone the regex per-call via a `re8000()` factory
that returns `new RegExp(source, flags)` each time. Same treatment for
the companion `re8123()`.
LOW: Replace the conditional `it()` registration (only created when the
package .env.example contained `:8000`) with an UNCONDITIONAL `it()`
that internally early-returns when there's nothing to rewrite. Vitest's
reporter registers the test either way so a future regression where every
package suddenly stopped matching the pattern would surface as "test
skipped" rather than silently vanishing from CI output.
M9: `ENV NODE_ENV=production` at the Dockerfile level propagates to every
process spawned by the entrypoint — including the Python agent. The
aimock toggle (aimock_toggle.py) refuses to apply when `NODE_ENV=production`
is set, so a Docker-based smoke test with `AIMOCK_URL` exported would
silently get a no-op toggle and hit real OpenAI instead of aimock.
Move `NODE_ENV=production` out of the global Dockerfile ENV blocks and
inline it on the `npx next start` invocation in entrypoint.sh. Next.js
still treats the server as production (that's all Next looks at), but the
Python sibling process is no longer subject to a production-guard that
only ever made sense for the JS runtime.
Affected: crewai-crews package Dockerfile + entrypoint.sh, plus the
starter template (Dockerfile.python + entrypoint.template.sh) — the
starter regen in a follow-up commit propagates this to all 17 starters.
M7: Every execution path (package `pnpm dev`, starter `pnpm dev`, Docker
entrypoint, CI workflow) invokes `python -m uvicorn agent_server:app ...`
directly via CLI. The `main()` wrapper reading PORT/RELOAD from env was
dead code that:
- was never exercised by any CI path;
- defaulted PORT=8000 while starter dev scripts bound :8123 (drift);
- carried a RELOAD gate whose only purpose was to guard `reload=True`
in a function that never ran.
Remove `main()` and the `if __name__ == "__main__"` dispatch entirely, plus
the now-unused `os` import. A comment at the bottom of the module documents
why no CLI entrypoint lives here so a future contributor doesn't add one
"for consistency" with other showcase packages.
M6: The prod-guard (`NODE_ENV=production` / `ENV=production` refuses to apply
the toggle) was entering its refusal branch even when AIMOCK_URL was unset
— returning a misleading `"aimock toggle refused"` reason when in fact
nothing was being applied. Move the `aimock_url` non-empty check BEFORE the
prod-guard block; the unset-URL path now falls through to the normal "not
configured" diagnostic, reserving the refusal message for actual refusals.
Regression pinned: `test_production_env_without_aimock_url_is_silent_no_op`
now asserts (a) no WARNING is logged, (b) reason text does not contain
"refused", and (c) neither OPENAI_BASE_URL nor LITELLM_API_BASE is mutated.
LOW: Rename `_AIMOCK_DUMMY_KEY` from `"sk-aimock-DO-NOT-SET-AIMOCK-URL-IN-PROD"`
to `"sk-aimock-dev-ci-only"`. The old name embedded a directive aimed at
operators but the value is never user-provided — the toggle itself injects
it when AIMOCK_URL is set and the prod-guard refuses production anyway.
Short, honest, still satisfies SDK versions that validate `sk-` prefix.
Test `test_dummy_key_does_not_mislead_with_replace_in_prod_suffix` extended
to also reject the old "DO-NOT-SET" directive.
LOW: Pin `LITELLM_API_BASE not in env` in
`test_production_env_refuses_toggle_even_with_aimock_url` so a future
regression that mutates LITELLM_API_BASE in the refusal branch (symmetric
with the existing OPENAI_BASE_URL check) fails the test.
The package's own `pnpm dev` script binds langgraph_cli on port 8123
(via package.json "dev"), but the .env.example documented port 8000.
A contributor running `cp .env.example .env && pnpm dev` would hit
ECONNREFUSED out of the box because nothing binds :8000 in this package.
Generated starters (showcase/starters/langgraph-fastapi/.env.example)
already point at :8123; this aligns the package-side .env.example with
both its own dev script and the scaffolded starter output.
HIGH fixes from CR5 round:
- H1: `pip install --only-binary :all:` forces wheel-only installs, blocking
source-build hooks (setup.py / PEP 517 build backends) from executing on
the runner. Unlike npm/pnpm there is no `pip --ignore-scripts` equivalent;
wheels are the closest mitigation. Residual risk documented inline.
- H2: Tighten `/test-aimock` match so `/test-aimocker` / `/test-aimock-like-this`
do NOT trigger runs. Require trailing space via job-level `if:` AND enforce
whole-word boundary in the shell-level slug extractor (belt-and-suspenders).
Remove the silent `crewai-crews` default fallback — a missing slug now FAILS
the workflow instead of silently running against the default.
- M4: Scope write perms (pull-requests + issues) to a SEPARATE `post-result`
job that only runs the comment post. The heavy test job now runs with
`contents: read` only, so a compromised transitive dep from `pip install`
on a PR-controlled requirements.txt cannot mutate PRs/issues with the
workflow token.
- M8: Use `npx next dev` directly in "Start dev server" step. `pnpm dev`
would spawn a SECOND uvicorn on :8000 via concurrently, racing the
already-bound agent from the prior step and silently picking whichever
won the port.
LOW fixes:
- Pin `npx` invocation to fully-scoped `@copilotkit/aimock@^1.14.3` so a
squatter on the unscoped `aimock` name can't be silently picked up if the
global install fails.
- Add rationale comment explaining why `steps.slug.outputs.slug` interpolation
is safe (slug is whitelisted + dir-validated before use).
showcase_aimock-e2e.yml:
- workflow_dispatch slug is now a choice-type enum restricted to Python
packages that ship aimock_toggle.py; a TS (mastra) or Java (spring-ai)
slug would have skipped the Python agent start step and then failed with
a misleading Playwright timeout. For comment-trigger paths, the pkg-type
step short-circuits with a clear ::error:: when the slug does not ship
aimock_toggle.py.
- Slug parsing replaced PCRE grep -oP '\\K\\S+' with a POSIX-safe
grep -oE + sed pipeline so BSD/Alpine grep works too (future-proof
against runner image changes).
- aimock pinned to @copilotkit/aimock@^1.14.3 (was @latest) with
--ignore-scripts; unpinned @latest let a bad aimock publish silently
poison CI for everyone.
- pnpm install now runs with --ignore-scripts — trusted commenter triggers
/test-aimock on untrusted PR content, so postinstall scripts must not
run on the runner.
- OPENAI_BASE_URL is no longer pre-set for Python agent start on packages
that ship aimock_toggle.py — forcing the toggle itself to do the
redirection proves the toggle works rather than masking it with env
already set.
- Dead AIMOCK_URL export removed from the Next.js dev-server step (the
runtime only reads OPENAI_BASE_URL).
- Python agent health-check extended 60s -> 90s (45 iter * 2s) for CrewAI
cold imports.
- actions/setup-python caches pip keyed on requirements.txt.
- github-script Post-result step passes slug + job status via env rather
than ${{ }} interpolation — even though the slug is already whitelisted,
the env pattern is the defensive default for dynamic values.
showcase_validate.yml:
- python-unit-tests is now a matrix on Python 3.10 + 3.12 (fail-fast
disabled) so the typing_extensions fallback branch gets CI coverage;
previously only 3.12 ran.
- pip install per package is a HARD FAIL (was ::warning:: + continue);
hiding broken requirements.txt behind a warning let a package ship green
with unresolvable runtime deps. A missing requirements.txt is still
handled gracefully via the -f guard.
- pip caching added via actions/setup-python.
Regenerated from showcase/scripts/generate-starters.ts with the updated
propagation rules. Every starter whose package ships .env.example now carries
one (port-rewritten to :8123 where AGENT_URL points at localhost), and the
crewai-crews starter additionally carries aimock_toggle.py alongside the
agent_server.
OPENAI_API_KEY placeholder text in every package + starter .env.example now
reads 'replace-with-your-key' (not a fake sk-... value), so users who skip
editing get a loud 401 from real OpenAI instead of a silent aimock success
masking misconfig.
Extends generate-starters.ts so every starter whose package ships .env.example
gets the file copied through (not just non-langgraph Python), and ships the
aimock_toggle.py alongside agent_server.py for any Python package that has one.
The .env.example copy rewrites AGENT_URL=http://(localhost|127.0.0.1):8000 to
:8123 during propagation because starter dev scripts bind the agent on 8123
while package dev scripts bind on 8000.
The port-rewrite regex is now exported as AGENT_URL_LOCALHOST_8000_RE and
imported by the starter-consistency test, so the two sides cannot drift.
Previously the test used a broader [^:\/]+ host pattern while the generator
correctly narrowed to localhost/127.0.0.1 — a future package documenting
AGENT_URL=https://api.corp.example:8000 would have tripped the test while the
generator correctly preserved the non-localhost hostname.
Adds an env-controlled aimock toggle for the CrewAI Python agent so dev/CI
traffic can be redirected to a local aimock server while production (AIMOCK_URL
unset) is unchanged. When AIMOCK_URL is set, the toggle populates
OPENAI_BASE_URL + LITELLM_API_BASE and injects a dummy OPENAI_API_KEY if one
isn't already set.
- aimock_toggle.py: configure_aimock() with production-env guard — NODE_ENV /
ENV set to prod/production refuses to apply the toggle (fail-safe WARNING)
so a misconfigured deploy can't silently redirect real traffic through a
mock. Dummy key renamed 'sk-aimock-DO-NOT-SET-AIMOCK-URL-IN-PROD' (was
'sk-aimock-dev-only-REPLACE-IN-PROD' — the old name implied operators
should substitute, but the value is never user-facing). Override warning
now symmetric across OPENAI_BASE_URL and LITELLM_API_BASE (both are
overwritten; both must report overrides).
- agent_server.py: uvicorn reload flag gated on RELOAD env (defaults false)
so 'python agent_server.py' in prod doesn't spawn the reloader.
- requirements.txt: typing_extensions>=4.6; python_version < "3.11" so the
NotRequired fallback only installs on older Pythons.
- Dockerfile: updated COPY comment to match reality (two COPY lines, not
three — agents/ is kept separate because it churns independently).
- pytest.ini + conftest.py cleanup: src/ now on the import path via
pythonpath= declarative config (pytest 7+), removing the sys.path.insert
mutation that leaked past test collection.
- 55 pytest cases (was 38), including red-green proofs for: prod-env guard,
LITELLM override warning, dummy-key rename, whitespace-wrapped falsy
USE_AIMOCK values, and the typing_extensions fallback import path.
## Summary
- Suppress per-run `✅ *Showcase deploy*: N service(s)
deployed to Railway` Slack posts from `showcase_deploy.yml`. During bulk
drift rebuilds, the smoke-monitor fans out one `showcase_deploy.yml` run
per stale service (up to ~18), producing ~18 noise messages in
#oss-alerts per drift cycle. The top-line `📦 Image drift
detected — N rebuilds triggered` in `showcase_smoke-monitor.yml` already
covers the aggregate success case.
- Also suppress the `no changes detected, nothing to deploy` post — rare
and uninformative.
- Preserve failure messages (pre-build failure, build failure) and the
mid-matrix cancellation info message (state transition worth seeing).
- Harden the Post step with `env.SLACK_WEBHOOK_OSS_ALERTS != ''` guard
so a missing secret fails closed rather than erroring the action.
Channel policy alignment: #oss-alerts surfaces only actionable state
(failures + transitions). Ad-hoc single-service successes stay quiet —
the Actions UI is the source of truth.
## Test plan
- [x] `actionlint .github/workflows/showcase_deploy.yml` — no new
findings introduced by this diff (pre-existing SC2086 style infos
unrelated to the edited region)
- [x] Local dry-run simulation of every branch in the payload step:
success (bulk) → suppressed, no-changes → suppressed, build-failure →
posted, detect/lockfile-failure → posted, mid-matrix cancel → posted,
pre-build cancel → suppressed
- [ ] Observe in production: next drift rebuild cycle should post one
top-line drift message and zero per-service greens; any failing leg
still surfaces
## Out of scope
- Smoke monitor verbosity (already correct: red + recovery only)
- Drift detection logic (unchanged)
- `crewai-crews` 502 investigation (unrelated)
The showcase_deploy.yml notify job posted a green success message on
every run. Bulk drift rebuilds fan out one showcase_deploy.yml run per
stale service (up to ~18), so a single drift cycle produced ~18
"Showcase deploy: 1 service(s) deployed to Railway" messages in
#oss-alerts — pure noise.
Align with the channel policy: surface only actionable state
(failures + state transitions). The bulk-rebuild aggregate is already
posted by showcase_smoke-monitor.yml as "📦 Image drift detected
— N rebuilds triggered". Ad-hoc single-service pushes/dispatches stay
quiet on success — the Actions UI is the source of truth.
Kept as-is: failure messages (pre-build and build), mid-matrix
cancellation info (state transition worth humans seeing). Also added an
empty-webhook guard on the Post step so a missing SLACK_WEBHOOK_OSS_ALERTS
secret fails closed instead of erroring out the step.
Second hotfix for #4018 — the Depot `id-token` addition in #4060 was
needed for later but wasn't the actual blocker.
## Root cause
Both new workflows used `secrets.SLACK_WEBHOOK_OSS_ALERTS != ''` in
step-level `if:` expressions. GitHub Actions rejects `secrets.*` as a
named-value inside `if:` on push events with:
Unrecognized named-value: 'secrets'
This aborted the workflow at parse time before any job could spawn (zero
jobs in the API, conclusion=failure). PR CI passed because pull_request
events validate `if:` expressions less strictly.
## Fix
Hoist the webhook into a job-level env var:
```yaml
env:
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK_OSS_ALERTS }}
```
Reference `env.SLACK_WEBHOOK` in every `if:`. `with: webhook:` keys keep
using `secrets.*` (valid context).
## Test plan
- [ ] Merging this triggers showcase_validate.yml on main (workflow
touches itself) — expect jobs to spawn and the Depot runner to start
- [ ] Weekly drift-report next Monday verifies the other workflow
end-to-end
The `secrets.*` context is not a valid named-value inside step-level
`if:` expressions on push events — GitHub Actions rejects it at
workflow parse time with "Unrecognized named-value: 'secrets'", which
caused both showcase_validate.yml and showcase_drift-report.yml to
fail at startup with zero jobs spawned after #4018 + #4060 merged.
Hoist the webhook into a job-level `env: SLACK_WEBHOOK` and reference
`env.SLACK_WEBHOOK` in every step-level `if:`. The `with: webhook:`
keys still use `secrets.*` directly (valid in that context).
PR CI didn't catch this because pull_request events parse if:
expressions less strictly than push events.
Hotfix for #4018.
The two workflows flipped to `runs-on: depot-ubuntu-24.04-4` in #4018
inherited the repo's default `contents: read` permissions, which blocks
Depot's OIDC handshake. Jobs failed to spawn on the first main-branch
push after merge.
Matches the pattern already used by `showcase_deploy.yml`'s Depot job.
## Test plan
- [x] Merging this should let `showcase_validate.yml` spawn a job on its
next trigger (any PR touching `showcase/**` or this workflow itself)
- [x] `showcase_drift-report.yml` next scheduled Monday run will verify
end-to-end
Jordan Ritter