- Narrow the mid-stream except around agent.llm_response_async from a
broad Exception to (openai.APIError, httpx.HTTPError,
asyncio.TimeoutError, pydantic.ValidationError). Programmer bugs
(AttributeError, NameError, TypeError) now propagate instead of
being sanitized as 'Agent run failed: <ClassName>' — keeps real
bugs visible in server logs and stack traces, where they belong.
Expected runtime errors still get the sanitized TEXT_MESSAGE +
RUN_FINISHED treatment so the UI never hangs mid-stream.
- Update _parse_tool_args call-site comment in handle_run: it used
to say 'returns None' when args could not be parsed; the function
now returns ParsedArgs with status in {ok, empty, malformed}, and
the caller already pattern-matches on .usable.
- Fix stale _try_parse_tool docstring that referenced the old
agent.agent_response() API; the adapter has used
llm_response_async() for a while now.
Tests: add test_llm_response_async_programmer_bug_propagates — mocks
llm_response_async to raise AttributeError('typo') and asserts the
exception propagates out of the SSE generator (not sanitized).
Update the existing mid-stream-failure test to raise
asyncio.TimeoutError (a covered exception) instead of RuntimeError
so it continues to exercise the sanitization path.
Add openai.OpenAIError (SDK base class) to the generate_a2ui except
tuple so config-time failures — e.g. OpenAI() construction with unset
or malformed OPENAI_API_KEY — become a structured a2ui_llm_error dict
instead of bubbling through the ADK tool machinery as an uncaught
exception. APIError and friends are subclasses of OpenAIError; the
constructor raises OpenAIError directly, which the narrowed tuple
previously missed. Parity fix with the strands sibling (R4).
Also:
- _A2uiError docstring now acknowledges 3-way sync with strands AND
langroid (all three carry the same TypedDict).
- search_flights docstring relaxed from 'exactly 2 flights' to '2-3
flights' so google-adk is consistent with ms-agent-dotnet (which
returns 3).
- Added test exercising openai.OpenAIError from the OpenAI()
constructor path; asserts a2ui_llm_error with full error shape.
- entrypoint.sh: bounded 10s SIGTERM grace window + SIGKILL fallback
when terminating the surviving sibling; prevents indefinite hang past
the platform SIGKILL grace period and preserves the structured
who-died log line.
- BoundedToolCallingManagerConfig: make the weakKeys() identity-vs-equals
footgun explicit in the comment; add JUnit concurrent-contention tests
(ExecutorService + CountDownLatch) asserting the cap invariant holds
under N-thread race on the same ChatOptions — no two threads exceed
cap-1 and both increment past N.
- WebClientConfig: switch the @Bean-time keepalive check from log-and-
proceed to fail-fast IllegalStateException unless COPILOTKIT_ALLOW_KEEPALIVE
opt-in is present (matches OpenAiApiKeyValidator philosophy). Narrow
isTruthy to 1/true (drop yes) since Spring/Java canonicalize on
true/false. Update tests to reflect the narrowed vocabulary and
fail-fast throw.
- application.properties: collapse the 8-line rationale block duplicating
OpenAiApiKeyValidator javadoc to a one-line pointer (drift risk).
- main.py: before_model_modifier no longer chops user suffix when the
prefix end_marker is missing (mangled/drifted prefix). Leaving
original_text untouched means worst-case is one duplicated signature
(non-stacking) instead of silent data loss.
- main.py: simple_after_model_modifier now logs llm_response.error_message
at WARNING with agent name before returning. Previously Gemini
quota/safety/context-overflow errors were silently swallowed.
- main.py: _A2uiError TypedDict now carries a sync-pointer comment to
the identical TypedDict in strands/src/agents/agent.py.
- entrypoint.sh: added explicit survivor cleanup with bounded 5s grace
window, mirroring the spring-ai pattern. Also escalated the
'both still running' wait -n race branch to ERROR + exit 1 so it
cannot silently mask the real child death.
- tests: added before_model_modifier suffix-preservation test and
simple_after_model_modifier error_message warning log tests.
Updated the test_generate_a2ui docstring to reflect the getattr()
guard pattern rather than the old AttributeError handler wording.
- _try_parse_tool function_call path now accepts (str, bytes, bytearray)
for arguments, matching _parse_tool_args. Real OpenAI/httpx stacks can
deliver bytes; the old narrow isinstance(args, str) guard fell through
to tool_cls(**args) with raw bytes and silently TypeError'd.
- Wrap agent.llm_response_async in try/except inside event_stream. On
failure after RUN_STARTED, emit a sanitized TEXT_MESSAGE triple and
RUN_FINISHED so the frontend never hangs waiting for stream closure.
- Wrap request.json() and RunAgentInput(**body) with explicit handlers
that return structured JSONResponse errors carrying an errorId for
client-side correlation (400 for bad JSON, 422 for schema mismatch).
- Defensive guard on run_input.messages: skip entries whose role or
content are not strings, with a warning. Prevents f-string coercion
of None/complex values into garbage prompts.
- Wrap the content-JSON-path asyncio.to_thread in try/except that emits
a sanitized error + RUN_FINISHED before re-raising, so scheduler-level
failures do not kill the SSE generator mid-stream.
- Duplicate-tool RuntimeError now includes fully qualified
module.Class identities for each colliding ToolMessage, instead of
just bare request names.
Tests: 30/30 pass (added 5 new tests covering bytes args, mid-stream
failure, malformed JSON body, invalid RunAgentInput, non-string
role/content, collision error identity).
entrypoint.sh:
- Disable errexit before wait -n so the diagnostic kill -0 probes and
which-died log line run regardless of child exit code.
- Kill the surviving sibling after one child dies to avoid orphan-reparent.
BoundedToolCallingManagerConfig.java:
- Use Caffeine.newBuilder().weakKeys() to force identity comparison on
cache keys. Without it, two concurrent turns with equal-but-distinct
ChatOptions references would collapse onto one counter and the cap
would fire early on the second turn.
- Narrow counter-cleanup catch from Throwable to Exception so we do not
run arbitrary cleanup code against a JVM in an undefined state
(OutOfMemoryError, StackOverflowError). Errors now unwind unhandled.
WebClientConfig.java:
- Accept 1 / true / yes (case-insensitive, trimmed) as opt-in values
for COPILOTKIT_ALLOW_KEEPALIVE instead of only the literal '1'.
- Add defensive runtime check in the @Bean method that logs ERROR if
jdk.httpclient.keepalive.timeout is not '0' at bean-construction
time — catches the case where the JVM arg path is dropped AND this
class loads after an HttpClient has already been constructed.
OpenAiApiKeyValidator.java / application.properties:
- Tighten comments to explicitly note that Spring treats
'?OPENAI_API_KEY must be set' as the DEFAULT VALUE (non-blank, would
pass a naive blank check) when the env var is unset.
Tests:
- Add equal-but-not-same ChatOptions counter-independence test.
- Replace Throwable-cleanup test with Error-propagation test.
- Add tests for true/yes/case-insensitive opt-in.
- Add tests for isTruthy helper.
- Add @Bean defensive-ERROR-log test.
- Result: 33 tests, all green (up from 26).
Verified:
- mvn test: 33/33 green
- docker build + smoke (integration-smoke -g spring-ai, LOCAL_PORTS=1): passing
Changes to src/agents/agui_adapter.py:
- Extract _execute_backend_tool / _run_backend_tool helpers so the
oai_tool_calls path and the content-JSON path share one sanitization
contract. The _try_parse_tool backend-tool path no longer runs
.handle() barefoot — it goes through _execute_backend_tool which
catches the narrowed (pydantic.ValidationError, ValueError), logs
the traceback server-side, and returns a sanitized JSON error payload.
- Narrow tool-execution except from
(RuntimeError, ValueError, TypeError, KeyError, pydantic.ValidationError)
to (pydantic.ValidationError, ValueError). RuntimeError/KeyError from
unrelated libraries or real config drift now propagate as real bugs
rather than being misclassified as user-facing tool failures.
- Replace dict | None three-way return of _parse_tool_args with a
ParsedArgs dataclass (args + status: 'ok' / 'empty' / 'malformed' +
.usable helper). Call sites pattern-match on status explicitly.
- Treat empty-string raw_args (legacy function_call path) as 'malformed'
instead of 'ok with {}' — firing a tool with empty args produces a
meaningless UI card, same rationale as the oai path.
- Drop TypeError from the json.loads except in _try_parse_tool (it
signals a programmer bug, not plain text). Add an
isinstance(content, (str, bytes, bytearray)) guard that logs a
WARNING rather than silencing the case.
- Drop pydantic.errors.PydanticUserError from tool-instantiation except
lists — it signals malformed model *definitions*, not runtime data
errors, and must surface loudly at startup.
- Freeze _TOOL_BY_NAME behind types.MappingProxyType so post-import
mutation raises TypeError (e.g. accidental monkeypatch / shadowing).
Changes to tests/python/conftest.py:
- Tighten the docstring cross-reference to src/agent_server.py (the
real entry point confirmed to import agents.agui_adapter identically).
Changes to tests/python/test_agui_adapter.py:
- Update helper tests to assert on ParsedArgs.status / .usable / .args.
- Flip empty-string expectation to 'malformed' per the new contract.
- Add: bytes-input parse, unknown-type 'empty' status, MappingProxyType
frozen-ness of _TOOL_BY_NAME, unified-helper happy path, helper-level
sanitization + traceback capture, helper propagates non-narrowed
RuntimeError, _try_parse_tool non-str/bytes guard warns and returns
None, and str(exc) never appears anywhere in the SSE stream bytes.
- Switch sanitized-error test's raise to ValueError (narrowed except
covers it) — previously RuntimeError leaked through by accident.
24/24 tests pass (16 existing + 8 new).
- BoundedToolCallingManagerConfig: use ConcurrentHashMap with atomic
compute() for per-turn counter; short-circuit null ChatOptions to
avoid shared null-key contamination; clear counter on delegate
exception + log before rethrow; make MAX iterations configurable via
copilotkit.tool.max-iterations (@Value); rename to
MAX_TOOL_ITERATIONS_BEFORE_RETURN_DIRECT for unambiguous semantics;
pin Spring-AI version in Javadoc and document invariant.
- WebClientConfig: keep static initializer as defensive belt-and-
suspenders but document entrypoint.sh JVM arg as authoritative path;
warn when a non-zero keepalive timeout is already set.
- application.properties: fail fast when OPENAI_API_KEY is unset via
${OPENAI_API_KEY:?…} placeholder.
- entrypoint.sh: drop bash 5.1+ PID args from 'wait -n'; log which
child (java/node) exited with which code; bump health-probe timeout
30s -> 60s to cover cold-start JVM warmup.
- Add JUnit 5 tests (spring-boot-starter-test, test scope): first/Nth
call, fresh-options reset, null-options safety, counter eviction on
cap and exception, HTTP/1.1 connector pin.
Address CR findings against the strands showcase package:
agent_server.py
- Replace throwaway lambda with named _disabled_instrument that returns
self instead of None (fluent callers no longer AttributeError).
- Add module docstring + inline warnings explaining the strict import
ordering invariant (instrumentor patch must precede ag_ui_strands /
strands imports).
- Add runtime assertion that the patch is actually installed.
- Annotate each # noqa: E402 with the concrete reason.
- Note that no upstream strands-agents issue has been filed yet.
agents/agent.py
- Wrap all module-level side effects (model init, agent construction,
per-thread dict patching) in build_showcase_agent() so import
failures are localized.
- Fail fast with RuntimeError when OPENAI_API_KEY is unset rather than
accepting an empty string silently.
- _ToolCallCapHook: protect _count with threading.Lock, document the
single-thread-at-a-time ag_ui_strands concurrency model, and emit a
logger.warning when the cap trips so operators see loops.
- _HookInjectingAgentDict: override update(), setdefault(), __ior__,
and __or__ in addition to __setitem__ so CPython's bulk-update C
paths (which bypass __setitem__) still route through hook injection.
- Preserve any pre-existing entries in _agents_by_thread when swapping
in the injecting dict (copy into the new dict first).
- Guard against double-injection on re-insert: before adding a cap
hook, check whether the Agent already has one attached.
- Narrow sales_state_from_args exception handler from bare Exception to
(json.JSONDecodeError, AttributeError, TypeError) and log a truncated
excerpt of the offending tool input at WARNING.
- Document that schedule_meeting duration is intentionally defaulted
in the showcase.
- Remove unused BaseModel / Field pydantic imports.
- Replace ASCII banner comments with section headers.
tests/python/
- New pytest suite exercising the cap hook counter semantics (fires at
max+1, resets on BeforeInvocationEvent, stop_event_loop sentinel),
the hook-injection dict (all mutation paths inject; existing entries
preserved; no double-injection on re-insert), and the threading
instrumentor patch (returns self, accepts arbitrary args, actually
replaces the method, agent_server module installs the patch).
- conftest.py stubs out strands / ag_ui_strands / uvicorn / dotenv so
the suite runs in environments where the heavy runtime deps aren't
installed.
- agui_adapter: compute thread_id once so RUN_STARTED/RUN_FINISHED never
disagree when the caller omits it.
- Narrow silent `except Exception: pass` sites, log parse/tool failures
with sanitized context, and drop the dead `agent.agent_response` path
(returns ChatDocument, not ToolMessage).
- Extract `_emit_text_block` helper DRYing the empty-delta guard, and a
`_parse_tool_args` helper with explicit warning on malformed input.
- Assert tool-class request-name uniqueness at import so collisions fail
loudly instead of silently shadowing.
- Replace `str(response)` fallback with documented empty default.
- Add unit tests covering the OAI tool-calls happy path, malformed args,
legacy function_call shape, empty-content skip, thread_id stability,
and tool-class uniqueness (red-green verified).
Spring-AI ignores the showcase-wide OPENAI_BASE_URL because its split
base-url + completions-path model would produce a double-/v1 against
aimock. Introduce SPRING_AI_OPENAI_BASE_URL carrying the host without
/v1 (e.g. http://aimock:4010), default to https://api.openai.com.
Additionally, pin WebClient's JDK HttpClient to HTTP/1.1 and disable
connection pooling:
- Without reactor-netty, Spring-AI's auto-configured WebClient sent
`Upgrade: h2c` on every cleartext request, which some fixtures (aimock)
reject with 404.
- Default JDK HttpClient pooling reused half-closed sockets between
fixture responses, tripping `Connection reset` on tool-result follow-ups.
L3 chat now passes locally against the stack. L4 tool-rendering still
needs additional work (tool-execution loop cap WIP in BoundedToolCallingManagerConfig).
The LLM could enter an infinite get_weather tool-call loop (300+ calls per
invocation) because strands has no upstream max-iterations knob, and
aimock's fuzzy fixture matching returns the same tool_call response
whenever the last user message contains the matched text. Direct OpenAI
usage can exhibit similar fixation under certain prompts.
Mitigations in agents/agent.py:
1. _ToolCallCapHook (HookProvider): counts tool calls per Agent
invocation (reset on BeforeInvocationEvent). On BeforeToolCallEvent
past the cap, sets event.cancel_tool so the tool returns a benign
error result. On AfterToolCallEvent past the cap, sets
invocation_state['request_state']['stop_event_loop'] = True which
strands' event_loop checks to terminate recursion.
2. ToolBehavior(stop_streaming_after_result=True) for get_weather: the
frontend weather card IS the response, so halting after the first
tool result also short-circuits any would-be recursion for the
tool-rendering demo.
3. _HookInjectingAgentDict: ag_ui_strands spawns a fresh Agent per
thread_id from the template WITHOUT copying hooks. Subclass the
per-thread dict and attach a fresh _ToolCallCapHook (stateful) on
every agent insertion.
Verified: L3 and L4 strands smoke tests pass 3x locally.
Langroid's OpenAI-backed LLM returns tool calls on
`response.oai_tool_calls` (OpenAI tools API) or `response.function_call`
(legacy function-calling API) with an empty `response.content`. The prior
adapter only inspected `response.content`, so pure tool-call turns
produced no TOOL_CALL events, the frontend never rendered the weather
card, and L4 (tool-rendering) smoke timed out waiting for an assistant
message.
Synthesize TOOL_CALL_START / TOOL_CALL_ARGS / TOOL_CALL_END events from
`oai_tool_calls` (and `function_call` as a fallback), preserving the
OpenAI-assigned call id and JSON-encoded arguments. For backend tools,
instantiate the matching ToolMessage subclass, execute its `.handle()`
off-thread, and stream the result back as a text message (skipping empty
deltas to stay schema-compliant). Frontend tool calls are forwarded as
events only, letting CopilotKit's client render them.
strands-agents 1.35.0 unconditionally calls ThreadingInstrumentor().instrument()
when constructing its Tracer (strands/telemetry/tracer.py). Combined with the
async model client dispatching work onto ThreadPoolExecutor, this wraps
ThreadPoolExecutor.submit such that it re-enters itself recursively, producing
RecursionError during tool-rendering requests and surfacing as an OpenAI
APIConnectionError.
Setting OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=threading does not help because
strands imports and instruments the class directly, bypassing the entry_point
autoloader. Monkey-patch the instrument() method to a no-op before strands is
imported.
Gemini 2.5-flash, given the SalesPipelineAgent's sales-heavy system prompt,
intermittently returned an empty response (FinishReason.STOP with no content
parts) for weather queries — about 70% of requests, causing the L4
tool-rendering smoke test to time out waiting for the assistant message.
Root causes and fixes:
1. System prompt priming: WEATHER guidance was buried after SALES TODOS.
Reordered to lead with a direct weather instruction ("When the user
asks about the weather, call the get_weather tool") and added an
explicit "ALWAYS provide a textual response after any tool call"
guard. The model now reliably invokes get_weather and emits a summary.
2. after_model_modifier over-termination: the old check ended the
invocation whenever parts[0].text was truthy, which fired on
streaming partial events and on mixed text+function_call responses,
stranding pending tool calls. Now skip partial events, and only
terminate when the response is role=model AND has text AND has no
pending function_call.
3. Added ADK_DISABLE_PROGRESSIVE_SSE_STREAMING=1 in entrypoint.sh to
suppress the ADK's progressive-SSE path, which aggregates cleaner
final events (removes the 'The last event is partial' warnings we
were seeing on every run).
Verified: 10/10 direct backend calls now return TOOL_CALL + TEXT_MESSAGE,
5/5 repeated L4 playwright runs pass, and L3 (agentic-chat /Hello/) still
passes.
SharedStateAgent forced ResponseFormat=json-schema for every request whenever
ag_ui_state was present in the options (which it always is for CopilotKit-
wrapped demos). For plain chat prompts like 'hello' the model dutifully
produced a SalesStateSnapshot shell that failed TryDeserialize, the code
hit yield break, and nothing was streamed to the client -- Playwright's L3
smoke waited 60s for [data-testid=copilot-assistant-message] and timed out.
Bypass the two-pass flow (run the agent normally) unless ag_ui_state.todos
is a non-empty array, i.e. there's actual sales pipeline data to sync. The
two-pass logic still runs for sales-pipeline demos that populate todos.
The AG-UI adapter was emitting TextMessageContentEvent with delta=""
whenever the Langroid response had empty content (e.g. a pure tool-call
turn) or a backend tool handler returned nothing. ag_ui.core validates
delta as min-length 1, so the event construction crashed with a pydantic
ValidationError and the SSE stream blew up mid-run, taking out the
tool-rendering L4 smoke test.
Guard both emission sites by skipping the entire
TEXT_MESSAGE_START/CONTENT/END block when the delta would be empty,
rather than padding with whitespace (which would leak a spurious blank
assistant bubble into the UI).
## Summary
Adds a one-command local smoke harness so the full 17-integration suite
can be exercised against Docker on the dev machine instead of Railway.
Useful when Railway is degraded (aimock OOM, rate limits, cold-start
drift) or when validating changes that haven't been deployed yet.
## Usage
```bash
# one-time
cp showcase/.env.example showcase/.env # fill in keys
pnpm --filter @showcase/e2e-smoke install
# full L1-L4 smoke
pnpm --filter @showcase/e2e-smoke smoke:local
# single level / keep containers up between runs
pnpm --filter @showcase/e2e-smoke smoke:local:L1
pnpm --filter @showcase/e2e-smoke smoke:local:keep
pnpm --filter @showcase/e2e-smoke smoke:local:nobuild
```
## What's in here
- **`docker-compose.local.yml`**: `aimock` added as 18th service →
integration containers reach `http://aimock:4010` on the compose
network, mirroring Railway's `showcase-aimock`.
- **`integration-smoke.spec.ts`**: `LOCAL_PORTS=1` env gates URL
rewriting from `https://showcase-<slug>-production.up.railway.app` →
`http://localhost:<port>` via `shared/local-ports.json`. Starters are
skipped under the flag because they're not in `local-ports.json`.
- **`scripts/smoke-local.sh`**: thin orchestrator — `build → up → wait
20s → playwright → down`. Flags: `--level=L1|L2|L3|L4`, `--keep`,
`--no-build`.
- **`tests/package.json`**: `pnpm smoke:local[:L1|:keep|:nobuild]`
wrappers.
- **`.env.example`**: documents optional
`OPENAI_BASE_URL`/`ANTHROPIC_BASE_URL` + `GitHubToken` (ms-agent-dotnet)
and `GOOGLE_API_KEY` (google-adk).
## Verification
Run locally against a fresh checkout of this branch:
- `LOCAL_PORTS=1 SMOKE_ALL=true npx playwright test integration-smoke
--grep @health` → **17/17 pass in 478ms**
- Full L1-L4 against the local stack → **42/51 pass** (9 failures in
L3/L4 for mastra, google-adk, ms-agent-dotnet, strands, langroid,
spring-ai — these are test-data / fixture gaps unrelated to this
infrastructure and will be filed separately)
- `docker compose -f showcase/docker-compose.local.yml config` validates
with 18 services
## Scope
Pure dev-ergonomics addition. No runtime behaviour changes in the
shipped containers. `LOCAL_PORTS` is opt-in; unset = existing
Railway-URL behaviour preserved.
## Test plan
- [ ] `Validate Showcase` CI still green (no package-source changes)
- [ ] No unrelated CI regressions
- [ ] Follow-up PR will investigate and fix the 9 L3/L4 failures
surfaced by local smoke
<!--
Thank you for sending the PR! We appreciate you spending the time to
work on these changes.
Help us understand your motivation by explaining why you decided to make
this change.
**Please PLEASE reach out to us first before starting any significant
work on new or existing features.**
By the time you've gotten here, you're looking at creating a pull
request so hopefully we're not too late.
We love community contributions! That said, we want to make sure we're
all on the same page before you start.
Investing a lot of time and effort just to find out it doesn't align
with the upstream project feels awful, and we don't want that to happen.
It also helps to make sure the work you're planning isn't already in
progress.
As described in our contributing guide, please file an issue first:
https://github.com/ag-ui-protocol/ag-ui/issues
Or, reach out to us on Discord: https://discord.com/invite/6dffbvGU3D
You can learn more about contributing to copilotkit here:
https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md
Happy contributing!
-->
## What does this PR do?
(Describe the changes introduced in this PR)
## Related PRs and Issues
- (Direct link to related PR or issue, if relevant)
## Checklist
- [ ] I have read the [Contribution
Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md)
- [ ] If the PR changes or adds functionality, I have updated the
relevant documentation
## Problem
vitest 3.2.4's hardcoded 60s birpc `onTaskUpdate` timeout
([vitest-dev/vitest#6129](https://github.com/vitest-dev/vitest/issues/6129))
was tripping `unit(20.x/22.x/24.x)` on CopilotKit showcase-scripts
tests. Root driver: cumulative RPC back-pressure from serial file
execution plus two oversized test files (`validate-pins.test.ts` 50.6s,
`audit.test.ts` 71.4s) individually approaching/exceeding the cliff.
Deterministic failure on #4068/#4018/#4079.
## Fix
- Split `validate-pins.test.ts` into 5 files (`cli`, `eacces`,
`parsers`, `r-scenarios`, `validate-all`) plus shared helpers. 50.6s →
12.8s.
- Split `audit.test.ts` into 3 files (`unit`, `audit-package`, `cli`)
plus shared helpers. 71.4s → ~16s.
- Isolated `bundle-demo-content.test.ts` / `generate-registry.test.ts` /
`create-integration.test.ts` from `.git/index.lock` races by serializing
every `restoreFromGitHead` invocation through a cross-process advisory
lock in `test-cleanup.ts`.
- Isolated `create-integration.test.ts` from the real
`showcase/packages/` tree. The generator now honors
`CREATE_INTEGRATION_{PACKAGES,WORKFLOWS}_DIR` env overrides; the test
runs entirely inside a per-suite `os.tmpdir()`.
- Flipped `fileParallelism: true` — safe now that `pool: 'forks'` gives
every file its own process AND FS isolation is in place.
## Results
- Full suite: 158s → 12s locally, 1061/1061 passing across 3 consecutive
`pnpm nx run @copilotkit/showcase-scripts:test --skip-nx-cache` runs.
- Zero `onTaskUpdate` timeouts, zero ENOENT, zero `index.lock`
contention across all 3 runs.
- Every parallel file gets a fresh 60s birpc budget, eliminating the
cumulative back-pressure cliff.
## Test plan
- [ ] CI green on `unit(20.x)`, `unit(22.x)`, `unit(24.x)`.
- [ ] Manual: `pnpm nx run @copilotkit/showcase-scripts:test
--skip-nx-cache` three times in a row — all 1061/1061, no timeout
warnings, no tree drift.
- test-cleanup.ts: `new Error(msg, { cause })` is ES2022; workspace lib is
ES2020 so the two-arg overload is missing. Replaced with an
`errorWithCause()` helper that assigns `.cause` after construction.
Runtime is identical (Node >=16.9); only the TS signature differs.
- test-cleanup.ts: retyped `SAFE_STDIO` as `StdioOptions` (still frozen at
runtime to keep `test-cleanup.test.ts` freeze assertion green) so
spreading `SAFE_EXEC_OPTS` into `execFileSync(..., opts)` no longer trips
the readonly-vs-mutable-array mismatch on `stdio` (fixes the error at
create-integration.test.ts:132).
- create-integration/index.ts: dropped unused `devCmd` local and unused
`args` parameter on `generateDemoPage` (+ call site); both were dead code
introduced during the parallel-isolation refactor.
- validate-pins.parsers.test.ts: annotated all `withTmp((tmp) => ...)`
callbacks as `(tmp: string)` for robustness under LSP module-resolution
glitches. Matches the contract in `validate-pins.shared.ts`.
Tests: 1061/1061 pass (`pnpm nx run @copilotkit/showcase-scripts:test`).
Prior reasons for `fileParallelism: false` are resolved:
- Env-var mutation races (VALIDATE_PINS_REPO_ROOT, SHOWCASE_AUDIT_ROOT,
VALIDATE_PARITY_REPO_ROOT) are moot under `pool: 'forks'` — every file
already gets its own node process with its own `process.env`.
- `.git/index.lock` races between suites that call `restoreFromGitHead`
are fixed by the cross-process lock in test-cleanup.ts.
- The create-integration vs generate-registry collision on
`showcase/packages/` is fixed by the create-integration tmpdir
isolation.
Under fork-per-file + parallel, each test file also gets a fresh 60s
birpc `onTaskUpdate` budget (vitest #6129), eliminating the cumulative
RPC back-pressure that tripped unit(20.x/22.x/24.x) on #4018/#4068/#4079.
Empirical: local full suite 158s → 12s, 1061/1061 passing across three
consecutive `--skip-nx-cache` runs with zero timeouts, zero ENOENT, zero
index.lock contention.
audit.test.ts is 3034 lines / 119 tests and on Node 22 CI its single-file
runtime grew from 36.7s (PR #4071) to 71.4s (PR #4081) — over the hardcoded
60s birpc onTaskUpdate RPC window (vitest #6129). Same cliff that motivated
the validate-pins split earlier in this PR.
Extract makeTmpTree, makeConfig, writePackage, makeExampleDir, anomalyStrings,
and the AUDIT_SCRIPT path constant into audit.shared.ts so the forthcoming
split files can share them without duplication. No behavior change — the
original audit.test.ts still re-declares its own local copies until the
split commit removes them.
Break the 3567-line validate-pins.test.ts into five smaller test files so
each one fits comfortably under vitest's hardcoded 60s birpc onTaskUpdate
RPC window (upstream vitest #6129: DEFAULT_TIMEOUT = 6e4 in the bundled
birpc). pool: 'forks' + fileParallelism: false already gives each file a
fresh 60s RPC budget; splitting ensures no single file is anywhere near
that cliff even when CI is slow.
Split buckets, chosen for logical cohesion and balanced subprocess load:
- validate-pins.parsers.test.ts - pure parsers (no validateAll, no subprocess)
- validate-pins.validate-all.test.ts - in-process validateAll + drift detection
- validate-pins.cli.test.ts - CLI subprocess exit codes
- validate-pins.eacces.test.ts - chmod/EACCES-routed subprocess tests
- validate-pins.r-scenarios.test.ts - R29/R33 regression scenarios
Test count is identical pre/post-split (134 tests). Behaviour-neutral
refactor: no test logic changes, only file boundaries + shared helper
imports.
Jordan Ritter