everyinc__compound-engineering-plugin/docs/solutions/developer-experience/idle-bound-test-suite-parallel-workers.md

title, date, category, module, problem_type, component, symptoms, root_cause, resolution_type, severity, tags

title	date	category	module	problem_type	component	symptoms	root_cause	resolution_type	severity	tags
CI test suite was idle-bound, not CPU-bound — parallel workers cut it 54%	2026-07-22	developer-experience	ci	performance_issue	testing_framework	CI `Run tests` step took 193-237s while every other step in the job totalled ~12s Suite wall time grew with the test count even though no individual test was slow A serial run reports low CPU utilization — `95.24s user 78.62s system 36% cpu`	config_error	config_change	medium	ci test-performance bun-test parallelism flaky-tests measurement

CI test suite was idle-bound, not CPU-bound — parallel workers cut it 54%

Problem

The Run tests step in .github/workflows/ci.yml effectively was the CI job: ~224s median across seven successful runs, against ~12s for checkout, install, release:validate, and plugin:validate combined. Every PR paid roughly four minutes to learn whether it was green.

Symptoms

• Run tests step durations of 193s, 215s, 218s, 224s, 227s, 228s, 237s on ubuntu-latest.
• No single slow test to blame — the cost was spread across 98 files and 2617 tests.
• The tell: a serial local run reports 95.24s user 78.62s system 36% cpu. Bun was consuming about a third of one core while the rest of the runner sat idle.

What Didn't Work

• Measuring on a developer laptop. The first instinct was to time bun test locally and A/B it. The machine happened to be at load average ~300 from unrelated concurrent work, so the serial run took 7m56s (vs. 224s on CI) and produced two spurious 5s-timeout failures that had nothing to do with any change. Local wall-clock was not just noisy, it was actively misleading. The A/B had to move to CI, where the runner is a dedicated 4-core VM — see Prevention.
• Turning the worker count up. More workers looks free on an idle-bound suite, so --parallel=8 on a 4-core runner was measured rather than assumed. It bought ~9% (102s -> 93s) but inflated total test-CPU from 223s to 343s and pushed five tests past bun's 5000ms default per-test timeout. That trades wall time for builds that go red when the runner is busy. Rejected.
• Looking for one hot file to fix. There wasn't a pathological test. The problem was structural — the runner had four cores and the test runner used one.

Solution

Put --parallel on the package test script and have CI invoke the script rather than the bare binary:

// package.json
"test": "bun test --parallel",

# .github/workflows/ci.yml
- name: Run tests
  run: bun run test

Keeping the flag in the package script rather than inline in the workflow is the point: the command CI runs and the command a contributor runs cannot drift apart. Bare bun test <file> stays available and is still the right tool for iterating on one file.

Note --parallel takes no value. It tracks the runner's core count, which is what you want — see Prevention.

Result, measured on the Run tests step (the only step whose duration moves):

	Run tests step
Before — main, two same-hour controlled runs	207s, 218s	median 212s
Before — last 7 successful runs	193s, 215s, 218s, 224s, 227s, 228s, 237s	median 224s
After — four runs	97s, 102s, 103s, 108s	median 103s

51% against the controlled pair, 54% against the historical median. 2617 pass / 0 fail in every run.

Why This Works

The suite spends its wall time blocked, not computing. It shells out constantly — python3 script invocations, bash sandbox scripts, git fixture repos, bun run src/index.ts CLI spawns — and several suites contain deliberate real sleeps to exercise timeout, liveness, and detached-process behavior. While a test file waits on a subprocess, the process running it has nothing to do.

36% cpu on a serial run is the whole diagnosis in one number: roughly two-thirds of the elapsed time was a single process waiting. Distributing files across worker processes fills that idle time with other files' work. Total work was 223 test-CPU-seconds; packed onto 4 cores it lands at ~100s wall.

This is why the fix is a flag and not a rewrite. Nothing was written inefficiently — the runner was just serial.

Prevention

• Check CPU utilization before optimizing a slow suite. time on a serial run answers "is this idle-bound or CPU-bound?" in one command, and the two diagnoses lead to completely different fixes. A low %cpu means look at concurrency; a high one means look at the work itself. Do not start by hunting for slow tests.

• Measure CI changes on CI. A developer laptop is a shared, unmetered machine — background builds, other worktrees, and editor indexing move wall-clock by multiples and can manufacture timeout failures that look like real regressions. gh workflow run ci.yml --ref <branch> gives a controlled A/B on identical dedicated runners; gh run view <id> --json jobs reads the exact per-step durations. Dispatch the baseline branch and the change branch in the same window so runner-pool conditions match. gh run view <id> --log also carries bun's per-test timings, which is how the critical path below was found.

• Do not pin a worker count. Bare --parallel follows the runner's core count and keeps working if the runner size changes. Raising it looks free on an idle-bound suite and is not — it inflates per-test latency and converts runner busyness into red builds.

• Parallel workers make cross-file isolation load-bearing. --parallel implies --isolate. A test file may no longer depend on another file's leftovers, and a test that writes outside its own mktemp directory becomes a latent flake rather than a harmless one. Audit for repo-root and fixed-path writes before enabling it.

• Give subprocess-heavy suites an explicit timeout. Parallelism narrows the margin on any test already running near the default. tests/skills/peer-job-runner.test.ts drives real detached processes and bounded waits on bun's 5000ms default; its bounded-wait case measured 4653ms on CI — a 347ms margin, meaning whether it passed was a question of runner busyness rather than correctness. A file whose tests legitimately run for seconds should declare setDefaultTimeout(20_000) the way the other subprocess-heavy suites do. Scan for near-timeout tests with:

  gh run view <run-id> --log | grep -oE '\[[0-9]{4}\.[0-9]+ms\]' | sort -rn | head
  

• After parallelizing, the floor is the slowest single file — a file never splits across workers. The ce-work-unit-workspace suite is already sharded (ce-work-unit-workspace-*.test.ts). AGENTS.md now reports ~81s wall time with ce-babysit-pr-snapshot.test.ts (~36s) as the remaining ceiling; further splits have small returns.

Related Issues

• PR EveryInc/compound-engineering-plugin#1235 — the change described here.
• AGENTS.md > "CI and Quality Gates" carries the operating rules (unpinned worker count, isolation contract, current critical path).