Files
redwoodjs__local-ci/packages/cli/runner-image.md
Peter Pistorius ef2ee43835 Catch any missing tool in runner image hint (#231)
* fix: skip orphan cleanup when running inside a Docker container

When agent-ci runs nested inside a container (e.g. smoke-bun-setup step 8),
it shares the host's Docker socket. killOrphanedContainers() checks
agent-ci.pid labels using process.kill(pid, 0), but host PIDs don't
exist in the container's PID namespace — so every container looks like
an orphan. This caused the nested agent-ci to force-remove its own
parent container (exit 137).

Both pruneOrphanedDockerResources() and killOrphanedContainers() now
bail out when /.dockerenv exists.

Fixes #228

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: ignore inherited AGENT_CI_DTU_HOST inside containers

When agent-ci runs nested inside a container (e.g. smoke-bun-setup),
the outer container's AGENT_CI_DTU_HOST=host.docker.internal is
inherited. The nested agent-ci used this to tell the inner container
where the DTU is, but the nested DTU is inside the outer container,
not on the host. The inner container got "Connection refused" trying
to reach host.docker.internal:PORT.

Now when /.dockerenv exists, AGENT_CI_DTU_HOST is ignored and the
container's own IP (via hostname -I) is used instead, so sibling
containers can reach the DTU.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: move /.dockerenv guard to call site so tests still work

The /.dockerenv check inside killOrphanedContainers() broke the
shutdown integration tests — they run inside a Docker container (the
test runner) and need the function to work. Moved the guard to the
session bootstrap in cli.ts where the nested agent-ci problem occurs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add global concurrency limiter across all workflows

In --all mode, each workflow created its own per-wave limiter, allowing
20+ containers to run simultaneously and exhaust available memory (OOM
kills with exit 137). Now a single global limiter (CPU/2) is created in
the orchestrator and shared by all handleWorkflow calls, including
single-job workflows and cold-cache first jobs.

Closes #225

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: factor Docker VM memory into concurrency limit

CPU-based limit alone (cpuCount/2 = 7) still causes OOM kills. Now
reads MemAvailable from /proc/meminfo inside the Docker VM and caps
at floor(available / 3GB per container). Falls back to docker info
with a 4GB reserve if busybox isn't available.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: guard orphan cleanup in the function, not just the call site

The /.dockerenv guard was only in cli.ts, but killOrphanedContainers
is also called directly by the Docker integration tests (via a spawned
tsx subprocess). Inside a container, the test's subprocess would see
the parent container's PID label, misidentify it as an orphan, and
kill the container running the tests.

Put the guard back in both functions AND keep the cli.ts guard.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: increase per-container memory estimate to 4 GB

3 GB per container wasn't enough — tests.yml (vitest + Docker
integration tests) and smoke-binary.yml (full build) both got killed
at 3 concurrent containers. 4 GB gives 2 concurrent containers on a
12 GB Docker VM, with ~5.4 GB each.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: mock fs.existsSync in shutdown unit tests for /.dockerenv guard

The unit tests mock execSync but not fs. Inside a runner container,
the real fs.existsSync("/.dockerenv") returns true, causing
killOrphanedContainers to bail before reaching the mocked execSync.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: skip Docker integration tests for orphan cleanup inside containers

The integration tests spawn a real killOrphanedContainers() subprocess.
Inside a container, the function bails due to /.dockerenv (PID namespace
mismatch), so the tests can't exercise it. Skip them with insideDocker().

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add --jobs flag and document concurrency

Adds -j/--jobs <n> to override the auto-detected concurrency limit.
Adds a Concurrency section to the README explaining how the limit is
calculated and how to override it.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: catch any missing tool in runner image hint, add Nix recipe and docs

Closes #229

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Revert "feat: catch any missing tool in runner image hint, add Nix recipe and docs"

This reverts commit ca261639e9.

* feat: generic missing-tool detection with Nix/xz recipe

Replace the hardcoded per-tool pattern list with a single generic
"command not found" regex that catches any missing tool. Add detection
for the Nix installer's "you do not have 'xz' installed" message.
Update hint to point to runner-image.md docs instead of inlining a
Dockerfile snippet. Add Nix recipe to runner-image.md and a runner
image section to the README.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 12:48:26 +02:00

150 lines
6.2 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Runner image
agent-ci runs each job inside a Docker container. By default that container is `ghcr.io/actions/actions-runner:latest` — the official GitHub self-hosted runner image. It ships the runner agent, Node.js, git, curl, jq, unzip, and not much else. In particular, it does **not** include `build-essential`, `gcc`, `cc`, `python3`, `clang`, or any language toolchain.
This is very different from GitHub's hosted `ubuntu-latest` runner, which is a ~30GB VM image preloaded with hundreds of tools (see [`actions/runner-images`](https://github.com/actions/runner-images)). That VM is **not published as a container image** — there is nothing to pull — so agent-ci cannot use it directly.
If a workflow that runs green on GitHub fails locally with `linker 'cc' not found`, `python: command not found`, `you do not have 'xz' installed`, or similar, this gap is why.
## Adding tools: `.github/agent-ci.Dockerfile`
Create a Dockerfile at `.github/agent-ci.Dockerfile` in your repo. agent-ci picks it up automatically on the next run, builds it once, and caches the result.
```dockerfile
FROM ghcr.io/actions/actions-runner:latest
RUN sudo apt-get update \
&& sudo apt-get install -y --no-install-recommends \
build-essential \
pkg-config \
&& sudo rm -rf /var/lib/apt/lists/*
```
That's it. No config file. No flags.
- The image is tagged `agent-ci-runner:<sha-of-Dockerfile-contents>`, so every edit forces a rebuild and identical contents reuse the cached build.
- **The first run is slow — agent-ci has to build the image and cache it (~60–90s for a common toolchain). Every run after that reuses the cached image and takes ~0s.** agent-ci prints a one-line notice while the build is running so you can tell why it's pausing.
- All jobs in your repo use this image unless they set an explicit `container:` directive (which still works — see [Per-job overrides](#per-job-overrides)).
## Directory form: `.github/agent-ci/Dockerfile`
If you need to `COPY` files into the image — corporate CA certs, a pinned binary, a lockfile for apt — use the directory form:
```
.github/
agent-ci/
Dockerfile
ca-bundle.pem
...
```
```dockerfile
FROM ghcr.io/actions/actions-runner:latest
COPY ca-bundle.pem /usr/local/share/ca-certificates/corp.crt
RUN sudo update-ca-certificates \
&& sudo apt-get update && sudo apt-get install -y build-essential
```
agent-ci builds it with `.github/agent-ci/` as the context, so everything next to the Dockerfile is copyable. The hash includes every file in the context, so adding or changing any of them forces a rebuild.
If both forms are present, the directory form wins.
## Common recipes
These are copy-paste examples — agent-ci does not ship any of them as a default.
**Rust / cargo (`cc-rs`, native build scripts):**
```dockerfile
FROM ghcr.io/actions/actions-runner:latest
RUN sudo apt-get update \
&& sudo apt-get install -y --no-install-recommends build-essential pkg-config \
&& sudo rm -rf /var/lib/apt/lists/*
```
**Node.js native modules (`node-gyp`, `sharp`, `bcrypt`):**
```dockerfile
FROM ghcr.io/actions/actions-runner:latest
RUN sudo apt-get update \
&& sudo apt-get install -y --no-install-recommends build-essential python3 \
&& sudo rm -rf /var/lib/apt/lists/*
```
**Go with cgo:**
```dockerfile
FROM ghcr.io/actions/actions-runner:latest
RUN sudo apt-get update \
&& sudo apt-get install -y --no-install-recommends build-essential \
&& sudo rm -rf /var/lib/apt/lists/*
```
**Nix (`cachix/install-nix-action`, `DeterminateSystems/nix-installer-action`):**
```dockerfile
FROM ghcr.io/actions/actions-runner:latest
RUN sudo apt-get update \
&& sudo apt-get install -y --no-install-recommends xz-utils \
&& sudo rm -rf /var/lib/apt/lists/*
```
**Ruby native gems:**
```dockerfile
FROM ghcr.io/actions/actions-runner:latest
RUN sudo apt-get update \
&& sudo apt-get install -y --no-install-recommends build-essential libffi-dev libyaml-dev \
&& sudo rm -rf /var/lib/apt/lists/*
```
## Per-job overrides
The standard GitHub Actions `jobs.<id>.container:` directive is still honored — and takes priority over `.github/agent-ci.Dockerfile`. Use it when a single job needs a different environment than the rest of your repo:
```yaml
jobs:
deploy:
runs-on: ubuntu-latest
container: my-org/deploy-runner:v2
steps:
- run: ./deploy.sh
```
When a job sets `container:`, agent-ci runs that image **verbatim**. It does **not** layer `.github/agent-ci.Dockerfile` on top, and it does **not** build anything. Whatever you put in `container:` is what runs, exactly as on GitHub-hosted runners. This keeps the "same YAML runs locally and on GitHub" parity promise — locally layering would silently change the environment relative to GitHub.
If you want both your own base image **and** agent-ci's extras, pick one:
- Skip `container:` and put everything in `.github/agent-ci.Dockerfile` (one image, applies to all jobs).
- Or build your own image that inherits from your preferred base and adds what you need, then reference it with `container:`.
## CI escape hatch: `AGENT_CI_RUNNER_IMAGE`
Setting the `AGENT_CI_RUNNER_IMAGE` environment variable overrides all discovery and uses that image directly:
```bash
AGENT_CI_RUNNER_IMAGE=my-org/internal-runner:v3 agent-ci run --all
```
Highest priority (after per-job `container:`). Useful for CI pipelines that want to force a specific image without touching files in the repo.
## Resolution order
Highest priority wins:
1. `jobs.<id>.container:` in workflow YAML (per-job override)
2. `AGENT_CI_RUNNER_IMAGE` environment variable
3. `.github/agent-ci/Dockerfile` (directory form)
4. `.github/agent-ci.Dockerfile` (simple form)
5. `ghcr.io/actions/actions-runner:latest` (built-in default)
## Build caching
When agent-ci builds from one of your Dockerfiles, it:
1. Hashes the Dockerfile contents (simple form) or Dockerfile + all context files (directory form), sorted deterministically.
2. Tags the image `agent-ci-runner:<hash-prefix>`.
3. Checks if that tag already exists locally — if so, reuses it.
4. Otherwise runs `docker build`, pulling `ghcr.io/actions/actions-runner:latest` first if needed (most Dockerfiles inherit from it).
Clean up old builds with `docker image prune` or `docker rmi agent-ci-runner:<tag>` as normal — they're regular Docker images with no special lifecycle.