Jude Gao 0921733a94 [Evals] Standardize Evals in Next.js (#90883) ...

Fixtures now live next to the code they test, like e2e. `pnpm eval
<name>` packs the local `next` build, generates baseline + agents-md
experiment configs on the fly, and runs both in a sandbox. The agents-md
variant drops an `AGENTS.md` that points the agent at the bundled docs
in `node_modules/next/dist/docs/` — comparing the two variants tells you
whether shipping a doc actually changes agent behavior.

`run-evals.js` mirrors `run-tests.js`: pack once, pass the tarball path
to the child via `NEXT_EVAL_TARBALL` env, forward flags. We only pack
`next`, not the whole workspace — the sandbox is remote Linux, so a
local `@next/swc` darwin binary wouldn't run there anyway; the sandbox
downloads the right one at runtime. The experiment config uses `sandbox:
'auto'`, which picks Vercel sandboxes when credentials are present and
falls back to local Docker otherwise, so external contributors can run
the same evals with just Docker + `ANTHROPIC_API_KEY`.

`experiments/` is generated fresh each run and gitignored so we don't
maintain N committed config files that differ by one line. Fixture code
is excluded from eslint since it's deliberately imperfect code for
agents to fix, and `EVAL.ts` uses vitest rather than jest. Fixture
`package.json` files use `"next": "^16"` rather than a pinned canary so
agents reading `package.json` to infer capabilities aren't misled by a
stale version string; the tarball install overlays it regardless.

next-evals-oss stays as the full benchmark runner for nextjs.org/evals;
it'll pull fixtures from here instead of keeping its own copy.

2026-03-06 12:05:36 -05:00

..

setup.ts

[Evals] Standardize Evals in Next.js (#90883)

2026-03-06 12:05:36 -05:00