mirror of
https://github.com/vercel/next.js.git
synced 2026-09-20 02:25:18 +08:00
Jude Gao
0921733a94
Fixtures now live next to the code they test, like e2e. `pnpm eval <name>` packs the local `next` build, generates baseline + agents-md experiment configs on the fly, and runs both in a sandbox. The agents-md variant drops an `AGENTS.md` that points the agent at the bundled docs in `node_modules/next/dist/docs/` — comparing the two variants tells you whether shipping a doc actually changes agent behavior. `run-evals.js` mirrors `run-tests.js`: pack once, pass the tarball path to the child via `NEXT_EVAL_TARBALL` env, forward flags. We only pack `next`, not the whole workspace — the sandbox is remote Linux, so a local `@next/swc` darwin binary wouldn't run there anyway; the sandbox downloads the right one at runtime. The experiment config uses `sandbox: 'auto'`, which picks Vercel sandboxes when credentials are present and falls back to local Docker otherwise, so external contributors can run the same evals with just Docker + `ANTHROPIC_API_KEY`. `experiments/` is generated fresh each run and gitignored so we don't maintain N committed config files that differ by one line. Fixture code is excluded from eslint since it's deliberately imperfect code for agents to fix, and `EVAL.ts` uses vitest rather than jest. Fixture `package.json` files use `"next": "^16"` rather than a pinned canary so agents reading `package.json` to infer capabilities aren't misled by a stale version string; the tarball install overlays it regardless. next-evals-oss stays as the full benchmark runner for nextjs.org/evals; it'll pull fixtures from here instead of keeping its own copy.
87 lines
1.0 KiB
Plaintext
87 lines
1.0 KiB
Plaintext
# build output
|
|
dist
|
|
!.github/actions/*/dist
|
|
.next
|
|
target
|
|
packages/next/wasm/@next
|
|
tarballs/
|
|
packages/**/*.tgz
|
|
.errors/
|
|
|
|
# rust compiler crashes
|
|
rustc-ice-*.txt
|
|
|
|
# dependencies
|
|
node_modules
|
|
package-lock.json
|
|
yarn.lock
|
|
!/yarn.lock
|
|
test/node_modules
|
|
.pnpm-store/
|
|
.github/**/node_modules
|
|
|
|
# logs & pids
|
|
*.log
|
|
pids
|
|
*.cpuprofile
|
|
*.heapsnapshot
|
|
|
|
# coverage
|
|
.nyc_output
|
|
coverage
|
|
|
|
# test output
|
|
test/**/out/*
|
|
test/**/next-env.d.ts
|
|
test/**/.next*
|
|
.DS_Store
|
|
/e2e-tests
|
|
test/tmp/**
|
|
test/.trace
|
|
test/traces
|
|
|
|
# Editors
|
|
**/.idea
|
|
**/.#*
|
|
.nvmrc
|
|
|
|
# examples
|
|
examples/**/out/*
|
|
|
|
# env
|
|
.env*.local
|
|
|
|
pr-stats.md
|
|
test-timings.json
|
|
|
|
# Vercel
|
|
.vercel
|
|
.now
|
|
|
|
# Cache
|
|
*.tsbuildinfo
|
|
.swc/
|
|
.turbo
|
|
|
|
# Storybook
|
|
*storybook.log
|
|
|
|
# Claude Code (local settings and session files)
|
|
.claude/settings.local.json
|
|
.claude/plans/
|
|
.claude/todos.json
|
|
.claude/worktrees
|
|
CLAUDE.local.md
|
|
|
|
# Conductor (worktrees created by Conductor app, but keep scripts/)
|
|
.conductor/*/
|
|
!.conductor/scripts/
|
|
|
|
index.scip
|
|
|
|
# Evals
|
|
evals/.tarballs/
|
|
evals/experiments/
|
|
evals/results/
|
|
evals/agent-results.json
|