Make the ppr test accept diverse solutions as long as they lead to a
static shell. Achieved by `expect(environment).toSatisfyCriterion`
(a.k.a., llm-as-judge)
Fixes false-negative regex assertions — observed rejecting correct
solutions in recent eval runs — by grading those specific checks with
`@vercel/agent-eval`'s agentic LLM judge instead. Only the assertions
with demonstrated false negatives change; every deterministic check
stays.
- `agent-034-async-cookies`: `/await\s+cookies\s*\(\s*\)/` only matched
the naive `await cookies()` and rejected the correct (arguably better)
`await Promise.all([cookies(), headers()])` — observed failing most runs
of a strong model — and the no-sync-call lookbehind wrongly flagged the
bare `cookies()` inside the array. The four await-mechanics tests
collapse into one judge criterion: the promises must actually be awaited
before use, in any correct form. The theme-cookie and Accept-Language
content checks stay regex.
- `agent-030-app-router-migration-hard`: the home-page test rejected
solutions that didn't match one exact shape (e.g. data fetching
extracted to a helper means no literal `fetch(` in `page.tsx`). Its body
becomes one judge criterion (async Server Component, fetches during
server render, no `getServerSideProps`); the file-existence check and
the other seven tests stay deterministic.
The judge is pinned to **`claude-haiku-4-5`** in the generated
experiment configs — a cheap fixed grader, identical for every run
regardless of the model under test. Each criterion states the
requirement semantically, includes one reference solution to ground the
small grader, and fixes the standard of judgment (runtime correctness,
not style) — no enumerated failure modes, so the judge stays flexible
about equivalent forms.
Supporting changes: bumps `@vercel/agent-eval` `0.9.5` → `1.3.0` (ships
the judge runtime) and adapts `run-evals.js` to the 1.3 CLI (`run
baseline agents-md --force`; `--dry` maps to `status`); generated
experiments now run through the Vercel AI Gateway so `vc env pull` is
the only credential setup; converted fixtures exclude `EVAL.ts` from
tsconfig (the `@vercel/agent-eval/eval` import has no type declarations
— vercel-labs/agent-eval#166 — and `next build` type-checks the fixture,
so experiments running `scripts: ['build']` would otherwise fail on it,
same reason `EVAL.tsx` was already excluded). Not converted: `agent-040`
(already redesigned upstream in #94578) and `agent-041` (no diagnosed
false negative in the report).
Verified end-to-end from this repo: converted `agent-034` with codegen
`claude-sonnet-4-5` via the AI Gateway in a Vercel sandbox, judge pinned
to `claude-haiku-4-5` — passes, with the judge clause executing
in-sandbox.
**Running it:**
```bash
pnpm install # picks up @vercel/agent-eval 1.3.0
pnpm --filter=next build # pnpm eval packs the locally built next
vc env pull .env.local # the only credential: sandbox + AI Gateway (codegen and judge)
pnpm eval agent-034-async-cookies # baseline + AGENTS.md variants
pnpm eval agent-034-async-cookies --dry # preview only
```
<!-- NEXT_JS_LLM_PR -->
## Summary
- Adds `agent-043-view-transitions`, a new eval that tests whether
agents can add React View Transitions to a Next.js product gallery app
- Starter app is a simple product list/detail gallery with Suspense
boundaries, no view transitions
- EVAL.ts checks 8 patterns: `viewTransition` config flag,
`ViewTransition` imported from `react`, shared element `name` props,
`transitionTypes` on Link, Suspense enter/exit animations,
`default="none"` isolation, `prefers-reduced-motion` CSS, and
`::view-transition-*` CSS pseudo-elements
## Test plan
- [x] Starter app builds with `next build`
- [x] All 8 EVAL.ts tests fail on unmodified starter (no false
positives)
- [x] All 8 EVAL.ts tests pass on a correct golden solution
- [x] Golden solution builds with `next build` (viewTransition
experiment detected)
- [x] Commit passes lint-staged (prettier + eslint)
Upgrades `@vercel/agent-eval` from 0.8.0 to 0.9.5. The new version loads
`.env` / `.env.local` from its own cwd (`evals/`) rather than inheriting
from the parent process, so the old `process.loadEnvFile()` call no
longer works. Instead, `run-evals.js` now symlinks the root env files
into `evals/` before spawning the harness. `evals/.env` is added to
`.gitignore` to keep secrets out of the tree.
Two new evals exercise PPR knowledge: `agent-041-optimize-ppr-shell`
checks that the agent decomposes a monolithic `loading.tsx` into
granular `Suspense` boundaries, and `agent-042-enable-ppr` verifies the
agent knows PPR is enabled via `cacheComponents: true` (not the old
`experimental.ppr` flag).
Fixtures now live next to the code they test, like e2e. `pnpm eval
<name>` packs the local `next` build, generates baseline + agents-md
experiment configs on the fly, and runs both in a sandbox. The agents-md
variant drops an `AGENTS.md` that points the agent at the bundled docs
in `node_modules/next/dist/docs/` — comparing the two variants tells you
whether shipping a doc actually changes agent behavior.
`run-evals.js` mirrors `run-tests.js`: pack once, pass the tarball path
to the child via `NEXT_EVAL_TARBALL` env, forward flags. We only pack
`next`, not the whole workspace — the sandbox is remote Linux, so a
local `@next/swc` darwin binary wouldn't run there anyway; the sandbox
downloads the right one at runtime. The experiment config uses `sandbox:
'auto'`, which picks Vercel sandboxes when credentials are present and
falls back to local Docker otherwise, so external contributors can run
the same evals with just Docker + `ANTHROPIC_API_KEY`.
`experiments/` is generated fresh each run and gitignored so we don't
maintain N committed config files that differ by one line. Fixture code
is excluded from eslint since it's deliberately imperfect code for
agents to fix, and `EVAL.ts` uses vitest rather than jest. Fixture
`package.json` files use `"next": "^16"` rather than a pinned canary so
agents reading `package.json` to infer capabilities aren't misled by a
stale version string; the tarball install overlays it regardless.
next-evals-oss stays as the full benchmark runner for nextjs.org/evals;
it'll pull fixtures from here instead of keeping its own copy.
Jude Gao