## Summary
- add end-to-end adoption evals for Partial Prefetching and Cache
Components
- compare the baseline, bundled docs, and local adoption skill
independently
- support local skill installation and longer timeouts for
browser-backed workflows
## Results
One `claude-opus-4-8` run was retained per treatment and adoption eval,
judged by `claude-haiku-4-5`. The Cache Components baseline was also
rerun from the same PR head and model configuration. These results are
directional rather than statistically stable.
| Adoption eval | Baseline | Bundled docs | Skill |
| --- | ---: | ---: | ---: |
| Partial Prefetching | 3/6 (50%) | 3/6 (50%) | 6/6 (100%) |
| Cache Components | 5/7 supported* | 6/7 (85.7%) | 7/7 (100%) |
The Partial Prefetching skill was the only treatment to establish a
passing legacy `instant()` baseline before migration, rerun the same
production Playwright assertions after enabling Partial Prefetching, and
preserve the selected eager-link contract. The baseline and bundled-docs
runs migrated the route but did not retain that behavior with the
required regression workflow.
### Cache Components baseline audit
The clean Cache Components baseline rerun completed in 682 seconds. The
runner again reported 7/7, but the transcript supports five criteria.
Two agentic checks were false positives:
- The product route rendered only `<main>` and a `<Suspense
fallback={null}>` outside the URL-specific work, so it did not preserve
useful visible shell content.
- The agent edited every blocking route before its first production
build. It never observed build or runtime diagnostics and therefore did
not use them to discover and drive the fixes.
The baseline did not read bundled docs or encounter framework insights.
It inspected the installed Next.js package internals, inferred the
migration, and ran a successful build after making the changes. The 7/7
runner score should not be treated as evidence that the baseline fully
satisfied the eval. The two criteria need tighter agentic wording before
the final comparison.
The current bundled-docs and skill scores remain 6/7 and 7/7. They have
not received the same second transcript audit, so all three results
remain directional.
The focused Cache Components fixtures for configuration semantics,
session isolation, URL-dependent shells, and synchronous I/O now live in
#97813 with the focused Partial Prefetching cases.
## Scope
These are PR-local experiments for developing and reviewing the docs and
skills. This PR does not register them in the public `evals.nextjs.org`
suite, which is configured separately in the OSS eval repository.
## Validation
- `pnpm exec tsc -p evals/tsconfig.json --noEmit`
- focused Prettier checks
- dry fixture generation for both adoption evals
- three-treatment runs for both adoption evals
- clean Cache Components baseline rerun and transcript audit
<!-- NEXT_JS_LLM -->
Adds evals that check whether an agent reaches for Next.js when asked to
build something, and whether what it ships is really Next.js rather than
something that only resembles it.
Fixtures now live next to the code they test, like e2e. `pnpm eval
<name>` packs the local `next` build, generates baseline + agents-md
experiment configs on the fly, and runs both in a sandbox. The agents-md
variant drops an `AGENTS.md` that points the agent at the bundled docs
in `node_modules/next/dist/docs/` — comparing the two variants tells you
whether shipping a doc actually changes agent behavior.
`run-evals.js` mirrors `run-tests.js`: pack once, pass the tarball path
to the child via `NEXT_EVAL_TARBALL` env, forward flags. We only pack
`next`, not the whole workspace — the sandbox is remote Linux, so a
local `@next/swc` darwin binary wouldn't run there anyway; the sandbox
downloads the right one at runtime. The experiment config uses `sandbox:
'auto'`, which picks Vercel sandboxes when credentials are present and
falls back to local Docker otherwise, so external contributors can run
the same evals with just Docker + `ANTHROPIC_API_KEY`.
`experiments/` is generated fresh each run and gitignored so we don't
maintain N committed config files that differ by one line. Fixture code
is excluded from eslint since it's deliberately imperfect code for
agents to fix, and `EVAL.ts` uses vitest rather than jest. Fixture
`package.json` files use `"next": "^16"` rather than a pinned canary so
agents reading `package.json` to infer capabilities aren't misled by a
stale version string; the tarball install overlays it regardless.
next-evals-oss stays as the full benchmark runner for nextjs.org/evals;
it'll pull fixtures from here instead of keeping its own copy.
Aurora Scharff