### What?
Adds eval coverage for the experimental agent feedback workflow:
- Routine debugging should not produce a report
- Qualifying friction should produce an anonymized structured report
- Distinct issues should produce separate review forms
Also adds repeat-run and variant controls so trigger frequency can be
measured across multiple runs.
Local-Skill and agent-feedback/privacy evals are marked `publish:
false`, with scoped `evals/AGENTS.md` instructions preventing their
fixtures, transcripts, or scores from being exported to the public
benchmark.
### Why?
We need to measure trigger precision, anonymization, issue splitting,
and duplicate prevention before expanding the experiment.
### How?
Depends on #98582.
The `agent-feedback` treatment uses the managed block and bundled
reporting protocol from the parent PR. Only the remote rollout gate is
forced on inside the eval sandbox so runs are deterministic.
| Eval | Baseline | Agent rules | Agent feedback | 10-run treatment |
| --- | --- | --- | --- | --- |
| Routine debugging | Pending | Pending | 1/1 passed | Pending |
| Anonymization | Pending | Pending | 2/2 reporting checks passed |
Pending |
| Distinct issues | Pending | Pending | 1/1 passed | Pending |
The first attempted run did not reach the agent because the local
sandbox was not linked to a Vercel project. It is infrastructure setup
and is not included in the results above.
The first distinct-issues run produced the expected two separate
payloads. Its scorer rejected them because the parser did not allow the
existing `token` query parameter and the browser criterion required an
open attempt even when the agent environment exposed no browser
capability. After correcting those assertions and clarifying the
stopping-point wording, the scored rerun passed.
Both anonymization treatments produced one valid payload with none of
the seeded customer, project, route, local-path, internal-URL, or secret
values. The corrected rerun also selected `misleading-error` and passed
every reporting assertion. An inherited `.next`-preservation assertion
was removed from this fixture because it measures `next-dev-loop`
behavior, not agent-feedback anonymization; Skill queue coordination
belongs in a separate focused eval.
Stacked on #98643.
This PR adds `experimental.agenticAutoUpgrade = 'future'` config which
enables nudging the agents to notify the user when there's unadopted
future default(s). The nudge will include guiding to upgrade via `next
upgrade --ai` (run "future" by detecting config).
The method of nudging leverages the agents behavior where they tend to
listen to messages from fatal errors that blocks the process compared to
general error/warning logs. Whenever the agents run `next dev` or `next
build`, Next.js will detect the condition and nudge the agent using this
method.
Afterwards it's up to the user whether to proceed the upgrade or not,
it's 100% up to the user how to run it e.g. subagent, background agent,
etc. and the process should not enforce any that affects user's
workflow.
Enabling `experimental.agenticAutoUpgrade = 'future'` also enables
security check and latest version check.
Stacked on #98640.
This PR adds `next upgrade --ai="future"` flag, which is targeted to
help users leverage agents to upgrade their app to adopt the future
defaults when available. Just like latest version upgrade, it covers
running codemods and a migration checklist for major-to-major upgrades
to support breaking changes more reliably.
This PR currently covers Cache Components only for the future default.
Stacked on #98633.
This PR adds `experimental.agenticAutoUpgrade = 'latest'` config which
enables nudging the agents to notify the user when there's new
major/minor Next.js version available to upgrade. The nudge will include
guiding to upgrade via `next upgrade --ai` (run "latest" by detecting
config).
The method of nudging leverages the agents behavior where they tend to
listen to messages from fatal errors that blocks the process compared to
general error/warning logs. Whenever the agents run `next dev` or `next
build`, Next.js will detect the condition and nudge the agent using this
method.
Afterwards it's up to the user whether to proceed the upgrade or not,
it's 100% up to the user how to run it e.g. subagent, background agent,
etc. and the process should not enforce any that affects user's
workflow.
Enabling `experimental.agenticAutoUpgrade = 'latest'` also enables
security check.
Stacked on #98637.
This PR adds `next upgrade --ai="latest"` flag, which is targeted to
help users leverage agents to upgrade their app to the latest major
version when available. Just like security upgrade, it covers running
codemods and a migration checklist for major-to-major upgrades to
support breaking changes more reliably.
Stacked on #98562.
> [!TIP]
> Recommended to review commit by commit.
This PR adds `experimental.agenticAutoUpgrade = 'security'` config which
enables nudging the agents to notify the user when the app's Next.js
version has any security advisories. The nudge will include guiding to
upgrade via `next upgrade --ai` (run "security" by detecting config).
The method of nudging leverages the agents behavior where they tend to
listen to messages from fatal errors that blocks the process compared to
general error/warning logs. Whenever the agents run `next dev` or `next
build`, Next.js will detect the condition and nudge the agent using this
method.
Afterwards it's up to the user whether to proceed the upgrade or not,
it's 100% up to the user how to run it e.g. subagent, background agent,
etc. and the process should not enforce any that affects user's
workflow.
> [!TIP]
> Recommended to review commit by commit.
This PR adds `next upgrade --experimental-ai="security"` flag (alias
`--ai`), which is targeted to help users leverage agents to upgrade
their app to the safe major version when their app's Next.js version has
any security advisories.
Once the command is ran from the user, Next.js will detect the installed
agent harness in user's device, currently limited to Codex and Claude,
and will proceed with starting an agent session once approved. If it is
called within an agent session, the work will continue off within that
agent.
`next upgrade --ai` simply does two things:
- prepare the relevant context to temporary dir
- print hand off prompt, guiding to read those context
The context will guide the agent to run relevant codemods and migration
checklist to proceed. This PR is a base core of the workflow, and will
have wrappers of entry point around this. Also, will add "latest" and
"future" as follow up, which will cover the app to be always latest, and
adopt the future defaults like Cache Components.
This PR also sets up the evals infra and adds evals.
## Summary
- add a Partial Prefetching optimizer Skill for selected source Links
and destinations
- keep prefetch stages and Link policy in the public [Optimizing
prefetching
guide](https://nextjs.org/docs/app/guides/optimizing-prefetching), and
rely on the same static-shell documentation as the Cache Components
optimizer for cache placement, Suspense, loading UI, and authorization
- keep the Skill focused on the product decision, production `instant()`
loop, differential verification, and mutation freshness when an
optimization adds a cache around mutable data
- reference the canonical
[`unstable_prefetch()`](https://nextjs.org/docs/app/api-reference/functions/prefetch)
and
[`unstable_navigation()`](https://nextjs.org/docs/app/api-reference/functions/navigation)
docs instead of explaining the APIs again in the Skill
- link cache-writing optimizations to the canonical
[Revalidating](https://nextjs.org/docs/app/getting-started/caching-and-revalidating#revalidating)
documentation and verify a populated-cache mutation in the Skill and
eval
- bundle a self-contained production rig aligned with the related
adoption and optimizer Skills, tuned to record each source-link and
prefetch-stage contract
- add an eval that compares baseline, the bundled guide, and the local
Skill against the same prefetched-UI and cache-freshness goal
- document the optimizer in the AI agents guide
This follows the same guide/Skill split as #97737. Partial Prefetching
adoption preserves existing behavior; this optimizer is the later
decision about what is worth preparing before a click and what should
wait for navigation.
Related: #97712, #97737, #97759, #98122, #98175
## Eval
One run per treatment with Claude Opus 4.8 and Claude Sonnet 4.5, using
a fixed Claude Haiku 4.5 judge. The bundled-guide and Skill treatments
use a local Next.js tarball containing the new guide and API references.
| Model | Treatment | Criteria | Score change | Overall | Duration |
Tokens processed | Total run cost | Cost change |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Opus 4.8 | Baseline | 10/10 (100%) | — | Pass | 18m 42s | 3.39M |
$3.46 | — |
| Opus 4.8 | Bundled guide | 9/10 (90%) | -10 pp | Fail | 13m 37s |
3.02M | $2.84 | -18.0% |
| Opus 4.8 | Local Skill | 10/10 (100%) | +10 pp | Pass | 17m 22s |
2.80M | $2.54 | -10.6% |
| Sonnet 4.5 | Baseline | 2/10 (20%) | — | Fail | 15m 55s | 1.97M |
$1.06 | — |
| Sonnet 4.5 | Bundled guide | 5/10 (50%) | +30 pp | Fail | 15m 49s |
2.17M | $1.11 | +4.5% |
| Sonnet 4.5 | Local Skill | 8/10 (80%) | +30 pp | Fail | 15m 31s |
1.41M | $0.77 | -30.4% |
Token totals count each unique model message once and include uncached
input, cache creation, cache reads, and output. Cache reads account for
most of the total in every treatment. Total run cost applies the current
AI Gateway base rates to each token category: Opus 4.8 costs $5/M input,
$6.25/M cache writes, $0.50/M cache reads, and $25/M output; Sonnet 4.5
costs $3/M input, $3.75/M cache writes, $0.30/M cache reads, and $15/M
output. The six agent runs cost $11.78 in total.
Compared with baseline, the Opus Skill run maintained a 100% criteria
pass rate while reducing cost by 26.7%. The Sonnet Skill run improved
the criteria pass rate from 20% to 80%, a 60 percentage-point increase,
while reducing cost by 27.3%.
The stronger model could solve the fixture without the Skill, although
it independently searched the bundled Next.js documentation and took
slightly longer than the Skill treatment. The Opus guide-only run
implemented the correct behavior but missed the required
RED-before-GREEN proof.
The Sonnet treatments show a clearer progression. Baseline did not
implement the required caching, navigation stage, `instant()` contract,
or cache invalidation. The bundled guide added the cached summary and
production test, but missed the navigation stage, deferred-state
assertions, tag invalidation, and RED-before-GREEN proof. The Skill
treatment completed the `instant()` RED-to-GREEN loop and final
production behavior, but left `connection()` in the related-sessions
read and used path revalidation instead of tagging the cached read. It
therefore passed 8 of 10 criteria but failed the overall eval.
All six measured runs completed through grading. Chromium and its system
dependencies are provisioned before the agent starts, so missing browser
libraries did not determine these results.
## Verification
- `pnpm --filter=next build`
- `pnpm prettier --check` for the changed guide, prompt, Skill, and eval
fixture
- `pnpm exec tsc -p
evals/evals/agent-051-optimize-partial-prefetching/tsconfig.json
--noEmit`
- `node --check run-evals.js`
- `git diff --check`
- packed a local Next.js tarball and confirmed it contains the optimizer
guide plus the `navigation()` and `prefetch()` references
- verified the `navigation()` contract in the minimal reproduction and
NextBeats on Next.js 16.4.0-canary.22
- merged the latest `canary`, preserving both generic fixture setup and
Playwright provisioning
<!-- NEXT_JS_LLM -->
## Summary
- Explain that removing `dynamic = 'force-dynamic'` moves a route from
full request-time rendering to Cache Components' Partial Prerendering
model, then document the choices for its request-time work.
- Preserve routes that previously used `dynamic = 'force-static'` or
`dynamic = 'error'` instead of leaving them under `instant = false`,
which can move rendering to request time and reduce prefetched UI.
- Clarify that existing `fetch` and `unstable_cache` caching can remain
in place during adoption.
- Combine the incremental checkpoint and completion coverage into one
eval that verifies both stages of the migration.
- Make the adoption skill honor an explicit incremental strategy before
continuing through a complete migration.
- Remove the unrelated `.env*` ignore and restore the existing agent-054
eval registration.
## Eval results
`agent-047-adopt-cache-components` runs the complete incremental
migration with `openai/gpt-5.6-luna` through the Codex harness. It
retains the original five checks for the first shippable checkpoint and
the original seven checks for completing the migration.
| Treatment | Result | Notes |
| --- | --- | --- |
| Baseline | 11/12 | Preserved the final behavior, but treated the work
as a fresh migration instead of explicitly continuing from the recorded
first checkpoint. |
| Guide (`AGENTS.md`) | 8/12 | Built successfully, but rewrote the
existing `unstable_cache`, changed the catalog's hourly cadence, moved
its timestamp to request time, and skipped the recorded incremental
checkpoint. |
| Adoption skill | 12/12 | Preserved both cache lifetimes and static
contracts, recorded a passing incremental checkpoint, removed the
remaining opt-outs, and completed the migration. |
## Verification
- `pnpm build`
- Luna eval runs for the baseline, guide, and adoption-skill treatments
- Prettier and ESLint checks for the changed guide, skill, and eval
files
- `git diff --check`
<!-- NEXT_JS_LLM -->
## Summary
- update existing evals for synchronous I/O, instant shell validation,
and Cache Components configuration instead of adding duplicate fixtures
- add focused Partial Prefetching coverage for URL-data inclusion, the
prefetch stage, and the navigation stage
- retain focused Cache Components coverage for session isolation and
URL-dependent shells
- rebase onto current `canary` so the Partial Prefetching evals can use
the public `unstable_prefetch()` and `unstable_navigation()` API
references
## Coverage
The latest results use Claude Sonnet 4.6 for the coding agent and Claude
Haiku 4.5 for the agentic judge. The Partial Prefetching cases have been
rerun after the new API references landed, and the corrected Cache
Components criteria have also been rerun.
| Eval | Scenario | Baseline | Bundled docs |
| --- | --- | ---: | ---: |
| `agent-035-connection-dynamic` | Adopt Cache Components while keeping
synchronous request-time values fresh and stable navigation in the shell
| 5/5 | 5/5 |
| `agent-040-instant` | Follow default validation and keep a stable
product title ahead of live inventory | 4/4 | 4/4 |
| `agent-042-enable-ppr` | Translate legacy route configuration while
preserving hourly and request-specific behavior | 4/4 | 4/4 |
| `agent-048-prefetch-url-data` | Include cached URL-specific category
content in a selected per-link prefetch | 3/6 | 6/6 |
| `agent-049-prefetch-stage` | Exclude cached content from the App Shell
but include it in a selected prefetch | 1/5 | 5/5 |
| `agent-051-prefetch-until-navigation` | Prefetch a selected title
while deferring the rest until navigation | 0/5 | 5/5 |
| `agent-052-cache-components-session-isolation` | Reuse shared data
without placing member-specific request data in a public cache | 5/5 |
5/5 |
| `agent-053-cache-components-url-shell` | Keep URL data below a useful
shell and include every output-changing value in the cache key | 3/5 |
5/5 |
After the public API references landed, the prefetch-stage and
navigation-stage evals show the intended documentation signal. Baseline
Sonnet passed 1/5 criteria for `unstable_prefetch()` and 0/5 for
`unstable_navigation()`. With the canonical bundled-docs instruction,
Sonnet found the new references and passed all 5/5 criteria in both
cases. After clarifying that the URL-data navigation must not block and
linking the general navigation docs to the optimizing-prefetching guide,
the URL-data case improved from 3/6 without docs to 6/6 with bundled
docs.
## Result interpretation
- `agent-048` now passes 6/6 with bundled docs. The scoped documentation
link led the agent to the URL-data prefetch pattern instead of a
blocking render. Baseline Sonnet added `prefetch={true}` but relied on
the previous full-payload behavior and missed both the Partial
Prefetching configuration and cached URL data, resulting in 3/6.
- The low baseline scores for `agent-049` and `agent-051` are the
intended signal. Without the new references, the agents did not discover
the prefetch and navigation stages. With bundled docs, both cases pass
5/5.
- `agent-053` is also an intended documentation signal: the baseline hid
the useful shell behind a broad fallback, while bundled docs preserved
the shell and passed 5/5.
- `agent-035` now passes 5/5 in both conditions after removing an
unnecessary requirement for visible loading UI in one small metadata
region. `agent-052` also passes 5/5 in both conditions after removing
two failures for an omitted explicit `cacheLife()`, because the default
cache profile still applies, and clarifying for the judge that an
uncached component reading `cookies()` or `headers()` remains
request-specific without another opt-out marker.
## Overlap decisions
- `agent-035` now carries the synchronous-I/O adoption case, so the
duplicate `agent-051` fixture was removed.
- `agent-042` now carries the route-configuration migration case, so the
duplicate `agent-048` fixture was removed.
- `agent-034` remains focused on correct async `cookies()` and
`headers()` usage. `agent-052` tests the separate risk of leaking
request-specific values through a public cache.
- `agent-040` tests one meaningful shell under default validation.
`agent-041` still tests independent streaming across several dashboard
sections.
- `agent-048` intentionally brings cached URL-specific content into a
selected prefetch. `agent-053` tests the opposite shared-shell
requirement during Cache Components adoption.
## Verification
- Dry fixture generation passes for all eight cases in the table.
- `pnpm build-all`
- `agent-049-prefetch-stage`: baseline 1/5, bundled docs 5/5
- `agent-051-prefetch-until-navigation`: baseline 0/5, bundled docs 5/5
- `agent-035-connection-dynamic`: baseline 5/5, bundled docs 5/5
- `agent-048-prefetch-url-data`: baseline 3/6, bundled docs 6/6
- `agent-052-cache-components-session-isolation`: baseline 5/5, bundled
docs 5/5
- `pnpm exec tsc -p
evals/evals/agent-035-connection-dynamic/tsconfig.json --noEmit`
- `pnpm exec tsc -p evals/evals/agent-042-enable-ppr/tsconfig.json
--noEmit`
<!-- NEXT_JS_LLM -->
Fixes false failures in `agent-030-app-router-migration-hard` when
`app/error.tsx` delegates to a shared component through an import,
re-export, or wrapper. Replaces the source-text regex with a semantic
check that follows the exported implementation and validates recovery
against the installed Next.js API, accepting supported `retry` or
`reset` callbacks.
Validated with the corrected Astra runs and nine additional experiment
reruns in https://github.com/vercel/next-evals-oss/pull/120. All nine
passed under pass@4; published scores are unchanged. Four Grok
experiments were blocked by provider access restrictions.
## Summary
Explain in the Building guide that `.next/dev` contains the active
development server state and incremental compilation caches. Deleting or
moving `.next` makes the development server recreate that state, while a
separate `distDir` provides isolated production output.
Make the `next-dev-loop` Skill explicit that moving `.next` to a backup
is still a reset. Keep the AI coding agents guide focused on discovering
and inspecting the running development server.
Update the agentic regression eval so it leaves two working routes and
their development server running before introducing an invalid
`generateStaticParams` signature. The production error points into
`.next/types`, but the source contains the defect. The eval
independently checks that the agent fixes the source, preserves `.next`,
and verifies the result through the existing development session.
## Eval results
| Model | Treatment | Result | Source fix | Preserve `.next` | Existing
dev verification |
| --- | --- | --- | --- | --- | --- |
| GPT-5.6 Luna | Baseline | 1/3 | Fail | Fail | Pass |
| GPT-5.6 Luna | Bundled Guide via `AGENTS.md` | 2/3 | Pass | Fail |
Pass |
| GPT-5.6 Luna | `next-dev-loop` Skill | 3/3 | Pass | Pass | Pass |
The baseline tried to clear `.next`, moved it aside when deletion was
blocked, and did not leave a valid source fix. The Guide treatment read
the relevant `generateStaticParams` API reference and fixed the source,
but still moved `.next` because it did not retrieve the general Building
guide. The Skill treatment read the explicit preservation guidance, kept
the original `.next` directory and development server active, used a
separate `distDir` for the production build, and verified both routes
through the running app.
## Verification
- `pnpm --filter=next build`
- Agentic eval with GPT-5.6 Luna: baseline, bundled Guide, and
`next-dev-loop` Skill
- Prettier and ESLint checks on the changed files
- `git diff --check`
<!-- NEXT_JS_LLM -->
### What?
Updates the App Router migration eval to accept the generated
LayoutProps helper as valid typing for root layout children.
### Why?
The current docs support both an inline ReactNode annotation and the
globally available LayoutProps helper. Opus 5 followed that guidance in
all four AGENTS.md runs, but the eval rejected LayoutProps and reported
a false regression.
### How?
The assertion still requires children and now accepts either documented
type form.
### Testing
- Prettier check
- ESLint
- Focused assertion check covering both valid forms and an untyped
invalid form
<!-- NEXT_JS_LLM -->
## Summary
Adds an agent eval covering migration of the legacy on-demand ISR
pattern: `force-static`, `revalidate`, and an empty
`generateStaticParams` result.
The assertions require the agent to enable Cache Components, preserve
the `generateStaticParams` export, and replace its empty result with at
least one parameter instead of deleting the function. The fixture also
runs the Cache Components adoption skill with the established 30-minute
timeout.
Updates the Cache Components migration guide to link each legacy route
segment config to its migration instructions. The `generateStaticParams`
section now explains that removing the export opts a dynamic route out
of ISR and links to the ISR with Cache Components guide.
## Eval results
### Before the guide change
| Treatment | Result | Duration | Behavior |
| --- | --- | ---: | --- |
| Baseline | Passed (1/1) | 482.9s | Preserved `generateStaticParams`
and returned a real seed. |
| Bundled docs / AGENTS.md | Failed (0/1) | 444.7s | Deleted
`generateStaticParams`, incorrectly treating omission as equivalent to
returning an empty array. |
| Cache Components skill | Passed (1/1) | 527.6s | Preserved
`generateStaticParams` and returned a real seed. |
The failed treatment was classified as a model failure. It read the
bundled migration guide but concluded that deleting
`generateStaticParams` preserved the defer-all-paths-to-runtime
behavior. The eval catches that loss of on-demand ISR semantics.
The full run used the identical fixture immediately before its numeric
identifier was changed from `agent-044` to the final `agent-054` name.
The fixture and assertions were unchanged.
### After the guide change
| Treatment | Result | Duration | Behavior |
| --- | --- | ---: | --- |
| Baseline | Passed (1/1) | 739.5s | Preserved `generateStaticParams`,
returned `FEATURED_EVENT_SLUG`, and compared the migrated route with the
original behavior. |
| Bundled docs / AGENTS.md | Passed (1/1) | 393.2s | Read the revised
migration and ISR guides, preserved `generateStaticParams`, and returned
`FEATURED_EVENT_SLUG`. |
| Cache Components skill | Passed (1/1) | 547.3s | Preserved
`generateStaticParams`, returned `FEATURED_EVENT_SLUG`, and verified the
build and representative routes in development. |
In the first controlled pair, the bundled-docs treatment changed from
failing to passing. Its final migration kept the route eligible for ISR,
preserved the 60-second revalidation behavior with `use cache` and
`cacheLife`, and produced a passing build.
### Additional variance check
The bundled-docs treatment was run once more against each guide version:
| Guide version | First sample | Second sample | Observed result |
| --- | --- | --- | --- |
| Before this change | Failed (444.7s) | Passed (455.6s) | 1/2 passed |
| After this change | Passed (393.2s) | Passed (425.2s) | 2/2 passed |
The second pre-change sample shows that the previous wording could still
lead the agent to the correct migration. The revised wording passed both
observed samples, suggesting that the explicit ISR consequence improves
reliability. Two samples per version are not enough to claim a stable
pass rate.
## Verification
- `pnpm --filter=next build`
- `pnpm eval agent-054-cache-components-empty-static-params --dry`
- `pnpm eval agent-054-cache-components-empty-static-params`
<!-- NEXT_JS_LLM -->
The view-transitions eval kept requiring a config option that #96098
removed, so models following the current docs were failed for correct
solutions. Evals only, no runtime change.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
- add end-to-end adoption evals for Partial Prefetching and Cache
Components
- compare the baseline, bundled docs, and local adoption skill
independently
- support local skill installation and longer timeouts for
browser-backed workflows
## Results
One `claude-opus-4-8` run was retained per treatment and adoption eval,
judged by `claude-haiku-4-5`. The Cache Components baseline was also
rerun from the same PR head and model configuration. These results are
directional rather than statistically stable.
| Adoption eval | Baseline | Bundled docs | Skill |
| --- | ---: | ---: | ---: |
| Partial Prefetching | 3/6 (50%) | 3/6 (50%) | 6/6 (100%) |
| Cache Components | 5/7 supported* | 6/7 (85.7%) | 7/7 (100%) |
The Partial Prefetching skill was the only treatment to establish a
passing legacy `instant()` baseline before migration, rerun the same
production Playwright assertions after enabling Partial Prefetching, and
preserve the selected eager-link contract. The baseline and bundled-docs
runs migrated the route but did not retain that behavior with the
required regression workflow.
### Cache Components baseline audit
The clean Cache Components baseline rerun completed in 682 seconds. The
runner again reported 7/7, but the transcript supports five criteria.
Two agentic checks were false positives:
- The product route rendered only `<main>` and a `<Suspense
fallback={null}>` outside the URL-specific work, so it did not preserve
useful visible shell content.
- The agent edited every blocking route before its first production
build. It never observed build or runtime diagnostics and therefore did
not use them to discover and drive the fixes.
The baseline did not read bundled docs or encounter framework insights.
It inspected the installed Next.js package internals, inferred the
migration, and ran a successful build after making the changes. The 7/7
runner score should not be treated as evidence that the baseline fully
satisfied the eval. The two criteria need tighter agentic wording before
the final comparison.
The current bundled-docs and skill scores remain 6/7 and 7/7. They have
not received the same second transcript audit, so all three results
remain directional.
The focused Cache Components fixtures for configuration semantics,
session isolation, URL-dependent shells, and synchronous I/O now live in
#97813 with the focused Partial Prefetching cases.
## Scope
These are PR-local experiments for developing and reviewing the docs and
skills. This PR does not register them in the public `evals.nextjs.org`
suite, which is configured separately in the OSS eval repository.
## Validation
- `pnpm exec tsc -p evals/tsconfig.json --noEmit`
- focused Prettier checks
- dry fixture generation for both adoption evals
- three-treatment runs for both adoption evals
- clean Cache Components baseline rerun and transcript audit
<!-- NEXT_JS_LLM -->
Adds evals that check whether an agent reaches for Next.js when asked to
build something, and whether what it ships is really Next.js rather than
something that only resembles it.
Makes a few of the evals judge what the code actually does rather than
how it happens to be written, so correct solutions stop failing on
style. One of them was already being vetoed by leftover assertions the
judge was meant to replace.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Make the ppr test accept diverse solutions as long as they lead to a
static shell. Achieved by `expect(environment).toSatisfyCriterion`
(a.k.a., llm-as-judge)
Fixes false-negative regex assertions — observed rejecting correct
solutions in recent eval runs — by grading those specific checks with
`@vercel/agent-eval`'s agentic LLM judge instead. Only the assertions
with demonstrated false negatives change; every deterministic check
stays.
- `agent-034-async-cookies`: `/await\s+cookies\s*\(\s*\)/` only matched
the naive `await cookies()` and rejected the correct (arguably better)
`await Promise.all([cookies(), headers()])` — observed failing most runs
of a strong model — and the no-sync-call lookbehind wrongly flagged the
bare `cookies()` inside the array. The four await-mechanics tests
collapse into one judge criterion: the promises must actually be awaited
before use, in any correct form. The theme-cookie and Accept-Language
content checks stay regex.
- `agent-030-app-router-migration-hard`: the home-page test rejected
solutions that didn't match one exact shape (e.g. data fetching
extracted to a helper means no literal `fetch(` in `page.tsx`). Its body
becomes one judge criterion (async Server Component, fetches during
server render, no `getServerSideProps`); the file-existence check and
the other seven tests stay deterministic.
The judge is pinned to **`claude-haiku-4-5`** in the generated
experiment configs — a cheap fixed grader, identical for every run
regardless of the model under test. Each criterion states the
requirement semantically, includes one reference solution to ground the
small grader, and fixes the standard of judgment (runtime correctness,
not style) — no enumerated failure modes, so the judge stays flexible
about equivalent forms.
Supporting changes: bumps `@vercel/agent-eval` `0.9.5` → `1.3.0` (ships
the judge runtime) and adapts `run-evals.js` to the 1.3 CLI (`run
baseline agents-md --force`; `--dry` maps to `status`); generated
experiments now run through the Vercel AI Gateway so `vc env pull` is
the only credential setup; converted fixtures exclude `EVAL.ts` from
tsconfig (the `@vercel/agent-eval/eval` import has no type declarations
— vercel-labs/agent-eval#166 — and `next build` type-checks the fixture,
so experiments running `scripts: ['build']` would otherwise fail on it,
same reason `EVAL.tsx` was already excluded). Not converted: `agent-040`
(already redesigned upstream in #94578) and `agent-041` (no diagnosed
false negative in the report).
Verified end-to-end from this repo: converted `agent-034` with codegen
`claude-sonnet-4-5` via the AI Gateway in a Vercel sandbox, judge pinned
to `claude-haiku-4-5` — passes, with the judge clause executing
in-sandbox.
**Running it:**
```bash
pnpm install # picks up @vercel/agent-eval 1.3.0
pnpm --filter=next build # pnpm eval packs the locally built next
vc env pull .env.local # the only credential: sandbox + AI Gateway (codegen and judge)
pnpm eval agent-034-async-cookies # baseline + AGENTS.md variants
pnpm eval agent-034-async-cookies --dry # preview only
```
<!-- NEXT_JS_LLM_PR -->
## Summary
- Adds `agent-043-view-transitions`, a new eval that tests whether
agents can add React View Transitions to a Next.js product gallery app
- Starter app is a simple product list/detail gallery with Suspense
boundaries, no view transitions
- EVAL.ts checks 8 patterns: `viewTransition` config flag,
`ViewTransition` imported from `react`, shared element `name` props,
`transitionTypes` on Link, Suspense enter/exit animations,
`default="none"` isolation, `prefers-reduced-motion` CSS, and
`::view-transition-*` CSS pseudo-elements
## Test plan
- [x] Starter app builds with `next build`
- [x] All 8 EVAL.ts tests fail on unmodified starter (no false
positives)
- [x] All 8 EVAL.ts tests pass on a correct golden solution
- [x] Golden solution builds with `next build` (viewTransition
experiment detected)
- [x] Commit passes lint-staged (prettier + eslint)
Upgrades `@vercel/agent-eval` from 0.8.0 to 0.9.5. The new version loads
`.env` / `.env.local` from its own cwd (`evals/`) rather than inheriting
from the parent process, so the old `process.loadEnvFile()` call no
longer works. Instead, `run-evals.js` now symlinks the root env files
into `evals/` before spawning the harness. `evals/.env` is added to
`.gitignore` to keep secrets out of the tree.
Two new evals exercise PPR knowledge: `agent-041-optimize-ppr-shell`
checks that the agent decomposes a monolithic `loading.tsx` into
granular `Suspense` boundaries, and `agent-042-enable-ppr` verifies the
agent knows PPR is enabled via `cacheComponents: true` (not the old
`experimental.ppr` flag).
Fixtures now live next to the code they test, like e2e. `pnpm eval
<name>` packs the local `next` build, generates baseline + agents-md
experiment configs on the fly, and runs both in a sandbox. The agents-md
variant drops an `AGENTS.md` that points the agent at the bundled docs
in `node_modules/next/dist/docs/` — comparing the two variants tells you
whether shipping a doc actually changes agent behavior.
`run-evals.js` mirrors `run-tests.js`: pack once, pass the tarball path
to the child via `NEXT_EVAL_TARBALL` env, forward flags. We only pack
`next`, not the whole workspace — the sandbox is remote Linux, so a
local `@next/swc` darwin binary wouldn't run there anyway; the sandbox
downloads the right one at runtime. The experiment config uses `sandbox:
'auto'`, which picks Vercel sandboxes when credentials are present and
falls back to local Docker otherwise, so external contributors can run
the same evals with just Docker + `ANTHROPIC_API_KEY`.
`experiments/` is generated fresh each run and gitignored so we don't
maintain N committed config files that differ by one line. Fixture code
is excluded from eslint since it's deliberately imperfect code for
agents to fix, and `EVAL.ts` uses vitest rather than jest. Fixture
`package.json` files use `"next": "^16"` rather than a pinned canary so
agents reading `package.json` to infer capabilities aren't misled by a
stale version string; the tarball install overlays it regardless.
next-evals-oss stays as the full benchmark runner for nextjs.org/evals;
it'll pull fixtures from here instead of keeping its own copy.
Aurora Scharff