7 Commits

Author	SHA1	Message	Date
Aurora Scharff	4c2e9ccdc6	Add agent feedback behavior evals (#98887) ... ### What? Adds eval coverage for the experimental agent feedback workflow: - Routine debugging should not produce a report - Qualifying friction should produce an anonymized structured report - Distinct issues should produce separate review forms Also adds repeat-run and variant controls so trigger frequency can be measured across multiple runs. Local-Skill and agent-feedback/privacy evals are marked `publish: false`, with scoped `evals/AGENTS.md` instructions preventing their fixtures, transcripts, or scores from being exported to the public benchmark. ### Why? We need to measure trigger precision, anonymization, issue splitting, and duplicate prevention before expanding the experiment. ### How? Depends on #98582. The `agent-feedback` treatment uses the managed block and bundled reporting protocol from the parent PR. Only the remote rollout gate is forced on inside the eval sandbox so runs are deterministic. | Eval | Baseline | Agent rules | Agent feedback | 10-run treatment | | --- | --- | --- | --- | --- | | Routine debugging | Pending | Pending | 1/1 passed | Pending | | Anonymization | Pending | Pending | 2/2 reporting checks passed | Pending | | Distinct issues | Pending | Pending | 1/1 passed | Pending | The first attempted run did not reach the agent because the local sandbox was not linked to a Vercel project. It is infrastructure setup and is not included in the results above. The first distinct-issues run produced the expected two separate payloads. Its scorer rejected them because the parser did not allow the existing `token` query parameter and the browser criterion required an open attempt even when the agent environment exposed no browser capability. After correcting those assertions and clarifying the stopping-point wording, the scored rerun passed. Both anonymization treatments produced one valid payload with none of the seeded customer, project, route, local-path, internal-URL, or secret values. The corrected rerun also selected `misleading-error` and passed every reporting assertion. An inherited `.next`-preservation assertion was removed from this fixture because it measures `next-dev-loop` behavior, not agent-feedback anonymization; Skill queue coordination belongs in a separate focused eval.	2026-09-18 16:38:50 +00:00
Jiwon Choi	5d9ab72cef	Add next upgrade --ai and security vulnerability coverage (#98562) ... > [!TIP] > Recommended to review commit by commit. This PR adds `next upgrade --experimental-ai="security"` flag (alias `--ai`), which is targeted to help users leverage agents to upgrade their app to the safe major version when their app's Next.js version has any security advisories. Once the command is ran from the user, Next.js will detect the installed agent harness in user's device, currently limited to Codex and Claude, and will proceed with starting an agent session once approved. If it is called within an agent session, the work will continue off within that agent. `next upgrade --ai` simply does two things: - prepare the relevant context to temporary dir - print hand off prompt, guiding to read those context The context will guide the agent to run relevant codemods and migration checklist to proceed. This PR is a base core of the workflow, and will have wrappers of entry point around this. Also, will add "latest" and "future" as follow up, which will cover the app to be always latest, and adopt the future defaults like Cache Components. This PR also sets up the evals infra and adds evals.	2026-09-18 00:52:20 +02:00
Aurora Scharff	393de1ebe0	skill: add Partial Prefetching optimizer (#96471) ... ## Summary - add a Partial Prefetching optimizer Skill for selected source Links and destinations - keep prefetch stages and Link policy in the public [Optimizing prefetching guide](https://nextjs.org/docs/app/guides/optimizing-prefetching), and rely on the same static-shell documentation as the Cache Components optimizer for cache placement, Suspense, loading UI, and authorization - keep the Skill focused on the product decision, production `instant()` loop, differential verification, and mutation freshness when an optimization adds a cache around mutable data - reference the canonical [`unstable_prefetch()`](https://nextjs.org/docs/app/api-reference/functions/prefetch) and [`unstable_navigation()`](https://nextjs.org/docs/app/api-reference/functions/navigation) docs instead of explaining the APIs again in the Skill - link cache-writing optimizations to the canonical [Revalidating](https://nextjs.org/docs/app/getting-started/caching-and-revalidating#revalidating) documentation and verify a populated-cache mutation in the Skill and eval - bundle a self-contained production rig aligned with the related adoption and optimizer Skills, tuned to record each source-link and prefetch-stage contract - add an eval that compares baseline, the bundled guide, and the local Skill against the same prefetched-UI and cache-freshness goal - document the optimizer in the AI agents guide This follows the same guide/Skill split as #97737. Partial Prefetching adoption preserves existing behavior; this optimizer is the later decision about what is worth preparing before a click and what should wait for navigation. Related: #97712, #97737, #97759, #98122, #98175 ## Eval One run per treatment with Claude Opus 4.8 and Claude Sonnet 4.5, using a fixed Claude Haiku 4.5 judge. The bundled-guide and Skill treatments use a local Next.js tarball containing the new guide and API references. | Model | Treatment | Criteria | Score change | Overall | Duration | Tokens processed | Total run cost | Cost change | | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | Opus 4.8 | Baseline | 10/10 (100%) | — | Pass | 18m 42s | 3.39M | $3.46 | — | | Opus 4.8 | Bundled guide | 9/10 (90%) | -10 pp | Fail | 13m 37s | 3.02M | $2.84 | -18.0% | | Opus 4.8 | Local Skill | 10/10 (100%) | +10 pp | Pass | 17m 22s | 2.80M | $2.54 | -10.6% | | Sonnet 4.5 | Baseline | 2/10 (20%) | — | Fail | 15m 55s | 1.97M | $1.06 | — | | Sonnet 4.5 | Bundled guide | 5/10 (50%) | +30 pp | Fail | 15m 49s | 2.17M | $1.11 | +4.5% | | Sonnet 4.5 | Local Skill | 8/10 (80%) | +30 pp | Fail | 15m 31s | 1.41M | $0.77 | -30.4% | Token totals count each unique model message once and include uncached input, cache creation, cache reads, and output. Cache reads account for most of the total in every treatment. Total run cost applies the current AI Gateway base rates to each token category: Opus 4.8 costs $5/M input, $6.25/M cache writes, $0.50/M cache reads, and $25/M output; Sonnet 4.5 costs $3/M input, $3.75/M cache writes, $0.30/M cache reads, and $15/M output. The six agent runs cost $11.78 in total. Compared with baseline, the Opus Skill run maintained a 100% criteria pass rate while reducing cost by 26.7%. The Sonnet Skill run improved the criteria pass rate from 20% to 80%, a 60 percentage-point increase, while reducing cost by 27.3%. The stronger model could solve the fixture without the Skill, although it independently searched the bundled Next.js documentation and took slightly longer than the Skill treatment. The Opus guide-only run implemented the correct behavior but missed the required RED-before-GREEN proof. The Sonnet treatments show a clearer progression. Baseline did not implement the required caching, navigation stage, `instant()` contract, or cache invalidation. The bundled guide added the cached summary and production test, but missed the navigation stage, deferred-state assertions, tag invalidation, and RED-before-GREEN proof. The Skill treatment completed the `instant()` RED-to-GREEN loop and final production behavior, but left `connection()` in the related-sessions read and used path revalidation instead of tagging the cached read. It therefore passed 8 of 10 criteria but failed the overall eval. All six measured runs completed through grading. Chromium and its system dependencies are provisioned before the agent starts, so missing browser libraries did not determine these results. ## Verification - `pnpm --filter=next build` - `pnpm prettier --check` for the changed guide, prompt, Skill, and eval fixture - `pnpm exec tsc -p evals/evals/agent-051-optimize-partial-prefetching/tsconfig.json --noEmit` - `node --check run-evals.js` - `git diff --check` - packed a local Next.js tarball and confirmed it contains the optimizer guide plus the `navigation()` and `prefetch()` references - verified the `navigation()` contract in the minimal reproduction and NextBeats on Next.js 16.4.0-canary.22 - merged the latest `canary`, preserving both generic fixture setup and Playwright provisioning <!-- NEXT_JS_LLM -->	2026-09-10 22:27:36 +02:00
Aurora Scharff	552dd3538d	docs: preserve Next.js development state (#98276) ... ## Summary Explain in the Building guide that `.next/dev` contains the active development server state and incremental compilation caches. Deleting or moving `.next` makes the development server recreate that state, while a separate `distDir` provides isolated production output. Make the `next-dev-loop` Skill explicit that moving `.next` to a backup is still a reset. Keep the AI coding agents guide focused on discovering and inspecting the running development server. Update the agentic regression eval so it leaves two working routes and their development server running before introducing an invalid `generateStaticParams` signature. The production error points into `.next/types`, but the source contains the defect. The eval independently checks that the agent fixes the source, preserves `.next`, and verifies the result through the existing development session. ## Eval results | Model | Treatment | Result | Source fix | Preserve `.next` | Existing dev verification | | --- | --- | --- | --- | --- | --- | | GPT-5.6 Luna | Baseline | 1/3 | Fail | Fail | Pass | | GPT-5.6 Luna | Bundled Guide via `AGENTS.md` | 2/3 | Pass | Fail | Pass | | GPT-5.6 Luna | `next-dev-loop` Skill | 3/3 | Pass | Pass | Pass | The baseline tried to clear `.next`, moved it aside when deletion was blocked, and did not leave a valid source fix. The Guide treatment read the relevant `generateStaticParams` API reference and fixed the source, but still moved `.next` because it did not retrieve the general Building guide. The Skill treatment read the explicit preservation guidance, kept the original `.next` directory and development server active, used a separate `distDir` for the production build, and verified both routes through the running app. ## Verification - `pnpm --filter=next build` - Agentic eval with GPT-5.6 Luna: baseline, bundled Guide, and `next-dev-loop` Skill - Prettier and ESLint checks on the changed files - `git diff --check` <!-- NEXT_JS_LLM -->	2026-09-08 23:54:37 +02:00
Aurora Scharff	d434afa837	test: add adoption skill evals (#97759) ... ## Summary - add end-to-end adoption evals for Partial Prefetching and Cache Components - compare the baseline, bundled docs, and local adoption skill independently - support local skill installation and longer timeouts for browser-backed workflows ## Results One `claude-opus-4-8` run was retained per treatment and adoption eval, judged by `claude-haiku-4-5`. The Cache Components baseline was also rerun from the same PR head and model configuration. These results are directional rather than statistically stable. | Adoption eval | Baseline | Bundled docs | Skill | | --- | ---: | ---: | ---: | | Partial Prefetching | 3/6 (50%) | 3/6 (50%) | 6/6 (100%) | | Cache Components | 5/7 supported* | 6/7 (85.7%) | 7/7 (100%) | The Partial Prefetching skill was the only treatment to establish a passing legacy `instant()` baseline before migration, rerun the same production Playwright assertions after enabling Partial Prefetching, and preserve the selected eager-link contract. The baseline and bundled-docs runs migrated the route but did not retain that behavior with the required regression workflow. ### Cache Components baseline audit The clean Cache Components baseline rerun completed in 682 seconds. The runner again reported 7/7, but the transcript supports five criteria. Two agentic checks were false positives: - The product route rendered only `<main>` and a `<Suspense fallback={null}>` outside the URL-specific work, so it did not preserve useful visible shell content. - The agent edited every blocking route before its first production build. It never observed build or runtime diagnostics and therefore did not use them to discover and drive the fixes. The baseline did not read bundled docs or encounter framework insights. It inspected the installed Next.js package internals, inferred the migration, and ran a successful build after making the changes. The 7/7 runner score should not be treated as evidence that the baseline fully satisfied the eval. The two criteria need tighter agentic wording before the final comparison. The current bundled-docs and skill scores remain 6/7 and 7/7. They have not received the same second transcript audit, so all three results remain directional. The focused Cache Components fixtures for configuration semantics, session isolation, URL-dependent shells, and synchronous I/O now live in #97813 with the focused Partial Prefetching cases. ## Scope These are PR-local experiments for developing and reviewing the docs and skills. This PR does not register them in the public `evals.nextjs.org` suite, which is configured separately in the OSS eval repository. ## Validation - `pnpm exec tsc -p evals/tsconfig.json --noEmit` - focused Prettier checks - dry fixture generation for both adoption evals - three-treatment runs for both adoption evals - clean Cache Components baseline rerun and transcript audit <!-- NEXT_JS_LLM -->	2026-08-31 20:52:36 +02:00
Jude Gao	f7ff5fd588	Add evals for whether agents actually use Next.js (#97816) ... Adds evals that check whether an agent reaches for Next.js when asked to build something, and whether what it ships is really Next.js rather than something that only resembles it.	2026-08-25 20:38:51 +00:00
Jude Gao	0921733a94	[Evals] Standardize Evals in Next.js (#90883) ... Fixtures now live next to the code they test, like e2e. `pnpm eval <name>` packs the local `next` build, generates baseline + agents-md experiment configs on the fly, and runs both in a sandbox. The agents-md variant drops an `AGENTS.md` that points the agent at the bundled docs in `node_modules/next/dist/docs/` — comparing the two variants tells you whether shipping a doc actually changes agent behavior. `run-evals.js` mirrors `run-tests.js`: pack once, pass the tarball path to the child via `NEXT_EVAL_TARBALL` env, forward flags. We only pack `next`, not the whole workspace — the sandbox is remote Linux, so a local `@next/swc` darwin binary wouldn't run there anyway; the sandbox downloads the right one at runtime. The experiment config uses `sandbox: 'auto'`, which picks Vercel sandboxes when credentials are present and falls back to local Docker otherwise, so external contributors can run the same evals with just Docker + `ANTHROPIC_API_KEY`. `experiments/` is generated fresh each run and gitignored so we don't maintain N committed config files that differ by one line. Fixture code is excluded from eslint since it's deliberately imperfect code for agents to fix, and `EVAL.ts` uses vitest rather than jest. Fixture `package.json` files use `"next": "^16"` rather than a pinned canary so agents reading `package.json` to infer capabilities aren't misled by a stale version string; the tarball install overlays it regardless. next-evals-oss stays as the full benchmark runner for nextjs.org/evals; it'll pull fixtures from here instead of keeping its own copy.	2026-03-06 12:05:36 -05:00