mirror of
https://github.com/vercel/next.js.git
synced 2026-09-20 02:25:18 +08:00
Aurora Scharff
4c2e9ccdc6
### What? Adds eval coverage for the experimental agent feedback workflow: - Routine debugging should not produce a report - Qualifying friction should produce an anonymized structured report - Distinct issues should produce separate review forms Also adds repeat-run and variant controls so trigger frequency can be measured across multiple runs. Local-Skill and agent-feedback/privacy evals are marked `publish: false`, with scoped `evals/AGENTS.md` instructions preventing their fixtures, transcripts, or scores from being exported to the public benchmark. ### Why? We need to measure trigger precision, anonymization, issue splitting, and duplicate prevention before expanding the experiment. ### How? Depends on #98582. The `agent-feedback` treatment uses the managed block and bundled reporting protocol from the parent PR. Only the remote rollout gate is forced on inside the eval sandbox so runs are deterministic. | Eval | Baseline | Agent rules | Agent feedback | 10-run treatment | | --- | --- | --- | --- | --- | | Routine debugging | Pending | Pending | 1/1 passed | Pending | | Anonymization | Pending | Pending | 2/2 reporting checks passed | Pending | | Distinct issues | Pending | Pending | 1/1 passed | Pending | The first attempted run did not reach the agent because the local sandbox was not linked to a Vercel project. It is infrastructure setup and is not included in the results above. The first distinct-issues run produced the expected two separate payloads. Its scorer rejected them because the parser did not allow the existing `token` query parameter and the browser criterion required an open attempt even when the agent environment exposed no browser capability. After correcting those assertions and clarifying the stopping-point wording, the scored rerun passed. Both anonymization treatments produced one valid payload with none of the seeded customer, project, route, local-path, internal-URL, or secret values. The corrected rerun also selected `misleading-error` and passed every reporting assertion. An inherited `.next`-preservation assertion was removed from this fixture because it measures `next-dev-loop` behavior, not agent-feedback anonymization; Skill queue coordination belongs in a separate focused eval.