mirror of
https://github.com/vercel/next.js.git
synced 2026-09-20 02:25:18 +08:00
Aurora Scharff
4c2e9ccdc6
### What? Adds eval coverage for the experimental agent feedback workflow: - Routine debugging should not produce a report - Qualifying friction should produce an anonymized structured report - Distinct issues should produce separate review forms Also adds repeat-run and variant controls so trigger frequency can be measured across multiple runs. Local-Skill and agent-feedback/privacy evals are marked `publish: false`, with scoped `evals/AGENTS.md` instructions preventing their fixtures, transcripts, or scores from being exported to the public benchmark. ### Why? We need to measure trigger precision, anonymization, issue splitting, and duplicate prevention before expanding the experiment. ### How? Depends on #98582. The `agent-feedback` treatment uses the managed block and bundled reporting protocol from the parent PR. Only the remote rollout gate is forced on inside the eval sandbox so runs are deterministic. | Eval | Baseline | Agent rules | Agent feedback | 10-run treatment | | --- | --- | --- | --- | --- | | Routine debugging | Pending | Pending | 1/1 passed | Pending | | Anonymization | Pending | Pending | 2/2 reporting checks passed | Pending | | Distinct issues | Pending | Pending | 1/1 passed | Pending | The first attempted run did not reach the agent because the local sandbox was not linked to a Vercel project. It is infrastructure setup and is not included in the results above. The first distinct-issues run produced the expected two separate payloads. Its scorer rejected them because the parser did not allow the existing `token` query parameter and the browser criterion required an open attempt even when the agent environment exposed no browser capability. After correcting those assertions and clarifying the stopping-point wording, the scored rerun passed. Both anonymization treatments produced one valid payload with none of the seeded customer, project, route, local-path, internal-URL, or secret values. The corrected rerun also selected `misleading-error` and passed every reporting assertion. An inherited `.next`-preservation assertion was removed from this fixture because it measures `next-dev-loop` behavior, not agent-feedback anonymization; Skill queue coordination belongs in a separate focused eval.
42 lines
1.0 KiB
JSON
42 lines
1.0 KiB
JSON
{
|
|
"agent-046-adopt-partial-prefetching": {
|
|
"skills": ["next-partial-prefetching-adoption"],
|
|
"publish": false,
|
|
"timeout": 1800
|
|
},
|
|
"agent-047-adopt-cache-components": {
|
|
"skills": ["next-cache-components-adoption"],
|
|
"publish": false,
|
|
"timeout": 1800
|
|
},
|
|
"agent-051-optimize-partial-prefetching": {
|
|
"skills": ["next-partial-prefetching-optimizer"],
|
|
"publish": false,
|
|
"timeout": 1800
|
|
},
|
|
"agent-054-cache-components-empty-static-params": {
|
|
"skills": ["next-cache-components-adoption"],
|
|
"timeout": 1800
|
|
},
|
|
"agent-055-stale-build-error": {
|
|
"skills": ["next-dev-loop"],
|
|
"publish": false,
|
|
"timeout": 900
|
|
},
|
|
"agent-056-agent-feedback-routine-debugging": {
|
|
"agentFeedback": true,
|
|
"publish": false,
|
|
"timeout": 900
|
|
},
|
|
"agent-057-agent-feedback-anonymization": {
|
|
"agentFeedback": true,
|
|
"publish": false,
|
|
"timeout": 1200
|
|
},
|
|
"agent-058-agent-feedback-distinct-issues": {
|
|
"agentFeedback": true,
|
|
"publish": false,
|
|
"timeout": 900
|
|
}
|
|
}
|