Fixes false-negative regex assertions — observed rejecting correct
solutions in recent eval runs — by grading those specific checks with
`@vercel/agent-eval`'s agentic LLM judge instead. Only the assertions
with demonstrated false negatives change; every deterministic check
stays.
- `agent-034-async-cookies`: `/await\s+cookies\s*\(\s*\)/` only matched
the naive `await cookies()` and rejected the correct (arguably better)
`await Promise.all([cookies(), headers()])` — observed failing most runs
of a strong model — and the no-sync-call lookbehind wrongly flagged the
bare `cookies()` inside the array. The four await-mechanics tests
collapse into one judge criterion: the promises must actually be awaited
before use, in any correct form. The theme-cookie and Accept-Language
content checks stay regex.
- `agent-030-app-router-migration-hard`: the home-page test rejected
solutions that didn't match one exact shape (e.g. data fetching
extracted to a helper means no literal `fetch(` in `page.tsx`). Its body
becomes one judge criterion (async Server Component, fetches during
server render, no `getServerSideProps`); the file-existence check and
the other seven tests stay deterministic.
The judge is pinned to **`claude-haiku-4-5`** in the generated
experiment configs — a cheap fixed grader, identical for every run
regardless of the model under test. Each criterion states the
requirement semantically, includes one reference solution to ground the
small grader, and fixes the standard of judgment (runtime correctness,
not style) — no enumerated failure modes, so the judge stays flexible
about equivalent forms.
Supporting changes: bumps `@vercel/agent-eval` `0.9.5` → `1.3.0` (ships
the judge runtime) and adapts `run-evals.js` to the 1.3 CLI (`run
baseline agents-md --force`; `--dry` maps to `status`); generated
experiments now run through the Vercel AI Gateway so `vc env pull` is
the only credential setup; converted fixtures exclude `EVAL.ts` from
tsconfig (the `@vercel/agent-eval/eval` import has no type declarations
— vercel-labs/agent-eval#166 — and `next build` type-checks the fixture,
so experiments running `scripts: ['build']` would otherwise fail on it,
same reason `EVAL.tsx` was already excluded). Not converted: `agent-040`
(already redesigned upstream in #94578) and `agent-041` (no diagnosed
false negative in the report).
Verified end-to-end from this repo: converted `agent-034` with codegen
`claude-sonnet-4-5` via the AI Gateway in a Vercel sandbox, judge pinned
to `claude-haiku-4-5` — passes, with the judge clause executing
in-sandbox.
**Running it:**
```bash
pnpm install # picks up @vercel/agent-eval 1.3.0
pnpm --filter=next build # pnpm eval packs the locally built next
vc env pull .env.local # the only credential: sandbox + AI Gateway (codegen and judge)
pnpm eval agent-034-async-cookies # baseline + AGENTS.md variants
pnpm eval agent-034-async-cookies --dry # preview only
```
<!-- NEXT_JS_LLM_PR -->
`@typescript-eslint/switch-exhaustiveness-check` (enabled in
eslint.cli.config.mjs) already guarantees complete switch coverage on
enums and discriminated unions, so requiring an additional default case
on those switches in TypeScript just forced dead branches.
We turn on its `requireDefaultForNonUnion` option so that switches on
plain types such as `string` or `number` still require a default,
preserving the coverage that `default-case` previously gave us. We scope
the `default-case: off` override to exactly the files where the
type-checked rule runs — the same `files` and `ignores` as the
switch-exhaustiveness check — and cross-reference the two config blocks
so the globs stay in sync. That way `default-case` stays enabled for
JavaScript files, `.mts` files, and the directories the type-checked
config skips, so no switch is left unchecked.
The two `// eslint-disable-next-line default-case` directives in
`postcss-loader` and `head.tsx` are no longer the right escape hatch:
with `requireDefaultForNonUnion` the exhaustiveness check now wants
those switches handled, and a directive for that rule would be reported
as unused under the editor config where it does not run. We give each
switch an explicit `default` case instead, which both rules accept and
which makes the previously implicit fall-through behavior explicit.
Jude Gao