Files
cloudflare__vinext/tests/classify-nextjs-suites.test.ts
T
James Anderson f27c40a77f chore: split e2e compatibility by App Router vs Pages Router (#1321)
* feat(compat): split e2e compatibility by App Router vs Pages Router

The /compatibility page previously showed one undifferentiated grid of
~1000 Next.js test files. This change classifies each test by which
router(s) its fixture exercises (App, Pages, both, or unknown) and
surfaces the breakdown in the UI.

How:

- A new `router` column on `compat_file_results` (enum: app | pages |
  both | unknown). Defaults to 'unknown' so pre-classifier rows still
  render — they just show up under 'Other'.
- A new `scripts/classify-nextjs-suites.mjs` walks each test's fixture
  directory and looks for app/page.tsx, app/route.ts, app/layout.tsx,
  pages/*.tsx (excluding _app / _document / _error specials). A fixture
  with real routes in both folders is classified as 'both' — these are
  the genuine parity tests and counting them only once would hide a
  router-specific failure.
- Edge cases handled in the classifier:
    * pageExtensions naming (layout.page.tsx → layout)
    * Suites whose .test.ts lives in a test/ subdir alongside the fixture
    * Inline-fixture suites under test/e2e/app-dir/ (no on-disk routes,
      but path convention says App Router)
    * APP_ROUTER_NON_APP_DIR_SUITES curated override
    * Skips node_modules and .next when scanning fixtures
- The nightly workflow now runs the classifier in the build job once
  per run (the cheap part — happens with Next.js still on disk),
  uploads the suite → router map as an artifact, and the report job
  joins it into the ingest payload before POSTing.
- The compatibility page gains a 'By router' card row showing per-router
  pass rates and file counts (parity tests are counted toward both
  router buckets — adding them exceeds the total, see the explainer
  card). The contribution grid grows filter chips above it for
  interactive narrowing.

Verified:

- vp check (format + type + lint) clean
- 14 new classifier unit tests pass (vp test run tests/classify-nextjs-suites.test.ts)
- apps/web build:vinext succeeds end-to-end
- Local classifier run against .nextjs-ref reports
  app=576 / pages=325 / both=113 / unknown=22 out of 1036 suites, with
  spot-checked classifications matching expectations.

Backward compatibility: the ingest endpoint treats `router` as optional;
existing workflow runs and historical rows continue to function and
render as 'Other' until the next nightly classifies them.

* refactor(compat): store router classification in its own table

Replaces the inline `router` column on `compat_file_results` (introduced
in the previous commit) with a dedicated `compat_suite_meta` table,
classification keyed by `suite`. The /compatibility UI LEFT JOINs the
two tables at query time.

Rationale (per design discussion in PR #1321):

- Classifications conceptually describe test files, not test runs. Storing
  them per-row coupled their cadence to results ingestion, which is wrong
  when (a) the Next.js ref bumps and re-classifies everything, (b) an
  override fix lands without re-running tests, or (c) a partial test run
  still wants fresh classifications.
- One row per suite (PK on `suite`) means re-classifying is an upsert,
  not a backfill loop. Provenance (`next_ref`, `classified_at`) is
  stored on the row for debugging.
- Decoupling lets the workflow POST classifications from the build job
  immediately after running the classifier, without round-tripping through
  the report job. Results ingestion stays focused on results.

Endpoints:
- `POST /api/compatibility` no longer accepts `router` per file
  (reverted to its pre-PR shape).
- `POST /api/compatibility/classify` (new): accepts
  `{ nextRef, classifiedAt?, suites: [{ suite, router }] }`,
  upserts in chunks (25 rows per INSERT to fit SQLite's variable cap),
  shares auth with the results endpoint via a new `_auth.ts` helper.

Workflow:
- The build job now POSTs the classification map directly to the new
  endpoint after running the classifier. Same guardrails as the results
  POST (only full-suite, only against main).
- The report job no longer needs to download / merge the classification
  artifact — it's already in D1 by the time results land.
- The classification JSON is still uploaded as a workflow artifact for
  manual re-submission / debugging.

Schema:
- New table `compat_suite_meta(suite PK, router, next_ref, classified_at)`
  with an index on `router` for the per-router count queries.
- `compat_file_results` reverts to its pre-PR columns. No data migration
  needed because the column was only ever populated on this branch.

Trade-off: classification changes are now retroactive — re-classifying a
suite updates how it appears in every historical run. This is usually
what you want (corrections heal the whole history) but means the trend
chart isn't a strict point-in-time record. The provenance fields on the
meta row let you tie a reclassification back to a specific Next.js ref.

Verified:
- vp check clean
- 14 classifier unit tests still pass
- apps/web build:vinext succeeds; /api/compatibility/classify shows up
  in the route list

* refactor(compat): drop redundant next_ref from compat_suite_meta

The Next.js ref a classification was produced against is already
recoverable from the run history — compat_runs records next_ref per
run, and the most recent classification's ref is implicit (it's the
ref of the most recent classify POST, which the workflow always pairs
with a run).

The future-proofing case for per-ref classifications would want a
composite (suite, next_ref) PK rather than a single global row anyway,
so this column doesn't help with that scenario either. classified_at
is enough for the debug case ('when was this last computed?').

Changes:
- Drop next_ref column from compat_suite_meta (3 columns now:
  suite PK, router, classified_at)
- /api/compatibility/classify no longer requires nextRef in the body
- Workflow no longer passes NEXT_REF when building the classify payload
- Migration regenerated as 0001_romantic_skullbuster.sql (one fewer
  column on the CREATE TABLE)
- Chunk size bumped to 33 rows/INSERT (100-var cap / 3 columns)

Verified: vp check clean, classifier tests pass, apps/web build succeeds.

* feat(compat-ui): share router filter between grid and trend chart

Lifts the router-filter state out of ContributionGrid into a new
CompatibilityViews client wrapper that owns the Kumo segmented Tabs
control. The grid and the line chart both consume the active filter
as a prop, so changing the tab updates both visualisations in lockstep.

The line chart now plots per-router series. The trend query was
rewritten to aggregate via JOIN against compat_suite_meta — one row
per run with app/pages/both/unknown rollups in a single round-trip,
~90 runs × ~1000 file rows over an indexed join. The TrendPoint
carries all five series; the chart picks one based on the filter
without re-fetching.

Other UI tweaks in this commit:
- Replaced the hand-rolled pill row with Kumo Tabs (variant='segmented',
  size='sm'). Fixes a weird active state (bg-kumo-default + text-kumo-base
  was using the text color for backgrounds, producing a saturated
  inversion).
- 'Parity' → 'Mixed' in user-visible labels. Internal identifiers
  (DB enum 'both', bucket variable 'parity') unchanged.
- Removed the standalone 'Other' stat card; the segmented Tabs still
  expose the 'Other' filter so unclassified suites are reachable.
- Reorganised the page into one 'Test files and trend' section
  containing both visualisations, gated by the shared Tabs.
- .gitignore: ignore .dev.vars (Wrangler's local-secrets convention)
  so the COMPAT_INGEST_SECRET we use for local testing never gets
  committed by accident.

Verified:
- vp check clean (format + type + lint)
- 14 classifier unit tests pass
- apps/web build:vinext succeeds; both API endpoints + page render

* fix(compat): address PR review comments

Addresses actionable items from the two /bigbonk review passes:

1. scanFixture: skip recursion into already-checked app/ and pages/
   (with a wrinkle around fixture wrappers).

   The previous code unconditionally pushed every child onto the walk
   stack, which (a) wasted work re-walking subtrees the route checker
   already covered and (b) risked a false positive on App Router route
   groups literally named 'pages' (e.g. app/pages/index.tsx would have
   tripped the Pages Router detector).

   Naive fix (skip descent into any app/ or pages/) regressed ~10 real
   Next.js fixtures that use a wrapping directory literally named 'app'
   as the test app's project root, with the real app/ and pages/ nested
   inside (test/e2e/og-api, test/e2e/middleware-static-files, etc.).

   Final fix: a directory named app or pages is treated as a fixture
   wrapper (and recursed into) if it contains a top-level next.config.*
   OR an inner app/ alongside an inner pages/. Otherwise it's handed to
   the route checker, which either finds real routes or returns nothing
   and the walk skips it. The wrapper detector intentionally does NOT
   treat middleware.{js,ts} as a wrapper signal, because Next.js tests
   put noop middleware.js files inside real App Router app/ to assert
   the file is ignored at that level (test/e2e/app-dir/app-middleware).

   Verified against the full test/e2e/ tree of the local Next.js
   checkout: 1036 suites classify identically to the pre-fix output
   (576 app / 325 pages / 113 both / 22 unknown), with the
   app/pages/index.tsx false-positive now correctly handled.

2. list-nextjs-e2e-suites.mjs: add the same import.meta.url guard the
   classifier already has, so importing the module programmatically
   doesn't immediately parse process.argv and write a file.

3. _auth.ts: early-exit when the X-Compat-Secret header is missing
   or empty, before paying two SHA-256 digests. The constant-time
   guarantee we care about (don't leak the contents of the expected
   secret via length or prefix matching) is preserved — we only
   short-circuit on values we already know can't match a non-empty
   secret.

4. Migration 0001_romantic_skullbuster.sql: add trailing newline.

Tests:
- 3 new regression tests added (18 total, all pass):
  * fixture wrapper with next.config.js + inner app/ + pages/
  * fixture wrapper with next.config.js + inner pages/ only
  * App Router app/ containing a route group named 'pages'
  * App Router app/ containing a noop middleware.js
- vp check clean
- Classifier output diff against current Next.js HEAD: empty

* fix(compat-ui): unexport cellMatchesFilter (knip)

CI's knip step flagged `cellMatchesFilter` as an unused export. It's only
used inside contribution-grid.tsx itself — the shared wrapper has its
own bucketing logic in compatibility-views.tsx. Drop the `export`.

* fix(compat): address third-pass review comments

1. sqlExcluded: narrow parameter type to a closed union
   ("router" | "classified_at") so the no-user-input invariant for
   sql.raw is compiler-enforced rather than relying on a code comment.

2. ContributionGrid: gate the SVG render on visibleCells.length > 0,
   and also clamp svgWidth/svgHeight to >= 0. Previously, when a
   filter emptied the grid, the SVG rendered with width=-3/height=-3
   alongside the placeholder div (browsers clamped silently but it
   was invalid SVG).

3. compatibility/page.tsx 'How this works' card: add a short note
   that per-router trend lines use the latest classification, so
   reclassifying a suite updates how it appears in historical runs.
   The aggregate "All" line is unaffected. Avoids a confusing
   support question if users notice old per-router numbers shift
   after a classifier improvement.

4. scanFixture readability: simplify the early-return checks from
   'if (hasApp && hasPages)' to 'if (hasPages)' / 'if (hasApp)',
   since the other flag was just set on the line above. No
   behavioural change; classifier output against the full Next.js
   test/e2e tree is byte-identical (1036 suites, 576/325/113/22).

5. /api/compatibility/classify body validator: bounds-check
   classifiedAt. Reject NaN/Infinity and any timestamp earlier
   than 2023-01-01 UTC. Catches the common seconds-vs-milliseconds
   mistake and accidental 0/-1 values from buggy callers. The
   only legitimate caller (the GH workflow) sends Date.now() so
   this is purely defensive.

Verified:
- vp check clean (format + type + lint + knip)
- 18 classifier unit tests pass
- apps/web build:vinext succeeds
- Classifier output against current Next.js HEAD is unchanged

* refactor(compat-ui): extract shared router bucketing module

Addresses items 1, 3, and 4 from the fourth review pass.

1. Trend GROUP BY note (item 1): added an inline comment explaining
   that selecting r.created_at while grouping by r.id only is a SQLite
   functional-dependency affordance. Standard SQL (postgres etc.)
   would reject it; the comment heads off a future 'fix' that breaks
   the query if it's ever ported.

2. Unified router bucketing (item 3): the three places that sliced
   cells by router filter all used slightly different naming
   conventions ('parity' vs 'both', 'other' vs 'unknown'). Extracted
   the canonical logic to a new ./router-buckets module with one
   shared vocabulary (the same as RouterKind), and updated all three
   call sites:

     - contribution-grid.tsx: cellMatchesFilter from shared module
     - compatibility-views.tsx: countByFilter for tab labels
     - page.tsx: bucketByRouter + bucketPassRate for stat cards,
       byRouter.parity/other renamed to byRouter.both/unknown

   Doc comment at the top of router-buckets.ts pins the
   'Mixed counts in both app and pages' rule in one place. No
   behavioural change.

3. passRate function rename (item 4): renamed the line chart's local
   ratio helper from passRate -> computePassRateRatio to avoid
   shadowing the passRate variable in page.tsx (and to distinguish
   it from bucketPassRate in router-buckets which returns a
   percentage, not a ratio).

Not addressed (intentionally):
- 'Mixed (both)' vs 'Parity'/'Interop' label (item 2): the user
  explicitly chose 'Mixed' over 'Parity' in an earlier message.
- trendRowsDesc rename (item 5): reviewer self-tagged 'very minor';
  renaming would muddle the diff.
- deriveSuiteGroup inconsistent levels (item 5b): pre-existing
  tooltip-only behaviour, not introduced by this PR.

Verified:
- vp check clean (format + type + lint + knip)
- 18 classifier unit tests pass
- apps/web build:vinext succeeds
2026-05-20 13:52:11 +01:00

237 lines
11 KiB
TypeScript

/**
* Tests for scripts/classify-nextjs-suites.mjs.
*
* The classifier reads from a Next.js checkout on disk, so each test creates
* a tiny fake Next.js checkout under a tmpdir, populates it with a fixture
* directory structure that exercises a specific bucket (see strategy notes
* in the script header), and asserts the right router kind is returned.
*
* Buckets covered:
* - App Router only (real app/, no pages/)
* - Pages Router only (real pages/, no app/)
* - Both (genuine parity test fixture)
* - App Router test with stub pages/ that has no real routes (still "app")
* - Pages Router test with stub app/ that has no real routes (still "pages")
* - app-dir/ inline fixture (no on-disk routes, falls back to "app")
* - Truly unclassifiable (config/build test, returns "unknown")
* - pageExtensions (page.page.js / layout.page.js still recognised)
* - test/ subdir (fixture lives one level up from the .test.ts)
* - APP_ROUTER_NON_APP_DIR_SUITES curated override
*/
import fsp from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { describe, it, expect, beforeEach, afterEach } from "vite-plus/test";
import { classifySuite } from "../scripts/classify-nextjs-suites.mjs";
async function writeFile(file: string, contents = "") {
await fsp.mkdir(path.dirname(file), { recursive: true });
await fsp.writeFile(file, contents);
}
describe("classifySuite", () => {
let root: string;
beforeEach(async () => {
root = await fsp.mkdtemp(path.join(os.tmpdir(), "classify-nextjs-"));
});
afterEach(async () => {
await fsp.rm(root, { recursive: true, force: true });
});
it("classifies an App Router-only fixture as 'app'", async () => {
const suite = "test/e2e/my-app-test/my-app-test.test.ts";
await writeFile(path.join(root, suite));
await writeFile(path.join(root, "test/e2e/my-app-test/app/page.tsx"));
await writeFile(path.join(root, "test/e2e/my-app-test/app/layout.tsx"));
expect(classifySuite(root, suite)).toBe("app");
});
it("classifies a Pages Router-only fixture as 'pages'", async () => {
const suite = "test/e2e/my-pages-test/my-pages-test.test.ts";
await writeFile(path.join(root, suite));
await writeFile(path.join(root, "test/e2e/my-pages-test/pages/index.tsx"));
await writeFile(path.join(root, "test/e2e/my-pages-test/pages/blog/[slug].tsx"));
expect(classifySuite(root, suite)).toBe("pages");
});
it("classifies a fixture with both real app/ and pages/ as 'both'", async () => {
const suite = "test/e2e/parity-test/parity-test.test.ts";
await writeFile(path.join(root, suite));
await writeFile(path.join(root, "test/e2e/parity-test/app/page.tsx"));
await writeFile(path.join(root, "test/e2e/parity-test/pages/old.tsx"));
expect(classifySuite(root, suite)).toBe("both");
});
it("ignores a stub pages/ with only Pages-Router specials (_app, _document)", async () => {
const suite = "test/e2e/stub-pages/stub-pages.test.ts";
await writeFile(path.join(root, suite));
await writeFile(path.join(root, "test/e2e/stub-pages/app/page.tsx"));
await writeFile(path.join(root, "test/e2e/stub-pages/pages/_app.tsx"));
await writeFile(path.join(root, "test/e2e/stub-pages/pages/_document.tsx"));
expect(classifySuite(root, suite)).toBe("app");
});
it("ignores a stub app/ with no real route files", async () => {
const suite = "test/e2e/stub-app/stub-app.test.ts";
await writeFile(path.join(root, suite));
await writeFile(path.join(root, "test/e2e/stub-app/pages/index.tsx"));
// app/ exists but has no page/layout/route/default — just a helper.
await writeFile(path.join(root, "test/e2e/stub-app/app/helper.ts"));
expect(classifySuite(root, suite)).toBe("pages");
});
it("falls back to 'app' for inline-fixture suites under test/e2e/app-dir/", async () => {
// The .test.ts builds its fixture via nextTestSetup({ files: { ... } })
// — no on-disk app/ or pages/ exists. Path convention says App Router.
const suite = "test/e2e/app-dir/inline-fixture/inline-fixture.test.ts";
await writeFile(path.join(root, suite));
expect(classifySuite(root, suite)).toBe("app");
});
it("returns 'unknown' for inline-fixture suites outside app-dir/", async () => {
// No path convention to fall back on, no on-disk fixture.
const suite = "test/e2e/config-test/config-test.test.ts";
await writeFile(path.join(root, suite));
expect(classifySuite(root, suite)).toBe("unknown");
});
it("recognises pageExtensions-style file names (layout.page.tsx, page.page.js)", async () => {
const suite = "test/e2e/app-dir/page-ext/index.test.ts";
await writeFile(path.join(root, suite));
await writeFile(path.join(root, "test/e2e/app-dir/page-ext/app/layout.page.tsx"));
await writeFile(path.join(root, "test/e2e/app-dir/page-ext/app/foo/page.page.js"));
expect(classifySuite(root, suite)).toBe("app");
});
it("walks up from `test/` subdir when the .test.ts lives in a test/ folder", async () => {
// Mirrors Next.js's pattern:
// test/e2e/middleware-base-path/test/index.test.ts
// test/e2e/middleware-base-path/app/...
const suite = "test/e2e/with-test-subdir/test/index.test.ts";
await writeFile(path.join(root, suite));
await writeFile(path.join(root, "test/e2e/with-test-subdir/app/page.tsx"));
expect(classifySuite(root, suite)).toBe("app");
});
it("honours the curated APP_ROUTER_NON_APP_DIR_SUITES list", async () => {
// This suite is hand-marked as App Router even though it doesn't live
// under test/e2e/app-dir/ and has no on-disk fixture.
const suite = "test/e2e/next-form/default/next-form-prefetch.test.ts";
// Don't write the file — verify the fallback fires when statSync fails.
expect(classifySuite(root, suite)).toBe("app");
});
it("respects an explicit overrides map", async () => {
const suite = "test/e2e/some/file.test.ts";
await writeFile(path.join(root, suite));
await writeFile(path.join(root, "test/e2e/some/pages/index.tsx"));
// Heuristic would say "pages"; overrides force "both".
expect(classifySuite(root, suite, { [suite]: "both" })).toBe("both");
});
it("classifies pages/api routes as real Pages Router routes", async () => {
// pages/api/* are real routes — fixtures that only have API routes
// are still Pages Router fixtures.
const suite = "test/e2e/api-only/index.test.ts";
await writeFile(path.join(root, suite));
await writeFile(path.join(root, "test/e2e/api-only/pages/api/hello.ts"));
expect(classifySuite(root, suite)).toBe("pages");
});
it("returns 'unknown' when the test file does not exist", async () => {
// Missing test file with no app-dir/ prefix → unknown.
expect(classifySuite(root, "test/e2e/missing/missing.test.ts")).toBe("unknown");
});
it("treats a fixture-wrapper 'app/' (contains next.config.js) as a project root, not App Router", async () => {
// Mirrors test/e2e/og-api/ in Next.js — the outer directory named
// `app` holds the Next.js project (next.config.js + pages/) and the
// inner `app/` is the real App Router root. Classification must walk
// through the wrapper to find both routers inside.
const suite = "test/e2e/wrapper-fixture/index.test.ts";
await writeFile(path.join(root, suite));
// Outer `app/` is the wrapper — next.config.js identifies it
await writeFile(path.join(root, "test/e2e/wrapper-fixture/app/next.config.js"));
// Real App Router inside
await writeFile(path.join(root, "test/e2e/wrapper-fixture/app/app/og/route.js"));
// Real Pages Router inside
await writeFile(path.join(root, "test/e2e/wrapper-fixture/app/pages/index.js"));
expect(classifySuite(root, suite)).toBe("both");
});
it("treats a fixture-wrapper 'app/' containing only pages/ as Pages Router", async () => {
// Mirrors test/e2e/browserslist/ in Next.js — outer `app/` is the
// wrapper, inner `pages/` is the real Pages Router fixture.
const suite = "test/e2e/wrapper-pages/index.test.ts";
await writeFile(path.join(root, suite));
await writeFile(path.join(root, "test/e2e/wrapper-pages/app/next.config.js"));
await writeFile(path.join(root, "test/e2e/wrapper-pages/app/pages/index.js"));
expect(classifySuite(root, suite)).toBe("pages");
});
it("does not treat an App Router route group named 'pages' as Pages Router", async () => {
// Regression test for a classifier bug where scanFixture recursed
// into `app/` looking for nested `app`/`pages` directories. An App
// Router fixture that uses a route group literally named `pages`
// (i.e. app/pages/...) would have incorrectly tripped the pages
// detector and been classified "both".
const suite = "test/e2e/with-app-route-group/with-app-route-group.test.ts";
await writeFile(path.join(root, suite));
await writeFile(path.join(root, "test/e2e/with-app-route-group/app/page.tsx"));
// A route group named "pages" — this is a directory inside app/,
// NOT a Pages Router directory, so classification must stay "app".
await writeFile(path.join(root, "test/e2e/with-app-route-group/app/pages/inner/page.tsx"));
expect(classifySuite(root, suite)).toBe("app");
});
it("does not mistake an App Router app/ that contains a noop middleware.js for a wrapper", async () => {
// Mirrors test/e2e/app-dir/app-middleware/ — the App Router app/
// contains a noop middleware.js file (used to assert that Next.js
// doesn't pick it up as middleware). The wrapper detector must NOT
// treat that as a project-root signal, because the directory is a
// real App Router app/, not a wrapper.
const suite = "test/e2e/app-dir/app-with-noop-middleware/app-with-noop-middleware.test.ts";
await writeFile(path.join(root, suite));
await writeFile(path.join(root, "test/e2e/app-dir/app-with-noop-middleware/app/layout.js"));
await writeFile(
path.join(root, "test/e2e/app-dir/app-with-noop-middleware/app/headers/page.js"),
);
// The noop middleware file inside app/ — must not flip wrapper detection
await writeFile(path.join(root, "test/e2e/app-dir/app-with-noop-middleware/app/middleware.js"));
// Sibling pages/ with a real route
await writeFile(path.join(root, "test/e2e/app-dir/app-with-noop-middleware/pages/[slug].js"));
expect(classifySuite(root, suite)).toBe("both");
});
it("does not walk into node_modules / .next when scanning fixtures", async () => {
const suite = "test/e2e/with-noise/with-noise.test.ts";
await writeFile(path.join(root, suite));
await writeFile(path.join(root, "test/e2e/with-noise/pages/index.tsx"));
// A node_modules dir contains a package with its own app/page.tsx —
// this must NOT cause the suite to be classified as "both".
await writeFile(path.join(root, "test/e2e/with-noise/node_modules/some-pkg/app/page.tsx"));
await writeFile(path.join(root, "test/e2e/with-noise/.next/server/app/page.js"));
expect(classifySuite(root, suite)).toBe("pages");
});
});