James Anderson f27c40a77f chore: split e2e compatibility by App Router vs Pages Router (#1321) ...

* feat(compat): split e2e compatibility by App Router vs Pages Router

The /compatibility page previously showed one undifferentiated grid of
~1000 Next.js test files. This change classifies each test by which
router(s) its fixture exercises (App, Pages, both, or unknown) and
surfaces the breakdown in the UI.

How:

- A new `router` column on `compat_file_results` (enum: app | pages |
  both | unknown). Defaults to 'unknown' so pre-classifier rows still
  render — they just show up under 'Other'.
- A new `scripts/classify-nextjs-suites.mjs` walks each test's fixture
  directory and looks for app/page.tsx, app/route.ts, app/layout.tsx,
  pages/*.tsx (excluding _app / _document / _error specials). A fixture
  with real routes in both folders is classified as 'both' — these are
  the genuine parity tests and counting them only once would hide a
  router-specific failure.
- Edge cases handled in the classifier:
    * pageExtensions naming (layout.page.tsx → layout)
    * Suites whose .test.ts lives in a test/ subdir alongside the fixture
    * Inline-fixture suites under test/e2e/app-dir/ (no on-disk routes,
      but path convention says App Router)
    * APP_ROUTER_NON_APP_DIR_SUITES curated override
    * Skips node_modules and .next when scanning fixtures
- The nightly workflow now runs the classifier in the build job once
  per run (the cheap part — happens with Next.js still on disk),
  uploads the suite → router map as an artifact, and the report job
  joins it into the ingest payload before POSTing.
- The compatibility page gains a 'By router' card row showing per-router
  pass rates and file counts (parity tests are counted toward both
  router buckets — adding them exceeds the total, see the explainer
  card). The contribution grid grows filter chips above it for
  interactive narrowing.

Verified:

- vp check (format + type + lint) clean
- 14 new classifier unit tests pass (vp test run tests/classify-nextjs-suites.test.ts)
- apps/web build:vinext succeeds end-to-end
- Local classifier run against .nextjs-ref reports
  app=576 / pages=325 / both=113 / unknown=22 out of 1036 suites, with
  spot-checked classifications matching expectations.

Backward compatibility: the ingest endpoint treats `router` as optional;
existing workflow runs and historical rows continue to function and
render as 'Other' until the next nightly classifies them.

* refactor(compat): store router classification in its own table

Replaces the inline `router` column on `compat_file_results` (introduced
in the previous commit) with a dedicated `compat_suite_meta` table,
classification keyed by `suite`. The /compatibility UI LEFT JOINs the
two tables at query time.

Rationale (per design discussion in PR #1321):

- Classifications conceptually describe test files, not test runs. Storing
  them per-row coupled their cadence to results ingestion, which is wrong
  when (a) the Next.js ref bumps and re-classifies everything, (b) an
  override fix lands without re-running tests, or (c) a partial test run
  still wants fresh classifications.
- One row per suite (PK on `suite`) means re-classifying is an upsert,
  not a backfill loop. Provenance (`next_ref`, `classified_at`) is
  stored on the row for debugging.
- Decoupling lets the workflow POST classifications from the build job
  immediately after running the classifier, without round-tripping through
  the report job. Results ingestion stays focused on results.

Endpoints:
- `POST /api/compatibility` no longer accepts `router` per file
  (reverted to its pre-PR shape).
- `POST /api/compatibility/classify` (new): accepts
  `{ nextRef, classifiedAt?, suites: [{ suite, router }] }`,
  upserts in chunks (25 rows per INSERT to fit SQLite's variable cap),
  shares auth with the results endpoint via a new `_auth.ts` helper.

Workflow:
- The build job now POSTs the classification map directly to the new
  endpoint after running the classifier. Same guardrails as the results
  POST (only full-suite, only against main).
- The report job no longer needs to download / merge the classification
  artifact — it's already in D1 by the time results land.
- The classification JSON is still uploaded as a workflow artifact for
  manual re-submission / debugging.

Schema:
- New table `compat_suite_meta(suite PK, router, next_ref, classified_at)`
  with an index on `router` for the per-router count queries.
- `compat_file_results` reverts to its pre-PR columns. No data migration
  needed because the column was only ever populated on this branch.

Trade-off: classification changes are now retroactive — re-classifying a
suite updates how it appears in every historical run. This is usually
what you want (corrections heal the whole history) but means the trend
chart isn't a strict point-in-time record. The provenance fields on the
meta row let you tie a reclassification back to a specific Next.js ref.

Verified:
- vp check clean
- 14 classifier unit tests still pass
- apps/web build:vinext succeeds; /api/compatibility/classify shows up
  in the route list

* refactor(compat): drop redundant next_ref from compat_suite_meta

The Next.js ref a classification was produced against is already
recoverable from the run history — compat_runs records next_ref per
run, and the most recent classification's ref is implicit (it's the
ref of the most recent classify POST, which the workflow always pairs
with a run).

The future-proofing case for per-ref classifications would want a
composite (suite, next_ref) PK rather than a single global row anyway,
so this column doesn't help with that scenario either. classified_at
is enough for the debug case ('when was this last computed?').

Changes:
- Drop next_ref column from compat_suite_meta (3 columns now:
  suite PK, router, classified_at)
- /api/compatibility/classify no longer requires nextRef in the body
- Workflow no longer passes NEXT_REF when building the classify payload
- Migration regenerated as 0001_romantic_skullbuster.sql (one fewer
  column on the CREATE TABLE)
- Chunk size bumped to 33 rows/INSERT (100-var cap / 3 columns)

Verified: vp check clean, classifier tests pass, apps/web build succeeds.

* feat(compat-ui): share router filter between grid and trend chart

Lifts the router-filter state out of ContributionGrid into a new
CompatibilityViews client wrapper that owns the Kumo segmented Tabs
control. The grid and the line chart both consume the active filter
as a prop, so changing the tab updates both visualisations in lockstep.

The line chart now plots per-router series. The trend query was
rewritten to aggregate via JOIN against compat_suite_meta — one row
per run with app/pages/both/unknown rollups in a single round-trip,
~90 runs × ~1000 file rows over an indexed join. The TrendPoint
carries all five series; the chart picks one based on the filter
without re-fetching.

Other UI tweaks in this commit:
- Replaced the hand-rolled pill row with Kumo Tabs (variant='segmented',
  size='sm'). Fixes a weird active state (bg-kumo-default + text-kumo-base
  was using the text color for backgrounds, producing a saturated
  inversion).
- 'Parity' → 'Mixed' in user-visible labels. Internal identifiers
  (DB enum 'both', bucket variable 'parity') unchanged.
- Removed the standalone 'Other' stat card; the segmented Tabs still
  expose the 'Other' filter so unclassified suites are reachable.
- Reorganised the page into one 'Test files and trend' section
  containing both visualisations, gated by the shared Tabs.
- .gitignore: ignore .dev.vars (Wrangler's local-secrets convention)
  so the COMPAT_INGEST_SECRET we use for local testing never gets
  committed by accident.

Verified:
- vp check clean (format + type + lint)
- 14 classifier unit tests pass
- apps/web build:vinext succeeds; both API endpoints + page render

* fix(compat): address PR review comments

Addresses actionable items from the two /bigbonk review passes:

1. scanFixture: skip recursion into already-checked app/ and pages/
   (with a wrinkle around fixture wrappers).

   The previous code unconditionally pushed every child onto the walk
   stack, which (a) wasted work re-walking subtrees the route checker
   already covered and (b) risked a false positive on App Router route
   groups literally named 'pages' (e.g. app/pages/index.tsx would have
   tripped the Pages Router detector).

   Naive fix (skip descent into any app/ or pages/) regressed ~10 real
   Next.js fixtures that use a wrapping directory literally named 'app'
   as the test app's project root, with the real app/ and pages/ nested
   inside (test/e2e/og-api, test/e2e/middleware-static-files, etc.).

   Final fix: a directory named app or pages is treated as a fixture
   wrapper (and recursed into) if it contains a top-level next.config.*
   OR an inner app/ alongside an inner pages/. Otherwise it's handed to
   the route checker, which either finds real routes or returns nothing
   and the walk skips it. The wrapper detector intentionally does NOT
   treat middleware.{js,ts} as a wrapper signal, because Next.js tests
   put noop middleware.js files inside real App Router app/ to assert
   the file is ignored at that level (test/e2e/app-dir/app-middleware).

   Verified against the full test/e2e/ tree of the local Next.js
   checkout: 1036 suites classify identically to the pre-fix output
   (576 app / 325 pages / 113 both / 22 unknown), with the
   app/pages/index.tsx false-positive now correctly handled.

2. list-nextjs-e2e-suites.mjs: add the same import.meta.url guard the
   classifier already has, so importing the module programmatically
   doesn't immediately parse process.argv and write a file.

3. _auth.ts: early-exit when the X-Compat-Secret header is missing
   or empty, before paying two SHA-256 digests. The constant-time
   guarantee we care about (don't leak the contents of the expected
   secret via length or prefix matching) is preserved — we only
   short-circuit on values we already know can't match a non-empty
   secret.

4. Migration 0001_romantic_skullbuster.sql: add trailing newline.

Tests:
- 3 new regression tests added (18 total, all pass):
  * fixture wrapper with next.config.js + inner app/ + pages/
  * fixture wrapper with next.config.js + inner pages/ only
  * App Router app/ containing a route group named 'pages'
  * App Router app/ containing a noop middleware.js
- vp check clean
- Classifier output diff against current Next.js HEAD: empty

* fix(compat-ui): unexport cellMatchesFilter (knip)

CI's knip step flagged `cellMatchesFilter` as an unused export. It's only
used inside contribution-grid.tsx itself — the shared wrapper has its
own bucketing logic in compatibility-views.tsx. Drop the `export`.

* fix(compat): address third-pass review comments

1. sqlExcluded: narrow parameter type to a closed union
   ("router" | "classified_at") so the no-user-input invariant for
   sql.raw is compiler-enforced rather than relying on a code comment.

2. ContributionGrid: gate the SVG render on visibleCells.length > 0,
   and also clamp svgWidth/svgHeight to >= 0. Previously, when a
   filter emptied the grid, the SVG rendered with width=-3/height=-3
   alongside the placeholder div (browsers clamped silently but it
   was invalid SVG).

3. compatibility/page.tsx 'How this works' card: add a short note
   that per-router trend lines use the latest classification, so
   reclassifying a suite updates how it appears in historical runs.
   The aggregate "All" line is unaffected. Avoids a confusing
   support question if users notice old per-router numbers shift
   after a classifier improvement.

4. scanFixture readability: simplify the early-return checks from
   'if (hasApp && hasPages)' to 'if (hasPages)' / 'if (hasApp)',
   since the other flag was just set on the line above. No
   behavioural change; classifier output against the full Next.js
   test/e2e tree is byte-identical (1036 suites, 576/325/113/22).

5. /api/compatibility/classify body validator: bounds-check
   classifiedAt. Reject NaN/Infinity and any timestamp earlier
   than 2023-01-01 UTC. Catches the common seconds-vs-milliseconds
   mistake and accidental 0/-1 values from buggy callers. The
   only legitimate caller (the GH workflow) sends Date.now() so
   this is purely defensive.

Verified:
- vp check clean (format + type + lint + knip)
- 18 classifier unit tests pass
- apps/web build:vinext succeeds
- Classifier output against current Next.js HEAD is unchanged

* refactor(compat-ui): extract shared router bucketing module

Addresses items 1, 3, and 4 from the fourth review pass.

1. Trend GROUP BY note (item 1): added an inline comment explaining
   that selecting r.created_at while grouping by r.id only is a SQLite
   functional-dependency affordance. Standard SQL (postgres etc.)
   would reject it; the comment heads off a future 'fix' that breaks
   the query if it's ever ported.

2. Unified router bucketing (item 3): the three places that sliced
   cells by router filter all used slightly different naming
   conventions ('parity' vs 'both', 'other' vs 'unknown'). Extracted
   the canonical logic to a new ./router-buckets module with one
   shared vocabulary (the same as RouterKind), and updated all three
   call sites:

     - contribution-grid.tsx: cellMatchesFilter from shared module
     - compatibility-views.tsx: countByFilter for tab labels
     - page.tsx: bucketByRouter + bucketPassRate for stat cards,
       byRouter.parity/other renamed to byRouter.both/unknown

   Doc comment at the top of router-buckets.ts pins the
   'Mixed counts in both app and pages' rule in one place. No
   behavioural change.

3. passRate function rename (item 4): renamed the line chart's local
   ratio helper from passRate -> computePassRateRatio to avoid
   shadowing the passRate variable in page.tsx (and to distinguish
   it from bucketPassRate in router-buckets which returns a
   percentage, not a ratio).

Not addressed (intentionally):
- 'Mixed (both)' vs 'Parity'/'Interop' label (item 2): the user
  explicitly chose 'Mixed' over 'Parity' in an earlier message.
- trendRowsDesc rename (item 5): reviewer self-tagged 'very minor';
  renaming would muddle the diff.
- deriveSuiteGroup inconsistent levels (item 5b): pre-existing
  tooltip-only behaviour, not introduced by this PR.

Verified:
- vp check clean (format + type + lint + knip)
- 18 classifier unit tests pass
- apps/web build:vinext succeeds

2026-05-20 13:52:11 +01:00

..

route.ts

chore: split e2e compatibility by App Router vs Pages Router (#1321)

2026-05-20 13:52:11 +01:00