- The `pr-status` can be very slow if you have a lot of test failures on
a PR. If the user asks a targeted question about PR review comments,
just fetch that information, it's much faster.
- Context bloat: Rely on the agent to pull in the `pr-status-triage`
skill if they need it, don't bloat the AGENTS.md, except to mention the
skill.
### What?
Lower the GitHub Actions jobs page size used by `scripts/pr-status.js`
and keep walking pages until all jobs are fetched. Reuse the paginated
job fetch for flaky-test detection.
### Why?
The previous `per_page=100&page=1` request for run `26093602659` returns
HTTP 502 from GitHub because the query is too slow.
### How?
Add a shared `JOBS_PAGE_SIZE = 30`, update the pagination stop condition
to match it, and route flaky-test failed job lookup through
`getFailedJobs()`.
### Verification
- `node --check scripts/pr-status.js`
- `pnpm prettier --with-node-modules --ignore-path .prettierignore
--write scripts/pr-status.js`
- `npx eslint --config eslint.config.mjs --fix scripts/pr-status.js`
- `gh api
"repos/vercel/next.js/actions/runs/26093602659/jobs?per_page=30&page=<n>"
--jq ".jobs | length"` across pages 1-5 captured 137 jobs, matching
`total_count: 137`
- Confirmed the old `per_page=100&page=1` request still returns HTTP 502
- Not run: full Next.js test suite (scripts-only helper change)
<!-- NEXT_JS_LLM_PR -->
### What?
Converts every test under `test/integration/` to an isolated test
running through `nextTestSetup` (under `test/e2e/`, `test/production/`,
`test/development/`, or `test/unit/`), then deletes `test/integration/`
along with the legacy CI orchestration that was specific to it.
- `test/integration/` removed entirely (~327 test suites)
- New isolated suites added across the existing folders:
- `test/e2e/` — 175
- `test/production/` — 130
- `test/development/` — 43
- `test/unit/` — 1
- `.github/workflows/build_and_test.yml` and `run-tests.js` no longer
have any `integration` branches
- `nextTestSetup` gained a `baseUrl` option on `next.browser()` so a
small number of tests that drive their own proxy/static-export server
can keep using `next.browser(...)` instead of importing `next-webdriver`
directly
### Why?
`test/integration/` predated `nextTestSetup` and ran tests directly
against the source checkout via custom helpers (`launchApp`,
`nextBuild`, `nextStart`, `runNextCommand`, `webdriver`, `fetchViaHTTP`,
…). Each suite hand-rolled its own dev/start/build orchestration,
fixture mutation, and process management.
The isolated test model used by the rest of the repo gives each suite an
isolated working directory containing a packed `next.tgz` install, a
uniform `next.start()` / `next.build()` / `next.fetch()` /
`next.browser()` API, and the same lifecycle for dev, start, and deploy
modes — so a single set of assertions covers all three. Deploy-mode
skips and per-feature gates are expressed declaratively
(`skipDeployment`, `disableAutoSkewProtection`, `if (skipped) return`)
instead of branching on `process.env`.
Removing `test/integration/` lets us:
- Delete the bespoke orchestration code in the CI workflow and
`run-tests.js`
- Run every converted suite consistently in dev, start, and deploy modes
(where applicable)
- Reproduce every test locally with the same `pnpm
test-{dev,start}-{turbo,webpack}` commands; no separate `integration`
path
- Open the door to running `test/production` against deployments in the
future (the converted suites already declare `skipDeployment` so they
can be flipped on)
### How?
Mechanical conversion per suite, with targeted clean-ups:
1. **Per-suite conversion.** Each
`test/integration/<name>/test/index.test.{js,ts}` was rewritten into a
single `<name>.test.ts` under the right folder based on what the
original exercised:
- `launchApp` / dev-only assertions → `test/development/`
- `nextBuild` + `nextStart` / start-only assertions → `test/production/`
- Both → `test/e2e/`
- The one pure jsdom render check (`link-without-router`) → `test/unit/`
2. **API mapping.** Custom helpers were replaced by `nextTestSetup`
equivalents: `launchApp` → `next.start()`, `nextBuild` → `next.build()`,
`runNextCommand` → `next.runCommand`, `fetchViaHTTP` → `next.fetch`,
`webdriver(...)` → `next.browser(...)`. Fixture mutations switched from
raw `fs.writeFile`/`fs.rename` to `next.patchFile` (with the 3-arg
`runWithTempContent` callback when the change has a defined scope) and
`next.deleteFile`.
3. **Deploy-mode handling.** Suites that can't run in deploy mode (use
`patchFile` / `next.build()` / depend on local CLI output) declare
`skipDeployment: true` and early-return on the `skipped` boolean. Suites
where Vercel's edge mutates URLs (`&dpl=`, immutable assets) declare
`disableAutoSkewProtection: true`.
4. **`next.browser({ baseUrl })`.** A handful of tests
(`prerender-export`, `cdn-cache-busting`, `preload-viewport`, both
`react-virtualized` suites) need to drive a separate server (a
static-export server or an `http-proxy` instance) rather than the
Next.js process. Instead of importing `next-webdriver` directly, those
tests now pass `{ baseUrl: <port|url> }` to `next.browser()`. For the
proxy cases, the proxy was moved into `server.js` inside the fixture and
`http-proxy` declared via the `dependencies` option of `nextTestSetup`,
so the test runs with a fully isolated dependency graph.
5. **CI clean-up.** With `test/integration` gone, the `test
integration*` jobs and `integration-tests-manifest`-related logic in
`.github/workflows/build_and_test.yml` were removed, and `run-tests.js`
no longer has the `integration` test-folder branch.
6. **Validation.** The PR was iterated against multiple full CI runs;
the remaining failures on the latest run are pre-existing flakes
(segment-cache 60s `act` timeouts in turbopack-prod) or transient
infrastructure issues unrelated to the conversion.
### What?
Adds a `reply-and-resolve-thread` subcommand to `scripts/pr-status.js`
that combines replying and resolving a review thread in a single
command. Also updates the generated `thread-N.md` files to show this
command alongside the existing separate `reply-thread` and
`resolve-thread` options, and documents the command in the agent skill
files.
### Why?
When addressing PR review comments, the typical workflow is to reply
with a description of what was done and then immediately resolve the
thread. This previously required two separate commands:
```bash
node scripts/pr-status.js reply-thread <id> "Done -- ..."
node scripts/pr-status.js resolve-thread <id>
```
Having a single command reduces friction for agents (and humans) closing
out review threads after addressing feedback.
### How?
- Added `reply-and-resolve-thread` subcommand in `main()` that calls the
existing `replyToThread()` and `resolveThread()` functions in sequence.
- Updated `generateThreadMd()` to include a "Reply and resolve in one
step" code block in the Commands section of each unresolved thread file
(alongside the existing separate commands, not replacing them).
- Updated `.agents/skills/pr-status-triage/SKILL.md` to mention the
combined command in workflow step 6 and added a "Thread interaction"
quick-command reference section.
- Updated `.agents/skills/pr-status-triage/workflow.md` to show the
one-step alternative in the "Resolving Review Threads" section.
<!-- NEXT_JS_LLM_PR -->
---------
Co-authored-by: Tobias Koppers <sokra@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
### What?
Fixes two bugs in `scripts/pr-status.js` that caused the script to
silently report zero failures when there were actual CI failures.
### Why?
When running `node scripts/pr-status.js 92080`, the script reported "No
failed jobs found" despite the PR having real failures. Two root causes:
1. **Transient API errors silently swallowed.** `getAllJobs()` had a
bare `catch { break }` that returned an empty array on any GitHub API
error (e.g., HTTP 502). Since the GitHub Actions jobs API for large runs
frequently returns transient 502s, this caused false "no failures"
reports.
2. **`timed_out` and `startup_failure` jobs were invisible.** The script
only checked for `conclusion === 'failure'`, but GitHub uses distinct
conclusion values like `timed_out` (job exceeded timeout) and
`startup_failure` (runner failed to start). These jobs fell through all
filters and were silently omitted from reports.
### How?
**Retry logic in `getAllJobs()`:**
- Retries each paginated API call up to 3 times with 2s/4s backoff
- If all retries fail on the first page, throws an error (no silent
empty results)
- If later pages fail after partial data is collected, warns and returns
what was fetched
**Broader failure detection with `FAILED_CONCLUSIONS`:**
- Added a shared `FAILED_CONCLUSIONS` set: `{'failure', 'timed_out',
'startup_failure'}`
- Used in `getFailedJobs()`, `categorizeJobs()`, and flaky test
detection
- Jobs with non-`failure` conclusions are annotated in the report table
(e.g., "(timed_out)")
- `getFailedJobs()` now also returns the `conclusion` field so
downstream code can distinguish failure types
Co-authored-by: Tobias Koppers <sokra@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
`getFailedJobs()` in `scripts/pr-status.js` used jq's `select(.conclusion == "failure")` filter during GitHub API pagination. When a page had 100 jobs but none were failures, jq returned empty output, and the `if (!output.trim()) break` check terminated the pagination loop early — never fetching page 2+ where actual failures existed.
For example, on PR #90755 (129 total jobs), all 100 jobs on page 1 were success/skipped, while the 2 failures (`test unit windows (22) / build` and `thank you, next`) were on page 2 (jobs 101-129). The script reported "Found 0 failed jobs" when there were actually 2.
## Fix
`getFailedJobs()` now delegates to `getAllJobs()` (which fetches all jobs without jq filtering, so pagination works correctly) and filters for failures in JavaScript afterward. This is simpler and avoids the class of bug where jq pre-filtering interacts badly with pagination.
## Test Plan
- Ran `node scripts/pr-status.js 90755` before fix: reported 0 failed jobs
- Ran `node scripts/pr-status.js 90755` after fix: correctly reported 2 failed jobs (`test unit windows (22) / build` and `thank you, next`)
### What?
Prefixes replies posted by `scripts/pr-status.js` with the `🤖` (🤖) emoji.
### Why?
When the pr-status script replies to PR review threads, there's no visual indicator that the reply was generated by AI rather than a human. Adding the robot emoji makes it immediately clear.
### How?
One-line change in the `replyToThread()` function in `scripts/pr-status.js` that prepends `🤖 ` to the reply body before posting it via the GitHub GraphQL API.
## Summary
- Add a `--wait` flag to `scripts/pr-status.js` that writes a partial
report with currently available CI results, then blocks on `gh run
watch` until CI completes, and re-runs the full analysis to produce the
final report
- Refactor `main()` into a reusable `runAnalysis()` function so it can
be called twice (partial + final)
- Update the `/pr-status` command to run the script with `--wait` in the
background, present the partial report immediately while CI is still
running, then poll for completion with 5-minute timeouts
### How it works
1. `/pr-status` launches `node scripts/pr-status.js --wait` in the
background (1-minute Bash timeout)
2. The script writes the initial report (with whatever jobs have
completed so far) and prints `Output written to ...`
3. The command polls for that message using `TaskOutput` with 20-second
timeouts, then reads `index.md` and analyzes the partial results
4. Meanwhile, the script blocks on `gh run watch` until CI finishes,
then re-runs the full analysis
5. After presenting the partial analysis, the command polls for the
background script to complete using `TaskOutput` with 5-minute timeouts
(repeating if needed)
6. When it finishes, the command re-reads the final report and analyzes
any newly failed jobs
### Files changed
- `scripts/pr-status.js` — Extracted `runAnalysis()` from `main()`,
added `--wait` flag with `gh run watch` blocking and re-analysis
- `.claude/commands/pr-status.md` — Updated workflow to use `--wait`
with background execution, poll for readiness (20s timeouts), and poll
for final results (5-min timeouts)
## Test plan
- [ ] Run `node scripts/pr-status.js <completed-pr-number>` (no
`--wait`) — should behave identically to before
- [ ] Run `node scripts/pr-status.js <completed-pr-number> --wait` —
should produce report and exit immediately (no waiting needed since CI
is done)
- [ ] Run `node scripts/pr-status.js --wait` on a branch with
in-progress CI — should write partial report, block on `gh run watch`,
then write final report
- [ ] Run `/pr-status` command — should show partial results, then
update with final results after CI completes
---------
Co-authored-by: Claude <noreply@anthropic.com>
### What?
Adds two new subcommands to `scripts/pr-status.js`:
- `reply-thread <threadNodeId> <body>` — posts a reply to a PR review
thread
- `resolve-thread <threadNodeId>` — marks a review thread as resolved
Each generated `thread-N.md` file now includes a `## Commands` section
at the bottom with ready-to-use commands pre-populated with the correct
GraphQL node IDs. The resolve command is only shown for threads that are
still open.
Also updates the pr-status-triage skill to remind agents to reply to
review threads with a description of actions taken before resolving
them.
### Why?
When an agent addresses review feedback, it should close the loop by
replying to the thread (describing what was done) and resolving it.
Previously there was no way to do this from the pr-status workflow — the
agent would need to manually construct `gh api graphql` calls.
### How?
- Fetches GraphQL node IDs for review threads (added `id` to the
thread-level query)
- Uses `execFileSync` with argument arrays (not shell strings) to safely
pass the reply body to `gh api graphql` without shell escaping issues
- GraphQL mutations used: `addPullRequestReviewThreadReply` and
`resolveReviewThread`
- Subcommand dispatch at the top of `main()` checks `process.argv[2]`
against known subcommand names before falling through to the existing
PR-number behavior
## Summary
Fixes `spawnSync /bin/sh ENOBUFS` errors in `scripts/pr-status.js` when
fetching large CI job logs.
The script uses `execSync` (which internally uses `spawnSync`) to fetch
CI job logs via `gh api .../jobs/{id}/logs`. `execSync` has a hard
`maxBuffer` limit (set to 50MB), and when CI job logs exceed that size,
Node.js throws an `ENOBUFS` error. This causes the flaky test detection
to silently fail, reporting 0 flaky tests.
**Before:**
```
Fetching logs for 37 failed jobs...
[stderr] Command failed: gh api "repos/vercel/next.js/actions/jobs/65135029618/logs"
spawnSync /bin/sh ENOBUFS
[stderr] Command failed: gh api "repos/vercel/next.js/actions/jobs/65135029722/logs"
spawnSync /bin/sh ENOBUFS
Found 0 flaky tests (failing on 2+ different branches)
```
**After:**
```
Fetching logs for 18 failed jobs...
Found 0 flaky tests (failing on 2+ different branches)
```
## Changes
- Added an `execAsync()` helper that uses `child_process.spawn` instead
of `execSync`. `spawn` streams data through pipes with no built-in
buffer limit, avoiding `ENOBUFS` entirely.
- Updated `getJobLogs()` and the log-fetching loop in `getFlakyTests()`
to use `execAsync` instead of `exec`.
- All other API calls (small JSON responses) continue using the
synchronous `exec()` helper since they are well within the 50MB limit.
## Test Plan
Ran `node scripts/pr-status.js 90617` against a PR with multiple failed
jobs — completes successfully with no ENOBUFS errors.
## Summary
- **Agent skills**: 9 new skill files in `.agents/skills/` covering DCE,
flags, react-vendoring, runtime-debug, PR triage, and skill authoring
- **PR status tooling**: `scripts/pr-status.js` script and
`.claude/commands/pr-status.md` command
- **AGENTS.md**: updated with skill references and development
guidelines
## Test plan
- [ ] No runtime behavior changes
- [ ] Docs and tooling only
---------
Co-authored-by: Tim Neutkens <tim@timneutkens.nl>
Benjamin Woodruff