microsoft__playwright/.github/workflows/fix-flakes-prompt.md

Playwright: Fix a Flaky or Red Test

Turn CI test-results data into one concrete fix: pick a high-impact flaky-or-red test that is reproducible on your OS, confirm nobody's on it, fix the root cause or scope a skip, pick a reviewer, and hand off a single commit that becomes the PR. Fully autonomous — no approval stops.

The GitHub CLI (gh) is not authenticated in this job. Do not use it for GitHub API operations; use GitHub MCP tools instead.

1. Pick one target

Query the DB following the patterns in .claude/skills/playwright-test-results/SKILL.md. Two families:

• Cross-run flake — the final verdict (after retries) flips between runs.
• Consistent red — expected_status = 'passed' yet failing in ~every run.

Rank by impact: fail %, run count (floor runs >= 10 so it isn't a one-off), and how many bots/PRs it disrupts — not by "has a tidy error message". Pick a candidate whose failing bot_name OS matches yours so you can reproduce it. Keep the ranked list — you fall back down it in step 2. Say why you picked the top one. Once you have the target, use the "Generate a linked emoji run history" recipe in .claude/skills/playwright-test-results/SKILL.md. Include the complete output in the PR body: never trim it or remove green runs, and check the square count matches the per-run row count.

2. Check nobody's on it — and that it isn't already fixed

Two dead ends to rule out before touching a candidate. If either fires, drop it and go down your step-1 ranking, re-checking each; only stop once the whole shortlist is covered.

• Already being worked on — search PRs/issues (open and recently merged/closed) for the test title words and file path, plus any issue linked in the test's annotation.
• Already fixed — the DB is a rolling window that still holds the failing runs from before a fix landed, so a test fixed mid-window looks maximally bimodal (old fails + new passes), which is exactly what floats it up step 1. Check whether a fix has landed since it last flaked: find the commit its most recent failing run ran on, and look at what's changed in that test's file since. Only move on if enough subsequent runs show the failures stopped; a browser roll alone is not evidence of a fix.

Compare the test across every bot that runs it, not only the failing bot. A clean boundary between stable, beta, and dev browser channels often points to a browser-version regression. Read the exact versions from the job logs and record the failing and passing versions; channel names alone do not identify the affected behaviour.

3. Reproduce on this OS

Read the test and its error_message.

Build first (npm run build; watch is not running — if you touch generated-code files see CLAUDE.md). Then reproduce, scoped to the failing target.

The DB project_name is the Playwright --project; bot_name encodes the OS. Always pass a <file>:<line> filter — a bare run launches the whole suite. The browser test scripts are project-locked, so use the one matching the failing bot:

Failing target	Command
chromium	npm run ctest -- <file>:<line>
firefox	npm run ftest -- <file>:<line>
webkit	npm run wtest -- <file>:<line>
tests/playwright-test/**	npm run ttest -- <file>:<line>
tests/mcp/**	npm run test-mcp -- --project=<name> <file>:<line>

Other suites (electron etest, etc.): see package.json scripts. Add --repeat-each=N to force a flake's flip.

You can only reproduce what this OS and architecture reaches. Match both from bot_name; non-reproduction under lighter local load or on another architecture is not itself a skip candidate. Keep any OS/architecture handling keyed on the current environment, never hardcoded. Broader OS coverage comes from different agent runs on other OSes, not from you.

4. Fix — root cause or scoped skip

• Tractable → fix the source. Usually a test-side race: a missing await, waiting on the wrong signal, an under-specified locator, leaked state. Fix the test (or the product bug).

• Timeout → check the budget before skipping. Compare passing durations and neighboring tests; if working tests share an expensive operation near the timeout, prefer a scoped it.slow().

• Engine/OS-specific, unreachable here, or genuinely hard → scope a skip, narrowed to the exact failing condition — never a whole file or browser:

  it.fixme(browserName === 'webkit' && isLinux, 'https://github.com/microsoft/playwright/issues/NNNNN');
  

  For browser-specific failures, prefer the narrowest observed version predicate over a channel predicate. Channels roll forward, so channel === 'chrome' can keep skipping the test after the browser fixes the bug. If stable fails while beta/dev passes, use browserVersion or browserMajorVersion and gate only the affected version:

  it.fixme(browserName === 'chromium' && browserMajorVersion === 150, 'https://github.com/microsoft/playwright/issues/NNNNN');
  

  Do not invent a wider threshold for untested versions. Keep a channel predicate only when the same browser version behaves differently between channels.

  Link an issue if one exists or explain why the test is skipped. Default to fixme — it parks the debt and stays greppable. Use skip only if reproduction shows the test is genuinely mis-scoped for that config (skip claims "this failure is expected and correct," which a flake-fighter rarely is). Never skip just to turn a red green.

Verify locally — your PR gets no CI (step 6), so this is the only proof: flake → re-run with --repeat-each and confirm it's stable; slow → confirm it still runs and passes; skip → confirm it's skipped on the target config and still runs elsewhere. Then run npm run flint. Record exactly what you ran — it goes in the PR body.

5. Pick a reviewer

No CODEOWNERS. Derive one from the touched file(s) — recent authors and frequent committers:

git log --format='%an %ae' -n 20 -- <file>
git log --format='%an' -n 200 -- <file> | sort | uniq -c | sort -rn

Also skim recent merged PRs on those paths for who reviewed them. Pick someone with real recency

• ownership. Record it as a git trailer so it survives the handoff:

Suggested-reviewer: dgozman

6. Commit and hand off (you never open the PR in CI)

You never push or open a PR in CI or locally. The harness does that for you, after you commit.

• Exactly one commit on the current branch (CLAUDE.md conventions; no co-author / "generated with" trailers; never amend).
• The commit message is the PR — the harness runs gh pr create --fill, mapping subject→title and body→description. Write both in the Playwright bot voice (.github/workflows/bot-voice.md) — verdict first, short, no slop. The body must carry: the DB evidence (fail %, runs, bots) + any related issue; the linked emoji run history for the selected test and bot; what you verified locally and on which OS.
• Nothing actionable? Make no commit — the harness then skips the PR.

Report what you committed, the reviewer, and why.

Guardrails

• One target, one commit. Don't batch.
• Scoped skips only — narrow to the failing condition, never a whole file or browser matrix.
• No unverified PRs — if you cannot validate a scoped mitigation, make no commit.