2 Commits

Author	SHA1	Message	Date
MK (fengmk2)	6fff7bb88f	chore: upgrade vite-plus to 0.2.1 (#2149) ... v0.2.1 consumes upstream Vitest directly: the @voidzero-dev/vite-plus-test wrapper is removed and vitest now comes in transitively through vite-plus. - catalog: vite-plus 0.1.24 -> 0.2.1, vite -> @voidzero-dev/vite-plus-core@0.2.1 - catalog: vitest -> upstream 4.1.9 (was the vite-plus-test alias), @vitest/coverage-istanbul 4.1.6 -> 4.1.9 - drop the vitest peerDependencyRules accommodations for the wrapper/old vitest - decode Vitest's flatted blob format directly in scripts/lib/vitest-blob-timings.mjs (the removed wrapper bundled the parser this script previously reached into)	2026-06-18 16:37:34 +01:00
Nathan Nguyen	df066cb486	ci: optimize CI sharding (#1768) ... * ci: add fifth integration shard * ci: weight integration shards from timing data * ci: shard app-router e2e * ci: rerun optimization experiment * ci: add sixth weighted integration shard * ci: rebalance weighted integration shards * ci: shard unit tests * ci: rebalance integration shards from current timings * ci: move weighted integration shard list from YAML into script + timing manifest * ci: move weighted integration shard list from YAML into script + timing manifest * feat(ci): derive integration shard weights from real CI timings with provenance Integration shard weights lived in a hand-seeded flat path->ms map ("aggregation": "manual seed"). A reviewer could not tell a measured number from a guess, and the guesses were wrong: favicon-short-circuit was seeded at 5s but runs ~35s in CI across five runs, a 7x under-weight that mis-packed the shards. The seed had no provenance and no way to regenerate from real data. Restructure the manifest to a v2 provenance model: per file estimateMs (the weight the planner uses), plus medianMs/p75Ms/samples and a generatedFrom.runs list, an estimator metric, and generatedAt. Add scripts/ci-integration-timings-refresh.mjs to aggregate Vitest blob reports downloaded from successful CI runs (p75 per file, nearest-rank) and rewrite the manifest deterministically, failing closed when the blobs do not cover every discovered file. The manifest here was regenerated from 5 successful runs (30 blobs); all six shards now pack to 84s. Extract planning and blob parsing into scripts/lib/* so the fragile Vite+ blob-parser probe lives in one place. Replace the O(files*shards) lightest-group scan with an O(n log m) binary min-heap and collapse the three duplicated local-search move/swap helpers into one makespanAfter + transfer primitive. Behavior preserved: the --check gate still verifies every file lands in exactly one shard. Harden --check to fail closed on no discovered files, missing, stale, malformed/zero/negative timings, shard-count drift, and bucket coverage. Add an advisory --recommend mode that models the optimal shard count from real weights and flags when integration has dropped below the competing cross-job bottleneck. It is advisory only and never runs in CI; the count stays declarative in manifest.shardTotal with the matrix enforced against it. * ci: pass integration shard file list via env to avoid template injection The integration shard step expanded ${{ steps.shard.outputs.files }} directly into the run: block. That output is a list of test file paths discovered from `vp test list`, and on pull_request runs a filename is attacker-controllable: a fork PR adding a file whose name contains shell metacharacters would inject it into the runner shell. GitHub code scanning (zizmor) flagged this as template-injection, alert 163. Route the file list and the other computed values through env vars and reference them in the script, leaving $SHARD_FILES unquoted so the shell still word-splits it into separate file arguments. The shell now treats the value as data, never as script text. Verified with zizmor: the pre-fix workflow reports template-injection on this line, the fixed workflow reports no findings. * feat(ci): require refresh blobs to back the claimed --run provenance The refresh tool recorded every --run id as provenance but only checked that each discovered file had at least one timing sample. Passing five --run ids with blobs for a single complete run still produced a manifest claiming five-run provenance while every file held one sample. The manifest could claim stronger provenance than the blob directory backs. A test file runs in exactly one shard per run, so one complete run yields exactly one sample per file. Require samples === runIds.length for every discovered file: too few means a claimed run's blobs are missing, too many means the directory holds blobs beyond the claimed runs. --allow-partial relaxes the check to "at least one sample per file" for the re-run-failed-shard case while still recording the true per-file sample count. * experiment: run integration at 5 shards to benchmark the latency/cost knee Temporary, for benchmarking only. Repacks the same provenance weights into 5 integration shards instead of 6 (manifest shardTotal and matrix set to 5, Check gate updated to match) so the 5 vs 6 trade-off can be measured with the same weights, unit split, and E2E split. To be reverted to 6 after the run is captured. * experiment: go aggressive on wall-clock (8 integration, 3 unit, 3 app-router E2E) Runner minutes are free on this public repo, so the objective is pure wall-clock. Attack the whole critical-path cluster at once: integration to 8 shards (~63s test load each, near the per-file floor), unit to 3, and the app-router E2E project to 3-way so none of them becomes the new ceiling once the others drop. Report job left as-is. Benchmarking only; final counts settle after the run lands. * ci: set integration to 10 shards, the wall-clock floor on free CI Public repo, so runner minutes are free and the objective is pure wall-clock. At 10 shards each integration shard carries ~51s of test load; combined with the serial report tail this brings the integration critical path down to roughly where the un-shardable create-next-app (windows) job sits, so additional shards stop moving the overall wall. Keeps unit at 3 shards and the app-router E2E project at 3-way from the prior step. Benchmarking continues; counts can still change. * fix(ci): default refresh shard count to the existing manifest, not a constant ci-integration-timings-refresh.mjs defaulted --shard-total to a hardcoded 6. The documented refresh command in ci.yml omits --shard-total, so once the matrix moved past 6 shards, following the advertised workflow rewrote shardTotal: 6 into the manifest and the next run failed the Verify integration shard manifest step with shard-count drift. Default to the current manifest's shardTotal instead. manifest.shardTotal is the single source of truth for the count: the matrix mirrors it and --check enforces no drift, so a plain refresh now preserves whatever the matrix uses. An explicit --shard-total still overrides it for an intentional count change, and a missing count with no existing manifest now fails with a clear message instead of silently picking a number. Found by Codex review on a31c99c4. * refactor(ci): share integration shard CLI helpers * refactor(ci): clarify shard local search * refactor(ci): drop doubled flag prefix in refresh shard-total error The invalid --shard-total message reconstructed the flag as '--shard-total=<value>', printing a doubled prefix ('Invalid --shard-total: --shard-total=abc'). parseFlag already returns just the value, so print it directly to match the planner CLI's wording. * ci(shard): warn on timing drift, enforce shard count at selection The integration shard check fails closed when a discovered file is missing from the timing manifest, so adding one integration test reds CI until someone hand-refreshes scripts/ci-integration-timings.json. The per-file weights are only a load-balancing hint: a missing or stale weight costs a little shard balance, never test correctness or coverage. Gating on a freshness signal blocks contributors (and forks, which run the secret-free ci.yml against the committed manifest) for an imbalance worth a few seconds on one shard. checkPlan now returns warnings separately from errors. Missing and stale files become warnings; the structural invariants (schema, shard-count drift, zero discovery, dropped or duplicated file) stay fail-closed. The check job prints warnings as ::warning:: annotations and exits 0, so the plan stays valid and a maintainer refreshes the manifest at leisure. Separately, runShard packed into whatever N/M the workflow passed while only --check compared the manifest to --shard-total, so a future edit could drift the matrix count from the manifest and silently drop or double-run tests at the point tests are selected. Guard manifest.shardTotal against the requested total in runShard too, dying on a mismatch instead of producing a malformed plan. * docs(ci): clarify missing timing warning * fix(ci): harden integration shard refresh	2026-06-06 13:55:58 +01:00