mirror of
https://github.com/cloudflare/vinext.git
synced 2026-09-14 19:04:59 +08:00
Nathan Nguyen
df066cb486
* ci: add fifth integration shard
* ci: weight integration shards from timing data
* ci: shard app-router e2e
* ci: rerun optimization experiment
* ci: add sixth weighted integration shard
* ci: rebalance weighted integration shards
* ci: shard unit tests
* ci: rebalance integration shards from current timings
* ci: move weighted integration shard list from YAML into script + timing manifest
* ci: move weighted integration shard list from YAML into script + timing manifest
* feat(ci): derive integration shard weights from real CI timings with provenance
Integration shard weights lived in a hand-seeded flat path->ms map
("aggregation": "manual seed"). A reviewer could not tell a measured
number from a guess, and the guesses were wrong: favicon-short-circuit
was seeded at 5s but runs ~35s in CI across five runs, a 7x under-weight
that mis-packed the shards. The seed had no provenance and no way to
regenerate from real data.
Restructure the manifest to a v2 provenance model: per file estimateMs
(the weight the planner uses), plus medianMs/p75Ms/samples and a
generatedFrom.runs list, an estimator metric, and generatedAt. Add
scripts/ci-integration-timings-refresh.mjs to aggregate Vitest blob
reports downloaded from successful CI runs (p75 per file, nearest-rank)
and rewrite the manifest deterministically, failing closed when the
blobs do not cover every discovered file. The manifest here was
regenerated from 5 successful runs (30 blobs); all six shards now pack
to 84s.
Extract planning and blob parsing into scripts/lib/* so the fragile
Vite+ blob-parser probe lives in one place. Replace the O(files*shards)
lightest-group scan with an O(n log m) binary min-heap and collapse the
three duplicated local-search move/swap helpers into one makespanAfter +
transfer primitive. Behavior preserved: the --check gate still verifies
every file lands in exactly one shard.
Harden --check to fail closed on no discovered files, missing, stale,
malformed/zero/negative timings, shard-count drift, and bucket coverage.
Add an advisory --recommend mode that models the optimal shard count from
real weights and flags when integration has dropped below the competing
cross-job bottleneck. It is advisory only and never runs in CI; the count
stays declarative in manifest.shardTotal with the matrix enforced
against it.
* ci: pass integration shard file list via env to avoid template injection
The integration shard step expanded ${{ steps.shard.outputs.files }}
directly into the run: block. That output is a list of test file paths
discovered from `vp test list`, and on pull_request runs a filename is
attacker-controllable: a fork PR adding a file whose name contains shell
metacharacters would inject it into the runner shell. GitHub code
scanning (zizmor) flagged this as template-injection, alert 163.
Route the file list and the other computed values through env vars and
reference them in the script, leaving $SHARD_FILES unquoted so the shell
still word-splits it into separate file arguments. The shell now treats
the value as data, never as script text. Verified with zizmor: the
pre-fix workflow reports template-injection on this line, the fixed
workflow reports no findings.
* feat(ci): require refresh blobs to back the claimed --run provenance
The refresh tool recorded every --run id as provenance but only checked
that each discovered file had at least one timing sample. Passing five
--run ids with blobs for a single complete run still produced a manifest
claiming five-run provenance while every file held one sample. The
manifest could claim stronger provenance than the blob directory backs.
A test file runs in exactly one shard per run, so one complete run
yields exactly one sample per file. Require samples === runIds.length for
every discovered file: too few means a claimed run's blobs are missing,
too many means the directory holds blobs beyond the claimed runs.
--allow-partial relaxes the check to "at least one sample per file" for
the re-run-failed-shard case while still recording the true per-file
sample count.
* experiment: run integration at 5 shards to benchmark the latency/cost knee
Temporary, for benchmarking only. Repacks the same provenance weights
into 5 integration shards instead of 6 (manifest shardTotal and matrix
set to 5, Check gate updated to match) so the 5 vs 6 trade-off can be
measured with the same weights, unit split, and E2E split. To be
reverted to 6 after the run is captured.
* experiment: go aggressive on wall-clock (8 integration, 3 unit, 3 app-router E2E)
Runner minutes are free on this public repo, so the objective is pure
wall-clock. Attack the whole critical-path cluster at once: integration
to 8 shards (~63s test load each, near the per-file floor), unit to 3,
and the app-router E2E project to 3-way so none of them becomes the new
ceiling once the others drop. Report job left as-is. Benchmarking only;
final counts settle after the run lands.
* ci: set integration to 10 shards, the wall-clock floor on free CI
Public repo, so runner minutes are free and the objective is pure
wall-clock. At 10 shards each integration shard carries ~51s of test
load; combined with the serial report tail this brings the integration
critical path down to roughly where the un-shardable create-next-app
(windows) job sits, so additional shards stop moving the overall wall.
Keeps unit at 3 shards and the app-router E2E project at 3-way from the
prior step. Benchmarking continues; counts can still change.
* fix(ci): default refresh shard count to the existing manifest, not a constant
ci-integration-timings-refresh.mjs defaulted --shard-total to a hardcoded
6. The documented refresh command in ci.yml omits --shard-total, so once
the matrix moved past 6 shards, following the advertised workflow rewrote
shardTotal: 6 into the manifest and the next run failed the Verify
integration shard manifest step with shard-count drift.
Default to the current manifest's shardTotal instead. manifest.shardTotal
is the single source of truth for the count: the matrix mirrors it and
--check enforces no drift, so a plain refresh now preserves whatever the
matrix uses. An explicit --shard-total still overrides it for an
intentional count change, and a missing count with no existing manifest
now fails with a clear message instead of silently picking a number.
Found by Codex review on a31c99c4.
* refactor(ci): share integration shard CLI helpers
* refactor(ci): clarify shard local search
* refactor(ci): drop doubled flag prefix in refresh shard-total error
The invalid --shard-total message reconstructed the flag as
'--shard-total=<value>', printing a doubled prefix
('Invalid --shard-total: --shard-total=abc'). parseFlag already
returns just the value, so print it directly to match the planner
CLI's wording.
* ci(shard): warn on timing drift, enforce shard count at selection
The integration shard check fails closed when a discovered file is
missing from the timing manifest, so adding one integration test reds CI
until someone hand-refreshes scripts/ci-integration-timings.json. The
per-file weights are only a load-balancing hint: a missing or stale
weight costs a little shard balance, never test correctness or coverage.
Gating on a freshness signal blocks contributors (and forks, which run
the secret-free ci.yml against the committed manifest) for an imbalance
worth a few seconds on one shard.
checkPlan now returns warnings separately from errors. Missing and stale
files become warnings; the structural invariants (schema, shard-count
drift, zero discovery, dropped or duplicated file) stay fail-closed. The
check job prints warnings as ::warning:: annotations and exits 0, so the
plan stays valid and a maintainer refreshes the manifest at leisure.
Separately, runShard packed into whatever N/M the workflow passed while
only --check compared the manifest to --shard-total, so a future edit
could drift the matrix count from the manifest and silently drop or
double-run tests at the point tests are selected. Guard
manifest.shardTotal against the requested total in runShard too, dying on
a mismatch instead of producing a malformed plan.
* docs(ci): clarify missing timing warning
* fix(ci): harden integration shard refresh
447 B
447 B