skills/cloudflare__vinext
1
0
Fork 0
mirror of https://github.com/cloudflare/vinext.git synced 2026-09-14 19:04:59 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
vinext@0.1.0
cloudflare__vinext/scripts/ci-integration-timings.json
T
Nathan Nguyen df066cb486 ci: optimize CI sharding (#1768) 
* ci: add fifth integration shard

* ci: weight integration shards from timing data

* ci: shard app-router e2e

* ci: rerun optimization experiment

* ci: add sixth weighted integration shard

* ci: rebalance weighted integration shards

* ci: shard unit tests

* ci: rebalance integration shards from current timings

* ci: move weighted integration shard list from YAML into script + timing manifest

* ci: move weighted integration shard list from YAML into script + timing manifest

* feat(ci): derive integration shard weights from real CI timings with provenance

Integration shard weights lived in a hand-seeded flat path->ms map
("aggregation": "manual seed"). A reviewer could not tell a measured
number from a guess, and the guesses were wrong: favicon-short-circuit
was seeded at 5s but runs ~35s in CI across five runs, a 7x under-weight
that mis-packed the shards. The seed had no provenance and no way to
regenerate from real data.

Restructure the manifest to a v2 provenance model: per file estimateMs
(the weight the planner uses), plus medianMs/p75Ms/samples and a
generatedFrom.runs list, an estimator metric, and generatedAt. Add
scripts/ci-integration-timings-refresh.mjs to aggregate Vitest blob
reports downloaded from successful CI runs (p75 per file, nearest-rank)
and rewrite the manifest deterministically, failing closed when the
blobs do not cover every discovered file. The manifest here was
regenerated from 5 successful runs (30 blobs); all six shards now pack
to 84s.

Extract planning and blob parsing into scripts/lib/* so the fragile
Vite+ blob-parser probe lives in one place. Replace the O(files*shards)
lightest-group scan with an O(n log m) binary min-heap and collapse the
three duplicated local-search move/swap helpers into one makespanAfter +
transfer primitive. Behavior preserved: the --check gate still verifies
every file lands in exactly one shard.

Harden --check to fail closed on no discovered files, missing, stale,
malformed/zero/negative timings, shard-count drift, and bucket coverage.
Add an advisory --recommend mode that models the optimal shard count from
real weights and flags when integration has dropped below the competing
cross-job bottleneck. It is advisory only and never runs in CI; the count
stays declarative in manifest.shardTotal with the matrix enforced
against it.

* ci: pass integration shard file list via env to avoid template injection

The integration shard step expanded ${{ steps.shard.outputs.files }}
directly into the run: block. That output is a list of test file paths
discovered from `vp test list`, and on pull_request runs a filename is
attacker-controllable: a fork PR adding a file whose name contains shell
metacharacters would inject it into the runner shell. GitHub code
scanning (zizmor) flagged this as template-injection, alert 163.

Route the file list and the other computed values through env vars and
reference them in the script, leaving $SHARD_FILES unquoted so the shell
still word-splits it into separate file arguments. The shell now treats
the value as data, never as script text. Verified with zizmor: the
pre-fix workflow reports template-injection on this line, the fixed
workflow reports no findings.

* feat(ci): require refresh blobs to back the claimed --run provenance

The refresh tool recorded every --run id as provenance but only checked
that each discovered file had at least one timing sample. Passing five
--run ids with blobs for a single complete run still produced a manifest
claiming five-run provenance while every file held one sample. The
manifest could claim stronger provenance than the blob directory backs.

A test file runs in exactly one shard per run, so one complete run
yields exactly one sample per file. Require samples === runIds.length for
every discovered file: too few means a claimed run's blobs are missing,
too many means the directory holds blobs beyond the claimed runs.
--allow-partial relaxes the check to "at least one sample per file" for
the re-run-failed-shard case while still recording the true per-file
sample count.

* experiment: run integration at 5 shards to benchmark the latency/cost knee

Temporary, for benchmarking only. Repacks the same provenance weights
into 5 integration shards instead of 6 (manifest shardTotal and matrix
set to 5, Check gate updated to match) so the 5 vs 6 trade-off can be
measured with the same weights, unit split, and E2E split. To be
reverted to 6 after the run is captured.

* experiment: go aggressive on wall-clock (8 integration, 3 unit, 3 app-router E2E)

Runner minutes are free on this public repo, so the objective is pure
wall-clock. Attack the whole critical-path cluster at once: integration
to 8 shards (~63s test load each, near the per-file floor), unit to 3,
and the app-router E2E project to 3-way so none of them becomes the new
ceiling once the others drop. Report job left as-is. Benchmarking only;
final counts settle after the run lands.

* ci: set integration to 10 shards, the wall-clock floor on free CI

Public repo, so runner minutes are free and the objective is pure
wall-clock. At 10 shards each integration shard carries ~51s of test
load; combined with the serial report tail this brings the integration
critical path down to roughly where the un-shardable create-next-app
(windows) job sits, so additional shards stop moving the overall wall.
Keeps unit at 3 shards and the app-router E2E project at 3-way from the
prior step. Benchmarking continues; counts can still change.

* fix(ci): default refresh shard count to the existing manifest, not a constant

ci-integration-timings-refresh.mjs defaulted --shard-total to a hardcoded
6. The documented refresh command in ci.yml omits --shard-total, so once
the matrix moved past 6 shards, following the advertised workflow rewrote
shardTotal: 6 into the manifest and the next run failed the Verify
integration shard manifest step with shard-count drift.

Default to the current manifest's shardTotal instead. manifest.shardTotal
is the single source of truth for the count: the matrix mirrors it and
--check enforces no drift, so a plain refresh now preserves whatever the
matrix uses. An explicit --shard-total still overrides it for an
intentional count change, and a missing count with no existing manifest
now fails with a clear message instead of silently picking a number.

Found by Codex review on a31c99c4.

* refactor(ci): share integration shard CLI helpers

* refactor(ci): clarify shard local search

* refactor(ci): drop doubled flag prefix in refresh shard-total error

The invalid --shard-total message reconstructed the flag as
'--shard-total=<value>', printing a doubled prefix
('Invalid --shard-total: --shard-total=abc'). parseFlag already
returns just the value, so print it directly to match the planner
CLI's wording.

* ci(shard): warn on timing drift, enforce shard count at selection

The integration shard check fails closed when a discovered file is
missing from the timing manifest, so adding one integration test reds CI
until someone hand-refreshes scripts/ci-integration-timings.json. The
per-file weights are only a load-balancing hint: a missing or stale
weight costs a little shard balance, never test correctness or coverage.
Gating on a freshness signal blocks contributors (and forks, which run
the secret-free ci.yml against the committed manifest) for an imbalance
worth a few seconds on one shard.

checkPlan now returns warnings separately from errors. Missing and stale
files become warnings; the structural invariants (schema, shard-count
drift, zero discovery, dropped or duplicated file) stay fail-closed. The
check job prints warnings as ::warning:: annotations and exits 0, so the
plan stays valid and a maintainer refreshes the manifest at leisure.

Separately, runShard packed into whatever N/M the workflow passed while
only --check compared the manifest to --shard-total, so a future edit
could drift the matrix count from the manifest and silently drop or
double-run tests at the point tests are selected. Guard
manifest.shardTotal against the requested total in runShard too, dying on
a mismatch instead of producing a malformed plan.

* docs(ci): clarify missing timing warning

* fix(ci): harden integration shard refresh
2026-06-06 13:55:58 +01:00

390 lines
9.1 KiB
JSON
Raw Permalink Blame History

{
"version": 2,
"suite": "integration",
"shardTotal": 10,
"generatedAt": "2026-06-05T17:52:19.779Z",
"generatedFrom": {
"runs": [
{
"id": "27028007653",
"url": "https://github.com/cloudflare/vinext/actions/runs/27028007653"
},
{
"id": "27026520579",
"url": "https://github.com/cloudflare/vinext/actions/runs/27026520579"
},
{
"id": "27025678394",
"url": "https://github.com/cloudflare/vinext/actions/runs/27025678394"
},
{
"id": "27024991233",
"url": "https://github.com/cloudflare/vinext/actions/runs/27024991233"
},
{
"id": "27024621441",
"url": "https://github.com/cloudflare/vinext/actions/runs/27024621441"
}
]
},
"estimator": {
"metric": "p75"
},
"files": {
"tests/api-handler.test.ts": {
"estimateMs": 76,
"medianMs": 72,
"p75Ms": 76,
"samples": 5
},
"tests/app-router-client-preloading.test.ts": {
"estimateMs": 15,
"medianMs": 14,
"p75Ms": 15,
"samples": 5
},
"tests/app-router-dev-server.test.ts": {
"estimateMs": 12546,
"medianMs": 11924,
"p75Ms": 12546,
"samples": 5
},
"tests/app-router-external-rewrite.test.ts": {
"estimateMs": 989,
"medianMs": 976,
"p75Ms": 989,
"samples": 5
},
"tests/app-router-font-google-prod.test.ts": {
"estimateMs": 2396,
"medianMs": 2294,
"p75Ms": 2396,
"samples": 5
},
"tests/app-router-isr-codegen.test.ts": {
"estimateMs": 15,
"medianMs": 14,
"p75Ms": 15,
"samples": 5
},
"tests/app-router-malformed-url.test.ts": {
"estimateMs": 7157,
"medianMs": 7079,
"p75Ms": 7157,
"samples": 5
},
"tests/app-router-metadata-routes.test.ts": {
"estimateMs": 8054,
"medianMs": 7574,
"p75Ms": 8054,
"samples": 5
},
"tests/app-router-middleware-next-request.test.ts": {
"estimateMs": 7785,
"medianMs": 7717,
"p75Ms": 7785,
"samples": 5
},
"tests/app-router-next-config-codegen.test.ts": {
"estimateMs": 152,
"medianMs": 151,
"p75Ms": 152,
"samples": 5
},
"tests/app-router-next-config-dev.test.ts": {
"estimateMs": 7647,
"medianMs": 7500,
"p75Ms": 7647,
"samples": 5
},
"tests/app-router-origin-check.test.ts": {
"estimateMs": 7549,
"medianMs": 7309,
"p75Ms": 7549,
"samples": 5
},
"tests/app-router-production-build.test.ts": {
"estimateMs": 9615,
"medianMs": 9423,
"p75Ms": 9615,
"samples": 5
},
"tests/app-router-production-server.test.ts": {
"estimateMs": 27331,
"medianMs": 26778,
"p75Ms": 27331,
"samples": 5
},
"tests/app-router-rsc-flight-hint.test.ts": {
"estimateMs": 5,
"medianMs": 5,
"p75Ms": 5,
"samples": 5
},
"tests/app-router-rsc-plugin.test.ts": {
"estimateMs": 13534,
"medianMs": 13043,
"p75Ms": 13534,
"samples": 5
},
"tests/app-router-static-export.test.ts": {
"estimateMs": 14970,
"medianMs": 14891,
"p75Ms": 14970,
"samples": 5
},
"tests/app-router-worker-entry.test.ts": {
"estimateMs": 136,
"medianMs": 132,
"p75Ms": 136,
"samples": 5
},
"tests/cjs.test.ts": {
"estimateMs": 7461,
"medianMs": 7424,
"p75Ms": 7461,
"samples": 5
},
"tests/ecosystem.test.ts": {
"estimateMs": 35921,
"medianMs": 35661,
"p75Ms": 35921,
"samples": 5
},
"tests/entry-templates.test.ts": {
"estimateMs": 54,
"medianMs": 53,
"p75Ms": 54,
"samples": 5
},
"tests/favicon-short-circuit.test.ts": {
"estimateMs": 36378,
"medianMs": 35779,
"p75Ms": 36378,
"samples": 5
},
"tests/features.test.ts": {
"estimateMs": 18480,
"medianMs": 18359,
"p75Ms": 18480,
"samples": 5
},
"tests/image-optimization-parity.test.ts": {
"estimateMs": 8936,
"medianMs": 8855,
"p75Ms": 8936,
"samples": 5
},
"tests/kv-cache-handler.test.ts": {
"estimateMs": 2446,
"medianMs": 2443,
"p75Ms": 2446,
"samples": 5
},
"tests/nextjs-compat/action-headers.test.ts": {
"estimateMs": 7403,
"medianMs": 7380,
"p75Ms": 7403,
"samples": 5
},
"tests/nextjs-compat/als-isolation.test.ts": {
"estimateMs": 94,
"medianMs": 92,
"p75Ms": 94,
"samples": 5
},
"tests/nextjs-compat/app-css.test.ts": {
"estimateMs": 7496,
"medianMs": 7381,
"p75Ms": 7496,
"samples": 5
},
"tests/nextjs-compat/app-rendering.test.ts": {
"estimateMs": 9586,
"medianMs": 9296,
"p75Ms": 9586,
"samples": 5
},
"tests/nextjs-compat/app-routes.test.ts": {
"estimateMs": 8014,
"medianMs": 7951,
"p75Ms": 8014,
"samples": 5
},
"tests/nextjs-compat/basepath.test.ts": {
"estimateMs": 110,
"medianMs": 85,
"p75Ms": 110,
"samples": 5
},
"tests/nextjs-compat/draft-mode.test.ts": {
"estimateMs": 7630,
"medianMs": 7406,
"p75Ms": 7630,
"samples": 5
},
"tests/nextjs-compat/dynamic.test.ts": {
"estimateMs": 8086,
"medianMs": 7815,
"p75Ms": 8086,
"samples": 5
},
"tests/nextjs-compat/global-error.test.ts": {
"estimateMs": 15167,
"medianMs": 14845,
"p75Ms": 15167,
"samples": 5
},
"tests/nextjs-compat/global-not-found.test.ts": {
"estimateMs": 2551,
"medianMs": 2507,
"p75Ms": 2551,
"samples": 5
},
"tests/nextjs-compat/hooks.test.ts": {
"estimateMs": 8188,
"medianMs": 8079,
"p75Ms": 8188,
"samples": 5
},
"tests/nextjs-compat/layout-segments.test.ts": {
"estimateMs": 7443,
"medianMs": 7366,
"p75Ms": 7443,
"samples": 5
},
"tests/nextjs-compat/metadata-suspense.test.ts": {
"estimateMs": 7201,
"medianMs": 7126,
"p75Ms": 7201,
"samples": 5
},
"tests/nextjs-compat/metadata.test.ts": {
"estimateMs": 8721,
"medianMs": 8497,
"p75Ms": 8721,
"samples": 5
},
"tests/nextjs-compat/navigation.test.ts": {
"estimateMs": 7579,
"medianMs": 7384,
"p75Ms": 7579,
"samples": 5
},
"tests/nextjs-compat/not-found.test.ts": {
"estimateMs": 8405,
"medianMs": 8223,
"p75Ms": 8405,
"samples": 5
},
"tests/nextjs-compat/prefetch.test.ts": {
"estimateMs": 7345,
"medianMs": 7243,
"p75Ms": 7345,
"samples": 5
},
"tests/nextjs-compat/request-apis.test.ts": {
"estimateMs": 7630,
"medianMs": 7265,
"p75Ms": 7630,
"samples": 5
},
"tests/nextjs-compat/revalidate.test.ts": {
"estimateMs": 7431,
"medianMs": 7413,
"p75Ms": 7431,
"samples": 5
},
"tests/nextjs-compat/rsc-basic.test.ts": {
"estimateMs": 7405,
"medianMs": 7395,
"p75Ms": 7405,
"samples": 5
},
"tests/nextjs-compat/rsc-context-lazy-stream.test.ts": {
"estimateMs": 7507,
"medianMs": 7465,
"p75Ms": 7507,
"samples": 5
},
"tests/nextjs-compat/script-nonce.test.ts": {
"estimateMs": 7237,
"medianMs": 7233,
"p75Ms": 7237,
"samples": 5
},
"tests/nextjs-compat/script-preload.test.ts": {
"estimateMs": 7701,
"medianMs": 7363,
"p75Ms": 7701,
"samples": 5
},
"tests/nextjs-compat/set-cookies.test.ts": {
"estimateMs": 7237,
"medianMs": 7157,
"p75Ms": 7237,
"samples": 5
},
"tests/nextjs-compat/streaming.test.ts": {
"estimateMs": 10320,
"medianMs": 10302,
"p75Ms": 10320,
"samples": 5
},
"tests/nextjs-compat/use-client-page-pathname.test.ts": {
"estimateMs": 7485,
"medianMs": 7355,
"p75Ms": 7485,
"samples": 5
},
"tests/node-modules-css.test.ts": {
"estimateMs": 7742,
"medianMs": 7538,
"p75Ms": 7742,
"samples": 5
},
"tests/pages-i18n-prod.test.ts": {
"estimateMs": 1388,
"medianMs": 1364,
"p75Ms": 1388,
"samples": 5
},
"tests/pages-router-concurrency.test.ts": {
"estimateMs": 2858,
"medianMs": 2705,
"p75Ms": 2858,
"samples": 5
},
"tests/pages-router.test.ts": {
"estimateMs": 27384,
"medianMs": 26734,
"p75Ms": 27384,
"samples": 5
},
"tests/postcss-resolve.test.ts": {
"estimateMs": 445,
"medianMs": 435,
"p75Ms": 445,
"samples": 5
},
"tests/prerender.test.ts": {
"estimateMs": 27967,
"medianMs": 26676,
"p75Ms": 27967,
"samples": 5
},
"tests/static-export.test.ts": {
"estimateMs": 16984,
"medianMs": 16896,
"p75Ms": 16984,
"samples": 5
},
"tests/vite-hmr-websocket.test.ts": {
"estimateMs": 1242,
"medianMs": 611,
"p75Ms": 1242,
"samples": 5
}
}
}
Reference in New Issue View Git Blame Copy Permalink