mirror of
https://github.com/civitai/civitai.git
synced 2026-09-20 22:08:18 +08:00
93ee73cd84
* test(perf): instrument the unit suite and track the isolation migration
The unit suite spends 81% of its worker time importing, and nothing in the repo
could say which modules or which files. This adds the measurement that answers it,
plus a dashboard so the isolation migration has a burn-down nobody has to maintain.
The finding that motivated the shape of it: traced at 1 worker, the module BODIES
of two of the heaviest test files total ~0.4s against a 25.4s import phase. The
cost is vite-node's per-module fetch, which under the `forks` pool is a
child-process IPC round trip per module per file. So cost is linear in module
COUNT, not module weight, and the static import graph is the right thing to rank by.
- graph.mjs static first-party import graph + vi.mock inventory
- reporter.mjs per-file collect/setup/test timings from any run
- bench.mjs fixed 90-file stratified yardstick, so two measurements taken
hours apart by different people are comparable
- sweep.mjs pool x isolation x worker-count matrix, run back to back
- trace-* module-execution tracer; counts what actually ran, which the
static graph cannot know (a vi.mock factory stops the real
module and its subtree from executing)
- why.mjs shortest import path between two modules
- dashboard.mjs builds .test-perf/dashboard.html from whatever is on disk
Output goes to .test-perf/, gitignored. Nothing here runs in CI or changes how
any suite executes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(perf): add the per-worker union report, and record three dead ends
Under isolate:false a worker keeps one module registry, so its cost is the UNION
of what its files import rather than the sum. order.mjs reports that union, which
is the number that bounds what removing isolation can deliver: measured at ~36
seconds per worker to build, near-constant at 8 and at 24 workers, and a wall-clock
floor more workers cannot shrink.
Three things measured today that do NOT help, written down so the next person does
not pay for them again:
- Affinity file ordering. A greedy graph-similarity sequencer gave a mean per-worker
union of 1139 modules at 31 workers against alphabetical's 1084 - slightly worse.
Alphabetical already groups by directory and directory already correlates with the
graph. The sequencer is deleted; the measurement is kept.
- NODE_COMPILE_CACHE. Cold 26.8s, warm 51.4s, warm again 33.9s on the yardstick. The
cache filled (5.3MB) so it was active; vite-node does not evaluate through the
loader it covers.
- vmThreads. Two clean 90-file runs, then the five sharp-executing files crashed or
passed on identical input at 2 and 3 workers. It is a race, and CI's 4 vCPU
resolves to the width measured at 1-in-3 SIGSEGV.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(test-perf): three measurement rules this project had to learn twice
- The 90-file yardstick understates isolate:false and cannot judge it. That flag
amortises the registry build across the files a worker runs, so its win scales
with files-per-worker: 90 files at 16 workers is ~6 each and measures 1.65x,
while 1065 files at 8 workers is ~133 each and measures 16x on the same phase.
- No --no-isolate number is quotable without a per-file collected count. Its damage
is not only failing assertions: files silently collect ZERO tests, and how many is
width-dependent (9 of 90 at forks/4, 14 of 90 at threads/4, 0 at threads/16, same
input). A summary line cannot show this.
- The noise floor on a shared box is +/-30%: one configuration measured 53.3s and
76.6s in a single session. Below ~20%, quote phase numbers or nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(perf): show the guard allowlist as the migration burn-down
The dashboard was deriving migration status from the static vi.mock inventory,
which is an estimate. The authoritative number is the length of the guard's
allowlist: it ratchets in both directions, so a new direct mock fails and a
migrated file left on the list fails too, and it therefore cannot drift from what
the suite will actually accept.
Reads it from the working tree when present, otherwise from the branch carrying
it, so a dashboard built on main still shows the real number rather than nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(test-perf): name the baseline, retract the IPC mechanism, fix the yardstick's promise
Three defects found by an adversarial review of this PR, all in the class of
asserting something nobody re-checked.
- The README quoted one baseline while shipping an artifact with different
numbers for the same tree - 4565.5s against 5476.3s of import, 36% apart. The
tree did not change; the box did. All three measurements of `3863adcbb0` are
now tabulated with the rule that a main-relative figure must name the run it
was measured against, because that spread is the same size as several of the
effects being measured against it. The derived import share is 81% or 84.3%
depending which row you pick, and both are now stated.
- The per-module IPC mechanism was retracted in mail hours ago and left standing
here. The pool sweep refutes it: `threads` beat `vmThreads` while paying a cold
fetch, which shipping module source cannot explain. Now labelled inferred, with
the competing reading and the note that V8 compile time was never instrumented.
- The yardstick claimed it made measurements "hours apart" comparable, three
bullets above a +/-30% noise floor measured inside one session. It fixes what is
measured, not when. Also records that a null from a 90-of-1065 sample is not
evidence of no effect - two changes measured flat there were later shown real.
And the gitignore claim is now scoped to "by this change", since it is false on
main until this merges.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(test-perf): count the modules a worker loads, not the ones a bundler compiles
The static graph over-counted a page-gate test by ~75x, which put four of the
cheapest files in the suite at the top of the closure ranking. Their measured
worker time ranks 202-572 of 1065.
Four independent causes, each removing modules the naive walk counted:
- lazy `import()` is not followed, EXCEPT from a test file itself. A
`dynamic(() => import())` in a page never runs; an `await import()` in a
test body is what the test exists to do. Collapsing the two makes those
files either the top of the ranking or 1 module each.
- a `vi.mock` factory without `importOriginal` truncates the subtree behind
it. The mocked module itself is still counted -- registering the mock is
what causes the transform -- and is counted even when nothing imports it.
- `import { type X } from` erases the statement. Only `import type {` was
handled, and the inline form is the common shape here.
- a line filter cannot strip a multi-line `import type`: it leaves
`} from '...'` behind, and IMPORT_RE's lazy `[\s\S]*?` glues that orphan
onto the previous import, inventing an edge. Stripped as statements now.
Plus `event-engine-common` in SRC_DIRS -- a submodule imported by relative
path from src/server/services, so its modules and the civitai-db-queries
files reachable only through it were invisible.
Validated by diffing the model against a transform-hook trace as SETS, not
counts: 3 of 5 files exact with empty diffs both ways, 16 modules of
symmetric error across 691 traced (2.32%). Counts alone agreed often enough
to hide two of the four causes -- a count agreeing is not the rule agreeing.
`graphModules` is now the honest count and stays the default field, so
dashboard.mjs is corrected without a change of its own; `graphModulesRaw`
keeps the bundler view. closures.json gains `mode: 'real'` and a note
describing the truncation rules, so a consumer can refuse a naive one.
Also flags the two order.mjs union figures as pre-honest levels needing a
re-run; the ordering conclusion is a ratio and survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(test-perf): stop the type-statement stripper eating real code, and record maxWorkers honestly
Three review findings, all mine.
1. graph.mjs -- the type-statement stripper swallowed spans of real code.
`TYPE_STATEMENT_RE` also started matching on a bodyless `export type X = ...`
(no `from`), and the lazy `[\s\S]*?from '...'` then scanned FORWARD across the
file to the next `from '...'`-shaped text -- inside comments and strings
included -- deleting every statement in between before IMPORT_RE saw them.
src/server/common/enums.ts lost a real `@civitai/notifications/constants`
edge across a 7,881-character span.
The gap is now tempered: between `type` and `from` a real type-import
statement holds only a binding clause, so it may not cross `;`, `=`, or
another `import`/`export` keyword. Measured across 5,294 files, spans of
more than 12 lines fall from 73 to 28, and every one of the 28 remaining is
a genuinely long multi-line type-import list. Re-validated against the
transform-hook traces: symmetric error 2.32% -> 2.03% over 691 traced
modules, so the fix is strictly in the right direction.
Direction of the bug was UNDER-count, and `graphModules` is the ranking key
for the dashboard, order.mjs and bench.mjs --make-subset. The earlier
validation did not cover it: 5 files of 1,065, none of the affected ones.
2. reporter.mjs -- the one unguarded filesystem call. It sits inside an awaited
vitest lifecycle hook, so a throw (read-only cwd, `.test-perf` existing as a
file, ENOSPC, an AV EPERM) would red a run whose tests all passed, and the
failure would be attributed to the code under test. A measurement tool must
never be able to fail a suite; losing the recording is the correct trade.
3. reporter.mjs -- `maxWorkers` is not on vitest 4's `ctx.config`. Verified by
execution: the emitted config was `{isolate, pool, argv}`. Every run ever
recorded therefore stored null, and dashboard.mjs rendered all of them as
"(default) workers" -- claiming a fact nobody measured. Recovered from argv
(`--max-workers=8`, `--max-workers 8`, `--maxWorkers=8`), falling back to
VITEST_MAX_WORKERS, and reported as 'unknown' rather than null when neither
is present. The env var and the flag are recorded separately because they do
not behave the same -- the flag reaches a queued run and the env var does
not. The dashboard now shows historical nulls as 'unrecorded', not
'default'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
145 lines
5.0 KiB
JavaScript
145 lines
5.0 KiB
JavaScript
#!/usr/bin/env node
|
|
/**
|
|
* Why does <file> import <target>?
|
|
*
|
|
* node scripts/test-perf/why.mjs src/server/__tests__/x.test.ts @mantine/core
|
|
* node scripts/test-perf/why.mjs src/server/__tests__/x.test.ts ~/server/services/image.service
|
|
* node scripts/test-perf/why.mjs src/server/__tests__/x.test.ts --ui
|
|
*
|
|
* Prints the SHORTEST import path, which is the edge to argue about. `--ui` looks for the first
|
|
* browser-only package reached from a server-side test, which is the usual accident.
|
|
*
|
|
* Static, so it does not know that a `vi.mock` factory would stop the real module executing — use
|
|
* the tracer for that. It is still the right tool for finding the edge to cut.
|
|
*/
|
|
import { readFileSync, readdirSync, statSync } from 'fs';
|
|
import path from 'path';
|
|
import { fileURLToPath } from 'url';
|
|
|
|
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../..');
|
|
const EXT = ['.ts', '.tsx', '.mts', '.cts', '.js', '.jsx', '.mjs'];
|
|
const norm = (p) => p.replace(/\\/g, '/');
|
|
|
|
function walk(dir, out = []) {
|
|
let entries;
|
|
try {
|
|
entries = readdirSync(dir, { withFileTypes: true });
|
|
} catch {
|
|
return out;
|
|
}
|
|
for (const e of entries) {
|
|
if (['node_modules', '.next', 'dist', '.git'].includes(e.name)) continue;
|
|
const p = path.join(dir, e.name);
|
|
if (e.isDirectory()) walk(p, out);
|
|
else if (EXT.includes(path.extname(e.name))) out.push(norm(p));
|
|
}
|
|
return out;
|
|
}
|
|
|
|
const fileSet = new Set(['src', 'scripts', 'packages'].flatMap((d) => walk(path.join(repoRoot, d))));
|
|
|
|
function tryFile(base) {
|
|
const b = norm(base);
|
|
for (const e of EXT) if (fileSet.has(b + e)) return b + e;
|
|
if (fileSet.has(b)) return b;
|
|
for (const e of EXT) if (fileSet.has(`${b}/index${e}`)) return `${b}/index${e}`;
|
|
return null;
|
|
}
|
|
|
|
function resolveSpecifier(spec, fromFile) {
|
|
if (spec.startsWith('~/')) return tryFile(path.join(repoRoot, 'src', spec.slice(2)));
|
|
if (spec.startsWith('@civitai/')) {
|
|
const [pkg, ...sub] = spec.slice('@civitai/'.length).split('/');
|
|
const base = path.join(repoRoot, 'packages', `civitai-${pkg}`, 'src');
|
|
return tryFile(sub.length ? path.join(base, sub.join('/')) : path.join(base, 'index'));
|
|
}
|
|
if (spec.startsWith('.')) return tryFile(path.resolve(path.dirname(fromFile), spec));
|
|
return null;
|
|
}
|
|
|
|
const IMPORT_RE =
|
|
/(?:^|\n)\s*import\s+(?:[\s\S]*?\sfrom\s+)?['"]([^'"]+)['"]|(?:^|[^.\w])(?:import|require)\s*\(\s*['"]([^'"]+)['"]\s*\)|(?:^|\n)\s*export\s+[\s\S]*?\sfrom\s+['"]([^'"]+)['"]/g;
|
|
|
|
const edgeCache = new Map();
|
|
function edges(file) {
|
|
if (edgeCache.has(file)) return edgeCache.get(file);
|
|
let text = '';
|
|
try {
|
|
text = readFileSync(file, 'utf8');
|
|
} catch {}
|
|
const lines = text
|
|
.split('\n')
|
|
.filter((l) => !/^\s*import\s+type\s/.test(l) && !/^\s*export\s+type\s/.test(l));
|
|
const scan = lines.join('\n');
|
|
const out = [];
|
|
let m;
|
|
IMPORT_RE.lastIndex = 0;
|
|
while ((m = IMPORT_RE.exec(scan))) {
|
|
const spec = m[1] || m[2] || m[3];
|
|
if (!spec) continue;
|
|
const resolved = resolveSpecifier(spec, file);
|
|
out.push({ spec, resolved });
|
|
}
|
|
edgeCache.set(file, out);
|
|
return out;
|
|
}
|
|
|
|
const UI_PACKAGES = [
|
|
'@mantine/', '@dnd-kit/', '@tabler/icons-react', 'react-dom', '@tiptap/', 'chart.js',
|
|
'@grafana/faro-web-sdk', 'embla-carousel', 'framer-motion', 'react-easy-crop', '@hello-pangea/',
|
|
'next/router', 'next/image', 'react-hook-form', '@mantine/core',
|
|
];
|
|
|
|
const [rawStart, rawTarget] = process.argv.slice(2);
|
|
if (!rawStart || !rawTarget) {
|
|
console.error('usage: why.mjs <test-or-source-file> <target specifier | --ui>');
|
|
process.exit(2);
|
|
}
|
|
const start = tryFile(path.resolve(repoRoot, rawStart)) ?? norm(path.resolve(repoRoot, rawStart));
|
|
|
|
const wantUi = rawTarget === '--ui';
|
|
const targetResolved = wantUi ? null : resolveSpecifier(rawTarget, start);
|
|
|
|
// BFS keeps the path SHORTEST, which is the one worth arguing about: a long path usually means a
|
|
// legitimate chain, a short one usually means a single wrong import.
|
|
const prev = new Map([[start, null]]);
|
|
const queue = [start];
|
|
let hit = null;
|
|
let hitSpec = null;
|
|
while (queue.length && !hit) {
|
|
const cur = queue.shift();
|
|
for (const { spec, resolved } of edges(cur)) {
|
|
if (wantUi) {
|
|
if (UI_PACKAGES.some((p) => spec.startsWith(p))) {
|
|
hit = cur;
|
|
hitSpec = spec;
|
|
break;
|
|
}
|
|
} else if (resolved && targetResolved && resolved === targetResolved) {
|
|
hit = resolved;
|
|
prev.set(resolved, cur);
|
|
break;
|
|
} else if (!targetResolved && spec === rawTarget) {
|
|
hit = cur;
|
|
hitSpec = spec;
|
|
break;
|
|
}
|
|
if (resolved && !prev.has(resolved)) {
|
|
prev.set(resolved, cur);
|
|
queue.push(resolved);
|
|
}
|
|
}
|
|
}
|
|
|
|
if (!hit) {
|
|
console.log(`no path from ${rawStart} to ${rawTarget}`);
|
|
process.exit(1);
|
|
}
|
|
|
|
const chain = [];
|
|
for (let n = hit; n != null; n = prev.get(n)) chain.push(path.relative(repoRoot, n).replace(/\\/g, '/'));
|
|
chain.reverse();
|
|
console.log(`shortest path (${chain.length} hops):\n`);
|
|
chain.forEach((c, i) => console.log(`${' '.repeat(i > 6 ? 6 : i)}${i === 0 ? '' : '-> '}${c}`));
|
|
if (hitSpec) console.log(`${' '.repeat(Math.min(chain.length, 6))}-> ${hitSpec} <-- target`);
|