civitai__civitai/docs/testing/shared-module-mocks.md

Shared-module mocks: running the unit suite without per-file isolation

Status: design + pilot. The system described here exists; the migration does not.

The unit suite spends ~81% of its worker-time importing modules. vitest --no-isolate collapses that (2266 worker-seconds to 229 at 8 workers) because each worker builds the module graph once instead of once per test file. It also breaks ~1,500 tests. This document is how to get the first without the second.

The mechanism, precisely

The one-line version — "vi.mock is per file, module instances are per worker" — is right but under-specified, and the missing detail decides the design. Measured with a two-file probe (--no-isolate --max-workers=1):

A test file that does not mock a module still gets the real module. The poisoning is not direct. A file with no vi.mock('./target') imports the genuine ./target, even after another file in the same worker mocked it.

The damage travels through shared NON-test modules. Any ordinary source module that imports a mocked module — ~/server/redis/fail-open-log importing safeError from ~/server/logging/client, say — is itself cached per worker. It captures its import bindings once, at its first evaluation, in whatever mock context that first evaluation happened to occur. Every later file reuses that cached module, still pointing at the first file's mock object.

The probe reproduces the exact error from the real suite:

Error: [vitest] No "beta" export is defined on the "./target" mock.
 ❯ useBeta consumer.ts:11:17
 ❯ 2-victim.test.ts:11:12

2-victim.test.ts contains no vi.mock at all. It failed because consumer.ts was first evaluated while 1-poisoner.test.ts's partial mock was installed.

Three consequences follow, and they are the whole specification:

1. The mocked shape must be complete, because a consumer may reach any export.
2. Function identities must be stable for the worker's lifetime, because consumers captured those bindings and will never re-read them. Behaviour has to be swapped by mutating a function, never by replacing it.
3. Per-file state must be reset between files, because the object outlives the file. This includes spy call counts, which is the expected "X" to be called 2 times failure class.

The damage has a silent form. When the poisoned import is reached at module scope rather than inside a test, the file throws while being imported and collects ZERO tests. The failure count does not rise, the summary line does not mention it, and the run reads as green while those tests simply did not happen — measured at nine files and 152 tests on the 90-file yardstick subset. So the success criterion for any run under --no-isolate is a per-file collected COUNT diffed against an isolated control, never a colour. Same shape as the repo's existing trap where a missing event-engine-common submodule makes blocks.router.workflow.test.ts collect 0 and the run still passes.

setupFiles re-run per test file under --no-isolate — verified in the same probe (setup run #1, setup run #2, same pid, same shared-module instance). That is the reset hook, and it is why this is tractable at all.

The shape

One canonical mock module per shared specifier. It is imported by the global setup file, which registers it for every test file and resets it between them. A test file never mocks these specifiers itself; it only declares behaviour.

src/__tests__/mocks/
  hybrid.ts        the auto-vivifying callable-proxy primitive
  db.mock.ts       ~/server/db/client
  redis.mock.ts    ~/server/redis/client
  logging.mock.ts  ~/server/logging/client
  index.ts         resetSharedMocks()

src/__tests__/setup.ts gains:

import { dbMock } from './mocks/db.mock';
import { resetSharedMocks } from './mocks';

vi.mock('~/server/db/client', () => dbMock);
resetSharedMocks();

🔴 Spread the package, never importOriginal of a shim

~/server/db/client and ~/server/redis/client are shims: they re-export their package wholesale and construct real clients at module scope. Registering them with an importOriginal spread evaluates the shim, which forces real Prisma/Redis construction into every test file — where before, only files without their own db mock paid it.

That failed in the worst available shape. A file whose own @prisma/client mock omitted PrismaClient died during module evaluation, so it collected zero tests: the failure count stayed at 0 and the run read as green.

The package re-exports are all the spread was ever protecting, so spread the package:

vi.mock('~/server/db/client', async () => ({
  ...(await import('@civitai/db/client')),
  dbRead: dbMock.dbRead,
  dbWrite: dbMock.dbWrite,
}));

Same exports, nothing constructed. This removes the precondition rather than enumerating the files that trip it — a file no longer needs its own db mock to be protected. Pinned by a test asserting neither shim's globalThis client cache is populated, since an absent global is direct evidence the module body never ran.

The primitive: a hybrid callable proxy

Prisma's surface is roughly 60 models times 15 methods; Redis has a few hundred commands. Hand-listing either is how the current mocks got partial in the first place. Instead one primitive covers all three modules: a node that is simultaneously a vi.fn() and a proxy that vivifies child nodes on property access.

• Property access that resolves to a real vi.fn member (mockResolvedValue, mock, mockReset, …) hits the function.
• Any other string property returns a cached child node — same object every time, so dbMock.dbRead.image.findMany is one stable identity forever.
• then, catch, finally and unknown symbols return undefined, so a node is never mistaken for a thenable and awaiting one cannot hang.

This gives dbRead.image.findMany(...), dbRead.$queryRaw(...) and redis.packed.get(...) without enumerating anything.

Known collision: a model or command literally named after a vi.fn member (name, call, bind, length, mock*) resolves to the function's own member instead of a child node. None of the current Prisma models or Redis wrappers hit this. The escape hatch is mockNode('dbRead.call.findMany'), which addresses a node by path and bypasses property lookup.

Reset

resetSharedMocks() calls mockReset() on every vivified node, then re-applies that node's registered default. It runs once per test file, from the setup file, before the test module is imported. Defaults exist so that an unmigrated code path reaching an undeclared method gets a plausible empty answer rather than undefined.map is not a function:

path shape	default
findMany, findManyAndCount	[]
findUnique, findFirst	null
count	0
$queryRaw, $queryRawUnsafe	[]
$executeRaw, $executeRawUnsafe	0
$transaction	runs the callback with dbWrite, or Promise.all for an array
everything else	undefined

Defaults are deliberately conservative. A test that cares about a return value must say so; the defaults only stop an unrelated code path from throwing.

What a call site looks like

Before (src/__tests__/pages/api/download/vault-blocklist.test.ts):

const mockFindUnique = vi.hoisted(() => vi.fn());
const mockVaultItemFindUnique = vi.hoisted(() => vi.fn());

vi.mock('~/server/db/client', () => ({
  dbRead: {
    keyValue: { findUnique: mockFindUnique },
    vaultItem: { findUnique: mockVaultItemFindUnique },
    modelVersion: { findUnique: mockModelVersionFindUnique },
  },
  dbWrite: { keyValue: { findUnique: mockFindUnique } },
}));

beforeEach(() => {
  mockFindUnique.mockReset();
  mockFindUnique.mockResolvedValue({ value: ['1'] });
});

After:

import { dbMock } from '~/__tests__/mocks/db.mock';

const mockFindUnique = dbMock.dbRead.keyValue.findUnique;
const mockVaultItemFindUnique = dbMock.dbRead.vaultItem.findUnique;

beforeEach(() => {
  mockFindUnique.mockResolvedValue({ value: ['1'] });
});

The vi.mock call and the vi.hoisted wrappers both go away. The local consts stay, so the body of the test file needs no edits at all — which is what makes the migration mechanical. The mockReset() in beforeEach also goes away, because the setup file already reset everything before this file was imported.

One real change of meaning: dbRead and dbWrite were often aliased to the same mock object, so a dbWrite call satisfied a dbRead assertion. They are now distinct. A file that relied on the aliasing has to name the client it actually exercises — that is a fixture collision being corrected, not a regression.

Migration is all-or-nothing per specifier

A single unmigrated file that still calls vi.mock('~/server/db/client', …) re-poisons every consumer module in its worker, and which files it takes down depends on sharding. So a specifier is only fixed when its last call site is converted — 441 files for ~/server/db/client, 200 for logging, 198 for redis.

That is enforced, not remembered: no-direct-shared-module-mock fails when a test file mocks a canonical specifier, carrying an explicit allowlist of the files not yet converted. The allowlist only ever shrinks, and its length is the migration's progress bar.

A green pilot run therefore proves the mechanism, not the suite. Converted files are run together, without unconverted ones, at two worker counts. Mixing them measures the residual, which is a different question — measured: the same 174 files went from 404 failures to 290 with 83 of them migrated, and the fully-migrated 83 alone came in at 11.

Measured result

83 files, migrated for all three specifiers, against the same files isolated at 4 workers:

run	files	tests	failed	zero-test files	wall
isolated, 4 workers (control)	83	856	0	0	62.6s
--no-isolate, 4 workers	83	846	11	1	22.9s
--no-isolate, 12 workers	83	853	1	1	18.8s

The residual failures are not mock shape. They are production modules holding state at module scope — eventloop-longtask.ts registers its metrics into a real prom registry on import, so under --no-isolate the first file to import it in a worker takes the registration and every later file sees an empty registry. That belongs with no-module-scope-cache. The one zero-collection file dies on ~/server/services/image.service, which is out of scope here.

Registering the canonical mocks is inert under isolation: the same files isolated at 4 workers report 0 failures.

Measuring

The yardstick is node scripts/test-perf/bench.mjs --label <name> --workers <n>, a fixed 90-file subset, with --no-isolate passed through. Every claim needs two worker counts because the failing set is order-dependent — 1574 failures at 8 workers against 1161 at 31, same code, same flag.

Do not read a suite result through | tail or | grep; that reports the pipe's exit code. Redirect to a file and read the file.

🔴 Never run two vitest runs in one worktree — the second one's result is not evidence.

A targeted run and a full suite sharing a checkout produce assertion failures in the targeted run that the code does not explain. Measured on one frozen SHA, five files, identical bytes on disk for every run: 10/10 green with nothing else in the tree, 1/5 red with a full suite running concurrently in the same worktree, and earlier the same day 5/10 and 4/5 red under that condition. A control rules out load as the cause: 6/6 green under 24 spinning CPU processes with no rival vitest.

The failure reads as a test bug, which is what makes it expensive. The symptom is that a dbRead lookup the test configured returns the canonical mock's empty default instead — so the code under test resolves nothing, a guard finds no owner, and an assertion that something is refused fails as "resolved instead of rejecting". It is not a consumed …Once queue: it reproduces on plain mockResolvedValue too. The mechanism is unproven; the shared node_modules/.vite cache is the obvious suspect and has not been demonstrated. CI runs one suite at a time, so this is a local hazard.

Two rules follow. Anyone's flakiness number measured in a shared worktree — including a green one — says nothing, so re-measure alone before believing it. And before attributing a red run to the diff, check whether anything else was running in that checkout: this condition sent two separate diagnoses the wrong way in one afternoon, once at a test file and once at run contention, and a tree that moves underneath a probe will do the same.

Not covered here

~/server/services/buzz.service (98 sites) and ~/server/services/image.service (88) lead the next tier. The infra clients above are uniform enough for one auto-vivifying primitive; a service module has a hand-written surface where the right canonical mock is a hand-written stub, so it is a different piece of work. (~/env/server has since been done — it needed a value table rather than a call surface; see the two-bucket rule in the migration doc.)

🔴 The PENDING list in guarded-specifiers.ts is a floor, not an inventory. It was assembled from the most-mocked specifiers in a static scan, and specifiers keep arriving from the other direction — as failures in a --no-isolate run of files already residual-clean for everything listed. ~/server/flipt/client arrived that way. ~/server/middleware/block-scope.middleware (27 files) is a live candidate found the same way, and is deliberately not added without someone verifying the mechanism first.

So the remaining work is discovered rather than known, and any estimate built on the current count is a lower bound. The method for finding the next one is the same each time: migrate a set clean for everything listed, run it under --no-isolate, and read what still fails.

The list was built from repo-wide mock counts, and that is why it misses what it misses. Two directories fail for different reasons:

	routers (27 files)	services (195 files)
distinct specifiers needed	33	101
not on the PENDING list	28	88
median shared per pair	10	1
pairs sharing nothing	0 (0%)	6,149 (32.5%)

Services is a fan-in problem — a few specifiers reached by very many files — and the PENDING list covers it well: every high-fan-in specifier is on it, and the first undiscovered one ranks 12th. Routers is a clique: its colliding specifiers are locally dense and repo-wide rare, so a count built from repo-wide frequency cannot see them.

So to find the missing ones, build the per-pair shared-specifier graph for the directory you are working on rather than sorting a repo-wide count. The two directories need different searches, not more of the same one.

💡 Untested, and the highest-value idea nobody has tried: a third of services pairs share no specifier at all. A worker assignment that groups non-overlapping files could make much of that suite clean under isolate: false without canonicalising anything. It is statically evaluable from the same graph, before anyone writes code.

The unit of work is the CLUSTER of specifiers a set of files shares, not one specifier. Canonicalising block-scope.middleware took one pair from 13 failures to 7, and the remaining 7 were a different class — so that pair shares at least one more poisoning specifier that nobody has named yet. 13 → 7 is the dangerous shape: it reads as progress and is not completion. A set is done when its --no-isolate failures reach zero, not when the specifier you were working on stops appearing.