Files
civitai__civitai/scripts/__tests__/dev-server-env-modes.test.ts
Justin Maier 6f54a21085 feat(dev-server): per-service dev/prod env modes on start (#3954)
* feat(dev-server): per-service dev/prod env modes on start

A session picks one .env and now applies a per-service overlay on top of it:
`start --prod buzz` moves Buzz to production and leaves everything else on dev.
Groups come from a gitignored env-modes.local (db, buzz, search, signals,
redis today), so adding a service is an edit to that file rather than to code.

Every group defaults to dev. DEVSERVER_PROD_GROUPS in the skill .env moves a
default when a dev service is unreliable, and a flag beats both. The flag
applies to that start only, including when it takes over a dead session, so a
prod choice never leaks into the next bare start. Asking for different modes
while a session is already running that worktree is refused rather than
answered with the running session.

The orchestrator, payments, S3, ClickHouse, the notifications DB, the feeds
proxy and OpenSearch have no dev counterpart at all, so the summary prints them
after every mode line: dev mode is not a claim that nothing here is production.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(dev-server): close the ways an env mode could lie about itself

Review found four paths where a mistake resolved quietly to the wrong
environment, three of them ending on production while the session reported dev.

A malformed section header (`[db prod]`, a missing bracket) matched neither the
section nor the KEY=VALUE pattern, and the parser left the previous section open
— so every following key landed under it, filing the production DATABASE_URL as
`[db.dev]`. Any parse error now closes the open section, and the daemon refuses
to start at all when the definitions file did not parse cleanly, rather than
logging a warning nobody reads and carrying on.

`all` was expanded before group names were validated, so `--prod all,typo`
discarded the typo and `--prod all` against a machine with no env-modes.local
reported success having moved nothing. Names are checked first, `all` errors
when it matches no groups, and it now selects only groups that have that mode
instead of failing the whole start over one dev-only group. `--prod=` with an
empty value errors like the spaced form already did. A DEVSERVER_PROD_GROUPS
entry that matches no group produces a note instead of silently leaving the
service on dev — the failure an operator setting it is specifically trying to
avoid.

The busy-session refusal compared flag text, so it refused `--prod all` against
a session started with every group named, and accepted a bare start against a
session whose modes a bare start would no longer produce. It compares resolved
modes now. That refusal also broke the dashboard: console.mjs sends no modes, so
a 409 sent it down a fallback that picked whichever session was first in the map
— another worktree, another branch, its logs, silently. It watches the refused
session when one is named and otherwise looks only at this worktree.

The example understated each group's key set: db without DATABASE_IS_PROD (which
gates the S3 delete paths) or DATABASE_REPLICA_LONG_URL left a session writing
to production and reading long queries from dev, and redis without REDIS_CLUSTER
would point a single-node client at a cluster.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(dev-server): stop the mode guard reporting a stale or partial answer

Second review round, on the hardening itself.

The 409 mismatch check read `session.modes`, which only start() writes. On the
takeover path the request's modes were stored and restart() awaited, so for the
seconds spent stopping and reclaiming the port the session still described the
run being torn down — long enough for another agent's bare start to match it and
be handed 200 for a session coming up on production. The resolved modes are
stamped before the await now, and cleared at the top of start() so a session
that errors out stops reporting an env it never applied. The endpoint also
resolved against a cached skill config while start() reloaded it, so an edited
DEVSERVER_PROD_GROUPS made the two disagree; it reloads first.

`--prod all --dev search` — everything on prod except one — threw a conflict,
because the conflict check ran after `all` expanded. Expansion now skips groups
named on the other flag and conflicts are judged on explicit names only, so the
exception form works and `--prod all --dev all` is rejected on its own terms. A
group named beside `all` that has no section for that mode is an error rather
than a silent fallback to dev.

Parser errors no longer echo the offending line: they reach an HTTP 400 body and
a log buffer any agent can read, and a stray connection-string line splits on
its first `=` with the password on the left.

The example understated three more groups: search without METRICS_SEARCH_* is a
second Meilisearch the app writes to, redis without REDIS_SYS_SENTINELS leaves
the system client on production (with sentinels set, the URL supplies only the
password), and db without DATAPACKET_DATABASE_RO_URL keeps a read pool on
production Postgres. The dashboard counts a `base` group as production, since
base means the .env value stood and this .env is production.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(dev-server): pin the env-mode rules, and stop the guard overreaching

Third review round. The resolver was the most intricate code on the branch and
had no test file — the checks proving it lived in a scratchpad, which protects
nothing after this session ends. `scripts/__tests__/dev-server-env-modes.test.ts`
now covers the resolution rules, the overlay, the parser and the session
comparison, beside the queue's tests for the same reason (the daemon is not in
the app's module graph). Both mutations fail legibly: reverting the parser fix
prints `expected 'PROD-host' to be 'dev-host'`, and reverting the all/dev
exception prints the wrong mode map.

Fixes with it:

The session comparison compared whole maps, so ADDING a group to env-modes.local
made every already-running session disagree with every new bare start — `start`,
which is meant to be idempotent, would 409 everywhere until each session was
restarted. It compares the groups both sides resolved. A group only one side
knows about is a definitions edit, not two agents wanting different environments.

`DEVSERVER_PROD_GROUPS` is pinned on the session when it is created rather than
re-read on every start, because start() also runs unattended — a branch switch,
a crash restart — and an edit to that file would otherwise move a LIVE session
onto production with nobody having asked.

A start refused over its env modes answered 201 and the CLI exited 0, so
`start && curl` proceeded as though a server had come up. It answers 5xx with the
session's own error line, on both the new-session and the reuse path.

The takeover stamp is `pendingModes`, separate from `modes`: a crashed session's
process can still be serving on the old env for the length of the port wait, so
`modes` stays true to what is running and the mismatch check reads what is coming.

`PROD_ONLY_GROUPS` is filtered by what the definitions file defines, so defining
`[s3.*]` no longer produces a summary that says `s3=dev` and `always prod: s3` on
the same line.

Documented rather than fixed: the auth hub is one shared process reading its own
apps/auth/.env, so it cannot follow a per-session db mode — a `--prod db` login
mints a token for a user id from the other database. And the build dir is keyed on
branch, not mode, so changing search/signals mode on a warm `.next` can leave the
previous NEXT_PUBLIC_* host inlined in client chunks; keying it on mode would
multiply an 8GB cache per combination.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(dev-server): close the fail-open half of the mode comparison

Fourth review round, and the worst finding was in the previous round's fix.

Comparing only the groups both sides resolved made an EMPTY request match
anything: with env-modes.local momentarily absent — an atomic-rename save, a
move, a git clean — a bare start resolved to no groups, the mismatch guard did
not fire, and the daemon handed back the running session as though it were the
dev one asked for, while it ran on production. The comparison is asymmetric now.
A group the request has and the session does not is an added section and still
matches, which is what keeps `start` idempotent; a group the SESSION has and the
request lost does not.

`auth-hub` stays in the always-prod list whatever env-modes.local says. Defining
[auth-hub.dev] cannot move the hub — it is a separate shared process reading its
own apps/auth/.env — so letting a config edit delete that warning would leave the
summary asserting something the mechanism cannot deliver.

`--prod all` now leaves a note naming each group it could not move for want of a
section, instead of being the one unhonourable request in this resolver that says
nothing. The takeover stamp is cleared in a `finally`, so a restart that throws
cannot leave a session advertising modes that will never be applied. And the
dashboard reports a refused start even when it finds a session to attach to
anyway — a malformed definitions file returns no session in the body, and
repainting a healthy dashboard over the pre-edit session read as the edit having
worked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(dev-server): close the remaining fail-open directions in the mode guard

Fifth review round, and again the findings were the mirror image of the previous
round's fixes. Each is the direction where a mistake resolved to production
while the summary said otherwise.

A session that resolved NO groups came up before env-modes.local existed and is
running the base .env. `every` over no keys is vacuously true, so once the file
appeared, the first bare start was told its dev modes held while the process was
still on the .env. An empty running side now matches only an empty request.

`--dev all` left any group with no [x.dev] section on the .env — production —
and succeeded with a note, while `--dev <group>` for the same group threw. The
safety direction should not be the one that fails open, so it throws too.
Skipping stays a note in the prod direction, where the group stays on dev.

`formatModeSummary` dropped a prod-only service from the always-prod tail
whenever it resolved to `base`. `base` means no section applied and the .env
value stands, which for those services is production — so a half-written
definition removed the warning exactly when it was most needed. Only a group
that actually moved leaves the list now.

`auth-hub` was described as unmovable but nothing enforced it: defining
[auth-hub.dev] resolved normally and wrote its keys into the MAIN app's env,
repointing the app at a hub that is not listening while deleting the warning.
Defining it is refused at load.

And a reuse that threw left the failed request's --prod set pinned to the
session, so the next unattended restart would have brought it up on production
off the back of a start that errored. The previous overrides are restored on
that path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(dev-server): a named group the session never had is a mismatch

Sixth review round, two more directions into the same guard.

Leniency about a group the running session does not know is only defensible
when nobody asked for it. `--dev search` against a session that started before
[search.*] was defined came back 200 "existing", CLI exit 0, while search was
still on the base .env — production Meilisearch. Groups named on a flag are a
hard requirement now; a bare start against a session missing a newly-defined
group is still idempotent, which was the point of the leniency.

The override restore lived only in `catch`, but start() reports a mode failure
by setting status and RETURNING rather than throwing — the exact case the 500
branch handles. So a failed `--prod db` left the session pinned to it, and the
dashboard's restart key would have brought it up on production off a start the
CLI reported as failed.

Also: the dashboard's mismatch notice was a 3-second flash, after which a
dashboard attached to a session that is not what it asked for looked identical
to a healthy one — it carries a sticky `!env` marker in the header now. And the
redis example restates REDIS_CLUSTER_NODES and REDIS_SYS_SENTINEL_PASSWORD in
both blocks, since the overlay applies onto the .env rather than onto the other
mode, and the file's own rule says every key of a service moves together.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-15 08:32:59 -06:00

250 lines
11 KiB
TypeScript

import { describe, expect, it } from 'vitest';
// Lives under scripts/ because the daemon is not part of the app's module graph — same arrangement
// as the queue's tests beside it.
const {
parseModeDefinitions,
loadModeDefinitions,
resolveSessionModes,
applyModes,
formatModeSummary,
sameResolvedModes,
} = await import('../../.claude/skills/dev-server/scripts/env-modes.mjs');
const defs = (text: string) => ({
path: 'env-modes.local',
exists: true,
...parseModeDefinitions(text),
});
const TWO_GROUPS = defs(
[
'[db.dev]',
'DATABASE_URL=postgres://dev-host/civitai',
'DATABASE_IS_PROD=false',
'[db.prod]',
'DATABASE_URL=postgres://prod-host/civitai',
'DATABASE_IS_PROD=true',
'[buzz.dev]',
'BUZZ_ENDPOINT=http://localhost:28081',
'[buzz.prod]',
'BUZZ_ENDPOINT=http://localhost:28080',
].join('\n')
);
const modesOf = (result: { modes: Record<string, { mode: string }> }) =>
Object.fromEntries(Object.entries(result.modes).map(([group, choice]) => [group, choice.mode]));
const resolve = (opts: Record<string, unknown> = {}) =>
resolveSessionModes({ definitions: TWO_GROUPS, prod: [], dev: [], ...opts });
describe('env mode resolution', () => {
it('puts every group on dev when nothing is asked for', () => {
expect(modesOf(resolve())).toEqual({ db: 'dev', buzz: 'dev' });
});
it('moves only the groups named on --prod', () => {
expect(modesOf(resolve({ prod: ['db'] }))).toEqual({ db: 'prod', buzz: 'dev' });
});
it('takes the skill default, and lets a flag beat it', () => {
expect(modesOf(resolve({ defaultProdGroups: ['buzz'] })).buzz).toBe('prod');
expect(modesOf(resolve({ dev: ['buzz'], defaultProdGroups: ['buzz'] })).buzz).toBe('dev');
});
it('reads --prod all --dev x as "all of it except x"', () => {
expect(modesOf(resolve({ prod: ['all'], dev: ['buzz'] }))).toEqual({
db: 'prod',
buzz: 'dev',
});
});
it('rejects a group asked for both ways, and all asked for both ways', () => {
expect(() => resolve({ prod: ['db'], dev: ['db'] })).toThrow(/both --dev and --prod/);
expect(() => resolve({ prod: ['all'], dev: ['all'] })).toThrow(/opposite things/);
});
it('rejects an unknown group even when all is alongside it', () => {
expect(() => resolve({ prod: ['bogus'] })).toThrow(/Unknown env group\(s\): bogus/);
expect(() => resolve({ prod: ['all', 'bogus'] })).toThrow(/Unknown env group\(s\): bogus/);
});
it('rejects all when the definitions file defines nothing', () => {
const empty = { path: 'env-modes.local', exists: false, groups: {}, errors: [] };
expect(() => resolveSessionModes({ definitions: empty, prod: ['all'], dev: [] })).toThrow(
/matches no env groups/
);
});
it('skips a group that lacks the mode under all, but not one named beside it', () => {
const withGap = defs(['[only.dev]', 'K=v', '[db.dev]', 'DATABASE_URL=x'].join('\n'));
expect(modesOf(resolveSessionModes({ definitions: withGap, prod: ['all'], dev: [] }))).toEqual({
only: 'dev',
db: 'dev',
});
expect(() =>
resolveSessionModes({ definitions: withGap, prod: ['all', 'only'], dev: [] })
).toThrow(/No \[only\.prod\] section/);
});
it('says so when DEVSERVER_PROD_GROUPS names something undefined', () => {
const { notes } = resolve({ defaultProdGroups: ['buzzz'] });
expect(notes.join(' ')).toMatch(/buzzz/);
});
});
describe('the overlay', () => {
it('applies the chosen mode and leaves everything else alone', () => {
const dev = applyModes({ OTHER: 'untouched' }, TWO_GROUPS, resolve().modes).envVars;
const prod = applyModes({}, TWO_GROUPS, resolve({ prod: ['db'] }).modes).envVars;
expect(dev.OTHER).toBe('untouched');
expect(dev.DATABASE_URL).toBe('postgres://dev-host/civitai');
expect(prod.DATABASE_URL).toBe('postgres://prod-host/civitai');
expect(prod.BUZZ_ENDPOINT).toBe('http://localhost:28081');
});
it('carries every key of the group, not just the first', () => {
expect(
applyModes({}, TWO_GROUPS, resolve({ prod: ['db'] }).modes).envVars.DATABASE_IS_PROD
).toBe('true');
});
});
describe('the definitions parser', () => {
it('closes the open section on a malformed header', () => {
// `[db prod]` matches neither pattern. Leaving the section open would file the production URL
// under [db.dev], and a bare start would run on production while reporting dev.
const parsed = parseModeDefinitions(
['[db.dev]', 'DATABASE_URL=dev-host', '[db prod]', 'DATABASE_URL=PROD-host'].join('\n')
);
expect(parsed.groups.db.dev.DATABASE_URL).toBe('dev-host');
// Two: the header itself, then the key it orphaned by closing the section.
expect(parsed.errors).toHaveLength(2);
expect(parsed.errors[0]).toMatch(/line 3/);
expect(parsed.errors[1]).toMatch(/outside any/);
});
it('refuses a group an overlay cannot actually move', async () => {
// Defining [auth-hub.*] would not move the hub — it reads its own .env — it would only repoint
// the app at a hub that is not listening, and delete the always-prod warning while doing it.
const { mkdtempSync, writeFileSync } = await import('fs');
const { tmpdir } = await import('os');
const { join } = await import('path');
const dir = mkdtempSync(join(tmpdir(), 'env-modes-'));
writeFileSync(
join(dir, 'env-modes.local'),
'[auth-hub.dev]\nNEXT_PUBLIC_AUTH_URL=http://nope\n'
);
const loaded = loadModeDefinitions(dir);
expect(loaded.groups['auth-hub']).toBeUndefined();
expect(loaded.errors.join(' ')).toMatch(/auth-hub/);
});
it('never echoes the offending line, which may carry a password', () => {
const stray = parseModeDefinitions('DATABASE_URL=postgres://user:hunter2@host/db');
const junk = parseModeDefinitions('[db.dev]\npostgres://user:hunter2@host/db\n');
expect(stray.errors[0]).not.toMatch(/hunter2/);
expect(junk.errors[0]).not.toMatch(/hunter2/);
});
});
describe('the summary and the session comparison', () => {
it('names the services that have no dev target at all', () => {
expect(formatModeSummary(resolve().modes)).toMatch(
/always prod \(no dev target\): orchestrator/
);
});
it('stops calling a group prod-only once someone defines it', () => {
const s3 = defs(
['[s3.dev]', 'S3_UPLOAD_BUCKET=dev', '[s3.prod]', 'S3_UPLOAD_BUCKET=prod'].join('\n')
);
const summary = formatModeSummary(
resolveSessionModes({ definitions: s3, prod: [], dev: [] }).modes
);
expect(summary).toMatch(/s3=dev/);
expect(summary).not.toMatch(/no dev target\):.*\bs3\b/);
});
it('treats the same resolved modes as the same session however they were spelled', () => {
expect(
sameResolvedModes(resolve({ prod: ['all'] }).modes, resolve({ prod: ['db', 'buzz'] }).modes)
).toBe(true);
expect(sameResolvedModes(resolve({ prod: ['db'] }).modes, resolve().modes)).toBe(false);
});
it('does not fail a bare start just because a new group was defined', () => {
// Adding a section to env-modes.local must not make `start` — which is meant to be idempotent —
// start refusing against every session that was already running.
const running = resolve().modes;
const afterEdit = { ...running, mongo: { mode: 'dev' } };
expect(sameResolvedModes(running, afterEdit)).toBe(true);
});
it('refuses to match a request that lost a group the session has', () => {
// The other direction is the dangerous one: env-modes.local gone (an atomic-rename save mid
// flight, a git clean) resolves the request to nothing, and an empty request matching a
// production session is exactly the answer the guard exists to prevent.
const running = resolve({ prod: ['db'] }).modes;
expect(sameResolvedModes(running, {})).toBe(false);
expect(sameResolvedModes(running, resolve({ prod: ['db'] }).modes)).toBe(true);
});
it('keeps warning about the auth hub even if someone defines it as a group', () => {
// Defining [auth-hub.*] cannot move the hub — it is a separate process reading its own .env —
// so the warning must not be deletable by a config edit.
const hub = defs(['[auth-hub.dev]', 'X=1', '[auth-hub.prod]', 'X=2'].join('\n'));
const summary = formatModeSummary(
resolveSessionModes({ definitions: hub, prod: [], dev: [] }).modes
);
expect(summary).toMatch(/always prod \(no dev target\):.*auth-hub/);
});
it('says which groups "all" could not move', () => {
const withGap = defs(['[only.dev]', 'K=v', '[db.dev]', 'D=1', '[db.prod]', 'D=2'].join('\n'));
const { notes } = resolveSessionModes({ definitions: withGap, prod: ['all'], dev: [] });
expect(notes.join(' ')).toMatch(/"all" skipped only/);
});
it('refuses --dev all when a group cannot go to dev', () => {
// Skipping is survivable in the prod direction — the group stays on dev. Skipping in the dev
// direction leaves it on the .env, which is production, so the request asking for safety would
// be the one that quietly failed to deliver it.
const prodOnly = defs(['[buzz.prod]', 'BUZZ_ENDPOINT=prod'].join('\n'));
expect(() => resolveSessionModes({ definitions: prodOnly, prod: [], dev: ['all'] })).toThrow(
/--dev all cannot move buzz/
);
});
it('refuses when a group named on a flag is one the session never had', () => {
// Leniency about an unknown group is only defensible when nobody asked for it. `--dev search`
// against a session predating [search.*] would otherwise report search=dev while it runs on the
// base .env — production Meilisearch.
const running = resolve().modes;
const requested = { ...running, search: { mode: 'dev' } };
expect(sameResolvedModes(running, requested)).toBe(true);
expect(sameResolvedModes(running, requested, ['search'])).toBe(false);
expect(sameResolvedModes(running, requested, ['db'])).toBe(true);
});
it('does not match a session that resolved nothing against one that resolved groups', () => {
// A session that came up before env-modes.local existed is running the base .env. Once the file
// appears, a bare start resolves real modes — the two are not the same session.
expect(sameResolvedModes({}, resolve().modes)).toBe(false);
expect(sameResolvedModes({}, {})).toBe(true);
});
it('keeps a half-defined prod-only service in the warning', () => {
// [s3.prod] alone resolves s3 to `base`: nothing applied, the .env value stands, and the .env
// is production. Dropping it from the tail would under-report exactly then.
const halfS3 = defs(['[s3.prod]', 'S3_UPLOAD_BUCKET=prod'].join('\n'));
const summary = formatModeSummary(
resolveSessionModes({ definitions: halfS3, prod: [], dev: [] }).modes
);
expect(summary).toMatch(/s3=base/);
expect(summary).toMatch(/always prod \(no dev target\):.*\bs3\b/);
});
});