Files
proffesor-for-testing__agen…/tests/unit/init/phases/code-intelligence-watchdog.test.ts
Dragan Spiridonov 05dcfe92dd fix(init): release-gate corpus + --json contract (#401)
Addresses the v3.9.1–v3.9.4 init regression series by building the
verification layer that #401's post-mortem identified as structurally
missing. The load-bearing changes are the real-public-repo corpus, the
--json contract, and the restructured error handling in phase 06 that
stops lying about success when a non-critical phase fails.

What landed
-----------
* `aqe init --json` — structured InitResult output with schema version 1,
  stricter exit code semantics (exit 1 on any step.status='error', not
  just on critical failures). Stable contract for the gate and other CI
  consumers. See InitJsonOutput in src/cli/handlers/init-handler.ts.

* Phase 06 no longer swallows errors as `{status:'skipped',entries:0}`.
  Exceptions now propagate to BasePhase.execute() and surface as real
  step errors in result.steps[]. Fixed a resource leak where kgService
  and memory handles were leaked on the old throw path — now always
  released via try/finally.

* `tests/fixtures/init-corpus/` — 4 pinned real public repo fixtures:
  - tiny-ts (developit/mitt@b240473)
  - mid-ts (sindresorhus/p-queue@55d306b)
  - multi-lang-real (ruvnet/RuView@2a05378 — contains the exact 28,745
    byte / 776 line ruview_live.py that triggered the v3.9.4 hang,
    sha256-pinned to detect content drift)
  - self-dogfood (agentic-qe HEAD)

* 22 gate assertions per fixture (A1–A22) including per-step JSON error
  inspection (A10), snapshot-based KG threshold with tolerance (A12),
  skills/agents/MCP/CLAUDE.md/workers/config.yaml presence (A13–A20),
  subthreshold stall detection (A21), and a second init pass that
  exercises the delta-scan path of phase 06 — the actual surface that
  hung in v3.9.1 ruview, never reached on a first init (A22).

* `.github/workflows/npm-publish.yml` — new pre-publish-gate job that
  runs the corpus against the freshly-built tarball before publish; new
  tests-on-tag-sha job that re-runs fast unit tests on release events
  (replaces a buggy gh-api conditional caught by devils-advocate review).

* `.github/workflows/post-publish-canary.yml` — fires on real publishes
  only (not workflow_dispatch dry runs), polls npm CDN via
  `npm install --dry-run` (actual CDN edge, not just registry metadata),
  runs the corpus against the published package, opens P0 issue on
  failure via heredoc-based body.

* Watchdog effectiveness unit test
  (tests/unit/init/phases/code-intelligence-watchdog.test.ts) — 4 cases
  proving the async-stall branches of runBoundedScan work and
  documenting the sync-block limitation that requires AQE_SKIP_CODE_INDEX
  or (future) worker isolation.

* `.github/PULL_REQUEST_TEMPLATE.md` — single load-bearing checkbox:
  every PR failure mode must have a test or tracking issue. Honor-system
  today; CI enforcement tracked in #408.

* AQE_SKIP_CODE_INDEX promoted from "temporary escape hatch" to
  permanent supported flag in docs, CLI help, and source comments.

Verification
------------
* `npm run typecheck` — clean
* `npx eslint` on all touched files — clean (plus fixed two pre-existing
  unused-var warnings in init-handler.ts)
* `aqe security --sast` on all touched source + workflows — 0 findings
* `tests/unit/init/` — 377/377 pass
* `tests/unit/init/phases/code-intelligence-watchdog.test.ts` — 4/4 pass
* Local gate against `npm pack`'d tarball — 4/4 fixtures pass all 22
  assertions including double-init delta path and sha256 regression
  marker

Failure paths independently verified (F1–F5 injection tests):
* F1 wrong sha256 → A3 fires, exit 2, actionable diagnostic
* F2 absurd expectedKgEntries → A12 fires, exit 2, shows actual vs expected
* F3 expectedElapsedSec=0 → A21 fires on slow fixture, A22 fires on
  double-init — same `(( elapsed > soft_cap ))` math proven in both paths
* F4 injected throw in runCodeIntelligenceScan → reproduces the exact
  v3.9.3 failure shape (`success:true` at orchestrator level, phase 06
  step.status='error'). A7 catches via strict exit code, A10 catches as
  defense-in-depth via jq inspection. Both layers verified.
* F5 expectedSkillsInstalled=999 → A13 fires with OR-logic showing
  json=84, disk=84 vs expected 999

Deferred with tracking issues
-----------------------------
* Phase 06 worker_threads isolation — #407
* PR template CI enforcement — #408
* VERIFICATION.md + release-notes matrix — #409
* Chaos workflow for pathological inputs — #410
* Mirror init-corpus tarballs to GitHub Releases — #411

What is NOT verified
--------------------
* The CI workflows have never actually executed in a real Actions run.
  YAML syntactically validated via js-yaml, job dependency graph traced,
  but the first real execution is the first push of this branch.
* The canary on a real publish is unexecuted. First execution is the
  first real release after merge.

Refs: #401, #407, #408, #409, #410, #411

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 16:44:27 +00:00

268 lines
11 KiB
TypeScript
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/**
* Test: CodeIntelligencePhase.runBoundedScan watchdog effectiveness
*
* This test exists because of issue #401. The v3.9.3 release shipped a
* `Promise.race + setTimeout` watchdog for phase 06 that was marketed as
* "init cannot hang" but had never been tested against a stalling stub.
* The post-mortem (#401 Part 2 item 3) said: "This single test would have
* caught the v3.9.3 bug before shipping."
*
* Scope of what this test PROVES:
* 1. Per-file timeout fires when `kgService.index()` returns a Promise
* that never resolves (the async-stall case).
* 2. Per-file errors are caught and the loop continues to the next file.
* 3. Phase-level cap kicks in between files when individual files
* collectively exceed PHASE_TIMEOUT_MS.
*
* Scope of what this test does NOT prove and CANNOT prove:
*
* The original v3.9.3 deadlock was a SYNCHRONOUS native-code stall
* inside `@ruvector/router`'s NAPI HNSW path (a futex_wait inside Rust
* tokio threads, with a NULL timeout). When the Node event loop is
* blocked inside synchronous native code, NO JavaScript timer can fire
* — including the very timer this test is exercising. Writing a unit
* test that proves the watchdog catches sync blocks would require the
* test to itself execute synchronous-blocking code on the main thread,
* which would freeze the test runner.
*
* This is not a defect of the test — it is a fundamental property of
* the single-threaded watchdog design. Sync-block protection is
* delegated to the AQE_SKIP_CODE_INDEX escape hatch (which the user
* sets after a hang) and to the release-gate corpus at
* tests/fixtures/init-corpus/ (which exercises real-world file shapes
* in a real install before they reach users).
*
* Future work: phase 06 worker_threads isolation would enable a real
* sync-block test, because the test could use Atomics.wait inside the
* worker while the parent's timer kept running. That work is tracked
* in a follow-up issue and was deferred per #401's accepted scope —
* the dependency that caused the original deadlock was replaced with
* `hnswlib-node` in v3.9.6 (ADR-090) and the specific bug class is no
* longer reproducible.
*/
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { CodeIntelligencePhase } from '../../../../src/init/phases/06-code-intelligence.js';
import type { InitContext } from '../../../../src/init/phases/phase-interface.js';
// Constants from src/init/phases/06-code-intelligence.ts. Must stay in
// sync — these are the load-bearing values the test exercises.
const PER_FILE_TIMEOUT_MS = 30_000;
const PHASE_TIMEOUT_MS = 180_000;
function createMockContext(overrides: Partial<InitContext> = {}): InitContext {
return {
projectRoot: '/tmp/test-project',
options: {},
config: {},
enhancements: { claudeFlow: false, ruvector: false },
results: new Map(),
services: {
log: vi.fn(),
warn: vi.fn(),
error: vi.fn(),
},
...overrides,
};
}
interface StubKgService {
// Match the surface area runBoundedScan touches
clear: () => Promise<void>;
index: (args: { paths: string[]; incremental?: boolean; includeTests?: boolean }) => Promise<{
success: boolean;
value: { nodesCreated: number; edgesCreated: number };
}>;
}
/**
* Build a stub KG service whose `index()` behavior is controlled by the
* test. We pass it via `as any` because runBoundedScan is private and
* typed against the real KnowledgeGraphService — for a unit test of the
* watchdog logic alone, we only need the methods runBoundedScan calls.
*/
function buildStub(handlers: {
index: StubKgService['index'];
clear?: StubKgService['clear'];
}): StubKgService {
return {
clear: handlers.clear ?? (async () => {}),
index: handlers.index,
};
}
describe('CodeIntelligencePhase.runBoundedScan watchdog (issue #401)', () => {
let phase: CodeIntelligencePhase;
beforeEach(() => {
phase = new CodeIntelligencePhase();
vi.useFakeTimers();
});
afterEach(() => {
vi.useRealTimers();
});
it('per-file timeout fires when index() never resolves, and the loop continues', async () => {
// Arrange: 2 files. The first stalls forever (returns a Promise that
// never resolves). The second succeeds normally. The watchdog must
// fire on file 1 and we must still see file 2 attempted.
const indexCalls: string[] = [];
const stubKg = buildStub({
index: (args) => {
indexCalls.push(args.paths[0]);
if (args.paths[0] === '/tmp/test-project/stalls.ts') {
// Never resolves — simulates async stall in native code that
// does at least yield to the event loop.
return new Promise(() => {});
}
return Promise.resolve({
success: true,
value: { nodesCreated: 3, edgesCreated: 2 },
});
},
});
const ctx = createMockContext();
const files = ['/tmp/test-project/stalls.ts', '/tmp/test-project/ok.ts'];
// Act: drive runBoundedScan via the private accessor
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const promise = (phase as any).runBoundedScan(stubKg, files, false, ctx);
// Advance fake clock past the per-file timeout. The withTimeout
// helper schedules its setTimeout for PER_FILE_TIMEOUT_MS — once
// we cross that, the watchdog rejects the inner promise with the
// AQE_PER_FILE_TIMEOUT error and runBoundedScan moves on.
await vi.advanceTimersByTimeAsync(PER_FILE_TIMEOUT_MS + 1_000);
const result = await promise;
// Assert: status is 'indexed' (the loop completed even though one
// file timed out — that's the whole point of per-file isolation),
// the second file got attempted, and the warn log mentions the
// timeout.
expect(result.status).toBe('indexed');
expect(indexCalls).toEqual([
'/tmp/test-project/stalls.ts',
'/tmp/test-project/ok.ts',
]);
expect(ctx.services.warn).toHaveBeenCalledWith(
expect.stringContaining('Skipped'),
);
expect(ctx.services.warn).toHaveBeenCalledWith(
expect.stringContaining('stalls.ts'),
);
});
it('per-file errors that are NOT timeouts are caught and the loop continues', async () => {
// Arrange: file 1 throws synchronously from the inner promise
// (representing a parser error or similar). The watchdog must
// catch it via the catch in runBoundedScan and continue.
const stubKg = buildStub({
index: (args) => {
if (args.paths[0] === '/tmp/test-project/broken.ts') {
return Promise.reject(new Error('SyntaxError: unexpected token'));
}
return Promise.resolve({
success: true,
value: { nodesCreated: 1, edgesCreated: 0 },
});
},
});
const ctx = createMockContext();
const files = ['/tmp/test-project/broken.ts', '/tmp/test-project/ok.ts'];
// Act
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const promise = (phase as any).runBoundedScan(stubKg, files, false, ctx);
await vi.runAllTimersAsync();
const result = await promise;
// Assert: status indexed, warn fired with "Failed to index" (not
// "Skipped" — the message branch is different), one entry from the
// good file.
expect(result.status).toBe('indexed');
expect(result.entries).toBe(1); // 1 node + 0 edges from ok.ts
expect(ctx.services.warn).toHaveBeenCalledWith(
expect.stringContaining('Failed to index'),
);
expect(ctx.services.warn).toHaveBeenCalledWith(
expect.stringContaining('broken.ts'),
);
});
it('phase-level cap fires when cumulative file processing exceeds PHASE_TIMEOUT_MS', async () => {
// Arrange: 10 files where each "successfully" indexes in 25 seconds
// (under the 30s PER_FILE_TIMEOUT_MS so per-file timeout never fires).
// Cumulative: 25s × 8 = 200s > 180s PHASE_TIMEOUT_MS, so the
// between-files cap should fire after ~7-8 successful files.
//
// The math is deliberate: perFileMs MUST be < PER_FILE_TIMEOUT_MS
// so individual files complete normally; otherwise per-file timeout
// fires first and entries stays at 0 (invalidating the partial-
// progress invariant we're trying to assert).
const perFileMs = 25_000;
let processedCount = 0;
const stubKg = buildStub({
index: () => {
processedCount++;
return new Promise((resolve) => {
setTimeout(() => {
resolve({
success: true,
value: { nodesCreated: 1, edgesCreated: 1 },
});
}, perFileMs);
});
},
});
const ctx = createMockContext();
const files = Array.from({ length: 10 }, (_, i) => `/tmp/test-project/file${i}.ts`);
// Act. We can't drive the loop with one big advance because each
// iteration awaits an inner setTimeout — we need to advance step by
// step so each iteration's timer can resolve before the next runs.
// vi.advanceTimersByTimeAsync drains microtasks between timer
// firings, so a single large advance does work in practice; the
// safer pattern is to advance in perFileMs chunks.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const promise = (phase as any).runBoundedScan(stubKg, files, false, ctx);
for (let i = 0; i < 10; i++) {
await vi.advanceTimersByTimeAsync(perFileMs + 100);
}
const result = await promise;
// Assert: phase-level cap fired
expect(result.status).toBe('timeout');
expect(result.timeoutFile).toBeDefined();
// Bailed before all 10 files (cap fires around file 8)
expect(processedCount).toBeLessThan(10);
expect(processedCount).toBeGreaterThanOrEqual(7);
// Partial entries preserved — this is the documented invariant
// ("on timeout we return the entries indexed so far, not zero")
expect(result.entries).toBeGreaterThan(0);
});
it('documents the test scope: sync blocks are NOT testable from main thread', () => {
// This is a contract test. It exists to make the limitation of
// the watchdog explicit and to fail loudly if anyone "fixes" it
// by trying to make the watchdog catch sync blocks (which is
// architecturally impossible without worker isolation). See the
// file header comment for the full reasoning.
//
// If a future contributor moves phase 06 into a worker_threads
// worker, this test should be updated to add a 4th case that
// uses Atomics.wait inside the worker to genuinely block, and
// asserts the parent's worker.terminate() fires within the
// configured timeout.
// The watchdog uses a JS-side setTimeout. setTimeout fires on
// the same thread as the indexer.
expect(typeof setTimeout).toBe('function');
// The escape hatch is the user-facing protection for sync blocks.
expect(process.env.AQE_SKIP_CODE_INDEX).toBeUndefined(); // not set in test env
// The release-gate corpus is the load-bearing prevention layer.
// (No assertion — this is the contract documentation.)
});
});