mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
05dcfe92dd
Addresses the v3.9.1–v3.9.4 init regression series by building the verification layer that #401's post-mortem identified as structurally missing. The load-bearing changes are the real-public-repo corpus, the --json contract, and the restructured error handling in phase 06 that stops lying about success when a non-critical phase fails. What landed ----------- * `aqe init --json` — structured InitResult output with schema version 1, stricter exit code semantics (exit 1 on any step.status='error', not just on critical failures). Stable contract for the gate and other CI consumers. See InitJsonOutput in src/cli/handlers/init-handler.ts. * Phase 06 no longer swallows errors as `{status:'skipped',entries:0}`. Exceptions now propagate to BasePhase.execute() and surface as real step errors in result.steps[]. Fixed a resource leak where kgService and memory handles were leaked on the old throw path — now always released via try/finally. * `tests/fixtures/init-corpus/` — 4 pinned real public repo fixtures: - tiny-ts (developit/mitt@b240473) - mid-ts (sindresorhus/p-queue@55d306b) - multi-lang-real (ruvnet/RuView@2a05378 — contains the exact 28,745 byte / 776 line ruview_live.py that triggered the v3.9.4 hang, sha256-pinned to detect content drift) - self-dogfood (agentic-qe HEAD) * 22 gate assertions per fixture (A1–A22) including per-step JSON error inspection (A10), snapshot-based KG threshold with tolerance (A12), skills/agents/MCP/CLAUDE.md/workers/config.yaml presence (A13–A20), subthreshold stall detection (A21), and a second init pass that exercises the delta-scan path of phase 06 — the actual surface that hung in v3.9.1 ruview, never reached on a first init (A22). * `.github/workflows/npm-publish.yml` — new pre-publish-gate job that runs the corpus against the freshly-built tarball before publish; new tests-on-tag-sha job that re-runs fast unit tests on release events (replaces a buggy gh-api conditional caught by devils-advocate review). * `.github/workflows/post-publish-canary.yml` — fires on real publishes only (not workflow_dispatch dry runs), polls npm CDN via `npm install --dry-run` (actual CDN edge, not just registry metadata), runs the corpus against the published package, opens P0 issue on failure via heredoc-based body. * Watchdog effectiveness unit test (tests/unit/init/phases/code-intelligence-watchdog.test.ts) — 4 cases proving the async-stall branches of runBoundedScan work and documenting the sync-block limitation that requires AQE_SKIP_CODE_INDEX or (future) worker isolation. * `.github/PULL_REQUEST_TEMPLATE.md` — single load-bearing checkbox: every PR failure mode must have a test or tracking issue. Honor-system today; CI enforcement tracked in #408. * AQE_SKIP_CODE_INDEX promoted from "temporary escape hatch" to permanent supported flag in docs, CLI help, and source comments. Verification ------------ * `npm run typecheck` — clean * `npx eslint` on all touched files — clean (plus fixed two pre-existing unused-var warnings in init-handler.ts) * `aqe security --sast` on all touched source + workflows — 0 findings * `tests/unit/init/` — 377/377 pass * `tests/unit/init/phases/code-intelligence-watchdog.test.ts` — 4/4 pass * Local gate against `npm pack`'d tarball — 4/4 fixtures pass all 22 assertions including double-init delta path and sha256 regression marker Failure paths independently verified (F1–F5 injection tests): * F1 wrong sha256 → A3 fires, exit 2, actionable diagnostic * F2 absurd expectedKgEntries → A12 fires, exit 2, shows actual vs expected * F3 expectedElapsedSec=0 → A21 fires on slow fixture, A22 fires on double-init — same `(( elapsed > soft_cap ))` math proven in both paths * F4 injected throw in runCodeIntelligenceScan → reproduces the exact v3.9.3 failure shape (`success:true` at orchestrator level, phase 06 step.status='error'). A7 catches via strict exit code, A10 catches as defense-in-depth via jq inspection. Both layers verified. * F5 expectedSkillsInstalled=999 → A13 fires with OR-logic showing json=84, disk=84 vs expected 999 Deferred with tracking issues ----------------------------- * Phase 06 worker_threads isolation — #407 * PR template CI enforcement — #408 * VERIFICATION.md + release-notes matrix — #409 * Chaos workflow for pathological inputs — #410 * Mirror init-corpus tarballs to GitHub Releases — #411 What is NOT verified -------------------- * The CI workflows have never actually executed in a real Actions run. YAML syntactically validated via js-yaml, job dependency graph traced, but the first real execution is the first push of this branch. * The canary on a real publish is unexecuted. First execution is the first real release after merge. Refs: #401, #407, #408, #409, #410, #411 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
268 lines
11 KiB
TypeScript
268 lines
11 KiB
TypeScript
/**
|
||
* Test: CodeIntelligencePhase.runBoundedScan watchdog effectiveness
|
||
*
|
||
* This test exists because of issue #401. The v3.9.3 release shipped a
|
||
* `Promise.race + setTimeout` watchdog for phase 06 that was marketed as
|
||
* "init cannot hang" but had never been tested against a stalling stub.
|
||
* The post-mortem (#401 Part 2 item 3) said: "This single test would have
|
||
* caught the v3.9.3 bug before shipping."
|
||
*
|
||
* Scope of what this test PROVES:
|
||
* 1. Per-file timeout fires when `kgService.index()` returns a Promise
|
||
* that never resolves (the async-stall case).
|
||
* 2. Per-file errors are caught and the loop continues to the next file.
|
||
* 3. Phase-level cap kicks in between files when individual files
|
||
* collectively exceed PHASE_TIMEOUT_MS.
|
||
*
|
||
* Scope of what this test does NOT prove and CANNOT prove:
|
||
*
|
||
* The original v3.9.3 deadlock was a SYNCHRONOUS native-code stall
|
||
* inside `@ruvector/router`'s NAPI HNSW path (a futex_wait inside Rust
|
||
* tokio threads, with a NULL timeout). When the Node event loop is
|
||
* blocked inside synchronous native code, NO JavaScript timer can fire
|
||
* — including the very timer this test is exercising. Writing a unit
|
||
* test that proves the watchdog catches sync blocks would require the
|
||
* test to itself execute synchronous-blocking code on the main thread,
|
||
* which would freeze the test runner.
|
||
*
|
||
* This is not a defect of the test — it is a fundamental property of
|
||
* the single-threaded watchdog design. Sync-block protection is
|
||
* delegated to the AQE_SKIP_CODE_INDEX escape hatch (which the user
|
||
* sets after a hang) and to the release-gate corpus at
|
||
* tests/fixtures/init-corpus/ (which exercises real-world file shapes
|
||
* in a real install before they reach users).
|
||
*
|
||
* Future work: phase 06 worker_threads isolation would enable a real
|
||
* sync-block test, because the test could use Atomics.wait inside the
|
||
* worker while the parent's timer kept running. That work is tracked
|
||
* in a follow-up issue and was deferred per #401's accepted scope —
|
||
* the dependency that caused the original deadlock was replaced with
|
||
* `hnswlib-node` in v3.9.6 (ADR-090) and the specific bug class is no
|
||
* longer reproducible.
|
||
*/
|
||
|
||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||
import { CodeIntelligencePhase } from '../../../../src/init/phases/06-code-intelligence.js';
|
||
import type { InitContext } from '../../../../src/init/phases/phase-interface.js';
|
||
|
||
// Constants from src/init/phases/06-code-intelligence.ts. Must stay in
|
||
// sync — these are the load-bearing values the test exercises.
|
||
const PER_FILE_TIMEOUT_MS = 30_000;
|
||
const PHASE_TIMEOUT_MS = 180_000;
|
||
|
||
function createMockContext(overrides: Partial<InitContext> = {}): InitContext {
|
||
return {
|
||
projectRoot: '/tmp/test-project',
|
||
options: {},
|
||
config: {},
|
||
enhancements: { claudeFlow: false, ruvector: false },
|
||
results: new Map(),
|
||
services: {
|
||
log: vi.fn(),
|
||
warn: vi.fn(),
|
||
error: vi.fn(),
|
||
},
|
||
...overrides,
|
||
};
|
||
}
|
||
|
||
interface StubKgService {
|
||
// Match the surface area runBoundedScan touches
|
||
clear: () => Promise<void>;
|
||
index: (args: { paths: string[]; incremental?: boolean; includeTests?: boolean }) => Promise<{
|
||
success: boolean;
|
||
value: { nodesCreated: number; edgesCreated: number };
|
||
}>;
|
||
}
|
||
|
||
/**
|
||
* Build a stub KG service whose `index()` behavior is controlled by the
|
||
* test. We pass it via `as any` because runBoundedScan is private and
|
||
* typed against the real KnowledgeGraphService — for a unit test of the
|
||
* watchdog logic alone, we only need the methods runBoundedScan calls.
|
||
*/
|
||
function buildStub(handlers: {
|
||
index: StubKgService['index'];
|
||
clear?: StubKgService['clear'];
|
||
}): StubKgService {
|
||
return {
|
||
clear: handlers.clear ?? (async () => {}),
|
||
index: handlers.index,
|
||
};
|
||
}
|
||
|
||
describe('CodeIntelligencePhase.runBoundedScan watchdog (issue #401)', () => {
|
||
let phase: CodeIntelligencePhase;
|
||
|
||
beforeEach(() => {
|
||
phase = new CodeIntelligencePhase();
|
||
vi.useFakeTimers();
|
||
});
|
||
|
||
afterEach(() => {
|
||
vi.useRealTimers();
|
||
});
|
||
|
||
it('per-file timeout fires when index() never resolves, and the loop continues', async () => {
|
||
// Arrange: 2 files. The first stalls forever (returns a Promise that
|
||
// never resolves). The second succeeds normally. The watchdog must
|
||
// fire on file 1 and we must still see file 2 attempted.
|
||
const indexCalls: string[] = [];
|
||
const stubKg = buildStub({
|
||
index: (args) => {
|
||
indexCalls.push(args.paths[0]);
|
||
if (args.paths[0] === '/tmp/test-project/stalls.ts') {
|
||
// Never resolves — simulates async stall in native code that
|
||
// does at least yield to the event loop.
|
||
return new Promise(() => {});
|
||
}
|
||
return Promise.resolve({
|
||
success: true,
|
||
value: { nodesCreated: 3, edgesCreated: 2 },
|
||
});
|
||
},
|
||
});
|
||
const ctx = createMockContext();
|
||
const files = ['/tmp/test-project/stalls.ts', '/tmp/test-project/ok.ts'];
|
||
|
||
// Act: drive runBoundedScan via the private accessor
|
||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||
const promise = (phase as any).runBoundedScan(stubKg, files, false, ctx);
|
||
|
||
// Advance fake clock past the per-file timeout. The withTimeout
|
||
// helper schedules its setTimeout for PER_FILE_TIMEOUT_MS — once
|
||
// we cross that, the watchdog rejects the inner promise with the
|
||
// AQE_PER_FILE_TIMEOUT error and runBoundedScan moves on.
|
||
await vi.advanceTimersByTimeAsync(PER_FILE_TIMEOUT_MS + 1_000);
|
||
|
||
const result = await promise;
|
||
|
||
// Assert: status is 'indexed' (the loop completed even though one
|
||
// file timed out — that's the whole point of per-file isolation),
|
||
// the second file got attempted, and the warn log mentions the
|
||
// timeout.
|
||
expect(result.status).toBe('indexed');
|
||
expect(indexCalls).toEqual([
|
||
'/tmp/test-project/stalls.ts',
|
||
'/tmp/test-project/ok.ts',
|
||
]);
|
||
expect(ctx.services.warn).toHaveBeenCalledWith(
|
||
expect.stringContaining('Skipped'),
|
||
);
|
||
expect(ctx.services.warn).toHaveBeenCalledWith(
|
||
expect.stringContaining('stalls.ts'),
|
||
);
|
||
});
|
||
|
||
it('per-file errors that are NOT timeouts are caught and the loop continues', async () => {
|
||
// Arrange: file 1 throws synchronously from the inner promise
|
||
// (representing a parser error or similar). The watchdog must
|
||
// catch it via the catch in runBoundedScan and continue.
|
||
const stubKg = buildStub({
|
||
index: (args) => {
|
||
if (args.paths[0] === '/tmp/test-project/broken.ts') {
|
||
return Promise.reject(new Error('SyntaxError: unexpected token'));
|
||
}
|
||
return Promise.resolve({
|
||
success: true,
|
||
value: { nodesCreated: 1, edgesCreated: 0 },
|
||
});
|
||
},
|
||
});
|
||
const ctx = createMockContext();
|
||
const files = ['/tmp/test-project/broken.ts', '/tmp/test-project/ok.ts'];
|
||
|
||
// Act
|
||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||
const promise = (phase as any).runBoundedScan(stubKg, files, false, ctx);
|
||
await vi.runAllTimersAsync();
|
||
const result = await promise;
|
||
|
||
// Assert: status indexed, warn fired with "Failed to index" (not
|
||
// "Skipped" — the message branch is different), one entry from the
|
||
// good file.
|
||
expect(result.status).toBe('indexed');
|
||
expect(result.entries).toBe(1); // 1 node + 0 edges from ok.ts
|
||
expect(ctx.services.warn).toHaveBeenCalledWith(
|
||
expect.stringContaining('Failed to index'),
|
||
);
|
||
expect(ctx.services.warn).toHaveBeenCalledWith(
|
||
expect.stringContaining('broken.ts'),
|
||
);
|
||
});
|
||
|
||
it('phase-level cap fires when cumulative file processing exceeds PHASE_TIMEOUT_MS', async () => {
|
||
// Arrange: 10 files where each "successfully" indexes in 25 seconds
|
||
// (under the 30s PER_FILE_TIMEOUT_MS so per-file timeout never fires).
|
||
// Cumulative: 25s × 8 = 200s > 180s PHASE_TIMEOUT_MS, so the
|
||
// between-files cap should fire after ~7-8 successful files.
|
||
//
|
||
// The math is deliberate: perFileMs MUST be < PER_FILE_TIMEOUT_MS
|
||
// so individual files complete normally; otherwise per-file timeout
|
||
// fires first and entries stays at 0 (invalidating the partial-
|
||
// progress invariant we're trying to assert).
|
||
const perFileMs = 25_000;
|
||
let processedCount = 0;
|
||
const stubKg = buildStub({
|
||
index: () => {
|
||
processedCount++;
|
||
return new Promise((resolve) => {
|
||
setTimeout(() => {
|
||
resolve({
|
||
success: true,
|
||
value: { nodesCreated: 1, edgesCreated: 1 },
|
||
});
|
||
}, perFileMs);
|
||
});
|
||
},
|
||
});
|
||
const ctx = createMockContext();
|
||
const files = Array.from({ length: 10 }, (_, i) => `/tmp/test-project/file${i}.ts`);
|
||
|
||
// Act. We can't drive the loop with one big advance because each
|
||
// iteration awaits an inner setTimeout — we need to advance step by
|
||
// step so each iteration's timer can resolve before the next runs.
|
||
// vi.advanceTimersByTimeAsync drains microtasks between timer
|
||
// firings, so a single large advance does work in practice; the
|
||
// safer pattern is to advance in perFileMs chunks.
|
||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||
const promise = (phase as any).runBoundedScan(stubKg, files, false, ctx);
|
||
for (let i = 0; i < 10; i++) {
|
||
await vi.advanceTimersByTimeAsync(perFileMs + 100);
|
||
}
|
||
const result = await promise;
|
||
|
||
// Assert: phase-level cap fired
|
||
expect(result.status).toBe('timeout');
|
||
expect(result.timeoutFile).toBeDefined();
|
||
// Bailed before all 10 files (cap fires around file 8)
|
||
expect(processedCount).toBeLessThan(10);
|
||
expect(processedCount).toBeGreaterThanOrEqual(7);
|
||
// Partial entries preserved — this is the documented invariant
|
||
// ("on timeout we return the entries indexed so far, not zero")
|
||
expect(result.entries).toBeGreaterThan(0);
|
||
});
|
||
|
||
it('documents the test scope: sync blocks are NOT testable from main thread', () => {
|
||
// This is a contract test. It exists to make the limitation of
|
||
// the watchdog explicit and to fail loudly if anyone "fixes" it
|
||
// by trying to make the watchdog catch sync blocks (which is
|
||
// architecturally impossible without worker isolation). See the
|
||
// file header comment for the full reasoning.
|
||
//
|
||
// If a future contributor moves phase 06 into a worker_threads
|
||
// worker, this test should be updated to add a 4th case that
|
||
// uses Atomics.wait inside the worker to genuinely block, and
|
||
// asserts the parent's worker.terminate() fires within the
|
||
// configured timeout.
|
||
|
||
// The watchdog uses a JS-side setTimeout. setTimeout fires on
|
||
// the same thread as the indexer.
|
||
expect(typeof setTimeout).toBe('function');
|
||
// The escape hatch is the user-facing protection for sync blocks.
|
||
expect(process.env.AQE_SKIP_CODE_INDEX).toBeUndefined(); // not set in test env
|
||
// The release-gate corpus is the load-bearing prevention layer.
|
||
// (No assertion — this is the contract documentation.)
|
||
});
|
||
});
|