mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
Dragan Spiridonov
4f340bc183
Closes the two remaining init-corpus follow-ups from #401 in one PR: makes the verification layer visible to humans (#409) and adds adversarial-rare coverage to complement the everyday-real release gate (#410). # #410 — Chaos workflow Weekly cron-driven workflow that runs aqe init against 6 pathological project shapes the everyday-real corpus deliberately doesn't cover. Per-shape assertion is "exit anything except 124" — the watchdog is the load-bearing thing under test, not init's ability to make sense of garbage input. Fail-loudly in the Actions tab; no auto-issue creation, matching the #411 mirror-test pattern. Six shapes: utf16le-bom, mixed-line-endings, symlink-loop, binary-as-text, minified-bundle, control-chars. Generated at runtime into a tmpdir by tests/fixtures/init-chaos/generate.sh — synthetic generation is appropriate here per the chaos README's "Why synthetic generation here" section: chaos tests need controlled adversarial inputs that don't exist in any real public repo. Verified locally: ran the generator + ran aqe init against all 6 shapes. All 6 pass with rc=0 against the current build, proving the generator produces valid shapes and the existing watchdog handles them today. The workflow exists to catch the day a future change breaks one of these paths. # #409 — VERIFICATION.md + matrix embed New docs/VERIFICATION.md is the maintainer-facing entry point: gate architecture diagram, how to interpret a failed gate, how to add a new fixture (links to corpus README), how to embed the verification matrix into a release notes file. scripts/embed-verification-matrix.sh downloads the init-corpus-logs artifact from a given workflow run, parses summary.txt, and emits a markdown matrix. Status-only (option C from the proposal) — Time and KG-entries columns are tracked as future work because the gate's current summary.txt doesn't carry that data and extending it would touch load-bearing infrastructure that's out of scope. docs/policies/release-verification.md is fully rewritten around the post-#401 reality: the manual aqe init checklist that pre-dated the corpus is removed (the gate now does what it tried to do, with 22 assertions per fixture × 4 fixtures × the load-bearing doubleInit flag). The Version Update Policy + Workflow sections are preserved intact — they cover the one class of pre-release error the gate cannot catch. CONTRIBUTING.md links to the new VERIFICATION.md from the Documentation section. # Housekeeping Removes 3 stale rows from tests/fixtures/init-corpus/README.md's deferred-items table: #408 (already merged), #409 and #410 (this PR). Only #407 remains — its 4 trigger conditions still haven't fired, so it stays deferred per its own definition. # Verification of this PR (per #401 rules) - actionlint passed on .github/workflows/init-chaos.yml (#408 lesson: js-yaml is not enough for workflow files) - bash -n on both new shell scripts - generator ran cleanly and produced all 6 shapes with the expected byte content (verified BOM, line endings, symlink targets, PNG signature, single-line bundle size, embedded NULs) - embed script ran against synthetic summary files (PASS-only and mixed PASS/FAIL) — both render correctly - live smoke test: ran aqe init against all 6 chaos shapes locally using the globally-installed aqe binary, all 6 exited rc=0 within the watchdog budget - every link in docs/VERIFICATION.md and docs/policies/release- verification.md verified to point at a file that exists Closes #409 Closes #410 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
aqe init chaos corpus
Adversarial-rare project shapes for the weekly
init-chaos workflow.
The everyday-real
init-corpus catches release regressions
against pinned slices of real public repositories. This directory
catches a different class of failure: aqe init hanging or
crashing on pathological file shapes that don't exist in any real
codebase but could be planted by a malicious or careless user.
Why synthetic generation here
The init-corpus README explicitly bans synthetic fixtures because they hide real-world content bugs in the release gate. That ban applies to regression markers — fixtures meant to prove init keeps working on actual user code. It does not apply to chaos tests: chaos tests need controlled adversarial inputs that don't exist in any real public repo. The lesson from #401 was "don't validate init against synthetic toy projects", not "never use synthetic data". This is the one place where synthetic generation is the right tool.
The fixtures are NOT committed. generate.sh writes them into a
temp directory at the start of every workflow run. Commits stay
small and the chaos shapes can be tweaked without churn.
The six shapes
| ID | What it is | Why it might hang init |
|---|---|---|
utf16le-bom |
TS source file encoded UTF-16LE with FF FE BOM |
Tokenizers that assume UTF-8 may loop or read garbage |
mixed-line-endings |
TS source with \r\n in the middle of an \n file |
Line-counters that resync on \n may misalign and re-tokenize |
symlink-loop |
src/a → src/b → src/a (mutual symlinks) |
Naive recursive walks loop forever |
binary-as-text |
PNG signature bytes saved as decoy.ts |
Tokenizers that don't sniff content type may run on binary |
minified-bundle |
~256 KB single-line JS in src/bundle.min.js |
Tokenizers with quadratic backtracking on long lines stall |
control-chars |
NUL and ESC sequences inside identifier-shaped tokens | Parsers may early-terminate strings or loop on resync |
What "pass" means for a chaos test
For each shape, the workflow runs timeout 60 aqe init --auto --json
and asserts the exit code is anything except 124 (the timeout's
signal that init hung).
- Exit 0 → init handled the chaos shape gracefully (best case).
- Exit non-zero → init detected the bad input and failed fast (also fine).
- Exit 124 → init hung past the 60-second watchdog (the failure mode this workflow exists to catch).
The watchdog is the load-bearing thing under test, not init's ability to make sense of garbage input.
What to do when this workflow fails
- Download the
init-chaos-logsartifact from the failed run. - Find the shape that hung and read its
${shape}.log(stderr from init) and${shape}.json(whatever--jsonoutput it produced before being killed). - Two possible outcomes:
- Real regression in init's watchdog or per-file timeout.
Reproduce locally with
./tests/fixtures/init-chaos/generate.sh /tmp/chaos && cd /tmp/chaos/<shape> && timeout 60 aqe init --auto --json. Open a P0 bug. - Known limitation we accept. Document it here in this README
and quarantine the shape via a comment in
generate.sh. The workflow stays green but the limitation is now visible.
- Real regression in init's watchdog or per-file timeout.
Reproduce locally with
- Do NOT silence the workflow without doing one of the above.
Local development
# Generate shapes into a temp dir
./tests/fixtures/init-chaos/generate.sh /tmp/chaos
# Inspect what was generated
find /tmp/chaos -type f -o -type l | sort
# Run aqe init against one shape
cd /tmp/chaos/utf16le-bom
timeout 60 aqe init --auto --json
Refs
- Issue #410 — chaos workflow proposal and acceptance criteria
- Issue #401 — the v3.9.1–v3.9.4 init regression series that motivated all the verification work
- tests/fixtures/init-corpus/README.md — the everyday-real corpus this complements