Brings up a complete test infrastructure spanning four tiers, each
testing a different layer of the project. Built incrementally over
many smaller commits and squashed here.
Tier 1 — language fixtures (free, fast)
Per-construct fixtures under tests/fixtures/lang/ that round-trip
through `allium check`. Covers diagnostic codes the CLI emits, with
a drift mechanism that keeps the corpus in sync as 3.2.x evolves.
Tier 2 — doc-example validation (free, fast)
Extracts code blocks from skills/ and reference docs and validates
each against the language. Inline annotations and an out-of-line
override file (tests/fixtures/docs/overrides.json) handle the long
tail of "this example is illustrative, not buildable."
Tier 3 — skill behavioural evals (gated --live, ~$0.05–$0.50/scenario)
`claude -p` invokes a skill against a fixture workspace; a second
`claude -p` call scores the result against a rubric (LLM-as-judge).
Supports both bare-API and OAuth auth modes. Rubrics under
tests/fixtures/evals/rubrics/.
Tier 4 — end-to-end pipeline (gated --live, ~$1–$10/scenario)
Multi-step distill→weed→tend→propagate scenarios against checked-in
fixture projects. Cumulative workspace state at the end of the run
is compared file-by-file against accepted snapshots.
Tier 4 features developed across this work:
• Dual snapshot comparison: text + Allium model JSON. Either being
a match passes.
• Snapshot-failure judge: LLM classifies diffs as cosmetic /
structural / semantic and gates pass/fail accordingly. Default
haiku, ~$0.05 per failed snapshot. Verdict cached per kept run.
• Kept workspaces (--keep-workspace): preserves the workspace +
per-step checkpoints + a manifest at tests/.tier4-runs/<scenario>-<ts>/
so expensive generation output can be inspected, snapshotted,
and replayed without regenerating.
• Replay mode (--workspace <path>): re-runs the snapshot comparison
against a kept workspace at no API cost.
• Auto-resume (--workspace <path> after scenario steps were added):
detects checkpoint vs scenario-step count, resumes from the last
checkpoint as the working state.
• Variance scenarios (repeat: N): a scenario can opt into running
N independent times into runs/01..NN/ subdirs under one parent
kept dir, with a configurable concurrency cap (--concurrency N,
default 3). Used to investigate distill-output stability.
• Convergence-report script (scripts/tier4-convergence.mjs): diffs
a tracked file across consecutive checkpoints; with --judge,
classifies each transition. Verdicts cached.
• Variance-report script (scripts/tier4-variance.mjs): line-count
distribution + pairwise (N choose 2) judge verdicts + clustering
(cosmetic + structural pairs treated as behaviourally equivalent).
Verdicts cached. --pair drilldown for one specific pair.
Auth modes: both bare ANTHROPIC_API_KEY and OAuth (--oauth) supported
across Tier 3 and Tier 4. The --json-schema flag was found incompatible
with OAuth/agent mode and dropped; envelope JSON is parsed instead.
UX: pretty output (colors + glyphs) with --plain opt-out, an allium
CLI version banner, in-place heartbeat updates on interactive
terminals, --verbose to stream claude stderr + pass --debug.
Snapshot baselines committed for the three node-todo scenarios
(distill-only, distill-then-tend, full-pipeline) plus the two
variance-investigation scenarios (weed-bounded, weed-convergence).
Documentation: tests/README.md (running, scenario authoring, kept
workspaces, replay, variance), tests/RATIONALE.md (the why behind
the tier split), and four iteration-by-iteration variance reports
in tests/docs/ that drove the prompt changes in the follow-up
commits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Matt Ford