CONTRIBUTING.md, an issue form, and a PR template link into the existing
contribute pipeline (references/contribute.md, references/commands/contribute.md,
CLAUDE.md) rather than restating it, plus one README pointer. Keeps the docs
summary-free so they can't drift from the gate-checked flow they link to.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
silhouette_scan.py never imported _lang, so a Spanish document the other
scanners wave through with non_english:true could still be hard-blocked
by silhouette metrics computed on unvalidated tokens. Decline is
scan-first (after result = scan(...)) to avoid re-introducing the
low-function-word English slop bug already fixed once in the sibling
scanners at 4d514a2. Adds LANG-20 (Spanish decline) and LANG-21 (decline
must not preempt a genuine flag, pinned against a known AI fixture).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Insert `from __future__ import annotations` in the 17 files that use
PEP 604/585 annotations in module-level positions evaluated at import
time, so scripts/banned_phrase_scan.py and friends no longer raise
TypeError on Python 3.8/3.9. Add a 3.8 leg to the CI matrix so the
floor claim in README.md is actually gated, and correct the two
imprecise "439 deterministic cases" references to "440 deterministic
script cases (439 pass, 1 documented xfail)".
New scripts must carry the future-import until the floor is raised;
if the maintainer later chooses 3.10+, delete the CI 3.8 leg and
README claim together.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Run /unslop mimic on the README against .unslop/voice/clayton. Rewrite the
running prose only: shorter/mid-range sentences over long compounds, contractions
at the card's rate, no semicolons (the card's Never list), causal "because"/"so"
connectors. Every fact, number, command, table cell, file path, and link is
unchanged; quoted slop specimens stay quoted.
Per the owner's order, remove "The weight sits about 70/30 toward removal." and
rework the paragraph so the detection-is-the-trust-asset point stands without the
internal ratio.
Scored against the owner's profile (voice_score, seed 7, impostor pool): prose-only
composite 0.225 -> 0.182, with sentence_emd, punctuation, contraction, and mtld
z-scores all moving toward the voice and GI holding. Gates clean: banned_phrase
0 hard/0 soft, structure_scan --genre docs clean, silhouette_scan --genre docs
clean. Full adversarial suite unaffected (439 PASS / 1 XFAIL / 0 FAIL).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
- evals/TUNE-RESULTS.md gains the 2026-07-06 holdout run (objective +4.2,
judge-blended flat, contaminated-baseline caveat) so public claims trace to
a repo record
- README lift interpretation cites the recorded number
- .codex/ nemesis review specs untracked and gitignored: process artifacts,
not product
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebuild the README around the current four-verb product (teach, cleanup,
rewrite, mimic) instead of the pre-restructure audit/rewrite framing.
- Lead with what unslop is and the four commands.
- Document the three detection layers (phrase, structure, silhouette) with
live counts: 313 banned phrases, 77 structural patterns, and the committed
silhouette corpus separation (12/12 AI, 0/8 human).
- Add a "gamut" tour of removed pattern families with quoted examples so the
README passes its own phrase scanner.
- Cover do-no-harm protections, teach/mimic voice building with the
contamination guarantee and gaming-resistant scoring, the cleanup co-writer
contract gates, contribute flywheel, and the eval suite (439 cases, 24 gates,
33-case behavioral split).
- Record the measured 2026-07-06 model-tiering results and refresh the scripts
tour and project tree to current reality.
README passes banned_phrase_scan (0 hard) and structure_scan --genre docs clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Codex session JSONL (~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl) has a
different envelope shape than Claude Code JSONL: user/agent messages nest
under event_msg.payload.type, and response_item rows carry a role that can
be user, assistant, or developer. Codex also injects non-user content into
user-role turns (AGENTS.md dumps, <environment_context> banners, and
similar wrapper tags), which is a distinct contamination vector from plain
assistant authorship.
Detect claude-jsonl vs codex-jsonl per file by content shape, reuse the
existing filter/tripwire/dedup pipeline unchanged, and drop injected
wrapper content under a named instruction-injection drop reason instead of
silently discarding it. HARV-12..15 pin: assistant/base_instructions/
injection markers never reach candidates, substantive codex user prose is
harvested, injection wrappers are dropped with a named reason, and a mixed
claude+codex+text-folder run produces correctly attributed candidates from
all three sources.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Sub-flows stay reachable by natural name without being top-level verbs: a
Routing-by-phrase table maps audit, harvest, calibrate, refine, voice-check,
and found-a-new-tell phrasings to their command-file sections, pinned by the
router parity gate (required synonyms + real targets). mimic.md gains the
voice check: score a draft against a taught profile, report in plain words,
change nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Split the monolithic SKILL.md into a shared-doctrine router plus one
self-contained flow file per command under references/commands/. Bare
`/unslop <text>` defaults to rewrite; invoking a subcommand now requires
reading references/commands/<command>.md first.
Top-level verb surface (per product decision): teach, cleanup, rewrite,
mimic. contribute stays reachable as a maintenance path, not a verb.
cleanup folds the old audit report-only mode; mimic folds the refine
hill-climb; teach folds harvest (step 1) and offers calibrate when
coverage is thin. Flow text moved out of the deep references
(mimic/harvest/calibrate/contribute), which now hold internals plus a
pointer, so no content lives in two places.
SKILL.md keeps the constitution: register guards, validation gates and
blocking semantics, output formats, script and reference tables. 214
lines, within the DOC-05 budget; Quick Examples still pass DOC-02.
Eval-first parity gate: evals/check_commands.py pins the four-verb
surface and keeps argument-hint, the routing table, and the command
files in exact agreement (files exist both ways, <= 90 lines each, first
line `# /unslop <cmd>`). Added as DOC-13 (RED with the gate absent,
GREEN once implemented) and wired into list_gates() as
command-router-parity; CHECKS.md regenerated from --list-gates (24
gates, DOC-03 pins it).
Full deterministic battery green: adversarial 435 + 1 XFAIL, build
--check, taboo parity, pattern coverage, kata 4/4, pairs, seeded docs,
silhouette 12/0, voice all, skill examples, packs, gates doc, HARV /
CONTRIB / CAL / MIMIC / CARD slices, strict-leakage. All new and moved
command files pass banned_phrase_scan with zero hard violations.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
'Teach my voice' now carries the complete flow in SKILL.md and mimic.md: the
agent harvests samples when none are offered, runs the approval loop, creates
all directories, builds profile/card/provenance, surfaces low_confidence and
coverage gaps, and closes with a scored demo mimic. The user supplies only
approvals and answers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Applies a reviewed cleanup list without changing any script's CLI flags, JSON
keys, exit codes, or stderr messages (all pinned by eval rows).
Shared modules (dedup):
- scripts/_lang.py: ENGLISH_FUNCTION_WORDS/english_function_share/is_probably_english,
previously byte-identical copies in banned_phrase_scan.py and structure_scan.py.
- structure_scan.py now imports split_sentences from readability_metrics instead
of keeping its own copy.
- silhouette_scan.py's stopword set is a verified pure superset of
structure_scan's; SILHOUETTE_STOPWORDS = structure's set | the extras.
- harvest_classify.py imports recency_value/DATE_FLOOR from harvest_samples.
- evals/_check_support.py: ROOT, run(cmd, timeout=60), load_evals() -- the
timeout=60 safety net that only check_contrib.py had now covers check_pairs,
check_seeded_docs, check_mimic, check_contrib, check_voice, and
check_pattern_coverage too.
Small cleanups: contribute.py's row_fn drops its unused category param;
run_model_parity.py's resolve_models param renamed responses->payload;
harvest_classify.py's heuristic() attaches suspect_ai/dictated so
rank_enriched needs no reconstruction; check_pattern_coverage.py's paired flag
formulas become a plain "if neither: both = True"; check_contrib.py drops the
__import__("scripts.contribute", ...) spelling for a normal import;
run_mimic_refine.py computes docs_a/matrix_a once and passes it to both
make_live_source and write_outputs; calibrate_pairs.py factors its four
near-identical contraction-replacement closures into one _contraction_repl
helper used by both directions.
Altitude items: GENRE_SUPPRESSIONS lookup tables replace the inline
`genre != "..."` conditionals in structure_scan.py and silhouette_scan.py;
check_gates_doc.py additively verifies every *.py token in a gate command
exists under ROOT (behavioral-tune and rubric-judge are exempt -- neither
command has a .py token).
Efficiency (Phase 2), each verified against the same eval rows / diffed
outputs before landing:
- run_model_parity.py replaces its subprocess-per-scanner-call helpers with
in-process imports of banned_phrase_scan/structure_scan/validate_preservation
(mirrors run_mimic_refine's import pattern). PARITY slice: 14.2s -> 0.33s.
- voice_score.py's lcs_len (O(n*m) DP) is replaced by
has_common_substring_over(), an O(n+m) rolling-hash check for "any shared
substring longer than the 120-char threshold" (hash matches are verified
against the source text, so no false positives). Nothing pins the exact
longest_common_substring value (checked); it now reports the matched
threshold window length on a hit, 0 otherwise -- documented in the
docstring. The violation boolean is unchanged.
- gi_score() precomputes per-key distances once per candidate/impostor
instead of recomputing distances() from scratch every trial; trials do a
subset-weighted sum over the precomputed values. Arithmetically exact
(same RNG draw order, same float sums) -- verified the VOICE-08
determinism value and the full check_voice --separation/--gi/--gaming
output are byte-identical before/after.
- check_voice.py and check_pairs.py convert their subprocess-per-cell/row
scanner calls to in-process imports (voice_score/voice_profile,
banned_phrase_scan/structure_scan), mirroring the CLI's own decline/exit
logic so output stays byte-compatible.
Deferred (out of scope for a contract-safe pass): a protects-grain redesign,
giving silhouette_scan.py its own English-decline gate, and decomposing
run_mimic_refine.py's build_report().
python3 evals/run_adversarial.py: 434 PASS / 1 XFAIL / 0 FAIL, unchanged
throughout; wall time 54.8s -> 25.3s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
- Both scanners now decline as non-English only when the function-word
heuristic fails AND zero patterns fired: imperative headline stacks and
buzzword noun-lists are English slop with few function words, and the old
order silently defeated the shipped headline-cadence detection
(LANG-3a/3b pin both genres; Spanish still declines)
- 'tells a story' flat hard ban replaced with an inanimate-subject structural
pattern (data/numbers/charts/... tells a story): grandmothers and clinical
case notes stay clean (FP-85), the genuine false-agency form still flags
(REC-50)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
F1: run_mimic_refine's acceptance/divergence scoring used a raw weighted
distance with no General Impostors term, so a marker-stuffed candidate that
cleared every hard gate could be accepted over honest prose. Score DEV and A
with the full voice_score composite (0.5*(1-GI) + 0.5* clipped weighted
impostor-z) against a seeded impostor pool (--impostors, default the committed
pool). New MIMIC-10 pins the regression: a punctuation/repetition-stuffed
candidate (cand02) and honest prose (cand01) both pass the hard gates, but under
the GI composite honest wins (0.143 vs 0.733). Divergence fixtures re-derived to
keep the genuine two-style A-down/DEV-up dynamics under the new composite.
F2: voice_card now recomputes the profile from --samples and exits 2 with a
named field mismatch when the supplied --profile does not describe them, so a
stale profile cannot silently drive a card (CARD-08).
F3: implement the LIVE generation path (--generate-cmd): per iteration assemble
B prompts (draft + A-split card + k=2 nearest-A samples + directives), invoke
the generator per beam over stdin, then run the same gate/score/accept pipeline.
--baseline selects prompt samples through the same path. mock_generator.py makes
it deterministically testable (MIMIC-11).
F4: document --name, the .txt/.md requirement (+ zero-doc diagnostics in
voice_profile/voice_card), the coverage->prompt templates per taxonomy
dimension, low_confidence surfacing, and rewrite mimic.md Scoring/refine to
describe the GI-bearing composite and the now-live generation path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Eval-first per CLAUDE.md: extended/added CAL-01, CAL-04, CAL-09..11 to pin
each behavior change before touching product code.
D1: em_dash restricted to the paired-dash<->paired-comma path only. A lone
joiner dash has no comma-pair equivalent that preserves sentence count
(the old lone-dash->period path silently split one sentence into two);
lone-dash-only passages now decline with exit 3. base_em_dash.txt fixture
rewritten to use paired dashes on both sentences. CAL-01 now also asserts
sentence count is unchanged between A and B for em_dash.
D2: contraction rewrites now skip matches sitting inside a capitalized
multi-word span (e.g. "Venue Can't Stop"), so a proper noun that happens to
reuse a contractable word is left untouched instead of becoming "Venue
Cannot Stop". Separately, _verify_constraints_preserved now does a
whole-occurrence (token-boundary) comparison instead of substring
containment, since "Venue Can" (a proper noun clipped at an apostrophe by
extract_constraints) is a substring of the corrupted "Venue Cannot" too.
New CAL-09 + evals/fixtures/calibrate/proper_noun_contraction.txt.
D3: generate_pair now runs banned_phrase_scan on both variants and
annotates the output with a_flags/b_flags (category lists, empty when
clean) instead of declining flagged variants. references/calibrate.md gets
a "Voice overrides defaults" section: a user's consistent preference for a
flagged pole is recorded with its flags, surfaced once, and marked "user-
preference overrides register guard" on the card. New CAL-10 +
staccato_flagged.txt / contractions_clean.txt fixtures.
D4: connectives plain pole now emits "But " (no comma), avoiding "So,"/
"Also," forms that read as filler_opener.
D5: calibrate_score.py reports preferred: null (status "tied") when a
dimension's top two tallies are exactly equal, instead of silently picking
whichever label sorts last.
D6: aggregate() dedups preferences by pair_id, keeping the latest row by
ts, so a replayed round counts once and the user's latest choice wins. New
dedup_by_pair_id() + CAL-11.
D7: references/calibrate.md instructs randomizing A/B display order per
round (seeded, reproducible) and recording the mapping so choices resolve
to the correct pole.
preferences_canned.jsonl extended with a staccato tie case and a
connectives pair_id replay case; CAL-04 assertions extended accordingly.
CAL-07's tiebreak premise (connectives lowest-n) still holds since
connectives ends at n=1 post-dedup, still the fixture's minimum.
Full adversarial suite: 334 pass, 1 documented xfail, 0 regressions. All
16 blocking gates verified green; behavioral-tune/rubric-judge (non-
blocking) skipped -- calibrate has no shared-benchmark skill rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Implements WP10b on top of the WP10a voice scorer.
Teach (scripts/voice_card.py): distills a profile + samples into a LAYERED
voice card — a <=300-word always-loaded card.md core with a "when writing X,
read card/X.md" index table, plus one card/<situation>.md sheet per COVERED
situation from a 10-dimension taxonomy. Uncovered dimensions get no sheet and
are named in card.md, so the card never fabricates a voice without sample
evidence. --coverage emits a deterministic lexical coverage matrix that drives
teach prompts (misclassification only adds/drops sheets, never card claims).
--provenance writes an auditable sha256 manifest. Byte-identical outputs.
Mimic --refine (evals/run_mimic_refine.py): iterative hill-climb with seeded
A/DEV document splits, hard removal gates per candidate (banned-phrase,
structure, draft->candidate preservation, copy-gate vs A, 150-word floor),
DEV-only acceptance, and a divergence guard that halts on the reward-hacking
signature (A improves while DEV worsens two rounds) with reward_hacking_warning.
Fully deterministic in --candidates-dir dry-run mode. Derives ranked directives
with paired card-amendment lines; writes report.json, final.md, and the refined
card.
Stats (evals/mimic_stats.py): per-item paired deltas, BCa bootstrap, and
sign-flip permutation; improved iff CI_low>0 AND p<0.05.
Eval-first: MIMIC-01..09 and CARD-01..07 rows added to adversarial-evals.json
first (RED before check_mimic.py existed), driven by evals/check_mimic.py over
committed dry-run fixtures. New "mimic-logic" gate wired into list_gates and
CHECKS.md regenerated. references/mimic.md replaces the onslaught stub; SKILL.md
gains a Teach & Mimic subsection within the DOC-05 budget; .unslop/ gitignored.
Stdlib-only, no new xfail, all gates green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Adds scripts/calibrate_pairs.py (deterministic dimension-controlled minimal
pairs for contractions, em_dash, sentence_length, connectives, staccato --
constraint-preserving, exits 3 when a dimension isn't expressible in the given
passage) and scripts/calibrate_score.py (Wilson-lower-bound aggregation over a
preferences JSONL, insufficient-below-k gating, --next dimension ordering, and
--profile conflict detection against a measured stylometric profile).
references/calibrate.md documents the agent-hosted game flow and provenance
discipline (stated-preference vs measured-from-samples); SKILL.md gets one
pointer line. Eval rows CAL-01..08 cover per-dimension transform correctness,
the inexpressible-dimension exit code, seed determinism, canned-stream
aggregation, below-k insufficiency, conflict detection, --next tiebreaking,
and the missing-file exit code. Wired a calibrate-suite gate (--only CAL) into
list_gates() and regenerated the CHECKS.md matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
- docs/PRODUCT.md: the grill-session decisions as durable doctrine; passes all
three scanners (and dogfooding it surfaced the carve-out below)
- references/pipeline.md: Model Parity results table populated from the live
2026-07-06 benches; tiering guidance updated to measured conclusions
- Bare 'research indicates/shows/suggests' joins vague_attribution
(FN-31/FP-84/REC-49, eval-first) — caught live when a model paraphrased
around 'studies show' in the replacement bench
- silhouette_scan --genre docs suppresses callback_content only: doctrine/spec
register reprises opening themes by convention (SIL-9 pins the carve-out;
essay recap codas still flag by default)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scripts/voice_profile.py + voice_score.py: char-3gram cosine, cosine delta,
sentence EMD, punctuation/contraction/MTLD/word-length distances, z-normalized
vs an impostor pool, General Impostors rank as the dominant gaming-resistant
term; copy gate (4-gram overlap + LCS); seeded determinism. Three synthetic
author corpora (amara/boris/celia) as measurement fixtures; VOICE-01..11 rows
(3x3 separation, GI sanity, gaming guard, copy gate, determinism, drift-pinned
profiles via check_voice).
Adversarially verified: four fresh mechanical attacks lose by >=0.85 composite;
separation diagonal wins at 5 seeds. Known follow-ups (approved-with-fixes,
queued): replace the mislabeled stuffed fixture with a true mechanical attack,
same-genre impostor pool, real --background calibration for delta z-scores,
feature-space GI subsets, full length-floor nulling, function-word dedupe,
IDF-weighted char-3grams.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ship scripts/silhouette_scan.py: a macro-structure scanner one level above
structure_scan that scores idea arrangement, not surface cadence. Five
validated one-sided tells (scaffold_opener_share, callback_content,
role_entropy_bits, preview_fulfillment, heading_preview) reused verbatim from
the research prototype, composited as a weighted relu-of-z penalty against a
committed human reference; flags at silhouette_penalty >= 1.0.
Scale deviation (documented in the scanner and reference): these tells are
degenerate at zero across the human corpus, so a literal sample-IQR denominator
with a 0.05 floor over-fires (2/8 human false positives). The scorer scales each
metric by max(sample_iqr, human_fence) instead, which keeps the exact
weighted-relu shape, reproduces the research struct01/03/09/11 signature, and
holds 0/8 human false positives while catching 12/12 ai (>= the 8/12 target).
- evals/check_silhouette.py: --reference drift gate (regenerates
human_reference.json from named in-repo sources and asserts equality) and
--separation gate (>= 8/12 ai, 0/8 human on the copied-in corpus).
- evals/fixtures/silhouette/: committed human_reference.json, self-contained
ai/human corpus, and SIL fixtures.
- Eval rows SIL-01..08 (RED-first) plus SIL-05a/b proving the silhouette +
structure_scan anti-gaming pairing against a cue-deletion attack, and DOC-07
for check_silhouette. Two gates wired into list_gates(); CHECKS.md regenerated.
- Docs: silhouette subsection in references/taboo-phrases.md, detector pointers
in pack-structure.md, and silhouette_scan added to the SKILL.md validation
list as blocking with a genre carve-out.
Voice-fingerprint (per-author median/scale) integration is noted for the
teach/mimic branch and deliberately left unimplemented.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Add agent-invoked co-writer mode and a graceful non-English decline, both
under the existing scanner constitution (eval-first; all gates green).
scripts/suggest.py
Emit LSP-style structured suggestions {span, severity, category, rationale,
suggested_replacement, phrased_as_question} from banned_phrase_scan +
structure_scan. Detection is deterministic; replacement generation is
DELEGATED (suggested_replacement null) with a --apply-replacements FILE mode
that merges externally-produced replacements and light-validates them. Soft
findings are phrased as questions. Deterministic order, non-overlapping spans.
scripts/check_suggestions.py
Blocking contract gates, each a named failure: span-minimality (edit touches
only its span; whole-sentence rewrites fail), replacement-scanner (each
replacement passes both scanners in isolation and in context), accept-all
(applying every suggestion yields a doc passing both scanners with
validate_preservation exit 0), span-overlap.
English-only decline
Cheap function-word heuristic (english_function_share / is_probably_english)
added identically to banned_phrase_scan.py and structure_scan.py. Below the
conservative threshold -> {"non_english": true, "violations": []}, exit 0 with
a stderr note. Threshold tuned so ESL English still scans.
Eval rows (RED before these scripts existed): SUGG-01..05 (suggestion emission,
soft-as-question, oversized-replacement rejection, accept-all safety, overlap
rejection) and LANG-01a/01b + LANG-02a/02b (Spanish declined by both scanners,
ESL English scanned by both). Fixtures under evals/fixtures/suggest/.
SKILL.md gains a compact Co-writer Mode section (agent-invoked; cheap detection,
delegated replacements, blocking contract gates, suggestions surfaced never
silently applied).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Add evals/run_model_parity.py: measures whether cheap models match strong
models on the two model-dependent product surfaces — Tier-1 pack DETECTION
(Task A) and Tier-2 REPLACEMENT generation (Task B) — over a fixed seeded
corpus, graded deterministically.
- Config-driven matrix {name, kind, model_id}: kind claude-cli
(`claude -p --model <id>`, Anthropic spectrum) or openrouter (GPT spectrum,
key from macOS keychain service OPENROUTER_API_KEY).
- Task A: 6 seeded fixtures (phrases-core/voice/register-guards; 2 are
register-guard cases), pack + chunk per the Tier-1 contract, recall + false
findings vs a frozen manifest.
- Task B: 6 seeded findings; span-minimal replacement graded by the co-writer
contract — both scanners (no tell added, target removed), constraints
preserved, span-minimal.
- Outputs per-model/per-task JSON table + markdown summary. --dry-run grades
canned response fixtures with no network; --no-network proves graceful
degradation; --check-determinism proves stable grading.
Eval rows (dry-run, network-free): PARITY-01 grading correct on canned
good/bad responses (both tasks), PARITY-02 determinism, PARITY-03 missing
key/network never breaks the suite.
Docs: references/pipeline.md gains a Model Parity section, binds any change to
co-writer/mimic/pack features to running the harness across both spectrums, and
ties the detection/replacement tiering rows to its data. CLAUDE.md + AGENTS.md
get one identical required-checks line for those features.
Gates green: run_adversarial (203 pass, 1 xfail), build_shared_benchmark
--check, check_taboo_parity, check_gates_doc, check_skill_examples,
check_packs, skill-benchmark validate --strict-leakage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
- evals/check_pattern_coverage.py: every structural pattern must match >=1
eval-row input and every banned phrase must appear in the row corpus (no
grandfathering — 28 DET rows, 28 REC coverage packs, 31 FP protection rows
added to reach 75/75 patterns, 313/313 phrases); every category must be
claimed by an FP row's protects field (52/52, backfilled)
- evals/kata_add_pattern.py: rehearses the maintenance procedure in a temp
copy and asserts each safety net fires in order (coverage -> parity -> green
-> red-on-removal); deterministic transcript
- Coverage gate immediately caught a dead pattern: filler_opener matched
literal 'So' against lowercased text; resurrected comma-gated and soft with
FP-83 protecting ordinary sentence-initial 'So'
- DOC-09/DOC-10 gates wired into --list-gates and CHECKS.md; maintenance.md
documents the conventions
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verifying a 3-headline document exposed a visibility bug: when a broad
anti_slop_register match spanned every headline line, the cross-category
containment de-dup filter erased all three headline_cadence occurrences, so the
document-level tell vanished on tight documents (and only survived in the FN
fixture by luck of span boundaries). A frequency-gated structural finding
(min_matches > 1) describes the whole document, not one span, so it must not be
suppressed by an unrelated larger match. Exempt those categories
(headline_cadence, fragment_template) from containment removal. The exemption
only prevents a frequency-gated finding from being dropped; it never adds or
removes any other category, so no other detection changes.
Full gate sweep still green: adversarial 200 PASS / 1 expected XFAIL / 0 FAIL;
shared benchmark unchanged; pack integrity, taboo parity, gates doc, skill
examples, and strict-leakage validate all pass. A tight 3-headline document now
reports headline_cadence x3; a 2-headline document stays below the gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AMn6wt4fMCxg7rxj9igFtS
A page of "Short statement. Short statement." headlines passed the scanner
clean: individually each pair fell below the two-beat imperative_slogan rule
(which spares noun-phrase and subject-first beats) and below the 3-sentence
staccato threshold, so a whole landing page of slogan headlines read clean.
Add headline_cadence (soft) to STRUCTURAL_PATTERNS. It generalizes
imperative_slogan to ANY standalone line built wholly from two-to-four short
sentences (each <=5 words; subject-verb, imperative, or a bare noun phrase),
and fires only at DOCUMENT frequency >=3 via the min_matches machinery. One
such headline is voice; three is template grammar.
Encoding + honest limit: the pattern is anchored to a full line and the first
beat must begin at line start, so a line that opens with any long sentence, or
a short-sentence run flowing inside a paragraph, is spared (end-of-line anchor).
In plain text "standalone headline" can only be approximated as "a line whose
whole content is the short-sentence pair," so the tell is gated purely on
frequency, not on proof the line is a real heading. That limit is documented in
taboo-phrases.md and the pattern comment.
evals: +1 FN production regression (the six shipped Artifacture headlines
composed into a realistic multi-headline document; source-tagged
"regression: artifacture landing 2026-07-05") and +2 FP controls — the
frequency gate (one headline + prose stays clean) and the line-shape guard (a
short-sentence pair flowing inside a paragraph stays clean). The existing
single-headline FP controls FP-45/FP-46 stay green because one line is below
the frequency gate. Map headline_cadence to pack-voice; document the family in
taboo-phrases.md and pack-voice.md.
Full gate sweep green: adversarial 200 PASS / 1 expected XFAIL / 0 FAIL;
shared benchmark unchanged (--check passes); pack integrity (52 categories),
taboo parity, gates doc, skill examples, and strict-leakage validate all pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AMn6wt4fMCxg7rxj9igFtS
Owner follow-up: the five marketing-page tells must live in the primary
must-catch suite as production regressions, not just as anonymous FN rows.
Per the repo's own docs, evals/evals.json is the legacy happy-path rewrite-
quality suite ("never edit it"); adversarial-evals.json is the source of
truth and run_adversarial.py is explicitly the regression harness. So the
regression identity lives there:
- FN-23/24/26/28/29 retitled "Production regression: ..." and given a
source tag ("regression: artifacture landing 2026-07-04"). Their stdin
was already the verbatim shipped copy.
- Each now also asserts expected severity (violations.0.severity == soft)
alongside the existing category assertion.
- Suite description documents the source-tag convention: rows carrying a
'source' field are real shipped copy that previously passed the scanner
clean and must stay caught.
Full gate sweep green: adversarial 197 PASS / 1 expected XFAIL / 0 FAIL;
shared benchmark regenerates byte-identically; pack integrity, taboo
parity, gates doc, skill examples, and strict-leakage validate all pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AMn6wt4fMCxg7rxj9igFtS
A marketing page's copy passed the scanner clean despite five AI tells the
library should catch. Add five structural patterns (all soft) to
STRUCTURAL_PATTERNS, with must-catch/must-not-catch eval coverage:
- contrastive_definition: sentence-final "X, not Y." tail (copula and
verb-fronted) plus the "X isn't a Y, it's a Z." comma variant. Guarded
against imperative corrections ("Use pnpm, not npm."), numeric corrections
("Latency fell 40%, not 4%."), and the authentication idiom ("The painting
is real, not a forgery.").
- imperative_slogan: two back-to-back <=4-word imperative-first sentences
("Emit 1,100 tokens. Ship 237KB."). Curated verb list so subject-first
pairs and imperative+noun-list slogans do not fire.
- fragment_template: repeated "<plural noun> that <verb>." headline fragment,
frequency-gated at 2+ via min_matches (one instance stays clean).
- numeric_parallelism: "One X, N Y." Narrowed to the leading-"one" shape so
"Six aesthetics, zero prompting." is honestly excluded, not overfit.
- anthropomorphic_shipping: an abstraction that "ships inside/with" something.
Concrete changelog "ships with" is spared.
evals: +7 FN (must-catch) and +8 FP (must-not-catch, incl. the frequency
gate and both narrowing decisions). Map the five new categories to pack-voice
in the manifest; document the families in taboo-phrases.md and pack-voice.md.
Full gate sweep green: adversarial 197 PASS / 1 expected XFAIL / 0 FAIL,
pack integrity, taboo parity, shared-benchmark --check, gates doc, skill
examples, and strict-leakage validate all pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AMn6wt4fMCxg7rxj9igFtS
- whether-you're pattern requires a role noun phrase (a/an/just starting) and
matches sentence-initially: 'Whether you're right or wrong' and activity
gerunds are clean, canonical audience flattery still flags (FP-39/40, REC-19)
- Bare 'bolstered' ban dropped in favor of the tense-aware collocation gate;
literal reinforcement clean, rhetorical support flags (FP-41, REC-20);
catalog row updated
- double-down collocation inflected (doubled/doubling) with its/their/a
determiners (FN-22)
- and/or extracted as a preservation constraint: collapse to plain 'and'
fails, rewording to 'or (or both)' passes (PRES-28/29)
- structure_scan opener-repetition ignores enumeration openers
(Section/Chapter/Step/Figure...); academic roadmap abstracts clean
(STRUCT-17)
- run_local.py fails a case on CLI error output (Not logged in etc.) instead
of grading the error page; stale DONOHARM-01 tune artifact regenerated with
a real answer (similarity 1.0)
- SKILL.md preservation-gate wording aligned with --strict semantics
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- SKILL.md 295 -> 180 lines: one home per rule, pointers to authoritative
references; single-agent path complete without pipeline.md; comma-splice
guard, assets/examples pointer, and maintenance pointers restored after
loss audit
- references/pipeline.md: Tier 0 deterministic scripts -> Tier 1 small
detector agents (one pack x one chunk, JSON findings) -> Tier 2 single
strong rewriter -> Tier 0 gates; model-tier table, escalation rule, cost
math
- references/packs/: five self-contained detector rule-packs (phrases-core,
structure with concrete thresholds, register-guards, voice, facts) +
manifest mapping all 46 scanner categories
- references/maintenance.md: add-phrase/add-structure/list/wiki-sync
procedures, eval-first
- scripts/check_packs.py + DOC-04/05/06: pack integrity, size budgets
(SKILL<=300, packs<=120), CLAUDE/AGENTS body identity — all deterministic
gates; pack gate in --list-gates and CHECKS.md
- SKILL-TIER-01 behavioral case: a register-guards-scoped detector must
surface the legal hedge and not touch out-of-scope slop; judge + deterministic
backstops
Grounded in OSS spot-check: small models lose register guards in a monolith
and cannot self-check macro structure without explicit thresholds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- evals/run_behavioral.sh: one-command behavioral run (prepare -> run_local ->
judge -> null-score coercion -> benchmark); holdback sealed behind
UNSLOP_CONFIRM_HOLDBACK=1
- evals/CHECKS.md: gate matrix mirroring --list-gates plus parallel check
protocol for small-context sub-agents; DOC-03 + check_gates_doc.py pin the
doc to the live matrix
- DOC-02 + check_skill_examples.py: SKILL.md example outputs must pass the
skill's own blocking gates (the old crisp example tripped both)
- CI runs all deterministic gates including strict-leakage validate and
taboo parity
- CLAUDE.md/AGENTS.md: consistent decision table for smallest useful eval
coverage, live-gate guidance, explicit behavioral-run rule; maintenance
commands are now an eval-first procedure; CRITIQUE.md counts point at the
runner instead of going stale
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- build_shared_benchmark: DETERMINISTIC_ASSERTIONS per case (fact tokens,
banned-string excludes, regex guards, word-count and difflib-similarity
scripts); validate_preservation --strict wired for legal negation; fixture
sync guard covers every fixtures/skill/ reference, not just preservation
- run_local: <final> marker extraction — output.md is the rewrite,
answer_full.md the full answer; audit-mode assertions read answer_full
- Three fresh holdout cases (SKILL-FRAGMENT-02, SKILL-DONOHARM-02,
SKILL-INJECT-02) so anti_slop_register / do_no_harm / prompt_injection are
no longer tune-only; FRAGMENT-02 uses a distinct bait frame
- Judge rubrics rewritten binary/observable; MODE-01 accepts both sanctioned
branches and its deterministic proxy accepts the ask-first branch
Verified by adversarial discrimination matrix: for 8 representative cases the
correct output passes and the target-failure output fails at least one
deterministic assertion, including the three previously vacuous backstops.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Clayton Kim