Add two scanner patterns for marketing personification of machinery — an
inanimate tool/artifact subject given human agency — wild-caught by the owner
across his own marketing pages.
Tier A (tool_agency_reflexive, soft, fires anywhere): a tool-noun subject that
acts on itself directly after the verb — "The suite defends itself.", "The rules
update themselves.", "It graded its own reflection." Adjacency plus a "...out"
lookahead spare the dev idioms "the test cleans up after itself" and "it sorts
itself out"; ordinary tool-verb prose ("the gate fails the build") has no
reflexive object and stays clean.
Tier B (tool_agency_volitional, soft, standalone-line only): a strongly-volitional
verb (decides/hunts/wants/knows/believes/cares/refuses/judges/thinks) with a
tool-noun or bare "It" subject on a headline line — "The bench decides which
model does which job.", "It hunts instances, not word lists." The everyday
technical verbs (reads/runs/checks/returns/learns/...) are excluded so
"the parser reads the file" and "the model learns the distribution" stay clean;
the tool-noun requirement keeps human roles out ("the judge decides the case").
Eval-first per CLAUDE.md: OWNER-05/06 (FN), REC-51/52/53 (recall), FP-88..91
(domain protection) added before the scanner; full battery green (adversarial
478 pass / 1 xfail, coverage 82/82, packs, kata, taboo parity, schema, gates
doc, build --check, strict-leakage). Documented in taboo-phrases.md and
pack-voice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
FINDING: the cheapest model that recovers macro structure under the climb
is claude-haiku-4-5 -- it converges in 5 rounds given --max-rounds 8 (was
capped-at-1-residual at cap 4; one round short, not a hard ceiling). On the
GPT side only the frontier (gpt-5.5) converges; gpt-5.4-mini oscillates
(trajectories 4->1->5->2 at cap 4 and 3->3->2->1->6->2->1->5 at cap 8) and
never reaches clean. "Cheap tiers partially recover" was model-dependent,
not a property of cheap: read the two families separately.
Codex CLI as the GPT-side generator (OpenRouter still 402-blocked):
evals/model_generate.py gains a `codex` kind. codex exec is an agent CLI
whose stdout is an interleaved transcript, so call_codex extracts ONLY the
final message via -o/--output-last-message (verified clean by hand), runs
read-only/ephemeral, and wraps the call in its own process group with a
hard timeout that SIGKILLs the group -- the known silent-hang failure mode
becomes an honest failed round. CLIMB-07 gates the extraction and the hang
kill offline against a fake codex binary (evals/fixtures/climb/fake_codex.py).
All four live runs (gpt-5.5 cap 4 converged 4->1->1->0; gpt-5.4-mini caps 4
and 8 capped; haiku-4-5 cap 8 converged 1->1->1->1->0) preserved every
source fact every round, on the same SKILL-MACRO-01 fixture and settings as
the recorded matrix. pipeline.md's climb section gains the follow-up table
and the cheapest-viable doctrine; the tiering row and PRODUCT.md bullet
updated to the model-dependent reading.
Gates: adversarial 469 pass / 1 documented xfail (CLIMB-07 new), schema,
gates-doc, taboo parity, build --check, strict-leakage all green; 0 scanner
hits (hard or soft) on both edited docs. Behavioral tune: 33/34 tasks ran;
the one failure was a transient API server error mid-response on
SKILL-MACRO-01/with_skill (runner infra, not a model answer), which aborted
the non-blocking judge step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
THESIS: the recorded parity matrix showed single-pass macro-structure
self-checking failed on every model tested -- told in prose to drop a
conclusion coda / connective scaffold / outline echo, each shipped it
anyway. But structure_scan + silhouette_scan SEE those failures precisely,
so the failure is correctable even where it is not self-checkable.
PROOF: evals/run_structure_climb.py closes the loop -- generate -> scan both
-> turn each finding into a TARGETED directive that names WHERE and WHAT
(the model can't see the tell, so the directive locates it) -> feed the
directives + current draft back -> regenerate, capped at N rounds, with a
validate_preservation guard so climbing can never eat a fact. Honest
terminal states (converged / capped / preservation_violation) carry
distinct exit codes.
Core IP is the directive builder: a deterministic map from every macro flag
both scanners can emit to a location-naming instruction (coda -> names the
final paragraph; connective openers -> names the paragraph numbers;
callback_content -> names the recapped opening vocabulary). CLIMB-06 gates
1:1 flag->directive coverage.
Eval-first (red then green): six CLIMB-* rows drive the loop offline through
a mock generator (evals/fixtures/climb/) -- converge (monotonic consumption,
exit 0), capped-still-dirty (honest, nonzero), single-pass control (dirty
without the loop), preservation abort (names the dropped fact), directive
location-naming, and full directive coverage. Registered a blocking
mock-only gate (structure-climb) and synced evals/CHECKS.md.
LIVE (2026-07-07, same SKILL-MACRO-01 fixture as Task C, source-anchored
preservation): single pass ships macro tells on both Claude tiers; the climb
recovers macro on the frontier where single pass failed (sonnet-4-5
converged 2->1->0, facts preserved) -- macro is now machine-CORRECTED, not
merely machine-gated. On the cheap tier the recovery is partial: haiku-4-5
cleared silhouette immediately but oscillated on surface structure and
capped at one residual flag; honest numbers recorded. The OpenRouter
spectrum (gpt-5.5, glm-5.2) is pending on account billing (HTTP 402);
exact reproduction commands recorded in references/pipeline.md.
Docs: pipeline.md tiering macro row + measured-conclusions bullet updated to
machine-corrected; new "Macro structure under the climb" subsection with the
loop, directive builder, and before/after evidence; PRODUCT.md macro bullet;
one-line SKILL.md pointer in the validation flow.
Battery green: adversarial 468 pass / 1 documented xfail, schema, gates-doc,
coverage, kata, taboo parity, build --check, strict-leakage; 0 hard scanner
hits on edited docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Live run of evals/run_model_parity.py against the current open-weights
flagships on OpenRouter. All three clear Task B (span replacement) 6/6;
GLM 5.2 hits 1.00 Task A recall, DeepSeek and Kimi 0.92 (one voice-span
miss each in fixture A4). Adds the models file and the dated Recorded
results subsection; also clears the two pre-existing em-dash scanner
flags so pipeline.md scans 0 hard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Three AI-ism patterns wild-caught by the owner on his own marketing page,
added eval-first (rows appended red, then made green):
- slogan_fragment (banned_phrase_scan STRUCTURAL_PATTERNS, soft): standalone
"N X, one Y." slogan-cadence line/header ("Four presets, one input.").
- spec_fragment (banned_phrase_scan STRUCTURAL_PATTERNS, soft): standalone
"N noun-phrase, past-participle ..." spec fragment ("Eight criteria,
scored 1 to 5.").
- every_template_openers (structure_scan, soft, min 2 paragraphs): repeated
paragraph-initial "Every <noun> <verb>" template opener; a lone
"Every child deserves a good school." stays clean.
Both banned-phrase patterns fire only on whole-line/header contexts, so
prose-embedded counts ("The unit has two bedrooms, one bath, and a den.",
"We rated eight criteria, scored 1 to 5, before deciding.") stay clean.
Regexes are linear (bounded interior runs, no adjacent unbounded overlaps).
Rows OWNER-01/02/03 (FN specimens) went red first, then green after the
scanner changes; OWNER-04 + FP-86/FP-87 guard the false-positive twins.
New categories mapped into pack-voice and documented in taboo-phrases.md.
Suite: 462 pass / 1 xfail / 0 fail. Coverage, schema, taboo parity, packs,
kata, seeded-docs, pairs, silhouette, voice, benchmark, model-parity green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Plan 013: the Growth promise in docs/PRODUCT.md relied on someone
remembering to run wiki_sync, re-run the parity bench, and funnel misses
through contribute. This lands the three missing pieces:
- references/refresh.md: the agent-runnable refresh procedure — staleness
check, wiki-sync flow, parity-bench flow with the eval-gating rule,
every-miss-becomes-a-row routing through contribute, and result-recording
conventions (pipeline.md tables, TUNE-RESULTS.md style).
- scripts/refresh_status.py: stdlib, network-free staleness reporter.
JSON with wiki_sync / parity_bench / newest_pattern_row, each carrying
last/days/stale. Sources: .wiki_sync_state.json (last_timestamp, mtime
fallback), the "Live matrix recorded **YYYY-MM-DD**" line in
references/pipeline.md, and git log -1 --format=%cs on
evals/adversarial-evals.json (argv list, no shell). Thresholds (90/180/60
days) are commented guidance; always exits 0 — reporter, not gate.
- docs/DECISIONS.md: durable home for decided tradeoffs, each entry with
Decision / Why / Revisit-when / Source. Seeds the PRODUCT.md decisions
(agent-invoked only, English-only, no packaging, no rights checks,
removal-dominant axis) plus the commit-message-only ones: WP8 fixture
word-floor/length-balance queue (05f2363), voice impostor/background
calibration queue (41b60d1), protects-grain + build_report deferrals
(d6f6321), and FP-06 as the single intentional xfail (efd7e0e,
evals/CRITIQUE.md).
- One-line pointers to references/refresh.md from references/maintenance.md
and docs/PRODUCT.md (Growth).
All four touched/new docs pass banned_phrase_scan and
structure_scan --genre docs. Suite: 439 PASS / 1 XFAIL (FP-06) / 0 FAIL.
New decisions go in DECISIONS.md at decision time; the reporter is the
building block if the operator ever wants scheduled refresh outside this
repo.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Codex session JSONL (~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl) has a
different envelope shape than Claude Code JSONL: user/agent messages nest
under event_msg.payload.type, and response_item rows carry a role that can
be user, assistant, or developer. Codex also injects non-user content into
user-role turns (AGENTS.md dumps, <environment_context> banners, and
similar wrapper tags), which is a distinct contamination vector from plain
assistant authorship.
Detect claude-jsonl vs codex-jsonl per file by content shape, reuse the
existing filter/tripwire/dedup pipeline unchanged, and drop injected
wrapper content under a named instruction-injection drop reason instead of
silently discarding it. HARV-12..15 pin: assistant/base_instructions/
injection markers never reach candidates, substantive codex user prose is
harvested, injection wrappers are dropped with a named reason, and a mixed
claude+codex+text-folder run produces correctly attributed candidates from
all three sources.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Sub-flows stay reachable by natural name without being top-level verbs: a
Routing-by-phrase table maps audit, harvest, calibrate, refine, voice-check,
and found-a-new-tell phrasings to their command-file sections, pinned by the
router parity gate (required synonyms + real targets). mimic.md gains the
voice check: score a draft against a taught profile, report in plain words,
change nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Split the monolithic SKILL.md into a shared-doctrine router plus one
self-contained flow file per command under references/commands/. Bare
`/unslop <text>` defaults to rewrite; invoking a subcommand now requires
reading references/commands/<command>.md first.
Top-level verb surface (per product decision): teach, cleanup, rewrite,
mimic. contribute stays reachable as a maintenance path, not a verb.
cleanup folds the old audit report-only mode; mimic folds the refine
hill-climb; teach folds harvest (step 1) and offers calibrate when
coverage is thin. Flow text moved out of the deep references
(mimic/harvest/calibrate/contribute), which now hold internals plus a
pointer, so no content lives in two places.
SKILL.md keeps the constitution: register guards, validation gates and
blocking semantics, output formats, script and reference tables. 214
lines, within the DOC-05 budget; Quick Examples still pass DOC-02.
Eval-first parity gate: evals/check_commands.py pins the four-verb
surface and keeps argument-hint, the routing table, and the command
files in exact agreement (files exist both ways, <= 90 lines each, first
line `# /unslop <cmd>`). Added as DOC-13 (RED with the gate absent,
GREEN once implemented) and wired into list_gates() as
command-router-parity; CHECKS.md regenerated from --list-gates (24
gates, DOC-03 pins it).
Full deterministic battery green: adversarial 435 + 1 XFAIL, build
--check, taboo parity, pattern coverage, kata 4/4, pairs, seeded docs,
silhouette 12/0, voice all, skill examples, packs, gates doc, HARV /
CONTRIB / CAL / MIMIC / CARD slices, strict-leakage. All new and moved
command files pass banned_phrase_scan with zero hard violations.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
'Teach my voice' now carries the complete flow in SKILL.md and mimic.md: the
agent harvests samples when none are offered, runs the approval loop, creates
all directories, builds profile/card/provenance, surfaces low_confidence and
coverage gaps, and closes with a scored demo mimic. The user supplies only
approvals and answers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
F1: run_mimic_refine's acceptance/divergence scoring used a raw weighted
distance with no General Impostors term, so a marker-stuffed candidate that
cleared every hard gate could be accepted over honest prose. Score DEV and A
with the full voice_score composite (0.5*(1-GI) + 0.5* clipped weighted
impostor-z) against a seeded impostor pool (--impostors, default the committed
pool). New MIMIC-10 pins the regression: a punctuation/repetition-stuffed
candidate (cand02) and honest prose (cand01) both pass the hard gates, but under
the GI composite honest wins (0.143 vs 0.733). Divergence fixtures re-derived to
keep the genuine two-style A-down/DEV-up dynamics under the new composite.
F2: voice_card now recomputes the profile from --samples and exits 2 with a
named field mismatch when the supplied --profile does not describe them, so a
stale profile cannot silently drive a card (CARD-08).
F3: implement the LIVE generation path (--generate-cmd): per iteration assemble
B prompts (draft + A-split card + k=2 nearest-A samples + directives), invoke
the generator per beam over stdin, then run the same gate/score/accept pipeline.
--baseline selects prompt samples through the same path. mock_generator.py makes
it deterministically testable (MIMIC-11).
F4: document --name, the .txt/.md requirement (+ zero-doc diagnostics in
voice_profile/voice_card), the coverage->prompt templates per taxonomy
dimension, low_confidence surfacing, and rewrite mimic.md Scoring/refine to
describe the GI-bearing composite and the now-live generation path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Eval-first per CLAUDE.md: extended/added CAL-01, CAL-04, CAL-09..11 to pin
each behavior change before touching product code.
D1: em_dash restricted to the paired-dash<->paired-comma path only. A lone
joiner dash has no comma-pair equivalent that preserves sentence count
(the old lone-dash->period path silently split one sentence into two);
lone-dash-only passages now decline with exit 3. base_em_dash.txt fixture
rewritten to use paired dashes on both sentences. CAL-01 now also asserts
sentence count is unchanged between A and B for em_dash.
D2: contraction rewrites now skip matches sitting inside a capitalized
multi-word span (e.g. "Venue Can't Stop"), so a proper noun that happens to
reuse a contractable word is left untouched instead of becoming "Venue
Cannot Stop". Separately, _verify_constraints_preserved now does a
whole-occurrence (token-boundary) comparison instead of substring
containment, since "Venue Can" (a proper noun clipped at an apostrophe by
extract_constraints) is a substring of the corrupted "Venue Cannot" too.
New CAL-09 + evals/fixtures/calibrate/proper_noun_contraction.txt.
D3: generate_pair now runs banned_phrase_scan on both variants and
annotates the output with a_flags/b_flags (category lists, empty when
clean) instead of declining flagged variants. references/calibrate.md gets
a "Voice overrides defaults" section: a user's consistent preference for a
flagged pole is recorded with its flags, surfaced once, and marked "user-
preference overrides register guard" on the card. New CAL-10 +
staccato_flagged.txt / contractions_clean.txt fixtures.
D4: connectives plain pole now emits "But " (no comma), avoiding "So,"/
"Also," forms that read as filler_opener.
D5: calibrate_score.py reports preferred: null (status "tied") when a
dimension's top two tallies are exactly equal, instead of silently picking
whichever label sorts last.
D6: aggregate() dedups preferences by pair_id, keeping the latest row by
ts, so a replayed round counts once and the user's latest choice wins. New
dedup_by_pair_id() + CAL-11.
D7: references/calibrate.md instructs randomizing A/B display order per
round (seeded, reproducible) and recording the mapping so choices resolve
to the correct pole.
preferences_canned.jsonl extended with a staccato tie case and a
connectives pair_id replay case; CAL-04 assertions extended accordingly.
CAL-07's tiebreak premise (connectives lowest-n) still holds since
connectives ends at n=1 post-dedup, still the fixture's minimum.
Full adversarial suite: 334 pass, 1 documented xfail, 0 regressions. All
16 blocking gates verified green; behavioral-tune/rubric-judge (non-
blocking) skipped -- calibrate has no shared-benchmark skill rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Implements WP10b on top of the WP10a voice scorer.
Teach (scripts/voice_card.py): distills a profile + samples into a LAYERED
voice card — a <=300-word always-loaded card.md core with a "when writing X,
read card/X.md" index table, plus one card/<situation>.md sheet per COVERED
situation from a 10-dimension taxonomy. Uncovered dimensions get no sheet and
are named in card.md, so the card never fabricates a voice without sample
evidence. --coverage emits a deterministic lexical coverage matrix that drives
teach prompts (misclassification only adds/drops sheets, never card claims).
--provenance writes an auditable sha256 manifest. Byte-identical outputs.
Mimic --refine (evals/run_mimic_refine.py): iterative hill-climb with seeded
A/DEV document splits, hard removal gates per candidate (banned-phrase,
structure, draft->candidate preservation, copy-gate vs A, 150-word floor),
DEV-only acceptance, and a divergence guard that halts on the reward-hacking
signature (A improves while DEV worsens two rounds) with reward_hacking_warning.
Fully deterministic in --candidates-dir dry-run mode. Derives ranked directives
with paired card-amendment lines; writes report.json, final.md, and the refined
card.
Stats (evals/mimic_stats.py): per-item paired deltas, BCa bootstrap, and
sign-flip permutation; improved iff CI_low>0 AND p<0.05.
Eval-first: MIMIC-01..09 and CARD-01..07 rows added to adversarial-evals.json
first (RED before check_mimic.py existed), driven by evals/check_mimic.py over
committed dry-run fixtures. New "mimic-logic" gate wired into list_gates and
CHECKS.md regenerated. references/mimic.md replaces the onslaught stub; SKILL.md
gains a Teach & Mimic subsection within the DOC-05 budget; .unslop/ gitignored.
Stdlib-only, no new xfail, all gates green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Adds scripts/calibrate_pairs.py (deterministic dimension-controlled minimal
pairs for contractions, em_dash, sentence_length, connectives, staccato --
constraint-preserving, exits 3 when a dimension isn't expressible in the given
passage) and scripts/calibrate_score.py (Wilson-lower-bound aggregation over a
preferences JSONL, insufficient-below-k gating, --next dimension ordering, and
--profile conflict detection against a measured stylometric profile).
references/calibrate.md documents the agent-hosted game flow and provenance
discipline (stated-preference vs measured-from-samples); SKILL.md gets one
pointer line. Eval rows CAL-01..08 cover per-dimension transform correctness,
the inexpressible-dimension exit code, seed determinism, canned-stream
aggregation, below-k insufficiency, conflict detection, --next tiebreaking,
and the missing-file exit code. Wired a calibrate-suite gate (--only CAL) into
list_gates() and regenerated the CHECKS.md matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
- docs/PRODUCT.md: the grill-session decisions as durable doctrine; passes all
three scanners (and dogfooding it surfaced the carve-out below)
- references/pipeline.md: Model Parity results table populated from the live
2026-07-06 benches; tiering guidance updated to measured conclusions
- Bare 'research indicates/shows/suggests' joins vague_attribution
(FN-31/FP-84/REC-49, eval-first) — caught live when a model paraphrased
around 'studies show' in the replacement bench
- silhouette_scan --genre docs suppresses callback_content only: doctrine/spec
register reprises opening themes by convention (SIL-9 pins the carve-out;
essay recap codas still flag by default)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scripts/voice_profile.py + voice_score.py: char-3gram cosine, cosine delta,
sentence EMD, punctuation/contraction/MTLD/word-length distances, z-normalized
vs an impostor pool, General Impostors rank as the dominant gaming-resistant
term; copy gate (4-gram overlap + LCS); seeded determinism. Three synthetic
author corpora (amara/boris/celia) as measurement fixtures; VOICE-01..11 rows
(3x3 separation, GI sanity, gaming guard, copy gate, determinism, drift-pinned
profiles via check_voice).
Adversarially verified: four fresh mechanical attacks lose by >=0.85 composite;
separation diagonal wins at 5 seeds. Known follow-ups (approved-with-fixes,
queued): replace the mislabeled stuffed fixture with a true mechanical attack,
same-genre impostor pool, real --background calibration for delta z-scores,
feature-space GI subsets, full length-floor nulling, function-word dedupe,
IDF-weighted char-3grams.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ship scripts/silhouette_scan.py: a macro-structure scanner one level above
structure_scan that scores idea arrangement, not surface cadence. Five
validated one-sided tells (scaffold_opener_share, callback_content,
role_entropy_bits, preview_fulfillment, heading_preview) reused verbatim from
the research prototype, composited as a weighted relu-of-z penalty against a
committed human reference; flags at silhouette_penalty >= 1.0.
Scale deviation (documented in the scanner and reference): these tells are
degenerate at zero across the human corpus, so a literal sample-IQR denominator
with a 0.05 floor over-fires (2/8 human false positives). The scorer scales each
metric by max(sample_iqr, human_fence) instead, which keeps the exact
weighted-relu shape, reproduces the research struct01/03/09/11 signature, and
holds 0/8 human false positives while catching 12/12 ai (>= the 8/12 target).
- evals/check_silhouette.py: --reference drift gate (regenerates
human_reference.json from named in-repo sources and asserts equality) and
--separation gate (>= 8/12 ai, 0/8 human on the copied-in corpus).
- evals/fixtures/silhouette/: committed human_reference.json, self-contained
ai/human corpus, and SIL fixtures.
- Eval rows SIL-01..08 (RED-first) plus SIL-05a/b proving the silhouette +
structure_scan anti-gaming pairing against a cue-deletion attack, and DOC-07
for check_silhouette. Two gates wired into list_gates(); CHECKS.md regenerated.
- Docs: silhouette subsection in references/taboo-phrases.md, detector pointers
in pack-structure.md, and silhouette_scan added to the SKILL.md validation
list as blocking with a genre carve-out.
Voice-fingerprint (per-author median/scale) integration is noted for the
teach/mimic branch and deliberately left unimplemented.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Add evals/run_model_parity.py: measures whether cheap models match strong
models on the two model-dependent product surfaces — Tier-1 pack DETECTION
(Task A) and Tier-2 REPLACEMENT generation (Task B) — over a fixed seeded
corpus, graded deterministically.
- Config-driven matrix {name, kind, model_id}: kind claude-cli
(`claude -p --model <id>`, Anthropic spectrum) or openrouter (GPT spectrum,
key from macOS keychain service OPENROUTER_API_KEY).
- Task A: 6 seeded fixtures (phrases-core/voice/register-guards; 2 are
register-guard cases), pack + chunk per the Tier-1 contract, recall + false
findings vs a frozen manifest.
- Task B: 6 seeded findings; span-minimal replacement graded by the co-writer
contract — both scanners (no tell added, target removed), constraints
preserved, span-minimal.
- Outputs per-model/per-task JSON table + markdown summary. --dry-run grades
canned response fixtures with no network; --no-network proves graceful
degradation; --check-determinism proves stable grading.
Eval rows (dry-run, network-free): PARITY-01 grading correct on canned
good/bad responses (both tasks), PARITY-02 determinism, PARITY-03 missing
key/network never breaks the suite.
Docs: references/pipeline.md gains a Model Parity section, binds any change to
co-writer/mimic/pack features to running the harness across both spectrums, and
ties the detection/replacement tiering rows to its data. CLAUDE.md + AGENTS.md
get one identical required-checks line for those features.
Gates green: run_adversarial (203 pass, 1 xfail), build_shared_benchmark
--check, check_taboo_parity, check_gates_doc, check_skill_examples,
check_packs, skill-benchmark validate --strict-leakage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
- evals/check_pattern_coverage.py: every structural pattern must match >=1
eval-row input and every banned phrase must appear in the row corpus (no
grandfathering — 28 DET rows, 28 REC coverage packs, 31 FP protection rows
added to reach 75/75 patterns, 313/313 phrases); every category must be
claimed by an FP row's protects field (52/52, backfilled)
- evals/kata_add_pattern.py: rehearses the maintenance procedure in a temp
copy and asserts each safety net fires in order (coverage -> parity -> green
-> red-on-removal); deterministic transcript
- Coverage gate immediately caught a dead pattern: filler_opener matched
literal 'So' against lowercased text; resurrected comma-gated and soft with
FP-83 protecting ordinary sentence-initial 'So'
- DOC-09/DOC-10 gates wired into --list-gates and CHECKS.md; maintenance.md
documents the conventions
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A page of "Short statement. Short statement." headlines passed the scanner
clean: individually each pair fell below the two-beat imperative_slogan rule
(which spares noun-phrase and subject-first beats) and below the 3-sentence
staccato threshold, so a whole landing page of slogan headlines read clean.
Add headline_cadence (soft) to STRUCTURAL_PATTERNS. It generalizes
imperative_slogan to ANY standalone line built wholly from two-to-four short
sentences (each <=5 words; subject-verb, imperative, or a bare noun phrase),
and fires only at DOCUMENT frequency >=3 via the min_matches machinery. One
such headline is voice; three is template grammar.
Encoding + honest limit: the pattern is anchored to a full line and the first
beat must begin at line start, so a line that opens with any long sentence, or
a short-sentence run flowing inside a paragraph, is spared (end-of-line anchor).
In plain text "standalone headline" can only be approximated as "a line whose
whole content is the short-sentence pair," so the tell is gated purely on
frequency, not on proof the line is a real heading. That limit is documented in
taboo-phrases.md and the pattern comment.
evals: +1 FN production regression (the six shipped Artifacture headlines
composed into a realistic multi-headline document; source-tagged
"regression: artifacture landing 2026-07-05") and +2 FP controls — the
frequency gate (one headline + prose stays clean) and the line-shape guard (a
short-sentence pair flowing inside a paragraph stays clean). The existing
single-headline FP controls FP-45/FP-46 stay green because one line is below
the frequency gate. Map headline_cadence to pack-voice; document the family in
taboo-phrases.md and pack-voice.md.
Full gate sweep green: adversarial 200 PASS / 1 expected XFAIL / 0 FAIL;
shared benchmark unchanged (--check passes); pack integrity (52 categories),
taboo parity, gates doc, skill examples, and strict-leakage validate all pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AMn6wt4fMCxg7rxj9igFtS
A marketing page's copy passed the scanner clean despite five AI tells the
library should catch. Add five structural patterns (all soft) to
STRUCTURAL_PATTERNS, with must-catch/must-not-catch eval coverage:
- contrastive_definition: sentence-final "X, not Y." tail (copula and
verb-fronted) plus the "X isn't a Y, it's a Z." comma variant. Guarded
against imperative corrections ("Use pnpm, not npm."), numeric corrections
("Latency fell 40%, not 4%."), and the authentication idiom ("The painting
is real, not a forgery.").
- imperative_slogan: two back-to-back <=4-word imperative-first sentences
("Emit 1,100 tokens. Ship 237KB."). Curated verb list so subject-first
pairs and imperative+noun-list slogans do not fire.
- fragment_template: repeated "<plural noun> that <verb>." headline fragment,
frequency-gated at 2+ via min_matches (one instance stays clean).
- numeric_parallelism: "One X, N Y." Narrowed to the leading-"one" shape so
"Six aesthetics, zero prompting." is honestly excluded, not overfit.
- anthropomorphic_shipping: an abstraction that "ships inside/with" something.
Concrete changelog "ships with" is spared.
evals: +7 FN (must-catch) and +8 FP (must-not-catch, incl. the frequency
gate and both narrowing decisions). Map the five new categories to pack-voice
in the manifest; document the families in taboo-phrases.md and pack-voice.md.
Full gate sweep green: adversarial 197 PASS / 1 expected XFAIL / 0 FAIL,
pack integrity, taboo parity, shared-benchmark --check, gates doc, skill
examples, and strict-leakage validate all pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AMn6wt4fMCxg7rxj9igFtS
- whether-you're pattern requires a role noun phrase (a/an/just starting) and
matches sentence-initially: 'Whether you're right or wrong' and activity
gerunds are clean, canonical audience flattery still flags (FP-39/40, REC-19)
- Bare 'bolstered' ban dropped in favor of the tense-aware collocation gate;
literal reinforcement clean, rhetorical support flags (FP-41, REC-20);
catalog row updated
- double-down collocation inflected (doubled/doubling) with its/their/a
determiners (FN-22)
- and/or extracted as a preservation constraint: collapse to plain 'and'
fails, rewording to 'or (or both)' passes (PRES-28/29)
- structure_scan opener-repetition ignores enumeration openers
(Section/Chapter/Step/Figure...); academic roadmap abstracts clean
(STRUCT-17)
- run_local.py fails a case on CLI error output (Not logged in etc.) instead
of grading the error page; stale DONOHARM-01 tune artifact regenerated with
a real answer (similarity 1.0)
- SKILL.md preservation-gate wording aligned with --strict semantics
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- SKILL.md 295 -> 180 lines: one home per rule, pointers to authoritative
references; single-agent path complete without pipeline.md; comma-splice
guard, assets/examples pointer, and maintenance pointers restored after
loss audit
- references/pipeline.md: Tier 0 deterministic scripts -> Tier 1 small
detector agents (one pack x one chunk, JSON findings) -> Tier 2 single
strong rewriter -> Tier 0 gates; model-tier table, escalation rule, cost
math
- references/packs/: five self-contained detector rule-packs (phrases-core,
structure with concrete thresholds, register-guards, voice, facts) +
manifest mapping all 46 scanner categories
- references/maintenance.md: add-phrase/add-structure/list/wiki-sync
procedures, eval-first
- scripts/check_packs.py + DOC-04/05/06: pack integrity, size budgets
(SKILL<=300, packs<=120), CLAUDE/AGENTS body identity — all deterministic
gates; pack gate in --list-gates and CHECKS.md
- SKILL-TIER-01 behavioral case: a register-guards-scoped detector must
surface the legal hedge and not touch out-of-scope slop; judge + deterministic
backstops
Grounded in OSS spot-check: small models lose register guards in a monolith
and cannot self-check macro structure without explicit thresholds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The meme-derived "the struggle is real" / "the stakes are real" punch line
asserts significance instead of showing it. Catalog it as a soft emphasis
crutch, restricted to abstract-emphasis subjects so the literal "is it
genuine?" sense (a painting that is real, not a forgery) is spared.
- banned_phrase_scan.py: new STRUCTURAL_PATTERNS regex (soft, emphasis_crutch)
- taboo-phrases.md: Emphasis Crutches table rows + detection-regex entry
- adversarial-evals.json: FN-06 (detection) + FP-14 (literal-sense guard)
- SKILL.md: 2.1.0 -> 2.2.0
Adversarial harness: PASS 54, FAIL 0; benchmark in sync.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the root cause behind the adversarial writing audit's B1: the skill
TAUGHT the anti-slop register. Its gold examples modeled bare fragment contrasts
and staccato triplets, so faithful operators reproduced them.
- edit-library.md: rewrote the three worst After examples (Patterns 1, 3, 12)
from fragment/staccato cadence into single varied sentences; tightened Pattern
2; added a "Vary the rhythm" transformation principle.
- crisp-human.md (the engine): sentence-length guidance now requires variety
("never three sub-six-word sentences in a row; avoid 'Not X. Y.'"); "one idea
per sentence / no compound thoughts" relaxed to allow flowing clauses; the two
antithesis gold examples rewritten; checklist gains rhythm + load-bearing-hedge
items.
- warm-human.md: rewrote the gold example that tripped binary_contrast.
- personality-guide.md: added two guardrails (don't invent a first-person voice
for impersonal copy; don't mistake de-hedging for voice in regulated content);
scoped the "show uncertainty" example.
- banned_phrase_scan.py: dropped the single-em-dash soft flag, keeping only the
2+ overuse flag, to match the revised "one appositive dash is fine" rule.
Result: all 0 gold After examples trip the scanner (was 8). Harness unchanged
(50 PASS / 1 XFAIL / 0 FAIL).
Adds research-sourced AI-writing tells the scanner was missing, applied
conservatively to keep false positives low, plus detection/FP-guard evals.
New patterns (scanner + references/taboo-phrases.md, with sources):
- Wikipedia "Signs of AI writing": plays a {vital,crucial,significant} role,
enduring/lasting legacy, leaves a lasting impact, watershed moment, in the
realm of, rich cultural heritage, stunning natural beauty, has been described
as, is widely regarded as, some critics argue, observers have noted, would be
complete without, in this article we will explore, as an AI language model
- Berens & Kobak (Science Advances 2024, excess-vocab study of 14M PubMed
abstracts): elucidate, delineate, underpin, unveil, seamless, invaluable,
noteworthy, revolutionize, holds (great) promise, new avenues, shed new light,
warrants further, it should be noted that, in recent years, with the advent
of, taken together, garnered significant attention, a deeper understanding of
Restraint to avoid false positives:
- academic single-words are soft (flag when clustered, not on one use)
- gated "serves as a"/"stands as a" behind significance nouns, so "serves as a
kitchen" is clean but "serves as a testament" still flags
- skipped cluster-only words (notably, particularly, demonstrate, exhibit)
Evals: DET-01..07 detection cases, FP-10/11 false-positive guards.
Suite now 62 cases (44 script: 43 PASS / 1 XFAIL / 0 FAIL, 18 behavioral skill).
Flips 19 of the 20 deterministic xfail cases to PASS (FP-06, literal
"delve into a place", stays xfail as an accepted regex sense-ambiguity limit).
Final: 20 PASS, 1 XFAIL, 0 FAIL.
Fact preservation (validate_preservation.py, extract_constraints.py):
- currency compares absolute magnitude, so $47.3M no longer equals $47.3 billion
- percentages require an exact token match (12% no longer matches 120%)
- dates require the month, not just the year (March 3 2020 != December 2020)
- bare integers and whole phone numbers are now tracked as constraints
- faithful rewordings ($47.3M -> $47.3 million) still pass, no false missing
- missing input files exit cleanly instead of raising a traceback
Scanner (banned_phrase_scan.py):
- removed over-broad entries: "the real" (hit "the real estate"), bare
"period."/"full stop." (hit any sentence ending in "period")
- gated context-sensitive words (leverage, navigate, tapestry, boasts) behind
jargon collocations so literal/financial senses aren't flagged
- added missing tells: in conclusion, firstly/secondly, underscore the
importance, treasure trove, ever-evolving, rich mosaic, plus stop-slop's
false-agency family (numbers speak for themselves, data tells a story)
- quote masking: dropped the 500-char cap, allowed multi-line quotes, and
stopped masking single-quoted prose that was hiding real slop
Robustness (readability_metrics.py, diff_check.py):
- numeric-only text counts as words instead of reading as empty
- punctuation-only edits register as change instead of 0%
Kept references/taboo-phrases.md in sync with the new scanner patterns.
Clayton Kim