Add two scanner patterns for marketing personification of machinery — an
inanimate tool/artifact subject given human agency — wild-caught by the owner
across his own marketing pages.
Tier A (tool_agency_reflexive, soft, fires anywhere): a tool-noun subject that
acts on itself directly after the verb — "The suite defends itself.", "The rules
update themselves.", "It graded its own reflection." Adjacency plus a "...out"
lookahead spare the dev idioms "the test cleans up after itself" and "it sorts
itself out"; ordinary tool-verb prose ("the gate fails the build") has no
reflexive object and stays clean.
Tier B (tool_agency_volitional, soft, standalone-line only): a strongly-volitional
verb (decides/hunts/wants/knows/believes/cares/refuses/judges/thinks) with a
tool-noun or bare "It" subject on a headline line — "The bench decides which
model does which job.", "It hunts instances, not word lists." The everyday
technical verbs (reads/runs/checks/returns/learns/...) are excluded so
"the parser reads the file" and "the model learns the distribution" stay clean;
the tool-noun requirement keeps human roles out ("the judge decides the case").
Eval-first per CLAUDE.md: OWNER-05/06 (FN), REC-51/52/53 (recall), FP-88..91
(domain protection) added before the scanner; full battery green (adversarial
478 pass / 1 xfail, coverage 82/82, packs, kata, taboo parity, schema, gates
doc, build --check, strict-leakage). Documented in taboo-phrases.md and
pack-voice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Three AI-ism patterns wild-caught by the owner on his own marketing page,
added eval-first (rows appended red, then made green):
- slogan_fragment (banned_phrase_scan STRUCTURAL_PATTERNS, soft): standalone
"N X, one Y." slogan-cadence line/header ("Four presets, one input.").
- spec_fragment (banned_phrase_scan STRUCTURAL_PATTERNS, soft): standalone
"N noun-phrase, past-participle ..." spec fragment ("Eight criteria,
scored 1 to 5.").
- every_template_openers (structure_scan, soft, min 2 paragraphs): repeated
paragraph-initial "Every <noun> <verb>" template opener; a lone
"Every child deserves a good school." stays clean.
Both banned-phrase patterns fire only on whole-line/header contexts, so
prose-embedded counts ("The unit has two bedrooms, one bath, and a den.",
"We rated eight criteria, scored 1 to 5, before deciding.") stay clean.
Regexes are linear (bounded interior runs, no adjacent unbounded overlaps).
Rows OWNER-01/02/03 (FN specimens) went red first, then green after the
scanner changes; OWNER-04 + FP-86/FP-87 guard the false-positive twins.
New categories mapped into pack-voice and documented in taboo-phrases.md.
Suite: 462 pass / 1 xfail / 0 fail. Coverage, schema, taboo parity, packs,
kata, seeded-docs, pairs, silhouette, voice, benchmark, model-parity green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
scan_for_violations recomputed every violation's character span from scratch
via text.splitlines() (quadratic) and re-checked containment against every
other span (quadratic) -- 45.9M comparisons on a 6k-sentence doc. Both also
derived positions independently of the actual regex match, so CRLF documents
(splitlines() strips \r\n as one break but the reconstruction assumed a
1-char terminator) and case-expanding folds (str.lower() on U+0130 "İ" adds
a char) could corrupt line/column/context.
Fix: carry match.start()/match.end() as the single source of truth.
- Precompute line_starts once over the original text; derive line/column/
context per match via bisect (O(log n) instead of re-splitting the doc).
- Match phrases and structural patterns case-insensitively directly against
scan_text (masking is length-preserving) instead of a separately lowered
copy, so match offsets are valid against the original text with no
re-derivation. Added _phrase_pattern_ci as a call-site-only cache so
_phrase_pattern's case-sensitive default is unchanged for other callers.
STRUCTURAL_PATTERNS phrase field is explicitly re-lowered to preserve its
pre-fix casing (it was always matched against a lowered copy before).
- Containment is now a single O(n log n) sweep (sort by start asc/end desc,
track the tallest end seen so far) instead of an O(n^2) any(). Verified
equivalent to the old any()-based filter by diffing both filters' output
across all 227 script-eval fixture texts plus synthetic duplicate/nested/
adjacent-span stress cases -- zero diffs.
Perf (scan_for_violations, "a testament to progress." sentence corpus):
8,000 sentences: ~10.3s -> 0.92s
10,000 sentences: ~14.8s -> 1.17s (target: < 1.5s)
Adds SPAN-01 (CRLF line/column), SPAN-02 (İ case-fold context), and SPAN-03
(8k-sentence perf, < 2.0s) to evals/adversarial-evals.json. SPAN-01 was
already correct at HEAD (the CRLF bug lived only in the now-deleted span
reconstruction, not the line/column computed directly for JSON output) --
verified it still catches a regression against a deliberately-broken scratch
variant. SPAN-02 and SPAN-03 were red before this change.
python3 evals/run_adversarial.py: 442 PASS, 1 XFAIL (FP-06, unchanged), 0 FAIL.
Plan 013: the Growth promise in docs/PRODUCT.md relied on someone
remembering to run wiki_sync, re-run the parity bench, and funnel misses
through contribute. This lands the three missing pieces:
- references/refresh.md: the agent-runnable refresh procedure — staleness
check, wiki-sync flow, parity-bench flow with the eval-gating rule,
every-miss-becomes-a-row routing through contribute, and result-recording
conventions (pipeline.md tables, TUNE-RESULTS.md style).
- scripts/refresh_status.py: stdlib, network-free staleness reporter.
JSON with wiki_sync / parity_bench / newest_pattern_row, each carrying
last/days/stale. Sources: .wiki_sync_state.json (last_timestamp, mtime
fallback), the "Live matrix recorded **YYYY-MM-DD**" line in
references/pipeline.md, and git log -1 --format=%cs on
evals/adversarial-evals.json (argv list, no shell). Thresholds (90/180/60
days) are commented guidance; always exits 0 — reporter, not gate.
- docs/DECISIONS.md: durable home for decided tradeoffs, each entry with
Decision / Why / Revisit-when / Source. Seeds the PRODUCT.md decisions
(agent-invoked only, English-only, no packaging, no rights checks,
removal-dominant axis) plus the commit-message-only ones: WP8 fixture
word-floor/length-balance queue (05f2363), voice impostor/background
calibration queue (41b60d1), protects-grain + build_report deferrals
(d6f6321), and FP-06 as the single intentional xfail (efd7e0e,
evals/CRITIQUE.md).
- One-line pointers to references/refresh.md from references/maintenance.md
and docs/PRODUCT.md (Growth).
All four touched/new docs pass banned_phrase_scan and
structure_scan --genre docs. Suite: 439 PASS / 1 XFAIL (FP-06) / 0 FAIL.
New decisions go in DECISIONS.md at decision time; the reporter is the
building block if the operator ever wants scheduled refresh outside this
repo.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
All three scripts previously treated -h/--help as a filename argument:
check_suggestions.py --help printed "Missing file: --help" and exited 2;
readability_metrics.py and extract_constraints.py emitted a JSON error.
Every sibling script uses argparse and gets proper usage text.
Mirrors structure_scan.py's minimal shape (optional positional path,
stdin fallback). Existing invocations are byte-identical before/after
(verified via before/after snapshots on file-arg, stdin, and
missing-file cases) — only -h/--help behavior changes.
wiki_sync.py fetched and parsed external wikitext with no eval reaching
its parser or SECTION_MAP; a drift there would silently emit wrong
diffs/prompts. Add a thin --from-file seam to check/diff/prompt that
reads wikitext from disk, skips the network fetch, and skips the sync
state write, then add an invented offline fixture and WIKI-01..03 rows
exercising the mapped-section and unmapped-section paths through it.
The fixture pins the parser, not the world; live-page drift detection
remains cmd_check. Never copy real wikitext into fixtures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
structure_scan.py and silhouette_scan.py each carried a private copy of
words(), a markdown-to-prose stripper, and a paragraph splitter. The
strippers had drifted: structure blanked blockquote lines and
silhouette didn't; silhouette stripped **bold** and structure didn't.
Move words(), strip_markdown_for_prose(), and paragraphs() into
_lang.py as the UNION of both original code paths, gated behind
blank_blockquotes / strip_bold flags so each caller keeps its exact
prior behavior on purpose instead of silently converging. structure_scan
now calls paragraphs(text, blank_blockquotes=True); silhouette_scan calls
paragraphs(text, strip_bold=True). silhouette's STOPWORDS import from
structure_scan is untouched (out of scope).
Verified byte-identical: snapshotted `structure_scan.py` and
`silhouette_scan.py` stdout for all 12 AI fixtures, all 8 human
fixtures, and README.md before and after the swap (42 outputs) --
`diff -r` empty. check_silhouette.py --reference still reports
"reference ok: 5 metrics over 15 human sources"; --separation still
12/12 AI flagged / 0/8 human flagged with identical per-file penalties
(e.g. 05_essay.txt 14.90, 05_readme.txt human 0.62, unchanged). Full
adversarial suite: 439 PASS, 1 XFAIL (known), 0 FAIL. Dual-mode import
(package import and direct script invocation) verified for both
scanners.
Open maintainer question, deliberately unresolved: converge both
scanners on one prose view? Doing so changes silhouette's committed
reference and needs deliberate regeneration + separation re-verification
-- its own eval-first plan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Eight scripts (banned_phrase_scan, structure_scan, silhouette_scan,
readability_metrics, extract_constraints, suggest, check_suggestions,
voice_score) raised a raw UnicodeDecodeError traceback on non-UTF-8
stdin or file input. Switch stdin reads to
sys.stdin.buffer.read().decode("utf-8", errors="replace") and CLI
file reads to errors="replace", matching the idiom already used by
voice_profile.py, voice_card.py, and run_mimic_refine.py.
harvest_samples.py aborted an entire batch when one sibling file had
bad encoding or was unreadable: apply errors="replace" at its three
read sites and wrap per-file dispatch in collect_sources with a
try/except (OSError, UnicodeDecodeError) that records a new
"unreadable" drop-stat and warning (reusing the existing
stats/warnings plumbing that already tracks "instruction-injection")
and continues with the rest of the batch.
evals/run_local.py tracebacked when the claude CLI binary was
missing; catch (FileNotFoundError, OSError) alongside the existing
TimeoutExpired handling and degrade that one task instead of crashing
the whole run.
Add six ENC-01..06 eval rows (evals/adversarial-evals.json) covering
each fix, plus a small harvest fixture
(evals/fixtures/harvest/fixture_bad_encoding/good.md) whose marker
sentence must survive a sibling bad-encoding file generated inline by
the eval row (not committed as binary). Confirmed red against
unfixed code via git stash before restoring the fixes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
contribute.py joined --pattern-name into a filesystem path before any
validation, so a value like ../evil escaped .unslop/contrib/ and wrote
four bundle files there before the only containment check (a
relative_to call in the success print) ever ran. Reject non-slug names
up front with the file's existing error-JSON convention, exit 2, no
write.
The exclamation_overuse structural pattern had two adjacent unbounded
quantifiers over overlapping character classes (\s is a subset of
[^.]), causing quadratic backtracking on a long unclosed exclamation
run (5.7s at 32k chars, in-process callers like harvest and refine run
it with no timeout). Excluding ! from the interior class removes the
overlap and keeps identical match spans on FP-37 and REC-17.
New rows SLUG-01 and REGX-01 pin both fixes; REGX-01 times the live
STRUCTURAL_PATTERNS entry directly so it tracks the production regex.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
silhouette_scan.py never imported _lang, so a Spanish document the other
scanners wave through with non_english:true could still be hard-blocked
by silhouette metrics computed on unvalidated tokens. Decline is
scan-first (after result = scan(...)) to avoid re-introducing the
low-function-word English slop bug already fixed once in the sibling
scanners at 4d514a2. Adds LANG-20 (Spanish decline) and LANG-21 (decline
must not preempt a genuine flag, pinned against a known AI fixture).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Insert `from __future__ import annotations` in the 17 files that use
PEP 604/585 annotations in module-level positions evaluated at import
time, so scripts/banned_phrase_scan.py and friends no longer raise
TypeError on Python 3.8/3.9. Add a 3.8 leg to the CI matrix so the
floor claim in README.md is actually gated, and correct the two
imprecise "439 deterministic cases" references to "440 deterministic
script cases (439 pass, 1 documented xfail)".
New scripts must carry the future-import until the floor is raised;
if the maintainer later chooses 3.10+, delete the CI 3.8 leg and
README claim together.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Codex session JSONL (~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl) has a
different envelope shape than Claude Code JSONL: user/agent messages nest
under event_msg.payload.type, and response_item rows carry a role that can
be user, assistant, or developer. Codex also injects non-user content into
user-role turns (AGENTS.md dumps, <environment_context> banners, and
similar wrapper tags), which is a distinct contamination vector from plain
assistant authorship.
Detect claude-jsonl vs codex-jsonl per file by content shape, reuse the
existing filter/tripwire/dedup pipeline unchanged, and drop injected
wrapper content under a named instruction-injection drop reason instead of
silently discarding it. HARV-12..15 pin: assistant/base_instructions/
injection markers never reach candidates, substantive codex user prose is
harvested, injection wrappers are dropped with a named reason, and a mixed
claude+codex+text-folder run produces correctly attributed candidates from
all three sources.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Applies a reviewed cleanup list without changing any script's CLI flags, JSON
keys, exit codes, or stderr messages (all pinned by eval rows).
Shared modules (dedup):
- scripts/_lang.py: ENGLISH_FUNCTION_WORDS/english_function_share/is_probably_english,
previously byte-identical copies in banned_phrase_scan.py and structure_scan.py.
- structure_scan.py now imports split_sentences from readability_metrics instead
of keeping its own copy.
- silhouette_scan.py's stopword set is a verified pure superset of
structure_scan's; SILHOUETTE_STOPWORDS = structure's set | the extras.
- harvest_classify.py imports recency_value/DATE_FLOOR from harvest_samples.
- evals/_check_support.py: ROOT, run(cmd, timeout=60), load_evals() -- the
timeout=60 safety net that only check_contrib.py had now covers check_pairs,
check_seeded_docs, check_mimic, check_contrib, check_voice, and
check_pattern_coverage too.
Small cleanups: contribute.py's row_fn drops its unused category param;
run_model_parity.py's resolve_models param renamed responses->payload;
harvest_classify.py's heuristic() attaches suspect_ai/dictated so
rank_enriched needs no reconstruction; check_pattern_coverage.py's paired flag
formulas become a plain "if neither: both = True"; check_contrib.py drops the
__import__("scripts.contribute", ...) spelling for a normal import;
run_mimic_refine.py computes docs_a/matrix_a once and passes it to both
make_live_source and write_outputs; calibrate_pairs.py factors its four
near-identical contraction-replacement closures into one _contraction_repl
helper used by both directions.
Altitude items: GENRE_SUPPRESSIONS lookup tables replace the inline
`genre != "..."` conditionals in structure_scan.py and silhouette_scan.py;
check_gates_doc.py additively verifies every *.py token in a gate command
exists under ROOT (behavioral-tune and rubric-judge are exempt -- neither
command has a .py token).
Efficiency (Phase 2), each verified against the same eval rows / diffed
outputs before landing:
- run_model_parity.py replaces its subprocess-per-scanner-call helpers with
in-process imports of banned_phrase_scan/structure_scan/validate_preservation
(mirrors run_mimic_refine's import pattern). PARITY slice: 14.2s -> 0.33s.
- voice_score.py's lcs_len (O(n*m) DP) is replaced by
has_common_substring_over(), an O(n+m) rolling-hash check for "any shared
substring longer than the 120-char threshold" (hash matches are verified
against the source text, so no false positives). Nothing pins the exact
longest_common_substring value (checked); it now reports the matched
threshold window length on a hit, 0 otherwise -- documented in the
docstring. The violation boolean is unchanged.
- gi_score() precomputes per-key distances once per candidate/impostor
instead of recomputing distances() from scratch every trial; trials do a
subset-weighted sum over the precomputed values. Arithmetically exact
(same RNG draw order, same float sums) -- verified the VOICE-08
determinism value and the full check_voice --separation/--gi/--gaming
output are byte-identical before/after.
- check_voice.py and check_pairs.py convert their subprocess-per-cell/row
scanner calls to in-process imports (voice_score/voice_profile,
banned_phrase_scan/structure_scan), mirroring the CLI's own decline/exit
logic so output stays byte-compatible.
Deferred (out of scope for a contract-safe pass): a protects-grain redesign,
giving silhouette_scan.py its own English-decline gate, and decomposing
run_mimic_refine.py's build_report().
python3 evals/run_adversarial.py: 434 PASS / 1 XFAIL / 0 FAIL, unchanged
throughout; wall time 54.8s -> 25.3s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
- Both scanners now decline as non-English only when the function-word
heuristic fails AND zero patterns fired: imperative headline stacks and
buzzword noun-lists are English slop with few function words, and the old
order silently defeated the shipped headline-cadence detection
(LANG-3a/3b pin both genres; Spanish still declines)
- 'tells a story' flat hard ban replaced with an inanimate-subject structural
pattern (data/numbers/charts/... tells a story): grandmothers and clinical
case notes stay clean (FP-85), the genuine false-agency form still flags
(REC-50)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
F1: run_mimic_refine's acceptance/divergence scoring used a raw weighted
distance with no General Impostors term, so a marker-stuffed candidate that
cleared every hard gate could be accepted over honest prose. Score DEV and A
with the full voice_score composite (0.5*(1-GI) + 0.5* clipped weighted
impostor-z) against a seeded impostor pool (--impostors, default the committed
pool). New MIMIC-10 pins the regression: a punctuation/repetition-stuffed
candidate (cand02) and honest prose (cand01) both pass the hard gates, but under
the GI composite honest wins (0.143 vs 0.733). Divergence fixtures re-derived to
keep the genuine two-style A-down/DEV-up dynamics under the new composite.
F2: voice_card now recomputes the profile from --samples and exits 2 with a
named field mismatch when the supplied --profile does not describe them, so a
stale profile cannot silently drive a card (CARD-08).
F3: implement the LIVE generation path (--generate-cmd): per iteration assemble
B prompts (draft + A-split card + k=2 nearest-A samples + directives), invoke
the generator per beam over stdin, then run the same gate/score/accept pipeline.
--baseline selects prompt samples through the same path. mock_generator.py makes
it deterministically testable (MIMIC-11).
F4: document --name, the .txt/.md requirement (+ zero-doc diagnostics in
voice_profile/voice_card), the coverage->prompt templates per taxonomy
dimension, low_confidence surfacing, and rewrite mimic.md Scoring/refine to
describe the GI-bearing composite and the now-live generation path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Eval-first per CLAUDE.md: extended/added CAL-01, CAL-04, CAL-09..11 to pin
each behavior change before touching product code.
D1: em_dash restricted to the paired-dash<->paired-comma path only. A lone
joiner dash has no comma-pair equivalent that preserves sentence count
(the old lone-dash->period path silently split one sentence into two);
lone-dash-only passages now decline with exit 3. base_em_dash.txt fixture
rewritten to use paired dashes on both sentences. CAL-01 now also asserts
sentence count is unchanged between A and B for em_dash.
D2: contraction rewrites now skip matches sitting inside a capitalized
multi-word span (e.g. "Venue Can't Stop"), so a proper noun that happens to
reuse a contractable word is left untouched instead of becoming "Venue
Cannot Stop". Separately, _verify_constraints_preserved now does a
whole-occurrence (token-boundary) comparison instead of substring
containment, since "Venue Can" (a proper noun clipped at an apostrophe by
extract_constraints) is a substring of the corrupted "Venue Cannot" too.
New CAL-09 + evals/fixtures/calibrate/proper_noun_contraction.txt.
D3: generate_pair now runs banned_phrase_scan on both variants and
annotates the output with a_flags/b_flags (category lists, empty when
clean) instead of declining flagged variants. references/calibrate.md gets
a "Voice overrides defaults" section: a user's consistent preference for a
flagged pole is recorded with its flags, surfaced once, and marked "user-
preference overrides register guard" on the card. New CAL-10 +
staccato_flagged.txt / contractions_clean.txt fixtures.
D4: connectives plain pole now emits "But " (no comma), avoiding "So,"/
"Also," forms that read as filler_opener.
D5: calibrate_score.py reports preferred: null (status "tied") when a
dimension's top two tallies are exactly equal, instead of silently picking
whichever label sorts last.
D6: aggregate() dedups preferences by pair_id, keeping the latest row by
ts, so a replayed round counts once and the user's latest choice wins. New
dedup_by_pair_id() + CAL-11.
D7: references/calibrate.md instructs randomizing A/B display order per
round (seeded, reproducible) and recording the mapping so choices resolve
to the correct pole.
preferences_canned.jsonl extended with a staccato tie case and a
connectives pair_id replay case; CAL-04 assertions extended accordingly.
CAL-07's tiebreak premise (connectives lowest-n) still holds since
connectives ends at n=1 post-dedup, still the fixture's minimum.
Full adversarial suite: 334 pass, 1 documented xfail, 0 regressions. All
16 blocking gates verified green; behavioral-tune/rubric-judge (non-
blocking) skipped -- calibrate has no shared-benchmark skill rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Implements WP10b on top of the WP10a voice scorer.
Teach (scripts/voice_card.py): distills a profile + samples into a LAYERED
voice card — a <=300-word always-loaded card.md core with a "when writing X,
read card/X.md" index table, plus one card/<situation>.md sheet per COVERED
situation from a 10-dimension taxonomy. Uncovered dimensions get no sheet and
are named in card.md, so the card never fabricates a voice without sample
evidence. --coverage emits a deterministic lexical coverage matrix that drives
teach prompts (misclassification only adds/drops sheets, never card claims).
--provenance writes an auditable sha256 manifest. Byte-identical outputs.
Mimic --refine (evals/run_mimic_refine.py): iterative hill-climb with seeded
A/DEV document splits, hard removal gates per candidate (banned-phrase,
structure, draft->candidate preservation, copy-gate vs A, 150-word floor),
DEV-only acceptance, and a divergence guard that halts on the reward-hacking
signature (A improves while DEV worsens two rounds) with reward_hacking_warning.
Fully deterministic in --candidates-dir dry-run mode. Derives ranked directives
with paired card-amendment lines; writes report.json, final.md, and the refined
card.
Stats (evals/mimic_stats.py): per-item paired deltas, BCa bootstrap, and
sign-flip permutation; improved iff CI_low>0 AND p<0.05.
Eval-first: MIMIC-01..09 and CARD-01..07 rows added to adversarial-evals.json
first (RED before check_mimic.py existed), driven by evals/check_mimic.py over
committed dry-run fixtures. New "mimic-logic" gate wired into list_gates and
CHECKS.md regenerated. references/mimic.md replaces the onslaught stub; SKILL.md
gains a Teach & Mimic subsection within the DOC-05 budget; .unslop/ gitignored.
Stdlib-only, no new xfail, all gates green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Adds scripts/calibrate_pairs.py (deterministic dimension-controlled minimal
pairs for contractions, em_dash, sentence_length, connectives, staccato --
constraint-preserving, exits 3 when a dimension isn't expressible in the given
passage) and scripts/calibrate_score.py (Wilson-lower-bound aggregation over a
preferences JSONL, insufficient-below-k gating, --next dimension ordering, and
--profile conflict detection against a measured stylometric profile).
references/calibrate.md documents the agent-hosted game flow and provenance
discipline (stated-preference vs measured-from-samples); SKILL.md gets one
pointer line. Eval rows CAL-01..08 cover per-dimension transform correctness,
the inexpressible-dimension exit code, seed determinism, canned-stream
aggregation, below-k insufficiency, conflict detection, --next tiebreaking,
and the missing-file exit code. Wired a calibrate-suite gate (--only CAL) into
list_gates() and regenerated the CHECKS.md matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
- docs/PRODUCT.md: the grill-session decisions as durable doctrine; passes all
three scanners (and dogfooding it surfaced the carve-out below)
- references/pipeline.md: Model Parity results table populated from the live
2026-07-06 benches; tiering guidance updated to measured conclusions
- Bare 'research indicates/shows/suggests' joins vague_attribution
(FN-31/FP-84/REC-49, eval-first) — caught live when a model paraphrased
around 'studies show' in the replacement bench
- silhouette_scan --genre docs suppresses callback_content only: doctrine/spec
register reprises opening themes by convention (SIL-9 pins the carve-out;
essay recap codas still flag by default)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scripts/voice_profile.py + voice_score.py: char-3gram cosine, cosine delta,
sentence EMD, punctuation/contraction/MTLD/word-length distances, z-normalized
vs an impostor pool, General Impostors rank as the dominant gaming-resistant
term; copy gate (4-gram overlap + LCS); seeded determinism. Three synthetic
author corpora (amara/boris/celia) as measurement fixtures; VOICE-01..11 rows
(3x3 separation, GI sanity, gaming guard, copy gate, determinism, drift-pinned
profiles via check_voice).
Adversarially verified: four fresh mechanical attacks lose by >=0.85 composite;
separation diagonal wins at 5 seeds. Known follow-ups (approved-with-fixes,
queued): replace the mislabeled stuffed fixture with a true mechanical attack,
same-genre impostor pool, real --background calibration for delta z-scores,
feature-space GI subsets, full length-floor nulling, function-word dedupe,
IDF-weighted char-3grams.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ship scripts/silhouette_scan.py: a macro-structure scanner one level above
structure_scan that scores idea arrangement, not surface cadence. Five
validated one-sided tells (scaffold_opener_share, callback_content,
role_entropy_bits, preview_fulfillment, heading_preview) reused verbatim from
the research prototype, composited as a weighted relu-of-z penalty against a
committed human reference; flags at silhouette_penalty >= 1.0.
Scale deviation (documented in the scanner and reference): these tells are
degenerate at zero across the human corpus, so a literal sample-IQR denominator
with a 0.05 floor over-fires (2/8 human false positives). The scorer scales each
metric by max(sample_iqr, human_fence) instead, which keeps the exact
weighted-relu shape, reproduces the research struct01/03/09/11 signature, and
holds 0/8 human false positives while catching 12/12 ai (>= the 8/12 target).
- evals/check_silhouette.py: --reference drift gate (regenerates
human_reference.json from named in-repo sources and asserts equality) and
--separation gate (>= 8/12 ai, 0/8 human on the copied-in corpus).
- evals/fixtures/silhouette/: committed human_reference.json, self-contained
ai/human corpus, and SIL fixtures.
- Eval rows SIL-01..08 (RED-first) plus SIL-05a/b proving the silhouette +
structure_scan anti-gaming pairing against a cue-deletion attack, and DOC-07
for check_silhouette. Two gates wired into list_gates(); CHECKS.md regenerated.
- Docs: silhouette subsection in references/taboo-phrases.md, detector pointers
in pack-structure.md, and silhouette_scan added to the SKILL.md validation
list as blocking with a genre carve-out.
Voice-fingerprint (per-author median/scale) integration is noted for the
teach/mimic branch and deliberately left unimplemented.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Add agent-invoked co-writer mode and a graceful non-English decline, both
under the existing scanner constitution (eval-first; all gates green).
scripts/suggest.py
Emit LSP-style structured suggestions {span, severity, category, rationale,
suggested_replacement, phrased_as_question} from banned_phrase_scan +
structure_scan. Detection is deterministic; replacement generation is
DELEGATED (suggested_replacement null) with a --apply-replacements FILE mode
that merges externally-produced replacements and light-validates them. Soft
findings are phrased as questions. Deterministic order, non-overlapping spans.
scripts/check_suggestions.py
Blocking contract gates, each a named failure: span-minimality (edit touches
only its span; whole-sentence rewrites fail), replacement-scanner (each
replacement passes both scanners in isolation and in context), accept-all
(applying every suggestion yields a doc passing both scanners with
validate_preservation exit 0), span-overlap.
English-only decline
Cheap function-word heuristic (english_function_share / is_probably_english)
added identically to banned_phrase_scan.py and structure_scan.py. Below the
conservative threshold -> {"non_english": true, "violations": []}, exit 0 with
a stderr note. Threshold tuned so ESL English still scans.
Eval rows (RED before these scripts existed): SUGG-01..05 (suggestion emission,
soft-as-question, oversized-replacement rejection, accept-all safety, overlap
rejection) and LANG-01a/01b + LANG-02a/02b (Spanish declined by both scanners,
ESL English scanned by both). Fixtures under evals/fixtures/suggest/.
SKILL.md gains a compact Co-writer Mode section (agent-invoked; cheap detection,
delegated replacements, blocking contract gates, suggestions surfaced never
silently applied).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
- evals/check_pattern_coverage.py: every structural pattern must match >=1
eval-row input and every banned phrase must appear in the row corpus (no
grandfathering — 28 DET rows, 28 REC coverage packs, 31 FP protection rows
added to reach 75/75 patterns, 313/313 phrases); every category must be
claimed by an FP row's protects field (52/52, backfilled)
- evals/kata_add_pattern.py: rehearses the maintenance procedure in a temp
copy and asserts each safety net fires in order (coverage -> parity -> green
-> red-on-removal); deterministic transcript
- Coverage gate immediately caught a dead pattern: filler_opener matched
literal 'So' against lowercased text; resurrected comma-gated and soft with
FP-83 protecting ordinary sentence-initial 'So'
- DOC-09/DOC-10 gates wired into --list-gates and CHECKS.md; maintenance.md
documents the conventions
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verifying a 3-headline document exposed a visibility bug: when a broad
anti_slop_register match spanned every headline line, the cross-category
containment de-dup filter erased all three headline_cadence occurrences, so the
document-level tell vanished on tight documents (and only survived in the FN
fixture by luck of span boundaries). A frequency-gated structural finding
(min_matches > 1) describes the whole document, not one span, so it must not be
suppressed by an unrelated larger match. Exempt those categories
(headline_cadence, fragment_template) from containment removal. The exemption
only prevents a frequency-gated finding from being dropped; it never adds or
removes any other category, so no other detection changes.
Full gate sweep still green: adversarial 200 PASS / 1 expected XFAIL / 0 FAIL;
shared benchmark unchanged; pack integrity, taboo parity, gates doc, skill
examples, and strict-leakage validate all pass. A tight 3-headline document now
reports headline_cadence x3; a 2-headline document stays below the gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AMn6wt4fMCxg7rxj9igFtS
Clayton Kim