Add two scanner patterns for marketing personification of machinery — an
inanimate tool/artifact subject given human agency — wild-caught by the owner
across his own marketing pages.
Tier A (tool_agency_reflexive, soft, fires anywhere): a tool-noun subject that
acts on itself directly after the verb — "The suite defends itself.", "The rules
update themselves.", "It graded its own reflection." Adjacency plus a "...out"
lookahead spare the dev idioms "the test cleans up after itself" and "it sorts
itself out"; ordinary tool-verb prose ("the gate fails the build") has no
reflexive object and stays clean.
Tier B (tool_agency_volitional, soft, standalone-line only): a strongly-volitional
verb (decides/hunts/wants/knows/believes/cares/refuses/judges/thinks) with a
tool-noun or bare "It" subject on a headline line — "The bench decides which
model does which job.", "It hunts instances, not word lists." The everyday
technical verbs (reads/runs/checks/returns/learns/...) are excluded so
"the parser reads the file" and "the model learns the distribution" stay clean;
the tool-noun requirement keeps human roles out ("the judge decides the case").
Eval-first per CLAUDE.md: OWNER-05/06 (FN), REC-51/52/53 (recall), FP-88..91
(domain protection) added before the scanner; full battery green (adversarial
478 pass / 1 xfail, coverage 82/82, packs, kata, taboo parity, schema, gates
doc, build --check, strict-leakage). Documented in taboo-phrases.md and
pack-voice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
FINDING: the cheapest model that recovers macro structure under the climb
is claude-haiku-4-5 -- it converges in 5 rounds given --max-rounds 8 (was
capped-at-1-residual at cap 4; one round short, not a hard ceiling). On the
GPT side only the frontier (gpt-5.5) converges; gpt-5.4-mini oscillates
(trajectories 4->1->5->2 at cap 4 and 3->3->2->1->6->2->1->5 at cap 8) and
never reaches clean. "Cheap tiers partially recover" was model-dependent,
not a property of cheap: read the two families separately.
Codex CLI as the GPT-side generator (OpenRouter still 402-blocked):
evals/model_generate.py gains a `codex` kind. codex exec is an agent CLI
whose stdout is an interleaved transcript, so call_codex extracts ONLY the
final message via -o/--output-last-message (verified clean by hand), runs
read-only/ephemeral, and wraps the call in its own process group with a
hard timeout that SIGKILLs the group -- the known silent-hang failure mode
becomes an honest failed round. CLIMB-07 gates the extraction and the hang
kill offline against a fake codex binary (evals/fixtures/climb/fake_codex.py).
All four live runs (gpt-5.5 cap 4 converged 4->1->1->0; gpt-5.4-mini caps 4
and 8 capped; haiku-4-5 cap 8 converged 1->1->1->1->0) preserved every
source fact every round, on the same SKILL-MACRO-01 fixture and settings as
the recorded matrix. pipeline.md's climb section gains the follow-up table
and the cheapest-viable doctrine; the tiering row and PRODUCT.md bullet
updated to the model-dependent reading.
Gates: adversarial 469 pass / 1 documented xfail (CLIMB-07 new), schema,
gates-doc, taboo parity, build --check, strict-leakage all green; 0 scanner
hits (hard or soft) on both edited docs. Behavioral tune: 33/34 tasks ran;
the one failure was a transient API server error mid-response on
SKILL-MACRO-01/with_skill (runner infra, not a model answer), which aborted
the non-blocking judge step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
THESIS: the recorded parity matrix showed single-pass macro-structure
self-checking failed on every model tested -- told in prose to drop a
conclusion coda / connective scaffold / outline echo, each shipped it
anyway. But structure_scan + silhouette_scan SEE those failures precisely,
so the failure is correctable even where it is not self-checkable.
PROOF: evals/run_structure_climb.py closes the loop -- generate -> scan both
-> turn each finding into a TARGETED directive that names WHERE and WHAT
(the model can't see the tell, so the directive locates it) -> feed the
directives + current draft back -> regenerate, capped at N rounds, with a
validate_preservation guard so climbing can never eat a fact. Honest
terminal states (converged / capped / preservation_violation) carry
distinct exit codes.
Core IP is the directive builder: a deterministic map from every macro flag
both scanners can emit to a location-naming instruction (coda -> names the
final paragraph; connective openers -> names the paragraph numbers;
callback_content -> names the recapped opening vocabulary). CLIMB-06 gates
1:1 flag->directive coverage.
Eval-first (red then green): six CLIMB-* rows drive the loop offline through
a mock generator (evals/fixtures/climb/) -- converge (monotonic consumption,
exit 0), capped-still-dirty (honest, nonzero), single-pass control (dirty
without the loop), preservation abort (names the dropped fact), directive
location-naming, and full directive coverage. Registered a blocking
mock-only gate (structure-climb) and synced evals/CHECKS.md.
LIVE (2026-07-07, same SKILL-MACRO-01 fixture as Task C, source-anchored
preservation): single pass ships macro tells on both Claude tiers; the climb
recovers macro on the frontier where single pass failed (sonnet-4-5
converged 2->1->0, facts preserved) -- macro is now machine-CORRECTED, not
merely machine-gated. On the cheap tier the recovery is partial: haiku-4-5
cleared silhouette immediately but oscillated on surface structure and
capped at one residual flag; honest numbers recorded. The OpenRouter
spectrum (gpt-5.5, glm-5.2) is pending on account billing (HTTP 402);
exact reproduction commands recorded in references/pipeline.md.
Docs: pipeline.md tiering macro row + measured-conclusions bullet updated to
machine-corrected; new "Macro structure under the climb" subsection with the
loop, directive builder, and before/after evidence; PRODUCT.md macro bullet;
one-line SKILL.md pointer in the validation flow.
Battery green: adversarial 468 pass / 1 documented xfail, schema, gates-doc,
coverage, kata, taboo parity, build --check, strict-leakage; 0 hard scanner
hits on edited docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Grounds the growth-flywheel description in the actual OWNER-01/slogan_fragment/
FP-86 specimen: the maintainer's own marketing-page miss, the red-first eval
row, the line-anchored regex fix, and the FP-86 protection, plus a pointer to
the CONTRIB-*/SLUG-01 rows that cover the contribute pipeline itself.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Live run of evals/run_model_parity.py against the current open-weights
flagships on OpenRouter. All three clear Task B (span replacement) 6/6;
GLM 5.2 hits 1.00 Task A recall, DeepSeek and Kimi 0.92 (one voice-span
miss each in fixture A4). Adds the models file and the dated Recorded
results subsection; also clears the two pre-existing em-dash scanner
flags so pipeline.md scans 0 hard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Three AI-ism patterns wild-caught by the owner on his own marketing page,
added eval-first (rows appended red, then made green):
- slogan_fragment (banned_phrase_scan STRUCTURAL_PATTERNS, soft): standalone
"N X, one Y." slogan-cadence line/header ("Four presets, one input.").
- spec_fragment (banned_phrase_scan STRUCTURAL_PATTERNS, soft): standalone
"N noun-phrase, past-participle ..." spec fragment ("Eight criteria,
scored 1 to 5.").
- every_template_openers (structure_scan, soft, min 2 paragraphs): repeated
paragraph-initial "Every <noun> <verb>" template opener; a lone
"Every child deserves a good school." stays clean.
Both banned-phrase patterns fire only on whole-line/header contexts, so
prose-embedded counts ("The unit has two bedrooms, one bath, and a den.",
"We rated eight criteria, scored 1 to 5, before deciding.") stay clean.
Regexes are linear (bounded interior runs, no adjacent unbounded overlaps).
Rows OWNER-01/02/03 (FN specimens) went red first, then green after the
scanner changes; OWNER-04 + FP-86/FP-87 guard the false-positive twins.
New categories mapped into pack-voice and documented in taboo-phrases.md.
Suite: 462 pass / 1 xfail / 0 fail. Coverage, schema, taboo parity, packs,
kata, seeded-docs, pairs, silhouette, voice, benchmark, model-parity green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
15 plans written against 217c218, executed by dispatched agents in isolated
worktrees, reviewed and approved individually, integrated and battery-proven
on advisor/integration. plans/README.md carries the full status ledger, the
findings surfaced during execution, and the one known follow-up (in-process
stdin shim needs a .buffer for 011's full speed win).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scan_for_violations recomputed every violation's character span from scratch
via text.splitlines() (quadratic) and re-checked containment against every
other span (quadratic) -- 45.9M comparisons on a 6k-sentence doc. Both also
derived positions independently of the actual regex match, so CRLF documents
(splitlines() strips \r\n as one break but the reconstruction assumed a
1-char terminator) and case-expanding folds (str.lower() on U+0130 "İ" adds
a char) could corrupt line/column/context.
Fix: carry match.start()/match.end() as the single source of truth.
- Precompute line_starts once over the original text; derive line/column/
context per match via bisect (O(log n) instead of re-splitting the doc).
- Match phrases and structural patterns case-insensitively directly against
scan_text (masking is length-preserving) instead of a separately lowered
copy, so match offsets are valid against the original text with no
re-derivation. Added _phrase_pattern_ci as a call-site-only cache so
_phrase_pattern's case-sensitive default is unchanged for other callers.
STRUCTURAL_PATTERNS phrase field is explicitly re-lowered to preserve its
pre-fix casing (it was always matched against a lowered copy before).
- Containment is now a single O(n log n) sweep (sort by start asc/end desc,
track the tallest end seen so far) instead of an O(n^2) any(). Verified
equivalent to the old any()-based filter by diffing both filters' output
across all 227 script-eval fixture texts plus synthetic duplicate/nested/
adjacent-span stress cases -- zero diffs.
Perf (scan_for_violations, "a testament to progress." sentence corpus):
8,000 sentences: ~10.3s -> 0.92s
10,000 sentences: ~14.8s -> 1.17s (target: < 1.5s)
Adds SPAN-01 (CRLF line/column), SPAN-02 (İ case-fold context), and SPAN-03
(8k-sentence perf, < 2.0s) to evals/adversarial-evals.json. SPAN-01 was
already correct at HEAD (the CRLF bug lived only in the now-deleted span
reconstruction, not the line/column computed directly for JSON output) --
verified it still catches a regression against a deliberately-broken scratch
variant. SPAN-02 and SPAN-03 were red before this change.
python3 evals/run_adversarial.py: 442 PASS, 1 XFAIL (FP-06, unchanged), 0 FAIL.
check_taboo_parity.py, check_silhouette.py, check_gates_doc.py,
check_packs.py, and check_skill_examples.py each hand-rolled a local
ROOT = Path(__file__).resolve().parent.parent instead of importing ROOT
from evals/_check_support.py, leaving three competing import precedents
with no written rule. Converts all five to the _check_support seam
(imports only, zero logic changes; every converted check's stdout/stderr/
exit is byte-identical before and after) and adds a "Writing a New Check"
subsection to evals/CHECKS.md plus a pointer line in CLAUDE.md/AGENTS.md's
"Add a New Pattern" section.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Plan 013: the Growth promise in docs/PRODUCT.md relied on someone
remembering to run wiki_sync, re-run the parity bench, and funnel misses
through contribute. This lands the three missing pieces:
- references/refresh.md: the agent-runnable refresh procedure — staleness
check, wiki-sync flow, parity-bench flow with the eval-gating rule,
every-miss-becomes-a-row routing through contribute, and result-recording
conventions (pipeline.md tables, TUNE-RESULTS.md style).
- scripts/refresh_status.py: stdlib, network-free staleness reporter.
JSON with wiki_sync / parity_bench / newest_pattern_row, each carrying
last/days/stale. Sources: .wiki_sync_state.json (last_timestamp, mtime
fallback), the "Live matrix recorded **YYYY-MM-DD**" line in
references/pipeline.md, and git log -1 --format=%cs on
evals/adversarial-evals.json (argv list, no shell). Thresholds (90/180/60
days) are commented guidance; always exits 0 — reporter, not gate.
- docs/DECISIONS.md: durable home for decided tradeoffs, each entry with
Decision / Why / Revisit-when / Source. Seeds the PRODUCT.md decisions
(agent-invoked only, English-only, no packaging, no rights checks,
removal-dominant axis) plus the commit-message-only ones: WP8 fixture
word-floor/length-balance queue (05f2363), voice impostor/background
calibration queue (41b60d1), protects-grain + build_report deferrals
(d6f6321), and FP-06 as the single intentional xfail (efd7e0e,
evals/CRITIQUE.md).
- One-line pointers to references/refresh.md from references/maintenance.md
and docs/PRODUCT.md (Growth).
All four touched/new docs pass banned_phrase_scan and
structure_scan --genre docs. Suite: 439 PASS / 1 XFAIL (FP-06) / 0 FAIL.
New decisions go in DECISIONS.md at decision time; the reporter is the
building block if the operator ever wants scheduled refresh outside this
repo.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
SPAN-01 pins CRLF line/column (already correct at HEAD; verified against
a deliberately-broken scratch variant that it does catch a regression).
SPAN-02 (İ case-fold) and SPAN-03 (8k-sentence perf, ~10s) are red,
exposing the offset and quadratic-cost bugs scan_for_violations has today.
Adds an in-process dispatcher to evals/run_adversarial.py that imports each
allowlisted scripts/*.py scanner once (importlib, cached) and invokes its
main() directly per case, instead of spawning a fresh python3 subprocess for
every one of the 440 script-target cases. Interpreter/import startup was
~50ms x 440 spawns (~22s of the ~26-32s baseline).
Allowlist (14 scripts, chosen by command-shape census, then extended past the
top-3 starting point to hit the runtime target): banned_phrase_scan.py,
structure_scan.py, validate_preservation.py, silhouette_scan.py,
readability_metrics.py, diff_check.py, harvest_samples.py,
calibrate_score.py, check_suggestions.py, extract_constraints.py, suggest.py,
harvest_classify.py, calibrate_pairs.py, voice_score.py. Together they cover
380/440 script cases; the remaining 60 (evals/*.py helpers and `python3 -c`
one-liners) stay on subprocess.
Global-state audit: grepped every allowlisted script for mutation
(.append/.update/.add/.extend/[key]=) against its module-level dict/list
constants (STOPWORDS, GENRE_SUPPRESSIONS, ROLE_CUES, _MAGNITUDES, PATTERNS,
CELLS, DIMENSIONS, POLES, WEIGHTS, etc.) — all are read-only after import
(.get/.values/.items only). The two scripts using `random` (calibrate_pairs,
voice_score) seed a local random.Random(seed) per call, not the global
generator. No leaking mutable state found; nothing needed resetting from the
runner.
Mechanism: per case, patch sys.argv (basename + args), sys.stdin (StringIO),
and os.chdir(ROOT) — matching the old subprocess cwd=ROOT — then call the
module's main() (detected via inspect.signature whether it takes an argv
list or reads sys.argv itself), under redirect_stdout/stderr, with a
SIGALRM-based timeout guard. Restoration of argv/stdin/cwd happens in a
finally so a case can never corrupt the next one's environment. Any dispatch
exception, or a case that ever times out, falls back to subprocess
transparently (the latter permanently, for the rest of the run) and is
counted; the run prints "dispatch: N in-process, M subprocess (K fallbacks)"
at the end. Added `--subprocess` to force the old path everywhere, and a
hidden `--repeat N` for the isolation audit.
Verification:
- `--only FP` in-process vs `--subprocess --only FP`: per-case status lines
identical (diff empty).
- Full suite in-process vs `--subprocess`: 439 PASS + 1 XFAIL both ways,
diff empty, 0 dispatch fallbacks.
- `--repeat 2`: identical results both passes in one process, 0 fallbacks —
no cross-case state leakage observed.
- Timing (best of several runs): baseline (--subprocess) ~28-32s wall;
dispatched ~10.2-10.7s wall, i.e. ~33-38% of baseline (target < 40%).
- Zero edits under scripts/; list_gates() untouched; `--only HARV/CONTRIB/
CAL` and `--only MIMIC --only CARD` gate subsets still pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
36 "# noqa: E402 / BLE001" markers in scripts/ and evals/ previously
annotated a lint gate that didn't exist (no config, no CI step) — a
false signal to anyone reading the code. Add ruff.toml selecting E, F,
BLE (the codes the noqa markers assume) and wire one CI step after
py_compile.
Style-only codes (E401/E501/E701/E702/E731) are disabled outright:
this codebase's small check_*.py helpers deliberately use compact
one-line style (semicolon joins, single-line if/lambda bodies), and
those codes fire on that choice rather than a defect.
Everything else that fired is a real, pre-existing finding, scoped to
its file via per-file-ignores rather than fixed here (out of scope for
this plan — baseline must pass today via rule selection, not source
edits). Worth a maintainer's attention later:
evals/check_contrib.py F821 `subprocess` referenced only in
a `from __future__ import
annotations` return type, never
imported; F401 unused `Path`.
evals/check_pattern_coverage.py F401 unused `Path`.
evals/check_seeded_docs.py BLE001 two bare `except Exception`.
evals/run_model_parity.py F401 unused `os`.
scripts/banned_phrase_scan.py F401 two unused `_lang` imports;
F601 `"needless to say"` key literal
repeated in BANNED_PHRASES (same
value both times, harmless but dead).
scripts/calibrate_score.py F401 unused `POLES`.
scripts/harvest_classify.py F401 unused `DATE_FLOOR`.
scripts/structure_scan.py F401 two unused `_lang` imports.
scripts/suggest.py F401 unused `re`.
scripts/voice_profile.py F401 unused `math`.
scripts/wiki_sync.py F541 three f-strings with no
placeholders.
Baseline is deliberately minimal; tightening is maintainer taste later.
All three scripts previously treated -h/--help as a filename argument:
check_suggestions.py --help printed "Missing file: --help" and exited 2;
readability_metrics.py and extract_constraints.py emitted a JSON error.
Every sibling script uses argparse and gets proper usage text.
Mirrors structure_scan.py's minimal shape (optional positional path,
stdin fallback). Existing invocations are byte-identical before/after
(verified via before/after snapshots on file-arg, stdin, and
missing-file cases) — only -h/--help behavior changes.
wiki_sync.py fetched and parsed external wikitext with no eval reaching
its parser or SECTION_MAP; a drift there would silently emit wrong
diffs/prompts. Add a thin --from-file seam to check/diff/prompt that
reads wikitext from disk, skips the network fetch, and skips the sync
state write, then add an invented offline fixture and WIKI-01..03 rows
exercising the mapped-section and unmapped-section paths through it.
The fixture pins the parser, not the world; live-page drift detection
remains cmd_check. Never copy real wikitext into fixtures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
structure_scan.py and silhouette_scan.py each carried a private copy of
words(), a markdown-to-prose stripper, and a paragraph splitter. The
strippers had drifted: structure blanked blockquote lines and
silhouette didn't; silhouette stripped **bold** and structure didn't.
Move words(), strip_markdown_for_prose(), and paragraphs() into
_lang.py as the UNION of both original code paths, gated behind
blank_blockquotes / strip_bold flags so each caller keeps its exact
prior behavior on purpose instead of silently converging. structure_scan
now calls paragraphs(text, blank_blockquotes=True); silhouette_scan calls
paragraphs(text, strip_bold=True). silhouette's STOPWORDS import from
structure_scan is untouched (out of scope).
Verified byte-identical: snapshotted `structure_scan.py` and
`silhouette_scan.py` stdout for all 12 AI fixtures, all 8 human
fixtures, and README.md before and after the swap (42 outputs) --
`diff -r` empty. check_silhouette.py --reference still reports
"reference ok: 5 metrics over 15 human sources"; --separation still
12/12 AI flagged / 0/8 human flagged with identical per-file penalties
(e.g. 05_essay.txt 14.90, 05_readme.txt human 0.62, unchanged). Full
adversarial suite: 439 PASS, 1 XFAIL (known), 0 FAIL. Dual-mode import
(package import and direct script invocation) verified for both
scanners.
Open maintainer question, deliberately unresolved: converge both
scanners on one prose view? Doing so changes silhouette's committed
reference and needs deliberate regeneration + separation re-verification
-- its own eval-first plan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Adds evals/LIVE-MIMIC-PROTOCOL.md: the acceptance bar for measuring mimic
against a real author's held-out writing, written before any live generation
runs. Covers corpus spec (N_teach vs N_held), the four conditions (single-pass
mimic, mimic+refine, retrieval few-shot, zero-shot), six committed
genre-matched task prompts, held-out scoring, paired mimic-vs-retrieval
statistics, and four pre-registered outcomes with roadmap consequences fixed
ahead of any number.
Section 8 wires the whole pipeline end to end against the synthetic voice
fixtures and the committed mock generator (no live model calls, no author
text) to prove the plumbing produces the report/stats shape the protocol
depends on. That rehearsal surfaced four real harness gaps, documented in
section 7 rather than patched (out of scope for this spike): the live path
has no way to omit the voice card for a true card-free baseline; scoring the
same candidate through run_mimic_refine.py's internal profile vs an
externally built one can disagree by a measurable margin due to dict
iteration-order tie-breaking in the function-word distance; the refine loop's
automated hard gates are a subset of the single-pass gate battery documented
in references/commands/mimic.md (missing readability_metrics.py and
diff_check.py); and there is no dedicated flag to reserve a genuinely unseen
held-out corpus.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
adversarial-evals.json is hand-merged and hand-numbered with no
validation: a duplicate id runs twice silently and collapses in the
EXPECTED_XFAIL comparison, and a malformed row crashes at runtime
instead of failing a gate cleanly. evals/check_evals_schema.py checks
row uniqueness, id shape, required keys, and per-target shape
(script command/assertions against check_assertion's real type set,
skill rows against build_shared_benchmark.py's loader), registered as
the new "evals-schema" gate (DOC-24).
CI previously named only 4 gate steps by hand while ~20 other blocking
gates ran only transitively through DOC-* rows, so deleting one DOC
row could silently drop a gate from CI. The workflow now parses
--list-gates and runs every blocking gate directly, skipping the
handful whose commands template a live rewrite session's own output
check (transformed.txt/original.txt aren't repo artifacts; their
scanner logic is already exercised by the suite's FP-*/STRUCT-*/SIL-*/
PRES-*/ROB-*/AS-* rows).
Eight scripts (banned_phrase_scan, structure_scan, silhouette_scan,
readability_metrics, extract_constraints, suggest, check_suggestions,
voice_score) raised a raw UnicodeDecodeError traceback on non-UTF-8
stdin or file input. Switch stdin reads to
sys.stdin.buffer.read().decode("utf-8", errors="replace") and CLI
file reads to errors="replace", matching the idiom already used by
voice_profile.py, voice_card.py, and run_mimic_refine.py.
harvest_samples.py aborted an entire batch when one sibling file had
bad encoding or was unreadable: apply errors="replace" at its three
read sites and wrap per-file dispatch in collect_sources with a
try/except (OSError, UnicodeDecodeError) that records a new
"unreadable" drop-stat and warning (reusing the existing
stats/warnings plumbing that already tracks "instruction-injection")
and continues with the rest of the batch.
evals/run_local.py tracebacked when the claude CLI binary was
missing; catch (FileNotFoundError, OSError) alongside the existing
TimeoutExpired handling and degrade that one task instead of crashing
the whole run.
Add six ENC-01..06 eval rows (evals/adversarial-evals.json) covering
each fix, plus a small harvest fixture
(evals/fixtures/harvest/fixture_bad_encoding/good.md) whose marker
sentence must survive a sibling bad-encoding file generated inline by
the eval row (not committed as binary). Confirmed red against
unfixed code via git stash before restoring the fixes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Read-only spike harness computes the five silhouette_scan.py metrics
over three synthetic voice authors plus the human silhouette fixtures
(as a 4th pseudo-author), then checks extraction (median/IQR
degeneracy), stability (jackknife + subsample swings vs. the
pre-registered 50%-of-fence threshold), and discrimination
(nearest-reference accuracy over held-out docs).
Recommendation: no-go for now. Discrimination is the decisive test and
finds no working signal once leakage (human_fixtures is itself a
human_reference.json build source) and degenerate-tie artifacts are
accounted for -- 2 of 3 independent synthetic authors never matched
their own held-out doc to their own reference. Nothing under scripts/
or evals/ is touched; this is measurement only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
contribute.py joined --pattern-name into a filesystem path before any
validation, so a value like ../evil escaped .unslop/contrib/ and wrote
four bundle files there before the only containment check (a
relative_to call in the success print) ever ran. Reject non-slug names
up front with the file's existing error-JSON convention, exit 2, no
write.
The exclamation_overuse structural pattern had two adjacent unbounded
quantifiers over overlapping character classes (\s is a subset of
[^.]), causing quadratic backtracking on a long unclosed exclamation
run (5.7s at 32k chars, in-process callers like harvest and refine run
it with no timeout). Excluding ! from the interior class removes the
overlap and keeps identical match spans on FP-37 and REC-17.
New rows SLUG-01 and REGX-01 pin both fixes; REGX-01 times the live
STRUCTURAL_PATTERNS entry directly so it tracks the production regex.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Clayton Kim