The staleness order still let --since suppress a missing output: a source
older than the cutoff with no output file was skipped, leaving the very bug
this branch fixes (never-extracted older sessions) partly alive and
contradicting needs_extraction's own docstring.
Move the "output missing -> extract" check ahead of the --since gate, so a
missing output is always (re)extracted and --since only narrows files that
already have an output. Add a regression test: missing output + future
--since still extracts.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
extract-jsonl.py decided re-extraction from the .extract-manifest.json
extracted_at wallclock. A session whose output was missing or that landed
after the first run was treated as already-done and never re-extracted, so
downstream ingest saw it as missing until a manual rerun (e.g. session
2d83ed0d). --since additionally acted as a hard gate that hid never-extracted
older sessions.
Make needs_extraction idempotent on the output file: (re)extract when the
output is missing or the source is newer than it; --since only narrows the
scan range. The manifest (read by nothing else) is removed. Add
test_extract_staleness.py covering missing/newer/older cases.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
* feat(wiki-ingest): optional PageIndex long-PDF preprocessing
Implements the proposal from issue #97. When PAGEINDEX_REPO is set and a
PDF has >= PAGEINDEX_MIN_PAGES pages, wiki-ingest builds a structure-aware
TOC tree (section titles + summaries + page ranges) before reading, so only
relevant sections are fetched — yielding page-cited provenance at a fraction
of context cost. Falls back to direct page reads if unset or on error.
- wiki-ingest/SKILL.md: gated step-1 branch before the Academic papers section
- wiki-ingest/references/pageindex.md: full recipe (run command, JSON shape,
step-by-step reasoning approach, caching notes)
- .env.example: PAGEINDEX_REPO, PAGEINDEX_MODEL, PAGEINDEX_MIN_PAGES,
PAGEINDEX_WORKSPACE keys
Fully additive and backward-compatible — unset PAGEINDEX_REPO = no change.
Closes#97
* feat(ingest): add extract-jsonl.py pre-processor for 50-200x JSONL compression
Raw Claude Code JSONL files are 80-90% noise (tool_use, thinking, progress,
file-history-snapshot entries). This adds a stdlib pre-extraction script that
strips all noise and writes compact signal-only JSON, enabling the ingest skill
to process 5-10x more conversations per run within the same token budget.
Measured on real sessions:
- 559 KB JSONL → 8.6 KB extracted (65x)
- 12 MB JSONL → 64 KB extracted (188x)
scripts/extract-jsonl.py:
- Scans ~/.claude/projects/*/*.jsonl, extracts user + assistant text turns only
- Writes ~/.claude/extracted/<project>/<session-id>.json (compact JSON)
- --since DATE for incremental runs (only files modified after the date)
- --skip A,B to exclude projects (also reads WIKI_SKIP_PROJECTS env var)
- --dry-run / --verbose for inspection
- Own extract-manifest at ~/.claude/extracted/.extract-manifest.json tracks
which source files have been extracted, preventing redundant re-extraction
- Pure stdlib, no dependencies; follows the same conventions as manifest.py
.skills/claude-history-ingest/SKILL.md:
- New "Pre-extraction" section with usage examples and the extracted format
- Step 3 updated: check ~/.claude/extracted/<proj>/<uuid>.json first,
fall back to raw JSONL if not present
- Sampling heuristic fixed: projects with memory files now also ingest new
conversations not yet in the manifest (previously all conversations were
silently skipped once any memory file existed)
Yician Wang