mirror of
https://github.com/Ar9av/obsidian-wiki.git
synced 2026-09-14 20:36:34 +08:00
Arnav
db1bf4d9c4
* feat(wiki-ingest): optional PageIndex long-PDF preprocessing Implements the proposal from issue #97. When PAGEINDEX_REPO is set and a PDF has >= PAGEINDEX_MIN_PAGES pages, wiki-ingest builds a structure-aware TOC tree (section titles + summaries + page ranges) before reading, so only relevant sections are fetched — yielding page-cited provenance at a fraction of context cost. Falls back to direct page reads if unset or on error. - wiki-ingest/SKILL.md: gated step-1 branch before the Academic papers section - wiki-ingest/references/pageindex.md: full recipe (run command, JSON shape, step-by-step reasoning approach, caching notes) - .env.example: PAGEINDEX_REPO, PAGEINDEX_MODEL, PAGEINDEX_MIN_PAGES, PAGEINDEX_WORKSPACE keys Fully additive and backward-compatible — unset PAGEINDEX_REPO = no change. Closes #97 * feat(ingest): add extract-jsonl.py pre-processor for 50-200x JSONL compression Raw Claude Code JSONL files are 80-90% noise (tool_use, thinking, progress, file-history-snapshot entries). This adds a stdlib pre-extraction script that strips all noise and writes compact signal-only JSON, enabling the ingest skill to process 5-10x more conversations per run within the same token budget. Measured on real sessions: - 559 KB JSONL → 8.6 KB extracted (65x) - 12 MB JSONL → 64 KB extracted (188x) scripts/extract-jsonl.py: - Scans ~/.claude/projects/*/*.jsonl, extracts user + assistant text turns only - Writes ~/.claude/extracted/<project>/<session-id>.json (compact JSON) - --since DATE for incremental runs (only files modified after the date) - --skip A,B to exclude projects (also reads WIKI_SKIP_PROJECTS env var) - --dry-run / --verbose for inspection - Own extract-manifest at ~/.claude/extracted/.extract-manifest.json tracks which source files have been extracted, preventing redundant re-extraction - Pure stdlib, no dependencies; follows the same conventions as manifest.py .skills/claude-history-ingest/SKILL.md: - New "Pre-extraction" section with usage examples and the extracted format - Step 3 updated: check ~/.claude/extracted/<proj>/<uuid>.json first, fall back to raw JSONL if not present - Sampling heuristic fixed: projects with memory files now also ingest new conversations not yet in the manifest (previously all conversations were silently skipped once any memory file existed)