Two strands of work.
Session brain — a topic graph over raw agent session history, so you can
find the session where something happened instead of remembering its
UUID. Reads every Claude transcript plus the pruned sessions that survive
only as prompts in history.jsonl, clusters them with local TF-IDF and
community detection (no embeddings, no API calls, no new dependencies),
and writes a sidecar at ~/.claude/session-brain/ — the vault is never
touched. Ships five modules, the sessions-* CLI commands, an interactive
graph.html, and the /session-brain and /wiki-sessions skills. This is
retrieval, deliberately distinct from wiki-history-ingest, which distils
sessions into permanent vault pages.
Docs — README.md was 598 lines and put trust-check flags and a 17-row
agent matrix in front of anyone evaluating the project. Cut it to a 129-
line landing page (problem statement, 60-second install, four blocks of
runnable commands, graph screenshot, why-not-a-notes-folder) and move the
reference material into docs/: installation, agents, skills, cli,
configuration, architecture, session-brain, contributing. README_TW.md
mirrors the new structure; docs/ stays English-only for now.
The restructure also closed real drift:
- Ten skills existed but appeared in no reference — session-brain,
session-search, wiki-dedup, wiki-digest, wiki-import, wiki-context-pack,
wiki-stage-commit, wiki-switch, vault-skill-factory, and
obsidian-layout-adjustment.
- SETUP.md listed 12 of 39 skills and still claimed "no scripts, no
dependencies" despite the pip package. It is now a signpost into docs/.
- Eight config variables were undocumented anywhere, including
WIKI_STAGED_WRITES, which wiki-setup prompts for and three skills read.
Added to .env.example and docs/configuration.md.
- sessions-name and a number of CLI flags were missing from the docs.
Two details worth noting. README links are absolute GitHub URLs because
pyproject sets readme = "README.md" as the PyPI long_description, and PyPI
resolves relative hrefs against pypi.org rather than the repo — docs/
links would have 404'd on the package page. And three doc-drift guards
asserted their facts against README.md; since those facts moved rather
than disappeared, the assertions were repointed at the page that now owns
each one instead of padding content back into the README.
Also gitignore /vault/ (a local knowledge vault, not a fixture) and the
packed browser extension.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The staleness order still let --since suppress a missing output: a source
older than the cutoff with no output file was skipped, leaving the very bug
this branch fixes (never-extracted older sessions) partly alive and
contradicting needs_extraction's own docstring.
Move the "output missing -> extract" check ahead of the --since gate, so a
missing output is always (re)extracted and --since only narrows files that
already have an output. Add a regression test: missing output + future
--since still extracts.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
extract-jsonl.py decided re-extraction from the .extract-manifest.json
extracted_at wallclock. A session whose output was missing or that landed
after the first run was treated as already-done and never re-extracted, so
downstream ingest saw it as missing until a manual rerun (e.g. session
2d83ed0d). --since additionally acted as a hard gate that hid never-extracted
older sessions.
Make needs_extraction idempotent on the output file: (re)extract when the
output is missing or the source is newer than it; --since only narrows the
scan range. The manifest (read by nothing else) is removed. Add
test_extract_staleness.py covering missing/newer/older cases.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
* feat(wiki-ingest): optional PageIndex long-PDF preprocessing
Implements the proposal from issue #97. When PAGEINDEX_REPO is set and a
PDF has >= PAGEINDEX_MIN_PAGES pages, wiki-ingest builds a structure-aware
TOC tree (section titles + summaries + page ranges) before reading, so only
relevant sections are fetched — yielding page-cited provenance at a fraction
of context cost. Falls back to direct page reads if unset or on error.
- wiki-ingest/SKILL.md: gated step-1 branch before the Academic papers section
- wiki-ingest/references/pageindex.md: full recipe (run command, JSON shape,
step-by-step reasoning approach, caching notes)
- .env.example: PAGEINDEX_REPO, PAGEINDEX_MODEL, PAGEINDEX_MIN_PAGES,
PAGEINDEX_WORKSPACE keys
Fully additive and backward-compatible — unset PAGEINDEX_REPO = no change.
Closes#97
* feat(ingest): add extract-jsonl.py pre-processor for 50-200x JSONL compression
Raw Claude Code JSONL files are 80-90% noise (tool_use, thinking, progress,
file-history-snapshot entries). This adds a stdlib pre-extraction script that
strips all noise and writes compact signal-only JSON, enabling the ingest skill
to process 5-10x more conversations per run within the same token budget.
Measured on real sessions:
- 559 KB JSONL → 8.6 KB extracted (65x)
- 12 MB JSONL → 64 KB extracted (188x)
scripts/extract-jsonl.py:
- Scans ~/.claude/projects/*/*.jsonl, extracts user + assistant text turns only
- Writes ~/.claude/extracted/<project>/<session-id>.json (compact JSON)
- --since DATE for incremental runs (only files modified after the date)
- --skip A,B to exclude projects (also reads WIKI_SKIP_PROJECTS env var)
- --dry-run / --verbose for inspection
- Own extract-manifest at ~/.claude/extracted/.extract-manifest.json tracks
which source files have been extracted, preventing redundant re-extraction
- Pure stdlib, no dependencies; follows the same conventions as manifest.py
.skills/claude-history-ingest/SKILL.md:
- New "Pre-extraction" section with usage examples and the extracted format
- Step 3 updated: check ~/.claude/extracted/<proj>/<uuid>.json first,
fall back to raw JSONL if not present
- Sampling heuristic fixed: projects with memory files now also ingest new
conversations not yet in the manifest (previously all conversations were
silently skipped once any memory file existed)
ar9av