* feat(config): resolve global config from an XDG-style location (#172)
The global config lived at ~/.obsidian-wiki, which isn't XDG-compliant.
It now resolves to $XDG_CONFIG_HOME/obsidian-wiki (default
~/.config/obsidian-wiki).
Existing installs are unaffected: if ~/.obsidian-wiki is present and the
XDG path is not, the legacy directory is used as-is. No migration is
required, and nothing moves on upgrade. Once the XDG path exists it wins,
so `mv ~/.obsidian-wiki ~/.config/obsidian-wiki` is all a migration takes.
Resolution is implemented identically in Python (cli.py), bash (setup.sh,
scripts/daily-update.sh, scripts/wiki-notify.sh), and the Config
Resolution Protocol that every skill defers to, so vault-scoped state and
named `config.<name>` profiles stay in one place.
* fix(config): stop setup from wiping user-added config keys
`write_config` rewrote the global config with only the three setup-managed
keys, so every `obsidian-wiki setup` — including the one a pip upgrade
prompts for — silently destroyed anything else in the file:
OBSIDIAN_LINK_FORMAT, QMD_WIKI_COLLECTION, WIKI_SKIP_PROJECTS, sync
settings, and comments.
Setup now owns only OBSIDIAN_VAULT_PATH, OBSIDIAN_WIKI_REPO and
OBSIDIAN_WIKI_VERSION, updating those in place and carrying every other
line over untouched. Duplicate definitions of a managed key collapse to
one.
Predates the XDG change (same behavior on main) but is fixed here because
this branch already touches config writing, and the XDG work makes
re-running setup more likely.
Also pins PYTHONPATH in both new test files so they exercise this
checkout rather than whatever obsidian_wiki happens to be installed.
Two strands of work.
Session brain — a topic graph over raw agent session history, so you can
find the session where something happened instead of remembering its
UUID. Reads every Claude transcript plus the pruned sessions that survive
only as prompts in history.jsonl, clusters them with local TF-IDF and
community detection (no embeddings, no API calls, no new dependencies),
and writes a sidecar at ~/.claude/session-brain/ — the vault is never
touched. Ships five modules, the sessions-* CLI commands, an interactive
graph.html, and the /session-brain and /wiki-sessions skills. This is
retrieval, deliberately distinct from wiki-history-ingest, which distils
sessions into permanent vault pages.
Docs — README.md was 598 lines and put trust-check flags and a 17-row
agent matrix in front of anyone evaluating the project. Cut it to a 129-
line landing page (problem statement, 60-second install, four blocks of
runnable commands, graph screenshot, why-not-a-notes-folder) and move the
reference material into docs/: installation, agents, skills, cli,
configuration, architecture, session-brain, contributing. README_TW.md
mirrors the new structure; docs/ stays English-only for now.
The restructure also closed real drift:
- Ten skills existed but appeared in no reference — session-brain,
session-search, wiki-dedup, wiki-digest, wiki-import, wiki-context-pack,
wiki-stage-commit, wiki-switch, vault-skill-factory, and
obsidian-layout-adjustment.
- SETUP.md listed 12 of 39 skills and still claimed "no scripts, no
dependencies" despite the pip package. It is now a signpost into docs/.
- Eight config variables were undocumented anywhere, including
WIKI_STAGED_WRITES, which wiki-setup prompts for and three skills read.
Added to .env.example and docs/configuration.md.
- sessions-name and a number of CLI flags were missing from the docs.
Two details worth noting. README links are absolute GitHub URLs because
pyproject sets readme = "README.md" as the PyPI long_description, and PyPI
resolves relative hrefs against pypi.org rather than the repo — docs/
links would have 404'd on the package page. And three doc-drift guards
asserted their facts against README.md; since those facts moved rather
than disappeared, the assertions were repointed at the page that now owns
each one instead of padding content back into the README.
Also gitignore /vault/ (a local knowledge vault, not a fixture) and the
packed browser extension.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The staleness order still let --since suppress a missing output: a source
older than the cutoff with no output file was skipped, leaving the very bug
this branch fixes (never-extracted older sessions) partly alive and
contradicting needs_extraction's own docstring.
Move the "output missing -> extract" check ahead of the --since gate, so a
missing output is always (re)extracted and --since only narrows files that
already have an output. Add a regression test: missing output + future
--since still extracts.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
cmd_normalize warned on relative keys but still ran them through
canonical() (os.path.abspath), which resolves them against the CWD and
rewrote e.g. "-Users-x-github/abc.jsonl" to "/cwd/-Users-x-github/abc.jsonl",
counted it as re-keyed, and persisted the corruption on a non-dry-run write.
Preserve relative keys untouched (ckey = key), skip the re-key count for
them, and only canonicalize absolute keys. normalize's dedup/canonicalize
purpose applies to absolute keys only; for relative keys it is now a safe
no-op. Collision merge logic is unchanged.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
extract-jsonl.py decided re-extraction from the .extract-manifest.json
extracted_at wallclock. A session whose output was missing or that landed
after the first run was treated as already-done and never re-extracted, so
downstream ingest saw it as missing until a manual rerun (e.g. session
2d83ed0d). --since additionally acted as a hard gate that hid never-extracted
older sessions.
Make needs_extraction idempotent on the output file: (re)extract when the
output is missing or the source is newer than it; --since only narrows the
scan range. The manifest (read by nothing else) is removed. Add
test_extract_staleness.py covering missing/newer/older cases.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
cmd_delta keyed every manifest source through canonical() (os.path.abspath),
which resolves relative keys (e.g. "-Users-x-github/abc.jsonl" under
~/.claude/projects/) against the CWD. Real vaults store most keys relatively,
so every already-ingested file was misreported as NEW (446 vs 384 known on the
live vault).
Add a basename-indexed suffix fallback: when a scanned absolute path has no
direct canonical match, look it up by basename and confirm the relative key is
a path suffix (O(1) avg, no O(n^2) scan). Absolute keys keep their exact prior
behavior. cmd_normalize is unchanged but now warns on relative keys, which it
would otherwise rewrite to CWD-based garbage.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
A naive last_updated (no tz offset) passes fromisoformat cleanly but throws
TypeError at mtime > last_updated since mtime is always UTC-aware. Guard added
right after parse: if tzinfo is None, replace with UTC. No change for already-aware timestamps.
Closes#101
* feat(wiki-ingest): optional PageIndex long-PDF preprocessing
Implements the proposal from issue #97. When PAGEINDEX_REPO is set and a
PDF has >= PAGEINDEX_MIN_PAGES pages, wiki-ingest builds a structure-aware
TOC tree (section titles + summaries + page ranges) before reading, so only
relevant sections are fetched — yielding page-cited provenance at a fraction
of context cost. Falls back to direct page reads if unset or on error.
- wiki-ingest/SKILL.md: gated step-1 branch before the Academic papers section
- wiki-ingest/references/pageindex.md: full recipe (run command, JSON shape,
step-by-step reasoning approach, caching notes)
- .env.example: PAGEINDEX_REPO, PAGEINDEX_MODEL, PAGEINDEX_MIN_PAGES,
PAGEINDEX_WORKSPACE keys
Fully additive and backward-compatible — unset PAGEINDEX_REPO = no change.
Closes#97
* feat(ingest): add extract-jsonl.py pre-processor for 50-200x JSONL compression
Raw Claude Code JSONL files are 80-90% noise (tool_use, thinking, progress,
file-history-snapshot entries). This adds a stdlib pre-extraction script that
strips all noise and writes compact signal-only JSON, enabling the ingest skill
to process 5-10x more conversations per run within the same token budget.
Measured on real sessions:
- 559 KB JSONL → 8.6 KB extracted (65x)
- 12 MB JSONL → 64 KB extracted (188x)
scripts/extract-jsonl.py:
- Scans ~/.claude/projects/*/*.jsonl, extracts user + assistant text turns only
- Writes ~/.claude/extracted/<project>/<session-id>.json (compact JSON)
- --since DATE for incremental runs (only files modified after the date)
- --skip A,B to exclude projects (also reads WIKI_SKIP_PROJECTS env var)
- --dry-run / --verbose for inspection
- Own extract-manifest at ~/.claude/extracted/.extract-manifest.json tracks
which source files have been extracted, preventing redundant re-extraction
- Pure stdlib, no dependencies; follows the same conventions as manifest.py
.skills/claude-history-ingest/SKILL.md:
- New "Pre-extraction" section with usage examples and the extracted format
- Step 3 updated: check ~/.claude/extracted/<proj>/<uuid>.json first,
fall back to raw JSONL if not present
- Sampling heuristic fixed: projects with memory files now also ingest new
conversations not yet in the manifest (previously all conversations were
silently skipped once any memory file existed)
Source keys could be stored as both ~-relative and absolute paths, so the
same file got tracked twice and re-ingested in append mode (5 such collisions
found in a real 137-source vault). Also no supported way to exclude projects
from history ingest, and manifest updates were ad-hoc inline Python.
- scripts/manifest.py: stdlib helper. `normalize` rewrites source keys to
canonical absolute paths and merges collisions (keeps newest ingested_at,
unions pages lists); `delta` lists new/modified sources honoring
WIKI_SKIP_PROJECTS + --skip.
- llm-wiki/SKILL.md: canonical-source-key rule + provenance recording note;
WIKI_SKIP_PROJECTS in env table.
- wiki-status/SKILL.md: schema example now uses absolute paths consistently
(was mixing ~ and absolute); canonical-key callout.
- claude-history-ingest/SKILL.md: Project Scoping step, canonical-path compare
note (with helper usage), conversation sampling heuristic, desktop-session
pre-check, hot.md frontmatter-timestamp reminder.
- .env.example: WIKI_SKIP_PROJECTS.
Closes#86, #87, #88
When .manifest.json stores source paths as relative (the common case),
os.path.exists() would silently fail for any CWD other than the vault root —
including the launchd default of /, causing stale_count to always be 0.
Pass OBSIDIAN_VAULT_PATH into the Python heredoc's env and join it onto
any relative path before the existence check. Absolute and ~/... paths
are unaffected. Closes#35.
All skills now follow the shared Config Resolution Protocol defined in
llm-wiki/SKILL.md: walk up CWD for .env → ~/.obsidian-wiki/config →
prompt setup. Eliminates the drift between skills that read only .env,
only global config, or inconsistent combinations.
- llm-wiki: add canonical Config Resolution Protocol + vault-scoped
state algorithm; document all env vars
- 14 skills updated to reference the protocol instead of hard-coding
their own config loading logic
- daily-update: state files are now vault-scoped under
~/.obsidian-wiki/state/<vault-id>/ — multi-vault setups get
independent state, not a single global .last_update
- daily-update setup mode: terminal notification is now optional,
shell-aware (zsh/bash/fish), and skipped for headless/VPS envs
- scripts/daily-update.sh: implement CWD-walk config resolution in bash
- scripts/wiki-notify.sh: iterate all vault state dirs; shows stale
vaults by name; handles multi-vault setups
- memory-bridge: browse wiki by source tool; diff mode surfaces knowledge
blind spots between AI tools (Claude vs Codex vs Hermes etc.)
- impl-validator: quality-checking subagent spawned by other skills to
verify output correctness before presenting to the user
- daily-update: daily maintenance cycle (freshness check, index reconcile,
hot.md refresh) with launchd cron and zshrc terminal notification
- scripts/: daily-update.sh, wiki-notify.sh, launchd plist
All three skills registered across .claude, .kiro, .agents, .cursor, .windsurf.
source_type mappings verified against actual manifest specs in each history-ingest skill.
Arnav