A naive last_updated (no tz offset) passes fromisoformat cleanly but throws
TypeError at mtime > last_updated since mtime is always UTC-aware. Guard added
right after parse: if tzinfo is None, replace with UTC. No change for already-aware timestamps.
Closes#101
* feat(wiki-ingest): optional PageIndex long-PDF preprocessing
Implements the proposal from issue #97. When PAGEINDEX_REPO is set and a
PDF has >= PAGEINDEX_MIN_PAGES pages, wiki-ingest builds a structure-aware
TOC tree (section titles + summaries + page ranges) before reading, so only
relevant sections are fetched — yielding page-cited provenance at a fraction
of context cost. Falls back to direct page reads if unset or on error.
- wiki-ingest/SKILL.md: gated step-1 branch before the Academic papers section
- wiki-ingest/references/pageindex.md: full recipe (run command, JSON shape,
step-by-step reasoning approach, caching notes)
- .env.example: PAGEINDEX_REPO, PAGEINDEX_MODEL, PAGEINDEX_MIN_PAGES,
PAGEINDEX_WORKSPACE keys
Fully additive and backward-compatible — unset PAGEINDEX_REPO = no change.
Closes#97
* feat(ingest): add extract-jsonl.py pre-processor for 50-200x JSONL compression
Raw Claude Code JSONL files are 80-90% noise (tool_use, thinking, progress,
file-history-snapshot entries). This adds a stdlib pre-extraction script that
strips all noise and writes compact signal-only JSON, enabling the ingest skill
to process 5-10x more conversations per run within the same token budget.
Measured on real sessions:
- 559 KB JSONL → 8.6 KB extracted (65x)
- 12 MB JSONL → 64 KB extracted (188x)
scripts/extract-jsonl.py:
- Scans ~/.claude/projects/*/*.jsonl, extracts user + assistant text turns only
- Writes ~/.claude/extracted/<project>/<session-id>.json (compact JSON)
- --since DATE for incremental runs (only files modified after the date)
- --skip A,B to exclude projects (also reads WIKI_SKIP_PROJECTS env var)
- --dry-run / --verbose for inspection
- Own extract-manifest at ~/.claude/extracted/.extract-manifest.json tracks
which source files have been extracted, preventing redundant re-extraction
- Pure stdlib, no dependencies; follows the same conventions as manifest.py
.skills/claude-history-ingest/SKILL.md:
- New "Pre-extraction" section with usage examples and the extracted format
- Step 3 updated: check ~/.claude/extracted/<proj>/<uuid>.json first,
fall back to raw JSONL if not present
- Sampling heuristic fixed: projects with memory files now also ingest new
conversations not yet in the manifest (previously all conversations were
silently skipped once any memory file existed)
Source keys could be stored as both ~-relative and absolute paths, so the
same file got tracked twice and re-ingested in append mode (5 such collisions
found in a real 137-source vault). Also no supported way to exclude projects
from history ingest, and manifest updates were ad-hoc inline Python.
- scripts/manifest.py: stdlib helper. `normalize` rewrites source keys to
canonical absolute paths and merges collisions (keeps newest ingested_at,
unions pages lists); `delta` lists new/modified sources honoring
WIKI_SKIP_PROJECTS + --skip.
- llm-wiki/SKILL.md: canonical-source-key rule + provenance recording note;
WIKI_SKIP_PROJECTS in env table.
- wiki-status/SKILL.md: schema example now uses absolute paths consistently
(was mixing ~ and absolute); canonical-key callout.
- claude-history-ingest/SKILL.md: Project Scoping step, canonical-path compare
note (with helper usage), conversation sampling heuristic, desktop-session
pre-check, hot.md frontmatter-timestamp reminder.
- .env.example: WIKI_SKIP_PROJECTS.
Closes#86, #87, #88
When .manifest.json stores source paths as relative (the common case),
os.path.exists() would silently fail for any CWD other than the vault root —
including the launchd default of /, causing stale_count to always be 0.
Pass OBSIDIAN_VAULT_PATH into the Python heredoc's env and join it onto
any relative path before the existence check. Absolute and ~/... paths
are unaffected. Closes#35.
All skills now follow the shared Config Resolution Protocol defined in
llm-wiki/SKILL.md: walk up CWD for .env → ~/.obsidian-wiki/config →
prompt setup. Eliminates the drift between skills that read only .env,
only global config, or inconsistent combinations.
- llm-wiki: add canonical Config Resolution Protocol + vault-scoped
state algorithm; document all env vars
- 14 skills updated to reference the protocol instead of hard-coding
their own config loading logic
- daily-update: state files are now vault-scoped under
~/.obsidian-wiki/state/<vault-id>/ — multi-vault setups get
independent state, not a single global .last_update
- daily-update setup mode: terminal notification is now optional,
shell-aware (zsh/bash/fish), and skipped for headless/VPS envs
- scripts/daily-update.sh: implement CWD-walk config resolution in bash
- scripts/wiki-notify.sh: iterate all vault state dirs; shows stale
vaults by name; handles multi-vault setups
- memory-bridge: browse wiki by source tool; diff mode surfaces knowledge
blind spots between AI tools (Claude vs Codex vs Hermes etc.)
- impl-validator: quality-checking subagent spawned by other skills to
verify output correctness before presenting to the user
- daily-update: daily maintenance cycle (freshness check, index reconcile,
hot.md refresh) with launchd cron and zshrc terminal notification
- scripts/: daily-update.sh, wiki-notify.sh, launchd plist
All three skills registered across .claude, .kiro, .agents, .cursor, .windsurf.
source_type mappings verified against actual manifest specs in each history-ingest skill.
Arnav