The staleness order still let --since suppress a missing output: a source
older than the cutoff with no output file was skipped, leaving the very bug
this branch fixes (never-extracted older sessions) partly alive and
contradicting needs_extraction's own docstring.
Move the "output missing -> extract" check ahead of the --since gate, so a
missing output is always (re)extracted and --since only narrows files that
already have an output. Add a regression test: missing output + future
--since still extracts.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
cmd_normalize warned on relative keys but still ran them through
canonical() (os.path.abspath), which resolves them against the CWD and
rewrote e.g. "-Users-x-github/abc.jsonl" to "/cwd/-Users-x-github/abc.jsonl",
counted it as re-keyed, and persisted the corruption on a non-dry-run write.
Preserve relative keys untouched (ckey = key), skip the re-key count for
them, and only canonicalize absolute keys. normalize's dedup/canonicalize
purpose applies to absolute keys only; for relative keys it is now a safe
no-op. Collision merge logic is unchanged.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
extract-jsonl.py decided re-extraction from the .extract-manifest.json
extracted_at wallclock. A session whose output was missing or that landed
after the first run was treated as already-done and never re-extracted, so
downstream ingest saw it as missing until a manual rerun (e.g. session
2d83ed0d). --since additionally acted as a hard gate that hid never-extracted
older sessions.
Make needs_extraction idempotent on the output file: (re)extract when the
output is missing or the source is newer than it; --since only narrows the
scan range. The manifest (read by nothing else) is removed. Add
test_extract_staleness.py covering missing/newer/older cases.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
cmd_delta keyed every manifest source through canonical() (os.path.abspath),
which resolves relative keys (e.g. "-Users-x-github/abc.jsonl" under
~/.claude/projects/) against the CWD. Real vaults store most keys relatively,
so every already-ingested file was misreported as NEW (446 vs 384 known on the
live vault).
Add a basename-indexed suffix fallback: when a scanned absolute path has no
direct canonical match, look it up by basename and confirm the relative key is
a path suffix (O(1) avg, no O(n^2) scan). Absolute keys keep their exact prior
behavior. cmd_normalize is unchanged but now warns on relative keys, which it
would otherwise rewrite to CWD-based garbage.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
A naive last_updated (no tz offset) passes fromisoformat cleanly but throws
TypeError at mtime > last_updated since mtime is always UTC-aware. Guard added
right after parse: if tzinfo is None, replace with UTC. No change for already-aware timestamps.
Closes#101
* feat(wiki-ingest): optional PageIndex long-PDF preprocessing
Implements the proposal from issue #97. When PAGEINDEX_REPO is set and a
PDF has >= PAGEINDEX_MIN_PAGES pages, wiki-ingest builds a structure-aware
TOC tree (section titles + summaries + page ranges) before reading, so only
relevant sections are fetched — yielding page-cited provenance at a fraction
of context cost. Falls back to direct page reads if unset or on error.
- wiki-ingest/SKILL.md: gated step-1 branch before the Academic papers section
- wiki-ingest/references/pageindex.md: full recipe (run command, JSON shape,
step-by-step reasoning approach, caching notes)
- .env.example: PAGEINDEX_REPO, PAGEINDEX_MODEL, PAGEINDEX_MIN_PAGES,
PAGEINDEX_WORKSPACE keys
Fully additive and backward-compatible — unset PAGEINDEX_REPO = no change.
Closes#97
* feat(ingest): add extract-jsonl.py pre-processor for 50-200x JSONL compression
Raw Claude Code JSONL files are 80-90% noise (tool_use, thinking, progress,
file-history-snapshot entries). This adds a stdlib pre-extraction script that
strips all noise and writes compact signal-only JSON, enabling the ingest skill
to process 5-10x more conversations per run within the same token budget.
Measured on real sessions:
- 559 KB JSONL → 8.6 KB extracted (65x)
- 12 MB JSONL → 64 KB extracted (188x)
scripts/extract-jsonl.py:
- Scans ~/.claude/projects/*/*.jsonl, extracts user + assistant text turns only
- Writes ~/.claude/extracted/<project>/<session-id>.json (compact JSON)
- --since DATE for incremental runs (only files modified after the date)
- --skip A,B to exclude projects (also reads WIKI_SKIP_PROJECTS env var)
- --dry-run / --verbose for inspection
- Own extract-manifest at ~/.claude/extracted/.extract-manifest.json tracks
which source files have been extracted, preventing redundant re-extraction
- Pure stdlib, no dependencies; follows the same conventions as manifest.py
.skills/claude-history-ingest/SKILL.md:
- New "Pre-extraction" section with usage examples and the extracted format
- Step 3 updated: check ~/.claude/extracted/<proj>/<uuid>.json first,
fall back to raw JSONL if not present
- Sampling heuristic fixed: projects with memory files now also ingest new
conversations not yet in the manifest (previously all conversations were
silently skipped once any memory file existed)
Source keys could be stored as both ~-relative and absolute paths, so the
same file got tracked twice and re-ingested in append mode (5 such collisions
found in a real 137-source vault). Also no supported way to exclude projects
from history ingest, and manifest updates were ad-hoc inline Python.
- scripts/manifest.py: stdlib helper. `normalize` rewrites source keys to
canonical absolute paths and merges collisions (keeps newest ingested_at,
unions pages lists); `delta` lists new/modified sources honoring
WIKI_SKIP_PROJECTS + --skip.
- llm-wiki/SKILL.md: canonical-source-key rule + provenance recording note;
WIKI_SKIP_PROJECTS in env table.
- wiki-status/SKILL.md: schema example now uses absolute paths consistently
(was mixing ~ and absolute); canonical-key callout.
- claude-history-ingest/SKILL.md: Project Scoping step, canonical-path compare
note (with helper usage), conversation sampling heuristic, desktop-session
pre-check, hot.md frontmatter-timestamp reminder.
- .env.example: WIKI_SKIP_PROJECTS.
Closes#86, #87, #88
When .manifest.json stores source paths as relative (the common case),
os.path.exists() would silently fail for any CWD other than the vault root —
including the launchd default of /, causing stale_count to always be 0.
Pass OBSIDIAN_VAULT_PATH into the Python heredoc's env and join it onto
any relative path before the existence check. Absolute and ~/... paths
are unaffected. Closes#35.
All skills now follow the shared Config Resolution Protocol defined in
llm-wiki/SKILL.md: walk up CWD for .env → ~/.obsidian-wiki/config →
prompt setup. Eliminates the drift between skills that read only .env,
only global config, or inconsistent combinations.
- llm-wiki: add canonical Config Resolution Protocol + vault-scoped
state algorithm; document all env vars
- 14 skills updated to reference the protocol instead of hard-coding
their own config loading logic
- daily-update: state files are now vault-scoped under
~/.obsidian-wiki/state/<vault-id>/ — multi-vault setups get
independent state, not a single global .last_update
- daily-update setup mode: terminal notification is now optional,
shell-aware (zsh/bash/fish), and skipped for headless/VPS envs
- scripts/daily-update.sh: implement CWD-walk config resolution in bash
- scripts/wiki-notify.sh: iterate all vault state dirs; shows stale
vaults by name; handles multi-vault setups
- memory-bridge: browse wiki by source tool; diff mode surfaces knowledge
blind spots between AI tools (Claude vs Codex vs Hermes etc.)
- impl-validator: quality-checking subagent spawned by other skills to
verify output correctness before presenting to the user
- daily-update: daily maintenance cycle (freshness check, index reconcile,
hot.md refresh) with launchd cron and zshrc terminal notification
- scripts/: daily-update.sh, wiki-notify.sh, launchd plist
All three skills registered across .claude, .kiro, .agents, .cursor, .windsurf.
source_type mappings verified against actual manifest specs in each history-ingest skill.
Yician Wang