setup.sh had an interactive git-sync flow (git init, .gitignore, remote,
a generated sync.sh, optional cron) that `obsidian-wiki setup` — the entry
point pip/uv installs actually run — never had, so those installs silently
skipped it and .env.example documented a var nothing ever read.
Move the git plumbing into obsidian_wiki/sync.py (configure_sync/run_sync/
get_remote) and expose it as `obsidian-wiki sync-setup <url>` and
`obsidian-wiki sync`. `obsidian-wiki setup` now offers the same interactive
prompt setup.sh always had (or accepts --remote non-interactively), and
setup.sh's own prompt delegates into the same module via the CLI instead of
re-implementing it. Sync status is read live from the vault's own `git
remote` (via `obsidian-wiki info`) rather than a config var that could drift.
wiki-ingest previously walked repo folders with a hardcoded SKIP_DIRS
list, so custom-ignored paths (anything not literally node_modules/
.venv/dist/...) leaked into the ingest plan. batch-plan now detects a
`.git` folder and enumerates files via `git ls-files`, so a repo's own
.gitignore decides what's skipped. Docs updated to spell out the
"clone locally, add to OBSIDIAN_SOURCES_DIR" workflow for public/private
repos on any host.
Implements the proposal from issue #96. Turns a cluster of mature, curated
Obsidian wiki pages into a portable Agent Skill (SKILL.md + references/),
the inverse of wiki-capture.
- .skills/vault-skill-factory/SKILL.md: new skill with hard guardrails
(never writes to .skills/, never auto-installs), QMD-based cluster
discovery, maturity filtering, and provenance manifest.
- .env.example: SKILL_FACTORY_OUTPUT_DIR and SKILL_FACTORY_MATURITY keys.
- AGENTS.md: routing row for /vault-skill-factory and related phrasings.
Closes#96
* feat(wiki-ingest): optional PageIndex long-PDF preprocessing
Implements the proposal from issue #97. When PAGEINDEX_REPO is set and a
PDF has >= PAGEINDEX_MIN_PAGES pages, wiki-ingest builds a structure-aware
TOC tree (section titles + summaries + page ranges) before reading, so only
relevant sections are fetched — yielding page-cited provenance at a fraction
of context cost. Falls back to direct page reads if unset or on error.
- wiki-ingest/SKILL.md: gated step-1 branch before the Academic papers section
- wiki-ingest/references/pageindex.md: full recipe (run command, JSON shape,
step-by-step reasoning approach, caching notes)
- .env.example: PAGEINDEX_REPO, PAGEINDEX_MODEL, PAGEINDEX_MIN_PAGES,
PAGEINDEX_WORKSPACE keys
Fully additive and backward-compatible — unset PAGEINDEX_REPO = no change.
Closes#97
* feat(ingest): add extract-jsonl.py pre-processor for 50-200x JSONL compression
Raw Claude Code JSONL files are 80-90% noise (tool_use, thinking, progress,
file-history-snapshot entries). This adds a stdlib pre-extraction script that
strips all noise and writes compact signal-only JSON, enabling the ingest skill
to process 5-10x more conversations per run within the same token budget.
Measured on real sessions:
- 559 KB JSONL → 8.6 KB extracted (65x)
- 12 MB JSONL → 64 KB extracted (188x)
scripts/extract-jsonl.py:
- Scans ~/.claude/projects/*/*.jsonl, extracts user + assistant text turns only
- Writes ~/.claude/extracted/<project>/<session-id>.json (compact JSON)
- --since DATE for incremental runs (only files modified after the date)
- --skip A,B to exclude projects (also reads WIKI_SKIP_PROJECTS env var)
- --dry-run / --verbose for inspection
- Own extract-manifest at ~/.claude/extracted/.extract-manifest.json tracks
which source files have been extracted, preventing redundant re-extraction
- Pure stdlib, no dependencies; follows the same conventions as manifest.py
.skills/claude-history-ingest/SKILL.md:
- New "Pre-extraction" section with usage examples and the extracted format
- Step 3 updated: check ~/.claude/extracted/<proj>/<uuid>.json first,
fall back to raw JSONL if not present
- Sampling heuristic fixed: projects with memory files now also ingest new
conversations not yet in the manifest (previously all conversations were
silently skipped once any memory file existed)
Source keys could be stored as both ~-relative and absolute paths, so the
same file got tracked twice and re-ingested in append mode (5 such collisions
found in a real 137-source vault). Also no supported way to exclude projects
from history ingest, and manifest updates were ad-hoc inline Python.
- scripts/manifest.py: stdlib helper. `normalize` rewrites source keys to
canonical absolute paths and merges collisions (keeps newest ingested_at,
unions pages lists); `delta` lists new/modified sources honoring
WIKI_SKIP_PROJECTS + --skip.
- llm-wiki/SKILL.md: canonical-source-key rule + provenance recording note;
WIKI_SKIP_PROJECTS in env table.
- wiki-status/SKILL.md: schema example now uses absolute paths consistently
(was mixing ~ and absolute); canonical-key callout.
- claude-history-ingest/SKILL.md: Project Scoping step, canonical-path compare
note (with helper usage), conversation sampling heuristic, desktop-session
pre-check, hot.md frontmatter-timestamp reminder.
- .env.example: WIKI_SKIP_PROJECTS.
Closes#86, #87, #88
Adds an optional GitHub sync step to setup.sh that mirrors the common
`brainsync` alias pattern but reads vault path from ~/.obsidian-wiki/config.
- Prompts during setup: repo URL, shell alias, optional hourly cron
- git-inits vault if needed, writes .gitignore for Obsidian workspace files
- Writes ~/.obsidian-wiki/sync.sh (idempotent, skips if nothing staged)
- Adds wiki-sync alias to .zshrc/.bashrc on confirmation
- Summary line shows wiki-sync if sync was configured
- README: new "Syncing your vault to GitHub" section with manual setup notes
- .env.example: documents VAULT_GITHUB_REMOTE config key
Thanks for the well-structured PR — mirrors the existing `claude-history-ingest` / `codex-history-ingest` pattern cleanly and the Pi session-format details (tree JSONL with id/parentId, message roles, compaction/branch_summary) line up with pi.dev.
Reviewed:
- New `pi-history-ingest` skill, routing across `wiki-history-ingest`, `wiki-agent` (`/wiki-pi`), and `memory-bridge`
- `setup.sh`, `.env.example`, `AGENTS.md`, `README.md`, `SETUP.md`, and all four IDE rule files updated consistently
- 38 committed symlinks under `.pi/skills/` + 5 cross-agent symlinks
One small follow-up: the SKILL.md cites `references/pi-data-format.md` but the file isn't shipped (the claude/codex skills both ship that companion reference). I'll add it in a follow-up so the family stays consistent.
Adds a new `OBSIDIAN_LINK_FORMAT` config variable (default: `wikilink`)
that lets users opt into standard relative Markdown links instead of
Obsidian `[[wikilinks]]` for all newly generated or updated pages.
Closes#26
- .env.example: document QMD_WIKI_COLLECTION, QMD_PAPERS_COLLECTION, and
OBSIDIAN_RAW_DIR with install instructions and fallback notes
- wiki-ingest: add Raw Mode for promoting _raw/ drafts to wiki pages;
add no-QMD fallback note pointing to grep path and .env.example
- wiki-query: add no-QMD fallback note in Step 2b directing to grep path
- wiki-setup: create _raw/ on vault init; ask about QMD during setup
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Agent-readable skills for building and maintaining an AI-powered Obsidian
knowledge base using Karpathy's LLM Wiki pattern. No scripts, no API keys —
the agent is the LLM.
10 skills: llm-wiki, obsidian-setup, obsidian-ingest, data-ingest,
claude-history-ingest, wiki-status, wiki-rebuild, obsidian-query,
obsidian-lint, skill-creator.
Features: delta tracking via manifest, project-based organization,
archive/rebuild/restore, any text source ingestion.
Arnav