mirror of
https://github.com/Ar9av/obsidian-wiki.git
synced 2026-09-14 20:36:34 +08:00
Arnav
77d74e0261
* feat(config): resolve global config from an XDG-style location (#172) The global config lived at ~/.obsidian-wiki, which isn't XDG-compliant. It now resolves to $XDG_CONFIG_HOME/obsidian-wiki (default ~/.config/obsidian-wiki). Existing installs are unaffected: if ~/.obsidian-wiki is present and the XDG path is not, the legacy directory is used as-is. No migration is required, and nothing moves on upgrade. Once the XDG path exists it wins, so `mv ~/.obsidian-wiki ~/.config/obsidian-wiki` is all a migration takes. Resolution is implemented identically in Python (cli.py), bash (setup.sh, scripts/daily-update.sh, scripts/wiki-notify.sh), and the Config Resolution Protocol that every skill defers to, so vault-scoped state and named `config.<name>` profiles stay in one place. * fix(config): stop setup from wiping user-added config keys `write_config` rewrote the global config with only the three setup-managed keys, so every `obsidian-wiki setup` — including the one a pip upgrade prompts for — silently destroyed anything else in the file: OBSIDIAN_LINK_FORMAT, QMD_WIKI_COLLECTION, WIKI_SKIP_PROJECTS, sync settings, and comments. Setup now owns only OBSIDIAN_VAULT_PATH, OBSIDIAN_WIKI_REPO and OBSIDIAN_WIKI_VERSION, updating those in place and carrying every other line over untouched. Duplicate definitions of a managed key collapse to one. Predates the XDG change (same behavior on main) but is fixed here because this branch already touches config writing, and the XDG work makes re-running setup more likely. Also pins PYTHONPATH in both new test files so they exercise this checkout rather than whatever obsidian_wiki happens to be installed.
163 lines
12 KiB
Markdown
163 lines
12 KiB
Markdown
# Architecture
|
||
|
||
The wiki is the artifact. The agent is the maintainer. Obsidian is the viewer.
|
||
|
||
No scripts run your knowledge pipeline — the skills are markdown files that tell an AI agent *how* to operate on your vault. The agent uses the same read/write/search tools it already has.
|
||
|
||
## The four stages
|
||
|
||
Every time you feed the brain, it runs through these:
|
||
|
||
### 1. Ingest
|
||
|
||
The agent reads your source material directly — markdown, PDFs (with page ranges), JSONL conversation exports, plain text logs, chat exports, meeting transcripts, and images (screenshots, whiteboard photos, diagrams; vision-capable model required). No preprocessing step, no pipeline to run. The agent reads the file the same way it reads code.
|
||
|
||
### 2. Pull information
|
||
|
||
From the raw source, the agent pulls out concepts, entities, claims, relationships, and open questions. A conversation about debugging a React hook yields a "stale closure" pattern. A research paper yields the key idea and its caveats. A work log yields decisions and their rationale. Noise gets dropped, signal gets kept.
|
||
|
||
Each page also gets a 1–2 sentence `summary:` in its frontmatter at write time — later queries use this to preview pages without opening them.
|
||
|
||
### 3. Merge
|
||
|
||
New knowledge merges against what's already there. If a concept page exists, the agent updates it: merging new information, noting contradictions, strengthening cross-references. If it's genuinely new, a page gets created. Nothing is duplicated. Sources are tracked in frontmatter so every claim stays attributable.
|
||
|
||
### 4. Schema
|
||
|
||
The schema isn't fixed upfront. It emerges from your sources and evolves as you add more. The agent maintains coherence: categories stay consistent, wikilinks point to real pages, the index reflects what's actually there. When you add a new domain, the schema expands without breaking what exists.
|
||
|
||
A `.manifest.json` tracks every source that's been ingested — path, timestamps, which pages it produced. On the next run, the agent computes the delta and only processes what's new or changed.
|
||
|
||
## The loop
|
||
|
||
1. Agent resolves the vault path (`@name` → `.env` → global config)
|
||
2. Agent reads `.manifest.json` to know what's already been done
|
||
3. Agent reads the relevant skill for instructions
|
||
4. Agent uses its built-in tools to do the work
|
||
5. Agent updates `.manifest.json`, `index.md`, `log.md`, and `hot.md`
|
||
6. Output is standard Obsidian-compatible markdown with frontmatter and `[[wikilinks]]`
|
||
|
||
## Vault structure
|
||
|
||
```
|
||
$OBSIDIAN_VAULT_PATH/
|
||
├── index.md # Master index — every page, always current
|
||
├── log.md # Chronological activity log
|
||
├── hot.md # ~500-word semantic snapshot of recent activity
|
||
├── .manifest.json # Ingest ledger: path, timestamps, pages produced
|
||
├── _meta/
|
||
│ ├── taxonomy.md # Controlled tag vocabulary
|
||
│ └── *.base # Obsidian Bases dashboard definitions
|
||
├── _insights.md # Graph analysis: hubs, bridges, dead ends
|
||
├── _raw/ # Staging — drop rough notes, next ingest promotes them
|
||
├── _staging/ # Review queue when WIKI_STAGED_WRITES=true
|
||
├── _archives/ # Timestamped snapshots for rebuild/restore
|
||
├── _readouts/ # Narrative readouts from wiki-narrate
|
||
├── concepts/ # Abstract ideas, patterns, mental models
|
||
├── entities/ # Concrete things — people, tools, libraries, companies
|
||
├── skills/ # How-to knowledge, techniques, procedures
|
||
├── references/ # Factual lookups — specs, APIs, configs
|
||
├── synthesis/ # Cross-cutting analysis connecting multiple concepts
|
||
├── journal/ # Time-bound entries — daily logs, session notes
|
||
└── projects/
|
||
└── <project-name>.md # One page per project, synced via wiki-update
|
||
```
|
||
|
||
Knowledge that's project-specific goes under `projects/`. Knowledge that's general goes in the global category directories. Both are cross-referenced with `[[wikilinks]]`.
|
||
|
||
Every page carries required frontmatter: `title`, `category`, `tags`, `sources`, `created`, `updated`.
|
||
|
||
`hot.md` deserves a mention — it's a running semantic snapshot every write skill updates, so the next session picks up where the last one left off without crawling the whole vault.
|
||
|
||
## Core principles
|
||
|
||
- **Compile, don't retrieve.** The wiki is pre-compiled knowledge. Update existing pages — don't append or duplicate.
|
||
- **Track everything.** `.manifest.json` after ingesting; `index.md`, `log.md`, and `hot.md` after any write.
|
||
- **Connect with `[[wikilinks]]`.** This is what makes it a knowledge graph rather than a folder of files.
|
||
- **Frontmatter is required.** Every page, every time.
|
||
- **Single source of truth.** Visibility tags shape how content surfaces — they never duplicate or separate it.
|
||
|
||
## What we added on top of Karpathy's pattern
|
||
|
||
The [original gist](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) is the seed: compile knowledge once into interconnected markdown and keep it current, instead of asking an LLM the same questions repeatedly or running RAG every time. Here's what got built around it.
|
||
|
||
- **Delta tracking.** A manifest tracks every source file ingested. Come back later and it computes the delta, processing only what's new or changed. You're not re-ingesting your document library every time.
|
||
|
||
- **Project-based organization.** Knowledge is filed under projects when project-specific, globally when not. Both cross-referenced. Ten codebases, ten spaces in the vault.
|
||
|
||
- **Archive and rebuild.** When the wiki drifts too far from its sources, archive the whole thing (timestamped snapshot, nothing lost) and rebuild. Or restore any previous archive.
|
||
|
||
- **Multi-agent ingest.** Documents, PDFs, Claude Code history, Codex sessions, Hermes memories, OpenClaw `MEMORY.md`, Pi sessions, Copilot CLI history, Windsurf data, ChatGPT exports, Slack logs, meeting transcripts, raw text. Dedicated skills for each agent, plus a catch-all for arbitrary exports.
|
||
|
||
- **Cross-agent targeted search.** `/wiki-codex "rust ownership"` from inside Claude Code finds your Codex sessions on that topic, extracts the relevant blobs, distills them into pages, and returns a synthesized answer. Topic-first, not session-first. Each agent has its own extraction strategy. Pair with `/memory-bridge diff` to see what each tool uniquely contributed.
|
||
|
||
- **Audit and lint.** Orphaned pages, broken wikilinks, stale content, contradictions, missing frontmatter — plus a dashboard of what's ingested vs. pending.
|
||
|
||
- **Identity resolution.** `wiki-dedup` finds pages covering the same concept under different names ("RSC" vs. "React Server Components") and merges them.
|
||
|
||
- **Automated cross-linking.** After ingest, the cross-linker scans for unlinked mentions and weaves them into the graph.
|
||
|
||
- **Tag taxonomy.** A controlled vocabulary in `_meta/taxonomy.md`, with a skill that audits and normalizes tags vault-wide.
|
||
|
||
- **Provenance tracking.** Every claim is tagged: extracted (default), `^[inferred]` (LLM synthesis), or `^[ambiguous]` (sources disagree). A `provenance:` block in frontmatter summarizes the mix per page, and `wiki-lint` flags pages drifting into mostly speculation. You can always tell what your wiki knows from what it guessed.
|
||
|
||
- **Trust ledger.** `obsidian-wiki trust-record` / `trust-check` record and validate human-approved confidence reviews against material fingerprints, so CI can gate on "a person actually checked this."
|
||
|
||
- **Multimodal sources.** Screenshots, whiteboard photos, slide captures, and diagrams ingest like text — visible text transcribed verbatim, interpreted content tagged as inferred.
|
||
|
||
- **Wiki insights.** `wiki-status` can analyze the shape of the vault itself: top hubs, bridge pages (nodes whose removal would partition the graph), tag cluster cohesion, scored surprising connections, a graph delta since last run, and questions the structure is uniquely positioned to answer. Output goes to `_insights.md`.
|
||
|
||
- **Graph export and import.** `wiki-export` turns the wikilink graph into `graph.json`, `graph.graphml` (Gephi/yEd), `cypher.txt` (Neo4j), a self-contained interactive `graph.html`, or an OKF bundle. `wiki-import` reads any of it back.
|
||
|
||
- **Tiered retrieval.** `wiki-query` reads titles, tags, and summaries first, opening page bodies only when the cheap pass can't answer. Say "quick answer" to force index-only mode. Query cost stays roughly flat from 20 pages to 2000.
|
||
|
||
- **Session brain.** A topic graph over your raw agent session history, so you can find the session where something happened. See [Session Brain](session-brain.md).
|
||
|
||
- **Staged writes.** Set `WIKI_STAGED_WRITES=true` and LLM-written pages queue in `_staging/` for review before landing in the live vault.
|
||
|
||
## Open Knowledge Format
|
||
|
||
The vault format is structurally conformant with [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) — markdown with YAML frontmatter, category subfolders, reserved `index.md`/`log.md`.
|
||
|
||
`wiki-export` (OKF mode) and `wiki-import` are the bridge: they translate between native frontmatter (`title`/`category`/`tags`/`sources`/`created`/`updated` + `summary`) and OKF (`type`/`title`/`description`/`resource`/`tags`/`timestamp`), making vaults exchangeable with any OKF tool.
|
||
|
||
The OKF round-trip is lossless. The `graph.json` round-trip is not — it carries structure, not page bodies.
|
||
|
||
## Repo layout
|
||
|
||
```
|
||
obsidian-wiki/
|
||
├── .skills/ # ← Canonical skill definitions (source of truth)
|
||
│ └── <skill-name>/SKILL.md # 39 skills — see docs/skills.md
|
||
│
|
||
├── obsidian_wiki/ # Python package — CLI, setup, sync, session brain
|
||
├── extensions/brain/ # Zero-build Chrome extension: capture + form fill
|
||
├── tools/check_readme_sync.py # Translation drift reporter
|
||
│
|
||
├── CLAUDE.md # Bootstrap → Claude Code / Kilocode (→ AGENTS.md)
|
||
├── GEMINI.md # Bootstrap → Gemini CLI (→ AGENTS.md)
|
||
├── AGENTS.md # Bootstrap → Codex, OpenCode, Aider, Droid, Trae, Hermes, OpenClaw
|
||
├── .hermes.md # Bootstrap → Hermes (symlink → AGENTS.md)
|
||
├── .cursor/rules/obsidian-wiki.mdc # Always-on → Cursor (alwaysApply: true)
|
||
├── .windsurf/rules/obsidian-wiki.md # Always-on → Windsurf
|
||
├── .kiro/steering/obsidian-wiki.md # Always-on → Kiro (inclusion: always)
|
||
├── .agent/rules/obsidian-wiki.md # Always-on → Google Antigravity
|
||
├── .agent/workflows/obsidian-wiki.md # Slash-command registry → Antigravity
|
||
├── .github/copilot-instructions.md # Always-on → GitHub Copilot (VS Code Chat)
|
||
│
|
||
├── .claude/skills/ → symlinks to .skills/* (created by setup)
|
||
├── .cursor/skills/ → symlinks to .skills/*
|
||
├── .windsurf/skills/ → symlinks to .skills/*
|
||
├── .agents/skills/ → symlinks to .skills/*
|
||
├── .pi/skills/ → symlinks to .skills/*
|
||
├── .kiro/skills/ → symlinks to .skills/*
|
||
│
|
||
├── setup.sh # One-command agent setup
|
||
├── .env.example # Configuration template
|
||
└── docs/ # You are here
|
||
```
|
||
|
||
Global symlink targets created by setup are listed in [Installation](installation.md#what-setupsh-wires-up).
|
||
|
||
For the full pattern — three-layer architecture, page templates, project organization — read [`.skills/llm-wiki/SKILL.md`](../.skills/llm-wiki/SKILL.md).
|