Files
ar9av__obsidian-wiki/obsidian_wiki/sync.py
Arnav 5cab5c16c4 feat(graph): make the vault graph queryable , and stop routing paths through index.md (#175)
* chore(sync): ignore the regenerable graph cache

New vaults get `.graph-cache.json` in their .gitignore — it is derived data
that churns on every graph change and should never enter history.

An existing .gitignore is the user's file and is left byte-identical; the
command prints a tip instead of editing it.

* feat(graph): bridges, cohesion, paths, neighbourhoods and a betweenness cache

Brings the graph analyser up to the algorithm family graphify runs on code
graphs, in pure Python (no networkx/numpy — the package stays dependency-free):

- `bridges` — exact Brandes betweenness centrality. The pages that sit on the
  most shortest paths; removing one fragments the vault.
- `cohesion` per community — intra-cluster edge density; < 0.15 with 5+ pages
  marks a fragmented cluster.
- `--path A B` — shortest wikilink chain between two pages.
- `--around PAGE --depth N --direction in|out|both` — N-hop neighbourhood;
  `--direction in` is the blast radius if the page were renamed or removed.
- `--diff-against` / `--snapshot` — compare against a previous `_insights.md`
  snapshot (added/removed pages and links, newly connected, lost incoming).
- `suggested_questions` — derived from bridges, sink hubs, isolates and
  low-cohesion clusters.

Two correctness fixes fall out of this:

- Vault bookkeeping files (`index`, `log`, `hot`, `_insights` at the vault
  root, plus `_meta/` and `_readouts/`) are no longer graph nodes. They link
  to nearly every page, so they dominated every degree and centrality ranking.
  A nested `concepts/index.md` is still an ordinary page.
- Pages are now iterated in sorted order rather than filesystem order, so
  results are reproducible across machines. That exposed a latent flaw in the
  label-propagation fallback: a sequential sweep cascades one label along
  chains and collapsed a barbell graph into a single community. Nodes are now
  visited in a seeded shuffle — deterministic, and clusters survive.

Betweenness is O(V*E) and the only expensive step in the module (~43s on a
5,000-page vault, versus 0.13s to parse it and 0.03s for communities), so it
is memoised in `.graph-cache.json`. The cache key is a hash of the graph
topology rather than file mtimes, which makes a stale hit impossible by
construction: retargeting any link changes the key, while editing prose
without touching links keeps the cached result valid. The file is only
written when the computation actually exceeded 0.5s, so small and medium
vaults never accumulate one; it is bounded to 3 keys, written atomically,
and ignored if corrupt.

Sampled betweenness (k pivots) was measured as an alternative and rejected:
25-50x faster but only 60-90% top-10 overlap and wrong ordering even on a
38-page vault, which is not acceptable for a ranked list a human reads.

* fix(query): stop routing paths through index.md; answer structural questions

`graph-query` is the front door `wiki-query` calls first, and it built its
own graph in graphrag.py with its own page-selection rules, its own slug
helper and its own BFS. Those rules did not exclude the vault bookkeeping
files, and `index.md` links to every page — so almost any two pages looked
about two hops apart and the returned "connection" was meaningless:

    owasp-agentic-quotes -> prismor-overview
    before: owasp-agentic-quotes -> index -> arnav -> prismor-overview
    after:  owasp-agentic-quotes -> agentic-governance-maturity
            -> prismor -> arnav -> prismor-overview

On a 38-page vault, 52 of 630 page pairs (8%) routed through a bookkeeping
file, and 51 of those were shorter than the real semantic path, so BFS
preferred them. The share grows with vault size.

graphrag now uses the shared page selection from graph_analysis and delegates
to its `shortest_path`, so there is one graph and one BFS in the codebase.

On top of that, `graph-query` now answers questions about the shape of the
vault, phrased in plain English, and returns them fully computed in a new
`graph` field (null for ordinary queries, so the response stays backwards
compatible):

  impact      "what breaks if I delete X" / "what depends on X"
  bridges     "which pages bridge my clusters" / "what would fragment my vault"
  hubs        "what's central" / "top hubs" / "my main topics"
  clusters    "what clusters do I have" / "how is my wiki organised"
  surprising  "surprising connections" / "unexpected links"

These are computed lazily: only `bridges` needs betweenness, so every other
intent stays as cheap as an ordinary lookup even on a 5,000-page vault
(<= 0.71s measured). A structural question that names an unresolvable page
falls back to ordinary retrieval rather than guessing.

* docs: document the structural intents, query modes and graph cache

- docs/cli.md: the new graph-analyse flags, the graph-query structural
  questions, and a section on the cache (what it stores, why the key is the
  graph topology, and when the file is written).
- wiki-query: route structural questions to graph-query and report the
  `graph` payload directly instead of re-deriving it from page reads.
- wiki-status: insights mode now reads bridges, cohesion, suggested questions
  and the graph delta straight from the analyser rather than instructing the
  model to compute them by hand.

* docs(readme): add the structural-query benchmark vs a plain agent

Same model, same vault, same plain-English questions — the only difference is
whether obsidian-wiki is installed: 81s -> 19s, 44% -> 83% correct, at
unchanged API cost.

Includes the per-question breakdown and, in a collapsed block, the method and
its limits: ground truth verified against networkx rather than this project's
own code, n=2 per cell on a single 38-page vault, and the one run in the
"with" column that failed outright. The accuracy gap is attributed to its
actual cause — the plain agent routing through index.md, which links to every
page.

Both READMEs updated together per the translation-parity rule.
2026-08-17 06:19:41 -07:00

125 lines
4.6 KiB
Python

"""GitHub sync for an Obsidian vault.
Single implementation of the git-sync flow, shared by `obsidian-wiki setup` /
`obsidian-wiki sync-setup` (pip/uv installs) and `setup.sh` (source/curl
installs) — see issue #153. Both entrypoints call into this module instead of
each maintaining their own copy of the git plumbing.
The vault's own `git remote` is the source of truth for "is sync configured
and where does it push" — nothing here caches the remote URL elsewhere, so
there's nothing to go stale if the user changes it with plain git commands.
"""
from __future__ import annotations
import subprocess
from datetime import datetime
from pathlib import Path
GITIGNORE_CONTENT = (
".obsidian/workspace.json\n"
".obsidian/workspace-mobile.json\n"
".obsidian/cache\n"
".trash/\n"
".graph-cache.json\n"
)
#: Derived files that should never be committed — they are regenerable and
#: churn on every graph change. Backfilled into vaults whose .gitignore
#: predates them.
_DERIVED_IGNORES = (".graph-cache.json",)
def _git(vault_path: Path, *args: str) -> subprocess.CompletedProcess:
return subprocess.run(
["git", "-C", str(vault_path), *args],
capture_output=True, text=True,
)
def get_remote(vault_path: Path) -> str | None:
"""Return the vault's `origin` remote URL, or None if unset/not a git repo."""
if not (vault_path / ".git").is_dir():
return None
proc = _git(vault_path, "remote", "get-url", "origin")
if proc.returncode != 0:
return None
return proc.stdout.strip() or None
def configure_sync(vault_path: Path, remote: str) -> list[str]:
"""Init git in the vault (if needed), write a default .gitignore, and wire
`origin` at `remote`. Returns human-readable confirmation lines.
Raises FileNotFoundError if vault_path doesn't exist, ValueError if remote
is blank.
"""
if not vault_path.is_dir():
raise FileNotFoundError(f"vault not found: {vault_path}")
remote = remote.strip()
if not remote:
raise ValueError("remote URL must not be blank")
messages = []
if not (vault_path / ".git").is_dir():
proc = _git(vault_path, "init", "-q")
if proc.returncode != 0:
raise RuntimeError(f"git init failed: {proc.stderr.strip()}")
messages.append("Initialized git repo in vault")
gitignore = vault_path / ".gitignore"
if not gitignore.is_file():
gitignore.write_text(GITIGNORE_CONTENT)
messages.append("Created .gitignore in vault")
else:
# An existing .gitignore is the user's file — never edit it. Just point
# out regenerable files that would otherwise be committed.
existing = gitignore.read_text(encoding="utf-8", errors="replace").split()
missing = [p for p in _DERIVED_IGNORES if p not in existing]
if missing:
messages.append(
"Tip: add "
+ ", ".join(missing)
+ " to .gitignore — regenerable, and it churns on every graph change"
)
has_origin = _git(vault_path, "remote", "get-url", "origin").returncode == 0
verb = "set-url" if has_origin else "add"
proc = _git(vault_path, "remote", verb, "origin", remote)
if proc.returncode != 0:
raise RuntimeError(f"git remote {verb} failed: {proc.stderr.strip()}")
messages.append(f"Git remote → {remote}")
return messages
def run_sync(vault_path: Path) -> tuple[int, str]:
"""Commit and push any pending vault changes. Returns (exit_code, message)."""
if not vault_path.is_dir():
return 1, f"wiki-sync: vault not found at '{vault_path}'"
if not (vault_path / ".git").is_dir():
return 1, "wiki-sync: vault is not a git repo — run `obsidian-wiki sync-setup <remote>` first"
add = _git(vault_path, "add", "-A")
if add.returncode != 0:
return 1, f"wiki-sync: git add failed: {add.stderr.strip()}"
diff = _git(vault_path, "diff", "--cached", "--quiet")
if diff.returncode == 0:
return 0, "wiki-sync: nothing to commit"
commit_msg = f"sync {datetime.now():%Y-%m-%d %H:%M}"
commit = _git(vault_path, "commit", "-m", commit_msg)
if commit.returncode != 0:
return 1, f"wiki-sync: git commit failed: {commit.stderr.strip()}"
has_upstream = _git(vault_path, "rev-parse", "--abbrev-ref", "@{u}").returncode == 0
push_args = ("push",) if has_upstream else ("push", "-u", "origin", "HEAD")
push = _git(vault_path, *push_args)
if push.returncode != 0:
return 1, f"wiki-sync: git push failed: {push.stderr.strip()}"
remote = get_remote(vault_path) or "origin"
return 0, f"wiki-sync: pushed to {remote}"