mirror of
https://github.com/Ar9av/obsidian-wiki.git
synced 2026-09-14 20:36:34 +08:00
fix/codegraph-source-json-gate
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 stemForge
|
81b18b649e |
fix(graphrag): strip trailing punctuation on the gap and list query paths (#190)
classify_query() strips punctuation on the direct path but not on the gap and list paths, where both end in .strip().split() -- and .strip() without arguments removes only whitespace. "What do I know about mise?" therefore yields the term "mise?", which matches no title, tag or summary, so the query returns nothing even when the page exists. Dropping the question mark makes the same query work, which is how the asymmetry surfaced. Both paths now go through a shared _split_terms() helper that splits on whitespace and strips the same punctuation set the direct path already used, now named _TERM_PUNCT. The direct path keeps its behaviour unchanged: its len(w) > 2 guard still applies to the unstripped token, so only the inline literal is replaced by the constant. Gap queries nearly always end in a question mark, so this path was broken for its ordinary input. Three regression tests cover the gap path, the list path, and agreement between the gap and direct paths on the same subject. Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Arnav
|
5cab5c16c4 |
feat(graph): make the vault graph queryable , and stop routing paths through index.md (#175)
* chore(sync): ignore the regenerable graph cache
New vaults get `.graph-cache.json` in their .gitignore — it is derived data
that churns on every graph change and should never enter history.
An existing .gitignore is the user's file and is left byte-identical; the
command prints a tip instead of editing it.
* feat(graph): bridges, cohesion, paths, neighbourhoods and a betweenness cache
Brings the graph analyser up to the algorithm family graphify runs on code
graphs, in pure Python (no networkx/numpy — the package stays dependency-free):
- `bridges` — exact Brandes betweenness centrality. The pages that sit on the
most shortest paths; removing one fragments the vault.
- `cohesion` per community — intra-cluster edge density; < 0.15 with 5+ pages
marks a fragmented cluster.
- `--path A B` — shortest wikilink chain between two pages.
- `--around PAGE --depth N --direction in|out|both` — N-hop neighbourhood;
`--direction in` is the blast radius if the page were renamed or removed.
- `--diff-against` / `--snapshot` — compare against a previous `_insights.md`
snapshot (added/removed pages and links, newly connected, lost incoming).
- `suggested_questions` — derived from bridges, sink hubs, isolates and
low-cohesion clusters.
Two correctness fixes fall out of this:
- Vault bookkeeping files (`index`, `log`, `hot`, `_insights` at the vault
root, plus `_meta/` and `_readouts/`) are no longer graph nodes. They link
to nearly every page, so they dominated every degree and centrality ranking.
A nested `concepts/index.md` is still an ordinary page.
- Pages are now iterated in sorted order rather than filesystem order, so
results are reproducible across machines. That exposed a latent flaw in the
label-propagation fallback: a sequential sweep cascades one label along
chains and collapsed a barbell graph into a single community. Nodes are now
visited in a seeded shuffle — deterministic, and clusters survive.
Betweenness is O(V*E) and the only expensive step in the module (~43s on a
5,000-page vault, versus 0.13s to parse it and 0.03s for communities), so it
is memoised in `.graph-cache.json`. The cache key is a hash of the graph
topology rather than file mtimes, which makes a stale hit impossible by
construction: retargeting any link changes the key, while editing prose
without touching links keeps the cached result valid. The file is only
written when the computation actually exceeded 0.5s, so small and medium
vaults never accumulate one; it is bounded to 3 keys, written atomically,
and ignored if corrupt.
Sampled betweenness (k pivots) was measured as an alternative and rejected:
25-50x faster but only 60-90% top-10 overlap and wrong ordering even on a
38-page vault, which is not acceptable for a ranked list a human reads.
* fix(query): stop routing paths through index.md; answer structural questions
`graph-query` is the front door `wiki-query` calls first, and it built its
own graph in graphrag.py with its own page-selection rules, its own slug
helper and its own BFS. Those rules did not exclude the vault bookkeeping
files, and `index.md` links to every page — so almost any two pages looked
about two hops apart and the returned "connection" was meaningless:
owasp-agentic-quotes -> prismor-overview
before: owasp-agentic-quotes -> index -> arnav -> prismor-overview
after: owasp-agentic-quotes -> agentic-governance-maturity
-> prismor -> arnav -> prismor-overview
On a 38-page vault, 52 of 630 page pairs (8%) routed through a bookkeeping
file, and 51 of those were shorter than the real semantic path, so BFS
preferred them. The share grows with vault size.
graphrag now uses the shared page selection from graph_analysis and delegates
to its `shortest_path`, so there is one graph and one BFS in the codebase.
On top of that, `graph-query` now answers questions about the shape of the
vault, phrased in plain English, and returns them fully computed in a new
`graph` field (null for ordinary queries, so the response stays backwards
compatible):
impact "what breaks if I delete X" / "what depends on X"
bridges "which pages bridge my clusters" / "what would fragment my vault"
hubs "what's central" / "top hubs" / "my main topics"
clusters "what clusters do I have" / "how is my wiki organised"
surprising "surprising connections" / "unexpected links"
These are computed lazily: only `bridges` needs betweenness, so every other
intent stays as cheap as an ordinary lookup even on a 5,000-page vault
(<= 0.71s measured). A structural question that names an unresolvable page
falls back to ordinary retrieval rather than guessing.
* docs: document the structural intents, query modes and graph cache
- docs/cli.md: the new graph-analyse flags, the graph-query structural
questions, and a section on the cache (what it stores, why the key is the
graph topology, and when the file is written).
- wiki-query: route structural questions to graph-query and report the
`graph` payload directly instead of re-deriving it from page reads.
- wiki-status: insights mode now reads bridges, cohesion, suggested questions
and the graph delta straight from the analyser rather than instructing the
model to compute them by hand.
* docs(readme): add the structural-query benchmark vs a plain agent
Same model, same vault, same plain-English questions — the only difference is
whether obsidian-wiki is installed: 81s -> 19s, 44% -> 83% correct, at
unchanged API cost.
Includes the per-question breakdown and, in a collapsed block, the method and
its limits: ground truth verified against networkx rather than this project's
own code, n=2 per cell on a single 38-page vault, and the one run in the
"with" column that failed outright. The accuracy gap is attributed to its
actual cause — the plain agent routing through index.md, which links to every
page.
Both READMEs updated together per the translation-parity rule.
|
||
|
Arnav
|
ea1633a8d4 |
fix(tests): isolate CLI subprocess tests, and actually run the suite in CI (#174)
* fix(tests): isolate CLI subprocess tests, and run the suite in CI
12 tests failed on a clean checkout. Neither was a product bug — both were
test-harness isolation gaps, and CI never caught them because the suite was
never run there.
1. Wrong package under test. The 18 test files that spawn
`python -m obsidian_wiki.cli` each build their own env from
os.environ.copy() and never set PYTHONPATH, so sys.path[0] was the
subprocess cwd. Any test running from a temp dir imported whatever
obsidian_wiki happened to be pip-installed instead of the working tree —
here a stale build predating `code-understand`, so all 8
test_code_understand_cli tests died at argparse with "invalid choice",
plus 4 in test_context_pack_cli. One autouse session fixture in
conftest.py now pins PYTHONPATH for every helper at once.
2. Developer .env leaking in. test_lint's `_run` and test_context_pack's
`run_cli` inherited pytest's cwd (the repo), so config resolution walked
up and found a developer's own untracked .env, pointing the run at their
real vault instead of the fixture. Both now run from the fake HOME.
Verified the fix isn't masking anything: sabotaging the config resolver makes
the relevant tests fail, and the suite passes in a venv with no obsidian_wiki
installed at all — so PYTHONPATH is genuinely doing the work.
Adds .github/workflows/tests.yml to run pytest on PRs and main. The workflows
only ran setup.sh, wheel publish, and the README-sync checker, so nothing
executed tests/ — the real reason this drifted.
Also drops the stale "RED by design / command does not exist yet" docstring
in test_code_understand_cli.py; the wiring landed in
|
||
|
ar9av
|
723ec49080 |
fix(graphrag): fold YAML block scalars when reading title/summary
_TITLE_RE and _SUMMARY_RE captured whatever followed `title:`/`summary:` on the same line, so a folded/literal block scalar (`>-`, `|-`, ...) returned the indicator itself instead of the text on the following indented lines — e.g. `summary: >-` read back as the literal string ">-". Replace both regexes with a shared _extract_scalar() that detects the block-scalar form and folds the continuation lines. Fixes #156 |
||
|
ar9av
|
a7fb1d69fe |
feat(graphrag): graph-index query pre-pass eliminates speculative page reads
Adds obsidian-wiki graph-query <vault> "<question>" — queries the compiled wikilink index before opening any page bodies, returning ranked candidates, a should_read shortlist, and multi-hop paths. Replaces the current approach of opening 5-10 pages speculatively per query. - New obsidian_wiki/graphrag.py: build_index() (frontmatter + wikilinks, in/out degree, tier), rank_candidates() (score = title/tag/summary match + degree bonus + tier weight), find_path() (BFS multi-hop, max_depth=4), classify_query() (direct/path/gap/list routing), query() top-level entry. - index_only=true when top candidate has exact title match + non-empty summary — skips page reads entirely for lookup queries. - CLI: graph-query <vault> "<question>" [--top N] [--max-read N] [--pretty] - wiki-query SKILL.md gets Step 0: run graph-query first; use should_read instead of speculative reads; skip to Step 5 when index_only=true. - 34 new tests: index building, ranking, path finding, query classification, integration, and CLI. |