codegraph 1.5.0 never writes .codegraph/source.json, so initialized was
permanently False, doctor's warning was cosmetic-looking but real, and
ensure_index() could never reach its sync branch — the index silently
never updated. source.json is now optional: used as a mismatch guard
when present, no longer required for initialized.
Fixes#192
Windows agent installs can contain directory junctions, and standard accounts
often cannot create new symbolic links. Detect link-like entries safely, remove
junctions without traversing their targets, fall back to copied skills only for
WinError 1314, and make status output tolerant of GBK consoles.
Constraint: Windows users may lack Developer Mode or Create symbolic links privilege
Rejected: Require elevation or Developer Mode | blocks standard Windows accounts
Confidence: high
Scope-risk: moderate
Reversibility: clean
Directive: Keep --copy as the explicit deterministic mode; only WinError 1314 may trigger automatic fallback
Tested: pytest -q tests/test_cli_install.py tests/test_ast_extractor.py tests/test_cache.py tests/test_graph_analysis.py tests/test_graphrag.py (196 passed); py_compile; actual obsidian-wiki setup with the target vault (exit 0)
Not-tested: Full suite still has unrelated Windows doctor/environment failures
Related: #186#187#188
Co-authored-by: Chen Li <george_lichne@163.com>
classify_query() strips punctuation on the direct path but not on the gap and
list paths, where both end in .strip().split() -- and .strip() without arguments
removes only whitespace. "What do I know about mise?" therefore yields the term
"mise?", which matches no title, tag or summary, so the query returns nothing
even when the page exists. Dropping the question mark makes the same query work,
which is how the asymmetry surfaced.
Both paths now go through a shared _split_terms() helper that splits on
whitespace and strips the same punctuation set the direct path already used, now
named _TERM_PUNCT. The direct path keeps its behaviour unchanged: its len(w) > 2
guard still applies to the unstripped token, so only the inline literal is
replaced by the constant.
Gap queries nearly always end in a question mark, so this path was broken for
its ordinary input. Three regression tests cover the gap path, the list path,
and agreement between the gap and direct paths on the same subject.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Everything else here is local: skills on your machine, a vault on your disk.
This puts a vault behind a URL so an agent elsewhere can search and write it.
One container, one vault, one API key. The vault stays plain markdown on a
mounted volume, so Obsidian still opens it. The container does no LLM work --
search and packing delegate to the existing graphrag/context_pack modules, and
the caller's agent does the thinking.
- obsidian_wiki/server.py: four vault functions, five REST routes and four MCP
tools as transport over them. Auth is one bearer key; the process refuses to
start without WIKI_API_KEY unless WIKI_ALLOW_ANONYMOUS=1. Every path is
resolved inside the vault or refused.
- Dockerfile, docker-compose.yml, .dockerignore, and a ghcr push on v* tags.
- New [server] extra; the core wheel stays dependency-free.
- docs/deployment.md, plus the WIKI_* vars in configuration.md and .env.example.
Requires mcp>=2.0: 2.0 renamed FastMCP to MCPServer, and session_manager only
exists once streamable_http_app() has been called -- a mounted sub-app's own
lifespan is ignored, so it is wired into the FastAPI lifespan by hand.
Fixes#178. sha256_dir() used str(fp.relative_to(path)), which yields
backslash-separated paths on Windows and forward-slash on POSIX, and sorted
entries by that same OS-native string. Both the hash input and the iteration
order now use as_posix(), so a directory source hashes identically across
platforms and unchanged vaults no longer re-trigger incremental ingest after
a cross-platform sync.
Both were documented in .env.example and docs/configuration.md but never
read anywhere in the project (#179).
- wiki-ingest: Step 4's page-planning cap now reads
OBSIDIAN_MAX_PAGES_PER_INGEST (default 15) instead of a hard-coded
"10-15 pages" literal.
- daily-update: new Step 4a runs wiki-lint on the LINT_SCHEDULE cadence
(daily/weekly/manual, default weekly), tracked via a
$STATE_DIR/.last_lint timestamp, and folds the result into the log
line and user-facing report.
Static, idempotent postgres.sql output (wiki_pages/wiki_edges tables,
ON CONFLICT upserts) loadable via psql — no driver dependency needed.
Documents the format in README, README_TW, and docs/.
Introduces an optional vault file `references/research-backends.md`
that registers extra CLI retrieval backends — name, binary, invoke
template, when-to-use, cost tier, and env key. In Rounds 1–2 the
skill checks active backends against each angle/URL and folds their
output into the same claims/concepts/entities/contradictions extraction.
Key guarantees:
- No `research-backends.md` → zero change in behaviour
- Missing binary / unset env key / non-zero exit → backend skipped,
noted in run summary, research continues via WebSearch/WebFetch
- Free-first ordering: free backends evaluated before paid ones per angle
- A backend failure never aborts a run
Ships a starter YAML registry template (defuddle, yt-dlp, perplexity-sonar)
inline in the skill for users to copy into their vault.
Closes#106
The OBSIDIAN_RAW_DIR comment still described the pre-#113 behaviour. wiki-ingest
moves a promoted draft into _raw/_archived/ instead of deleting it
(.skills/wiki-ingest/SKILL.md), wiki-status excludes that subdirectory from
pending work, and llm-wiki states the same. .env.example was the last place
still promising deletion, which is the outcome #113 was filed about.
* docs(vault-skill-factory): resolve skill-creator scripts path for both install layouts
Step 4 pointed at $OBSIDIAN_WIKI_REPO/.skills/skill-creator/scripts/, which only
exists for a source checkout. For a wheel install, skills_dir() resolves to
<pkg>/_data/skills and OBSIDIAN_WIKI_REPO is set to its parent, so the bundled
skills sit under skills/ with no leading dot (pyproject force-include maps
".skills" to "obsidian_wiki/_data/skills").
List both layouts and take the first that exists, matching how wiki-setup
already documents the wiki-stop-capture.sh lookup.
* docs(vault-skill-factory): separate the layout paths from the script list
The two layout bullets and the three script-name bullets rendered as one
flat five-item list, so a reader could not tell locations from filenames.
---------
Co-authored-by: ar9av <ar9avg@gmail.com>
* chore(sync): ignore the regenerable graph cache
New vaults get `.graph-cache.json` in their .gitignore — it is derived data
that churns on every graph change and should never enter history.
An existing .gitignore is the user's file and is left byte-identical; the
command prints a tip instead of editing it.
* feat(graph): bridges, cohesion, paths, neighbourhoods and a betweenness cache
Brings the graph analyser up to the algorithm family graphify runs on code
graphs, in pure Python (no networkx/numpy — the package stays dependency-free):
- `bridges` — exact Brandes betweenness centrality. The pages that sit on the
most shortest paths; removing one fragments the vault.
- `cohesion` per community — intra-cluster edge density; < 0.15 with 5+ pages
marks a fragmented cluster.
- `--path A B` — shortest wikilink chain between two pages.
- `--around PAGE --depth N --direction in|out|both` — N-hop neighbourhood;
`--direction in` is the blast radius if the page were renamed or removed.
- `--diff-against` / `--snapshot` — compare against a previous `_insights.md`
snapshot (added/removed pages and links, newly connected, lost incoming).
- `suggested_questions` — derived from bridges, sink hubs, isolates and
low-cohesion clusters.
Two correctness fixes fall out of this:
- Vault bookkeeping files (`index`, `log`, `hot`, `_insights` at the vault
root, plus `_meta/` and `_readouts/`) are no longer graph nodes. They link
to nearly every page, so they dominated every degree and centrality ranking.
A nested `concepts/index.md` is still an ordinary page.
- Pages are now iterated in sorted order rather than filesystem order, so
results are reproducible across machines. That exposed a latent flaw in the
label-propagation fallback: a sequential sweep cascades one label along
chains and collapsed a barbell graph into a single community. Nodes are now
visited in a seeded shuffle — deterministic, and clusters survive.
Betweenness is O(V*E) and the only expensive step in the module (~43s on a
5,000-page vault, versus 0.13s to parse it and 0.03s for communities), so it
is memoised in `.graph-cache.json`. The cache key is a hash of the graph
topology rather than file mtimes, which makes a stale hit impossible by
construction: retargeting any link changes the key, while editing prose
without touching links keeps the cached result valid. The file is only
written when the computation actually exceeded 0.5s, so small and medium
vaults never accumulate one; it is bounded to 3 keys, written atomically,
and ignored if corrupt.
Sampled betweenness (k pivots) was measured as an alternative and rejected:
25-50x faster but only 60-90% top-10 overlap and wrong ordering even on a
38-page vault, which is not acceptable for a ranked list a human reads.
* fix(query): stop routing paths through index.md; answer structural questions
`graph-query` is the front door `wiki-query` calls first, and it built its
own graph in graphrag.py with its own page-selection rules, its own slug
helper and its own BFS. Those rules did not exclude the vault bookkeeping
files, and `index.md` links to every page — so almost any two pages looked
about two hops apart and the returned "connection" was meaningless:
owasp-agentic-quotes -> prismor-overview
before: owasp-agentic-quotes -> index -> arnav -> prismor-overview
after: owasp-agentic-quotes -> agentic-governance-maturity
-> prismor -> arnav -> prismor-overview
On a 38-page vault, 52 of 630 page pairs (8%) routed through a bookkeeping
file, and 51 of those were shorter than the real semantic path, so BFS
preferred them. The share grows with vault size.
graphrag now uses the shared page selection from graph_analysis and delegates
to its `shortest_path`, so there is one graph and one BFS in the codebase.
On top of that, `graph-query` now answers questions about the shape of the
vault, phrased in plain English, and returns them fully computed in a new
`graph` field (null for ordinary queries, so the response stays backwards
compatible):
impact "what breaks if I delete X" / "what depends on X"
bridges "which pages bridge my clusters" / "what would fragment my vault"
hubs "what's central" / "top hubs" / "my main topics"
clusters "what clusters do I have" / "how is my wiki organised"
surprising "surprising connections" / "unexpected links"
These are computed lazily: only `bridges` needs betweenness, so every other
intent stays as cheap as an ordinary lookup even on a 5,000-page vault
(<= 0.71s measured). A structural question that names an unresolvable page
falls back to ordinary retrieval rather than guessing.
* docs: document the structural intents, query modes and graph cache
- docs/cli.md: the new graph-analyse flags, the graph-query structural
questions, and a section on the cache (what it stores, why the key is the
graph topology, and when the file is written).
- wiki-query: route structural questions to graph-query and report the
`graph` payload directly instead of re-deriving it from page reads.
- wiki-status: insights mode now reads bridges, cohesion, suggested questions
and the graph delta straight from the analyser rather than instructing the
model to compute them by hand.
* docs(readme): add the structural-query benchmark vs a plain agent
Same model, same vault, same plain-English questions — the only difference is
whether obsidian-wiki is installed: 81s -> 19s, 44% -> 83% correct, at
unchanged API cost.
Includes the per-question breakdown and, in a collapsed block, the method and
its limits: ground truth verified against networkx rather than this project's
own code, n=2 per cell on a single 38-page vault, and the one run in the
"with" column that failed outright. The accuracy gap is attributed to its
actual cause — the plain agent routing through index.md, which links to every
page.
Both READMEs updated together per the translation-parity rule.
* fix(tests): isolate CLI subprocess tests, and run the suite in CI
12 tests failed on a clean checkout. Neither was a product bug — both were
test-harness isolation gaps, and CI never caught them because the suite was
never run there.
1. Wrong package under test. The 18 test files that spawn
`python -m obsidian_wiki.cli` each build their own env from
os.environ.copy() and never set PYTHONPATH, so sys.path[0] was the
subprocess cwd. Any test running from a temp dir imported whatever
obsidian_wiki happened to be pip-installed instead of the working tree —
here a stale build predating `code-understand`, so all 8
test_code_understand_cli tests died at argparse with "invalid choice",
plus 4 in test_context_pack_cli. One autouse session fixture in
conftest.py now pins PYTHONPATH for every helper at once.
2. Developer .env leaking in. test_lint's `_run` and test_context_pack's
`run_cli` inherited pytest's cwd (the repo), so config resolution walked
up and found a developer's own untracked .env, pointing the run at their
real vault instead of the fixture. Both now run from the fake HOME.
Verified the fix isn't masking anything: sabotaging the config resolver makes
the relevant tests fail, and the suite passes in a venv with no obsidian_wiki
installed at all — so PYTHONPATH is genuinely doing the work.
Adds .github/workflows/tests.yml to run pytest on PRs and main. The workflows
only ran setup.sh, wheel publish, and the README-sync checker, so nothing
executed tests/ — the real reason this drifted.
Also drops the stale "RED by design / command does not exist yet" docstring
in test_code_understand_cli.py; the wiring landed in fd30720.
566 passed, 1 skipped (was 554 passed / 12 failed).
* fix(tests): make three test modules importable on Python 3.9
test_batch, test_graph_analysis and test_graphrag annotate helpers with
`int | None` / `list[str] | None` but lack `from __future__ import
annotations`, so 3.9 evaluates the union at import time and collection dies
with "unsupported operand type(s) for |". Every module under obsidian_wiki/
already has the future import; only these three tests were missing it.
Surfaced by the new pytest CI job — the 3.9 leg (the requires-python floor)
failed at collection while 3.12 passed.
* feat(config): resolve global config from an XDG-style location (#172)
The global config lived at ~/.obsidian-wiki, which isn't XDG-compliant.
It now resolves to $XDG_CONFIG_HOME/obsidian-wiki (default
~/.config/obsidian-wiki).
Existing installs are unaffected: if ~/.obsidian-wiki is present and the
XDG path is not, the legacy directory is used as-is. No migration is
required, and nothing moves on upgrade. Once the XDG path exists it wins,
so `mv ~/.obsidian-wiki ~/.config/obsidian-wiki` is all a migration takes.
Resolution is implemented identically in Python (cli.py), bash (setup.sh,
scripts/daily-update.sh, scripts/wiki-notify.sh), and the Config
Resolution Protocol that every skill defers to, so vault-scoped state and
named `config.<name>` profiles stay in one place.
* fix(config): stop setup from wiping user-added config keys
`write_config` rewrote the global config with only the three setup-managed
keys, so every `obsidian-wiki setup` — including the one a pip upgrade
prompts for — silently destroyed anything else in the file:
OBSIDIAN_LINK_FORMAT, QMD_WIKI_COLLECTION, WIKI_SKIP_PROJECTS, sync
settings, and comments.
Setup now owns only OBSIDIAN_VAULT_PATH, OBSIDIAN_WIKI_REPO and
OBSIDIAN_WIKI_VERSION, updating those in place and carrying every other
line over untouched. Duplicate definitions of a managed key collapse to
one.
Predates the XDG change (same behavior on main) but is fixed here because
this branch already touches config writing, and the XDG work makes
re-running setup more likely.
Also pins PYTHONPATH in both new test files so they exercise this
checkout rather than whatever obsidian_wiki happens to be installed.
A host that answers `vaults` with an error is a staged copy older than the
popup, which looked exactly like "you only have one vault" — the picker
silently fell back to its single default option. A missing host stays silent,
since pressing Fill already explains that one.
Also document that a second vault needs a ~/.obsidian-wiki/config.<name>
profile before it can appear in the list.
The fill flow always used whatever ~/.obsidian-wiki/config resolved to, so
answering a work questionnaire from the work vault meant re-pointing the
wiki-switch symlink for everything else too. This is the extension's
equivalent of the @name override.
The host gains a `vaults` request that lists the active-config default plus
every ~/.obsidian-wiki/config.<name> profile with its resolved path and an
existence flag, and `fill` accepts a vault name. Named resolution never falls
back to the default: silently answering from the wrong vault is exactly what
picking a vault is meant to prevent.
The popup remembers the choice per browser profile, disables profiles whose
path is gone, and resets to the default (loudly) if the remembered one
disappears. If the host is not installed the picker stays on its single
default option, so fill behaves as before.
* test: add code-understanding fixtures (fake codegraph CLI, repo builder)
* feat: add code_understanding module with builtin + codegraph providers
Provider abstraction behind code_understand(): builtin uses the regex
ast_extractor plus rg cross-file references (dependency-free fallback);
codegraph shells out to the codegraph CLI with defensive JSON parsing.
Focus-map output is ranked symbols with file:line citations, capped by
max_symbols, and never contains source bodies.
* feat(cli): wire code-understand command and doctor code-understanding checks
New 'obsidian-wiki code-understand --project' subcommand emits the focus
map JSON; doctor --project gains 5 code-understanding capability checks
(builtin, rg, codegraph, codegraph-index, codegraph-fresh). Missing
optional CodeGraph reports warn (exit 0); explicit CODE_UNDERSTANDING_
BACKEND=codegraph with a broken backend fails (exit 1).
* feat(skills): add optional code-understanding focus-map step to wiki-update
Step 3b runs 'obsidian-wiki code-understand --since <last_commit_synced>'
between deciding what to distill and distilling. GUARD-soft-fails like
wiki-ingest Step 1c; mandates file:line citations for architectural
claims and a pruning rule that removes relationships no longer
supported by the codebase. Registers CODE_UNDERSTANDING_BACKEND and
CODE_UNDERSTANDING_CODEGRAPH_BIN in the llm-wiki env reference.
* docs: document CODE_UNDERSTANDING_BACKEND and code-understand command
* docs: add CodeGraph install steps; teach wiki-update to offer install
configuration.md gains a Setup subsection under Code understanding
(npm install -g @colbymchenry/codegraph + verify + CODE_UNDERSTANDING_
CODEGRAPH_BIN fallback). wiki-update Step 3b gains an optional offer:
when codegraph is missing and the user wants the enhanced backend, the
agent may install it (with consent) and re-run the step.
* docs(readme): note code-graph-aware /wiki-update (EN + zh-TW parity)
* docs(readme): clarify zh-TW /wiki-update codegraph note
* fix(codegraph): cap impact queries at 25 seeds on full-project scans
Full scans (no --changed/--since) seed from every tracked file, so an
unbounded impact loop fired one subprocess per symbol — hundreds of
codegraph CLI calls on real repos (observed 120s+ timeout on a 4.3k-node
project). Cap impact at 25 seeds, matching the existing callers/callees
cap (seeds[:5]), keeping full scans bounded while --changed/--since
delta scans stay exhaustive for their small seed sets.
Regression test: test_codegraph_caps_impact_queries_on_full_scan pins the
25-call ceiling (RED: 30 calls -> GREEN: <= 25).
* feat(skills): add standalone code-understand skill
Exposes the code-graph-assisted focus map directly to any coding
agent session, not just as wiki-update Step 3b — so it can be invoked
on demand ("help me understand this codebase", "what breaks if I
change X") independent of a wiki sync.
---------
Co-authored-by: ar9av <ar9avg@gmail.com>
Loading an unpacked extension needs a directory on disk, so pip users had no
way to reach the feature at all. The extension now ships under
_data/extension/ and `obsidian-wiki info` prints the path to point Chrome at.
install.sh now handles both origins: a checkout stages a copy of the package
(so the host survives the checkout later moving into a TCC-protected folder),
while a wheel install uses the already-importable package and stages nothing.
The private signing key moved out of the repo to ~/.obsidian-wiki/. This is
load-bearing, not tidiness: hatchling's force-include copies straight from the
working tree and honours neither .gitignore nor the target's exclude list, so
every file in a force-included directory ships. Verified empirically — the key
appeared in both wheel and sdist until it was moved out, and exclude patterns
did not stop it. pyproject now says so at the include site.
The brain-capture extension only went one way: browser -> vault. This adds
the return path — read the form on the current page, answer what the vault
supports, and leave the rest blank — and folds both directions into a single
two-tab extension renamed from "Brain Vault Capture" to "Brain".
Retrieval never calls a model. `obsidian-wiki context-pack --json` ranks the
vault deterministically and the model does one narrow job: map retrieved facts
onto fields. One `claude -p` / `codex exec` invocation per fill, billed to the
subscription the user already has. Nothing is ever submitted.
A native messaging host bridges the two, staged into ~/.obsidian-wiki/ rather
than run from the checkout. Chrome's children do not inherit your shell, and
all three consequences bite:
- macOS TCC: ~/Documents is mode 700 with an ACL, and a native host inherits
Chrome's TCC context, so a checkout there cannot be exec'd at all. Chrome
reports this as "Native host has exited", identical to a crash.
- PATH: Chrome passes a minimal PATH, hiding claude/codex in ~/.local/bin.
- USER/LOGNAME: Claude Code reads OAuth credentials from the macOS login
Keychain and needs these to find them, else "Not logged in" despite a valid
session.
The host bookends every run in its log so those failures stay distinguishable,
re-validates every returned fill against the real form schema (an injected
value cannot reach a field the page never advertised), and forwards the pack's
instruction_policy so vault excerpts stay data rather than instruction.
Form-schema extraction resolves labels via aria-labelledby before aria-label:
Google Forms puts aria-label="Your answer" on every paragraph field while the
real question sits behind aria-labelledby, so checking aria-label first
silently mislabelled exactly the textarea fields. Values are written through
the native setter with input/change dispatched, so React-controlled forms see
a genuine edit.
Docs: docs/browser-extension.md, linked from both READMEs and the docs index.
Also backfills the untranslated tagline from #159 into README_TW.md.
Track candidates_seen / candidates_used / dropped from the Step 2
frontmatter grep through the QMD semantic pass, surface them as a
Retrieval line in every answer, and log them in the Step 6 log.md
QUERY entry. Inspired by mnem's tokens_used/candidates_seen/dropped
retrieval transparency — lets users see when retrieval wasn't
exhaustive instead of silently trimming to the top-K.
The skill addressed a specific named user ("Dan") throughout its
instructions instead of "the user", left over from wherever it was
authored/copied from.
* fix(hooks): close 19 classifier escapes found by adversarial review
Tool-call layer:
- Task/Agent no longer blanket-neutral: a spawned subagent can edit files
or call write APIs, so only the read-only agent types (Explore, Plan)
keep the exemption; every other subagent_type is suspicious
- EnterWorktree/ExitWorktree removed from the neutral set (they create and
remove git branches/worktrees)
- read-verb-named SQL tools scan their serialized input: EXPLAIN ANALYZE
executes the statement, so DML keywords or "analyze": true in the
payload disable the exemption
- "prepare" added to write verbs (prepare_query_tuning provisions a
branch); "resolve" dropped from read verbs (resolving a review thread
mutates it)
Shell scanner:
- double-quote escaping now uses backslash parity: "a\\" closes the quote
(the backslashes escape each other), so a following command is seen
Command classifiers:
- python -c: .execute()/.executemany(), Path.open('w'), aliased os imports
(import os as x / from os import ...), case-insensitive SQL and pattern
matching (lowercase "delete from" caught)
- curl: --config/-K (config can carry any request), -Q/--quote (FTP
protocol commands like DELE), Q/K in short-flag clusters
- git: --upload-pack/--receive-pack/--exec inject remotely-executed
commands even under read-only subcommands like ls-remote
- date: BSD/macOS positional operand sets the clock (+format and -j forms
stay read-only)
- awk: redirects/pipes to variables (print > f, print | c), close(),
getline; numeric comparisons ($3 > 100) stay clean
- sed: w-detection works with any substitution delimiter (s#a#b#w out)
- xxd/uniq: a bare "-" stdin marker counts as positional, so the output
file argument is no longer missed
- less/more: -o/-O/--log-file copy input into a file
Twenty new transcript tests pin each escape and the read-only forms.
* feat(hooks): re-arm the capture nudge for long-running sessions
"Once per session" starves multi-day sessions: the single nudge fires
early, and everything after it is never captured automatically.
The sentinel now expires when BOTH hold: it is older than
WIKI_STOP_REARM_SECONDS (default 6h) and at least WIKI_STOP_REARM_EDITS
(default 10) new file edits happened since the nudge that claimed it.
Short sessions keep exactly one nudge; long busy sessions get roughly one
per work stretch; long idle sessions stay silent.
The edit count at nudge time is stored inside the sentinel directory;
sentinels predating the feature have no count and re-arm from zero. The
young-sentinel path stays the cheap pre-parse short-circuit, and an
expired sentinel is retired with an atomic mv so concurrent invocations
(duplicate hook registration) still produce exactly one re-nudge.
* fix(hooks): close a re-arm double-nudge race under duplicate registration
With the hook registered twice, both invocations can pass the sentinel
age check; the winner retires the expired sentinel, nudges, and claims a
fresh one — and the loser's mv then grabbed that FRESH sentinel and
nudged again.
After winning the mv, the invocation now verifies the retired sentinel
really is expired: a young one means a concurrent re-arm already nudged,
so it is moved back intact (count preserved) and the invocation stands
down. If the restore itself loses to yet another claim, the grabbed copy
is discarded and the newest claim wins.
A stress test runs 10 rounds of two concurrent invocations against an
expired sentinel and asserts exactly one nudge and a surviving sentinel
every round.
* fix(hooks): repair edit-count state when restoring a stolen fresh sentinel
The double-nudge guard restores a fresh sentinel it grabbed by mistake —
but if it grabbed it in the gap between the winner's mkdir and its count
write, the restored sentinel has no edits file, and the next re-arm would
compute its delta from zero and fire early.
Both invocations count the same stop's transcript, so the restorer's own
count is the correct stand-in: after a successful restore, write it if
the edits file is missing. The stress test now also asserts the surviving
sentinel holds the correct count every round, in every interleaving.
* test(hooks): make the re-arm race test actually concurrent
The stress test launched two hook processes but fed stdin through
sequential communicate() calls — and the hook's first statement is
INPUT=$(cat), so the second process blocked at that read until the first
had fully exited. The bodies ran one after another and the race path was
never exercised; the test only re-proved the sequential young-sentinel
path.
Both processes now get stdin from a pre-written payload file and execute
the full body simultaneously, with a 200-entry transcript so the parse
phase gives a wide overlap window between the age check and the claim.
Verified with bash -x tracing: both processes reach the claim phase in
every round (one wins the mv or mkdir, the other loses it or restores),
with exactly one nudge and intact edit-count state throughout.
* test(hooks): assert the race path actually executes, not just its outcome
The outcome invariants (one nudge, surviving sentinel, intact count) hold
in a serialized interleaving too, so the concurrent test could pass
without ever exercising the race. The hook now runs under bash -x and the
test inspects each process's xtrace: a process that reached the claim
phase shows an mv or mkdir on the sentinel path, while a serialized loser
exits at the top age check without touching it.
A round counts as raced only when BOTH processes reached the claim phase;
at least 3 of 10 rounds must race (unloaded machines hit 10/10, the slack
tolerates loaded-CI serialization) or the test fails loudly, flagging
that the harness degraded back to sequential execution.
* fix(hooks): fix BSD-first stat fallback returning wrong mtime on Linux
stat -f %m FILE || stat -c %Y FILE assumed the BSD form fails clean on
GNU/Linux. It doesn't: GNU stat's -f takes no format argument, so %m is
parsed as a second FILE operand. The command errors on that bogus
operand but still prints real filesystem info for the sentinel to
stdout, and exits nonzero overall — so `||` also runs stat -c, and
$(...) concatenates both outputs into one non-numeric string. The
^[0-9]+$ guard rejects that and silently falls back to $NOW, so the
sentinel always looks freshly-created and the age check never passes.
Flip the order: GNU stat -c is an invalid option on BSD/macOS and fails
with no stdout, so trying it first is safe on both platforms. Same
pattern existed twice (CLAIMED_AT, RET_AT).
Reproduced on a real Linux box: 4 of the PR's own re-arm tests
(test_rearm_after_age_and_new_edits,
test_rearm_updates_stored_count_and_rests_again,
test_pre_feature_sentinel_rearms_from_zero,
test_concurrent_rearm_produces_exactly_one_nudge) fail there and pass
after this fix; all 74 tests pass on macOS both before and after.
---------
Co-authored-by: ar9av <ar9avg@gmail.com>
* fix(hooks): exempt read-only sessions from the Stop-hook capture nudge
The Stop hook fired after pure research sessions (0 file edits, N shell
commands that only read state — gh/git log/grep/cat), forcing a full
wiki-capture skill invocation that predictably ends in SKIP. Shell-command
count alone is a poor signal of capture-worthy work.
The transcript scan now classifies each Bash command read-only vs mutating
(quote-aware pipeline splitting, redirect detection outside quotes,
git/gh/sed/python/curl subcommand and flag awareness; unknown commands
count as mutating). The nudge still fires on any Write/Edit/NotebookEdit,
and on >= 4 shell calls when at least one mutated state — but an edit-free
session whose shell activity is entirely read-only exits silently, acting
as a cheap pre-gate so a SKIP-destined session never costs a skill turn.
Behavioral tests run the hook against synthetic transcripts and pin the
exit-code contract, the classifier edges, and the existing sentinel and
stop_hook_active guards.
* fix(hooks): close read-only classifier gaps that hid mutating commands
Review of the exemption surfaced commands the classifier wrongly proved
read-only. Close them:
- command/process substitution: $(...), backticks, <(...) run arbitrary
inner commands (also inside double quotes) — now always mutating;
single-quoted occurrences stay literal
- find with -delete/-exec/-execdir/-ok/-fls/-fprint* mutates
- curl: glued -XPOST, --request, --json, --form-string now detected
- sort -o / --output writes a file; uniq with two positional args too
- env/nice/stdbuf are wrappers that run their argument command — unwrap
them (bare env stays read-only); date -s and sysctl -w mutate
- python -c payloads are scanned for write/exec patterns (os.remove,
shutil, subprocess, open(...,w), .write(), writing HTTP verbs)
- awk programs with internal redirection or system() mutate
Tokenization now uses shlex (with a naive fallback), and the classifier
moved from an inline $(python3 <<heredoc) into a temp file: bash 3.2
(macOS /bin/bash) scans a $() body naively and chokes on backticks or
unbalanced quotes in the embedded program text.
Eleven new transcript tests pin both directions of each case.
* fix(hooks): harden the classifier against remaining read-only escapes
A second adversarial pass found more commands the classifier wrongly
proved read-only, plus one common false positive:
- python -c payloads using exec()/eval()/__import__, open(mode="w"),
pandas/PIL-style .save()/.to_csv(), sqlite writes (INSERT/DROP/commit),
and urllib Request(data=...) now count as mutating
- the sed `w` script command writes files without -i: `sed -n 'w out'`,
`s/a/b/w out` now detected
- `git diff/log --output=file` writes without a shell redirect
- curl short-flag clusters (-sd, -sLo, -sT, -sD, glued -X) hide
data/output flags from exact-match checks — any cluster containing
d/D/o/O/T/F/X now counts as mutating
- shell-session state commands (cd, pushd, export, set, read, man, ...)
are read-only: they mutate nothing durable, so research chains like
`cd repo && git log` no longer lose the exemption
Eight new transcript tests pin both directions of each case.
* fix(hooks): exemption must consider non-Bash tools and remaining allowlist holes
The read-only exemption only inspected Bash commands, so a session that
mutated a real system through an MCP tool (a CRM update, an SQL INSERT)
while its shell activity stayed read-only was wrongly exempted — a nudge
the pre-exemption hook would have delivered.
Non-Bash tool calls now feed the decision: core harness read tools
(Read/Grep/WebFetch/planning/task bookkeeping) stay neutral; MCP tools are
classified by name verbs (write verbs beat read verbs, unknown counts as
suspicious), and any unknown harness tool counts as suspicious. Suspicious
calls never trigger a nudge on their own — they only disable the
exemption, so the hook still never fires where the original threshold
(edits >= 1 or bash >= 4) wouldn't have.
Also closes remaining bash allowlist holes: hostname with an argument sets
the hostname; fd -x/--exec runs commands; tree -o and a second xxd
positional write files; rg --pre executes a preprocessor.
Nine new transcript tests pin the MCP/unknown-tool matrix and each flag.
* fix(hooks): correct single-quote scanning and remaining mutating flag forms
The quote scanner treated a backslash before a closing single quote as an
escape, but the shell allows no escaping inside single quotes — so
constructs like: echo 'a\' ; rm x' kept the rm inside a "quoted" region
and classified read-only. Single quotes now always close; only double
quotes honor a backslash.
More mutating forms detected:
- git reflog expire / reflog delete (bare reflog and reflog show read)
- curl cookie/header/trace writers: -c clusters, --cookie-jar,
--dump-header, --trace*, --etag-save
- awk programs piping to a command or reading via getline pipe
- sed w-script glued into -e's/a/b/w out' (w-scan covers all tokens)
- history -c/-d/-w/-a/-r mutate history state (bare history reads)
- python -c payloads touching smtplib/ftplib/paramiko/os.kill
Eight new transcript tests pin each case in both directions.
Previously `setup --vault <path>` only wrote the config pointing at the
vault path, leaving the directory and its special files (index.md,
log.md, hot.md, .manifest.json, etc.) uncreated unless a full agent
session ran the wiki-setup skill. Add an idempotent scaffold_vault()
that mirrors wiki-setup/SKILL.md's directory/file layout so a plain
pip-installed CLI run leaves a working vault behind.
Two strands of work.
Session brain — a topic graph over raw agent session history, so you can
find the session where something happened instead of remembering its
UUID. Reads every Claude transcript plus the pruned sessions that survive
only as prompts in history.jsonl, clusters them with local TF-IDF and
community detection (no embeddings, no API calls, no new dependencies),
and writes a sidecar at ~/.claude/session-brain/ — the vault is never
touched. Ships five modules, the sessions-* CLI commands, an interactive
graph.html, and the /session-brain and /wiki-sessions skills. This is
retrieval, deliberately distinct from wiki-history-ingest, which distils
sessions into permanent vault pages.
Docs — README.md was 598 lines and put trust-check flags and a 17-row
agent matrix in front of anyone evaluating the project. Cut it to a 129-
line landing page (problem statement, 60-second install, four blocks of
runnable commands, graph screenshot, why-not-a-notes-folder) and move the
reference material into docs/: installation, agents, skills, cli,
configuration, architecture, session-brain, contributing. README_TW.md
mirrors the new structure; docs/ stays English-only for now.
The restructure also closed real drift:
- Ten skills existed but appeared in no reference — session-brain,
session-search, wiki-dedup, wiki-digest, wiki-import, wiki-context-pack,
wiki-stage-commit, wiki-switch, vault-skill-factory, and
obsidian-layout-adjustment.
- SETUP.md listed 12 of 39 skills and still claimed "no scripts, no
dependencies" despite the pip package. It is now a signpost into docs/.
- Eight config variables were undocumented anywhere, including
WIKI_STAGED_WRITES, which wiki-setup prompts for and three skills read.
Added to .env.example and docs/configuration.md.
- sessions-name and a number of CLI flags were missing from the docs.
Two details worth noting. README links are absolute GitHub URLs because
pyproject sets readme = "README.md" as the PyPI long_description, and PyPI
resolves relative hrefs against pypi.org rather than the repo — docs/
links would have 404'd on the package page. And three doc-drift guards
asserted their facts against README.md; since those facts moved rather
than disappeared, the assertions were repointed at the page that now owns
each one instead of padding content back into the README.
Also gitignore /vault/ (a local knowledge vault, not a fixture) and the
packed browser extension.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
_TITLE_RE and _SUMMARY_RE captured whatever followed `title:`/`summary:`
on the same line, so a folded/literal block scalar (`>-`, `|-`, ...)
returned the indicator itself instead of the text on the following
indented lines — e.g. `summary: >-` read back as the literal string
">-". Replace both regexes with a shared _extract_scalar() that
detects the block-scalar form and folds the continuation lines.
Fixes#156
QMD_WIKI_COLLECTION pointed at OBSIDIAN_VAULT_PATH, which contains
_raw/, so a literal reading of the setup docs merges the wiki and
papers collections wiki-query depends on keeping disjoint. Document
the ignore-based exclusion in .env.example (and fix the nonexistent
`qmd index` command name), have wiki-setup verify/create that
exclusion when QMD_WIKI_COLLECTION is configured, and have wiki-query
defensively drop any _raw/ path surfaced from the wiki collection so
a misconfigured collection degrades instead of citing stale drafts
as compiled knowledge.
Fixes#157
The "fire at most once per session" guard in wiki-stop-capture.sh was a
check-then-write with the two halves separated by the whole script: the
existence test ran near the top, the write only just before exit 2.
Claude Code merges project-level and user-level hook config rather than
letting one override the other, and runs matching hooks concurrently. Since
this repo ships its own .claude/settings.json registering the Stop hook,
following wiki-setup's instructions to also install it globally means two
invocations run in parallel for the same session_id. Both pass the existence
test before either performs the write, so the guard never fires and the user
gets two capture nudges.
Claim the sentinel with mkdir instead, which is atomic on POSIX filesystems:
exactly one invocation wins and the losers exit 0 silently.
Two placement details matter:
- The claim stays at the decision point rather than moving to the top of the
script. Claiming early would burn the session's single nudge on an early
below-threshold turn, silencing a later turn that did meaningful work.
- The cheap short-circuit check moves from -f to -e, since the sentinel is now
a directory. Left as -f it would silently stop matching.
Verified with six cases: two concurrent invocations yield exit codes 2 and 0
with exactly one nudge; a below-threshold turn followed by a work turn still
nudges; sequential re-fire stays suppressed; the stop_hook_active re-entry
guard holds; a missing session_id still nudges (no dedup possible); and a
legacy file-shaped sentinel from an in-flight session is still honored.
Also document the duplicate registration in wiki-setup, which previously only
warned about not clobbering an existing hooks.Stop array.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The daily-update skill's Setup Mode references scripts/daily-update.sh,
the launchd plist, and wiki-notify.sh via $OBSIDIAN_WIKI_REPO/scripts/ —
same class of gap as #143 (the Stop hook), which didn't cover this
directory. force-include scripts/ under obsidian_wiki/_data/scripts/ so
pip/uv installs actually have it, and exclude __pycache__/*.pyc from the
wheel target since force-include copies straight from the working tree
(a local scripts/__pycache__ was otherwise leaking stray .pyc files in).
setup.sh had an interactive git-sync flow (git init, .gitignore, remote,
a generated sync.sh, optional cron) that `obsidian-wiki setup` — the entry
point pip/uv installs actually run — never had, so those installs silently
skipped it and .env.example documented a var nothing ever read.
Move the git plumbing into obsidian_wiki/sync.py (configure_sync/run_sync/
get_remote) and expose it as `obsidian-wiki sync-setup <url>` and
`obsidian-wiki sync`. `obsidian-wiki setup` now offers the same interactive
prompt setup.sh always had (or accepts --remote non-interactively), and
setup.sh's own prompt delegates into the same module via the CLI instead of
re-implementing it. Sync status is read live from the vault's own `git
remote` (via `obsidian-wiki info`) rather than a config var that could drift.
wiki-ingest previously walked repo folders with a hardcoded SKIP_DIRS
list, so custom-ignored paths (anything not literally node_modules/
.venv/dist/...) leaked into the ingest plan. batch-plan now detects a
`.git` folder and enumerates files via `git ls-files`, so a repo's own
.gitignore decides what's skipped. Docs updated to spell out the
"clone locally, add to OBSIDIAN_SOURCES_DIR" workflow for public/private
repos on any host.
Legacy vaults fail-closed on every pre-existing page after the trust-ledger
rollout: missing base_confidence/lifecycle and a missing ledger became
mandatory failures with no opt-out, skipping the staged migration from #28.
Split confidence/lifecycle out of the core required-frontmatter check into a
separate confidence_missing_fields finding, and add a strict_trust switch to
lint_vault (default off): missing trust fields, ledger errors, stale reviews,
and score mismatches now warn by default and only fail once a vault owner
opts in via --strict-trust or OBSIDIAN_TRUST_STRICT=1. Ledger presence alone
no longer silently enables strict mode.
cache-check / cache-update assumed a dict-keyed manifest with bare hex
hashes, but the wiki-ingest skill writes 'sources' as a list of entry
objects with sha256:-prefixed content_hash and vault-relative paths.
Against a skill-built manifest the commands crashed, then (once past the
crash) reported every file as modified/new. Both shapes occur in real
vaults.
- _load/_iter: read both dict- and list-shaped 'sources' transparently
- compare content_hash prefix-insensitively (strip optional algo: prefix)
- match paths across raw / absolute / vault-relative forms
- update_source edits in place, preserving manifest shape, duplicate-path
entries, skill-written fields, and top-level keys (stats/version)
- don't flag URL / pseudo-path source keys as missing-on-disk
- add regression tests covering all four failure modes from the issue
The wiki-setup skill wires a Claude Code Stop hook pointing at
<REPO_PATH>/.claude/hooks/wiki-stop-capture.sh, but /.claude is pruned
from the sdist, so the script was missing from the published wheel and
the installed hook dangled at a nonexistent path.
- force-include the hook into the wheel under _data/hooks/ (and re-add it
to the sdist, which the wheel is built from)
- wiki-setup now resolves the packaged hooks/ path, falls back to the
source-checkout .claude/hooks/ path, then to a GitHub raw fetch
- wiki-setup now creates .manifest.json so 'obsidian-wiki doctor' passes
its required-core-files check (secondary note in the issue)
- add regression tests pinning the packaging config and skill wiring
The readme-sync workflow no longer fails when README.md and
README_TW.md don't change together. Instead, check_readme_sync.py
reports drift: it lists commits that touched README.md after the
last README_TW.md update, plus the pending English diff to translate
and backfill. CI surfaces drift as a warning annotation only.
Claude-Session: https://claude.ai/code/session_01PG9Pz6tcUfgpTHLjq3BT5Q
PR #129 added a "prefer ripgrep" policy to llm-wiki/SKILL.md, but six
other skills still hardcoded literal grep commands in their examples,
bypassing that policy. Swap them to rg with correct flag mapping
(--include="*.md" -> --glob '*.md'; -r dropped since rg recurses by
default).
ar9av