check_sources only matched a manifest entry by the passed key or its
abspath, and detected missing files with Path(key).exists() against CWD.
When the manifest stored vault-relative keys (e.g. _raw/articles/foo.md),
callers passing an absolute path got 'new' instead of 'unchanged', and
every relative key was reported 'missing' when CWD != vault root — so the
docstring's documented abs-or-rel key support was unusable.
Also try the vault-relative form on lookup, and resolve relative manifest
keys against the vault root when checking existence. Add relative-key
tests to close the CI gap (every prior test used absolute keys).
Fixes#136
The Step 7 example wrote ingested_at/pages_created/pages_updated, but
cache.py reads and writes content_hash/last_ingested/pages_produced. An
agent following the docs wrote fields cache-check never reads, so every
source looked new on the next run and incremental-skip detection died.
Rename the fields to match the code, drop the never-persisted
size_bytes/modified_at/pages_updated, and note that source_type/project
are advisory only.
Fixes#135
Validate updated timestamps before fingerprint comparison, support dash-only relationship list items, and reject ambiguous normalized qualified targets and unsupported relationship fields.
Reject symlinked writes and malformed or ambiguous trust data, enforce complete page coverage and strict timestamps/lifecycle values, and parse typed relationships independent of key order.
wiki-narrate saves derived readouts to _readouts/, but cross-linker,
wiki-export, and wiki-lint scanned every .md in the vault and would
have linked, exported, or lint-flagged them. Exclude the directory in
all three and document it in the AGENTS.md vault structure.
The staleness order still let --since suppress a missing output: a source
older than the cutoff with no output file was skipped, leaving the very bug
this branch fixes (never-extracted older sessions) partly alive and
contradicting needs_extraction's own docstring.
Move the "output missing -> extract" check ahead of the --since gate, so a
missing output is always (re)extracted and --since only narrows files that
already have an output. Add a regression test: missing output + future
--since still extracts.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
cmd_normalize warned on relative keys but still ran them through
canonical() (os.path.abspath), which resolves them against the CWD and
rewrote e.g. "-Users-x-github/abc.jsonl" to "/cwd/-Users-x-github/abc.jsonl",
counted it as re-keyed, and persisted the corruption on a non-dry-run write.
Preserve relative keys untouched (ckey = key), skip the re-key count for
them, and only canonicalize absolute keys. normalize's dedup/canonicalize
purpose applies to absolute keys only; for relative keys it is now a safe
no-op. Collision merge logic is unchanged.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
extract-jsonl.py decided re-extraction from the .extract-manifest.json
extracted_at wallclock. A session whose output was missing or that landed
after the first run was treated as already-done and never re-extracted, so
downstream ingest saw it as missing until a manual rerun (e.g. session
2d83ed0d). --since additionally acted as a hard gate that hid never-extracted
older sessions.
Make needs_extraction idempotent on the output file: (re)extract when the
output is missing or the source is newer than it; --since only narrows the
scan range. The manifest (read by nothing else) is removed. Add
test_extract_staleness.py covering missing/newer/older cases.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
cmd_delta keyed every manifest source through canonical() (os.path.abspath),
which resolves relative keys (e.g. "-Users-x-github/abc.jsonl" under
~/.claude/projects/) against the CWD. Real vaults store most keys relatively,
so every already-ingested file was misreported as NEW (446 vs 384 known on the
live vault).
Add a basename-indexed suffix fallback: when a scanned absolute path has no
direct canonical match, look it up by basename and confirm the relative key is
a path suffix (O(1) avg, no O(n^2) scan). Absolute keys keep their exact prior
behavior. cmd_normalize is unchanged but now warns on relative keys, which it
would otherwise rewrite to CWD-based garbage.
AI-Agent: Claude Code
AI-Session-IDs: session_01PPCNXmXUvemmyvWxrJ3D5E
Adds obsidian-wiki batch-plan <vault> <source-dir> — splits a source
directory into sized batches and emits a dispatch plan the skill uses
to spawn parallel Claude subagents, each handling one batch independently.
- New obsidian_wiki/batch.py: file classification (text/pdf/image/office/
code/skip), discover_sources() directory walker (respects SKIP_DIRS,
excludes code by default since ast-extract handles those), _make_batches()
(byte + file-count limits), plan_batches() top-level entry that integrates
with the cache module to skip unchanged files before splitting.
- CLI: batch-plan <vault> <source-dir> [--max-mb N] [--max-files N]
[--no-cache] [--include-code] [--pretty]
- wiki-ingest SKILL.md gets Step 0: run batch-plan for directories >20 files,
dispatch each batch as a parallel subagent, run cross-linker once all done.
- 30 new tests: classification, discovery, splitting, cache integration, CLI.
Adds obsidian-wiki graph-analyse <vault> — computes god nodes, community
clusters, surprising cross-community connections, dead-ends, and isolated
pages from the vault's wikilink graph. No LLM, no API calls.
- New obsidian_wiki/graph_analysis.py: parse_vault_graph() (wikilink + tag
extraction), god_nodes() (degree ranking), detect_communities_greedy()
(label-propagation, pure Python, O(n·k)), detect_communities() (tries
leidenalg+igraph first, falls back to greedy), surprising_connections()
(cross-community edges ranked by inverse sqrt of cross-degree product),
analyse_vault() top-level entry returning JSON-ready dict.
- CLI: graph-analyse <vault> [--top N] [--pretty]
- Optional obsidian-wiki[graph] extra for Leiden (leidenalg + igraph).
- wiki-status SKILL.md insights mode updated: now calls graph-analyse first
instead of building the wikilink graph manually, with fallback instructions.
- 27 new tests covering parsing, metrics, community detection, surprising
connections, full integration, and CLI.
Replaces the manual sha256sum shell dance with a dedicated Python module
and three CLI subcommands. Pure stdlib — no new runtime dependencies.
- New obsidian_wiki/cache.py: sha256_file/sha256_dir (chunked, memory-safe),
check_sources() classifies a list of paths as new/modified/unchanged/missing
vs .manifest.json; update_source() records hash + pages_produced + timestamp
after a successful ingest.
- CLI: cache-check <vault> <sources...> → JSON classification
cache-update <vault> <source> [--pages ...] → write hash to manifest
cache-hash <path> → just compute and return SHA-256
- wiki-ingest SKILL.md updated: append-mode section now uses cache-check /
cache-update instead of platform-specific sha256sum invocations. Falls back
to the old manual approach if obsidian-wiki is not installed.
- 25 new tests covering hash stability, mtime-independence, manifest round-trips,
and all three CLI commands.
Adds `obsidian-wiki ast-extract <path>` — a free, local pass that extracts
classes, functions, imports, and inheritance edges from code files using
deterministic regex patterns before any token is spent on the LLM.
- New `obsidian_wiki/ast_extractor.py`: Graph/Node/Edge datamodel + per-language
LangSpec patterns for Python, JS/TS, Go, Rust, Java, Kotlin, Ruby, C/C++,
Swift, Shell (12 languages). Outputs god_nodes (top hubs by degree), stats,
and a node-link JSON dict compatible with any graph tool.
- New `ast-extract` CLI subcommand with `--pretty` flag.
- Optional `obsidian-wiki[ast]` extra for tree-sitter upgrade path.
- wiki-ingest SKILL.md gets Step 1c: run ast-extract first on code sources,
seed entity pages from class nodes, skip sending code files to the LLM.
- 19 new tests covering file extraction, directory merge, CLI output, edge cases.
* fix(pi-history): require session-grounded provenance
Add a session knowledge closure rule, evidence ledger, and source verification gate so extracted wiki claims must cite Pi session entries instead of relying on model memory.
AI-Agent: Pi
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Arnav <f2016731@pilani.bits-pilani.ac.in>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Arnav