Commit Graph

2 Commits

Author SHA1 Message Date
Arnav 073a04cd37 feat(batch): make source discovery git-aware for repo ingestion (#148) (#150)
wiki-ingest previously walked repo folders with a hardcoded SKIP_DIRS
list, so custom-ignored paths (anything not literally node_modules/
.venv/dist/...) leaked into the ingest plan. batch-plan now detects a
`.git` folder and enumerates files via `git ls-files`, so a repo's own
.gitignore decides what's skipped. Docs updated to spell out the
"clone locally, add to OBSIDIAN_SOURCES_DIR" workflow for public/private
repos on any host.
2026-07-24 18:08:51 -07:00
ar9av 6069d775d6 feat(batch): parallel subagent batch planning for large-folder ingestion
Adds obsidian-wiki batch-plan <vault> <source-dir> — splits a source
directory into sized batches and emits a dispatch plan the skill uses
to spawn parallel Claude subagents, each handling one batch independently.

- New obsidian_wiki/batch.py: file classification (text/pdf/image/office/
  code/skip), discover_sources() directory walker (respects SKIP_DIRS,
  excludes code by default since ast-extract handles those), _make_batches()
  (byte + file-count limits), plan_batches() top-level entry that integrates
  with the cache module to skip unchanged files before splitting.
- CLI: batch-plan <vault> <source-dir> [--max-mb N] [--max-files N]
        [--no-cache] [--include-code] [--pretty]
- wiki-ingest SKILL.md gets Step 0: run batch-plan for directories >20 files,
  dispatch each batch as a parallel subagent, run cross-linker once all done.
- 30 new tests: classification, discovery, splitting, cache integration, CLI.
2026-06-30 13:49:53 -07:00