- extract_sql(): deterministic tree-sitter extraction of tables, views,
functions, foreign key references, and FROM/JOIN reads_from edges
- .sql added to CODE_EXTENSIONS and dispatch table
- tree-sitter-sql added as optional dep under [sql] extra
- xlsx_extract_structure(): extracts sheet/table/column nodes from .xlsx
(utility — pipeline wiring in follow-up)
- 6 new SQL tests, 447 total passing
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Token-budget chunking cuts the truncation rate but doesn't eliminate
it. Output token cost scales with extractable concept density rather
than input tokens — a chunk that lands on a directory of dense design
docs can pack under the input budget while needing more than
`max_completion_tokens=8192` to express every named concept, so the
response is truncated mid-string and `_parse_llm_json` returns an
empty fragment.
Pre-tuning chunk size to be conservative enough that this never
happens leaves throughput on the table for the common case. Adding a
hard `max_files_per_chunk` cap on top of `token_budget` reintroduces
the "tune a static constant" problem the previous commit set out to
fix.
The fix uses the API's own truncation signal:
1. `_call_openai_compat` and `_call_claude` now expose `finish_reason`
on the result dict (Anthropic's `stop_reason == "max_tokens"` is
normalised to `"length"`).
2. `_extract_with_adaptive_retry` checks it: when truncated, splits
the chunk in half and recurses on each half. Recursion is bounded
by `max_retry_depth` (default 3 → at most 8x fanout per top-level
chunk).
3. Single-file chunks that truncate can't recover and surface a
warning rather than infinite-loop.
4. `extract_corpus_parallel` routes every chunk through the retry
wrapper. The `on_chunk_done` callback fires once per top-level
chunk with the merged result — recursive splits are invisible to
callers.
Three independent improvements to extract_corpus_parallel:
1. Token-aware chunking. Replaces `chunk_size=20` static packing with
a greedy packer keyed on `token_budget` (default 60_000), grouped
by parent directory so related artefacts share a chunk. Pass
`token_budget=None` to fall back to fixed-count packing.
2. Optional tiktoken (added to the [kimi] extra). When available,
`_estimate_file_tokens` uses cl100k_base for accurate counts;
without it, the existing chars/4 heuristic kicks in. Kimi-K2 ships
a tiktoken-based tokenizer so estimates against Moonshot are very
close to truth.
3. True parallelism. The function name said "parallel" but the body
was a sequential for-loop. Now uses ThreadPoolExecutor capped at
`max_concurrency` (default 4 — conservative against provider rate
limits). `on_chunk_done(idx, total, result)` still fires once per
chunk with the original submission idx so progress UIs work
unchanged. `max_concurrency=1` skips the pool to preserve
sequential semantics.
Plus failure tolerance: a chunk raising is now caught, logged to
stderr, and the run continues. Other chunks' results merge as normal.
On a 162-file repo (~125k words), the same work that took ~36 min
sequential under the old code finishes in ~7 min.
Adds docs/docker-mcp-sqlite.md, a reproducible recipe for installing the
SQLite MCP server into Docker MCP Toolkit so any connected MCP client
(Claude Code, Cursor, VS Code, etc.) gains six SQLite tools alongside
graphify's knowledge-graph tools.
Notes the catalog has two SQLite images at time of writing: `mcp/sqlite`
(marked Archived but works) and `mcp/sqlite-mcp-server` (broken
entrypoint). Recommends the working one.
Linked from README.md under a new 'Optional integrations' section.
This is unrelated to the upstream graphify pipeline; it lives as an
optional companion runbook for users who want a lightweight persistent
SQL workspace exposed to their MCP-aware AI clients.
Co-Authored-By: Oz <oz-agent@warp.dev>
AST and semantic entries now write to cache/ast/ and cache/semantic/
respectively. Previously both used the flat cache/ dir causing semantic
results to overwrite AST entries for code files on mixed corpora, making
the shrink guard fire on every subsequent update run.
Migration: load_cached falls back to legacy flat cache/ for AST reads
so existing cache entries are not lost on upgrade.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Grep and Glob tools removed in CC v2.1.117; searches now go through Bash.
Hook now reads stdin tool_input and pattern-matches on search commands.
Uninstall/reinstall handles both old and new matcher for clean upgrades.
Closes#578
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
import(`./handlers/${name}`) previously produced a garbage edge to a
path containing the unresolved ${name} expression. Now detects
template_substitution child nodes and breaks without emitting an edge.
Static template literals (no interpolation) still resolve correctly.
Adds 2 new tests: one asserting dynamic templates produce no edge,
one asserting static templates resolve like plain strings.
Adds _dynamic_import_js() helper (65 lines) that detects import() call
expressions in JS/TS, resolves the module path (same logic as static
imports including .js→.ts mapping and tsconfig aliases), and emits
imports_from edges from the enclosing function. Hooked into walk_calls
for JS/TS configs.
Also adds tests/fixtures/dynamic_import.ts fixture and 5 new tests
in tests/test_languages.py (all passing alongside 110 existing tests).
- #550: _file_stem() includes parent dir to prevent node ID collisions for same-named files
- #555: extract() relativizes source_file paths before returning for cross-machine portability
- #562: to_json() returns bool; _rebuild_code() writes report/html only if json succeeded
- #563: skill prompts store rationale as node attribute, not separate node; enforce calls direction
- #566: Show All / Hide All buttons added to HTML community panel
- #575: _import_js() resolves tsconfig.json compilerOptions.paths aliases before external fallback
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
html.escape() the values that land in <title> and <h1>, and replace </
with <\/ in the JSON embedded inside <script> so crafted graph labels
or --label values cannot break out. Mirrors the _js_safe() pattern in
export.py.
Reported by Qodo on PR #557.
Adds a new `graphify tree` subcommand that emits a self-contained D3
v7 collapsible-tree HTML view of an existing graph.json.
Why
---
The existing `graph.html` (force-directed) is great for finding hubs
and unexpected connections. But for code review and onboarding, a
hierarchical tree-of-modules view is much faster: you can collapse
everything and expand only the package you care about, the depth-
based colour palette gives instant orientation, and the layout
mirrors the on-disk structure.
UX choices include expand-all / collapse-all / reset-view buttons,
multi-line `wrapText` labels with separately-coloured name and count,
a depth-based palette, click-to-toggle subtrees, and a hover-inspector
that surfaces the top-K outbound edges per symbol.
Implementation
--------------
- `graphify/tree_html.py` (575 LOC, single file, no new runtime
dependencies). D3 v7 is loaded from cdn.jsdelivr.net at view time.
- Hierarchy is built from `source_file` longest-common-prefix;
symbols are grouped by containing module so the tree mirrors the
on-disk layout exactly.
- Inspector pre-computes top-K outbound edges per symbol so the page
works fully offline once loaded.
- `__main__.py` adds the subcommand + help text after the
`check-update` block.
Configuration
-------------
- `--graph PATH` path to graph.json (default: graphify-out/graph.json)
- `--output HTML` output path (default: graphify-out/GRAPH_TREE.html)
- `--root PATH` filesystem root (default: LCP of source_files)
- `--max-children N` cap visible children per node (default: 200)
- `--top-k-edges N` per-symbol outbound edges in inspector (default: 12)
- `--label NAME` project label shown in the page header
Tested locally on a 17 641-node graph — emits a 4.9 MB HTML file
that renders smoothly in Firefox / Chromium.
Safi