#2543 (adopts PR #2546, thanks @michaelxer): a failed extraction is no
longer stamped in the incremental manifest as up-to-date, so graphify
update retries it instead of skipping it forever; a manifest already
poisoned by the old behavior is healed on the next run; genuinely
unchanged files are not re-processed. Extended to the watch save_manifest
paths too.
#2554 (adopts PR #2555, thanks @annieyii): the claude-cli backend now
inspects the stdout envelope for an is_error result (e.g. a rate limit
returned with exit code 0) and raises it on both the zero and non-zero
exit paths, instead of parsing it as an empty success and bisecting
against a live rate limit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cluster-only warns when --backend/--model/--batch-size are ignored on the
label-reuse path; the community-label prompt key no longer collides with
the discard sentinel (an echoed key was silently dropped); tree --root
exits non-zero when it matches no source file instead of flattening the
tree; and cluster-only stamps built_at_commit from the analysed graph, not
the shell cwd. Also folds in the cluster-only refused-write guard from
PR #2522 (thanks @aniJani).
Thanks @elecnix for the report.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#2486 (thanks @adminwat): normalize dict-shaped hyperedge members to ids
(or drop with a warning) so a malformed hyperedge can't abort a completed
merge with a TypeError.
#2484 (thanks @sortakool; approach from @oleksii-tumanov's #1691):
merge-graphs relabels hyperedge member ids and ids with the repo prefix,
unions both inputs' hyperedges instead of clobbering, and writes both
persistence slots.
#2485 (thanks @sortakool): build_from_json reads hyperedges from the
top-level and nested slots; a full validation wipeout is reported loudly.
#2490 (thanks @PapiScholz): the skill Step-5 flow passes curated
community_labels to to_json, so graph.json ships community_name.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#2411 (thanks @JensD-git): key the C# partial-class merge on assembly
(nearest ancestor .csproj/.fsproj/.vbproj) in addition to namespace and
name, so same-name partial classes in different assemblies stay distinct
while genuine partial halves within one project still merge. Fixes a
0.9.32 regression from #2332.
#2437/#2438 (thanks @aryanbonigala, builds on PR #2439): incremental
rebuilds no longer drop member-call and indirect_call edges from a
changed file into an unchanged target. Re-resolution now sees the
unchanged corpus (nodes, contains/method edges, and _callable markers,
which now persist to graph.json like _origin); edges to a genuinely
removed target are still evicted.
#2444/#2445 (thanks @Baziar, builds on PRs #2461/#2458): a
BrokenProcessPool triggers the sequential fallback instead of being
swallowed per future, a failed worker file is retried sequentially
rather than merged as empty, and a whole-pass AST failure on a fresh
build exits non-zero instead of writing a zero-node graph
(--allow-partial opts into a best-effort partial graph).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
graphify path decided hop direction from the persisted source/target order,
so a link stored in flipped order (pre-#563 graphs, raw dumps, merge-driver
output) printed backwards; explain had the same defect, and the query/merge
load shims clobbered in-file _src/_tgt markers. All now honor _src/_tgt (the
build-side direction truth), falling back to arc order for markerless files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`_find_node` ranks matches but never reports that a tie was broken, so callers
taking `matches[0]` present one arbitrary file as the answer. Two workspaces
that each define `MetricsPort` put both nodes in the same `exact` tier,
separated only by `G.nodes()` iteration order — reorder the graph and the same
query answers with a different file, equally confidently.
`affected` already declines this via `resolve_seed()` returning None, and
`path` warns when the top two scores are within 10%. Only `explain` (and the
MCP `get_neighbors` tool, which shares the matcher) was silent.
Split `_find_node` into `_find_node_tiers` (logic unchanged, tiers exposed) plus
a flattening wrapper, and add `find_node_ambiguity()`, which returns rivals only
when the winning tier spans multiple source files. Matches within one file (a
file node plus its members) stay ordinary precedence and resolve as before.
`_disambiguate_file_node_labels` (#2032) already relabels colliding *file*
nodes; this covers the symbol case it does not reach.
#2212: benchmark, the merge-driver graph load, and callflow_html crashed
or silently failed on a --no-cluster graph.json (edges stored under
'edges', not 'links'). A shared load_node_link_graph helper normalizes
links/edges before node_link_graph and is used at all three sites.
#2210: _stale_graph_sources compared graph source_file spellings to the
scan with a raw string test (no NFC), and pruned any non-match with no
liveness check, so alive files (macOS NFD paths, legacy basenames) were
pruned as 'deleted'. It now compares NFC-on-both-sides and is fail-closed:
a corpus-missing source whose file still exists is pruned only when the
exclusion is provable, else kept with a warning. Prune message corrected
to 'deleted or excluded'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An incremental `extract --no-cluster` wrote only the changed files over
graph.json with no merge, dropping every node/edge owned by an unchanged
file; and the id-canonicalization pass only learned batch files, so the
changed file's cross-file edges kept absolute-path target ids and
dangled. The raw path now merges the existing graph forward with the same
replace/prune semantics as the clustered path (new merge_raw_extraction
helper in build.py, shared loader), refuses to overwrite a corrupt
existing graph, and the remap pass now also learns in-root edge
target_file paths (existence-gated) so cross-file targets canonicalize.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The heuristic over-matched and silently dropped legitimate files:
- prose `.md`/`.rst` whose topic slug ends in a keyword (privacy-tokens.md,
token-economics.md) — only code was exempt, not prose;
- the unbounded Stage-2 `service.account` substring (regex `.` wildcard) matched
real source (google/oauth2/service_account.py) and prose slugs.
It also MISSED real secrets (.npmrc, .pypirc, secring, .git-credentials, and
case variants on case-insensitive filesystems), which were being indexed.
Fix: move service_account/aws_credentials to the boundary-checked keyword path
(so real source is spared, downloaded key files still drop), add a prose-note
carve-out (multi-word slugs indexed, bare `secrets.md`/`token.md` still dropped),
tighten id_rsa with a left boundary, add the missed secret dotfiles + secring,
lowercase the dir/segment comparisons, and count multi-dot slugs as multi-word.
Net effect is stricter on real secrets and stops the false-positive data loss.
Traceability: `graphify extract` now names the files skipped as sensitive (not
just a count), so a wrongly-flagged file is visible.
`graphify explain "<node>"` sorts a node's connections by neighbor
degree and shows only the top 20, appending a bare "... and N more"
for the rest. On a high-degree node (a logging/error helper called
from dozens of places is typical) that leaves the exact question
explain is meant to answer - "who calls this, what's the impact?" -
unanswered for 97% of the callers, with nothing pointing at where
they live (#2009).
The top-20 list and its ordering are unchanged (no behavior change
for nodes at or under the cutoff). Past it, the cut connections are
now grouped by (direction, source_file) with counts and printed
under a "Grouped by file:" section, sorted by count descending, so
the caller/callee distribution is visible without falling back to a
repo-wide grep. The aggregation itself is capped at 20 files with
its own "... and N more files" line for the pathological case where
the remainder is spread across more files than that.
Full per-caller detail behind a flag (--callers --all / --group-by=dir
as proposed in the issue) is left for a follow-up: it's a larger,
more opinionated surface (flag naming, pagination semantics) than
the literal complaint needs, and the default output no longer hides
the answer either way.
Four regression tests in tests/test_explain_cli.py: the pre-existing
truncation notice on a 30-connection node is unchanged, the grouped-
by-file output carries real counts (3 files, one at 4 and two at 3)
that sum back to exactly the cut total (no silent loss), a
byte-for-byte no-op check for nodes at/under the cutoff, and a
boundary check pinning the cutoff itself at exactly 20 connections
(no section) vs exactly 21 (one grouped entry) — the earlier tests
sit well clear of the edge and wouldn't catch a future off-by-one.
`graphify query` builds an undirected nx.Graph (so BFS/DFS can explore
both callers and callees of the seed node), but its text renderer
assumed the BFS/DFS visit order (u, v) was always the edge's
(source, target). On an undirected graph that assumption only holds
when the seed happens to be the caller: seeding on the callee makes
BFS/DFS visit the callee first, so a `caller --calls--> callee` edge
was rendered backwards as `callee --calls--> caller`. graph.json's
own source/target fields stay correct on disk; only the query
rendering was wrong.
`graphify path` and `graphify explain` don't have this problem
because they force directed=True on load (#849, #853), and the MCP
query_graph tool's _load_graph() does the same. Doing that for CLI
`query` too was tried and reverted: forcing a DiGraph makes
G.neighbors() return successors only, so a query seeded on a
leaf/sink node (no outgoing edges) found zero neighbors instead of
its callers — a recall regression, not just a display fix, and it
would make the CLI and MCP query tools diverge in what they discover
even though they'd render direction identically.
Fix instead mirrors the _src/_tgt pattern graphify/build.py already
uses for the same underlying problem (undirected storage loses
direction): the CLI now stashes each link's true source/target on
its edge data as _src/_tgt before constructing the (still undirected)
graph, and _subgraph_to_text renders EDGE lines from _src/_tgt when
present, falling back to (u, v) otherwise. Traversal itself is
unchanged, so recall is unaffected — verified against the unpatched
CLI, the node counts returned for the same seeds are identical before
and after this fix, only the printed edge direction changes.
Adds two regression tests in tests/test_query_cli.py seeding the same
`calls` edge from both endpoints; the callee-seeded case fails on the
prior code with the exact backwards-edge symptom above.
The top-level `graphify --help` has listed --code-only since #1734, but the
`graphify extract` usage string and the README never mentioned it. A user
evaluating graphify on a "can this run with no network call" constraint sees
the extract usage / README first and can conclude the flag doesn't exist.
Add --code-only to the extract usage line, name it in the Privacy section and
the command reference, and add a test asserting the usage advertises it.
Two correctness defects found in a head-to-head benchmark of 0.9.22.
Caller line numbers: explain/affected/get_neighbors/query printed the caller
node's def line for an incoming call, presented as a precise citation, so
click-through landed in the wrong place. The `calls` edge already carries the
true call-site line (engine.py sets it); every caller/relation listing now reads
the traversed edge's source_file:source_location, falling back to the node's own
line only when the edge lacks one.
Silent query truncation: rendered nodes were degree-ordered (a low-degree
definition node ranked last, cut first), the queried symbol wasn't guaranteed to
appear, and the truncation marker sat only at the end so silence read as absence.
Nodes are now ranked by hop distance from the seeds (deterministic), the seed the
question named is rendered first and never truncated, and a prominent TRUNCATED
notice at the top states shown/total counts and how to widen the budget. Also
rewires the seed-first ordering the renderer already supported — a branch merge
had silently dropped the `seeds=` argument, leaving it dead code.
A `cluster-only --no-label` run wrote "Community N" placeholders into
.graphify_labels.json plus a matching .sig, and the reuse path treated them as
fresh, so real labels were never regenerated on later runs. Two fixes: (a) don't
persist the labels sidecar (or its .sig) on a placeholder-only run, so a later
run generates real labels; (b) treat a stored "Community {cid}" as absent in the
reuse path so an already-polluted sidecar self-heals via the hub labeler while
genuine labels are still reused with no LLM call. The watch/update rebuild had
the same placeholder-perpetuation twin — fixed alongside.
`graphify path` (and the MCP shortest_path tool) ran shortest_path over
G.to_undirected(as_view=True), whose neighbor iteration is a hash-seeded set
union, so among equal-length paths BFS returned a route that varied per process.
Build a sorted, materialized undirected graph so the chosen path is canonical.
The hop label also printed a relation read from an arbitrarily-collapsed parallel
edge, so it could show `calls` on a pair that only carries `references`. Force
multigraph on the cli path reload so parallel links survive, and render the
ACTUAL stored relation(s) via edge_datas, falling back to an honest "related"
when the edge has none. Serve's shared graph is left untouched (its degree feeds
query-seed tie-breaks); the fix is applied locally in both path readers.
The build_from_json label pass only covered the clustered/update paths; the raw
`extract --no-cluster` path writes the merged node list directly, so colliding
basenames stayed un-disambiguated there. Factor the logic into a shared
_file_label_reassignments core with a list-based variant
(disambiguate_file_labels_in_nodes) and apply it on the raw merged nodes.
Caught by the clean-venv edge-case battery.
Part 2: `god_nodes` was an analyzer, an MCP tool, and a README-advertised
capability, but `graphify god_nodes` errored with "unknown command". Add a
read-only `god-nodes`/`god_nodes` subcommand mirroring `affected` (--graph,
--top, --json), routing labels through sanitize_label.
Part 3: `--output DIR` on `extract` was silently dropped (output fell back to
the default dir). It is now an alias of `--out` (both space and =forms), matching
what `graphify tree` already documents. Help/usage text updated.
Part 1 (affected/reverse-dep import-id mismatch) is deferred — a build-time
id-resolution change, tracked separately.
build_merge now prunes a deleted file's nodes, edges, and hyperedges regardless
of whether their stored source_file is absolute or relative. When the caller
passed no root (the --update runbook), a node that kept an absolute path slipped
past the relative prune set and the deleted file's graph survived silently.
Matching is now form-insensitive (raw, normalized-relative, then an absolute-
identity fallback); a re-extracted file is still never pruned (#1796 preserved).
`graphify extract` also writes the .graphify_root marker after every graph write
so a later build_merge relativizes deleted-file paths correctly even under a
custom --out (its grandparent-of-graph.json fallback pointed at the wrong dir).
Regression tests in tests/test_build_merge_hyperedges_and_prune.py.
With `graphify extract --out <dir>`, the semantic cache write and read
sides disagreed on both location and key anchoring, breaking the cache
round-trip in two ways:
- Checkpoints (#1990): `_checkpoint_chunk` called `save_semantic_cache`
with only `root=target`, so per-chunk recovery checkpoints were written
under `<corpus>/graphify-out/` while the reader consulted
`<out>/graphify-out/` — creating an unwanted graphify-out/ inside the
analyzed source tree and making every interrupted run re-extract (and
re-bill) completed chunks.
- Final save (#1991): cli.py passed `root=out_root`, so corpus-relative
`source_file` paths resolved against the --out directory, failed
`p.is_file()`, and every result group was silently skipped — the cache
the reader would consult was never populated at all, with no warning.
Fix, following the split the AST cache already uses (#1774):
- `save_semantic_cache` and `check_semantic_cache` gain a `cache_root`
parameter mirroring `load_cached`/`save_cached`: `root` stays the
source-key anchor (content-hash keys, source_file resolution and
relativization), `cache_root` selects where cache files live. Omitting
it keeps `root` for both, so existing callers are unchanged.
- `extract_corpus_parallel` plumbs `cache_root` into `_checkpoint_chunk`.
- cli.py extract passes `root=target, cache_root=out_root` at the cache
read, the checkpoint path, and the final save, and re-anchors the
prune sweep's live hashes to `target` (keys anchored to out_root would
mismatch every entry and sweep the fresh cache as orphaned).
- `save_semantic_cache` now warns loudly when every result group is
dropped because its source_file does not resolve to a real file — the
silent-0-writes failure mode #1991 asked to surface.
Regression tests cover: checkpoint written under cache_root (not the
corpus, no corpus graphify-out/ created), recovery read finds the
checkpoint via the same root/cache_root split, the final-save call shape
writes entries where the reader looks, the all-groups-dropped warning,
and backward compatibility when cache_root is omitted.
Fixes#1990Fixes#1991
ee1df22 narrowed the Claude Code search-guard matcher from "Glob|Grep" to
"Bash" on the premise that dedicated search tools were removed and searches
go through Bash. Current Claude Code routes content search through its
first-class Grep tool (its Bash tool description actively steers away from
shell grep), so the graphify-first nudge never fired on the agent's primary
exploration path and the graph was silently bypassed.
Three-part fix, per the issue's analysis:
- Matcher: "Bash" -> "Bash|Grep" in _claude_pretooluse_hooks. Glob already
fires the read nudge via "Read|Glob", so Grep was the only orphaned tool.
- Guard body: the hook-guard search branch only inspected tool_input.command,
which a Grep call doesn't carry (it has pattern/path/glob). A Grep-shaped
input (pattern present, no command) is now treated as a search — it IS one
by definition — and nudges whenever a fresh graph exists. The Bash
token-matching path is unchanged, and a command-carrying input never
triggers the Grep shape, so non-search Bash calls stay silent.
- Idempotency: "Bash|Grep" added to the four install/uninstall dedup filters
(claude + codebuddy), so upgrading replaces the stale "Bash" hook in place
instead of appending a duplicate — verified against a pre-fix settings.json.
Tests: new regression tests feed Grep-shaped tool_input through
hook-guard search and assert the nudge (with graph), silence (without),
valid PreToolUse JSON, and no blocking; plus a guard that a non-search Bash
command with a stray pattern key does not nudge. Existing matcher assertions
updated across test_search_hook/test_install/test_claude_md/test_codebuddy/
test_hook_strict. Hook+install suites: 397 passed. Full suite: 3224 passed;
the 13 failures are pre-existing on clean v8 in this environment.
Fixes#1986
The PR always wrote gitignore=not no_gitignore into .graphify_build.json, so a
flag-less `graphify extract` after `--no-gitignore` reset it to True and the
git-ignored code silently disappeared again — the exact #1971 complaint. Write
False only when the flag is set (None = leave as-is, mirroring #1886 excludes),
and honor the persisted value for the run when the flag is absent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agents routinely ignore the advisory "run graphify query first" nudge and read
raw files anyway. `graphify install --project --strict` (or `graphify claude
install --strict`) now installs a hook that BLOCKS the first raw source read of a
session via permissionDecision:"deny" with a redirect to graphify query, then
downgrades to the soft nudge — it fires at most once per session (atomic
per-session marker) so it can never strand the agent, and a recent
query/explain/path refreshes a stamp that suppresses it. Claude Code only;
Bash-grep and Glob stay nudge-only; Gemini/Codex/OpenCode are unchanged.
GRAPHIFY_HOOK_STRICT=1/0 toggles at runtime without a reinstall; default installs
are byte-identical (soft nudge).
Also fixes#1840 for the default soft nudge: the guard no longer fires for reads
of out-of-project files, and softens to a non-mandatory nudge when the graph is
stale for the target file. Gating is ~3 stat calls (no corpus walk) and fails
open. Begins the 0.9.19 cycle.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify extract <root> --out <dir>` reduced every node's `source_file` to a
bare filename, so a graph.json could no longer be resolved back to files on
disk by joining source_file onto the scan root.
--out passes the output dir as cache_root to relocate the cache, but that value
also anchored relativization. Every scanned file then failed relative_to(root),
fell through to the #1899 out-of-root fallback, tripped its `updepth > 3`
walk-up guard -- written for a stray ProjectReference, not a whole corpus -- and
collapsed to a basename. On Windows an --out on another drive hit the
cross-drive branch and basenamed unconditionally, which is what the reporter
saw: 0 of ~120k source_files kept a separator. The directory survived only in
the node id slug, lossily (`.`, `/`, `\`, `-`, spaces all map to `_`), and no
other field carried it -- origin_file is stripped (#1516) and the export has no
file table -- so 0% of nodes resolved.
extract() now takes an explicit `root` anchor for source_file/ids/symbol
resolution, which the CLI pins to the scan root independent of where the cache
lives. This completes the cache/anchor decoupling #1774 started and matches
build(root=target), which already anchored on the scan root -- extract was the
lone component keying off --out.
cache_root keeps its fallback-anchor role, so callers that pass the scan root as
cache_root (watch, the no---out CLI path, tests) are unchanged, as are cache
location and out-of-root portability (#1899).
The _partial item-marker approach couldn't fire on the most common truncation
shape: a mid-JSON cut parses to zero items, so a sliced document whose second
slice truncated empty was still stamped complete (the PR's own headline case).
- The adaptive-retry give-up sites now record the chunk's own source files in a
result-level _partial_files list, independent of parsed items; it propagates
through _merge_two / the recursion merges / _merge_into so it reaches both the
per-chunk checkpoint and the run-level manifest stamp.
- _partial_source_files unions _partial_files with the item markers.
- save_semantic_cache seeds an empty group for a named partial file with no
items so its entry is stamped partial, and carries a partial prev entry's flag
forward so a later clean slice merging over it can't re-promote it to complete.
- the CLI final save now passes partial_source_files (computed before the save)
so an empty-parse file isn't written back as a complete entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A chunk whose LLM response is truncated (`finish_reason="length"`) and can't be
recovered by splitting, or that hits the adaptive-retry depth cap, returns a
partial node set. Today that set is checkpointed and written to the content-hash
semantic cache + manifest-stamped as complete, so the incomplete nodes are
served forever until the file content changes or `--force`.
Truncated give-up results are now tagged with an internal `_partial` marker.
`save_semantic_cache` stamps the affected file's entry `partial: True` (detected
from the marker or an explicit `partial_source_files` arg), and `load_cached`
treats a partial entry as a cache MISS, so the file re-dispatches and retries.
The file is also left unstamped in the manifest (like a failed chunk, #933) so
detect_incremental re-queues it on the next incremental run — not only on a full
/ `--force` / content-change run. A file sliced across chunks accumulates via a
partial-aware `merge_existing` peek (`load_cached(allow_partial=True)`) so a
truncated slice is never dropped or silently promoted to complete. Self-heals: a
later complete extraction overwrites the same key with a non-partial entry. The
marker is stripped after the final save so it never leaks into graph.json.
The atomic-write PR left several writers on the old truncate-then-write path.
Route them through write_json_atomic so a crash mid-write can't corrupt them:
- the --no-cluster raw graph.json dump (a core graph.json writer)
- merge-graphs / merge-chunks / merge-semantic output
- .graphify_analysis.json and .graphify_labels.json sidecars
- global_graph.py's global-graph.json and global-manifest.json
write_json_atomic gains an ensure_ascii flag so the raw-UTF-8 writers
(labels, merge outputs) keep byte-for-byte output. Adds tests for the
Windows PermissionError copy fallback and ensure_ascii=False.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
graph.json (the clustered `to_json` write and the `--no-cluster`/merge raw
dumps) and manifest.json were written with a direct `open()`/`write_text`, so a
crash, kill, or disk-full mid-write left a truncated, unparseable file that the
next load or `detect_incremental` then failed on.
Add `write_text_atomic`/`write_json_atomic` in graphify.paths (temp file in the
same directory + `os.replace`; JSON is streamed into the temp, not materialized
as one string) and route the graph.json writers (export.to_json,
cli._prune_graph_json_sources, the merge driver) plus detect.save_manifest
through them. The helper preserves the destination's mode (an atomic replace
never tightens 0644 to mkstemp's 0600), writes through a symlinked destination
(shared-output setups), and falls back to copy-then-delete on a Windows
os.replace lock — matching graphify.cache's existing atomic writer. On failure
the previous file is left intact and the temp removed. Not a power-loss
durability guarantee (no fsync, consistent with the rest of the codebase).
Two gaps the review found in the incomplete-build shrink guard:
- existing_graph_node_count() returned None ("proceed") on a present-but-
unparseable graph.json, so the --no-cluster path could clobber a complete
graph whose file was corrupt/mid-write. It now returns a MALFORMED_GRAPH
sentinel and the caller fails closed, matching to_json's #479 handling.
- A walk that couldn't fully enumerate the corpus (permission-denied subtree,
I/O error) is now treated as an incomplete extraction: detect()/
detect_incremental() already record walk_errors; the extract path consumes
them so a walk-truncated graph can't force-overwrite a complete one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The clustered write is guarded by to_json's #479 shrink check, but the
`--no-cluster` raw-dump path writes graph.json directly and had no guard, so an
incomplete `--no-cluster` build could still overwrite a larger complete graph
with a partial one — the residual gap noted in the original change.
Add `existing_graph_node_count` in graphify.export (mirrors to_json's guard and
respects the graph-size cap) and, on the raw path, refuse the write with exit 1
before the manifest when the build was incomplete and the new graph has fewer
nodes than the existing one — unless --allow-partial is passed. Both write paths
now enforce the same guarantee.
A full `graphify extract` writes the final graph with `to_json(..., force=True)`,
which bypasses the #479 shrink guard. That is correct for a clean build that
legitimately shrinks (dedup collapse, deleted code), but when this run's
extraction was incomplete — an AST pass crashed, or some semantic chunks failed —
forcing the write lets a partial graph silently overwrite a good complete one.
The build now tracks incompleteness (AST-pass failure, semantic-pass crash, or
succeeded < total chunks) and falls back to the shrink guard (force=False) on an
incomplete run, so a smaller partial graph is refused rather than written. It
exits non-zero before the manifest is written, so the manifest is never stamped
for a graph we declined to write and the next run re-attempts. `--allow-partial`
restores force=True to override intentionally.
Note: the `--no-cluster` raw-dump path writes graph.json directly and has no
shrink guard; this change covers the normal clustered build path only.
An untrusted chunk with a non-numeric input_tokens/output_tokens would abort
the whole merge with a TypeError after other chunks had already merged. Coerce
to 0 so a bad token field can't defeat the per-chunk validation guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify merge-chunks` concatenates agent-written `.graphify_chunk_*.json`
files with only a JSON-decode guard, so an oversized payload or a crafted
node/edge id (e.g. `../../etc/passwd`) flowed straight into the merged graph.
Route each chunk through `load_validated_semantic_fragment`, which stats the
file size BEFORE reading it (a multi-GB chunk can't blow up memory), parses the
JSON, and validates the byte/count caps + the node/edge id charset that blocks
path traversal (#825). An invalid chunk is skipped with a warning (filter
semantics; never abort). Left OUT of build_from_json/load_graph_json on purpose:
those must keep loading valid pre-existing graphs.
Also relax two over-strict checks in the shared validator that would otherwise
silently drop whole legitimate chunks (a relaxation — never a new rejection):
- file_type is no longer gated: build coerces any value via _FILE_TYPE_SYNONYMS
(unknown -> "concept", #840), so synonyms like "markdown"/"tool" the loader
maps must not fail validation.
- the id charset now allows Unicode word chars (build's normalize_id preserves
CJK/Cyrillic/accented-Latin ids); the explicit path-separator/".." check still
blocks directory escape.
Corrects the stale module docstring (the validator serves the devin skill path
and merge-chunks, not skill-opencode/codex).
The semantic cache keyed entries on sha256(file content + path) alone, with
no component for the extraction prompt that produced them. After an upgrade
that changed the prompt, every unchanged file was a cache hit and replayed
the older prompt's extraction: the run exited 0, cost.json looked cheap, and
the graph silently carried two prompt generations side by side. The reporter
saw 506 of 512 docs replay an older vintage on a rebuild they expected to be
cold; deleting the whole cache was the only workaround.
output, and invalidating them on every release would re-bill extraction for
unchanged files. Fingerprinting the prompt itself keeps both properties:
entries survive releases that don't touch the prompt, and invalidate only
when it actually changed.
Semantic entries now live under cache/semantic/p{fingerprint}/, mirroring the
AST cache's v{version}/ layout. Both extraction paths pass their prompt: the
Python/CLI path from llm.py's _EXTRACTION_SYSTEM (shared by every backend),
and the skill path via a new prompt_file argument in Step B0/B3 naming the
references/extraction-spec.md the subagents were handed. The fingerprint
normalizes line endings so a CRLF checkout isn't mistaken for a new prompt.
Pre-existing entries predate fingerprinting and have unknowable vintage, so
they are still served rather than re-billing a whole corpus on upgrade — but
check_semantic_cache now warns with the count, turning "no signal at all"
into a visible one. merge_existing refuses to fuse such an entry into a
current-vintage write, which would mix two prompts inside one entry and then
attest the result to a prompt that produced half of it.
Old-fingerprint entries are pruned by liveness only, never swept wholesale
the way stale AST versions are: two hosts with different prompts (verbose vs
compact extraction-spec) can share one graphify-out/, and a wholesale sweep
would have each run delete the other's entries and re-bill on every
alternation. prune/clear/cached_files glob recursively so fingerprinted
entries can't become unprunable orphans (the #1527 failure mode).
The two monolith skills (aider, devin) inline their prompt instead of
shipping a spec sidecar and stay on the unfingerprinted path for now.
- #1925: a missing manifest.json no longer degrades `extract --code-only`
into a full scan that discards the committed semantic layer. An existing
graph.json is a sufficient incremental baseline (detect_incremental treats
an absent manifest as "all new / none deleted"), so out-of-scope doc/paper/
image nodes are preserved while genuinely deleted sources still evict.
- #1920: _stamped_manifest_files now counts hyperedge output, so a doc whose
only chunk output is a hyperedge is stamped instead of re-extracted forever.
- #1923: new namespace/use-aware PHP resolver (mirrors the Java resolver, runs
before the unique-name rewire) so App\Models\Page and an imported
Filament\Pages\Page stay distinct — no more false inherits/imports edge.
- #1922: detect() records ignored files/dirs in a new `ignored` diagnostic
field (the nested-ignore scoping bug itself shipped in 0.9.16 / #1873).
Regression tests added for each; full suite 3325 passed, 3 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify extract --mode deep` over a warm tree was a silent no-op, for
three stacked reasons:
1. The semantic cache ignored mode: deep runs were served standard-mode
entries (and vice versa). check_semantic_cache/save_semantic_cache now
take `mode` (default None, byte-identical when omitted so older
installed callers keep working) and map it to a namespaced kind —
cache/semantic/ for None, cache/semantic-{mode}/ otherwise. The
per-chunk checkpoint in llm.extract_corpus_parallel and the extract /
cache-check consumers thread the run's mode through (cache-check grows
--mode/--deep). cached_files, clear_cache, and prune_semantic_cache
sweep BOTH namespaces; prune uses the same live-hash set for both
(liveness is content-based, mode-independent) so semantic-deep/ can't
regrow the #1527 unbounded-orphan problem and inherits the
files_by_type-derived exclusion gating for free.
2. extract had no --force — the flag was silently swallowed by the
parser's unknown-arg fallthrough. It is now real (plus GRAPHIFY_FORCE
env parity with `update`): force disables the incremental gate so
detection is a full scan and skips the semantic cache READ so every
semantic file re-dispatches, while the post-run save and manifest
stamping still happen.
3. The incremental gate dispatched zero files on a warm unchanged tree
before the cache was ever consulted, so namespacing alone couldn't fix
the repro. In deep+incremental runs the semantic pass now widens to the
full live doc/paper/image set from detect_incremental's files_by_type
(already exclusion-filtered, #1908/#1909) and lets the mode-namespaced
cache decide hits/misses, with a loud count line so the first deep
run's full re-dispatch is visible.
Skill-side threading of mode is deliberately deferred to PR-2; mode
defaults keep generated skills byte-compatible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
.cjs was half-registered: the language maps in build.py and
extract.py already routed it to the JS grammar, but it was missing
from CODE_EXTENSIONS, the extractor _DISPATCH, _LANG_FAMILY, and the
JS resolution/cache suffix sets — so .cjs sources (Electron
main/preload scripts, CommonJS escape hatches in "type": "module"
packages) were silently skipped during a build: classified as
non-code and never handed to the JS extractor.
Add .cjs alongside .mjs in the six lists that gate/route JS files
(detect.py, extract.py _DISPATCH, analyze.py _LANG_FAMILY,
extractors/models.py cache-bypass, extractors/resolution.py resolve
exts, cli.py _HOOK_SOURCE_EXTS), with regression locks mirroring the
.mts/.cts fix (#1607).
Real-world impact: an Electron app whose ~2000-line main.cjs backend
was invisible went from 201 to 294 nodes and 256 to 441 edges on
rebuild.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two coupled excluded-vs-deleted fixes:
#1909 — incremental extract's prune set was derived from the manifest
alone (manifest - corpus), so a file that became excluded without ever
being manifest-listed (every pre-#1897 graph) kept its stale nodes in
graph.json forever. The prune set is now also derived from the existing
graph's own node source_files reconciled against the post-exclude detect
corpus (_stale_graph_sources), restricted to in-root paths; out-of-root
--include/symlinked entries and remote (://) sources are never pruned.
Relative source_files are anchored against both the scan root and the
--out root (the #555/#1899 relativized form). The --no-cluster
incremental early exit never runs build_merge, so an exclusion-only
change now prunes the raw graph.json in place instead.
#1908 — save_manifest retained any prior row whose file still existed
on disk, so an excluded-but-alive file survived as a permanent phantom
that detect_incremental reported as deleted on every run. Full-scan
callers (extract's saves, watch._rebuild_code's saves) now pass the RAW
detect corpus via a new scan_corpus parameter and in-root rows outside
it are dropped; the corpus is deliberately not the #933 stamp-filtered
files dict, so failed-chunk/omitted-doc rows and --code-only doc rows
survive. Subset saves (changed_paths hooks, #917) keep the seeding
default. detect_incremental now splits manifest rows that left the scan
into deleted_files (gone from disk) and excluded_files (alive but out of
scan), mirroring the watch-side #1795 distinction, and the extract
summaries report the two separately.
Ordering matters: extract's cleanup of newly-excluded nodes previously
worked only through the #1908 conflation, so the graph-source prune
lands together with the manifest split to avoid regressing #1909.
The #933 manifest filter built its extracted-set from node/edge
source_file values, which are root-relative on a fresh extraction, and
compared them against files_by_type entries, which are absolute (from
detect()). The raw string membership test therefore never matched, so
every freshly-extracted semantic doc was dropped from the manifest and
re-queued as changed on the next run; only code files and cache-replayed
docs got stamped. The filter now lives in _stamped_manifest_files(),
which resolves BOTH sides against the scan root before the membership
test — the same Path/is_absolute/resolve normalization the #1890
reconciliation uses in graphify.llm. Genuinely omitted zero-node docs
still have no source_file entry and stay unstamped, preserving the
intentional #933 re-queue behavior.
Regression tests: a CLI extract with a mocked corpus extractor returning
root-relative source_files lands the doc in manifest.json with a
non-empty semantic_hash while a zero-node doc stays out; a unit test
covers relative (fresh) and absolute (cache-hit) source_file shapes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
--exclude was honored only by the initial extract scan; update/watch/hook
rebuilds re-ran detect() without it and silently re-indexed excluded
paths. extract now writes the patterns to a graphify-out/.graphify_build
.json sidecar, and _rebuild_code reads and re-applies them on every
rebuild (layered after the ignore files, so they still win). Pre-existing
graphs without the sidecar behave as before. Adds a regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify path` committed each endpoint to _score_nodes()[0]. The full-query
bonus tier only fires when the query equals/prefixes a label, so a query that is
a token subset of the intended label ("Reject-everything judge" vs "Degenerate
Reject-Everything Judge") got no bonus and a node prefix-matching one rare token
("Rejection Summary") could out-score it on IDF alone — anchoring the path on an
unrelated, often disconnected node and returning a false "No path found".
_pick_scored_endpoint() scans the score-ordered list and takes the first
candidate whose label contains EVERY query token, falling back to scored[0] when
none does — so when the head already full-matches (the common case) resolution
is unchanged. Wired into both the `path` CLI and the MCP _tool_shortest_path.
The close-runner-up ambiguity warning now fires only when the picked endpoint is
the raw score head (a full-token override was chosen on coverage, not score, so
the head's margin is irrelevant).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Case 1 — `extract <corpus> --out <dir>`: the graph went to <dir> (cache_root is
already passed to the AST extractor), but detect()'s word-count/stat-index cache
uses the scan root, so a stray graphify-out/cache/ was created inside the corpus
(and left behind even when the run aborted at the no-LLM-key gate). Thread an
optional cache_root through detect() -> cached_word_count() -> _ensure_stat_index()
and pass out_root from the extract CLI, so the stat index lives under --out. Entry
keys are absolute paths, so relocating the index file is safe.
Case 2 — `cluster-only --graph <elsewhere>/graphify-out/graph.json`: outputs
(GRAPH_REPORT.md, re-clustered graph.json, labels, analysis, html) were written to
the CWD's graphify-out/, ignoring where --graph lives. They now write beside the
input graph when it sits in a graphify-out/ dir (another project/tenant's output),
while still falling back to the CWD for an arbitrary archived backup/graph.json —
the restore-into-place workflow #934 pins.
Regression tests for both cases; #934 still passes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
main() was a 2,760-line function that was almost entirely a 2,538-line
if/elif chain dispatching every non-install subcommand (query, path, explain,
diagnose, affected, reflect, save-result, extract, update, cluster-only, label,
build, global, merge-graphs, merge-driver, watch, tree, export, benchmark,
clone, prs, provider, hook*, check-update, and the `graphify <path>` redirect).
Move that chain, plus its 5 chain-only helpers (_StageTimer, _clone_repo,
_default_graph_path, _enforce_graph_size_cap_or_exit, _run_hook_guard) and 4
chain-only constants (_SEARCH_NUDGE, _READ_NUDGE, _HOOK_SOURCE_EXTS,
_GEMINI_NUDGE_TEXT), into graphify/cli.py as dispatch_command(cmd). main() now
does its setup (encoding, stale-skill check, version/help) then
`if dispatch_install_cli(cmd): return` else `dispatch_command(cmd)`.
The chain ends in its own unknown-command exit, so it's called as a statement —
no return-value plumbing. The one cli->__main__ coupling, the path-redirect's
recursive main() call, is handled by a lazy import (_reenter_main), so import
direction stays __main__ -> cli with no cycle. __main__ re-exports every moved
symbol, so importers/tests are unchanged.
__main__.py 3,388 -> 662 LOC (5,368 at branch start, -88%). Verified end-to-end:
version/help/unknown-command, the `graphify <path>` redirect (rebuilds a graph),
and query/path/explain. Full suite unchanged: 3036 passed, 29 skipped; skillgen OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
safishamsi