The #1943 carve-out rescues genuine source under secrets/ / credentials/, but
.tfvars sits in CODE_EXTENSIONS while being Terraform's canonical values store
(routinely real secrets), so it would now be indexed. Add .tfvars to
_SECRET_PRONE_DATA_EXTS so the shared graphable-source predicate drops it in
both Stage 1 and Stage 3; .tf/.hcl stay graphable as genuine infra source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
graph.json (the clustered `to_json` write and the `--no-cluster`/merge raw
dumps) and manifest.json were written with a direct `open()`/`write_text`, so a
crash, kill, or disk-full mid-write left a truncated, unparseable file that the
next load or `detect_incremental` then failed on.
Add `write_text_atomic`/`write_json_atomic` in graphify.paths (temp file in the
same directory + `os.replace`; JSON is streamed into the temp, not materialized
as one string) and route the graph.json writers (export.to_json,
cli._prune_graph_json_sources, the merge driver) plus detect.save_manifest
through them. The helper preserves the destination's mode (an atomic replace
never tightens 0644 to mkstemp's 0600), writes through a symlinked destination
(shared-output setups), and falls back to copy-then-delete on a Windows
os.replace lock — matching graphify.cache's existing atomic writer. On failure
the previous file is left intact and the temp removed. Not a power-loss
durability guarantee (no fsync, consistent with the rest of the codebase).
- #1925: a missing manifest.json no longer degrades `extract --code-only`
into a full scan that discards the committed semantic layer. An existing
graph.json is a sufficient incremental baseline (detect_incremental treats
an absent manifest as "all new / none deleted"), so out-of-scope doc/paper/
image nodes are preserved while genuinely deleted sources still evict.
- #1920: _stamped_manifest_files now counts hyperedge output, so a doc whose
only chunk output is a hyperedge is stamped instead of re-extracted forever.
- #1923: new namespace/use-aware PHP resolver (mirrors the Java resolver, runs
before the unique-name rewire) so App\Models\Page and an imported
Filament\Pages\Page stay distinct — no more false inherits/imports edge.
- #1922: detect() records ignored files/dirs in a new `ignored` diagnostic
field (the nested-ignore scoping bug itself shipped in 0.9.16 / #1873).
Regression tests added for each; full suite 3325 passed, 3 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
.cjs was half-registered: the language maps in build.py and
extract.py already routed it to the JS grammar, but it was missing
from CODE_EXTENSIONS, the extractor _DISPATCH, _LANG_FAMILY, and the
JS resolution/cache suffix sets — so .cjs sources (Electron
main/preload scripts, CommonJS escape hatches in "type": "module"
packages) were silently skipped during a build: classified as
non-code and never handed to the JS extractor.
Add .cjs alongside .mjs in the six lists that gate/route JS files
(detect.py, extract.py _DISPATCH, analyze.py _LANG_FAMILY,
extractors/models.py cache-bypass, extractors/resolution.py resolve
exts, cli.py _HOOK_SOURCE_EXTS), with regression locks mirroring the
.mts/.cts fix (#1607).
Real-world impact: an Electron app whose ~2000-line main.cjs backend
was invisible went from 201 to 294 nodes and 256 to 441 edges on
rebuild.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two coupled excluded-vs-deleted fixes:
#1909 — incremental extract's prune set was derived from the manifest
alone (manifest - corpus), so a file that became excluded without ever
being manifest-listed (every pre-#1897 graph) kept its stale nodes in
graph.json forever. The prune set is now also derived from the existing
graph's own node source_files reconciled against the post-exclude detect
corpus (_stale_graph_sources), restricted to in-root paths; out-of-root
--include/symlinked entries and remote (://) sources are never pruned.
Relative source_files are anchored against both the scan root and the
--out root (the #555/#1899 relativized form). The --no-cluster
incremental early exit never runs build_merge, so an exclusion-only
change now prunes the raw graph.json in place instead.
#1908 — save_manifest retained any prior row whose file still existed
on disk, so an excluded-but-alive file survived as a permanent phantom
that detect_incremental reported as deleted on every run. Full-scan
callers (extract's saves, watch._rebuild_code's saves) now pass the RAW
detect corpus via a new scan_corpus parameter and in-root rows outside
it are dropped; the corpus is deliberately not the #933 stamp-filtered
files dict, so failed-chunk/omitted-doc rows and --code-only doc rows
survive. Subset saves (changed_paths hooks, #917) keep the seeding
default. detect_incremental now splits manifest rows that left the scan
into deleted_files (gone from disk) and excluded_files (alive but out of
scan), mirroring the watch-side #1795 distinction, and the extract
summaries report the two separately.
Ordering matters: extract's cleanup of newly-excluded nodes previously
worked only through the #1908 conflation, so the graph-source prune
lands together with the manifest split to avoid regressing #1909.
- pg_introspect: the psycopg-missing ImportError suggested pip install
'graphify[postgres]', but the PyPI package is graphifyy (double-y)
- detect: add .skill (Markdown with YAML frontmatter agent files) to
DOC_EXTENSIONS so they are no longer dropped as unclassified
- extract: route .skill through extract_markdown for the structural
quick-scan
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Non-anchored patterns from a nested .gitignore/.graphifyignore were
matched against root-relative paths first, so a nested ignore file's
patterns leaked outside its directory. In the wild, .hypothesis/.gitignore
(a bare "*" auto-written by the hypothesis library) ignored the entire
repository and detect() returned 0 files.
Per gitignore semantics, patterns from A/.gitignore apply only to paths
under A. Match every pattern against the path relative to its own anchor
(the anchor dir itself exempt — an ignore file governs its directory's
contents, not the directory), and skip patterns whose anchor does not
contain the target.
Regression introduced with nested-ignore support in 8a5287a (#1206).
tests/test_detect.py: 143 passed, including the existing nested-ignore
and nested-negation tests plus two new regressions.
Fixes#1873
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
detect() only read .gitignore/.graphifyignore in the scan root and its
ancestor directories (up to the nearest VCS root), loaded once before the
walk began. A .gitignore sitting in a descendant directory — e.g.
vendor/sub/.gitignore — was never read, so files/dirs it excluded leaked
into the graph. Real git (and every other gitignore-aware tool) honors
.gitignore at every directory level, not just the ancestor chain.
Extracts the per-directory read+parse logic into a shared
_load_dir_own_ignore() helper (used by both the existing ancestor-chain
loader and the new call site) and invokes it live inside detect()'s
os.walk loop for every directory visited, before that directory's
children are pruned — so a nested ignore file governs its own subtree
with the same closer-file-wins precedence git uses.
Adds three regression tests: nested file exclude, nested directory prune
(the walk never descends into it), and nested negation overriding a
broader root-level rule.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
#1810 — detection read only .gitignore/.graphifyignore, never
.git/info/exclude, which is where git records local-only excludes and
where `git worktree add` writes nested worktree paths. graphify walked
into those worktree copies and the graph exploded (one 5-worktree repo:
9.4k nodes/10MB -> 210k nodes/311MB, ~77% duplicate). detect now loads
info/exclude at lowest precedence (below every per-dir .gitignore, per
git, so a nearer `!` still wins) and resolves the linked-worktree /
submodule case where `.git` is a file to the shared common git dir.
#1809 — two git-hook gaps: (a) post-checkout never honored
GRAPHIFY_SKIP_HOOK, so the var stopped commit rebuilds but not
branch-switch ones; now checked in both. (b) with core.hooksPath shared
across worktrees, a commit in any linked worktree fired post-commit,
which wrote a rogue delta-only graph.json into it and raced deploy/CI
`git clean` against the detached rebuild. Both hooks now short-circuit
in a linked worktree (git-dir != git-common-dir), comparing ABSOLUTE
paths so the primary checkout (where --git-common-dir is the relative
".git") is never false-positived and skipped.
Adds regression tests: info/exclude honored + negation precedence;
both hooks honor the skip env and carry the worktree guard; and an
end-to-end guard check against a real `git worktree`.
Reported by @cdahl86-cyber (#1810, #1809); the worktree guard was
co-developed with @Claude-Madera's PR #1806.
Co-Authored-By: cdahl86-cyber <cdahl86-cyber@users.noreply.github.com>
Co-Authored-By: Claude-Madera <Claude-Madera@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#1807 — piping graphify into a reader that stops early (head,
Select-Object -First N, sed q) disconnected stdout mid-write, raising an
unhandled BrokenPipeError (OSError(EINVAL) on Windows) and exiting 255,
so CI wrappers and agent harnesses read a successful query as a failure.
The console entry point now wraps the CLI body: a closed-pipe reader is
treated as success — stdout is redirected to devnull so shutdown flush
can't raise again, and the process exits 0. Adds a subprocess regression
test.
#1804 — .nox/ (nox virtualenvs, tox's successor, same .nox/ tree shape)
was missing from _SKIP_DIRS while .tox was present, so nox site-packages
got fully indexed (one repo came out 91% venv noise). Added next to .tox
with a regression test.
Reported by @varuntej07 (#1807) and @igorregoir-lgtm (#1804).
Co-Authored-By: varuntej07 <varuntej07@users.noreply.github.com>
Co-Authored-By: igorregoir-lgtm <igorregoir-lgtm@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
.rake files are plain Ruby (Rake's task DSL is ordinary method calls), but the
extension was gated out everywhere, so rake tasks were classified as
unsupported, skipped, and their calls invisible. Add `.rake` to all seven `.rb`
gates the reporter mapped:
- detect.CODE_EXTENSIONS (classification)
- extract._DISPATCH (extractor dispatch)
- extract._LANG_FAMILY_BY_EXT-adjacent language-name map (.rake -> ruby)
- the ruby_member_calls LanguageResolver suffix set
- both `.rb`-suffix filters in ruby_resolution.py (raw-call gather + class-def index)
- analyze language-stats map
- build repo-tag map
The extractor already parsed the content; this is purely extension routing.
Regression test: a `.rake` task's `Widget.tally` resolves cross-file to the
`.rb` definition.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Case 1 — `extract <corpus> --out <dir>`: the graph went to <dir> (cache_root is
already passed to the AST extractor), but detect()'s word-count/stat-index cache
uses the scan root, so a stray graphify-out/cache/ was created inside the corpus
(and left behind even when the run aborted at the no-LLM-key gate). Thread an
optional cache_root through detect() -> cached_word_count() -> _ensure_stat_index()
and pass out_root from the extract CLI, so the stat index lives under --out. Entry
keys are absolute paths, so relocating the index file is safe.
Case 2 — `cluster-only --graph <elsewhere>/graphify-out/graph.json`: outputs
(GRAPH_REPORT.md, re-clustered graph.json, labels, analysis, html) were written to
the CWD's graphify-out/, ignoring where --graph lives. They now write beside the
input graph when it sits in a graphify-out/ dir (another project/tenant's output),
while still falling back to the CWD for an arbitrary archived backup/graph.json —
the restore-into-place workflow #934 pins.
Regression tests for both cases; #934 still passes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two correctness fixes found while analysing the reported 'graphify update
occasionally writes a partial graph.json' bug.
Enumeration (P0): detect()'s os.walk had no onerror handler, so any os.scandir
failure -- a transient PermissionError, or a directory created/deleted mid-walk
by concurrent writes (e.g. benchmarking racing the scan) -- was silently
swallowed and that entire subtree dropped out of the file list with no log, no
error. Downstream that becomes a silently partial graph.json. The walk now
records each skipped directory (surfaced as walk_errors in detect()'s result)
and warns to stderr, while still enumerating the rest of the tree. This stays
visible even when a --force/GRAPHIFY_FORCE rebuild bypasses the shrink guards.
Relatedly, to_json's #479 anti-shrink guard was fail-OPEN: a non-empty but
unreadable existing graph.json (corrupt or mid-write) proceeded with the
overwrite. It now fails SAFE -- refuse and point at force=True -- while an
empty/whitespace existing file (no nodes to lose) still proceeds. The size-cap
check keeps running before any read, so an oversized existing file is not
loaded into memory.
Pascal edges (P1): a class method declared in the interface section and defined
in the implementation section each emitted a "method" edge to the same node id,
and the edge helpers (unlike the node helpers) did not dedup, so ~half of a
Pascal/Delphi graph's method edges were doubled -- inflating degree/centrality
and tripping the #1739 cross-file resolver's single-owner god-node guard. Both
extractors now dedup edges on (source, target, relation).
Adds regression tests for all three behaviours.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When classify_file() returned None — an extensionless, non-shebang file
(Dockerfile, Gemfile, Makefile, Rakefile, LICENSE, ...) or an unsupported
extension — the file left no trace at all: not counted, not listed, nothing.
A user had no way to tell from graphify's output that those files were even
considered.
detect() now collects these into an "unclassified" list in its result, and
`graphify extract` prints a one-line summary after the scan counts:
"N file(s) not classified (no supported extension or shebang), skipped:
Dockerfile, Makefile, ...". Real code/docs are unaffected. This is the
visibility half of the issue; wiring up extractors/manifest handling for
Dockerfile/Makefile-style files remains a separate feature.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@krishnateja7 root-caused this precisely: the files were never reaching
extraction, so the 0.9.7 no-cache-on-empty mitigation could not surface them.
Two discovery-layer filters were the cause:
(a) A bare `snapshots/` directory was pruned as a Jest/Vitest artifact, which
killed legitimate code namespaces like a Rails `app/services/snapshots/`. It is
now pruned only when it actually contains `.snap` files or sits directly under a
JS test root (`__tests__`/`__test__`). `__snapshots__` stays unconditionally pruned.
(b) `_is_sensitive` dropped files on a bare name-keyword hit (device_token.rb,
passwords_controller.rb) even when `classify_file` had already resolved them to
source code. A genuine programming-language source file is now exempt from the
weak keyword heuristic, while real secret stores in data/config formats
(credentials.json, secrets.yaml, .env, .pem, ...) are still caught — those route
through the CODE path for manifest parsing but are deliberately not exempted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#1649: detect_incremental tracks the converted markdown sidecar, and
convert_office_file early-returned whenever the sidecar existed — so a .docx/
.xlsx edited after its first conversion never updated its sidecar and was
reported "unchanged" forever, freezing the graph. It now re-converts when the
source is newer than the sidecar (bumping the sidecar so the hash check catches
it); an unchanged source still skips the rewrite (#1226).
#1655: _md5_file/save_manifest/count_words used plain open()/stat(), which the
Windows file APIs reject for absolute paths over 260 chars unless prefixed with
`\\?\`. Deeply-nested files never hashed, their manifest entry never stabilized,
and detect_incremental re-flagged them as changed every run. A new _os_path adds
the extended-length prefix on win32 for change-detection I/O (mirror of
cache._normalize_path, which strips it for keys). No-op elsewhere.
#1656: detect() re-parsed every PDF/docx/text file to size the corpus on each
run. Word counts are now memoized in the existing content-hash stat index (keyed
by size + mtime_ns), so an unchanged file is parsed once. file_hash's fastpath is
guarded so a word-count-only entry (no hash) can't KeyError, and both writers
augment a co-located entry in place instead of clobbering the other's field.
Full suite: 2906 passed, 3 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`.mts` (ESM) and `.cts` (CommonJS) are the TypeScript counterparts of the
already-supported `.mjs`/`.cjs`, but graphify handled them in neither the
code-extension detection nor the TypeScript grammar path — so they were either
skipped entirely (classified as non-code) or, once detected, parsed with the plain
JavaScript grammar, which silently drops every `type`/`interface`/type-alias
declaration.
Detection — add `.mts`/`.cts` alongside their siblings in:
- detect.CODE_EXTENSIONS (the code / non-code gate)
- analyze._LANG_FAMILY (cross-language edge family → "js")
- extract._JS_RESOLVE_EXTS (import target resolution)
- extract._JS_CACHE_BYPASS_SUFFIXES
- build.py per-node language map (→ "js")
TypeScript grammar — route `.mts`/`.cts` to the plain TypeScript grammar (like
`.ts`; neither is JSX), so their TS-only syntax is captured:
- extract_js() (grammar selector: .mts/.cts → _TS_CONFIG)
- extract._DISPATCH (.mts/.cts → extract_js)
- the use_ts tree-sitter selector
- the typescript_member_calls resolver frozenset
Result: a `.mts`/`.cts` file now extracts identically to the equivalent `.ts`.
Verified on a real 47 KB `.mts` source — 36 nodes for `.mts`/`.cts`, matching `.ts`
exactly (was 22 with the JS grammar; the 14 dropped nodes were TS type/interface
declarations). No new dependency — same tree-sitter-typescript grammar as `.ts`.
Tests: tests/test_typescript_module_extensions.py — membership regression locks for
the extension sets, dispatch routing, and end-to-end parity asserting `.mts`/`.cts`
capture the TS `type`/`interface` nodes and match the `.ts` node set.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Metal Shading Language is C++14, so .metal files are classified as code and routed
through the existing C++ extractor, mirroring the CUDA .cu/.cuh reuse. Also adds
.cu/.cuh/.metal to build.py's _LANG_FAMILY map (they were all missing), so the
cross-language phantom-`calls`-edge filter treats them as C++. README language
table updated.
Ported from PR #1480 by @jiangyq9. This supersedes the earlier #1450 by
@GoodOlClint (same .metal -> C++ approach and fixture) — credit to @GoodOlClint
for the original; #1480 additionally closes the CUDA family-map gap and updates
the docs, and merges clean against current v8.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Makes .xaml a first-class code input. extract_xaml() uses stdlib XML (no new
parser dependency) behind the same DOCTYPE/ENTITY and size guards as the .csproj
extractor, and captures: the root element, named controls (x:Name/Name) and their
control types, {Binding ...} references, x:Class, and -- the useful part -- a
bridge from the view markup to its .xaml.cs code-behind by resolving event-handler
attributes to the matching methods on the partial class.
Ported from PR #1460 by @MikeKatsoulakis onto current v8.
Maintainer hardening on top of the original PR: event resolution is now gated so
it can't fabricate edges. The original matched any attribute value against
code-behind method names, so Content="Save" next to a business method Save(), or
Tag="<a-handler-name>", produced spurious "event" edges. Resolution now requires
(a) the attribute is not a known free-form/identity property (Content, Text, Tag,
Title, ToolTip, Header, ...), (b) the value is a bare identifier, and (c) the
matched method actually has the .NET event-handler signature
(object sender, <T>EventArgs e) -- read from the code-behind source since the C#
extractor does not record parameter lists on method nodes. Added regression tests
for both false-positive cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CUDA is a C++ superset, so .cu/.cuh files parse cleanly with
tree-sitter-cpp (already a dependency). Two registrations wire it up:
- detect.py: add .cu/.cuh to CODE_EXTENSIONS so they're detected and
watched (watch.py's _WATCHED_EXTENSIONS derives from CODE_EXTENSIONS).
- extract.py: route .cu/.cuh through extract_cpp in _DISPATCH, which
also makes collect_files() pick them up (_EXTENSIONS = _DISPATCH.keys()).
Adds tests/fixtures/sample.cu (kernel + __device__/host functions +
struct + includes) and CUDA cases in test_languages.py covering kernel/
device function extraction, structs, includes, and host call edges.
Documents the new extensions in the README extension table and CHANGELOG.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The GRAPHIFY_OUT override (custom output-dir name / absolute path, #686) was only
respected by some readers. `graphify extract` and several commands hardcoded the
literal "graphify-out", so `GRAPHIFY_OUT=custom-out graphify extract` still wrote
to graphify-out/ and downstream query/serve/update looked in the wrong place.
Resolve the output-dir name through graphify.paths everywhere it matters:
- new graphify.paths.out_path()/default_graph_json() helpers
- __main__: extract write dir, cluster-only/label, query/affected/benchmark
defaults, save-result --memory-dir, uninstall --purge, cache-check
- detect: _MANIFEST_PATH, memory/ + converted/ dirs, and the scan-exclude (a
renamed output dir is no longer re-ingested as source input)
- transcribe._TRANSCRIPTS_DIR; build_merge/serve/benchmark/prs graph-path defaults
Default behaviour is unchanged: with no env var everything still uses graphify-out/.
Verified end-to-end (extract -> cluster-only -> query under GRAPHIFY_OUT=custom-out
writes/reads custom-out/, no stray graphify-out/) and added a CLI regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
apm.yml was a .yml document handled by the LLM, so the same package got a
different file-anchored node id from its own manifest than from each dependent's
dependency reference and split into duplicate nodes. New manifest_ingest module
parses apm.yml/pyproject.toml/go.mod/pom.xml deterministically into ONE package
node per package, keyed by name via ids.make_id, plus depends_on edges; routed
to the AST path (CODE) so the LLM never sees them. Package nodes are exempt from
the file-stem prefix remap so the canonical id is stable across manifests and
dedup collapses references to a single hub node.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A .graphifyignore made graphify skip that directory's .gitignore entirely, so a
file excluded only by .gitignore (including neutrally-named secrets the
sensitive-file heuristic misses) got indexed into the graph and could leak into
committed graph artifacts. Read .gitignore first and .graphifyignore last so
their patterns merge and graphifyignore negations still win.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Index PowerShell .psd1 manifests + emit Import-Module/dot-source edges (closes#1331). Builds on the shipped .psm1 support. Validated: full suite 2107 passed, 18 new tests. Thanks @geektan123.
- #1315: add .psm1 to CODE_EXTENSIONS + _DISPATCH so PowerShell modules are indexed
- #1327: synthesize a module node for Swift import targets (new LanguageConfig
flag synthesize_import_module_nodes) so imports edges survive build.py pruning;
strengthen the Swift dangling-edge test to also assert edge targets
- #1317: dedupe parallel edges by (source,target,relation) in the --no-cluster
and incremental update write paths so edge counts are deterministic and
`update` is idempotent
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
stat+MD5 across all files now fans out over a thread pool instead of running sequentially — I/O-bound work releases the GIL so this scales with available disk concurrency.
A single `!` rule in .graphifyignore set a blanket `has_negation` flag that
disabled directory-level pruning for EVERY ignored directory during the
os.walk in detect(). One unrelated `!docs/**` therefore made the walk descend
bin/, obj/, wwwroot/, generated/, … on large repos — a pathological slowdown.
Output stayed correct (the per-file `_is_ignored` filter still excluded those
files), but the walk visited the entire tree.
The bypass was unnecessary: `_is_ignored` already honours negations correctly —
last-match-wins lets `!dir/` un-ignore a directory (so it is not pruned), and
the gitignore parent-exclusion rule means a `!` cannot rescue a file beneath an
excluded directory, so descending an ignored dir to find a re-included file is
never needed. Prune purely on `_is_noise_dir` + `_is_ignored`.
Adds a regression test that tracks os.walk and asserts the ignored dir is never
descended while the negation still re-includes its target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- export.py: guard to_obsidian/to_canvas against dangling community member IDs
(KeyError crash when a node in communities dict is absent from graph, #1236)
- detect.py: NFC-normalize path before hashing Office sidecar filename to fix
macOS NFC/NFD mismatch causing --update to re-extract all Office files (#1226)
- extract.py: add _is_config_json() to skip data JSON files (only extract
package.json, tsconfig.json, eslint, deno, JSON Schema etc.) eliminating
561 orphan key-nodes on large repos (#1224)
- llm.py: add GRAPHIFY_LLM_TEMPERATURE env var + _resolve_temperature() helper;
auto-omit temperature for o1/o3/o4/gpt-5 reasoning models that reject temp=0;
mirrors GRAPHIFY_MAX_OUTPUT_TOKENS precedence pattern (#1191)
- tests: 20 new regression tests across obsidian, detect, extract, llm_backends
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- graphify/_minhash.py: self-contained MinHash/MinHashLSH using pure numpy,
byte-identical hash math to datasketch (sha1_hash32, Mersenne-prime permutation).
Drops datasketch + scipy transitive dep — eliminates EDR hang on Windows where
numpy.testing platform.machine() subprocess spawn was intercepted at import time
- dedup.py: import from graphify._minhash instead of datasketch
- pyproject.toml: replace datasketch>=1.6 with numpy>=1.21
- detect.py: memoize _is_ignored/_eval results in a dict[Path,bool] cache per
detect() call; each unique ancestor dir evaluated once across all sibling files,
eliminating ~42M redundant fnmatch calls on large repos (~34% whole-run speedup)
- tests/test_minhash.py: 11 tests including import-isolation guard asserting scipy
and numpy.testing are not loaded after import graphify.dedup
- tests/test_detect.py: 2 cache tests — correctness (cached==uncached with negation
patterns) and hit-count (each dir evaluated exactly once across siblings)
Closes#1234, closes#1235
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds support for the XML-based `.slnx` solution format (VS 2022 17.13+ replacement for `.sln`). Extracts project references as `contains` edges and build dependencies as `imports` edges. XXE-protected XML parsing with size cap. Wired into `_DISPATCH` and `CODE_EXTENSIONS`. 6 new tests passing.
Co-authored-by: bakgaard <bakgaard@users.noreply.github.com>
#1174: affected.py load_graph now forces directed=True before
node_link_graph, matching the identical fix in serve.py and __main__.py.
Undirected graphs (directed:false in graph.json) were causing in_edges
to fall back to a direction-blind scan, missing true callers and
reporting false positives. Regression test added.
#1173: post-commit and post-checkout hook bodies now read
graphify-out/.graphify_root before calling _rebuild_code, falling back
to Path('.') if absent. A scoped build (graphify src/) no longer gets
silently expanded to the full repo on the next commit. Tests added.
#1172: Step 9 cleanup split into rm -f for fixed files and
find -maxdepth 1 -delete for the chunk glob. Under fish/zsh an
unmatched glob aborts the entire rm -f line, leaving temp files on disk.
Fixed in the three skillgen source fragments and regenerated.
#1163: detect_incremental type guard on stored mtime — if the manifest
contains a dict-valued mtime (schema drift from older versions), coerce
to None rather than propagating a non-numeric into comparisons.
Regression test added.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
#1170 — replace nohup with cross-platform Python detach in git hooks.
Git for Windows MSYS has no nohup so post-commit/post-checkout hooks
silently failed. Now uses subprocess.Popen with DETACHED_PROCESS |
CREATE_NEW_PROCESS_GROUP on Windows, start_new_session=True on POSIX.
Quoting-safe (argv list). Fixes#1161.
#1169 — fix _is_sensitive false positives on topic-mentioning filenames.
token-economics-of-recall.md and password-policy-discussion.md were
silently dropped as secrets. Generic keywords (token/secret/password)
now only fire when the keyword ends the filename stem or the stem is
≤2 words. Specific patterns (.env/.pem/id_rsa etc.) remain unconditional.
#1165 — fix multi-word endpoint resolution in _score_nodes.
graphify path "AuthService" "UserRepo" never fired the exact-match bonus
because per-token comparison never equalled the full label. Now joins
normalized tokens and compares against the full label and its tokenized
form. O(1) per node, affects query_graph and shortest_path uniformly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
#1118 — prune stale AST nodes on full re-extraction (#1116)
Stamps every AST-extracted node with _origin="ast" in extract(). On a
full rebuild _rebuild_code drops any AST-marked node absent from the
fresh output even when its source file survives, fixing stale symbols.
Backward-compat: marker-less nodes from pre-1118 graphs survive one
cycle then self-heal.
#1110 — stop reading images and PDFs as garbage in headless extract
Images route through per-backend vision payloads (base64/data-URI/bytes
for claude/openai/bedrock); non-vision backends get _strip_pixels for
graceful degradation. PDFs reuse pypdf. 5MB cap, 20-image chunk limit.
#1159 — Salesforce Apex extractor (.cls, .trigger)
Regex-based extractor: classes, interfaces, enums, methods, triggers,
SOQL/DML edges. No new dependency. Dispatched as .cls and .trigger.
#1107 — Azure OpenAI Service backend (--backend azure)
Uses AzureOpenAI SDK client (from existing openai package). Auto-detects
when AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT both set. Uses
max_completion_tokens (not deprecated max_tokens).
#1103 — live PostgreSQL introspection (--postgres DSN)
graphify extract --postgres "postgresql://..." introspects tables, views,
routines, and FK relations via information_schema (SERIALIZABLE READ ONLY).
Credentials sanitized on error. New graphify[postgres] extra (psycopg3).
Union-resolved llm.py conflict: Azure functions + bedrock images= param.
Fixed test_image_vision.py mock to accept timeout= kwarg (our #1112).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fixes#777. Relativize manifest keys, .graphify_root, and cache source_file fields on persist; re-anchor on load. In-memory callers still see absolute paths. Symlink round-trip fixed in follow-up commit 8f09326.
F2: replace the header-trust ratio check with an authoritative bounded
streaming-decompression pass. The zip central-directory sizes are
attacker-controlled, so a member that under-declares its size could dodge the
declared-size checks; now every member is stream-decompressed with a hard byte
ceiling, so actual expansion past the cap is caught regardless of the headers.
The cheap declared-size pre-filter stays as a fast reject for honest bombs.
F3: _validate_ollama_base_url now resolves the host, so an alias that points at
a link-local/metadata IP is blocked too, not just literal IPs. The extract
command gains an early gate that turns the metadata block into a clean
'error: ...' exit(2) instead of a deep traceback; a warn toggle keeps the
single user-facing LAN warning in the in-flow call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
.docx/.xlsx are zip+XML containers parsed during a corpus scan with no size
guard, so a few-KB zip-bomb could decompress to gigabytes and OOM-kill the
process. Screen every office/PDF file before openpyxl/python-docx/pypdf touch
it: an on-disk size cap, a total-uncompressed cap, and a compression-ratio
check on the archive members. Files that exceed the caps are skipped (treated
as empty) rather than parsed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
os.walk order is non-deterministic across runs (filesystem b-tree order
shifts with cache/mount state), causing first-writer-wins decisions in
cross-file resolution to produce different node IDs each run; sorting
all_files and per-FileType lists stabilises graph.json without changing
any parent-child or connectivity semantics (closes#1090)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mzt006