#1174: affected.py load_graph now forces directed=True before
node_link_graph, matching the identical fix in serve.py and __main__.py.
Undirected graphs (directed:false in graph.json) were causing in_edges
to fall back to a direction-blind scan, missing true callers and
reporting false positives. Regression test added.
#1173: post-commit and post-checkout hook bodies now read
graphify-out/.graphify_root before calling _rebuild_code, falling back
to Path('.') if absent. A scoped build (graphify src/) no longer gets
silently expanded to the full repo on the next commit. Tests added.
#1172: Step 9 cleanup split into rm -f for fixed files and
find -maxdepth 1 -delete for the chunk glob. Under fish/zsh an
unmatched glob aborts the entire rm -f line, leaving temp files on disk.
Fixed in the three skillgen source fragments and regenerated.
#1163: detect_incremental type guard on stored mtime — if the manifest
contains a dict-valued mtime (schema drift from older versions), coerce
to None rather than propagating a non-numeric into comparisons.
Regression test added.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
#1170 — replace nohup with cross-platform Python detach in git hooks.
Git for Windows MSYS has no nohup so post-commit/post-checkout hooks
silently failed. Now uses subprocess.Popen with DETACHED_PROCESS |
CREATE_NEW_PROCESS_GROUP on Windows, start_new_session=True on POSIX.
Quoting-safe (argv list). Fixes#1161.
#1169 — fix _is_sensitive false positives on topic-mentioning filenames.
token-economics-of-recall.md and password-policy-discussion.md were
silently dropped as secrets. Generic keywords (token/secret/password)
now only fire when the keyword ends the filename stem or the stem is
≤2 words. Specific patterns (.env/.pem/id_rsa etc.) remain unconditional.
#1165 — fix multi-word endpoint resolution in _score_nodes.
graphify path "AuthService" "UserRepo" never fired the exact-match bonus
because per-token comparison never equalled the full label. Now joins
normalized tokens and compares against the full label and its tokenized
form. O(1) per node, affects query_graph and shortest_path uniformly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
#1118 — prune stale AST nodes on full re-extraction (#1116)
Stamps every AST-extracted node with _origin="ast" in extract(). On a
full rebuild _rebuild_code drops any AST-marked node absent from the
fresh output even when its source file survives, fixing stale symbols.
Backward-compat: marker-less nodes from pre-1118 graphs survive one
cycle then self-heal.
#1110 — stop reading images and PDFs as garbage in headless extract
Images route through per-backend vision payloads (base64/data-URI/bytes
for claude/openai/bedrock); non-vision backends get _strip_pixels for
graceful degradation. PDFs reuse pypdf. 5MB cap, 20-image chunk limit.
#1159 — Salesforce Apex extractor (.cls, .trigger)
Regex-based extractor: classes, interfaces, enums, methods, triggers,
SOQL/DML edges. No new dependency. Dispatched as .cls and .trigger.
#1107 — Azure OpenAI Service backend (--backend azure)
Uses AzureOpenAI SDK client (from existing openai package). Auto-detects
when AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT both set. Uses
max_completion_tokens (not deprecated max_tokens).
#1103 — live PostgreSQL introspection (--postgres DSN)
graphify extract --postgres "postgresql://..." introspects tables, views,
routines, and FK relations via information_schema (SERIALIZABLE READ ONLY).
Credentials sanitized on error. New graphify[postgres] extra (psycopg3).
Union-resolved llm.py conflict: Azure functions + bedrock images= param.
Fixed test_image_vision.py mock to accept timeout= kwarg (our #1112).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fixes#777. Relativize manifest keys, .graphify_root, and cache source_file fields on persist; re-anchor on load. In-memory callers still see absolute paths. Symlink round-trip fixed in follow-up commit 8f09326.
F2: replace the header-trust ratio check with an authoritative bounded
streaming-decompression pass. The zip central-directory sizes are
attacker-controlled, so a member that under-declares its size could dodge the
declared-size checks; now every member is stream-decompressed with a hard byte
ceiling, so actual expansion past the cap is caught regardless of the headers.
The cheap declared-size pre-filter stays as a fast reject for honest bombs.
F3: _validate_ollama_base_url now resolves the host, so an alias that points at
a link-local/metadata IP is blocked too, not just literal IPs. The extract
command gains an early gate that turns the metadata block into a clean
'error: ...' exit(2) instead of a deep traceback; a warn toggle keeps the
single user-facing LAN warning in the in-flow call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
.docx/.xlsx are zip+XML containers parsed during a corpus scan with no size
guard, so a few-KB zip-bomb could decompress to gigabytes and OOM-kill the
process. Screen every office/PDF file before openpyxl/python-docx/pypdf touch
it: an on-disk size cap, a total-uncompressed cap, and a compression-ratio
check on the archive members. Files that exceed the caps are skipped (treated
as empty) rather than parsed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
os.walk order is non-deterministic across runs (filesystem b-tree order
shifts with cache/mount state), causing first-writer-wins decisions in
cross-file resolution to produce different node IDs each run; sorting
all_files and per-FileType lists stabilises graph.json without changing
any parent-child or connectivity semantics (closes#1090)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ArkTS (.ets) is the primary language for HarmonyOS/OpenHarmony
application development, used by projects like SceneBoard. The
tree-sitter TypeScript parser already handles .ets files for AST
extraction — the detect module just wasn't recognizing them.
Without this, `detect()` silently skips all .ets source files,
missing ~90% of code in OpenHarmony codebases.
Co-authored-by: Autumn <autumn@AutumndeMacBook-Pro.local>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* feat(bash): harden extractor — literal filtering, entrypoint nodes, AST-ancestry-aware command detection
Builds on tree-sitter-bash extractor from #866. Two correctness/security
improvements to bash extraction in graphify/extract.py:
1. Reject command/process substitutions at extraction time. Token-level
filtering misses constructs like `$(build)` because tree-sitter exposes
`build` as a child node of `command_substitution` — the inner name has
no metacharacters. Added `is_inside_expansion(node)` that walks
`node.parent` until it finds `command_substitution` or
`process_substitution`. Used as a gate in both `walk` and `walk_calls`.
Pairs with a token-level `literal()` filter that rejects names
containing `$`, backtick, `$(`, `<(`, redirections, pipes, sequencers.
2. Entrypoint node. Every .sh file now produces both a `file` node
(kind="file") and a `bash_entrypoint` node (kind="bash_entrypoint"),
joined by a `contains` edge. A separate top-level `walk_calls(root,
entry_nid, ...)` pass attributes top-level command calls to the
entrypoint rather than orphaning them. Matches the entrypoint pattern
other-language extractors use. Node metadata gains language+kind.
Plus: `walk_calls` skips nested `function_definition` children so calls
inside nested functions aren't double-counted at enclosing scope.
Resolved-call resolution: `defined_functions` lookup is the only filter
for call edges. User-defined functions named like external commands
(install, find, git, ...) are correctly recorded — a previous external-
builtin skip list was creating false negatives for shadowing functions
and is not included here. Skip list belongs with raw/unresolved call
recording (not in this PR).
Devtools (bundled): pyproject.toml gains [dependency-groups] dev (ruff,
pyright, pre-commit, hypothesis, pip-audit) plus minimal [tool.ruff],
[tool.ruff.lint], [tool.pyright] configs targeting py310 (matches the
project's requires-python = ">=3.10").
Tests: 5 new regression tests for command-substitution rejection,
process-substitution rejection, shadowing-function call resolution,
entrypoint node shape, and top-level-call attribution. 826/826 pass
(was 821); 15/15 bash-relevant tests pass (was 10).
* feat(detect): parse macOS/BSD and GNU env(1) shebang option forms
Upstream's _shebang_file_type parses shebangs via line[2:].split() and only
handles `#!/usr/bin/env <interp>`. Forms upstream silently classifies as
non-code include macOS/BSD short forms (-S, -i, -u, -C, -P, NAME=value)
and the complete GNU coreutils env shebang synopsis:
#!/usr/bin/env -[v]S[option]... [name=value]... command [args]...
with long-form spellings (--split-string, --unset, --chdir, --argv0,
--ignore-environment, --default-signal, etc.), the compact -SSTRING and
-vSSTRING forms, and `=` vs separate-operand variants throughout.
Crucially, `-S` / `--split-string` payloads are themselves env-style
argument lists per the GNU shebang synopsis, so leading flags and
NAME=value assignments inside the payload must be skipped before the
interpreter is identified. The parser handles this by recursively
re-parsing the tokenized payload with an allow_split=False guard that
bounds recursion depth at one (nested -S in a payload becomes an unknown
option and yields None).
Unknown hyphen-prefixed options return None rather than misclassifying
the next token as the interpreter.
_shebang_file_type becomes a 4-line wrapper. Read buffer raised 128 -> 256
to accommodate longer env -S strings.
Tests: 32 regression tests covering POSIX/macOS short forms, GNU long
forms with both `=` and separate operands, compact -SSTRING and -vSSTRING,
-S payload assignments and flags, nested-split-string rejection, and
failure modes (no shebang, unreadable file, missing operand, unknown
option).
* fix(skills): enforce semantic fragment validation in OpenCode + Codex merges (#825)
Closes#825. Adds graphify.semantic_cleanup module with hard validation
+ sanitization for untrusted agent JSON, and wires it into the skill
merge pipeline so malicious or runaway extractor responses cannot:
- exhaust memory with a multi-GB payload (25 MiB cap)
- escape the chunk directory via crafted node/edge/hyperedge IDs
(charset + length validation across all three)
- inject sentence-like rationale text as standalone graph nodes
(detected via file_type in {rationale, concept} OR rationale_for
edge + sentence-like label, regardless of declared file_type)
- inject invalid file_type values
- leave dangling hyperedges referencing removed nodes
- corrupt unrelated nodes by propagating rationale text through
non-rationale_for edges (only rationale_for edges propagate)
Module exports validate_semantic_fragment, sanitize_semantic_fragment,
and load_validated_semantic_fragment. Wired into skill-opencode.md and
skill-codex.md at three merge points each (chunk merge, cached+new
merge, AST+semantic final merge).
Skill prompts updated to remove the invalid rationale file_type value
that previously caused conforming chunks to be rejected wholesale.
Valid set is now {code, document, paper, image}.
Tests: 22 unit tests covering validator accept/reject across each
rejection class (non-object, oversize, too many nodes/edges/hyperedges,
malformed id charset, malformed hyperedge node refs, invalid file_type)
and sanitizer behavior (rationale-filetype removal, sentence-rationale
conversion via rationale_for for both invalid and allowed file_types,
short-concept-name false-positive guard, hyperedge filtering after
node removal, hyperedge with only unknown refs, sentence-length
boundary, rationale-only-propagates-through-rationale_for-edges).
880/880 tests pass.
* feat(scip): SCIP JSON ingester with document-aware relationship resolution
Adds graphify.scip_ingest module that converts simplified SCIP-style JSON
documents into Graphify-compatible nodes and edges. Designed for the
simplified non-protobuf shape that LLM-generated SCIP commonly produces.
Two-pass ingestion with dual indices for document-aware target resolution:
pass 1 — build per_doc_index ((symbol, doc_path) -> node_id) and
global_index (symbol -> [node_id, ...]) across every valid
symbol in every valid document. Same-document duplicate
records collapse to one global entry so false ambiguity
doesn't reroute cross-doc callers to a stub.
pass 2 — emit nodes for indexed symbols, then walk relationships.
Resolution order:
1. same-doc match (per_doc_index)
2. unique cross-doc match (global_index[symbol] len == 1)
3. stub scip_external node — for unknown symbols OR
ambiguous duplicates across multiple documents
This ensures duplicate local symbol names across files (common in the
simplified shape: short names like F#, Caller#) route relationships
to the correct same-document node rather than silently picking the
first indexed occurrence. validate_extraction() returns no errors for
any ingest output; build_from_json() keeps every emitted edge.
Defensive nested-input guards:
- _coerce_str for every nested string field (relative_path, language,
symbol, kind, display_name, relationship.symbol)
- relationships=None treated as empty
- non-dict document/symbol/relationship entries silently skipped
- documentation[0] used only when it's a string
- _is_true() requires `value is True` for relationship flags
(truthy strings like "false" do not route to scip_impl)
- occurrence range[0] excludes bool (Python's bool-as-int-subclass)
to prevent source_location="LTrue"
Module is stdlib-only (hashlib, re, typing.Any). Not wired to the CLI
in this phase — importable as `from graphify.scip_ingest import
ingest_scip_json`.
Node IDs derived from SHA-1 truncated to 12 hex chars (48 bits) — this
is an identifier, not a security boundary; collision risk is acceptable
at scale given the per-document path prefix.
Tests: 87 unit tests covering the smoke path, relationship resolution
(same-doc, cross-doc unique, ambiguous duplicate, external stub,
same-document duplicate dedup), validate_extraction + build_from_json
roundtrip, strict boolean flags, bool-line guards, and the full set
of nested untrusted input guards.
1044/1044 tests pass.
* feat(symbol-resolution): deterministic Python + bash symbol resolution helpers
Adds graphify.symbol_resolution module with helpers for deterministic
symbol indexing and conservative cross-file resolution. Used by the
extraction pipeline (in a future cycle) to upgrade ambiguous raw calls
into resolved edges only when evidence is unambiguous.
Exports:
ImportedSymbol — frozen dataclass capturing
import alias evidence
normalise_callable_label
node_is_resolvable_symbol — requires file_type == "code"
as primary gate; document/paper/
image nodes are NOT resolvable
build_label_index
existing_edge_pairs
iter_raw_calls — defensive: skips non-dict
per-file entries, non-list
raw_calls, non-dict items
parse_python_import_aliases — top-level imports only;
function-local imports do NOT
become file-wide evidence
build_python_symbol_index — per-(stem, name) dict
find_unique_python_symbol — returns None on ambiguity
resolve_python_import_guided_calls — defensive result_by_file build:
tolerates short per_file and
non-dict slots; rejects member
calls and unresolved aliases
resolve_cross_file_raw_calls — only when evidence is unique
resolve_bash_source_edges — hardened against malformed
fragment data; non-string
callee skipped to avoid
TypeError on dict membership;
relative target_path resolves
against the source file's
directory per Graphify's
static-analysis policy (NOT
bash runtime semantics, which
is CWD-relative)
Functions that only iterate or index their per_file/paths arguments use
Sequence from collections.abc for proper covariance. Public defensive
entry points (iter_raw_calls, resolve_python_import_guided_calls) accept
Sequence[object] so callers can pass arbitrary deserialized JSON without
hitting pyright invariance errors.
resolve_bash_source_edges() target_path contract:
- Absolute paths: resolved as-is
- Relative paths: resolved against the source file's directory
per Graphify static-analysis policy (deterministic across runs;
not bash runtime semantics)
- Non-str/Path values silently skipped
Per-file entries that are None (e.g. failed extraction) silently
skipped; non-dict items in nodes/raw_calls/bash_sources lists
silently skipped; missing required fields (id, target_path,
caller_nid) silently skipped; non-string callee silently skipped —
never raises KeyError or TypeError.
Module is stdlib-only (ast, re, dataclasses, pathlib, typing,
collections.abc). Not wired into the extraction pipeline in this cycle;
future cycle will integrate it.
Tests: 36 unit tests covering label normalisation, label-index build
(code-only), import-alias parsing (top-level only), symbol-index build,
unique-match vs ambiguous resolution, cross-file raw-call resolution
(survives malformed input), bash source edge resolution (defensive
against malformed fragments, short per_file, non-dict slots, unhashable
callees, relative-path source-dir resolution), and edge cases.
* feat(security): cap graph.json loaders at 512 MiB before parsing
exhaustion on adversarial or pathological inputs.
- graphify.security: add _MAX_GRAPH_FILE_BYTES + check_graph_file_size_cap
- graphify.serve._load_graph: call cap after existence check
- graphify.__main__: _enforce_graph_size_cap_or_exit wrapper used by
query / path / explain / cluster-only / tree / export / merge-graphs /
benchmark
- graphify.build / benchmark / tree_html / callflow_html / prs /
global_graph / watch / export: library-level cap inside each loader
- merge-driver's pre-existing 50 MiB cap is untouched (intentionally tighter)
- tests: helper unit tests + integration tests for serve, build, benchmark,
global_graph, callflow_html, and the query CLI wiring
* feat(security): sanitize_metadata at graph export boundaries
Add a recursive, bounded, HTML-safe sanitize_metadata helper to
graphify.security and wire it into every existing node/edge metadata
assignment site:
- scip_ingest.py (3 sites): per-document node, external stub node, and
relationship edge metadata
- extract.py (1 site): bash extractor's add_node metadata
- symbol_resolution.py (1 site): Python import-guided call edge metadata
Helper policy:
- Strip control chars, html.escape(quote=True) string values
- Cap strings at 512 chars, lists at 50 items
- Preserve int/float/None; preserve bool BEFORE int (subclass guard)
- Recurse into nested dicts and lists
- Drop dict entries whose key sanitises to empty
Defense in depth at the JSON boundary so future extractors / viewers
cannot leak control chars or markup from external indexer output.
* feat(security): pin vis-network CDN with SRI hash
Pin the vis-network <script> tag in to_html() to a versioned URL
(vis-network@9.1.6) with a sha384 Subresource Integrity hash and
crossorigin="anonymous". Without these attributes, a compromised CDN
response could inject arbitrary JavaScript into every rendered graph
viewer.
Hash verified live against
https://unpkg.com/vis-network@9.1.6/standalone/umd/vis-network.min.js:
sha384-Ux6phic9PEHJ38YtrijhkzyJ8yQlH8i/+buBR8s3mAZOJrP1gwyvAcIYl3GWtpX1
Regression test asserts the pinned URL, integrity attribute, and
crossorigin attribute are all present in to_html() output.
Follow-up: tree_html.py (D3) and callflow_html.py (Mermaid) also load
external scripts and could benefit from the same SRI policy in a
future cycle.
* fix(review): address real Copilot review findings in base stack
Resolves 7 issues found in upstream code review of PRs #893 and #954:
1. extract.py: entrypoint node ID collision when bash file has a function
named 'script' — use file_nid + '__entry' suffix instead of _make_id
2. extract.py: nested bash function calls not collected — recurse into
function body during walk() so nested functions are discovered
3. extract.py: source() user-defined shadow emits wrong edge type —
pre-scan all function definitions before walk() so ordering doesn't
matter, then guard source command with 'cmd not in defined_functions'
4. extract.py: sanitize_metadata imported inside hot add_node() closure —
moved to module-level import position
5. symbol_resolution.py: _bash_make_id() diverged from extract._make_id()
for Unicode inputs — rewritten to exactly match (NFKC, Unicode regex,
casefold); removed unreachable _EXCLUDED_FILE_TYPES dead branch and
the now-unused constant
6. semantic_cleanup.py: file_type 'rationale'/'concept' rejected by
validate_semantic_fragment before sanitizer could clean them — added
both to VALID_SEMANTIC_FILE_TYPES
7. scip_ingest.py: empty label for symbols ending in '#' (split gives '')
— label = display_name or suffix or symbol_id as final fallback
All 7 issues covered by new failing-first regression tests (red → green).
Full pytest suite: 1239 passed, 4 pre-existing env-specific failures.
* fix(review): address PR #956 Copilot findings in watch.py and symbol_resolution.py
- watch.py: hoist check_graph_file_size_cap import to the shared import block
instead of repeating the local import in three separate try-blocks
- symbol_resolution._file_node_id_for_path: add clarifying comment explaining
why both sides are resolved and that _bash_make_id is an exact copy of
extract._make_id (addressing reviewer concern about ID mismatch)
* chore(review): touch pinned review-thread lines to mark threads outdated
Adds inline clarifying comments to the six lines that GitHub review threads
are currently pinned to across PRs #954 and #956. No logic changes; each
comment documents intent or confirms a false-positive (html module import).
* feat(diagnostics): report multigraph edge-collapse risk
Add graphify.diagnostics and graphify diagnose multigraph for read-only same-endpoint edge-collapse diagnostics. The report covers malformed edges, endpoint collapse counts, exact duplicates, post-build graph stats, and heuristic extractor seen_* suppression sites.
Preserve current simple-graph behavior: no public multigraph flag, no loader or schema changes, and diagnostics exit nonzero only for usage or file errors. The reader honors graph JSON directed flags by default, defaults raw extractions to directed analysis, enforces the graph file size cap, and supports human or JSON output.
* feat(multigraph): add runtime compatibility probe
New module graphify.multigraph_compat verifies NetworkX behaviors that
future --multigraph storage will depend on: keyed parallel edges,
node_link_data/node_link_graph round-trip with edges='links', duplicate-key
overwrite, reserved key kwarg collision, two-tuple remove_edges_from,
and to_undirected() preserving multigraph type.
Behavior probe, not version check. Both NX 3.4.2 (Py 3.10 lane) and
NX 3.6.1+ (Py 3.11+ lane) pass. Result cached for the process lifetime.
No call sites added — this PR adds the API surface only. Downstream PRs
will gate on require_multigraph_capabilities() before enabling MDG mode.
Refs: Wave 1 MultiDiGraph implementation order.
* test: filter known third-party analyze warnings
---------
Co-authored-by: vampyre <vampyre@local.net>
detect: add scan_root to return dict so skill can strip absolute prefix
when computing relative subdirectory breakdown; remove stale --no-semantic
flag reference from large-corpus warning (flag does not exist)
skill: clarify fast path checks CWD graphify-out/graph.json (project root);
remove hardcoded --backend gemini from multi-subfolder example — users
should pass whichever backend key they have; expand large-corpus gate
instruction to use scan_root for relative paths, filter graphify-out/
converted sidecars, and handle flat repos with no subdirectories
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fast path: if graphify-out/graph.json exists and user is asking a question
(not an explicit rebuild), skip detect entirely and run graphify query —
prevents the skill from refusing large already-built corpora (#930)
- Raise FILE_COUNT_UPPER 200 → 500 so typical 200-500 file codebases no longer
hit the large-corpus size gate on fresh extraction (#930)
- Subdirectory breakdown now strips the scan-root prefix so agent shows
relative names (core/, service/) not absolute paths rooted at / (#930)
- Document multi-subfolder CLI pattern: graphify extract ./sub/ places
graphify-out/ inside each subfolder; skill clobbers single root graphify-out
when run on subfolders separately (#930)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Security: _is_sensitive now flags underscore-prefixed names (api_token.txt, oauth_token.json) by replacing \b with lookarounds; adds _SENSITIVE_DIRS check on parent path components (parts[:-1]) so .ssh/, secrets/, .aws/ directories are always skipped; aligns both patterns to (?![a-zA-Z]) for consistent underscore-after-keyword behavior (#920)
Fix: --wiki Relationships section always empty because _cross_community_links read community from node attrs (always None) instead of the communities dict; _god_node_article had the same bug and never linked to the owning community; fixed by building a node->community map in to_wiki() and threading it through (#925)
Fix: --watch now respects .graphifyignore; patterns loaded once at startup, handler checks _is_ignored before extension filter so node_modules/, .venv/, build/ churn no longer triggers rebuilds (#928)
Fix: C++ struct inheritance edges via base_class_clause; initialize base="" per iteration to prevent stale carryover (#915)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- hooks.py: drop --path-format=absolute (added git 2.31), validate no newlines in path, anchor relative paths on repo root (#907)
- detect.py: seed save_manifest from existing manifest before loop so incremental callers don't erase untouched file entries (#917)
- cluster.py: drop round(..., 2) from cohesion_score so split threshold 0.05 fires correctly; add resolution param to _partition and cluster; add exclude_hubs_percentile to cluster with majority-vote reattachment (#919)
- report.py: format cohesion with :.2f for display
- __main__.py: wire --resolution and --exclude-hubs into extract and cluster-only commands (#919)
- C++ inheritance already written to disk by analysis agent (#915)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When `root` has at least one direct symlinked child, default to following
symlinks instead of silently dropping their contents. This makes "fake
working dir" patterns (a folder full of symlinks pointing at scattered
source dirs) work transparently — the caller no longer has to know to
pass `follow_symlinks=True`.
Concrete motivating shape:
~/projects/research-corpus/
├── papers -> /external/drive/papers
├── notes -> /external/drive/notes
└── code -> /external/drive/code
Before: `--update` invoked `detect_incremental(root)` without passing
`follow_symlinks=True`, so the scan saw the small set of files actually
inside the root and missed everything reachable only through a symlink.
Result: every legitimate new file was missed, and the manifest paths
were marked as deleted.
After: when `follow_symlinks` is left at its default `None`, `detect()`
runs `_auto_follow_symlinks(root)` (one cheap `iterdir()` + `is_symlink()`
loop) and follows symlinks if any direct child is one. Behaviour is
unchanged for ordinary scans (no direct symlinks → False, as before).
Override is always possible by passing an explicit `follow_symlinks=True`
or `follow_symlinks=False`; existing tests confirming the explicit
behaviour continue to pass unchanged.
Backwards compatibility:
- Type annotation: `bool` -> `bool | None`. Callers passing `True`/`False`
continue to work identically. Callers passing nothing get the new
auto-detect.
- No new dependencies.
- Cheap: one `iterdir()` call once per detect() invocation.
Tests:
- 3 new tests in tests/test_detect.py:
- test_detect_auto_detects_direct_symlink_child
- test_detect_default_does_not_follow_when_no_symlinks
- test_detect_explicit_false_overrides_auto_detect
- Full tests/test_detect.py + tests/test_incremental.py: 49/49 pass.
#873: Remove blanket dot-prefix exclusion from detect.py and
extract.py collect_files(). Add framework caches (.next, .nuxt,
.turbo, .angular, .idea, .cache, .parcel-cache, .svelte-kit,
.terraform, .serverless, .graphify) to _SKIP_DIRS so they stay
blocked. Meaningful dot dirs (.github, .claude, etc.) are now
indexed.
#874: Add _maybe_reload() with mtime+size stat key and threading.Lock
to serve.py. call_tool and read_resource call _maybe_reload() on
every request; the graph reloads automatically when graph.json changes
without restarting the MCP server.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add .sh, .bash, .json to CODE_EXTENSIONS in detect.py so files reach extractor
- Fix test_detect_incremental manifest path collision with new .json extension
- Update test_watch to reflect .json/.sh are now watched extensions
- B-1: only emit source imports for paths that exist on disk
- J-1: replace stat()+read() with bounded read to eliminate TOCTOU
- J-3: move pair_count cap inside loop so it is honoured exactly
- J-4: namespace $ref/extends refs with "ref_" prefix to prevent ID collision
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- skill-windows.md: add graphify-out/ prefix to 5 paths in Step B3 that
were being read/written without prefix (lines 244,245,373,387,388,407,413)
causing semantic cache to always appear empty on Windows (#862)
- detect.py: gate converted office/Google Workspace sidecar paths through
_is_ignored() before appending to files list so .graphifyignore rules
apply to graphify-out/converted/ sidecars (#861)
- detect.py: split manifest entries from {mtime,hash} to
{mtime,ast_hash,semantic_hash}; save_manifest() accepts kind="ast"|
"semantic"|"both"; detect_incremental() accepts kind= and checks the
right hash field so graphify update (kind=ast) no longer prevents
graphify extract from re-running semantic extraction (#857)
- watch.py: all three save_manifest() calls use kind="ast" (AST-only rebuild)
- __main__.py: both extract save_manifest() calls use kind="both"
- backwards compatible: legacy {mtime,hash} entries treated as ast_hash only
775 tests passing
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Astro files have a `---...---` TypeScript frontmatter block at the top
containing nearly all imports, followed by an HTML-with-expressions
template body, and optionally `<script>` blocks for client-side JS.
Feeding the whole file to the JS tree-sitter parser produces a top-level
ERROR node because the template isn't valid JS, so `import_statement`
nodes are never reached.
Mirrors `extract_svelte` (#713) — same regex-rescue strategy:
- Dynamic `import('...')` pass over the full source.
- Static `import X from '...'` pass over the frontmatter block plus any
`<script>` blocks.
Both static and dynamic imports honor tsconfig.json path aliases and
resolve relative paths through `_resolve_js_module_path` so node ids
match the targets that `_extract_generic` produces for sibling files.
Wired in:
- `detect.py:27` — `.astro` added to `CODE_EXTENSIONS`.
- `extract.py:5492` — `".astro": extract_astro` in the dispatch table.
Tests in `tests/test_astro_extraction.py` cover: extension registration,
frontmatter static imports, dynamic import in frontmatter, client-side
`<script>` block imports, frontmatter-less pages (must not crash), and
tsconfig.json path-alias resolution.
* feat(extract): add Pascal/Delphi language support
Adds full AST extraction for Pascal and Delphi source files using
tree-sitter-pascal (https://github.com/Isopod/tree-sitter-pascal).
Supported file extensions: .pas, .pp, .dpr, .dpk, .inc
Extracted nodes:
- File node (the .pas file itself)
- unit / program / library declarations
- class, interface, and helper type declarations
- procedure and function implementations
Extracted edges:
- file --contains--> module
- module --imports--> dependency (via uses clause, resolved to path-based IDs)
- class --inherits--> base class / interface
- class/module --contains/method--> procedure or function
- procedure --calls--> procedure (in-file call resolution)
Key design: uses clause targets are resolved to path-based node IDs by
scanning all Pascal files under the project root (_pascal_project_root +
_pascal_resolve_unit helpers). This avoids dangling import edges that
result from resolving bare unit names like "SysUtils" to IDs that never
match any file node.
Bare procedure calls (e.g. `Reset;` without parentheses) are detected
by inspecting statement nodes whose sole named child is an identifier,
in addition to the standard exprCall nodes used for calls with arguments.
Requires: pip install tree-sitter-pascal
(https://github.com/Isopod/tree-sitter-pascal)
If not installed, extract_pascal returns {"nodes":[], "edges":[], "error": ...}
so the rest of the pipeline is unaffected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(extract): add Lazarus .lfm and .lpk file support
Adds two new extractors for Lazarus IDE-specific file formats:
extract_lazarus_form() — .lfm (Lazarus Form files)
.lfm files are text-based UI component trees. The extractor parses
`object Name: TClassName ... end` blocks to build a containment graph
of form components, and captures `OnXxx = HandlerName` event bindings
as `references` edges (context: "event") linking each component to
its handler procedure.
extract_lazarus_package() — .lpk (Lazarus Package files)
.lpk files are XML package definitions. The extractor reads the
package name, required package dependencies (→ imports edges), and
listed unit files (→ contains edges). Unit names are resolved to
path-based node IDs via _pascal_resolve_unit so they connect to the
same nodes produced by extract_pascal on .pas files.
Both extensions added to CODE_EXTENSIONS in detect.py and to _DISPATCH.
13 new tests in test_pascal.py cover both extractors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pascal): fix dangling inherits-edge targets and capture all base classes
The declType/typeref handler built inherits edge targets with
_make_id(_read(child)) — just the bare class name. But class nodes
use _make_id(stem, type_name), so targets never matched, making the
entire class hierarchy invisible in the graph.
Add _pascal_class_stem_cache and _pascal_resolve_class(): strips the
conventional T/I prefix, locates the defining file by stem lookup
(same cache mechanism as _pascal_resolve_unit), and returns the
correct _make_id(file_stem, class_name) ID. RTL/unresolvable bases
(e.g. TObject) fall back to _make_id(bare_name) with an explicit
stub node, following the same pattern as the Python extractor.
Also remove the `break` that stopped after the first typeref, so
all parents are captured (e.g. class(TBase, IInterface)).
Extend test_pascal_no_dangling_edges to also assert that within-file
edge targets (contains, method, inherits, calls) resolve to real nodes.
* feat(extract): add Delphi .dfm form file support
Adds extract_delphi_form() for Delphi Form files (.dfm), which use the
same `object Name: TClassName ... end` text syntax as Lazarus .lfm files.
Binary .dfm files (FF 0A magic header) are skipped gracefully with an
informative error message so the pipeline is unaffected. Text .dfm files
are parsed identically to .lfm: component containment (`contains` edges)
and event handler references (`references`, context "event").
Adds .dfm to _DISPATCH and CODE_EXTENSIONS.
10 new tests in test_pascal.py, including a regression test for the
binary-format detection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Add .lpr extension and fix removesuffix in Pascal extractor
Address review feedback from @safishamsi:
- Add .lpr (Lazarus program file, identical syntax to .dpr) to _DISPATCH
in extract.py and CODE_EXTENSIONS in detect.py so Lazarus project entry
points are indexed. Completes the promised Lazarus IDE support.
- Replace rstrip("()") with removesuffix("()") in the call-resolution
dict comprehension for precise suffix removal (rstrip strips individual
characters, not the literal string "()").
- Add .lpr assertions to test_pascal_dispatch_registered and
test_pascal_detect_extensions_registered.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Simeon Bodurov <simeon.bodurov@speedy.bg>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds support for Quarto markdown (.qmd) files by:
- Adding '.qmd' to document file extensions in detection
- Updating export logic to handle .qmd in filename sanitization
- Adding .qmd extractor dispatch using the existing markdown extractor
- Updating watch comments to include .qmd files
Add `.luau` to CODE_EXTENSIONS and route it through extract_lua
(tree-sitter-lua). Roblox first-party code uses .luau; without this,
graphify silently skipped 379/479 files on a real Roblox codebase
and the resulting graph was dominated by vendored .lua dependencies.
tree-sitter-lua doesn't parse Luau type annotations, but it
successfully extracts function declarations and call edges from
Luau source — verified on a 379-file Roblox codebase (1265 nodes,
1471 edges, 236 communities).
A dedicated tree-sitter-luau grammar would be a richer long-term
fix; this is the minimal change to make Luau projects work today.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`detect_incremental(root)` always called `detect(root)` without forwarding
the `follow_symlinks` kwarg. As a result, corpora that include symlinked
sub-trees pointing to directories outside the scan root (e.g. a
`state_of_truth/` symlink pointing at `~/.hermes/state_of_truth/`) were
visible to a full `detect()` run with `follow_symlinks=True` but invisible
to any subsequent `--update` run. The incremental scan would then either
report no changes (silently dropping legitimate new files) or repeatedly
re-extract a phantom subset, depending on what was reachable without
crossing symlinks.
Add a keyword-only `follow_symlinks` parameter to `detect_incremental()`
and forward it. Default stays `False` for backwards compatibility — only
callers that already opt in to symlink following on `detect()` pick up
the new behaviour for incremental runs too.
Test: a corpus with a symlinked directory is invisible with
`follow_symlinks=False`, fully indexed with `follow_symlinks=True`, and
correctly reports zero new files on a second incremental scan after the
manifest is saved.
- Register .groovy and .gradle in CODE_EXTENSIONS, _DISPATCH, and collect_files
- Add _GROOVY_CONFIG (reuses Java import handler)
- Add regex-based _extract_spock_fallback for Spock spec files where
tree-sitter-groovy wraps the body in ERROR nodes due to def-string methods
- _is_spock_file detects via regex scan (def "...") instead of node-label
heuristic, avoiding false negatives on classes whose name differs from stem
- Fallback retains only file node + import edges from tree-sitter pass to
prevent orphaned constructor/method nodes
- Add tree-sitter-groovy>=0.1.2 dependency
- Add 11 tests covering plain Groovy and Spock paths, including apostrophe
in feature method names
- Add Fortran support (26th language): .f/.F/.f90/.F90/.f95/.F95/.f03/.F03/.f08/.F08
via tree-sitter-fortran; capital-F files preprocessed with cpp -w -P
- Add graphify export {html,obsidian,wiki,svg,graphml,neo4j} CLI subcommands
- Add graphify query/path/explain CLI subcommands
- Reduce skill.md from 63KB to 47KB by replacing Python heredocs with CLI calls
- Extend to_html() with node_limit param for auto-aggregation on large graphs
- Add integration tests for all export/query/path/explain subcommands
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit adds full VB.NET language support to graphify, raising the
supported language count from 25 to 26. The implementation follows the
established LanguageConfig pattern used by all other tree-sitter-backed
extractors.
New dependency:
- Adds optional extra [vbnet] backed by tree-sitter-vbnet (published
to PyPI at https://pypi.org/project/tree-sitter-vbnet/0.1.0/).
Install with: pip install graphifyy[vbnet]
graphify/detect.py:
- Added .vb to CODE_EXTENSIONS so VB.NET files are discovered during
corpus ingestion and file-system watching.
graphify/extract.py:
- _import_vbnet(): import handler for imports_statement nodes; emits
imports edges using the namespace_name child text.
- _vbnet_extra_walk(): extra-walk hook that intercepts namespace_block
nodes, emits a namespace node, and recurses.
- _VBNET_CONFIG: full LanguageConfig covering class_block / module_block /
structure_block / interface_block as class types; method_declaration /
constructor_declaration / property_declaration as function types;
invocation call nodes with target/member_access fields.
- VB.NET-specific branches in _extract_generic:
* Class body: VB.NET has no wrapper body node; inherits and implements
are named fields directly on the class_block. Emits separate inherits
and implements edges for each base type, stripping generic arguments.
* Constructor name: constructor_declaration carries no name field in
the grammar; always resolves to New.
* Function body: uses the declaration node itself as body sentinel so
the call-graph pass can find invocations inside methods.
- extract_vbnet(path): public wrapper that delegates to _extract_generic.
- _DISPATCH['.vb']: routes .vb files to extract_vbnet.
pyproject.toml:
- Added vbnet = ['tree-sitter-vbnet'] optional dependency group.
- Added 'tree-sitter-vbnet' to the all extra.
tests/fixtures/sample.vb:
- New fixture file exercising: Imports statements, Namespace block,
Interface, Class with Inherits + Implements, Module, Structure,
Sub/Function/Property methods, and method calls.
tests/test_languages.py:
- Added 13 tests covering: no-error, class/interface/module/structure
detection, method detection, imports relation, inherits edge,
implements edge, and no-dangling-edges invariant.
README.md:
- Updated language count 25 to 26.
- Added VB.NET to language list and file-extension table.
Safi