Commit Graph

70 Commits

Author SHA1 Message Date
Safi 6a549e42d5 fix: four bugs — affected direction, hook root, glob fish/zsh, manifest drift (#1174 #1173 #1172 #1163)
#1174: affected.py load_graph now forces directed=True before
node_link_graph, matching the identical fix in serve.py and __main__.py.
Undirected graphs (directed:false in graph.json) were causing in_edges
to fall back to a direction-blind scan, missing true callers and
reporting false positives. Regression test added.

#1173: post-commit and post-checkout hook bodies now read
graphify-out/.graphify_root before calling _rebuild_code, falling back
to Path('.') if absent. A scoped build (graphify src/) no longer gets
silently expanded to the full repo on the next commit. Tests added.

#1172: Step 9 cleanup split into rm -f for fixed files and
find -maxdepth 1 -delete for the chunk glob. Under fish/zsh an
unmatched glob aborts the entire rm -f line, leaving temp files on disk.
Fixed in the three skillgen source fragments and regenerated.

#1163: detect_incremental type guard on stored mtime — if the manifest
contains a dict-valued mtime (schema drift from older versions), coerce
to None rather than propagating a non-numeric into comparisons.
Regression test added.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-07 12:40:44 +01:00
Soham Patankar fbe1e9977f feat(skills): progressive-disclosure split for all platforms (generator + drift fence) (#1121)
All-platform progressive-disclosure skill split + generator (addresses #1106).

Splits each platform's skill into a lean core (~615 lines, full default pipeline inline) + on-demand references/, generated from a single source via tools/skillgen with a CI/pre-commit drift gate. 13 hosts split, aider/devin stay monoliths. Also fixes the stale bare-path bugs across the previously hand-maintained variants and moves the always-on blocks into packaged markdown.

Verified: all 5 generator guards pass, byte-verbatim load-bearing slices, lean cores self-sufficient on the default path across all 13 split hosts, references gated to non-default branches, description preserves the graphify-out-query-first clause. Supersedes #1119 (Claude-first subset).

Known follow-up applied on top: harden _always_on() against a missing packaged file so a partial install can't brick the CLI.
2026-06-02 20:48:13 +01:00
Safi a4a615da97 Fix Ukrainian README typo, Unicode vocab regex, add migration note for Java extends->inherits
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-24 20:40:49 +01:00
eugene-krivosheyev 238702b697 Constrained query expansion (#998)
Add constrained query expansion step to /graphify query skill

## Problem

`graphify query` matches via case-folded substring + IDF — no stemming, no synonyms, no cross-language match. When the user's question uses different vocabulary than the graph labels (Slavic → English, "handlers" → "handler", "обработчик" → "handler"), the literal matcher returns 0
hits and the LLM consumer either gets empty subgraph or improvises an ungrounded keyword list from training memory (e.g. expanding "auth" to `{passport, sso, saml, oauth, jwt, scim, …}` regardless of whether those tokens exist in the corpus).

## Fix

Adds a `Step 0 — Constrained query expansion` block to the skill's `/graphify query` section. The LLM consumer extracts vocabulary from graph labels (CamelCase/snake_case split, length-filtered) and is instructed to pick **only** tokens present in that vocabulary, explicitly forbidden from inventing terms.

Effects:
- Bounded improvisation — fantom tokens (terms not in corpus) cannot be expanded, even when LLM "knows" they're related to the intent.
- Honest negative signal — if vocab is poor on a query's topic,  expansion returns [] and the LLM tells the user, instead of  fabricating a search.
- Auditability — selected tokens are printed to the user, and saved into `save-result` for the next --update to graph as Q&A nodes.

## Scope

Patches the canonical `graphify/skill.md`. The 11 host-variant skills (skill-codex.md, skill-aider.md, …) follow the same query-section contract but inline Python rather than calling `graphify query` CLI; those need a parallel patch with the inline form. Happy to follow up in a separate PR after review on the canonical patch.

## Test

On a graph built from the graphify repo itself (1284 nodes, 1454 vocab tokens), an unconstrained expansion of "укрупненная архитектура аутентификации" yields {auth, oauth, jwt, saml, sso, ldap, scim, mfa, 2fa, pin, passport, session, login, token} — of which 11/15 are absent
from the corpus. Constrained expansion against the actual vocab yields {credential, security, token, signature, user, architecture, component, module, overview} — 9 tokens, 0 fantom. Same retrieval, dramatically higher precision.
2026-05-24 20:38:13 +01:00
Safi 4dce16f475 Fix case-sensitive call resolution and cross-language phantom calls (#993, #991)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-23 23:35:25 +01:00
Safi 71b4e57b67 fix Husky 9 hook path, skill.md INPUT_PATH literal, per-worker exception isolation
- hooks.py: add _user_hooks_dir() to target .husky/ instead of .husky/_ on Husky 9 repos (#987)
- skill.md: replace unsubstituted 'INPUT_PATH' literal with '.' in generate() calls (#986)
- extract.py: wrap future.result() per-future so a single worker failure prints a warning instead of falling back to full sequential re-extraction (#943)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-23 23:17:40 +01:00
Safi a234c5238e fix skill large-corpus path and detect output
detect: add scan_root to return dict so skill can strip absolute prefix
  when computing relative subdirectory breakdown; remove stale --no-semantic
  flag reference from large-corpus warning (flag does not exist)

skill: clarify fast path checks CWD graphify-out/graph.json (project root);
  remove hardcoded --backend gemini from multi-subfolder example — users
  should pass whichever backend key they have; expand large-corpus gate
  instruction to use scan_root for relative paths, filter graphify-out/
  converted sidecars, and handle flat repos with no subdirectories

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-18 17:45:29 +01:00
Safi 850c5457da skill: fast path for existing graphs, fix large-corpus gate, fix subfolder output
- Fast path: if graphify-out/graph.json exists and user is asking a question
  (not an explicit rebuild), skip detect entirely and run graphify query —
  prevents the skill from refusing large already-built corpora (#930)
- Raise FILE_COUNT_UPPER 200 → 500 so typical 200-500 file codebases no longer
  hit the large-corpus size gate on fresh extraction (#930)
- Subdirectory breakdown now strips the scan-root prefix so agent shows
  relative names (core/, service/) not absolute paths rooted at / (#930)
- Document multi-subfolder CLI pattern: graphify extract ./sub/ places
  graphify-out/ inside each subfolder; skill clobbers single root graphify-out
  when run on subfolders separately (#930)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-18 17:36:30 +01:00
Safi 299b6baa26 v0.8.4: SQL FK/trigger extraction, deletion pruning fixes, community label normalization
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-15 15:31:22 +01:00
Justin Stottlemyer a08f0de53b docs(skill): clarify that no provider API key is read for semantic extraction
Several host agents (Claude Code in particular) misread the current
phrasing and prompt the user for `ANTHROPIC_API_KEY` when no Gemini
key is set, instead of falling through to subagent dispatch. The
Gemini tip stays as-is; this just adds an explicit one-line callout
that no other provider keys are consulted, with `ANTHROPIC_API_KEY`
called out by name since that is the specific misread we hit in the
wild.

No behavior change — pure documentation.
2026-05-13 23:29:02 -07:00
Safi f7c9a9ad82 fix watch.py labels churn, edges/links schema, shrink-check duplication, and skill.md ID edge cases 2026-05-14 00:29:54 +01:00
Safi 7d1c1096bc fix node ID format in skill.md: parent_dir+stem, not filename-only (fixes ghost-duplicate root cause from #807) 2026-05-14 00:12:04 +01:00
Safi 822abd6e69 fix --update manifest shrink and align file_type enum (#837, #840)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 23:22:37 +01:00
Safi 994b17be46 fix #832 #831 #828 #826 #827: encoding, uv fallback, path scoring, cache, OpenCode trigger
- skill.md + skill-windows.md: add encoding="utf-8" to all read_text()/write_text()
  calls and ensure_ascii=False to json.dumps — bare calls defaulted to system
  codepage on Chinese-locale Windows, mojibaking non-ASCII content (#832)
- skill.md + skill-windows.md: prefer uv tool install --upgrade graphifyy over
  pip in the Step 1 install fallback — pip installs to the wrong env when
  graphify was installed via uv tool (#831)
- serve.py + __main__.py: replace flat substring scoring in _score_nodes with
  three-tier precedence (exact 1000 / prefix 100 / substring 1); _find_node
  returns results ordered exact→prefix→substring; both path CLI and MCP now
  emit a clear error when src and tgt resolve to the same node (#828)
- cache.py: normalize path key via .as_posix().lower() in file_hash so Windows
  junction/case variants hash identically; mirror abs-path normalization from
  save_semantic_cache into check_semantic_cache so relative source_file paths
  resolve the same way on both sides (#826)
- __main__.py: add /graphify skill trigger line to _AGENTS_MD_SECTION — affects
  all 7 AGENTS.md platforms (OpenCode, Codex, Aider, Trae, Hermes, Claw, Droid)
  so typing /graphify actually invokes the skill tool (#827)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-12 12:11:03 +01:00
Safi 95e2c5eb32 fix #811 #803 #801 #808: Unicode IDs, dedup edge keys, direction flip, chunk paths
- extract/_make_id + build/_normalize_id: use NFKC normalization and casefold
  so composed/decomposed Unicode forms produce the same ID; collapse consecutive
  underscores; both functions are now byte-for-byte equivalent (#811)
- dedup: use explicit key-presence check instead of `or` for source/from
  fallback; pop stale from/to keys so they don't leak into graph.json attrs (#803)
- skill --update: use build_merge() to avoid NetworkX round-trip direction flip;
  fix dict merge ordering so explicit source/target win; pull hyperedges from
  G.graph (merged) not new_extraction only (#801)
- skill subagents: inject absolute CHUNK_PATH so Write tool doesn't lose chunk
  files to undefined cwd (#808)
- __main__: skip skill version check during hook-check (runs on every editor
  tool use, must be silent); move warning to stderr

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 18:46:12 +01:00
Safi 23f598f3a0 fix MultiGraph crash, hollow LLM response retry, and skill --help (#796, #795, #792)
#796: add edge_data()/edge_datas() helpers in build.py that tolerate
MultiGraph/MultiDiGraph; replace all G.edges[u,v] 2-tuple call sites in
__main__.py, serve.py, wiki.py, export.py, analyze.py, benchmark.py;
fix same pattern in 10 skill file inline heredocs

#795: all 12 skill files now short-circuit on /graphify --help or -h
and print the Usage block without running any pipeline steps

#792 (hollow response): add _response_is_hollow() predicate in llm.py;
when Ollama (or any backend) returns empty/null/whitespace content or a
parsed result with no nodes/edges, rewrite finish_reason="length" so
_extract_with_adaptive_retry bisects the chunk instead of silently
dropping it; applied to _call_openai_compat, _call_claude, _call_bedrock

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-09 21:17:25 +01:00
Daniel Graham cc63a1711b Make Gemini extraction model configurable
The initial Gemini backend defaulted to 2.5 Flash, but large semantic extraction chunks can benefit from newer models and more output headroom. Move the default to Gemini 3 Flash Preview, add CLI and environment model overrides, and increase the Gemini completion budget while keeping low reasoning effort for cost control.

Constraint: Google exposes Gemini through an OpenAI-compatible chat-completions endpoint

Rejected: Hardcode Gemini 3.1 Pro as the default | higher cost for routine repository indexing

Confidence: medium

Scope-risk: narrow

Directive: Keep --model and GRAPHIFY_GEMINI_MODEL working before changing Gemini defaults again

Tested: uv run --directory vendor/graphify pytest tests/test_llm_backends.py tests/test_chunking.py -q

Not-tested: Live Gemini 3 extraction on the full cloud-edge repo before this commit
2026-05-05 10:11:12 -04:00
Daniel Graham a9cb692961 Prefer accessible semantic extraction backends
Gemini is often the cheaper available quota for low-stakes semantic graph extraction, while OpenAI is a useful fallback. Extend the direct extraction backend registry, CLI validation, docs, and tests so headless extraction can use GEMINI_API_KEY, GOOGLE_API_KEY, or OPENAI_API_KEY without changing the existing Claude and Kimi paths.

Constraint: Gemini supports OpenAI-compatible chat completions at the Google generative-language endpoint

Rejected: Native google-genai integration | higher dependency and response-shape churn for the same chat-completions path

Confidence: medium

Scope-risk: moderate

Directive: Keep backend detection explicit and test every accepted API-key environment variable before adding new providers

Tested: uv run --directory vendor/graphify pytest tests/test_llm_backends.py tests/test_chunking.py -q

Not-tested: Live Gemini/OpenAI API calls; no GEMINI_API_KEY or OPENAI_API_KEY present in this environment
2026-05-05 08:59:37 -04:00
Safi b6ffdbb8dd v0.7.2: Fortran support + export CLI subcommands + skill.md size reduction
- Add Fortran support (26th language): .f/.F/.f90/.F90/.f95/.F95/.f03/.F03/.f08/.F08
  via tree-sitter-fortran; capital-F files preprocessed with cpp -w -P
- Add graphify export {html,obsidian,wiki,svg,graphml,neo4j} CLI subcommands
- Add graphify query/path/explain CLI subcommands
- Reduce skill.md from 63KB to 47KB by replacing Python heredocs with CLI calls
- Extend to_html() with node_limit param for auto-aggregation on large graphs
- Add integration tests for all export/query/path/explain subcommands

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-04 11:17:06 +01:00
Safi d819827ea2 fix #655 #656 #657 #658: cache dir crash, sanitize_label None, rationale file_type, token counting
#655 and #656 already fixed in cache.py and security.py.
#657: add rationale to file_type schema in all 12 skill variants; warn against inventing concept.
#658: add explicit chunk-merge step with token summation before save_semantic_cache.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-02 17:10:20 +01:00
Safi 36e894aa62 v0.6.6: Windows skill bash rewrite, wiki fixes, rationale-node fix, hidden allowlist, --no-viz cluster-only
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-02 14:25:26 +01:00
Safi 14fd14a284 combine what+when in skill description 2026-05-01 18:47:40 +01:00
Safi 7f336acfd9 fix .graphifyignore: correct gitignore semantics + hermetic non-VCS scan + skill auto-invoke
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-01 18:45:12 +01:00
Safi f755aca58f fix kimi temperature 400 error and community label deletion on cleanup (fixes #610, #608)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-29 17:03:44 +01:00
Safi f9c344b546 Remember scan root so graphify update works without a path arg
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-29 09:43:51 +01:00
Safi 5904081d7a Add Kimi K2.6 backend, fix phantom god nodes (#598), fix concept file_type (#601)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-29 08:51:53 +01:00
Safi 4563b043f2 Fix 6 bugs: ID collisions, path portability, alias resolution, HTML controls, desync guard, rationale prompt
- #550: _file_stem() includes parent dir to prevent node ID collisions for same-named files
- #555: extract() relativizes source_file paths before returning for cross-machine portability
- #562: to_json() returns bool; _rebuild_code() writes report/html only if json succeeded
- #563: skill prompts store rationale as node attribute, not separate node; enforce calls direction
- #566: Show All / Hide All buttons added to HTML community panel
- #575: _import_js() resolves tsconfig.json compilerOptions.paths aliases before external fallback

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-27 21:44:22 +01:00
Safi df9b7ec54f fix #479 #451: build_merge(), pre-write shrink guard, label dedup, chunk-suffix prompt block 2026-04-23 20:04:43 +01:00
Safi 2faeed99a2 feat: cross-repo merge-graphs; fix #527 CLAUDE_CONFIG_DIR; fix #524 graphify-out excluded from source scan 2026-04-23 19:59:14 +01:00
Safi 2c49da24f0 feat: graphify clone <github-url> — clone any repo and run full pipeline on it 2026-04-23 19:49:56 +01:00
Safi 64f38acf3e implement #488 #482 #472 #490: legacy schema canonicalization, Java inheritance, aggregated HTML viz, check-update subcommand 2026-04-22 23:28:09 +01:00
Safi 9c3d0fa517 Fix chunk temp files not cleaned up after graph build
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 22:21:37 +01:00
Safi f0ebd07fd5 Improve packaging: uv tool install support, fix interpreter detection on Mac/Linux 2026-04-21 21:11:27 +01:00
Safi c24cde9d9e v0.4.22: fix #429 AST cache root, fix #428 add .mdx to DOC_EXTENSIONS 2026-04-18 09:47:06 +01:00
Safi 8666c6994c v0.4.21: fix #422 cluster-only KeyError total_files, fix #423 --update drops existing nodes 2026-04-17 20:33:06 +01:00
Safi 8a474eb1b3 v0.4.19: fix #390 #298 #410 #401 #385, team workflow docs, Windows/pipx tips
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 12:46:42 +01:00
Safi 01fb51ba78 Fix 9 issues: kiro package data, betweenness perf, wiki step, opencode plugin, cache root, PHP missing edges, Windows stability, cross-file calls
- #352: add skill-kiro.md to pyproject.toml package-data
- #341: guard edge_betweenness at >5000 nodes; use approximate k=100 for suggest_questions on large graphs
- #354/#229: add Step 6b in skill.md to call to_wiki() when --wiki given (before Step 9 cleanup)
- #356: call _install_opencode_plugin() from install --platform opencode path
- #350: add cache_root param to extract() so subdirectory runs keep cache at ./graphify-out/cache/
- #230: PHP class_constant_access_expression emits references_constant edges
- #232: PHP scoped_call_expression (static method calls) emits calls edges
- #287: os.replace fallback for Windows WinError 5; graphify update exits 1 on failure; templates use graphify update . instead of python3 -c
- #348: cross-file call resolution for all languages via raw_calls + global label map pass in extract()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 00:26:11 +01:00
Safi cede5b6ca6 Fix #188: collect_files() now respects .graphifyignore; fix #195: skill.md requires general-purpose subagent type for extraction dispatch
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 18:41:48 +01:00
Safi 2a7d2f4fdc Remove Anthropic API call from transcribe.py - agent generates Whisper prompt itself
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 15:55:22 +01:00
Safi cf5d721bee Add video/audio corpus support with yt-dlp download and Whisper transcription
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 15:40:18 +01:00
Safi 9c829d7a2e release 0.3.25: Aider + Copilot CLI, directed graphs, frontmatter cache, graphifyignore parent discovery, MCP fixes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 00:56:22 +01:00
Safi df26f85648 Add Gemini CLI support and sponsor nudge at pipeline completion (#105)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 17:40:45 +01:00
Safi be2058d5d5 Fix install step: try plain pip before --break-system-packages (#126)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 08:40:17 +01:00
Safi c3817d6144 Apply PRs #82 #93 #102 #109: extension drift, click detection, skill coverage, .graphify_python persistence
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 08:22:01 +01:00
Safi 91cbc74bb1 release 0.3.18: fix watch .graphifyignore, codex hook, trae link, Korean README, save-result CLI 2026-04-09 08:09:36 +01:00
Safi 7ff7bd2e8f v0.3.17: Julia support, smarter chunking, tree-sitter pin, progress output
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-08 22:54:18 +01:00
Safi 1896379283 v0.3.16: NetworkX <3.4 compat, .jsx support, pipx follow-up fix
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-08 22:27:08 +01:00
Safi 359ef86dd4 fix: shebang allowlist validation in all skill files 2026-04-08 19:57:33 +01:00
Safi 2d64c0cd10 fix: Codex PreToolUse hook + --update ghost node pruning (#86, #51) 2026-04-08 19:29:33 +01:00
Safi bd24ddb1d6 fix: hook JSON format, Go pkg scoping, xcassets PDF, cross-file guard, skill file paths (#83, #85, #52, #81) 2026-04-08 19:20:48 +01:00