#1118 — prune stale AST nodes on full re-extraction (#1116)
Stamps every AST-extracted node with _origin="ast" in extract(). On a
full rebuild _rebuild_code drops any AST-marked node absent from the
fresh output even when its source file survives, fixing stale symbols.
Backward-compat: marker-less nodes from pre-1118 graphs survive one
cycle then self-heal.
#1110 — stop reading images and PDFs as garbage in headless extract
Images route through per-backend vision payloads (base64/data-URI/bytes
for claude/openai/bedrock); non-vision backends get _strip_pixels for
graceful degradation. PDFs reuse pypdf. 5MB cap, 20-image chunk limit.
#1159 — Salesforce Apex extractor (.cls, .trigger)
Regex-based extractor: classes, interfaces, enums, methods, triggers,
SOQL/DML edges. No new dependency. Dispatched as .cls and .trigger.
#1107 — Azure OpenAI Service backend (--backend azure)
Uses AzureOpenAI SDK client (from existing openai package). Auto-detects
when AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT both set. Uses
max_completion_tokens (not deprecated max_tokens).
#1103 — live PostgreSQL introspection (--postgres DSN)
graphify extract --postgres "postgresql://..." introspects tables, views,
routines, and FK relations via information_schema (SERIALIZABLE READ ONLY).
Credentials sanitized on error. New graphify[postgres] extra (psycopg3).
Union-resolved llm.py conflict: Azure functions + bedrock images= param.
Fixed test_image_vision.py mock to accept timeout= kwarg (our #1112).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
uv tool install graphifyy runs graphify in an isolated venv, so when a backend
needs anthropic/openai/boto3 the old 'Run: pip install anthropic' advice never
reached it. Worse, claude was the only backend with no [extra] at all, so a user
with ANTHROPIC_API_KEY set could not satisfy it without --with anthropic.
Add an anthropic optional extra (and include it in [all]), and replace the
package-missing ImportError messages with a shared _backend_pkg_hint that points
at 'uv tool install graphifyy[<extra>] --force' first, then the pip/venv path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
F2: replace the header-trust ratio check with an authoritative bounded
streaming-decompression pass. The zip central-directory sizes are
attacker-controlled, so a member that under-declares its size could dodge the
declared-size checks; now every member is stream-decompressed with a hard byte
ceiling, so actual expansion past the cap is caught regardless of the headers.
The cheap declared-size pre-filter stays as a fast reject for honest bombs.
F3: _validate_ollama_base_url now resolves the host, so an alias that points at
a link-local/metadata IP is blocked too, not just literal IPs. The extract
command gains an early gate that turns the metadata block into a clean
'error: ...' exit(2) instead of a deep traceback; a warn toggle keeps the
single user-facing LAN warning in the in-flow call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
F3: OLLAMA_BASE_URL pointing at a link-local or cloud-metadata address
(169.254.x, metadata.google.*, 0.0.0.0) now fails closed instead of only
warning, since no real Ollama host lives there and it is a classic SSRF
target. General LAN hosts still warn-and-allow.
F5: the Fortran C-preprocessor call now passes an absolute path. cpp has no
'--' end-of-options terminator, so an attacker-named corpus file like
'-I/etc/x.F90' could otherwise be parsed as a cpp option; an absolute path
always begins with '/' and cannot be.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A custom provider's base_url is where the full corpus and the user's API key
are sent. A project-local ./.graphify/providers.json travels with a cloned or
shared repo, so loading it silently was a corpus/key exfiltration channel.
Require GRAPHIFY_ALLOW_LOCAL_PROVIDERS=1 to load the project-local file (the
user's own ~/.graphify/providers.json stays trusted), reject non-http(s)
schemes on load and on 'provider add', and warn on plaintext-http egress. We
deliberately do not apply the ingest SSRF guard here so legitimate on-prem
https LLM gateways still work.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
No behavior change. label_communities / generate_community_labels live next
to the backend dispatch (_call_llm, detect_backend) they use, so the lazy
import and separate file are no longer needed. Tests unchanged except the
import path; they still monkeypatch graphify.llm._call_llm.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three compounding bugs caused ~30-50% of semantic chunks to come back
as 'hollow responses' on the claude-cli backend, triggering adaptive
bisection that doubled or tripled the number of subprocess calls.
Root causes
-----------
1. _parse_llm_json only stripped markdown fences when raw.startswith('```').
Claude frequently prepends a short preamble before the fence
('Here are the extracted entities:\n\n```json\n{...}```'), making
the check fail. json.loads then drops the chunk. Each bisected half
may exhibit the same failure, so cost compounds.
2. _call_claude_cli used --append-system-prompt, which layers graphify's
extraction prompt on top of Claude Code's default interactive-agent
prompt ('use markdown formatting', 'output text to communicate with
the user'). The conflicting instructions explain ~50% of the
preambles and fences from (1). Switching to --system-prompt (replace)
eliminates the conflict at the source.
3. claude-cli defaults to Opus, which is overkill for the structured
JSON extraction graphify performs. New GRAPHIFY_CLAUDE_CLI_MODEL env
var lets users opt into haiku / sonnet for big builds. Default
behaviour unchanged when the env var is unset.
Fix
---
- Robust _parse_llm_json: strips fences regardless of position, with a
balanced-brace fallback that scans for the first complete JSON object
in the response. Handles preambles, trailing prose, prose-wrapped
JSON without fences. Diagnostic log on terminal failure includes the
first 200 chars of the response.
- _call_claude_cli switches to --system-prompt.
- _call_claude_cli respects GRAPHIFY_CLAUDE_CLI_MODEL when set.
Tests (tests/test_llm_parser.py)
--------------------------------
- The four PR-body failure modes: preamble+fence, prose+JSON, raw JSON,
total refusal.
- Bonus: uppercase fence tag, unclosed fence, empty response.
- argv shape: --system-prompt present, --append-system-prompt absent.
- argv shape: --model added iff GRAPHIFY_CLAUDE_CLI_MODEL is set.
19/19 tests pass (9 pre-existing in test_claude_cli_backend.py +
10 new). Verified end-to-end on a 800-file repo: 0 hollow responses
after, vs ~30-50% before; output tokens -93%; wall time 44 min -> 4 min.
Backends routed through _call_openai_compat (gemini, openai, kimi,
deepseek, ollama) silently ignored the documented env override when
their backend config dict carried a hardcoded max_completion_tokens.
The dispatcher used:
cfg.get("max_completion_tokens", max_out)
which always returned the config-dict value when the key was present,
shadowing the env-var-resolved max_out.
For gemini specifically, the hardcoded cap of 16384 truncated
extracted-graph JSON mid-response on multi-document chunks (~17 specs
of 100-1500 lines each pushing the output past 16k tokens). Symptom:
cascading 'LLM returned invalid JSON, skipping chunk: Unterminated
string at column 4XXXX' followed by bisect-retry storms that bill
input tokens without producing graph nodes.
Fix: route the same _resolve_max_tokens(...) call that the Claude and
Bedrock paths already use, so the override applies uniformly across
backends.
Verified with gemini-2.5-pro over a 20-doc / 76k-input-token chunk:
output of 36008 tokens emitted without truncation, producing 193
nodes / 223 edges / 23 communities in a single chunk.
The OpenAI-compatible API can return HTTP 200 with an empty `choices`
list or with `choices[0].message = None` (e.g. content-filtered
responses on Gemini, overwhelmed Ollama instances). Without a guard,
both sites raise an unhandled IndexError or AttributeError.
`_call_openai_compat` already documents this hazard ("Ollama can return
HTTP 200 with empty/null content") and has `_response_is_hollow` logic
downstream, but `_response_is_hollow` is unreachable when the choices
list itself is empty. The new guard closes that gap.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(extract): force UTF-8 encoding on subprocess + loud failure on chunk errors
On Windows cp1252, subprocess.run(..., text=True) without encoding= raises
UnicodeEncodeError for chars like → ✅ ≥ in chunk content. Both
_call_claude_cli (llm.py:426) and the _call_llm claude-cli branch (llm.py:959)
lacked encoding=.
- Add encoding="utf-8" to both subprocess.run sites.
- Track failed_chunks in extract_corpus_parallel merged result dict.
- Print [graphify] WARNING: N/M semantic chunk(s) failed summary to stderr at
end of run when any chunk failed, so silent partial failures are visible.
- Add tests/test_charmap_encoding.py: 10 regression tests covering subprocess
encoding kwarg, loud-failure summary, and substitution validation.
Closes#3
* style: drop fork-local issue refs from llm/test comments
- Universal -h/--help/-? guard after cmd dispatch: any help flag anywhere in
argv stops execution and prints "Run 'graphify --help'" instead of triggering
the subcommand — cursor/kiro/gemini install --help no longer silently installs;
benchmark --help no longer crashes with FileNotFoundError (#821)
- --version / -v / version subcommand: print graphify {__version__} and exit (#818)
- GRAPHIFY_OLLAMA_NUM_CTX=<invalid> now falls through to auto-derived num_ctx
instead of hardcoding 131072 (the cap that causes OOM on constrained VRAM);
pinned num_ctx < estimated input now triggers an explicit truncation warning
with a suggested --token-budget correction (#820)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
#796: add edge_data()/edge_datas() helpers in build.py that tolerate
MultiGraph/MultiDiGraph; replace all G.edges[u,v] 2-tuple call sites in
__main__.py, serve.py, wiki.py, export.py, analyze.py, benchmark.py;
fix same pattern in 10 skill file inline heredocs
#795: all 12 skill files now short-circuit on /graphify --help or -h
and print the Usage block without running any pipeline steps
#792 (hollow response): add _response_is_hollow() predicate in llm.py;
when Ollama (or any backend) returns empty/null/whitespace content or a
parsed result with no nodes/edges, rewrite finish_reason="length" so
_extract_with_adaptive_retry bisects the chunk instead of silently
dropping it; applied to _call_openai_compat, _call_claude, _call_bedrock
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
#791: per-repo fcntl flock in _rebuild_code prevents concurrent hook rebuilds
from exhausting memory; changed_paths wired through so only modified files are
re-extracted; stale nodes evicted on deletion; SIGALRM watchdog with
GRAPHIFY_REBUILD_TIMEOUT; Darwin-aware RLIMIT_DATA memory cap
#792: remove hard 8-worker cap (GRAPHIFY_MAX_WORKERS env var); add
--max-workers, --token-budget, --max-concurrency, --api-timeout CLI flags
to graphify extract; fix ollama API key gate for loopback URLs; explicit
timeout on OpenAI client (GRAPHIFY_API_TIMEOUT, default 600s); per-chunk
progress prints during extraction
#453 + #785: rename .agent -> .agents throughout antigravity install/uninstall;
add trigger:always_on YAML frontmatter to _ANTIGRAVITY_RULES so Antigravity
recognises the rules file
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adaptive retry only recovered from `finish_reason="length"` (output
truncation). It did not handle the other shape of overflow: the API
rejecting the prompt outright with a 400 because the input plus
`max_completion_tokens` doesn't fit in the model's context window.
This shows up immediately on local OpenAI-compatible servers (LM
Studio, llama.cpp, vLLM) where the default context is small (4K-32K)
and a 60K-token chunk packed for cloud Kimi/Claude blows past it.
Without retry the whole chunk fails with no output, even though the
two halves would each fit cleanly.
Catch a heuristic set of context-overflow exception messages,
classify them as the same kind of recoverable failure as
`finish_reason="length"`, and split-recurse on the same path. Single-
file overflow returns an empty fragment so the rest of the corpus
keeps running. Unrelated errors (rate limit, auth, etc.) still
propagate.
Tested with qwen3.5-9b on LM Studio (32K ctx) against a 215-file
corpus where chunks 4-12 of 12 previously failed; with this change
the overflowing chunks self-heal by splitting in half.
The initial Gemini backend defaulted to 2.5 Flash, but large semantic extraction chunks can benefit from newer models and more output headroom. Move the default to Gemini 3 Flash Preview, add CLI and environment model overrides, and increase the Gemini completion budget while keeping low reasoning effort for cost control.
Constraint: Google exposes Gemini through an OpenAI-compatible chat-completions endpoint
Rejected: Hardcode Gemini 3.1 Pro as the default | higher cost for routine repository indexing
Confidence: medium
Scope-risk: narrow
Directive: Keep --model and GRAPHIFY_GEMINI_MODEL working before changing Gemini defaults again
Tested: uv run --directory vendor/graphify pytest tests/test_llm_backends.py tests/test_chunking.py -q
Not-tested: Live Gemini 3 extraction on the full cloud-edge repo before this commit
Gemini is often the cheaper available quota for low-stakes semantic graph extraction, while OpenAI is a useful fallback. Extend the direct extraction backend registry, CLI validation, docs, and tests so headless extraction can use GEMINI_API_KEY, GOOGLE_API_KEY, or OPENAI_API_KEY without changing the existing Claude and Kimi paths.
Constraint: Gemini supports OpenAI-compatible chat completions at the Google generative-language endpoint
Rejected: Native google-genai integration | higher dependency and response-shape churn for the same chat-completions path
Confidence: medium
Scope-risk: moderate
Directive: Keep backend detection explicit and test every accepted API-key environment variable before adding new providers
Tested: uv run --directory vendor/graphify pytest tests/test_llm_backends.py tests/test_chunking.py -q
Not-tested: Live Gemini/OpenAI API calls; no GEMINI_API_KEY or OPENAI_API_KEY present in this environment
Token-budget chunking cuts the truncation rate but doesn't eliminate
it. Output token cost scales with extractable concept density rather
than input tokens — a chunk that lands on a directory of dense design
docs can pack under the input budget while needing more than
`max_completion_tokens=8192` to express every named concept, so the
response is truncated mid-string and `_parse_llm_json` returns an
empty fragment.
Pre-tuning chunk size to be conservative enough that this never
happens leaves throughput on the table for the common case. Adding a
hard `max_files_per_chunk` cap on top of `token_budget` reintroduces
the "tune a static constant" problem the previous commit set out to
fix.
The fix uses the API's own truncation signal:
1. `_call_openai_compat` and `_call_claude` now expose `finish_reason`
on the result dict (Anthropic's `stop_reason == "max_tokens"` is
normalised to `"length"`).
2. `_extract_with_adaptive_retry` checks it: when truncated, splits
the chunk in half and recurses on each half. Recursion is bounded
by `max_retry_depth` (default 3 → at most 8x fanout per top-level
chunk).
3. Single-file chunks that truncate can't recover and surface a
warning rather than infinite-loop.
4. `extract_corpus_parallel` routes every chunk through the retry
wrapper. The `on_chunk_done` callback fires once per top-level
chunk with the merged result — recursive splits are invisible to
callers.
Three independent improvements to extract_corpus_parallel:
1. Token-aware chunking. Replaces `chunk_size=20` static packing with
a greedy packer keyed on `token_budget` (default 60_000), grouped
by parent directory so related artefacts share a chunk. Pass
`token_budget=None` to fall back to fixed-count packing.
2. Optional tiktoken (added to the [kimi] extra). When available,
`_estimate_file_tokens` uses cl100k_base for accurate counts;
without it, the existing chars/4 heuristic kicks in. Kimi-K2 ships
a tiktoken-based tokenizer so estimates against Moonshot are very
close to truth.
3. True parallelism. The function name said "parallel" but the body
was a sequential for-loop. Now uses ThreadPoolExecutor capped at
`max_concurrency` (default 4 — conservative against provider rate
limits). `on_chunk_done(idx, total, result)` still fires once per
chunk with the original submission idx so progress UIs work
unchanged. `max_concurrency=1` skips the pool to preserve
sequential semantics.
Plus failure tolerance: a chunk raising is now caught, logged to
stderr, and the run continues. Other chunks' results merge as normal.
On a 162-file repo (~125k words), the same work that took ~36 min
sequential under the old code finishes in ~7 min.
Safi