Commit Graph

13 Commits

Author SHA1 Message Date
Colby McHenry 1601e4ad12 feat(extraction): add Pascal/Delphi to value-reference edges (unit/class const → constant)
Pascal unit/class `const` sections already extracted as `constant`, so wiring was
add-to-`VALUE_REF_LANGS` + shadow prune + two Pascal-specific fixes:

**`constant`-only targets** — the Pascal extractor emits function parameters
(`const ATarget: TControl`, `var Dest: …`) and class fields as `variable` at
enclosing scope, which would collapse to noisy file-wide targets. Genuine shared
values are all `constant` (`declConst`), so targets are restricted to that kind.
(Unit `var` globals are the rare cost; the parameter/field noise dominates.)

**Sibling-body fix (same as Dart)** — Pascal attaches a proc body (`block`) as a
next sibling of the `declProc` header (the reader scope), both under `defProc`,
not as a child. Extended the existing Dart `function_body` sibling-pull to also
walk a `block` sibling so reads inside the body are found.

Shadow prune wired for `declConst`/`declVar` — a function-local `const X`
shadows a unit `const X`.

Validated S/M/L (horse / mORMot2 / castle-engine): node count identical on/off,
precision sample 100% TP (font/crypto/DB/FFI binding consts); headline is FFI
library-name constants read by hundreds of `external` declarations (impact
2→1880, 1→358). Caveats: low same-file density on app code (cross-unit reads;
horse gave 4 edges), the `constant`-only restriction, a rare tree-sitter-pascal
misparse of `const` params in complex Delphi signatures, and case-insensitivity
(exact-text scan misses a differently-cased reference — a miss, never an FP).
Coverage now 15 languages.
2026-06-16 09:31:29 -05:00
Colby McHenry 2ec74a2c36 feat(extraction): add Dart to value-reference edges (static const / static final constants)
Dart's grammar cleanly separates the constant idiom: a `static_final_declaration`
is exactly a top-level or class-`static` `const`/`final`, while instance fields
and `var` use `initialized_identifier` and method-locals use
`initialized_variable_definition`. A `visitNode` rule maps `static_final_declaration`
to `constant`, so there are no instance/local leaks to guard.

The reader-scan needed a Dart-specific fix: Dart attaches a method/function body
as a NEXT SIBLING of the signature node (which is stored as the reader scope),
not as a child, so the scan saw only the signature and produced zero edges until
it was taught to also walk a `function_body` next-sibling. The shadow prune counts
all three Dart declarator nodes so a method-local `const X` drops a file `const X`.

Validated S/M/L (http / flame-engine/flame / flutter/packages): node count
identical on/off; the headline is a single `static const String iconFont` read by
every Cupertino icon definition (impact 1 -> 1790). The common Dart codegen
suffixes (.g.dart/.freezed.dart/.pb.dart) are already skipped; header-only-marked
generators (JNIGEN _bindings.dart, pigeon .gen.dart) are not, so generated FFI/JNI
bindings are noisy while real source stays clean. Coverage now 15 languages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 00:46:29 -05:00
Colby McHenry a3acc3626e feat(extraction): add Swift to value-reference edges (static let / enum-struct namespaced constants)
Top-level `let` and `static let` in a `struct`/`enum`/`class` (Swift's
shared-constant idiom — it namespaces constants in `enum`/`struct`) now extract
as `constant` and participate in impact analysis; instance stored `let`s stay
`field`, and computed properties (getter, no stored value) are skipped.

Reuses the Kotlin machinery: the name nests
`property_declaration → pattern → simple_identifier` (resolved via
firstSimpleIdentifier), and the reader-scan already matched `simple_identifier`.
Two Swift-specific touches: the value-ref target gate was widened to accept
`struct:`/`enum:` parents (every other language's targets sit at
file:/class:/module:), and computed properties are detected and skipped. Node
creation slots into the existing Swift property_declaration handler, so the
property-wrapper / type-annotation dependency extraction (@Published/@State) is
untouched.

Validated S/M/L (Alamofire / swift-argument-parser / swift-nio): node count
identical on/off, genuine static-let constants, computed properties skipped,
0-1 sibling-type collisions per repo. Coverage now 14 languages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 00:29:04 -05:00
Colby McHenry 907098a703 feat(extraction): add C, Java, C#, PHP, Scala, and Kotlin to value-reference edges
Extends same-file constant-reader impact edges to six more languages, each
requiring a different technique:

**C** — file-scope `static const` / global nodes weren't extracted at all (name
nests in `init_declarator`, the generic fallback missed it). Added a C branch in
`extractVariable` with `cDeclaratorIdentifier` to walk the declarator chain.
Skips bare-`identifier` declarators to kill the macro-prefixed-prototype misparse
FP cluster (a `MACRO RetType fn(args);` mints a spurious type-named global).
Prune switch gains `init_declarator`.

**Java** — `static final` fields already existed but as `field` kind, which the
value-ref gate rejects. An `isConst` predicate (`static` + `final` modifiers)
re-kinds the constant subset to `constant` in `extractField`. Instance `final`
fields stay `field`.

**C#** — same `field`→`constant` approach: `const` modifier or `static readonly`
modifier pair → `constant`; instance `readonly` stays `field`.

**PHP** — constants already extracted as `constant`; the only gap was the
reader-scan. PHP represents a constant reference as a `name` node (not
`identifier`), so bare `X` and the const half of `self::X` / `Foo::X` were
invisible. Added `name` to the scan. No prune wiring needed: a `$var` local is a
`variable_name` — a different namespace — and can never shadow a bare constant.

**Scala** — top-level `val` was already `constant`; `object` and `class` vals
both came out as `field`. Fixed by walking to the enclosing AST definition in the
`val_definition` handler: `object_definition` → `constant`/`variable`; `class`/
`trait`/`enum`/`given` → `field`. Prune switch gains `val_definition`/
`var_definition`.

**Kotlin** — properties weren't extracted at all (`property_declaration →
variable_declaration → simple_identifier` is too deep for the generic path).
Added a `property_declaration` handler in `visitNode`: pulls the nested name,
walks to the enclosing scope for kind (`object`/`companion object`/top-level →
`constant`/`variable`; `class` → `field`; function body → local, skipped). Reader-
scan gains `simple_identifier`; prune switch gains `property_declaration`.

Validated S/M/L across all six languages (hiredis/curl/redis, gson/commons-lang/
guava, AutoMapper/Newtonsoft/efcore, guzzle/monolog/laravel, upickle/cats/pekko,
okio/coroutines/ktor): node count identical on/off, precision guards held, 0
shadow leaks, impact wins confirmed (`INDEX_NOT_FOUND` 4→165, `Curl_ssl` 3→57,
`_resourceManager` 22→1664, `STATE_IN_QUEUE` 1→32, `BLOCKING_SHIFT` 1→24).

C++ was attempted and reverted — tree-sitter-cpp parse fidelity on
template/macro-heavy code leaks class members to file scope as bogus constants;
did not reach the precision bar.
2026-06-16 00:06:43 -05:00
Colby McHenry e8d137ce8d docs(extraction): add value-refs coverage tracker vs the README languages
Add a coverage section to the playbook (§2b) mapping value-reference-edge support
onto the README's 24-row Supported Languages table + the framework list:

- Done (7): TS, JS, tsx, Python, Go, Rust, Ruby.
- Done/inherited (3): Svelte, Vue, Astro — their extractors re-parse the
  <script>/frontmatter as typescript/javascript (in VALUE_REF_LANGS), so a const
  in a .svelte/.vue/.astro script edges its readers for free (verified on a
  synthetic .svelte). Noted in the design doc too.
- Remaining (easy/file-scope): C, Kotlin (top-level), Swift (top-level), Dart
  (top-level), Lua/Luau, R.
- Remaining (needs the Ruby class-scope treatment): Java, C#, Scala, PHP, C++,
  Pascal/Delphi, Objective-C.
- N/A: Liquid.

Plus a note that frameworks are NOT a value-refs axis — it's language-level and
framework-agnostic (validation already ran on Rails/Django/gin/express/…); the
README's framework list is the separate route-extraction feature.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:02:00 -05:00
Colby McHenry b92576f660 feat(extraction): add Ruby to value-reference edges (class-scope constants)
Ruby keeps almost all constants inside a class/module (jekyll's lib: 0 top-level,
58 class-internal), so this needed the class-scope extension flagged since Go.
Three Ruby-specific changes:
- Extract Ruby constants as symbols at all: a `CONST = …` has a `constant`-typed
  LHS (not `identifier`), so they were never extracted — including class-internal
  ones (the variableTypes dispatch skipped class-internal assignments; it now
  makes an exception for `constant`-LHS assignments).
- The value-ref target gate accepts `class:`/`module:` parents, not just `file:`.
- The reader-scan matches `constant` nodes (Ruby represents both a constant's
  definition AND its references as `constant`, not `identifier`).

Effectively Ruby-only: Rust impl consts are already `file:`-parented (ripgrep
stayed at 144 edges) and TS/Python class members aren't constant/variable kind.

Class-scope precision uses a FILE-WIDE target map (not strict same-class) because
Ruby constant lookup is lexical+ancestor — a nested class legitimately reads an
enclosing class's constant (jekyll ERBRenderer→ThemeBuilder::SCAFFOLD_DIRECTORIES;
sinatra AcceptEntry→Request::HEADER_PARAM), and strict matching would drop those.
The only real FP is the same name in sibling classes in one file (~1.7% of
targets on rails; a NameError in valid Ruby, so rare). Net ~98–100% precision.

Validated small/medium/large on public Ruby OSS (sinatra, jekyll, rails): node
count identical on/off, ~98–100% precision, impact win holds (HEADER_PARAM 1→5).
Adds a Ruby test (top-level + class-internal const reads). Design matrix,
playbook (class-scope section now resolved), and CHANGELOG updated. Suite green
(1,558 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 19:51:08 -05:00
Colby McHenry 6281a541e4 feat(extraction): add Rust to value-reference edges
Extend value-reference edges to Rust (module-level `const`/`static`). Rust's
declarators are `const_item`/`static_item` (the module consts) and
`let_declaration` (the local that shadows) — added to the per-grammar prune
switch. A synthetic probe caught the expected shadow FP (a `const TIMEOUT`
shadowed by a local `let TIMEOUT`), fixed by the addition.

No prune-RULE change needed: Rust's `#[cfg]`-conditional consts (`#[cfg(unix)]
const SEP = …; #[cfg(windows)] const SEP = …`) are kept correctly by the
Python-era file-scope-count rule (validated on tokio's io/interest.rs cfg-gated
flags). Consts written inside a config macro (`cfg_aio! { … }`) live in an
unparsed token tree, so the prune's syntax walk doesn't even see them.

Validated small/medium/large on public Rust OSS (ripgrep, tokio, rust-analyzer):
node count identical on/off (incl. 38.8k nodes on rust-analyzer), precision
samples all true positives, zero real shadow leaks, and the impact win reproduces
(rust-analyzer `INLINE_CAP` 2→183, tokio `PERMIT_SHIFT` 1→97).

Adds 2 Rust test cases (const/static read; the local-`let` shadow). Design
matrix, playbook, and CHANGELOG updated. Full suite green (1,557 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 13:21:34 -05:00
Colby McHenry 7b0f204604 feat(extraction): add Python to value-reference edges (+ conditional-def-safe prune)
Extend value-reference edges to Python (module-level `NAME = …`; the `assignment`
declarator added to the per-grammar shadow-prune switch). A synthetic probe found
the expected shadow FP — a module `CONFIG` shadowed by a local `CONFIG = …` —
fixed by that addition.

Python also exposed a flaw in the prune RULE. Python conditionally defines module
constants (`try: X = a; except: X = b`), a very common idiom that binds a name
twice AT MODULE SCOPE. The old "declarators > 1 → drop" over-pruned these,
dropping a real const and all its readers. Refine the rule to compare declarator
count against the number of FILE-SCOPE NODES the name has: a conditional def makes
them equal (kept); a real local shadow makes declarators exceed file-scope nodes
(pruned). Strictly more correct for every language. Track file-scope node count
per name in captureValueRefScope; also suppress same-name value-ref edges (the two
halves of a conditional def would otherwise cross-reference via their own names).

Validated small/medium/large on public Python OSS (requests, sqlalchemy, django):
node count identical on/off, precision samples all true positives, guard holds,
conditional defs now kept (django recall +18 edges), zero same-name edges, and the
impact win reproduces (django `_trans` 1→138, sqlalchemy `COMPARE_FAILED` 1→26).

Adds 3 Python test cases (module read; the local-`=` shadow, verified to fail
without the prune extension; the conditional-def keep case). Design matrix,
playbook, and CHANGELOG updated. Full suite green (1,555 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 13:01:56 -05:00
Colby McHenry c369d5d226 feat(extraction): add Go to value-reference edges (+ per-grammar shadow prune)
Extend value-reference edges to Go. Go's package-level const/var are file-scope,
so the target gate fits as-is. But Go's declarators are const_spec / var_spec /
short_var_declaration, not variable_declarator, so the shadow prune was a no-op
for Go — a package `const Timeout` shadowed by a local `Timeout := …` produced a
false positive (a synthetic probe caught it immediately). Generalise the prune's
declarator scan into a per-node-type switch (a file only contains its own
grammar's nodes, so listing all languages in one switch is safe).

Validated small/medium/large/very-large on public Go OSS (gin, hugo, prometheus,
kubernetes): node count identical on/off at every size (incl. 251k nodes on k8s),
precision samples all true positives, guard invariant clean, and the impact win
reproduces (k8s KubeletSubsystem 3→138, prometheus rdsLabelInstance 1→82).

Adds 2 Go test cases (package-const read; the shadow case, verified to fail
without the prune extension). Design matrix, playbook, and CHANGELOG updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 12:17:01 -05:00
Colby McHenry 3289589296 docs(extraction): add value-refs language-extension playbook (handoff runbook)
Self-contained "Start on language X" runbook: how the feature works + code map,
the precision guards and what is/isn't an FP, the full validation recipe
(probe.sh, FP-hunt SQL, impact-delta + agent-A/B), the per-language checklist
(file-scope-vs-class-scope decision, declarator node type per grammar, extractor
kind check), the git/PR workflow, and the traps already hit (the LIKE
'%valueRef%' filename false-match, SearchResult/.node, initSync ignoring opts,
require-bindings-are-not-FPs). Cross-linked from the design doc.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 11:53:54 -05:00
Colby McHenry 4ca424785b test(extraction): cover the tsx/JSX value-ref read path
tsx is in VALUE_REF_LANGS and was exercised at scale by the excalidraw sweep
(tablerIconProps lives in icons.tsx), but those readers were normal expressions.
Add a unit test for the one tsx-specific path — a const read ONLY inside JSX
(`<Foo x={CONST}/>`), which depends on the reader-scan descending into the JSX
subtree — so it's locked against regression. Passes; no code change needed.
Note in the design doc that tsx needs no separate repo sweep.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 10:16:16 -05:00
Colby McHenry 9b68286a0a docs(extraction): add JavaScript validation row to value-refs matrix
Sweep value-refs across small/medium/large JS (express, eslint, webpack),
indexing each on vs off. Clean on all three: node count identical on/off,
precision guards held, no minified-file or shadow FPs, and the impact-radius
win reproduces (webpack `LogType` 1->89, eslint `internalSlotsMap` 1->32).

No code fix needed — the #895 shadow prune and generated-file skip generalize
to JS unchanged.

Document the one JS-specific characteristic: CommonJS `var x = require(...)`
bindings and module-level `var` state are value-ref targets (hence JS's higher
~4-5% edge share vs TS's ~0.7-1.6%). These are correct, not noise — changing
such a binding genuinely affects its readers, and impact dedups against any
overlapping `calls` edge. TS `import`s dodge this (import-kind nodes, never
const/var targets).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 09:50:52 -05:00
Colby McHenry 7f9ec84f27 docs(extraction): record value-reference-edges design + TS validation matrix
Sweep the value-refs feature across small/medium/large TypeScript repos (ky,
excalidraw, vscode), indexing each with the flag on vs off. Result: clean on all
three — node count identical on/off (edges-only), edge growth <=1.6%, precision
guards held, and the impact-radius win reproduces at every size (e.g. vscode
`LayoutStateKeys` 1->85 affected, `CORE_WEIGHT` 1->52).

No new code fix needed: the #895 shadow prune (which fixed excalidraw's 23
bundled-file false positives) generalizes — vscode showed no param-shadow or
bundle FPs in the top 200 targets, and the one apparent anomaly was a probe
artifact (a filename containing "ValueRef"), not a real edge.

Document the design, the precision guards, the validation matrix, the agent-A/B
finding (impact-API correctness is the win, not agent read-reduction), the known
limitations (param-only shadowing left unguarded by intent), and the recipe to
extend the sweep to the next language.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 01:47:25 -05:00