proffesor-for-testing__agentic-qe/docs/plans/web-tree-sitter-migration.md

Migration Plan: tree-sitter to web-tree-sitter

Executive Summary

This document outlines the migration from native tree-sitter bindings to web-tree-sitter (WASM-based) to eliminate npm install warnings caused by native compilation requirements.

Problem Statement

Users installing agentic-qe@latest encounter warnings/errors:

prebuild-install WARN install No prebuilt binaries found
gyp ERR! build error

Solution

Replace native tree-sitter with web-tree-sitter which uses WebAssembly - no native compilation required.

---

Current State Analysis

Dependencies to Replace

Current Package	Version	Replacement
tree-sitter	^0.22.4	web-tree-sitter
tree-sitter-typescript	^0.23.2	.wasm file
tree-sitter-javascript	^0.23.0	.wasm file
tree-sitter-python	^0.23.5	.wasm file
tree-sitter-go	^0.23.3	.wasm file
tree-sitter-rust	^0.24.0	.wasm file

Files Requiring Changes

Core Parser (3 files):

1. src/code-intelligence/parser/TreeSitterParser.ts - Main parser class
2. src/code-intelligence/parser/extractors/BaseExtractor.ts - Base extractor types
3. src/code-intelligence/parser/index.ts - Exports

Language Extractors (5 files): 4. src/code-intelligence/parser/extractors/TypeScriptExtractor.ts 5. src/code-intelligence/parser/extractors/JavaScriptExtractor.ts 6. src/code-intelligence/parser/extractors/PythonExtractor.ts 7. src/code-intelligence/parser/extractors/GoExtractor.ts 8. src/code-intelligence/parser/extractors/RustExtractor.ts

Tests (3+ files): 9. tests/code-intelligence/parser/TreeSitterParser.test.ts 10. tests/code-intelligence/parser/validation.test.ts 11. tests/code-intelligence/benchmarks/chunking-accuracy.test.ts

Scripts (1 file): 12. scripts/validate-code-intelligence-setup.ts

---

API Differences

Initialization

// BEFORE: tree-sitter (synchronous)
import Parser from 'tree-sitter';
import TypeScript from 'tree-sitter-typescript';

const parser = new Parser();
parser.setLanguage(TypeScript.typescript);

// AFTER: web-tree-sitter (asynchronous)
import Parser from 'web-tree-sitter';

await Parser.init();  // REQUIRED: Must init WASM runtime first
const parser = new Parser();
const TypeScript = await Parser.Language.load('/path/to/tree-sitter-typescript.wasm');
parser.setLanguage(TypeScript);

Key Differences

Aspect	tree-sitter	web-tree-sitter
Initialization	Sync	Async (Parser.init())
Language Loading	Sync import	Async Language.load()
Performance	Native (fastest)	WASM (~slower in Node.js)
Platform Support	Needs compilation	Universal
Node Types	Parser.SyntaxNode	Same API
Tree Walking	Same	Same

Type Compatibility

The SyntaxNode interface is nearly identical:

• .type - node type string
• .text - node text content
• .startIndex, .endIndex - byte positions
• .startPosition, .endPosition - row/column positions
• .children - child nodes array
• .parent - parent node

---

Implementation Phases

Phase 1: Infrastructure Setup (Low Risk)

Duration: 1-2 hours

1. Add new dependencies:

   npm install web-tree-sitter
   npm uninstall tree-sitter tree-sitter-typescript tree-sitter-javascript \
                  tree-sitter-python tree-sitter-go tree-sitter-rust
   

2. Create WASM assets directory:

   assets/
   └── wasm/
       ├── tree-sitter.wasm
       ├── tree-sitter-typescript.wasm
       ├── tree-sitter-tsx.wasm
       ├── tree-sitter-javascript.wasm
       ├── tree-sitter-python.wasm
       ├── tree-sitter-go.wasm
       └── tree-sitter-rust.wasm
   

3. Generate WASM files:

   # Install tree-sitter-cli for WASM generation
   npm install -g tree-sitter-cli
   
   # Clone and build each grammar
   git clone https://github.com/tree-sitter/tree-sitter-typescript
   cd tree-sitter-typescript
   tree-sitter build --wasm
   # Copy tree-sitter-typescript.wasm to assets/wasm/
   

   Alternative: Use pre-built WASM from @aspect-build or generate at build time.

4. Update package.json files array:

   {
     "files": [
       "dist",
       "assets/wasm",
       ...
     ]
   }
   

Phase 2: Create Async Parser Wrapper (Medium Risk)

Duration: 2-3 hours

Create a new async-compatible parser that wraps web-tree-sitter:

// src/code-intelligence/parser/WebTreeSitterParser.ts

import Parser, { Language, Tree, SyntaxNode } from 'web-tree-sitter';
import { join } from 'path';
import { Language as CodeLanguage, ParseResult } from './types.js';

export class WebTreeSitterParser {
  private static initialized = false;
  private parsers: Map<CodeLanguage, Parser> = new Map();
  private languages: Map<CodeLanguage, Language> = new Map();
  private treeCache: Map<string, { tree: Tree; content: string; language: CodeLanguage }> = new Map();

  /**
   * Initialize the WASM runtime - MUST be called before parsing
   */
  static async initialize(): Promise<void> {
    if (this.initialized) return;
    await Parser.init();
    this.initialized = true;
  }

  /**
   * Load a language grammar
   */
  async loadLanguage(lang: CodeLanguage): Promise<void> {
    if (this.languages.has(lang)) return;

    await WebTreeSitterParser.initialize();

    const wasmPath = this.getWasmPath(lang);
    const language = await Parser.Language.load(wasmPath);
    this.languages.set(lang, language);

    const parser = new Parser();
    parser.setLanguage(language);
    this.parsers.set(lang, parser);
  }

  /**
   * Get WASM file path for a language
   */
  private getWasmPath(lang: CodeLanguage): string {
    const wasmDir = join(__dirname, '../../../assets/wasm');
    const files: Record<CodeLanguage, string> = {
      typescript: 'tree-sitter-typescript.wasm',
      javascript: 'tree-sitter-javascript.wasm',
      python: 'tree-sitter-python.wasm',
      go: 'tree-sitter-go.wasm',
      rust: 'tree-sitter-rust.wasm',
    };
    return join(wasmDir, files[lang]);
  }

  /**
   * Parse file (async version)
   */
  async parseFile(
    filePath: string,
    content: string,
    language?: string
  ): Promise<ParseResult> {
    const detectedLanguage = language as CodeLanguage ?? this.detectLanguage(filePath);
    if (!detectedLanguage) {
      throw new Error(`Cannot detect language for: ${filePath}`);
    }

    await this.loadLanguage(detectedLanguage);
    const parser = this.parsers.get(detectedLanguage)!;

    const tree = parser.parse(content);

    // Cache for incremental updates
    this.treeCache.set(filePath, {
      tree,
      content,
      language: detectedLanguage,
    });

    return this.extractEntities(tree.rootNode, content, filePath, detectedLanguage);
  }

  // ... rest of implementation follows same pattern as TreeSitterParser
}

Phase 3: Update Extractors (Low Risk)

Duration: 1-2 hours

The extractors use Parser.SyntaxNode type which is compatible. Update imports:

// BEFORE
import Parser from 'tree-sitter';

// AFTER
import type { SyntaxNode } from 'web-tree-sitter';

// Update method signatures
export abstract class BaseExtractor {
  abstract extractFunctions(node: SyntaxNode): ExtractedSymbol[];
  abstract extractClasses(node: SyntaxNode): ExtractedSymbol[];
  abstract extractMethods(classNode: SyntaxNode): ExtractedSymbol[];

  protected findChildByType(node: SyntaxNode, type: string): SyntaxNode | null {
    return node.children.find(child => child.type === type) || null;
  }
  // ...
}

Phase 4: Update Consumers (Medium Risk)

Duration: 2-3 hours

Update all code that uses TreeSitterParser:

// BEFORE
const parser = new TreeSitterParser();
const result = parser.parseFile(filePath, content);

// AFTER
const parser = new WebTreeSitterParser();
await parser.loadLanguage('typescript'); // Pre-load for performance
const result = await parser.parseFile(filePath, content);

Key consumers to update:

• src/code-intelligence/service/CodeIntelligenceService.ts
• src/code-intelligence/graph/GraphBuilder.ts
• Any CLI commands using parsing

Phase 5: Update Tests (Low Risk)

Duration: 1-2 hours

Convert test setup to async:

// BEFORE
describe('TreeSitterParser', () => {
  let parser: TreeSitterParser;

  beforeEach(() => {
    parser = new TreeSitterParser();
  });

  it('parses TypeScript', () => {
    const result = parser.parseFile('test.ts', 'const x = 1;');
    expect(result.entities).toHaveLength(1);
  });
});

// AFTER
describe('WebTreeSitterParser', () => {
  let parser: WebTreeSitterParser;

  beforeAll(async () => {
    await WebTreeSitterParser.initialize();
  });

  beforeEach(() => {
    parser = new WebTreeSitterParser();
  });

  it('parses TypeScript', async () => {
    const result = await parser.parseFile('test.ts', 'const x = 1;');
    expect(result.entities).toHaveLength(1);
  });
});

Phase 6: Remove Old Dependencies (Low Risk)

Duration: 30 minutes

1. Remove from package.json:

   // Remove these
   "tree-sitter": "^0.22.4",
   "tree-sitter-go": "^0.23.3",
   "tree-sitter-javascript": "^0.23.0",
   "tree-sitter-python": "^0.23.5",
   "tree-sitter-rust": "^0.24.0",
   "tree-sitter-typescript": "^0.23.2",
   

2. Remove overrides section (no longer needed):

   // Remove entire overrides block
   "overrides": {
     "tree-sitter-go": { ... },
     ...
   }
   

3. Run npm install to verify clean install

---

WASM File Distribution Options

Option A: Bundle in Package (Recommended)

Include WASM files in the npm package:

{
  "files": ["dist", "assets/wasm"]
}

Pros: Self-contained, works offline Cons: Larger package size (~5-10MB)

Option B: Download on First Use

Fetch WASM files on first parse:

async function downloadWasm(lang: string): Promise<Buffer> {
  const url = `https://cdn.example.com/wasm/tree-sitter-${lang}.wasm`;
  const response = await fetch(url);
  return Buffer.from(await response.arrayBuffer());
}

Pros: Smaller package Cons: Requires network, first-parse latency

Option C: Optional Peer Dependency

Make code-intelligence optional:

{
  "optionalDependencies": {
    "web-tree-sitter": "^0.26.3"
  }
}

Users who need parsing install it explicitly.

Pros: Zero impact for users not using code-intelligence Cons: More complex setup

---

Performance Considerations

Benchmarks (Expected)

Operation	tree-sitter (native)	web-tree-sitter (WASM)
First parse	1x (baseline)	~1.5-2x slower
Incremental	1x (baseline)	~1.5-2x slower
Memory	1x (baseline)	~1.2x

Mitigations

1. Lazy Loading: Only load languages when needed
2. Caching: Keep parsed trees in memory
3. Pre-initialization: Call Parser.init() at startup
4. Consider oxc-parser: For TS/JS only, 2x faster than even native tree-sitter

---

Rollback Strategy

If issues arise, revert by:

1. Restore tree-sitter dependencies in package.json
2. Git revert parser changes
3. Remove assets/wasm directory
4. Republish

---

Testing Checklist

• All existing parser tests pass
• Incremental parsing works correctly
• All 5 languages parse correctly (TS, JS, Python, Go, Rust)
• npm install agentic-qe@latest shows no warnings
• Package installs on: Linux, macOS, Windows
• Package installs on: Node 18, 20, 22
• CI pipeline passes
• Code-intelligence CLI commands work

---

Timeline Estimate

Phase	Duration	Risk
Phase 1: Infrastructure	1-2 hours	Low
Phase 2: Async Parser	2-3 hours	Medium
Phase 3: Update Extractors	1-2 hours	Low
Phase 4: Update Consumers	2-3 hours	Medium
Phase 5: Update Tests	1-2 hours	Low
Phase 6: Cleanup	30 min	Low
Total	8-12 hours

---

Alternative: Hybrid Approach

For optimal performance while eliminating warnings:

1. Use web-tree-sitter as the default (no warnings)
2. Optionally use native tree-sitter if available:

   let Parser: typeof import('tree-sitter') | typeof import('web-tree-sitter');
   try {
     Parser = await import('tree-sitter');
   } catch {
     Parser = await import('web-tree-sitter');
     await Parser.init();
   }
   

This gives users native performance if compilation succeeds, with WASM fallback.

---

Decision Required

Choose implementation approach:

1. Full WASM Migration - Simplest, eliminates all native deps
2. Hybrid with Fallback - Best performance, more complex
3. Optional Code Intelligence - Make entire feature opt-in

Recommendation: Option 1 (Full WASM Migration) for simplicity and reliability.