proffesor-for-testing__agentic-qe/docs/decisions/embedding-consolidation.md

Embedding Generation Consolidation (Issue #52)

Date: 2025-11-17 Issue: GitHub Issue #52 - Eliminate Duplicate Embedding Generation Status: ✅ Completed

Problem

Found 4 different implementations of the same hash-based embedding function across the codebase:

1. /src/utils/EmbeddingGenerator.ts - Simple utility function (384 dimensions)
2. /src/core/embeddings/EmbeddingGenerator.ts - Complex class with ML support
3. /src/agents/BaseAgent.ts - Using utility ✓
4. /src/agents/TestExecutorAgent.ts - Using utility ✓
5. /src/core/neural/NeuralTrainer.ts - Duplicate simpleHashEmbedding() method

Solution

Single Source of Truth: /src/utils/EmbeddingGenerator.ts

Why This Choice?

1. Already In Use: BaseAgent and TestExecutorAgent already import from this utility
2. Simple & Clear: Straightforward implementation without unnecessary complexity
3. Consistent Output: 384 dimensions (standard for sentence transformers)
4. Well-Documented: Clear production replacement guidance

Changes Made

NeuralTrainer.ts Refactoring

Removed (34 lines):

private simpleHashEmbedding(text: string): number[] {
  const dimensions = 384;
  const embedding = new Array(dimensions).fill(0);

  for (let i = 0; i < text.length; i++) {
    const charCode = text.charCodeAt(i);
    const index = (charCode * (i + 1)) % dimensions;
    embedding[index] += Math.sin(charCode * 0.1) * 0.1;
  }

  // Normalize
  const magnitude = Math.sqrt(embedding.reduce((sum, val) => sum + val * val, 0));
  if (magnitude > 0) {
    for (let i = 0; i < dimensions; i++) {
      embedding[i] /= magnitude;
    }
  }

  return embedding;
}

Added:

import { generateEmbedding } from '../../utils/EmbeddingGenerator.js';

Refactored Methods:

1. generateStateEmbedding() - Now uses generateEmbedding(stateStr)
2. generateExperienceEmbedding() - Now uses generateEmbedding(expStr)

Benefits

Code Quality

• -34 lines: Removed duplicate code
• Single Source of Truth: One place to update embedding logic
• Consistency: All agents use identical algorithm

Maintainability

• Easier Updates: Change embedding algorithm in one place
• Testing: Single function to test comprehensively
• Production Migration: Replace utility once when upgrading to real embeddings

Backward Compatibility

• ✅ 100% Compatible: Same 384-dimension output
• ✅ Same Algorithm: Identical hash-based implementation
• ✅ All Tests Pass: No breaking changes

Verification

TypeScript Compilation

npm run typecheck  # ✅ No new errors

Runtime Test

const { generateEmbedding } = require('./dist/utils/EmbeddingGenerator.js');
const embedding = generateEmbedding('test neural pattern');

// Results:
// - Dimensions: 384 ✅
// - Normalized: true ✅
// - Output: [-0.0173, 0.0000, 0.0000, ...] ✅

Future Work

The consolidated utility (/src/utils/EmbeddingGenerator.ts) includes guidance for production:

/**
 * In production, replace with:
 * - OpenAI embeddings API (text-embedding-ada-002)
 * - Cohere embeddings
 * - Local transformer models (sentence-transformers)
 * - Custom trained embeddings
 */

When upgrading, only this single file needs modification. All consumers (BaseAgent, TestExecutorAgent, NeuralTrainer) will automatically use the new implementation.

Files Modified

1. /src/core/neural/NeuralTrainer.ts
   • Added import for generateEmbedding
   • Removed simpleHashEmbedding() method
   • Updated generateStateEmbedding()
   • Updated generateExperienceEmbedding()

Architecture Note

The /src/core/embeddings/EmbeddingGenerator.ts class remains unchanged. It provides advanced features (ML models, caching, batch processing) and is used for different purposes. The simple utility is for basic embedding needs.

Coordination

Task tracked through:

• Memory namespace: aqe/swarm/issue52/embedding-consolidation
• Pre-task hook: ✓ Executed
• Post-task hook: ✓ Executed (database unavailable, expected)

---

Completed by: Claude Code Verified: Embedding generation working correctly (384D, normalized)