Files
Profa cdc537dab4 feat(cloud-sync): complete Phase 3 & 4 - embeddings and periodic sync
Phase 3 - Embedding Generation:
- Add SyncEmbeddingGenerator for pattern embeddings (384-dim)
- Generate embeddings using Xenova/all-MiniLM-L6-v2 model
- Implement vector similarity search with cosine similarity
- Add CLI script for embedding generation and search
- 67 patterns embedded (45 patterns + 22 sona_patterns)

Phase 4 - Periodic Sync:
- Add CloudSyncWorker background worker
- Default 1-hour interval (configurable via AQE_SYNC_INTERVAL)
- Enable with AQE_CLOUD_SYNC_ENABLED=true
- Graceful handling when gcloud CLI unavailable

Bug Fixes:
- Fix issue #226: Dynamic require of child_process error
- Use platform-based heuristics for global npm paths
- Exclude browser E2E tests from CI workflows

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 09:07:53 +00:00

28 KiB

Cloud Sync Plan: Local AQE → Cloud ruvector-postgres

Executive Summary

Sync learning data from local SQLite databases to cloud PostgreSQL with ruvector for centralized self-learning across environments.

Problem: Data is fragmented across 6+ locations. This plan consolidates all sources.


1. Data Inventory - Complete Fragmentation Analysis

⚠️ CRITICAL: Data Fragmentation Issue

Data is scattered across multiple locations. The sync system must consolidate:

Active Data Sources (by recency)

Location Last Modified Size Records Status
v3/.agentic-qe/memory.db Jan 24 (TODAY) 1.8 MB 1,186 ✅ PRIMARY (v3 runtime)
.claude-flow/memory/store.json Jan 23 247 KB 6,113 lines ✅ ACTIVE (daemon)
.agentic-qe/memory.db Jan 22 9.5 MB 2,060 ⚠️ HISTORICAL (v2 data)
v3/.ruvector/intelligence.json Jan 7 943 KB 57,233 lines ⚠️ STALE (RL patterns)
.swarm/memory.db Jan 15 385 KB 82 ❌ LEGACY (archive)
v2/data/ruvector-patterns.db Jan 11 17 MB - ❌ LEGACY (archive)

V3 Active Memory (PRIMARY SOURCE)

Table Records Priority Description
qe_patterns 1,073 HIGH QE-specific learned patterns
sona_patterns 68 HIGH Neural backbone patterns
goap_actions 40 HIGH Planning primitives
kv_store 5 MEDIUM Queen state, reports
mincut_* varies LOW Graph analysis

Root Memory (HISTORICAL DATA)

Table Records Priority Description
memory_entries 2,060 HIGH Historical patterns
events 1,082 MEDIUM Audit trail
learning_experiences 665 HIGH RL trajectories
goap_actions 61 HIGH Planning primitives
patterns 45 HIGH Learned behaviors
goap_plans 27 MEDIUM Execution traces

Claude-Flow JSON Data

File Lines Priority Description
memory/store.json 6,113 HIGH ADR analysis, agent patterns
daemon-state.json - MEDIUM Worker stats (2,059 runs)
metrics/*.json 9 files LOW Performance metrics

V3 Intelligence (RL Q-Learning)

File Lines Priority Description
.ruvector/intelligence.json 57,233 MEDIUM Q-values, action patterns

Data includes:

  • Q-learning state-action pairs with visit counts
  • File access memories with embeddings
  • Success/failure action patterns

Data Namespaces in Root memory_entries

agents, aqe, aqe-workflows, aqe/coverage, aqe/policies,
aqe/qx, aqe/test, baselines, coordination, default,
edge/poc, events, goap, learning, llm-independence,
ooda_cycles, optimizations, qx-analysis, ...

2. Cloud Schema Design

PostgreSQL Schema for ruvector-postgres

-- Enable extensions
CREATE EXTENSION IF NOT EXISTS ruvector;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";

-- Schema for AQE learning data
CREATE SCHEMA IF NOT EXISTS aqe;

-- Memory entries (key-value with namespaces)
CREATE TABLE aqe.memory_entries (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    key TEXT NOT NULL,
    partition TEXT NOT NULL DEFAULT 'default',
    value JSONB NOT NULL,
    metadata JSONB,
    embedding ruvector(384),  -- For semantic search (ruvector)
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),
    expires_at TIMESTAMPTZ,
    source_env TEXT NOT NULL,  -- 'devpod', 'laptop', 'ci'
    sync_version BIGINT DEFAULT 0,
    UNIQUE(key, partition, source_env)
);

-- Learning experiences (RL trajectories)
CREATE TABLE aqe.learning_experiences (
    id SERIAL PRIMARY KEY,
    agent_id TEXT NOT NULL,
    task_id TEXT,
    task_type TEXT NOT NULL,
    state JSONB NOT NULL,
    action JSONB NOT NULL,
    reward REAL NOT NULL,
    next_state JSONB NOT NULL,
    episode_id TEXT,
    metadata JSONB,
    source_env TEXT NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- GOAP actions (planning primitives)
CREATE TABLE aqe.goap_actions (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    description TEXT,
    agent_type TEXT NOT NULL,
    preconditions JSONB NOT NULL,
    effects JSONB NOT NULL,
    cost REAL DEFAULT 1.0,
    duration_estimate INTEGER,
    success_rate REAL DEFAULT 1.0,
    execution_count INTEGER DEFAULT 0,
    category TEXT,
    source_env TEXT NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- GOAP plans (execution traces)
CREATE TABLE aqe.goap_plans (
    id TEXT PRIMARY KEY,
    goal_id TEXT,
    sequence JSONB NOT NULL,
    initial_state JSONB,
    goal_state JSONB,
    action_sequence JSONB,
    total_cost REAL,
    estimated_duration INTEGER,
    actual_duration INTEGER,
    status TEXT DEFAULT 'pending',
    success BOOLEAN,
    failure_reason TEXT,
    execution_trace JSONB,
    source_env TEXT NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    completed_at TIMESTAMPTZ
);

-- Patterns (learned behaviors)
CREATE TABLE aqe.patterns (
    id TEXT PRIMARY KEY,
    pattern TEXT NOT NULL,
    confidence REAL NOT NULL,
    usage_count INTEGER DEFAULT 0,
    metadata JSONB,
    domain TEXT DEFAULT 'general',
    success_rate REAL DEFAULT 1.0,
    embedding ruvector(384),
    source_env TEXT NOT NULL,
    expires_at TIMESTAMPTZ,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Events (audit log)
CREATE TABLE aqe.events (
    id TEXT PRIMARY KEY,
    type TEXT NOT NULL,
    payload JSONB NOT NULL,
    source TEXT NOT NULL,
    source_env TEXT NOT NULL,
    timestamp TIMESTAMPTZ NOT NULL,
    expires_at TIMESTAMPTZ
);

-- Sync metadata
CREATE TABLE aqe.sync_state (
    source_env TEXT PRIMARY KEY,
    last_sync_at TIMESTAMPTZ,
    last_sync_version BIGINT DEFAULT 0,
    tables_synced JSONB,
    status TEXT DEFAULT 'idle'
);

-- Claude-Flow memory store (JSON → PostgreSQL)
CREATE TABLE aqe.claude_flow_memory (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    key TEXT NOT NULL,
    value JSONB NOT NULL,
    category TEXT,  -- 'adr-analysis', 'agent-patterns', etc.
    embedding ruvector(384),
    source_env TEXT NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    UNIQUE(key, source_env)
);

-- Claude-Flow daemon worker stats
CREATE TABLE aqe.claude_flow_workers (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    worker_type TEXT NOT NULL,  -- 'map', 'audit', 'optimize', etc.
    run_count INTEGER DEFAULT 0,
    success_count INTEGER DEFAULT 0,
    failure_count INTEGER DEFAULT 0,
    avg_duration_ms REAL,
    last_run TIMESTAMPTZ,
    source_env TEXT NOT NULL,
    captured_at TIMESTAMPTZ DEFAULT NOW(),
    UNIQUE(worker_type, source_env)
);

-- Q-Learning patterns from intelligence.json
CREATE TABLE aqe.qlearning_patterns (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    state TEXT NOT NULL,
    action TEXT NOT NULL,
    q_value REAL NOT NULL,
    visits INTEGER DEFAULT 0,
    last_update TIMESTAMPTZ,
    source_env TEXT NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    UNIQUE(state, action, source_env)
);

-- Memory embeddings from intelligence.json
CREATE TABLE aqe.intelligence_memories (
    id TEXT PRIMARY KEY,
    memory_type TEXT NOT NULL,  -- 'file_access', etc.
    content TEXT,
    embedding vector(64),  -- intelligence.json uses 64-dim
    metadata JSONB,
    source_env TEXT NOT NULL,
    timestamp TIMESTAMPTZ,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- SONA neural patterns (from v3 memory)
CREATE TABLE aqe.sona_patterns (
    id TEXT PRIMARY KEY,
    type TEXT NOT NULL,
    domain TEXT,
    state_embedding ruvector(384),
    action_embedding ruvector(384),
    action_type TEXT,
    action_value JSONB,
    outcome_reward REAL,
    outcome_success BOOLEAN,
    outcome_quality REAL,
    confidence REAL,
    usage_count INTEGER DEFAULT 0,
    success_count INTEGER DEFAULT 0,
    source_env TEXT NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- QE-specific patterns (from v3 memory)
CREATE TABLE aqe.qe_patterns (
    id TEXT PRIMARY KEY,
    pattern_type TEXT NOT NULL,
    qe_domain TEXT,  -- 'test-generation', 'coverage-analysis', etc.
    content JSONB NOT NULL,
    embedding ruvector(384),
    confidence REAL,
    usage_count INTEGER DEFAULT 0,
    success_rate REAL DEFAULT 1.0,
    metadata JSONB,
    source_env TEXT NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Indexes for vector similarity search
CREATE INDEX idx_memory_embedding ON aqe.memory_entries
    USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
CREATE INDEX idx_patterns_embedding ON aqe.patterns
    USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);

-- Indexes for queries
CREATE INDEX idx_memory_partition ON aqe.memory_entries(partition);
CREATE INDEX idx_memory_source ON aqe.memory_entries(source_env);
CREATE INDEX idx_learning_agent ON aqe.learning_experiences(agent_id);
CREATE INDEX idx_events_type ON aqe.events(type);
CREATE INDEX idx_patterns_domain ON aqe.patterns(domain);

3. Sync Architecture

Data Flow (Consolidated from 6+ Sources)

┌─────────────────────────────────────────────────────────────────────────┐
│                         Local Environment                                │
│                                                                         │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │                     PRIMARY (V3 Active)                          │   │
│  │  ┌─────────────────┐  ┌─────────────────┐                       │   │
│  │  │ v3/.agentic-qe/ │  │ v3/.ruvector/   │                       │   │
│  │  │   memory.db     │  │ intelligence.json│                       │   │
│  │  │  (1,186 records)│  │ (57K Q-values)  │                       │   │
│  │  └────────┬────────┘  └────────┬────────┘                       │   │
│  └───────────┼────────────────────┼─────────────────────────────────┘   │
│              │                    │                                     │
│  ┌───────────┼────────────────────┼─────────────────────────────────┐   │
│  │           │    CLAUDE-FLOW (Active)                              │   │
│  │  ┌────────┴────────┐  ┌────────┴────────┐  ┌──────────────┐     │   │
│  │  │ .claude-flow/   │  │ daemon-state    │  │   metrics/   │     │   │
│  │  │ memory/store.json│ │   .json         │  │  (9 files)   │     │   │
│  │  │ (6,113 lines)   │  │ (2,059 runs)    │  └──────────────┘     │   │
│  │  └────────┬────────┘  └────────┬────────┘                       │   │
│  └───────────┼────────────────────┼─────────────────────────────────┘   │
│              │                    │                                     │
│  ┌───────────┼────────────────────┼─────────────────────────────────┐   │
│  │           │     HISTORICAL (Root V2)                             │   │
│  │  ┌────────┴────────┐  ┌────────┴────────┐  ┌──────────────┐     │   │
│  │  │ .agentic-qe/    │  │ .swarm/         │  │ v2/data/     │     │   │
│  │  │   memory.db     │  │   memory.db     │  │ patterns.db  │     │   │
│  │  │ (2,060 records) │  │  (82 records)   │  │  (17 MB)     │     │   │
│  │  └────────┬────────┘  └────────┬────────┘  └──────┬───────┘     │   │
│  └───────────┼────────────────────┼──────────────────┼──────────────┘   │
│              │                    │                  │                  │
│              └────────────────────┼──────────────────┘                  │
│                                   │                                     │
│                          ┌────────▼────────┐                            │
│                          │   Sync Agent    │                            │
│                          │  (TypeScript)   │                            │
│                          │  Consolidates   │                            │
│                          │   All Sources   │                            │
│                          └────────┬────────┘                            │
└───────────────────────────────────┼─────────────────────────────────────┘
                                    │
                           ┌────────▼────────┐
                           │   IAP Tunnel    │
                           │  (Encrypted)    │
                           └────────┬────────┘
                                    │
┌───────────────────────────────────▼─────────────────────────────────────┐
│                           Google Cloud                                   │
│                                                                         │
│  ┌───────────────────────────────────────────────────────────────┐     │
│  │                    ruvector-postgres                           │     │
│  │  ┌─────────────────────────────────────────────────────────┐  │     │
│  │  │                     aqe schema                           │  │     │
│  │  │  ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐│  │     │
│  │  │  │memory_entries│ │qe_patterns │ │claude_flow_memory   ││  │     │
│  │  │  │patterns     │ │sona_patterns│ │qlearning_patterns   ││  │     │
│  │  │  │goap_*       │ │events       │ │intelligence_memories││  │     │
│  │  │  └─────────────┘ └─────────────┘ └─────────────────────┘│  │     │
│  │  └─────────────────────────────────────────────────────────┘  │     │
│  │  ┌─────────────┐                                              │     │
│  │  │  ruvector   │  ← Vector similarity search                  │     │
│  │  │ (indexes)   │                                              │     │
│  │  └─────────────┘                                              │     │
│  └───────────────────────────────────────────────────────────────┘     │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

Sync Modes

Mode Description Use Case
Full Sync all records Initial migration
Incremental Only changed records Regular sync
Append Only new records Events/logs
Bidirectional Merge from multiple sources Multi-env learning

4. Implementation Plan

Phase 1: Schema Migration (Day 1) ✅ COMPLETED

  • Create PostgreSQL schema in cloud DB (v3/src/sync/schema/cloud-schema.sql)
  • Set up ruvector HNSW indexes (using ruvector_cosine_ops)
  • Create sync_state tracking table
  • Apply migration for additional columns (migration-001.sql)

Phase 2: Sync Agent (Days 2-3) ✅ COMPLETED

  • Create TypeScript sync agent (v3/src/sync/sync-agent.ts)
  • Implement IAP tunnel connection (v3/src/sync/cloud/tunnel-manager.ts)
  • Add SQLite → PostgreSQL data conversion (v3/src/sync/readers/)
  • Handle JSON/JSONB transformations (auto-wrap non-JSON strings)
  • Handle timestamp conversions (Unix ms → ISO 8601)
  • Add conflict resolution logic (ON CONFLICT DO UPDATE)

Phase 3: Initial Migration (Day 4) ✅ COMPLETED

  • Full sync of all historical data (5,062 records total)
  • Verify data integrity (all tables verified)
  • Generate embeddings for patterns (2026-02-03)
  • Test vector similarity search (2026-02-03)

Embedding Generation Results (2026-02-03):

  • 45 patterns table embeddings generated (384-dim)
  • 22 sona_patterns table embeddings generated
  • Total: 67 embeddings
  • Avg generation time: 10.65ms per pattern
  • Model: Xenova/all-MiniLM-L6-v2
  • Vector similarity search tested and working

Phase 4: Incremental Sync (Day 5) ✅ COMPLETED

  • Implement change detection (incremental mode)
  • Set up periodic sync via background worker (2026-02-03)
  • Add sync status monitoring (aqe sync status)
  • Handle network failures gracefully (port connectivity check, retries)

Periodic Sync Implementation (2026-02-03):

  • Created CloudSyncWorker background worker (v3/src/workers/workers/cloud-sync.ts)
  • Default interval: 1 hour (configurable via AQE_SYNC_INTERVAL)
  • Enable with AQE_CLOUD_SYNC_ENABLED=true
  • Worker checks gcloud availability before syncing
  • Graceful handling when gcloud CLI not installed

Phase 5: Bidirectional Learning (Day 6+)

  • Enable pattern sharing across environments
  • Implement consensus for conflicting patterns
  • Add cross-environment success rate aggregation

Sync Results (2026-01-24)

Source Records Status
v3-qe-patterns 1,073 ✅
v3-sona-patterns 34 ✅
v3-goap-actions 40 ✅
claude-flow-memory 2 ✅
root-memory-entries 2,060 ✅
root-learning-experiences 665 ✅
root-goap-actions 61 ✅
root-patterns 45 ✅
root-events 1,082 ✅
Total 5,062 ✅

5. Sync Agent Design

Configuration

interface SyncConfig {
  // All local data sources (fragmented)
  local: {
    // PRIMARY - V3 active runtime
    v3MemoryDb: string;           // v3/.agentic-qe/memory.db

    // HISTORICAL - Root v2 data
    rootMemoryDb: string;         // .agentic-qe/memory.db
    rootCacheDb: string;          // .agentic-qe/ruvector-cache.db

    // CLAUDE-FLOW - JSON stores
    claudeFlowMemory: string;     // .claude-flow/memory/store.json
    claudeFlowDaemon: string;     // .claude-flow/daemon-state.json
    claudeFlowMetrics: string;    // .claude-flow/metrics/

    // Q-LEARNING - Intelligence patterns
    intelligenceJson: string;     // v3/.ruvector/intelligence.json

    // LEGACY (archive only)
    swarmMemoryDb: string;        // .swarm/memory.db
    v2PatternsDb: string;         // v2/data/ruvector-patterns.db
  };

  cloud: {
    project: string;              // <your-gcp-project-id>
    zone: string;                 // us-central1-a
    instance: string;             // ruvector-postgres
    database: string;             // aqe_learning
    user: string;                 // ruvector
    tunnelPort: number;           // 15432
  };

  sync: {
    mode: 'full' | 'incremental' | 'bidirectional';
    interval: string;             // '5m', '1h', etc.
    batchSize: number;            // 1000

    // Source priority (higher = sync first)
    sourcePriority: {
      v3Memory: 1,                // PRIMARY
      claudeFlowMemory: 2,        // ACTIVE
      rootMemory: 3,              // HISTORICAL
      intelligenceJson: 4,        // RL DATA
      legacy: 5                   // ARCHIVE
    };

    // Tables/sources to sync
    sources: SyncSource[];
  };

  environment: string;            // 'devpod', 'laptop', 'ci'
}

interface SyncSource {
  name: string;
  type: 'sqlite' | 'json';
  path: string;
  targetTable: string;
  priority: 'high' | 'medium' | 'low';
  mode: 'incremental' | 'full' | 'append';
  transform?: (data: any) => any;
}

Core Sync Operations

// Sync operations
async function syncTable(table: string, mode: SyncMode): Promise<SyncResult>;
async function getChangedRecords(table: string, since: Date): Promise<Record[]>;
async function upsertToCloud(table: string, records: Record[]): Promise<void>;
async function resolveConflicts(local: Record, cloud: Record): Record;

// Embedding generation
async function generateEmbedding(text: string): Promise<number[]>;
async function batchGenerateEmbeddings(texts: string[]): Promise<number[][]>;

// Tunnel management
async function startTunnel(): Promise<TunnelConnection>;
async function withTunnel<T>(fn: (conn: Connection) => Promise<T>): Promise<T>;

6. Data Transformation Rules

SQLite → PostgreSQL Type Mapping

SQLite Type PostgreSQL Type Notes
TEXT (JSON) JSONB Parse and validate
INTEGER (timestamp) TIMESTAMPTZ Unix ms → ISO
REAL REAL Direct
BLOB BYTEA Direct
TEXT (embedding) vector(384) Parse array

Conflict Resolution

Scenario Resolution
Same key, different values Higher confidence wins
Same pattern, different success_rate Weighted average by usage_count
Same event ID Skip (events are immutable)
Newer local vs older cloud Local wins

7. Security Considerations

  • IAP Tunnel: All traffic encrypted, requires Google auth
  • No public ports: Database not exposed to internet
  • Environment isolation: source_env column prevents cross-contamination
  • Credential storage: Use Secret Manager for production
  • Audit logging: All syncs logged to events table

8. Monitoring & Alerts

Metrics to Track

  • Sync duration per table
  • Records synced per run
  • Conflict rate
  • Failed sync attempts
  • Cloud DB storage usage

Alert Thresholds

Metric Warning Critical
Sync duration > 5 min > 15 min
Failure rate > 5% > 20%
Days since sync > 1 day > 3 days

9. CLI Commands ✅ IMPLEMENTED

# In v3/ directory (or use npm -w v3)
cd v3

# Initial setup
npm run sync:cloud:init        # Generate cloud schema SQL
npm run sync:cloud:config      # Show sync configuration

# Regular sync
npm run sync:cloud             # Incremental sync (default)
npm run sync:cloud:full        # Force full sync

# Status & verification
npm run sync:cloud:status      # Check sync state
npm run sync:cloud:verify      # Verify data integrity

# Or use the CLI directly:
npx tsx src/cli/index.ts sync           # Incremental sync
npx tsx src/cli/index.ts sync --full    # Full sync
npx tsx src/cli/index.ts sync status    # Check status
npx tsx src/cli/index.ts sync verify    # Verify integrity
npx tsx src/cli/index.ts sync config    # Show config

Environment Variables

# Required
export PGPASSWORD=aqe_secure_2024

# Optional (defaults shown)
export GCP_PROJECT=ferrous-griffin-480616-s9
export GCP_ZONE=us-central1-a
export GCP_INSTANCE=ruvector-postgres
export GCP_DATABASE=aqe_learning
export GCP_USER=ruvector
export GCP_TUNNEL_PORT=15432
export AQE_ENV=devpod

10. Next Steps

  1. Approve this plan ✅ DONE - Review and confirm approach
  2. Create cloud schema ✅ DONE - Schema applied with ruvector
  3. Build sync agent ✅ DONE - TypeScript implementation complete
  4. Initial migration ✅ DONE - 5,062 records synced
  5. Set up automation ✅ DONE - CloudSyncWorker (1-hour interval)
  6. Generate embeddings ✅ DONE - 67 patterns with 384-dim vectors
  7. Enable bidirectional sync - Multi-environment learning (TODO)

12. CLI Commands for Embeddings

# Generate embeddings for all patterns (skip existing)
npm run sync:embeddings

# Regenerate all embeddings (force)
npm run sync:embeddings:force

# Search for similar patterns
npm run sync:embeddings:search "your query here"

# Example: Find patterns related to test coverage
npm run sync:embeddings:search "test coverage gap detection"

13. Technical Details

Embedding Generator

Location: v3/src/sync/embeddings/sync-embedding-generator.ts

Features:

  • Uses @xenova/transformers with all-MiniLM-L6-v2 model
  • Generates 384-dimensional embeddings
  • ONNX acceleration for fast inference (<15ms per pattern)
  • Caching support for repeated embeddings
  • Batch processing with progress reporting
  • Vector similarity search using cosine similarity

API:

import { createSyncEmbeddingGenerator } from '@agentic-qe/v3/sync';

const generator = createSyncEmbeddingGenerator();

// Generate embeddings for all patterns
const stats = await generator.generateForAllPatterns('patterns', {
  force: false,      // Skip existing embeddings
  batchSize: 50,     // Process in batches
  verbose: true      // Show progress
});

// Search for similar patterns
const results = await generator.findSimilarPatterns('test coverage', {
  limit: 10,
  threshold: 0.3
});

11. Cloud Infrastructure

GCE VM Setup

The cloud database runs on a GCE VM with the ruvector-postgres Docker container:

# VM: ruvector-postgres
# Zone: us-central1-a
# Project: ferrous-griffin-480616-s9

# Docker container running on VM
docker run -d \
  --name ruvector-db \
  -e POSTGRES_USER=ruvector \
  -e POSTGRES_PASSWORD=aqe_secure_2024 \
  -e POSTGRES_DB=aqe_learning \
  -p 5432:5432 \
  ruvnet/ruvector-postgres:latest

# Access via IAP tunnel (no public IP needed)
gcloud compute start-iap-tunnel ruvector-postgres 5432 \
  --local-host-port=localhost:15432 \
  --zone=us-central1-a \
  --project=ferrous-griffin-480616-s9

Security

  • No public IP on the VM
  • Access only through IAP tunnel with Google authentication
  • Database credentials stored in environment variables