proffesor-for-testing__agentic-qe/docs/design/binary-cache-architecture.md

Binary Metadata Cache Architecture

Version: 1.0.0 Status: Design Phase (A1.1) Author: System Architecture Designer Date: 2025-12-12

Executive Summary

This document defines a binary caching system for pattern metadata to achieve 10x faster test discovery by reducing pattern load time from 32ms (current SQLite baseline) to <5ms (target).

Current State:

• Pattern bank reads from SQLite on every query: 32ms p95 latency
• Test discovery time: 500ms average
• No caching layer between SQLite and application

Target State:

• Pattern load time: <5ms (6.4x improvement)
• Test discovery time: 50ms (10x improvement)
• Binary cache with graceful SQLite fallback

---

1. Format Selection: MessagePack vs FlatBuffers

1.1 Analysis Criteria

Criterion	Weight	MessagePack	FlatBuffers
Serialization Speed	25%	8/10	9/10
Deserialization Speed	30%	7/10	10/10
Size Efficiency	15%	9/10	8/10
Schema Evolution	15%	6/10	9/10
Float32Array Support	10%	7/10	10/10
TypeScript Integration	5%	9/10	7/10
Weighted Score	-	7.45/10	8.85/10

1.2 Decision: FlatBuffers (Selected)

Rationale:

1. Zero-Copy Access (Critical for <5ms target)

   • FlatBuffers allows direct memory mapping without deserialization
   • No parsing overhead for reading frequently accessed fields
   • Essential for sub-5ms latency with large pattern banks (1000+ patterns)

2. Native Float32Array Support

   • Vector embeddings stored as binary buffers
   • No conversion overhead from JSON arrays
   • 4-8x faster embedding access than MessagePack

3. Schema Versioning

   • Built-in schema evolution with field deprecation
   • Forward/backward compatibility for cache migration
   • Critical for production deployments with rolling updates

4. Benchmark Justification (Estimated):

   Pattern Load Benchmark (1000 patterns):
   - SQLite (current):        32ms
   - MessagePack:            12-15ms  (2.1-2.7x improvement)
   - FlatBuffers:             4-6ms   (5.3-8x improvement)
   
   Embedding Access (768-dim vector):
   - SQLite + JSON parse:    2.5ms
   - MessagePack:            1.2ms
   - FlatBuffers (zero-copy): 0.15ms  (16x improvement)
   

5. Trade-offs:

   • Con: Requires schema compilation (.fbs → .ts)
   • Con: More complex setup than MessagePack
   • Pro: Future-proof for 10K+ pattern banks
   • Pro: Lower memory footprint with zero-copy

1.3 Alternative Considered: MessagePack

MessagePack was rejected due to:

• Parsing overhead for large pattern banks
• No zero-copy access for embeddings
• Limited schema evolution capabilities
• 2-3ms slower than FlatBuffers for target use case

Use Case Suitability:

• MessagePack: Better for <100 patterns, rapid prototyping
• FlatBuffers: Better for 1000+ patterns, production systems

---

2. Cache Data Structure Design

2.1 FlatBuffers Schema (cache.fbs)

namespace AgenticQE.Cache;

// Root table: Binary cache container
table BinaryCache {
  version: uint32;              // Cache format version (1)
  timestamp: uint64;            // Generation timestamp (Unix ms)
  checksum: string;             // SHA-256 checksum for validation
  patterns: [PatternEntry];     // Pattern metadata array
  agent_configs: [AgentConfig]; // Agent configuration cache
  indexes: IndexData;           // Pre-built search indexes
}

// Pattern metadata entry
table PatternEntry {
  id: string (key);             // Pattern unique ID
  type: string;                 // Pattern type (e.g., "unit-test")
  domain: string;               // Domain (e.g., "api")
  framework: string;            // Test framework (e.g., "jest")
  embedding: [float];           // 768-dim vector (Float32Array)
  content: string;              // Pattern content/template
  metadata: PatternMetadata;    // Extended metadata
}

// Pattern metadata
table PatternMetadata {
  coverage: float;              // Code coverage score (0-1)
  flakiness_score: float;       // Flakiness probability (0-1)
  verdict: Verdict;             // Test verdict enum
  created_at: uint64;           // Creation timestamp
  last_used: uint64;            // Last usage timestamp
  usage_count: uint32;          // Usage counter
  success_count: uint32;        // Success counter
}

// Test verdict enum
enum Verdict: byte {
  Success = 0,
  Failure = 1,
  Flaky = 2,
  Unknown = 3
}

// Agent configuration
table AgentConfig {
  agent_id: string (key);       // Agent unique ID
  type: string;                 // Agent type (e.g., "test-generator")
  config_json: string;          // JSON-serialized config
  version: string;              // Config version
  updated_at: uint64;           // Last update timestamp
}

// Pre-built indexes for fast lookup
table IndexData {
  domain_index: [DomainIndex];  // Domain → pattern ID mapping
  type_index: [TypeIndex];      // Type → pattern ID mapping
  framework_index: [FrameworkIndex]; // Framework → pattern ID mapping
}

table DomainIndex {
  domain: string (key);
  pattern_ids: [string];
}

table TypeIndex {
  type: string (key);
  pattern_ids: [string];
}

table FrameworkIndex {
  framework: string (key);
  pattern_ids: [string];
}

root_type BinaryCache;

2.2 Memory Layout Optimization

Design Principles:

1. Struct-of-Arrays (SoA): Group similar data types for cache-friendly access
2. Embedding Alignment: 16-byte alignment for SIMD operations
3. String Interning: Deduplicate domain/type/framework strings
4. Index Pre-computation: Build domain/type/framework lookup tables at cache generation

Memory Estimates (1000 patterns):

Component                    Size per Entry    Total (1000 entries)
─────────────────────────────────────────────────────────────────
Pattern ID (string)          32 bytes         32 KB
Embedding (768-dim float32)  3072 bytes       3.07 MB
Metadata                     64 bytes         64 KB
Content (avg 500 chars)      500 bytes        500 KB
Indexes                      -                50 KB (estimated)
FlatBuffers overhead         -                100 KB (estimated)
─────────────────────────────────────────────────────────────────
Total                                         ~3.9 MB

Cache Size Targets:

• 1000 patterns: ~4 MB
• 10K patterns: ~40 MB
• 100K patterns: ~400 MB (future scaling)

---

3. Versioning & Checksum Validation

3.1 Version Strategy

Semantic Versioning for Cache Format:

interface CacheVersion {
  major: number;  // Breaking schema changes (e.g., field removal)
  minor: number;  // Backward-compatible additions (new optional fields)
  patch: number;  // Non-breaking optimizations
}

Current Version: 1.0.0

Version Compatibility Matrix:

Cache Version	Code Version	Action
1.x.x	1.y.y	Compatible (load cache)
2.x.x	1.y.y	Incompatible (rebuild cache)
1.x.x	2.y.y	Deprecated (rebuild cache)

Version Encoding:

• Stored in BinaryCache.version as uint32
• Format: (major << 16) | (minor << 8) | patch
• Example: v1.2.3 → 0x00010203 (66051)

3.2 Checksum Validation

Two-Level Validation:

1. File-Level Checksum (SHA-256):

   function computeCacheChecksum(buffer: Uint8Array): string {
     const hash = crypto.subtle.digest('SHA-256', buffer);
     return Buffer.from(hash).toString('hex');
   }
   

2. Entry-Level Checksums (Optional):

   • Per-pattern CRC32 for partial validation
   • Stored in PatternMetadata (future extension)

Validation Flow:

1. Read cache file
2. Extract stored checksum from BinaryCache.checksum
3. Compute actual checksum (exclude checksum field)
4. Compare checksums
   ├─ Match: Load cache
   └─ Mismatch: Log error → Fallback to SQLite

Performance Impact:

• SHA-256 computation: ~1ms for 4MB cache
• Total validation overhead: <2ms (acceptable for 5ms target)

---

4. Invalidation Triggers & TTL Strategy

4.1 Invalidation Triggers

Event-Based Invalidation:

Trigger Event	Action	Priority
Pattern stored to SQLite	Mark cache stale	High
Pattern deleted	Mark cache stale	High
Agent config updated	Mark cache stale	Medium
SQLite schema migration	Force rebuild	Critical
Manual aqe optimize	Force rebuild	Low

Implementation:

interface CacheInvalidation {
  trigger: 'pattern_stored' | 'pattern_deleted' | 'config_updated' | 'schema_migration' | 'manual';
  timestamp: number;
  requiresRebuild: boolean;
}

class CacheInvalidator {
  private staleTimestamp: number = 0;

  markStale(trigger: CacheInvalidation): void {
    this.staleTimestamp = Date.now();
    if (trigger.requiresRebuild) {
      this.scheduleCacheRebuild();
    }
  }

  isCacheValid(cacheTimestamp: number): boolean {
    return cacheTimestamp > this.staleTimestamp;
  }
}

4.2 TTL Strategy

Time-Based Expiration:

interface TTLConfig {
  maxAge: number;           // 3600000ms (1 hour default)
  checkInterval: number;    // 300000ms (5 min)
  backgroundRebuild: boolean; // true (rebuild in background)
}

TTL Decision Matrix:

Scenario	TTL	Rationale
Development (CI/local)	10 minutes	Frequent pattern changes
Staging	1 hour	Moderate stability
Production	24 hours	High stability, background rebuild

Background Rebuild:

1. When cache age exceeds 80% of TTL:
   • Trigger background rebuild
   • Continue serving stale cache
   • Hot-swap cache after rebuild completes
2. Prevents cold start latency spikes

TTL Validation:

function isCacheFresh(cacheTimestamp: number, ttl: number): boolean {
  const age = Date.now() - cacheTimestamp;
  return age < ttl;
}

function shouldBackgroundRebuild(cacheTimestamp: number, ttl: number): boolean {
  const age = Date.now() - cacheTimestamp;
  return age > (ttl * 0.8); // 80% threshold
}

---

5. Graceful Fallback to SQLite

5.1 Fallback Triggers

Automatic Fallback Conditions:

1. Cache Corruption:

   • Checksum validation failure
   • Invalid FlatBuffers structure
   • Version incompatibility

2. Missing Cache File:

   • First initialization
   • Manual cache deletion

3. I/O Errors:

   • Permission denied
   • Disk full
   • File locked by another process

4. Performance Degradation:

   • Cache load time > 10ms (2x target)
   • Indicates potential corruption or disk issues

5.2 Fallback Implementation

Multi-Tier Architecture:

class PatternLoader {
  private cache: BinaryCache | null = null;
  private sqliteAdapter: SQLiteAdapter;

  async loadPattern(id: string): Promise<TestPattern | null> {
    // Tier 1: Binary cache (target: <5ms)
    if (this.cache && this.isCacheValid()) {
      try {
        return this.loadFromCache(id); // <5ms
      } catch (error) {
        this.logCacheFallback('cache_error', error);
      }
    }

    // Tier 2: SQLite fallback (baseline: 32ms)
    try {
      const pattern = await this.sqliteAdapter.getPattern(id);

      // Trigger background cache rebuild
      if (!this.cache) {
        this.scheduleCacheRebuild();
      }

      return pattern;
    } catch (error) {
      this.logFatalError('sqlite_error', error);
      return null;
    }
  }

  private logCacheFallback(reason: string, error: Error): void {
    console.warn(`[BinaryCache] Fallback to SQLite: ${reason}`, {
      error: error.message,
      timestamp: Date.now()
    });
  }
}

Fallback Decision Tree:

Pattern Load Request
│
├─ Cache available?
│  ├─ Yes
│  │  ├─ Cache valid?
│  │  │  ├─ Yes → Load from cache (<5ms)
│  │  │  └─ No → Fallback to SQLite (32ms) + Schedule rebuild
│  │  └─ Cache corrupt?
│  │     └─ Fallback to SQLite (32ms) + Log error
│  └─ No → Fallback to SQLite (32ms) + Build cache

5.3 Monitoring & Alerting

Metrics to Track:

interface CacheMetrics {
  cacheHitRate: number;        // Target: >95%
  avgCacheLoadTime: number;    // Target: <5ms
  avgSQLiteFallbackTime: number; // Baseline: 32ms
  cacheCorruptionCount: number; // Target: 0
  cacheRebuildCount: number;   // Monitor frequency
}

Alerting Rules:

Condition	Threshold	Action
Cache hit rate	<90%	Investigate invalidation frequency
Avg cache load time	>10ms	Check disk I/O, consider SSD
Corruption count	>1/day	Investigate disk health
Rebuild frequency	>10/hour	Optimize invalidation logic

---

6. Migration Plan from SQLite Queries

6.1 Migration Phases

Phase 1: Cache Generation (Week 1)

1. Implement FlatBuffers schema compiler integration
2. Create BinaryCacheBuilder to serialize SQLite data
3. Add cache file I/O with atomic writes
4. Initial cache generation on aqe init

Phase 2: Read-Only Cache (Week 2)

1. Implement BinaryCacheReader with zero-copy access
2. Add validation (checksum, version)
3. Integrate with existing IPatternStore interface
4. A/B testing in CI (50% cache, 50% SQLite)

Phase 3: Invalidation & Rebuild (Week 3)

1. Implement event-based invalidation hooks
2. Add TTL management with background rebuild
3. Graceful fallback to SQLite on errors
4. Performance benchmarking vs SQLite baseline

Phase 4: Production Rollout (Week 4)

1. Feature flag for cache enablement
2. Gradual rollout (10% → 50% → 100%)
3. Monitor cache hit rates and fallback frequency
4. Optimize based on production metrics

6.2 Backward Compatibility

Dual-Mode Operation:

interface PatternStoreConfig {
  enableBinaryCache: boolean;  // Feature flag
  cachePath: string;           // Default: .aqe/cache/patterns.bin
  fallbackToSQLite: boolean;   // Always true for safety
}

Compatibility Matrix:

Scenario	Behavior
New installation	Generate cache on first aqe init
Existing installation	Detect missing cache → Build cache → Use cache
Cache disabled	Use SQLite only (no performance degradation)
Cache corrupted	Fallback to SQLite + Log warning

6.3 Data Consistency Strategy

Write-Through Pattern (Future Extension):

class WriteThoughCache {
  async storePattern(pattern: TestPattern): Promise<void> {
    // 1. Write to SQLite (source of truth)
    await this.sqliteAdapter.storePattern(pattern);

    // 2. Invalidate cache (mark stale)
    this.invalidator.markStale({
      trigger: 'pattern_stored',
      timestamp: Date.now(),
      requiresRebuild: false
    });

    // 3. Optional: Hot-update cache (future optimization)
    // await this.cache.updateEntry(pattern);
  }
}

Current Implementation (Phase 1-3):

• SQLite is source of truth
• Cache is read-only, regenerated on invalidation
• No cache coherency issues (eventual consistency acceptable)

---

7. Performance Projections

7.1 Target vs Baseline Comparison

Metric	Baseline (SQLite)	Target (Cache)	Improvement
Pattern load (single)	2.5ms	<0.5ms	5x faster
Pattern load (batch 100)	32ms	<5ms	6.4x faster
Embedding access	2.5ms	<0.15ms	16x faster
Test discovery (1000 patterns)	500ms	<50ms	10x faster

7.2 Latency Breakdown (Projected)

SQLite Path (Current):

Total: 32ms
├─ SQLite query execution: 18ms (56%)
├─ Row deserialization: 8ms (25%)
├─ JSON parsing (embeddings): 4ms (12%)
└─ Object construction: 2ms (7%)

Binary Cache Path (Target):

Total: <5ms
├─ File mmap (if not cached): 2ms (40%)
├─ FlatBuffers deserialization: 0ms (zero-copy)
├─ Index lookup: 1ms (20%)
└─ Object construction: 2ms (40%)

7.3 Scaling Characteristics

Pattern Bank Size vs Load Time:

Patterns	SQLite (ms)	Binary Cache (ms)	Improvement
100	12	2	6x
1,000	32	5	6.4x
10,000	180	18	10x
100,000	1,800	80	22.5x

Note: FlatBuffers scales better than SQLite due to:

• Zero-copy access (no parsing overhead)
• Sequential memory layout (cache-friendly)
• Pre-built indexes (O(1) lookup vs O(log n) B-tree)

---

8. Risk Assessment & Mitigation

8.1 Risks

Risk	Severity	Probability	Mitigation
Cache corruption	High	Low	Checksum validation + SQLite fallback
Version incompatibility	Medium	Medium	Semantic versioning + migration
Disk space exhaustion	Low	Low	Cache size limits + cleanup
Stale cache serving	Medium	Medium	TTL + event-based invalidation
Performance regression	High	Low	A/B testing + gradual rollout

8.2 Mitigation Strategies

1. Cache Corruption:

   • SHA-256 checksum validation on load
   • Automatic fallback to SQLite
   • Background cache rebuild
   • Monitoring alerts on corruption frequency

2. Version Incompatibility:

   • Semantic versioning in cache header
   • Automatic cache rebuild on version mismatch
   • Migration scripts for major version changes

3. Stale Cache:

   • Event-based invalidation on pattern changes
   • TTL-based expiration
   • Manual cache clear via aqe optimize

4. Performance Regression:

   • Continuous benchmarking in CI
   • Feature flag for cache enablement
   • Gradual rollout (10% → 50% → 100%)
   • Instant rollback capability

---

9. Success Criteria

9.1 Functional Requirements

• ✅ Cache format specification complete (FlatBuffers)
• ✅ Versioning strategy defined (semantic versioning)
• ✅ Checksum validation strategy (SHA-256)
• ✅ Invalidation triggers documented (event-based + TTL)
• ✅ Graceful fallback to SQLite on corruption
• ✅ Migration plan with phased rollout

9.2 Performance Requirements

Requirement	Target	Validation Method
Pattern load time	<5ms	Benchmark suite
Test discovery time	<50ms	End-to-end test
Cache hit rate	>95%	Production metrics
Embedding access	<0.15ms	Microbenchmark
Memory footprint (1000 patterns)	<10MB	Memory profiler

9.3 Quality Attributes

• Reliability: Checksum validation + fallback ensures zero data loss
• Maintainability: Schema evolution via FlatBuffers versioning
• Scalability: Supports 100K+ patterns with linear scaling
• Observability: Metrics for cache hit rate, fallback frequency, rebuild count

---

10. Next Steps (A1.2 Implementation)

1. Setup FlatBuffers Toolchain:

   • Install flatc compiler
   • Integrate schema compilation into build process
   • Generate TypeScript bindings

2. Implement Core Interfaces:

   • BinaryCacheBuilder: SQLite → FlatBuffers serialization
   • BinaryCacheReader: Zero-copy cache access
   • CacheValidator: Checksum + version validation

3. Integration Testing:

   • Unit tests for serialization/deserialization
   • Integration tests for SQLite fallback
   • Performance benchmarks vs baseline

4. CI/CD Integration:

   • Cache generation in aqe init
   • Benchmark regression tests
   • Feature flag for gradual rollout

---

Appendix A: FlatBuffers vs MessagePack Benchmarks

Methodology:

• Benchmark tool: npm run benchmark:serialization
• Dataset: 1000 test patterns with 768-dim embeddings
• Hardware: 8-core CPU, 16GB RAM, NVMe SSD

Results (Projected):

Operation	MessagePack	FlatBuffers	Winner
Serialize 1000 patterns	45ms	38ms	FlatBuffers (1.2x)
Deserialize 1000 patterns	62ms	8ms	FlatBuffers (7.8x)
Access single embedding	1.2ms	0.15ms	FlatBuffers (8x)
Memory usage (1000 patterns)	4.8MB	3.9MB	FlatBuffers (1.2x)
Schema evolution support	Limited	Full	FlatBuffers

Conclusion: FlatBuffers is the clear winner for our use case due to zero-copy deserialization and native Float32Array support.

---

Appendix B: Cache File Format Specification

File Layout:

┌─────────────────────────────────────────────────────┐
│ File Header (64 bytes)                              │
├─────────────────────────────────────────────────────┤
│ Magic Number: 0x41514543 ("AQEC")           4 bytes│
│ Format Version: 0x00010000 (1.0.0)           4 bytes│
│ Checksum: SHA-256 hash                      32 bytes│
│ Timestamp: Unix milliseconds                 8 bytes│
│ Reserved                                    16 bytes│
├─────────────────────────────────────────────────────┤
│ FlatBuffers Payload (variable)                      │
├─────────────────────────────────────────────────────┤
│ - BinaryCache root table                            │
│ - PatternEntry array                                │
│ - AgentConfig array                                 │
│ - IndexData tables                                  │
└─────────────────────────────────────────────────────┘

File Naming Convention:

• Path: .aqe/cache/patterns.bin
• Versioned: .aqe/cache/patterns-v1.bin (future)
• Backup: .aqe/cache/patterns.bin.bak

---

Appendix C: Glossary

• Zero-Copy: Direct memory mapping without deserialization overhead
• HNSW: Hierarchical Navigable Small World (vector search algorithm)
• TTL: Time-To-Live (cache expiration time)
• MMR: Maximal Marginal Relevance (diversity-aware search)
• FlatBuffers: Memory-efficient serialization library by Google
• MessagePack: Binary JSON-like serialization format

---

Document History

Version	Date	Changes
1.0.0	2025-12-12	Initial design specification

---

End of Document