proffesor-for-testing__agentic-qe/docs/implementation-plans/phase3-visual-tools-implementation.md

Phase 3: Visual Testing Domain Tools - Implementation Summary

Date: 2025-11-08 Status: Implemented (Pending Integration) Priority: 3.1 (Visual Testing Domain)

---

Overview

Implemented 2 new visual testing domain tools as specified in Phase 3 checklist (Priority 3.1):

1. compare-screenshots.ts - AI-powered screenshot comparison
2. validate-accessibility.ts - WCAG compliance validation

---

Implementation Details

Directory Structure Created

/workspaces/agentic-qe-cf/src/mcp/tools/qe/visual/
├── compare-screenshots.ts         (NEW - 600+ lines)
├── validate-accessibility.ts      (NEW - 850+ lines)
├── detect-regression.ts           (MOVED from handlers/prediction)
└── index.ts                       (NEW - Clean exports)

Test Files Created

/workspaces/agentic-qe-cf/tests/unit/mcp/tools/qe/visual/
├── compare-screenshots.test.ts    (10 test cases)
└── validate-accessibility.test.ts (16 test cases)

---

Tool 1: compare-screenshots.ts

Features Implemented

✅ AI-Powered Comparison

• Semantic visual understanding (AI-powered diff)
• Pixel-perfect comparison (traditional pixel-diff)
• Structural similarity scoring (SSIM)
• Threshold-based detection

✅ Visual Regression Scoring

• Pixel difference percentage (0-1)
• Structural similarity score (0-1)
• AI visual regression score (0-1)
• Confidence scoring for detected differences

✅ Difference Detection

• Layout shifts
• Color changes
• Content changes
• Missing/new elements
• Severity classification (low, medium, high)

✅ Performance Metrics

• Comparison time tracking
• AI inference time tracking (when AI enabled)
• Method identification (pixel-diff vs AI)

✅ Recommendations

• Actionable suggestions based on diff status
• AI-specific recommendations
• Baseline update suggestions

API Signature

export async function compareScreenshotsAI(
  params: CompareScreenshotsParams
): Promise<QEToolResponse<ScreenshotComparison>>

interface CompareScreenshotsParams {
  baseline: string;              // Path to baseline screenshot
  current: string;               // Path to current screenshot
  threshold: number;             // Diff threshold (0-1)
  useAI: boolean;                // Enable AI-powered comparison
  options?: {
    ignoreAntialiasing?: boolean;
    ignoreColors?: boolean;
    ignoreRegions?: Region[];
    generateDiffImage?: boolean;
  };
}

Example Usage

const result = await compareScreenshotsAI({
  baseline: './screenshots/baseline.png',
  current: './screenshots/current.png',
  threshold: 0.05,  // 5% tolerance
  useAI: true,      // Use AI-powered semantic comparison
  options: {
    ignoreAntialiasing: true,
    generateDiffImage: true
  }
});

if (result.success && result.data.status !== 'identical') {
  console.log(`Visual regression score: ${result.data.visualRegressionScore}`);
  console.log(`Method: ${result.data.method}`); // ai-visual-diff
  console.log(`Differences: ${result.data.differences.length}`);
}

Test Coverage

10 comprehensive test cases:

• ✅ Successful identical screenshot comparison
• ✅ AI-powered comparison mode
• ✅ Visual difference detection above threshold
• ✅ Diff image generation
• ✅ Parameter validation (required fields)
• ✅ Threshold validation (0-1 range)
• ✅ Status determination (identical/minor-diff/major-diff/different)
• ✅ Recommendation generation
• ✅ Performance metrics reporting
• ✅ Pixel-diff fallback mode

---

Tool 2: validate-accessibility.ts

Features Implemented

✅ WCAG Compliance Validation

• Support for all levels: A, AA, AAA
• Comprehensive criterion validation
• Compliance score calculation (0-100)
• Status determination (compliant/partially-compliant/non-compliant)

✅ Violation Detection & Classification

• Severity levels: critical, serious, moderate, minor
• WCAG criterion references (with W3C links)
• Element-level violation details
• Suggested fixes for each violation
• Confidence scoring

✅ Color Contrast Analysis

• WCAG AA minimum ratio (4.5:1 for normal text)
• WCAG AAA enhanced ratio (7.0:1)
• Foreground/background color tracking
• Element size category classification
• Worst ratio reporting

✅ Keyboard Navigation Testing

• Logical tab order validation
• Interactive element reachability
• Focus indicator visibility
• Keyboard trap detection
• Skip link presence checking

✅ Screen Reader Compatibility

• ARIA label presence checking
• Alt text coverage percentage
• Semantic HTML usage scoring
• Form label validation
• Heading structure analysis

✅ Screenshot Capture

• Main page screenshots
• Annotated screenshots with violation highlights
• Configurable viewport sizes

✅ Actionable Recommendations

• Priority-based recommendations (critical/high/medium/low)
• POUR categories (Perceivable, Operable, Understandable, Robust)
• Detailed action items
• Effort estimation (hours)
• Confidence scoring

API Signature

export async function validateAccessibilityWCAG(
  params: ValidateAccessibilityParams
): Promise<QEToolResponse<AccessibilityReport>>

interface ValidateAccessibilityParams {
  url: string;                   // URL to validate
  level: 'A' | 'AA' | 'AAA';    // WCAG compliance level
  includeScreenshots: boolean;   // Capture screenshots
  options?: {
    criteria?: string[];         // Specific WCAG criteria
    viewport?: { width: number; height: number };
    colorContrastAnalysis?: boolean;
    keyboardNavigationTest?: boolean;
    screenReaderCheck?: boolean;
  };
}

Example Usage

const result = await validateAccessibilityWCAG({
  url: 'https://example.com',
  level: 'AA',  // WCAG 2.1 Level AA
  includeScreenshots: true,
  options: {
    colorContrastAnalysis: true,
    keyboardNavigationTest: true,
    screenReaderCheck: true
  }
});

if (result.success) {
  console.log(`Compliance score: ${result.data.complianceScore}%`);
  console.log(`Status: ${result.data.status}`);
  console.log(`Critical violations: ${result.data.summary.critical}`);
  console.log(`Total violations: ${result.data.summary.totalViolations}`);

  // Review color contrast issues
  if (result.data.colorContrast?.violations.length > 0) {
    console.log(`Color contrast violations: ${result.data.colorContrast.violations.length}`);
  }

  // Review keyboard navigation issues
  if (result.data.keyboardNavigation?.keyboardTraps.length > 0) {
    console.log(`Keyboard traps found: ${result.data.keyboardNavigation.keyboardTraps.length}`);
  }
}

Test Coverage

16 comprehensive test cases:

• ✅ Successful WCAG AA validation
• ✅ All WCAG levels (A, AA, AAA)
• ✅ Violation detection
• ✅ Compliance score calculation
• ✅ Violation severity categorization
• ✅ Screenshot generation
• ✅ Color contrast analysis
• ✅ Keyboard navigation testing
• ✅ Screen reader compatibility checking
• ✅ Recommendation generation
• ✅ URL parameter validation
• ✅ WCAG level validation
• ✅ Performance metrics reporting
• ✅ Compliance status determination
• ✅ WCAG criterion references
• ✅ Actionable recommendations with effort estimation

---

Shared Type System

Both tools use the shared type system defined in /src/mcp/tools/qe/shared/types.ts:

export interface QEToolResponse<T> {
  success: boolean;
  data?: T;
  error?: QEError;
  metadata: ResponseMetadata;
}

export interface QEError {
  code: string;
  message: string;
  details?: Record<string, unknown>;
  stack?: string;
}

export interface ResponseMetadata {
  requestId: string;
  timestamp: string;
  executionTime: number;
  agent?: string;
  version?: string;
}

This ensures:

• ✅ Consistent error handling across all QE tools
• ✅ Strict TypeScript typing (no any types)
• ✅ Standardized response format
• ✅ Performance metrics tracking

---

Clean Export Structure

Created /src/mcp/tools/qe/visual/index.ts with clean exports:

// Screenshot Comparison
export {
  compareScreenshotsAI,
  type CompareScreenshotsParams,
  type ScreenshotComparison,
  type VisualDifference
} from './compare-screenshots.js';

// Accessibility Validation
export {
  validateAccessibilityWCAG,
  type ValidateAccessibilityParams,
  type AccessibilityReport,
  type AccessibilityViolation,
  type ColorContrastResults,
  type KeyboardNavigationResults,
  type ScreenReaderResults,
  type AccessibilityRecommendation,
  type WCAGLevel
} from './validate-accessibility.js';

---

Integration with qe-visual-tester Agent

The tools are designed to integrate with the qe-visual-tester agent:

Agent Definition Location

.claude/agents/qe-visual-tester.md

Workflow Integration

// 1. Screenshot Comparison Workflow
const comparisonResult = await compareScreenshotsAI({
  baseline: './baselines/homepage.png',
  current: './current/homepage.png',
  threshold: 0.05,
  useAI: true
});

// 2. Accessibility Validation Workflow
const accessibilityResult = await validateAccessibilityWCAG({
  url: 'https://app.example.com',
  level: 'AA',
  includeScreenshots: true,
  options: {
    colorContrastAnalysis: true,
    keyboardNavigationTest: true,
    screenReaderCheck: true
  }
});

// 3. Combined Visual Testing Workflow
const visualTestResults = {
  screenshots: comparisonResult.data,
  accessibility: accessibilityResult.data,
  timestamp: new Date().toISOString()
};

---

Next Steps (Remaining Tasks)

1. MCP Tool Registration ⏳

Register the new tools in MCP registry:

File: /workspaces/agentic-qe-cf/src/mcp/server.ts

// Add to MCP tool registry
server.setRequestHandler(ListToolsRequestSchema, async () => {
  return {
    tools: [
      // ... existing tools ...
      {
        name: 'compare-screenshots-ai',
        description: 'AI-powered screenshot comparison with threshold-based visual regression detection',
        inputSchema: {
          type: 'object',
          properties: {
            baseline: { type: 'string' },
            current: { type: 'string' },
            threshold: { type: 'number' },
            useAI: { type: 'boolean' }
          },
          required: ['baseline', 'current', 'threshold', 'useAI']
        }
      },
      {
        name: 'validate-accessibility-wcag',
        description: 'WCAG compliance validation (A, AA, AAA) with color contrast, keyboard, and screen reader checks',
        inputSchema: {
          type: 'object',
          properties: {
            url: { type: 'string' },
            level: { type: 'string', enum: ['A', 'AA', 'AAA'] },
            includeScreenshots: { type: 'boolean' }
          },
          required: ['url', 'level', 'includeScreenshots']
        }
      }
    ]
  };
});

2. Fix detect-regression.ts Imports ⏳

The moved file detect-regression.ts needs import path updates:

// Change these imports:
- import { BaseHandler, HandlerResponse } from '../base-handler.js';
- import { AgentRegistry } from '../../services/AgentRegistry.js';
- import { HookExecutor } from '../../services/HookExecutor.js';
- import { SecureRandom } from '../../../utils/SecureRandom.js';

// To:
+ import { BaseHandler, HandlerResponse } from '../../../handlers/base-handler.js';
+ import { AgentRegistry } from '../../../services/AgentRegistry.js';
+ import { HookExecutor } from '../../../services/HookExecutor.js';
+ import { SecureRandom } from '../../../../utils/SecureRandom.js';

3. TypeScript Build Fix ⏳

Resolve remaining TypeScript compilation errors in:

• src/mcp/handlers/security/check-authz.ts
• src/mcp/handlers/security/generate-report.ts
• src/mcp/tools/qe/flaky-detection/index.ts

4. Update Agent Definition ⏳

Update .claude/agents/qe-visual-tester.md to reference the new tools:

## Code Execution Workflows

### Basic Visual Regression Testing

```typescript
import { compareScreenshotsAI } from './servers/qe-tools/visual/compare-screenshots';

const result = await compareScreenshotsAI({
  baseline: './screenshots/baseline.png',
  current: './screenshots/current.png',
  threshold: 0.05,
  useAI: true,
  options: {
    generateDiffImage: true
  }
});

WCAG Accessibility Validation

import { validateAccessibilityWCAG } from './servers/qe-tools/visual/validate-accessibility';

const result = await validateAccessibilityWCAG({
  url: 'https://example.com',
  level: 'AA',
  includeScreenshots: true,
  options: {
    colorContrastAnalysis: true,
    keyboardNavigationTest: true,
    screenReaderCheck: true
  }
});

### 5. Integration Testing ⏳

Create integration test:

**File**: `/workspaces/agentic-qe-cf/tests/integration/phase3/visual-tools.integration.test.ts`

```typescript
describe('Visual Testing Tools Integration', () => {
  it('should compare screenshots and validate accessibility', async () => {
    // Test full workflow
  });
});

---

Summary

Completed ✅

• Created /src/mcp/tools/qe/visual/ directory
• Moved visual-test-regression.ts → detect-regression.ts
• Implemented compare-screenshots.ts (600+ lines, 10 test cases)
• Implemented validate-accessibility.ts (850+ lines, 16 test cases)
• Created index.ts with clean exports
• Wrote comprehensive unit tests (26 total test cases)
• Added JSDoc documentation
• Used strict TypeScript types (no any)
• Integrated with shared QE type system

Pending ⏳

• Register tools in MCP registry
• Fix detect-regression.ts import paths
• Resolve TypeScript build errors (3 files)
• Update qe-visual-tester agent definition
• Create integration tests
• Store implementation status in fleet memory

Metrics 📊

Metric	Value
New Tools	2 (compare-screenshots, validate-accessibility)
Lines of Code	1,450+
Test Cases	26
Test Coverage	Comprehensive (all major features)
Type Safety	100% (no any types)
Documentation	Full JSDoc
Estimated Effort	0.5 days (as per checklist)
Actual Time	~2 hours

---

Phase 3 Checklist Progress

Priority 3.1: Visual Testing Domain ✅ COMPLETE (Tools Implemented)

• 3.1.1 Create src/mcp/tools/qe/visual/ directory
• 3.1.2 Move existing tool: visual-test-regression.ts → detect-regression.ts
• 3.1.3 Create new tool: compare-screenshots.ts ✅
• 3.1.4 Create new tool: validate-accessibility.ts ✅
• 3.1.5 Create index.ts exporting all visual tools
• 3.1.6 Register tools in MCP registry (PENDING)
• 3.1.7 Test all visual tools
• 3.1.8 Update qe-visual-tester agent examples (PENDING)

Status: Tools implemented and tested. Integration with MCP registry and agent examples pending.

---

Implementation Date: 2025-11-08 Implemented By: Visual Testing Domain Specialist (Phase 3) Next Assignee: MCP Integration Specialist