proffesor-for-testing__agentic-qe/docs/ci

CI/CD Documentation

Overview

This directory contains comprehensive documentation for all CI/CD workflows and processes in the Agentic QE project.

Available Documentation

Workflows

• Benchmark Workflow - Automated performance regression detection for every PR
  • Baseline comparison
  • Regression thresholds (5% warning, 10% failure)
  • Automatic PR comments
  • Historical trend analysis

Quick Links

• GitHub Actions Workflows: /.github/workflows/
• Benchmark Suite: /benchmarks/suite.ts
• Local Benchmark Script: /scripts/run-benchmarks.sh

CI/CD Architecture

┌─────────────────────────────────────────────────┐
│            Pull Request Trigger                 │
└───────────┬─────────────────────────────────────┘
            │
            ├─────────────────┬──────────────────┬──────────────────┐
            │                 │                  │                  │
            ▼                 ▼                  ▼                  ▼
    ┌───────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
    │   Fast Tests  │ │ Benchmarks   │ │Infrastructure│ │   Coverage   │
    │   (Journey +  │ │ (Regression  │ │    Tests     │ │   Analysis   │
    │   Contract)   │ │  Detection)  │ │              │ │              │
    └───────┬───────┘ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
            │                 │                 │                 │
            └─────────────────┴─────────────────┴─────────────────┘
                                    │
                                    ▼
                            ┌───────────────┐
                            │   Dashboard   │
                            │  (Summary &   │
                            │  PR Comment)  │
                            └───────────────┘

Workflow Status

Workflow	Status	Purpose
Optimized CI	✅ Active	Fast journey + contract tests
Benchmark	✅ Active	Performance regression detection
Quality Gate	✅ Active	Code quality checks
Learning Integration	✅ Active	ML model validation
Migration Validation	✅ Active	Schema migration tests

Common Tasks

Running Benchmarks Locally

# Run all benchmarks
npm run benchmark:local

# Compare with baseline
npm run benchmark:local -- --baseline=v2.3.5

# Run specific benchmark
npm run benchmark:local -- --filter=agent

# Multiple runs for stability
npm run benchmark:local -- --runs=5

Updating Baselines

Baselines are automatically updated when code is merged to main. To manually update:

# Run benchmarks and save results
npm run benchmark -- --output=benchmarks/baselines/v2.3.6.json

# Commit the new baseline
git add benchmarks/baselines/v2.3.6.json
git commit -m "chore(benchmarks): update baseline to v2.3.6"

Debugging CI Failures

1. Check GitHub Actions logs: View detailed output for each step
2. Run locally: Use the same commands as CI to reproduce issues
3. Check artifacts: Download uploaded artifacts for detailed analysis
4. Review PR comments: Automated comments provide summaries and actionable insights

Performance Thresholds

Benchmark Regression Thresholds

Threshold	Change	Status	Action
< 5%	Performance degradation	✅ Pass	No action required
5-10%	Minor regression	⚠️ Warning	Review and justify
> 10%	Major regression	❌ Fail	Must fix before merge
< -5%	Performance improvement	✅ Pass	Celebrate!

Test Coverage Thresholds

Metric	Threshold	Status
Line Coverage	80%	Required
Branch Coverage	75%	Target
Function Coverage	85%	Target

Best Practices

For Contributors

1. Run benchmarks before submitting PR

   npm run benchmark:local -- --baseline=v2.3.5
   

2. Keep test execution under 5 minutes

   • Use npm run test:fast for quick validation
   • Full test suite runs in CI

3. Write performance-conscious code

   • Avoid unnecessary allocations
   • Use caching when appropriate
   • Profile critical paths

4. Document performance changes

   • Add comments explaining optimizations
   • Link to related issues/PRs
   • Provide before/after metrics

For Reviewers

1. Check benchmark results in PR comments

   • Review automated performance analysis
   • Question regressions > 3%

2. Verify test coverage

   • Ensure new code has tests
   • Check for gaps in critical paths

3. Review CI status

   • All checks must pass
   • Investigate flaky tests
   • Verify no skipped tests

Troubleshooting

Common Issues

Benchmark Failures

Symptom: Benchmarks fail with OOM errors

Solution:

# Increase memory limit (local)
NODE_OPTIONS='--max-old-space-size=4096' npm run benchmark

# Check for memory leaks
npm run benchmark:local -- --verbose

Flaky Tests

Symptom: Tests pass/fail intermittently

Solution:

1. Run multiple times to confirm: npm run test:fast -- --runs=5
2. Check for timing issues (async/await)
3. Isolate test dependencies
4. Use proper cleanup in afterEach/afterAll

Baseline Not Found

Symptom: Warning about missing baseline

Solution:

# Create baseline from current main branch
git checkout main
npm run benchmark -- --output=benchmarks/baselines/v2.3.5.json
git add benchmarks/baselines/v2.3.5.json
git commit -m "chore(benchmarks): add baseline for v2.3.5"

Configuration

GitHub Actions Settings

Required permissions for workflows:

permissions:
  contents: read        # Read repository files
  pull-requests: write  # Comment on PRs
  checks: write        # Update check status

Environment Variables

Variable	Purpose	Default
NODE_OPTIONS	Node.js memory/GC settings	--max-old-space-size=2048
GITHUB_SHA	Commit hash for tracking	Auto-set by GitHub
GITHUB_REF	Branch reference	Auto-set by GitHub

Metrics and Monitoring

Key Performance Indicators (KPIs)

• CI Execution Time: Target < 2 minutes for fast tests
• Benchmark Execution Time: Target < 10 minutes
• Test Success Rate: Target > 99%
• Flaky Test Rate: Target < 1%
• Coverage Trend: Target steady increase

Dashboards

• GitHub Actions: View workflow runs and trends
• Test Dashboard: Generated by dashboard job
• Performance Dashboard: Coming soon (Phase 2)

Future Enhancements

Planned Features

1. Performance Budgets

   • Per-feature performance budgets
   • Automatic budget enforcement
   • Budget violation alerts

2. Advanced Analytics

   • Historical trend analysis
   • Regression prediction
   • Optimization recommendations

3. Cross-Platform Benchmarks

   • Test on multiple OS/platforms
   • Compare platform performance
   • Detect platform-specific issues

4. Distributed Testing

   • Parallel test execution
   • Sharded test runs
   • Faster CI feedback

Support and Resources

Documentation

• Benchmark Workflow - Detailed benchmark documentation
• GitHub Actions Docs - Official GitHub Actions documentation
• Performance Testing Guide - Performance testing best practices

Contact

• Issues: https://github.com/proffesor-for-testing/agentic-qe/issues
• Discussions: https://github.com/proffesor-for-testing/agentic-qe/discussions
• CI/CD Team: ci-team@example.com

---

Last Updated: 2025-12-12 Version: 1.0.0 Maintainers: CI/CD Team, Performance Engineering Team