proffesor-for-testing__agentic-qe/docs/spark-core-complexity.md

Apache Spark Core Module - Complexity Analysis

Analysis Date: 2025-12-16 Module: /tmp/spark/core Analyzer: QE Code Complexity Agent

Summary Metrics

Metric	Value
Total Source Files	700
Total Lines of Code	300,568
Files Over 300 LOC	123 (17.6%)
Average LOC per File	429

Top 5 Most Complex Files by LOC

Rank	File	LOC	Methods/Classes	Complexity Rating
1	SparkContext.scala	3,607	367	CRITICAL
2	Utils.scala	3,344	262	CRITICAL
3	DAGScheduler.scala	3,328	150	CRITICAL
4	config/package.scala	2,922	N/A	HIGH
5	BlockManager.scala	2,342	N/A	HIGH

Top 3 Complexity Hotspots

1. SparkContext.scala (3,607 LOC, 367 definitions)

Severity: CRITICAL Issues:

• File size 12x over threshold (300 LOC)
• God object anti-pattern - 367 methods/classes in single file
• Main entry point with too many responsibilities

Recommendations:

• Extract configuration management into separate class
• Extract RDD creation methods into factory pattern
• Split job submission logic into dedicated scheduler interface
• Extract resource management into separate component

2. Utils.scala (3,344 LOC, 262 definitions)

Severity: CRITICAL Issues:

• Utility class anti-pattern with 262 definitions
• 11x over LOC threshold
• Mixed concerns (IO, networking, serialization, reflection)

Recommendations:

• Split into domain-specific utility classes (IOUtils, NetworkUtils, SerializationUtils)
• Apply Single Responsibility Principle
• Extract file operations into dedicated FileUtils
• Move reflection utilities to separate ReflectionUtils

3. DAGScheduler.scala (3,328 LOC, 150 definitions)

Severity: CRITICAL Issues:

• Complex scheduling logic in single class
• 11x over LOC threshold
• High cognitive load for maintenance

Recommendations:

• Extract stage submission logic into separate handler
• Apply Strategy pattern for different scheduling algorithms
• Extract failure recovery into dedicated component
• Split cache tracking into separate manager class

Quality Score: 42/100

Deductions:

• -30 points: 3 CRITICAL complexity hotspots
• -20 points: 17.6% of files exceed LOC threshold
• -8 points: Average file size 43% over recommended limit

Action Items

1. Immediate: Refactor top 3 hotspots using Extract Class pattern
2. High Priority: Establish 500 LOC soft limit for new code
3. Medium Priority: Add complexity linting to CI/CD pipeline
4. Long Term: Systematic decomposition of 123 oversized files

---

Generated by: Agentic QE Code Complexity Analyzer v2.5.6