skills/proffesor-for-testing__agentic-qe
1
0
Fork 0
mirror of https://github.com/proffesor-for-testing/agentic-qe.git synced 2026-09-19 08:45:47 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
transformers-fastest-validation-session
proffesor-for-testing__agen…/scripts/adr-092-phase0a-report.json
T
Dragan Spiridonov 24e30e0584 feat(advisor): implement ADR-092 provider-agnostic advisor strategy 
Multi-model routing layer with circuit breaking, PII redaction, and
domain-specific prompts for the Queen coordinator.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 07:40:20 +00:00

394 lines
19 KiB
JSON
Raw Permalink Blame History

{
"adr": "ADR-092",
"phase": "phase-0a-agent-harness",
"fixture_repo": "https://github.com/ruvnet/RuView",
"fixture_sha": "2a05378bd229df07eff3e309d414d65510ce507c",
"timestamp": "2026-04-11T18:03:29.995Z",
"executor_provider": "openrouter",
"executor_model": "anthropic/claude-sonnet-4",
"advisor_provider": "openrouter",
"advisor_model": "anthropic/claude-opus-4",
"trials": 10,
"successes": 10,
"aggregate": {
"baseline_total_cost_usd": 0.19972600000000001,
"advisor_total_cost_usd": 0.23572699999999996,
"cost_delta_usd": 0.03600099999999995,
"cost_delta_pct": 18.02519451648756,
"baseline_ast_valid": "9/10",
"advisor_ast_valid": "10/10",
"avg_test_fn_count": {
"baseline": 32.6,
"advisor": 35.7,
"delta_pct": 9.509202453987733
},
"avg_assertion_count": {
"baseline": 58.9,
"advisor": 63.5,
"delta_pct": 7.809847198641768
},
"avg_mock_count": {
"baseline": 36.6,
"advisor": 67.8,
"delta_pct": 85.24590163934424
}
},
"gate": {
"quality_ge_baseline": true,
"cost_le_baseline_plus_10": false,
"pass": false
},
"results": [
{
"file": "v1/src/main.py",
"lines": 116,
"baseline": {
"ast_valid": true,
"test_fn_count": 13,
"assertion_count": 1,
"mock_count": 109,
"import_count": 8,
"output_lines": 301,
"tokens_in": 1084,
"tokens_out": 3406,
"cost_usd": 0.011302000000000001,
"latency_ms": 1111
},
"advisor": {
"ast_valid": true,
"test_fn_count": 15,
"assertion_count": 6,
"mock_count": 103,
"import_count": 10,
"output_lines": 287,
"advisor_tokens_in": 1059,
"advisor_tokens_out": 220,
"advisor_cost_usd": 0.001719,
"advisor_latency_ms": 746,
"executor_tokens_in": 1326,
"executor_tokens_out": 3476,
"executor_cost_usd": 0.011754,
"executor_latency_ms": 875,
"combined_cost_usd": 0.013473,
"combined_latency_ms": 1621,
"advisor_plan": "## Strategic Test Plan\n\n1. **Mock Dependencies**: Mock `get_settings()`, `setup_logging()`, `create_app()`, `ServiceOrchestrator`, `create_cli()`, `uvicorn.Server`, and `asyncio.run()`\n\n2. **Priority Methods**: \n - `main()` - core orchestration logic\n - `setup_signal_handlers()` - signal registration\n - `run()` - entry point wrapper\n\n3. **Edge Cases**:\n - `validate_settings()` returning issues in production vs development\n - CLI mode vs server mode detection (`len(sys.argv) > 1`)\n - Exception handling in `main()` with/without orchestrator in locals\n - Signal handler behavior (SIGINT/SIGTERM)\n - KeyboardInterrupt in both `main()` and `run()`\n - Uvicorn config variations based on settings (reload, workers, log_level)"
}
},
{
"file": "v1/src/sensing/classifier.py",
"lines": 201,
"baseline": {
"ast_valid": true,
"test_fn_count": 33,
"assertion_count": 75,
"mock_count": 0,
"import_count": 5,
"output_lines": 461,
"tokens_in": 1976,
"tokens_out": 4543,
"cost_usd": 0.015605,
"latency_ms": 910
},
"advisor": {
"ast_valid": true,
"test_fn_count": 32,
"assertion_count": 70,
"mock_count": 8,
"import_count": 4,
"output_lines": 444,
"advisor_tokens_in": 1951,
"advisor_tokens_out": 220,
"advisor_cost_usd": 0.002611,
"advisor_latency_ms": 825,
"executor_tokens_in": 2217,
"executor_tokens_out": 4437,
"executor_cost_usd": 0.015528,
"executor_latency_ms": 1098,
"combined_cost_usd": 0.018139,
"combined_latency_ms": 1923,
"advisor_plan": "1. **Mock dependencies**: Mock `RssiFeatures` from `v1.src.sensing.feature_extractor` - create test fixtures with controlled variance, motion_band_power, breathing_band_power, n_change_points values.\n\n2. **Priority methods**: \n - `classify()` - core logic with all classification rules\n - `_compute_confidence()` - complex weighted scoring algorithm\n\n3. **Edge cases to cover**:\n - Zero/negative thresholds in constructor\n - Variance exactly at `presence_variance_threshold` boundary\n - Motion energy exactly at `motion_energy_threshold` boundary\n - Empty `other_receiver_results` list vs None\n - Division by zero in confidence calculation (when thresholds = 0)\n - Confidence clamping to [0,1] range\n - Cross-receiver agreement with mismatched motion levels\n\n4. **Test property getters** for threshold access."
}
},
{
"file": "v1/src/hardware/router_interface.py",
"lines": 240,
"baseline": {
"ast_valid": true,
"test_fn_count": 38,
"assertion_count": 73,
"mock_count": 18,
"import_count": 8,
"output_lines": 470,
"tokens_in": 2320,
"tokens_out": 5123,
"cost_usd": 0.017689,
"latency_ms": 1007
},
"advisor": {
"ast_valid": true,
"test_fn_count": 44,
"assertion_count": 82,
"mock_count": 49,
"import_count": 7,
"output_lines": 620,
"advisor_tokens_in": 2293,
"advisor_tokens_out": 211,
"advisor_cost_usd": 0.0029259999999999998,
"advisor_latency_ms": 887,
"executor_tokens_in": 2552,
"executor_tokens_out": 6725,
"executor_cost_usd": 0.022726999999999997,
"executor_latency_ms": 921,
"combined_cost_usd": 0.025652999999999995,
"combined_latency_ms": 1808,
"advisor_plan": "1. **Mock dependencies**: `asyncssh.connect()`, `ssh_client.run()`, `ssh_client.close()`, and `CSIData` import\n2. **Priority methods**: `_validate_config()`, `connect()`, `execute_command()`, `disconnect()`\n3. **Test `_validate_config()`**: Missing fields, invalid port types/values, valid configs\n4. **Test connection flow**: Success/failure scenarios, timeout handling, connection state tracking\n5. **Test `execute_command()`**: Not connected state, retry mechanism (3 attempts), command failures, timeouts\n6. **Test `configure_csi_monitoring()`**: Channel validation (1-196 range), command injection prevention\n7. **Edge cases**: Connection drops during command execution, malformed responses, concurrent operations\n8. **Test `_parse_csi_response()`**: Always raises `RouterConnectionError` as documented"
}
},
{
"file": "v1/src/models/densepose_head.py",
"lines": 278,
"baseline": {
"ast_valid": true,
"test_fn_count": 37,
"assertion_count": 88,
"mock_count": 2,
"import_count": 6,
"output_lines": 447,
"tokens_in": 3170,
"tokens_out": 5033,
"cost_usd": 0.018269,
"latency_ms": 1071
},
"advisor": {
"ast_valid": true,
"test_fn_count": 31,
"assertion_count": 44,
"mock_count": 189,
"import_count": 6,
"output_lines": 422,
"advisor_tokens_in": 3141,
"advisor_tokens_out": 256,
"advisor_cost_usd": 0.003909,
"advisor_latency_ms": 800,
"executor_tokens_in": 3447,
"executor_tokens_out": 6888,
"executor_cost_usd": 0.024111,
"executor_latency_ms": 922,
"combined_cost_usd": 0.02802,
"combined_latency_ms": 1722,
"advisor_plan": "1. **Mock dependencies**: `torch.nn` modules (Conv2d, BatchNorm2d, etc.), `torch.nn.functional` (cross_entropy, l1_loss, softmax), `torch.nn.init` methods\n\n2. **Priority methods**: `__init__` (config validation), `forward` (main logic), `_validate_config`, loss computation methods (`compute_segmentation_loss`, `compute_uv_loss`, `compute_total_loss`)\n\n3. **Edge cases to test**:\n - Invalid config (missing fields, negative values)\n - Wrong input tensor shape in `forward` (should raise `DensePoseError`)\n - Empty `hidden_channels` list\n - FPN enabled/disabled paths\n - Zero division in `get_prediction_confidence` UV variance calculation\n - Tensor shape mismatches in loss functions\n\n4. **Test tensor operations**: Mock `torch.sigmoid`, `torch.argmax`, `torch.var`, `torch.max` for predictable outputs\n\n5. **Validate weight initialization**: Check `_initialize_weights` calls correct init methods"
}
},
{
"file": "v1/src/api/middleware/auth.py",
"lines": 306,
"baseline": {
"ast_valid": true,
"test_fn_count": 32,
"assertion_count": 45,
"mock_count": 24,
"import_count": 8,
"output_lines": 380,
"tokens_in": 2896,
"tokens_out": 4168,
"cost_usd": 0.0154,
"latency_ms": 1025
},
"advisor": {
"ast_valid": true,
"test_fn_count": 37,
"assertion_count": 71,
"mock_count": 30,
"import_count": 7,
"output_lines": 423,
"advisor_tokens_in": 2871,
"advisor_tokens_out": 261,
"advisor_cost_usd": 0.003654,
"advisor_latency_ms": 1179,
"executor_tokens_in": 3178,
"executor_tokens_out": 4823,
"executor_cost_usd": 0.017647,
"executor_latency_ms": 1248,
"combined_cost_usd": 0.021301,
"combined_latency_ms": 2427,
"advisor_plan": "1. **Mock dependencies**: `get_settings()`, `jwt.decode()`, `datetime.utcnow()`, `datetime.fromtimestamp()`, `logger`\n\n2. **Priority methods**: `dispatch()` (main flow), `_verify_token()` (JWT validation), `_extract_token()` (token extraction), `_is_public_path()`, `_is_protected_path()`\n\n3. **Mock Request/Response objects**: Use `unittest.mock.Mock` for FastAPI `Request` with `.url.path`, `.headers`, `.query_params`, `.cookies`, `.state`\n\n4. **Edge cases**: \n - Malformed JWT tokens in `_verify_token()`\n - Missing/invalid Authorization header formats\n - Token blacklist race conditions in `TokenBlacklist`\n - Path matching edge cases (trailing slashes, case sensitivity)\n - Exception handling in `dispatch()` when `call_next()` fails\n - Token expiration boundary conditions\n\n5. **Test `TokenBlacklist._cleanup_if_needed()` timing logic and `APIKeyAuth` CRUD operations**"
}
},
{
"file": "v1/src/config.py",
"lines": 309,
"baseline": {
"ast_valid": true,
"test_fn_count": 42,
"assertion_count": 67,
"mock_count": 68,
"import_count": 7,
"output_lines": 619,
"tokens_in": 2964,
"tokens_out": 7022,
"cost_usd": 0.02403,
"latency_ms": 863
},
"advisor": {
"ast_valid": true,
"test_fn_count": 43,
"assertion_count": 66,
"mock_count": 58,
"import_count": 6,
"output_lines": 585,
"advisor_tokens_in": 2939,
"advisor_tokens_out": 198,
"advisor_cost_usd": 0.003533,
"advisor_latency_ms": 1486,
"executor_tokens_in": 3183,
"executor_tokens_out": 6221,
"executor_cost_usd": 0.021845999999999997,
"executor_latency_ms": 1216,
"combined_cost_usd": 0.025379,
"combined_latency_ms": 2702,
"advisor_plan": "1. **Mock dependencies**: `get_settings()`, `get_domain_config()`, `os.environ`, `Path.mkdir()`, `validate_settings()`\n\n2. **Priority methods**:\n - `__init__` with lazy loading properties\n - `set_environment_override`/`clear_environment_overrides` \n - `validate_configuration` (complex validation logic)\n - `get_storage_config` (creates directories)\n\n3. **Edge cases**:\n - Property access when `_settings`/`_domain_config` is None\n - `get_redis_config()` when `redis_url` is None\n - `validate_configuration` exception handling at each step\n - Storage path creation failures\n - Environment override with non-string values\n - `get_full_config()` with partial failures"
}
},
{
"file": "v1/src/api/middleware/rate_limit.py",
"lines": 325,
"baseline": {
"ast_valid": true,
"test_fn_count": 38,
"assertion_count": 57,
"mock_count": 42,
"import_count": 9,
"output_lines": 431,
"tokens_in": 3359,
"tokens_out": 5182,
"cost_usd": 0.018905,
"latency_ms": 1079
},
"advisor": {
"ast_valid": true,
"test_fn_count": 37,
"assertion_count": 70,
"mock_count": 25,
"import_count": 8,
"output_lines": 500,
"advisor_tokens_in": 3332,
"advisor_tokens_out": 230,
"advisor_cost_usd": 0.0040219999999999995,
"advisor_latency_ms": 1640,
"executor_tokens_in": 3610,
"executor_tokens_out": 5252,
"executor_cost_usd": 0.019365999999999998,
"executor_latency_ms": 835,
"combined_cost_usd": 0.023388,
"combined_latency_ms": 2475,
"advisor_plan": "1. **Mock dependencies**: `get_settings()`, `time.time()`, `logger`, `Request.client.host`, `Request.state.user`, `Request.headers`\n\n2. **Priority methods**: `dispatch()`, `_check_rate_limits()`, `_check_limit()`, `_get_client_id()`, `_get_user_type()`\n\n3. **Mock FastAPI objects**: `Request`, `Response`, `JSONResponse`, use `AsyncMock` for `call_next`\n\n4. **Edge cases to cover**:\n - Missing `request.client` (None case)\n - Empty/missing user-agent headers\n - Time boundary conditions in sliding window\n - Concurrent requests at rate limit threshold\n - Block expiration edge timing\n - Empty deques in cleanup\n - Exception handling in `_add_rate_limit_headers()`\n\n5. **Test data structures**: Verify `defaultdict(deque)` behavior, blocked_clients dict state"
}
},
{
"file": "v1/src/core/phase_sanitizer.py",
"lines": 346,
"baseline": {
"ast_valid": true,
"test_fn_count": 61,
"assertion_count": 86,
"mock_count": 38,
"import_count": 6,
"output_lines": 597,
"tokens_in": 3687,
"tokens_out": 7979,
"cost_usd": 0.027624,
"latency_ms": 1017
},
"advisor": {
"ast_valid": true,
"test_fn_count": 42,
"assertion_count": 71,
"mock_count": 22,
"import_count": 5,
"output_lines": 352,
"advisor_tokens_in": 3659,
"advisor_tokens_out": 225,
"advisor_cost_usd": 0.004334,
"advisor_latency_ms": 1599,
"executor_tokens_in": 3933,
"executor_tokens_out": 5539,
"executor_cost_usd": 0.02055,
"executor_latency_ms": 1562,
"combined_cost_usd": 0.024884,
"combined_latency_ms": 3161,
"advisor_plan": "1. **Mock**: `logging.Logger`, `scipy.signal.butter`, `scipy.signal.filtfilt`\n\n2. **Priority methods**:\n - `__init__` with `_validate_config`\n - `sanitize_phase` (main pipeline)\n - `unwrap_phase` (all three methods: numpy/scipy/custom)\n - `validate_phase_data`\n\n3. **Edge cases**:\n - Empty arrays (`phase_data.size == 0`)\n - 1D arrays for `validate_phase_data`\n - Even `smoothing_window` in `_apply_moving_average`\n - Small arrays (`< 18 elements`) in `_apply_low_pass_filter`\n - Division by zero in `_detect_outliers` (handled by `+ 1e-8`)\n - No valid indices in `_interpolate_outliers`\n - Config with missing/invalid fields"
}
},
{
"file": "v1/src/services/orchestrator.py",
"lines": 394,
"baseline": {
"ast_valid": false,
"ast_error": "'(' was never closed (<unknown>, line 721)",
"test_fn_count": 0,
"assertion_count": 0,
"mock_count": 0,
"import_count": 0,
"output_lines": 721,
"tokens_in": 3495,
"tokens_out": 8192,
"cost_usd": 0.028071,
"latency_ms": 1339
},
"advisor": {
"ast_valid": true,
"test_fn_count": 47,
"assertion_count": 68,
"mock_count": 132,
"import_count": 6,
"output_lines": 655,
"advisor_tokens_in": 3469,
"advisor_tokens_out": 223,
"advisor_cost_usd": 0.004138,
"advisor_latency_ms": 10281,
"executor_tokens_in": 3739,
"executor_tokens_out": 6909,
"executor_cost_usd": 0.024466,
"executor_latency_ms": 10026,
"combined_cost_usd": 0.028604,
"combined_latency_ms": 20307,
"advisor_plan": "1. **Mock dependencies:**\n - `Settings` object\n - `HealthCheckService`, `MetricsService` \n - `get_hardware_service()`, `get_pose_service()`, `get_stream_service()`\n - `PoseStreamHandler`\n - `connection_manager`\n - `asyncio.create_task()`\n\n2. **Priority methods:**\n - `initialize()` - test double init, exception handling\n - `start()` - test double start, init-before-start\n - `shutdown()` - test task cancellation, service cleanup order\n - `service_context()` - test context manager flow\n\n3. **Edge cases:**\n - Background task exceptions in `_health_check_loop()`, `_metrics_collection_loop()`\n - Service without start/shutdown/reset methods\n - `restart_service()` with non-existent service\n - Concurrent shutdown during initialization"
}
},
{
"file": "v1/src/api/routers/health.py",
"lines": 420,
"baseline": {
"ast_valid": true,
"test_fn_count": 32,
"assertion_count": 97,
"mock_count": 65,
"import_count": 7,
"output_lines": 555,
"tokens_in": 3889,
"tokens_out": 6314,
"cost_usd": 0.022831,
"latency_ms": 1487
},
"advisor": {
"ast_valid": true,
"test_fn_count": 29,
"assertion_count": 87,
"mock_count": 62,
"import_count": 6,
"output_lines": 586,
"advisor_tokens_in": 3864,
"advisor_tokens_out": 236,
"advisor_cost_usd": 0.004572,
"advisor_latency_ms": 1702,
"executor_tokens_in": 4146,
"executor_tokens_out": 6056,
"executor_cost_usd": 0.022314,
"executor_latency_ms": 1313,
"combined_cost_usd": 0.026886,
"combined_latency_ms": 3015,
"advisor_plan": "1. **Mock dependencies:**\n - `request.app.state` (hardware_service, pose_service, stream_service)\n - `psutil` functions (cpu_percent, virtual_memory, disk_usage, etc.)\n - `get_current_user` dependency\n - `get_settings()`\n\n2. **Priority methods:**\n - `health_check()` - test all service states (healthy/unhealthy/unavailable)\n - `readiness_check()` - test ready/not-ready conditions\n - `get_system_metrics()` - test psutil exceptions\n\n3. **Edge cases:**\n - Services returning exceptions during health_check()\n - Missing app.state attributes (AttributeError)\n - psutil raising exceptions (OSError, AttributeError)\n - Memory/disk thresholds (89.9% vs 90.1%, 0.9GB vs 1.1GB)\n - Windows vs Unix differences (getloadavg, sensors_temperatures)"
}
}
]
}
Reference in New Issue View Git Blame Copy Permalink