2 Commits

Author SHA1 Message Date
Seth Hobson 31fdae14d6 fix(plugin-eval): populate model_usage from judge and Monte Carlo layers (#660) (#668)
* fix(plugin-eval): populate model_usage from judge and Monte Carlo layers (#660)

* fix(plugin-eval): attribute judge usage to SDK model, scope it per-call

Addresses CodeRabbit + Codex review on #668:

- query_llm now keys usage_sink by the SDK-reported model
  (collect_sdk_output(messages).model), falling back to the requested
  model only when the stream reported none. Keeps judge attribution
  consistent with the Monte Carlo layer, which already aggregates by
  SDK-reported model, and stops routing/fallback substitutions from
  misattributing tokens to the wrong model.
- JudgeAnalyzer no longer accumulates model_usage as instance state.
  analyze_skill() now creates a local dict and threads it through the
  four assess_* calls, so a reused analyzer (or concurrent
  analyze_skill calls) can no longer leak or mix token counts between
  runs.
- Parameterized a bare `dict` annotation introduced by this feature in
  test_judge.py's _result() helper.
- Added tests: SDK-reported model differing from the requested model,
  and repeated analyze_skill() calls on one analyzer not leaking usage.
2026-08-18 10:31:31 -04:00
Seth Hobson 88061a68ce feat(plugin-eval): implement eval engine with composite scoring and layer blending
Adds EvalEngine that coordinates static analysis, blends layer scores
across dimensions with renormalized weights, and produces PluginEvalResult
with composite score, badge, and per-dimension grades. Layer 2/3 stubs
ready for Tasks 8/9.
2026-03-25 17:42:11 -04:00