mirror of
https://github.com/wshobson/agents.git
synced 2026-09-14 18:04:20 +08:00
Seth Hobson
31fdae14d6
* fix(plugin-eval): populate model_usage from judge and Monte Carlo layers (#660) * fix(plugin-eval): attribute judge usage to SDK model, scope it per-call Addresses CodeRabbit + Codex review on #668: - query_llm now keys usage_sink by the SDK-reported model (collect_sdk_output(messages).model), falling back to the requested model only when the stream reported none. Keeps judge attribution consistent with the Monte Carlo layer, which already aggregates by SDK-reported model, and stops routing/fallback substitutions from misattributing tokens to the wrong model. - JudgeAnalyzer no longer accumulates model_usage as instance state. analyze_skill() now creates a local dict and threads it through the four assess_* calls, so a reused analyzer (or concurrent analyze_skill calls) can no longer leak or mix token counts between runs. - Parameterized a bare `dict` annotation introduced by this feature in test_judge.py's _result() helper. - Added tests: SDK-reported model differing from the requested model, and repeated analyze_skill() calls on one analyzer not leaking usage.