mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Nicolò Boschi
20bd4f3618
* fix(ci): compare OpenAPI against the merge-base; build benchmark role configs whole Two unrelated CI failures, both of which fail without anything being wrong with the code under test. OpenAPI compatibility diffed the branch's spec against the LIVE tip of the base branch, so every endpoint main gained after a branch was cut is reported as "Endpoint removed (breaks old clients)" by that branch. Three open PRs failed this way today on /health/live and /health/ready (added by #3329), none of which touch the spec at all; the only cure was an unrelated rebase. Compare against `git merge-base origin/$BASE_BRANCH HEAD` instead, which asks the question the check means to ask: did *this branch* remove something. Genuine removals still fail — verified both directions against the real specs. The scheduled LoComo benchmark has failed every night since at least Aug 8, before its first question: `LoComoAnswerGenerator()` raised "HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID is required". The workflow does export that variable — but each benchmark role built its LLMConfig from exactly four env vars (provider, api_key, base_url, model), and LLMConfig deliberately does not read the environment for provider-specific settings (from_env is where the API resolves them). Every Vertex AI value was therefore dropped. The same four-var construction was copy-pasted at three sites — locomo, longmemeval and the shared judge — so fixing only the crashing one would have moved the failure down a line. Replace all three with a shared builder that carries the Vertex AI project/region/service-account through. hindsight-dev/tests had no CI job, which is why a plain construction bug was left for a nightly benchmark to find hours later. Add one, so those tests (and the new regression test) actually run on PRs. * fix(ci): make the benchmark role-config tests hermetic They passed locally off the developer's HINDSIGHT_API_LLM_API_KEY and failed in the new test-dev job, where no key exists, with "API key is required for openai" — the tests were reading ambient environment instead of declaring what they need. Clear every HINDSIGHT_API_*LLM* var before each test and set the ones under test explicitly.
67 lines
2.7 KiB
Python
67 lines
2.7 KiB
Python
"""Build the benchmark-role LLM configs (answer generator, judge).
|
|
|
|
Benchmarks configure two LLMs independently of the API's own config system: the
|
|
answer generator (``HINDSIGHT_API_ANSWER_LLM_*``) and the judge
|
|
(``HINDSIGHT_API_JUDGE_LLM_*``), each falling back to the shared
|
|
``HINDSIGHT_API_LLM_*`` values.
|
|
|
|
Each call site used to build its own ``LLMConfig`` from four env vars —
|
|
provider, api_key, base_url, model — and nothing else. ``LLMConfig.__init__``
|
|
deliberately does not read the environment for provider-specific settings (its
|
|
docstring says the caller resolves them; ``LLMConfig.from_env`` is where the API
|
|
does it), so those hand-rolled configs silently dropped every Vertex AI setting.
|
|
With ``provider=vertexai`` the constructor then raised
|
|
"HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID is required" before a single question ran,
|
|
which is what broke the scheduled LoComo benchmark.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
|
|
from hindsight_api.engine.llm_wrapper import LLMConfig
|
|
|
|
# Roles a benchmark can configure separately from the engine's own LLM.
|
|
ANSWER_ROLE = "ANSWER"
|
|
JUDGE_ROLE = "JUDGE"
|
|
|
|
|
|
def _role_env(role: str, suffix: str, default: str = "") -> str:
|
|
"""Read ``HINDSIGHT_API_<ROLE>_LLM_<SUFFIX>``, falling back to the shared var."""
|
|
return os.getenv(
|
|
f"HINDSIGHT_API_{role}_LLM_{suffix}",
|
|
os.getenv(f"HINDSIGHT_API_LLM_{suffix}", default),
|
|
)
|
|
|
|
|
|
def build_role_llm_config(
|
|
role: str,
|
|
*,
|
|
default_model: str = "gpt-4o-mini",
|
|
reasoning_effort: str = "high",
|
|
) -> LLMConfig:
|
|
"""Build the ``LLMConfig`` for a benchmark role.
|
|
|
|
Args:
|
|
role: ``ANSWER_ROLE`` or ``JUDGE_ROLE`` — selects the env var prefix.
|
|
default_model: Model to use when neither the role nor the shared var is set.
|
|
reasoning_effort: Passed straight through to ``LLMConfig``.
|
|
|
|
Returns:
|
|
A config carrying the provider-specific settings as well as the common
|
|
ones, so providers that need more than an api_key (currently Vertex AI)
|
|
are usable for benchmark roles.
|
|
"""
|
|
return LLMConfig(
|
|
provider=_role_env(role, "PROVIDER", "openai"),
|
|
api_key=_role_env(role, "API_KEY"),
|
|
base_url=_role_env(role, "BASE_URL"),
|
|
model=_role_env(role, "MODEL", default_model),
|
|
reasoning_effort=reasoning_effort,
|
|
# Vertex AI authenticates with a project/region/service account rather
|
|
# than an api_key, and LLMConfig requires the project id up front.
|
|
vertexai_project_id=_role_env(role, "VERTEXAI_PROJECT_ID") or None,
|
|
vertexai_region=_role_env(role, "VERTEXAI_REGION") or None,
|
|
vertexai_service_account_key=_role_env(role, "VERTEXAI_SERVICE_ACCOUNT_KEY") or None,
|
|
)
|