mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
perf/profiling-env
43 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Nicolò Boschi
|
7a52b0a6b7 |
fix(embeddings): let litellm-sdk invoke a Bedrock inference profile ARN (#4114)
* fix(embeddings): let litellm-sdk invoke a Bedrock inference profile ARN (#4034) LiteLLM derives the Bedrock embedding request/response shape by pattern-matching a provider name (cohere/amazon/twelvelabs/nova) out of the `model` string, while the target it actually invokes comes from a separate `model_id` kwarg that defaults to `model`. `LiteLLMSDKEmbeddings` only ever exposed one string, so both were forced to the same value. That makes an application inference profile ARN unusable: the ARN is opaque, so provider detection fails on it, and orgs whose Service Control Policy denies `bedrock:InvokeModel` on the bare model id can't use the bare id either. Neither setting works, and there is no third one. Expose the split litellm already has: `HINDSIGHT_API_EMBEDDINGS_LITELLM_SDK_MODEL` stays a recognizable id for provider detection, and the new `HINDSIGHT_API_EMBEDDINGS_LITELLM_SDK_MODEL_ID` carries the ARN to invoke. Unset, nothing changes — no `model_id` kwarg is sent and litellm falls back to `model`. * docs(models): show how to point litellm-sdk embeddings at a Bedrock profile ARN Adds a plain Bedrock example to the embeddings config block, plus a tip for the application inference profile case: which of the two vars picks the request format, which one is invoked, and why they have to be separate. Notes that chat models need none of this (bedrock/converse/<arn> takes any model). |
||
|
Nicolò Boschi
|
30ce3b8d11 |
feat(llm): add Meta Model API as a first-class provider (#4081)
* feat(llm): add Meta Model API as a first-class provider Meta Model API (https://ai.developer.meta.com) serves the Muse family over an OpenAI-compatible chat/completions endpoint, so it slots into the existing OpenAICompatibleLLM path exactly like deepseek / zai / atlas. Set `HINDSIGHT_API_LLM_PROVIDER=meta` to route fact extraction, reflection and consolidation through it. The base URL defaults to https://api.meta.ai/v1 and the default model is muse-spark-1.3 (1M context). Two Meta-specific behaviours are worth knowing, and are documented rather than worked around: - Muse Spark always reasons. `reasoning_effort: "none"` is rejected with HTTP 400 — only minimal/low/medium/high/xhigh are accepted, or omit it. Reasoning tokens are billed against the output budget, so the per-operation max-token limits need headroom. - Chat Completions documents `max_tokens`, not `max_completion_tokens`. Muse Spark is a reasoning model but not one of the OpenAI products the frozen `_supports_reasoning_model` name list recognises, so the parameter name comes from the provider default. A test pins that, and pins that a configured reasoning_effort still reaches the request (the #3449 drop list only covers OpenAI's own non-reasoning products). The provider is registered on the chat/completions path only. Meta also serves a Responses and an Anthropic-Messages endpoint; a `meta-responses` variant mirroring openai/openai-responses was considered and deliberately left out. Capability flags are left off deliberately: Meta has no batch endpoint, and its prompt caching is automatic (no key, flag, or breakpoints), so it does not fit the explicit `get_or_create_cached_prefix` contract `supports_prompt_caching` describes — the benefit applies for free either way. Changes: - engine/llm_wrapper.py: register "meta" in create_llm_provider(), LLMProvider.valid_providers, and the default base_url map - engine/providers/openai_compatible_llm.py: register "meta" in valid_providers, default base_url, and the API-key-required check - config.py: PROVIDER_DEFAULT_MODELS["meta"] = muse-spark-1.3 - tests/test_meta_provider.py: default model/base URL, API-key requirement, the max_tokens parameter name, and reasoning_effort pass-through - hindsight-embed control center: add Meta Model API to the provider wizard - docs: add Meta to llmProviders.json (drives the providers grid, the capability table and the default-models table) and config examples in developer/models.mdx + developer/configuration.md - README + .env.example (+ the bundled embed copy): document the new provider Also colours the docs provider grid, which was previously monochrome. Each tile now carries its brand colour, taken from the Simple Icons dataset — the same project the marks themselves come from, so a tile's colour matches its mark. Near-black brands (OpenAI, Ollama, Anthropic, ...) get a dark-theme override so they do not vanish against the dark surface. Providers with no Simple Icons entry (Groq, Fireworks, Atlas Cloud, Requesty, opencode-go, Nous, llama.cpp, LiteLLM) keep the neutral inherited colour rather than an invented hex. The fallback is `inherit`, so the other IconGrid caller (ClientsGrid) is unchanged. Meta uses its own mark (SiMeta) rather than the generic OpenAI-compatible glyph. Not verified against the live API — no Meta API key was available — so muse-spark-1.3 is deliberately absent from the "Tested Models" table, which means models verified to work. Worth probing first with a real key: Meta rejects recursive JSON schemas in structured output with HTTP 400. * docs(llm): mark muse-spark-1.3 tested against the live Meta API Verified end-to-end through Hindsight's own create_llm_provider() against https://api.meta.ai/v1 (HTTP 200, valid content, token usage parsed including reasoning tokens), so muse-spark-1.3 now belongs in the Tested Models table. Four behaviours confirmed live, all matching the published docs: - Structured output with a flat json_schema works, and classified the probe input correctly (world vs experience). - Recursive JSON schemas are rejected: HTTP 400 "Recursive JSON schemas are not currently supported". Audited every Pydantic response model in engine/response_models.py for self-reference through $defs — none is recursive, so no Hindsight path is affected. - reasoning_effort "none" is rejected: HTTP 400 '"reasoning_effort" does not support "none" with this model.' Other levels are accepted. - Reasoning tokens are substantial and come out of the output budget: a trivial prompt spent 87 reasoning tokens against 11 visible output tokens, and at max_tokens=64 the response comes back with no content at all. Hindsight's defaults leave ample room (retain 64000; consolidation and reflect unbounded), so this only bites an operator who lowers the cap — which is what the configuration note added with the provider already warns about. Also records in the max-tokens test that Meta accepts max_completion_tokens as well, so sending max_tokens is a choice between two working names rather than a correctness fix. * fix(llm): reflect failed outright on Meta — tool_choice is auto-only Found by running a real Hindsight instance against Meta Model API and exercising retain, recall, reflect and consolidation end to end. Reflect returned HTTP 400 on every call: only `"auto"` is supported for `tool_choice`. `"none"`, `"required"`, and named function choices are not currently supported Reflect's agent loop forces a retrieval tool on its first turns, so the whole reflect surface was unusable on this provider. The unit tests could not have caught it: they cover provider construction and parameter naming, not the tool-calling path. This is the opposite failure mode to the one `_drops_tool_choice_required` handles. LM Studio and Ollama accept the field and silently ignore it, so reflect answers badly (#1563/#1179); Meta rejects the request outright, so reflect answers not at all. The two need separate predicates, hence `_rejects_non_auto_tool_choice` alongside the existing check rather than a widening of it. The field is dropped for any non-auto mode. A named choice has already been narrowed to a single tool by the block above, so the call stays practically forced under auto — the same reasoning the DeepSeek branch relies on. "none" cannot be expressed by omission and would become "auto"; no caller reaches this path with it (only the gemini, claude-code and github-copilot providers handle NONE), so that is documented in place rather than given an untested tools-stripping branch. tests/test_meta_tool_choice.py covers required, named and auto, and asserts the carve-out does not leak to other OpenAI-compatible endpoints. The LM Studio and required-downgrade suites still pass unchanged. Also documents the latency finding from the same run: Muse Spark reasons before every reply, and reflect's 30s default deadline is too short for its final synthesis — it timed out four times before failing. Raising HINDSIGHT_API_REFLECT_LLM_TIMEOUT and HINDSIGHT_API_LLM_TIMEOUT to 300 makes reflect return a correct grounded answer in ~60s. Verified end to end on the fixed build: retain 26s (3 facts, entities and a March 2026 temporal range), recall 1.7s (3 hits, correctly ranked), reflect 60s (grounded answer), consolidation completed (4 observations, 15 links, 0 failed operations). * docs(models): give Meta Model API its own setup section with the required knobs The provider's settings were inline comments inside the shared 20-provider config block, which is the wrong place for something an operator must act on: three of the four are required, not tuning, and one of them (the reflect deadline) is the difference between reflect working and reflect returning nothing at all. Adds a "Meta Model API Setup" section alongside the other providers that need one, with the required knobs as a table that states why each is required — every one of them a consequence of Muse Spark always reasoning before it replies. Also records the model lineup, the contributor-tier trade-off, and the four things worth knowing up front: prompt caching is automatic (which is why the capability table shows none), there is no batch or embeddings endpoint, recursive JSON schemas are rejected, and calls are slow. The shared config block keeps the two timeout exports, since they are required to be set, and now points at the section for the reasoning. |
||
|
Nicolò Boschi
|
c42e6323e9 |
feat(llm): per-provider Codex credentials directory (#3793) (#3983)
* feat(llm): per-provider Codex credentials directory (#3793) Codex auth resolves `auth.json` from the process-wide `CODEX_HOME`, so every `openai-codex` provider a Hindsight process builds reads the same store. A multi-LLM chain of two Codex members therefore authenticates twice as the same ChatGPT account: when the preferred profile hits its usage limit, failover just retries it. There was no way to express "prefer profile A, fall back to profile B". Add `codex_home` alongside the existing per-member provider settings (`VERTEXAI_*`, `LITELLMROUTER_CONFIG`), which is all this needs — the routing half of #3793 already exists: - `HINDSIGHT_API_LLM_CODEX_HOME` for the primary - `HINDSIGHT_API_LLM_<n>_CODEX_HOME` for indexed members (also under the `RETAIN_` / `REFLECT_` / `CONSOLIDATION_` prefixes) Each falls back to `CODEX_HOME`, then `~/.codex`, so nothing changes for existing deployments. Refresh is already coordinated per auth-file path (`_path_scoped_lock`), so two profiles refresh independently and cannot overwrite each other's tokens. The field is server-level only, deliberately not in `_CONFIGURABLE_FIELDS`: it is a filesystem path to a credential store, and accepting it over the bank config API would let a bank point the server at an arbitrary file. Scope: this is the credential-store seam only. Failover keeps its existing generic semantics — any `Exception` from a member advances to the next, with no quota-vs-terminal classification and no cooldown, so a rate-limited primary is re-tried at the head of every request before the fallback serves it. Batch retain already stays pinned to the member that submitted it, so batch affinity holds. The `openai-codex` embeddings provider still reads `CODEX_HOME` only — it has no member chain to span. Closes #3793 Claude-Session: https://claude.ai/code/session_011KDT484YujNcBxHzbfzNbk * fix(llm): append codex_home instead of inserting it mid-signature `create_llm_provider` and `LLMProvider.__init__` still have callers that pass the older settings positionally, and two guard tests in test_llm_wrapper.py assert exactly that. Inserting `codex_home` before `vertexai_project_id` pushed every later parameter one slot along, so a positional `timeout` landed on `cache_affinity` (`assert 120.0 == 7.5` / `assert None == 7.5`). Move it to the end of both signatures and leave a comment saying why new parameters go there, since the mistake is invisible at the call sites that use keywords. Claude-Session: https://claude.ai/code/session_011KDT484YujNcBxHzbfzNbk * chore(docs): regenerate docs skill for the --shm-size docker examples Drift inherited from main: the `--shm-size=1g` flag was added to the docker run examples in installation/quickstart without re-running the skill generator, so verify-generated-files is red on every PR branched after it. Regenerated output only — no source docs changed here. Claude-Session: https://claude.ai/code/session_011KDT484YujNcBxHzbfzNbk |
||
|
Max Marino
|
1d9a6a381c |
feat(llm): add GitHub Copilot subscription provider (#3597)
Adds a `github-copilot` LLM provider backed by the official GitHub Copilot SDK, using the signed-in Copilot entitlement with no `HINDSIGHT_API_LLM_API_KEY`. Verified end-to-end against a live Copilot subscription: retain, consolidation and reflect all run on the provider. Review fixes on top of the original submission: - default to `gpt-5.6-terra`; the submitted `gpt-5.6-sol` is not in the model list the runtime serves, so every call failed and the provider was unusable out of the box - make a rejected request configuration terminal instead of a runtime failure; an unavailable model was invalidating the shared runtime and respawning `copilot --headless` once per attempt (5 spawns at max_retries=3, ~12 at the default of 10) - let a token-authenticated host run without `~/.copilot`, so the documented COPILOT_GITHUB_TOKEN / GH_TOKEN / GITHUB_TOKEN path works in containers and CI - regenerate skills/hindsight-docs (the provider list also feeds the generated faq.md) Co-authored-by: Max Marino <dudujuju828@users.noreply.github.com> |
||
|
Parafee41
|
7850efcef6 |
fix: disable automatic cache affinity for Azure OpenAI (#3521)
* fix Azure cache affinity detection * docs: state why Azure hosts opt out of prompt_cache_key Azure OpenAI does accept prompt_cache_key on GPT deployments; what rejects it is a non-OpenAI Foundry model (DeepSeek) served over the same *.openai.azure.com endpoint. The host can't distinguish the two, so auto stays off there - but operators on an Azure GPT deployment can opt back in with openai_prompt_cache_key. --------- Co-authored-by: Nicolò Boschi <boschi1997@gmail.com> |
||
|
Nicolò Boschi
|
77bdda7042 |
docs(examples): add TEI embeddings + reranker docker-compose example (#3465)
* docs(examples): add TEI embeddings + reranker docker-compose example
Adds docker/docker-compose/tei/ — a runnable Compose stack that serves
embeddings and reranking from two HuggingFace Text Embeddings Inference
(TEI) sidecars, with the slim Hindsight image talking to them via
HINDSIGHT_API_{EMBEDDINGS,RERANKER}_PROVIDER=tei. Serves Hindsight's
default models so it's a drop-in 'move embeddings/reranking onto TEI'
demo. Links it from the TEI section of the models docs.
Verified end-to-end: retain + recall return the expected memory with
both TEI semantic and reranker scores populated.
* docs(examples): prod-like TEI tuning — bge-reranker-base + throughput flags
Swap the reranker to BAAI/bge-reranker-base (the cross-encoder commonly
paired with bge-small embeddings on dedicated inference servers) and carry
prod-like TEI throughput flags on both services (--max-concurrent-requests,
--max-batch-tokens, --max-client-batch-size) instead of TEI's bare defaults,
so the example doubles as a starting point for real deployments.
|
||
|
Nicolò Boschi
|
1d0dcbce25 |
fix(reflect): decouple page max_tokens from the provider output cap (#3365) (#3389)
On thinking models the provider's output budget covers reasoning tokens plus visible output, so passing the reflect/mental-model `max_tokens` straight through as `max_completion_tokens` let reasoning consume the whole budget and truncated the page mid-word — silently, since the call still returned success. Treat `max_tokens` as what it means to a user: a *visible* page-length target. It is now communicated to the model as a prompt directive and enforced by the existing post-hoc rewrite, never as a hard cap on the synthesis call. The transport-level cost cap is a separate, uncapped-by-default config (`HINDSIGHT_API_REFLECT_MAX_COMPLETION_TOKENS`) so reasoning never starves the answer. This fixes the truncation for every provider without model-specific detection. Also surface `finishReason == MAX_TOKENS` on Gemini as a warning instead of returning half-written text as a silent success. - config: add `reflect_max_completion_tokens` (None = uncapped) - reflect: synthesis + rewrite use the config cap, not the page budget; `build_final_prompt` carries the length directive - gemini: log a truncation warning on a non-empty MAX_TOKENS response - docs: models.mdx reasoning/`max_tokens` note + configuration.md entry - tests: prompt directive, config default/override, uncapped-by-default synthesis, config-cap override, Gemini truncation warning |
||
|
Nicolò Boschi
|
d4ac97d643 |
docs: correct the Azure OpenAI base URL (#3385)
Reported in #3377 and verified against a live Azure OpenAI resource. The OpenAI-compatible tip told users to point HINDSIGHT_API_LLM_BASE_URL at "your provider's endpoint", which for Azure reads as the resource root -- and Azure does not serve the API there, so it returns 404 Resource not found. Measured against a real resource (gpt-5-mini deployment): https://<res>.openai.azure.com 404 https://<res>.openai.azure.com/openai/deployments/<dep> 404 (no api-version) https://<res>.openai.azure.com/openai/v1 works .../openai/deployments/<dep>?api-version=2025-01-01-preview works Adds an Azure OpenAI Setup section with both working shapes and the three things that actually bite: the model is the *deployment* name, the key is the resource key (an APIM subscription key is a different setup), and gateways must preserve the path shape. Also records that Azure accepts the prompt_cache_key field sent under cache_affinity=auto (#3271) on every api-version from 2024-02-01 onward, so that default needs no Azure carve-out -- an explicitly untested risk when #3271 merged, now closed. |
||
|
JoshFunnell
|
82852c5225 |
feat(llm): xai-oauth provider — LLM lanes on a SuperGrok subscription (#3272)
Adds the `xai-oauth` provider: serves Hindsight's LLM lanes from a flat-rate
SuperGrok subscription via an RFC 8628 device-code grant against auth.x.ai, with
proactive/reactive refresh over a shared 0600 on-disk store. No API key. For
API-key access to the same endpoint, `provider: openai` with an api.x.ai base
URL still applies.
Reviewed and tested live against a real xAI subscription (grok-4.5): device-code
login, plain/structured/tool-calling completions, and a real token refresh.
Six follow-up fixes landed on top of the contribution, each with a RED-proven
regression test — see
|
||
|
Nicolò Boschi
|
55883dc517 |
feat(llm): add openai-responses provider (OpenAI Responses API) (#3121)
Add a provider that talks exclusively to the OpenAI Responses API (`client.responses.create` → `/v1/responses`) — never chat/completions. Motivation: reasoning models such as gpt-5.6-terra reject `reasoning_effort` combined with function tools on `/v1/chat/completions` (HTTP 400 unless `reasoning_effort="none"`, see #2983). Reflect is a tool-calling search loop, so that constraint forces the whole reflect operation — including the final synthesis — to run with reasoning disabled. The Responses API models the chain-of-thought as a first-class reasoning item, so reasoning and tools coexist; reflect's search loop can now run with a real reasoning effort. `OpenAIResponsesLLM` is a standalone `LLMInterface` implementation (OpenAI-only; it deliberately does NOT subclass the multi-vendor chat/completions provider, so it can never route through `chat.completions` and carries none of the groq/ollama/deepseek special-casing). It reuses only provider-agnostic pure helpers (text cleanup, quota-defer parsing). It translates the engine's chat-shaped inputs: - chat messages → `input` items (assistant `tool_calls` → `function_call`, `role="tool"` → `function_call_output` keyed by `call_id`), - nested `{"type":"function","function":{...}}` tools → flattened `{"type":"function","name":...,"parameters":...}`, - flat `reasoning_effort` → a `reasoning={"effort": ...}` object, - `response_format` → `text={"format": {...}}` (strict json_schema or the soft schema-in-prompt + json_object fallback), - reads `response.output_text` + `function_call` items from `response.output`. Generic LLM config flags are honored: `extra_body`, `timeout`, per-call `temperature`/`max_completion_tokens`/`max_retries`. It also wires two flags the chat/completions path drops — `openai_service_tier` (as the native Responses `service_tier`) and `default_headers` (on the SDK client). The conversation is replayed statelessly each turn (`store=False`, no `previous_response_id`); server-side reasoning reuse across turns is left as a future optimization. Wiring: provider registration + dispatch, `PROVIDER_DEFAULT_MODELS` default (`gpt-5.6`), docs + `.env.example` (+ embed template sync), and an `openai` floor bump to `>=1.66.0` for the Responses API surface. Unit tests mock `responses.create` (incl. the generic-flag wiring) and pin that reasoning + tools are sent together on the tool path — the combination chat/completions rejects. Validated live against real gpt-5.6-terra: plain + reasoning, strict structured output, reasoning+tools together with a stateless function_call replay, and a full run_reflect_agent end-to-end (stubbed retrieval, no DB) — tools drove to a correct synthesized answer. |
||
|
JiehoonKwak
|
c6a1b507ba | Fix Codex OAuth request identity (#2647) | ||
|
Nicolò Boschi
|
0c8699dc20 |
docs: document CODEX_HOME isolation for long-running Codex services (#2496)
Hindsight already honors CODEX_HOME (openai-codex LLM + embeddings), but it was only mentioned in the 0.8.3 changelog. Long-running services sharing ~/.codex/auth.json with another Codex process can get their refresh token rotated out, leaving /reflect broken while /health stays green. Add a 'Isolating Codex auth for long-running services' section to the Models docs and a pointer next to the openai-codex snippet in configuration. Refs #2476 |
||
|
lucaszhu-hue
|
f8d277697d |
feat: add Atlas Cloud as an OpenAI-compatible LLM provider (#2362)
Atlas Cloud (https://www.atlascloud.ai) exposes an OpenAI-compatible chat/completions endpoint, so it slots into the existing OpenAICompatibleLLM path exactly like deepseek / zai / opencode-go. Set `HINDSIGHT_API_LLM_PROVIDER=atlas` to route fact extraction, reflection and consolidation through Atlas Cloud. The base URL defaults to https://api.atlascloud.ai/v1 and the default model is deepseek-ai/deepseek-v4-pro (a reasoning model — give it enough max_tokens, >= 512). Changes: - engine/llm_wrapper.py: register "atlas" in create_llm_provider(), LLMProvider.valid_providers, and the default base_url map - engine/providers/openai_compatible_llm.py: register "atlas" in valid_providers, default base_url, and the API-key-required check - config.py: PROVIDER_DEFAULT_MODELS["atlas"] = deepseek-ai/deepseek-v4-pro - hindsight-embed control center: add Atlas Cloud to the provider wizard - docs: add Atlas Cloud to llmProviders.json (drives the providers grid and table) and a config example in developer/models.mdx - README + .env.example: document the new provider Verified end-to-end: instantiated the atlas provider through Hindsight's own create_llm_provider() and made a live call() to deepseek-ai/deepseek-v4-pro (HTTP 200, valid content + token usage). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Evo
|
2c2a20b290 |
docs(models): sync anthropic default model to claude-haiku-4-5 alias (#2326)
The runtime default for the anthropic provider is the self-updating alias `claude-haiku-4-5` (config.py PROVIDER_DEFAULT_MODELS, enforced by tests/test_provider_default_models.py), but the Models docs advertised the date-pinned snapshot `claude-haiku-4-5-20251001`. A pinned snapshot and a self-updating alias differ for pricing/retirement, and the page contradicted hermes.md (which already says `claude-haiku-4-5`). Sync the canonical sources (llmProviders.json default-model table + models.mdx examples) to the alias and regenerate the docs skill mirror. |
||
|
Nicolò Boschi
|
82c7df7266 |
feat(providers): add native Nous Portal provider (Codex-style OAuth, no hermes_cli dep) (#2102)
* feat(providers): add native Nous Portal provider (codex-style OAuth, no hermes_cli dep)
Adds a 'nous' provider that speaks the OpenAI-compatible wire format (thin
subclass of OpenAICompatibleLLM) and authenticates with the rotating,
inference-scoped JWT from a 'hermes portal' login — read natively from
~/.hermes/auth.json, exactly mirroring the Codex provider. No dependency on
the hermes_cli package.
- nous_auth.py: NousAuthManager reads providers.nous OAuth state, decodes the
JWT exp for proactive refresh, and refreshes via POST {portal}/api/oauth/token
(x-nous-refresh-token header). Atomic write-back of rotated tokens. Because
the Hermes auth store is shared with a possibly-running Hermes agent, refresh
takes the same ~/.hermes/auth.lock flock Hermes uses and re-reads the latest
refresh_token from disk before exchange (single-use RT reuse-detection safety).
- nous_llm.py: thin subclass; proactive refresh offloaded to a thread so the
event loop never blocks; one reactive refresh + retry on a 401.
- llm_wrapper.py: register nous in dispatch, validator, no-key set, base-url default.
- tests: auth-store load/refresh/persist/terminal-error + provider wiring (no
Hermes install or network needed).
* docs(providers): document nous provider + add default model
- config.py: add nous to PROVIDER_DEFAULT_MODELS (deepseek/deepseek-v4-flash)
so omitting HINDSIGHT_API_LLM_MODEL doesn't fall back to gpt-4o-mini.
- configuration.md: add nous to the provider list + an env example block.
- models.mdx: add a nous example + a 'Nous Portal Setup (Hermes)' section
covering the 'hermes portal' login, the no-API-key flow, and automatic
JWT refresh that coordinates with a running Hermes agent.
- skills/hindsight-docs: regenerated bundle from the docs sources.
|
||
|
Evo
|
a0d91408cf | docs: add gemini embedding 2 models (#2090) | ||
|
Mobilinkd LLC
|
27cb1c6843 |
Support service_tier selection for Amazon Bedrock (#2098)
* feat: add HINDSIGHT_API_LLM_BEDROCK_SERVICE_TIER env var Adds support for setting Bedrock service tier (flex/priority/reserved) via environment variable, following the same pattern as the existing Groq and OpenAI service tier support. - config.py: env constant, default, dataclass field, os.getenv() load - llm_wrapper.py: bedrock_service_tier param plumbing - litellm_llm.py: inject service_tier kwarg for bedrock/ models - configuration.md: table entry + Bedrock example block - models.md/mdx: Bedrock tip block update Closes #2072 * Add validation + tests for HINDSIGHT_API_LLM_BEDROCK_SERVICE_TIER - validate() rejects invalid values (e.g. 'standard') with clear error - Empty string treated as unset (matching llm_output_language pattern) - Tests: default, flex, priority, reserved, invalid value, empty string * fix(api): thread bedrock_service_tier from config into LLM providers The new HINDSIGHT_API_LLM_BEDROCK_SERVICE_TIER flag was plumbed through LLMProvider/create_llm_provider/LiteLLMLLM but nothing ever constructed an LLMProvider with the resolved config value, so the env var was inert (service_tier was never injected into the Bedrock call). - memory_engine.py: pass bedrock_service_tier=config.llm_bedrock_service_tier to all four LLMConfig constructions (default/retain/reflect/consolidation) - llm_wrapper.py: LLMProvider.from_env() reads the env var too, so ad-hoc constructions honor it - test_bedrock_service_tier.py: plumbing tests asserting the tier reaches the LiteLLM call kwargs for bedrock/ models, is omitted otherwise, and is guarded off non-Bedrock models * style(test): ruff format test_config_validation.py (fix verify-generated-files) --------- Co-authored-by: Hermes Agent (Rob) <hermes-sa@mobilinkd.com> Co-authored-by: Nicolò Boschi <boschi1997@gmail.com> |
||
|
Evo
|
bd60c7575c | docs(models): document the onnx embeddings provider (#2020) | ||
|
Evo
|
78c27bfa74 |
docs(models): sync gemini + vertexai default models to 3.x matching config.py (#2030)
* docs(models): sync gemini + vertexai default models to 3.x matching config.py * docs(models): regenerate skills mirror default-model table (gemini+vertexai 3.x) * docs(models): sync Vertex AI walkthrough + gemini examples to 3.x (complete #2030 scope) The defaults table fix (#2030) left the env-var examples and Vertex AI setup walkthrough still handing users the retired gemini-2.0-flash-001 (404 on Vertex) and stale gemini-2.0-flash. Sync the prose surface: - Vertex AI examples + google/ prefix note -> gemini-3.1-flash-lite (vertexai default) - Gemini AI Studio example -> gemini-3.5-flash (gemini default) Regenerated the CI-enforced skills mirror. |
||
|
Nicolò Boschi
|
3b2830c7d8 |
docs(models): note Groq free tier (8k TPM) is unsuitable for Hindsight (#2015)
Retain reserves max_completion_tokens (~64k) up front, and Groq's free-tier 8k TPM limit counts that reservation at admission, so every retain call is rejected with HTTP 413 'Request too large' even for a one-line message. Document that the free tier is unsuitable and a paid tier / other provider is required. Refs #1573. |
||
|
Derek Bouius
|
a3797e2014 |
docs: update Gemini model recommendations to 3.x series (#1787)
Replace deprecated Gemini models with their 3.x successors: - gemini-3-pro-preview → gemini-3.1-pro-preview (shut down March 2026) - gemini-2.5-flash → gemini-3.5-flash - gemini-2.5-flash-lite → gemini-3.1-flash-lite Also update default models in config.py for gemini and vertexai providers. |
||
|
Chris Bartholomew
|
d7a3aa5269 |
feat(llm): provider prompt-prefix caching — retain + consolidation + reflect (bank-agnostic, default-on) (#1936)
* feat(gemini): add context-cache foundation (GeminiCacheManager + opt-in call() arg)
Wraps the google-genai SDK's CachedContent API so callers can reuse a
stable (system_instruction + response_schema) prefix across many
requests. Cached input tokens are billed at a fraction of the standard
input rate, which makes workloads with a fixed-prefix / small-user-message
shape — fact extraction, structured tagging, classification — far
cheaper to run.
This PR is foundation-only: no caller is wired up yet. Default
behaviour for every existing path is unchanged because
`cached_content_name` defaults to `None` and the cache manager is
never instantiated until a follow-up wires it in.
What's here
-----------
- `gemini_cache.GeminiCacheManager`: per-process map of prefix
fingerprint → CachedContent resource name. Thread-safe via a single
asyncio.Lock. Refreshes proactively at TTL minus a safety margin.
Stable fingerprint normalisation strips auto-generated Pydantic
schema titles so dynamically-built schema classes with identical
shape hash to the same key (relevant for callers that rebuild the
schema class on every request).
- `gemini_llm.GeminiLLM.call(cached_content_name=...)`: new optional
arg. When set, the SDK config drops `system_instruction` and
`response_schema` (those live in the cache) and instead passes
`cached_content` to GenerateContentConfig. When unset, behaviour is
byte-identical to before.
- `tests/test_gemini_cache.py`: 10 unit tests covering fingerprint
stability, dict/list/Pydantic schema cases, get_or_create
caching/recreate, "minimum token count" soft-fallback, transient
SDK error soft-fallback, failed-create-doesn't-poison-cache, and
the TTL refresh boundary.
Failure handling
----------------
- Gemini rejects creates whose prefix is below the model's minimum
cacheable size with a "minimum"-style error message. The manager
catches this, logs at DEBUG, and returns None so the caller
transparently falls back to a non-cached call.
- Any other SDK error is logged at ERROR and also returns None — a
bad create never crashes a request. Callers are required to treat
None as "cache unavailable, use the normal path".
Not in this PR
--------------
- Wiring this into the fact-extraction pipeline (or any other caller)
- A metric for cached-token volume
Both will come in a focused follow-up so the foundation can land and
be reviewed independently.
* feat(gemini): wire retain fact-extraction to context cache; surface cached + thoughts tokens
Follow-on to the foundation commit on this branch — without this, the
cache manager is unreachable and the metric ignores half the cost
surface. This commit makes the change actually do something when the
flag is flipped on.
What lands
----------
1. Retain fact-extraction (engine/retain/fact_extraction.py) opts into
the cache. The system prompt and response schema are fingerprinted
and reused across calls; the user message is the only variable
part on the wire. A cache lookup failure or "prefix too small"
response from Gemini transparently falls back to the existing
uncached path — caching is a soft optimisation, never a blocker.
2. New top-level flag HINDSIGHT_API_LLM_GEMINI_PROMPT_CACHE_ENABLED
(also exposed as ``llm_gemini_prompt_cache_enabled`` on
HindsightConfig). Defaults to False so upgrade-and-do-nothing is a
no-op. Flipping to True opts every Gemini caller (currently only
retain) into context caching.
3. Two new metrics:
- hindsight.llm.tokens.cached_input — subset of input tokens billed
at the cached rate. Lets dashboards split cache-hit vs cache-miss
volume independently of total throughput.
- hindsight.llm.tokens.thoughts — reasoning tokens emitted by
Gemini 2.5+. Billed at the output rate by the provider but
invisible to candidates_token_count, so absent from output-token
dashboards today. Surfacing this is required for honest cost
attribution.
4. Provider plumbing: GeminiLLM gains a ``gemini_prompt_cache_enabled``
kwarg and a ``get_or_create_cached_prefix(...)`` accessor that lazy-
builds a GeminiCacheManager on first opt-in. LLMProvider /
create_llm_provider / ConfiguredLLMProvider pass the flag through
the standard plumbing alongside the existing safety_settings.
Verification
------------
- ``uv run ruff check`` — clean
- ``uv run pytest tests/test_gemini_cache.py`` — 12 tests including
two new integration tests that pin (a) flag-off → cache manager
never built, and (b) flag-on → manager lazy-built, second lookup
served from in-memory cache, no extra SDK call.
- ``uv run pytest tests/test_gemini_safety_settings.py`` — 13 tests
still green (no signature drift; the NoOp metrics collector was
updated alongside the real one).
Rollout
-------
- Land this commit. With the flag default-off, behaviour is identical
to today: cache code paths exist but are never reached.
- Flip the flag per-env. The metric goes non-zero on cached_input
within a few calls.
- Watch hindsight.llm.tokens.cached_input vs hindsight.llm.tokens.input
to confirm cache-hit rate.
What's deliberately NOT in this PR
----------------------------------
- Extending caching to other Gemini callers (reflect tool-call,
consolidation). Same mechanism applies — copy two lines from the
retain path. Leave for a follow-up so this lands in one focused PR.
- Cross-pod cache sharing. Each pod warms its own cache. The cost of
one extra full-price call per pod per fingerprint per TTL window is
negligible relative to steady-state savings.
* feat(gemini): extend context caching to the tool-calling reflect loop
Adds caching support to the agentic tool-loop path. The reflect agent's
``system_prompt + tools`` is stable for the duration of a single reflect
(and across reflects against the same bank), so caching them once and
reusing the cache name across every iteration of the loop collapses the
dominant input cost — the prefix repeated on every turn.
Mechanism
---------
1. ``GeminiCacheManager.fingerprint(...)`` now accepts ``tools`` and
includes the OpenAI-style tool list in the hash. A loop that swaps a
tool gets a fresh cache automatically; a loop that doesn't, hits the
cache deterministically. The tool list is serialised with sort_keys
so upstream dict-reordering doesn't cause phantom cache misses.
2. ``GeminiCacheManager.get_or_create(...)`` accepts ``tools`` and
converts the OpenAI-style entries into Gemini ``Tool`` /
``FunctionDeclaration`` shapes inside ``CreateCachedContentConfig``.
The cached prefix now holds system_instruction + tools, so the
subsequent ``call_with_tools(cached_content_name=...)`` invocation
skips resending both.
3. ``GeminiLLM.call_with_tools(...)`` gains ``cached_content_name``.
When set, ``system_instruction`` and ``tools`` are dropped from the
per-request config (the SDK rejects re-sending them alongside
``cached_content``); ``tool_config`` (mode / allowed_function_names)
stays per-request as it must.
4. ``GeminiLLM.get_or_create_cached_prefix(...)`` accepts ``tools``
and forwards them to the cache manager.
5. ``reflect/agent.py:run_reflect_agent`` looks up (or creates) the
cached prefix ONCE per reflect — right after the ``system_prompt``
and ``tools`` are built — and reuses the returned cache name across
every iteration of the agentic loop. The lookup is wrapped in a
try/except so a cache-side failure can never block a reflect.
6. ``call_with_tools`` now extracts ``cached_content_token_count``
and ``thoughts_token_count`` from ``usage_metadata`` and threads them
through ``metrics.record_llm_call`` — same as ``call()`` already
does. Without this the new ``hindsight.llm.tokens.cached_input`` and
``hindsight.llm.tokens.thoughts`` counters would never report the
reflect-side share of cached/thinking tokens.
Tests (3 new on top of the 12 from earlier on this branch)
----------------------------------------------------------
- ``test_fingerprint_changes_with_tools``: adding a tool changes the
fingerprint so a loop that adds a tool gets a fresh cache.
- ``test_fingerprint_stable_under_dict_reordering``: dict-key order in
the OpenAI-style tools list does NOT change the fingerprint.
- ``test_get_or_create_passes_tools_to_create``: the ``caches.create``
call actually receives the tools in its config — without this the
cache would silently lack the tool definitions and the first
``call_with_tools(cached_content_name=...)`` would 400.
Verification
------------
- ``uv run pytest tests/test_gemini_cache.py tests/test_gemini_safety_settings.py``
→ 28/28 pass (15 cache + 13 safety; the safety-settings suite
doubles as regression on the ``call_with_tools`` signature change).
- ``uv run ruff check`` on changed files — clean.
Behavioural envelope
--------------------
- Flag still defaults False — no caller is opted in by default.
- When flag is True, both ``retain_extract_facts`` (from the earlier
commit on this branch) and ``reflect_tool_call`` opt in.
- A cache-side failure (transient SDK error, prefix too small, manager
uninstantiated) returns None and the caller proceeds uncached. There
is no path by which caching can break reflect or retain.
* fix(gemini): make explicit prompt caching actually work end-to-end
The caching paths could never produce a cache hit:
- CreateCachedContentConfig was given response_schema/response_mime_type,
which the google-genai SDK forbids (extra_forbidden) — so every cache
create raised and soft-fell-back to an uncached call. Cache only holds
system_instruction (+ tools); response_schema is a generation-time
constraint and stays on the per-request GenerateContentConfig.
- call() dropped response_schema when a cache was in use (assuming the
schema lived in the cache — impossible). Keep it on the request; only
system_instruction moves into the cache. Structured output is preserved.
- cached_content_name was plumbed into the leaf GeminiLLM.call /
call_with_tools but NOT through the LLMProvider wrapper, so the real call
path raised "unexpected keyword argument 'cached_content_name'". Thread it
through both wrappers, forwarding only when set (other providers untouched).
With these, retain extraction caches the ~1.7k-token prefix at ~90%.
* feat(gemini): cache consolidation prefix + gate reflect cache to auto turns
- Consolidation: split the batch prompt into a stable system instruction
(mission + rules + decision guide + output format) and a per-batch user
message (facts + existing observations + capacity note). The system prefix
is byte-identical across batches in a run, so it is cached and reused; the
variable data and the per-batch response_schema stay out of the cached
surface so it never busts. Measures ~30-40% cached/input per batch (the
remainder is irreducible per-batch data).
- Reflect: Gemini rejects cached_content alongside a per-request tool_config
("CachedContent can not be used with ... tool_config"). The forced-retrieval
iterations set tool_config, so only the `auto` iterations can reference the
cache. Gate cached_content_name on tool_choice == "auto"; forced iterations
send the prefix inline.
* test(gemini): per-operation cached-ratio test + consolidation split coverage
- New tests/test_gemini_implicit_cache_ratio.py: measures cached/input token
ratio per operation (retain, reflect, consolidation) against real Gemini via
the LLM-request tracer. Dual mode: default records the implicit-cache baseline
(~0% for this access pattern); HINDSIGHT_GEMINI_EXPLICIT_CACHE=1 asserts the
explicit cache engages (cached_tokens > 0, per-op ratio floor). Gated behind
HINDSIGHT_RUN_GEMINI_EVALS=1 + a Gemini key.
- test_consolidation.py: unit test for the system/user prompt split (cacheable
byte-stable prefix; data only in the user message). Fix the inline mock LLM
callbacks to read facts from the user message(s) rather than messages[0], now
that the stable instructions are a separate system message.
* perf(consolidation): move stable observation-format note into cached prefix
The "## INPUT FORMAT" boilerplate (the explanation of the observation JSON
shape: id/text/proof_count/occurred_*/source_memories) was re-sent in every
per-batch user message. It's stable, so move it into the cached system prefix
(build_consolidation_system_prompt); the per-batch user message now carries
only the variable facts + observations data. Lifts the cached/input ratio a
couple of points without changing what the model sees.
* feat(gemini): make cached prefix bank-agnostic (mission → user message)
The retain and consolidation system prompts embedded the per-bank mission, so
each distinct mission produced a different cache fingerprint → one CachedContent
per bank. With many banks/missions that multiplies create + storage cost and
cached-object count, and makes default-on uneconomical.
Move the mission out of the cached prefix into the per-request user message:
- retain: _build_extraction_prompt_and_schema now returns a bank-agnostic prompt;
the mission rides in the user message via _retain_mission_preamble().
- consolidation: build_consolidation_system_prompt drops the mission param; the
mission moves into build_consolidation_input (the user message).
Result: the cached prefix is identical across all banks, so a single shared
CachedContent serves every bank — cardinality drops from O(missions) to O(1) per
operation, and the cost-inversion for many-low-volume-bank workloads goes away.
Behavioral note: the mission now appears in the user turn rather than the system
prompt. Validate mission-adherence against the accuracy benchmarks before flipping
the global default on. Tests updated to assert the new location + cross-bank
prefix sharing.
* test(retain): assert different missions yield one shared cache prefix
Extend the mission-relocation test to prove the payoff directly: two banks with
different retain missions produce a byte-identical system prompt → the same cache
fingerprint → a single shared CachedContent instead of one per mission.
* test(retain): cacheable prefix invariant to per-bank free-text (concise/verbose)
Parametrized over the concise and verbose modes: the cached system prompt must be
byte-identical regardless of the retain mission (any value, incl. JSON/unicode/
long text) and custom instructions, so per-bank free-text can never fragment the
shared Gemini cache. Structural toggles (causal/labels/language) are intentionally
out of scope — they legitimately partition the cache via the fingerprint.
* refactor(llm): make prompt-prefix caching a provider-interface feature
Hoist caching out of Gemini-specific duck-typing into the LLMInterface contract,
mirroring supports_batch_api():
- LLMInterface.supports_prompt_caching() -> bool (default False) and
get_or_create_cached_prefix(...) -> str | None (default None), with docs on how
explicit-cache (Gemini handle), automatic-cache (OpenAI), and inline-marker
(Anthropic cache_control) providers each map onto the hook.
- call()/call_with_tools() gain a provider-neutral cached_prefix handle (renamed
from the Gemini-flavoured cached_content_name); the wrapper forwards it only
when set so non-caching providers' signatures are untouched.
- GeminiLLM implements supports_prompt_caching(); the retain/consolidation/reflect
call sites gate on it instead of hasattr().
The engine already decides WHAT is cacheable (bank-agnostic system prefix), so a
new provider only implements HOW — e.g. OpenAI can benefit with no code (stable
leading prefix is auto-cached) or a thin override.
* docs(models): add per-provider capability table (batch API, prompt caching)
Adds a "Provider Capabilities" table to the LLM section of the models page
showing which providers support the Batch API (OpenAI/Groq/Fireworks) and
explicit prompt-prefix caching (Gemini/Vertex via CachedContent), with notes on
OpenAI's automatic prefix caching and the bank-agnostic shared-cache design.
Includes the regenerated skills/hindsight-docs mirror.
* docs(models): drive provider capability table from llmProviders.json
Replace the hand-written capability table with a data-driven one so adding a
provider stays a single-file edit. The capability flags (batchApi, promptCaching)
live in llmProviders.json — the existing single source of truth for the provider
grid and default-models table — and a new LLMProviderCapabilities component (plus
a matching renderer in generate-docs-skill.sh) renders them. Tool-calling dropped
(not differentiating here). Keep flags aligned with supports_batch_api() /
supports_prompt_caching() on the provider classes.
* feat(llm): generic, default-on prompt caching knob
Rename the Gemini-specific opt-in flag to a provider-agnostic, default-on knob,
modelled on HINDSIGHT_API_RETAIN_BATCH_ENABLED:
- HINDSIGHT_API_LLM_GEMINI_PROMPT_CACHE_ENABLED → HINDSIGHT_API_LLM_PROMPT_CACHE_ENABLED
(config field llm_gemini_prompt_cache_enabled → llm_prompt_cache_enabled, kwarg
gemini_prompt_cache_enabled → prompt_cache_enabled), single global knob (not per-op).
- DEFAULT_LLM_PROMPT_CACHE_ENABLED = True. Safe to default on: the cached prefix is
bank-agnostic (one shared cache) and creation soft-fails to an uncached call, so
it never breaks a request. Providers that don't implement caching ignore the flag.
- Resolve the flag for every provider (drop the gemini/vertexai restriction) so any
future provider that implements supports_prompt_caching() picks it up.
Docs: models page now says "on by default; disable with
HINDSIGHT_API_LLM_PROMPT_CACHE_ENABLED=false". The per-operation ratio test sets the
flag explicitly in both modes since the default is now on. Includes the regenerated
skills/hindsight-docs mirror.
* fix(gemini): fall back to uncached on a cached-request 400
A 400 from a generate request that references a CachedContent (expired/deleted
cache, cross-project mismatch, cache+tool_config incompatibility, ...) was treated
as a generic retryable error: the same cached request was retried, 400'd again,
and the whole operation failed. The soft-fallback only covered cache *creation*,
not the call that *uses* the cache.
Now, on the first 400 while a cache is in use, call()/call_with_tools():
- drop the cache and rebuild the request inline (re-send system prefix + schema/
tools) so the request still succeeds,
- invalidate the dead cache name (GeminiCacheManager.invalidate) so the next
operation recreates it instead of reusing the bad name,
- retry immediately (no backoff — it's a config switch, not a transient error).
If the uncached retry also 400s it's a genuine bad request and errors normally.
Supporting fix: system_instruction is now ALWAYS captured from the messages (it
was skipped when cached), so the fallback has the prefix to inline; the config
builder still omits it from the request while the cache carries it. New unit test
covers the 400 → uncached-retry → invalidate path. Cached success path unchanged
(real Gemini retain still 90.8%).
* fix(gemini): bound the cache-create call with a timeout
get_or_create holds the manager lock across the caches.create network call, which
correctly dedups concurrent callers (a 10-chunk retain batch produces exactly one
create, not ten). But with no timeout, a hung create would block every waiting
chunk indefinitely. Wrap the create in asyncio.wait_for (30s default, configurable
via create_timeout_seconds); on timeout it soft-fails to None and callers proceed
uncached instead of stalling the batch. Unit test covers the timeout path.
* style: ruff-format the prompt-cache config line (fixes verify-generated-files)
* test: fix consolidation-scope-parallelism mock + metrics counter count
- test_consolidation_scope_parallelism.py: the inline mock read facts from
messages[0], which is now the (cached) system message after the consolidation
prompt split — read the user message(s) instead.
- test_metrics.py: mock_meter provided 5 counter mocks but MetricsCollector now
creates 7 (the cached_input + thoughts counters), so create_counter.side_effect
ran out (StopIteration at setup). Bump both fixtures to 7.
---------
Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>
|
||
|
Octopus
|
d8dadc0a95 |
feat: upgrade MiniMax default model to M3 (#1914)
- Switch the minimax provider default from MiniMax-M2.7 to MiniMax-M3 in PROVIDER_DEFAULT_MODELS (hindsight-api-slim/hindsight_api/config.py). - Update the LiteLLM router test fixture to exercise MiniMax-M3. - Update provider docstrings and example .env entries to mention MiniMax-M3 while keeping MiniMax-M2.7 noted as a previous-generation option. - Refresh hindsight-docs (developer/models, integrations/hermes, llmProviders.json) and the docs-skill reference table to list MiniMax-M3 as the documented default. The deprecated MiniMax-M2.5 / M2.1 / M2 / M1 IDs are not referenced anywhere in the active codebase, so no removals are required. Co-authored-by: octo-patch <octo-patch@github.com> |
||
|
Evo
|
a1ee94ab3e |
docs(models): list openrouter, google, and jina-mlx in Cross-Encoder Supported Providers table (#1832)
* docs(models): list openrouter, google, jina-mlx in Cross-Encoder Supported Providers * docs(models): skills mirror — Cross-Encoder providers openrouter/google/jina-mlx |
||
|
Evo
|
9ec73e8455 |
docs(models): list openai-codex and openrouter in embeddings Supported Providers table (#1792)
* docs(models): list openai-codex and openrouter in embeddings Supported Providers table * docs(models): list openai-codex and openrouter in embeddings Supported Providers table |
||
|
Nicolò Boschi
|
374c013689 |
docs(docker): add docker-compose example for local llama.cpp sidecar (#1814)
Hindsight's published image deliberately omits llama-cpp-python to keep the image small, so setting HINDSIGHT_API_LLM_PROVIDER=llamacpp directly against ghcr.io/vectorize-io/hindsight fails with ModuleNotFoundError. Adds a docker-compose recipe that runs the official llama.cpp server container as a sidecar and points Hindsight's openai provider at it via HINDSIGHT_API_LLM_BASE_URL. Verified end-to-end against ghcr.io/ggml-org/llama.cpp:server pulling Gemma 4 E2B from HuggingFace. The named volume is mounted at /root/.cache/huggingface (where llama-server actually caches downloads) so the GGUF survives stack recreation. README documents the CPU perf reality and how to flip the relevant blocks for NVIDIA GPU acceleration. Also links the recipe from the "Built-in llama.cpp" tip in the models docs so users following the docs find the Docker setup. |
||
|
Mersad Ajanovic
|
ec49175fa3 | add zeroentropy embeddings provider (#1770) | ||
|
Nicolò Boschi
|
cb037290bb |
fix(ollama): add ollama-cloud provider and fix native API auth for cloud endpoints (#1734)
The Ollama provider's native API path (_call_ollama_native) used raw httpx without passing authentication headers, causing 401 errors when connecting to Ollama Cloud endpoints. The verify_connection call succeeded because it uses the OpenAI-compatible path (AsyncOpenAI client) which includes the API key, but structured output calls failed. - Pass Authorization Bearer header in native Ollama httpx calls when a real API key is provided (not the "local" dummy fallback) - Add ollama-cloud as a first-class provider that uses the OpenAI-compatible path exclusively (no native /api/chat fallback), requires an API key, and defaults to https://ollama.com/v1 Closes #1559 |
||
|
quicklyfast
|
b83bb87ddd |
feat(reranker): support alibaba qwen3-rerank (#1501)
* feat(reranker): support alibaba qwen3-rerank * feat(reranker): support alibaba qwen3-rerank * Fix formatting of Alibaba API key export line |
||
|
Teven Feng
|
c29c76e3fa | feat: add opencode-go LLM provider (#1652) | ||
|
Nicolò Boschi
|
98f33cbf6c |
feat(api): add litellmrouter provider for LLM fallback chains (#1537)
* feat(api): add litellmrouter provider for LLM fallback chains Closes #1464. New "litellmrouter" provider wraps LiteLLM Router with ordered fallback across a configurable chain of deployments. On transient errors (rate-limit, timeout, 5xx) the Router falls back to the next deployment in declared order; auth errors (401/403) are not retried so a misconfigured key cannot silently cascade through the chain. Configuration is provider-scoped (one-word LITELLMROUTER namespace to avoid clashing with the existing LITELLM_* settings used by the embeddings/reranker layers): HINDSIGHT_API_LLM_PROVIDER=litellmrouter HINDSIGHT_API_LLM_LITELLMROUTER_CHAIN=<json list of deployments> Per-operation chains are supported via the same pattern that already exists for retain/reflect/consolidation: HINDSIGHT_API_RETAIN_LLM_LITELLMROUTER_CHAIN=... HINDSIGHT_API_REFLECT_LLM_LITELLMROUTER_CHAIN=... HINDSIGHT_API_CONSOLIDATION_LLM_LITELLMROUTER_CHAIN=... Each per-op chain falls back to the default chain when unset, mirroring the existing per-op provider/model overrides. Chain entries are tagged as credential fields and are never exposed via the bank-config API. Batch APIs are intentionally unsupported in router mode; users that need batch retain should configure a single provider. * refactor(api): dedup litellmrouter on top of LiteLLMLLM, accept arbitrary chain keys, add CI matrix entry The retry/parse/metrics loop in LiteLLMRouterLLM was a near-verbatim copy of LiteLLMLLM. Extract three small hooks on the base class (_acompletion, _resolve_completion_model, _stage_label) and have the Router provider inherit + override only what differs. Drop strict validation of chain entries. The parser now requires only 'provider' and 'model'; everything else passes through to LiteLLM Router unchanged. Top-level keys (rpm, tpm, weight, model_info, ...) flow to the deployment record; an optional 'litellm_params' sub-object merges into the inner params dict. Documented and tested. Add a litellmrouter row to the LLM acceptance matrix using a single OpenAI deployment in the chain. The chain JSON is built from secrets in a dedicated step and masked in logs before being written to GITHUB_ENV. * refactor(api): pure pass-through to litellm.Router, drop translation layer Replace the chain-with-Hindsight-shape API with a thin pass-through to litellm.Router. The HINDSIGHT_API_LLM_LITELLMROUTER_CONFIG env var is now a JSON object forwarded verbatim to Router(**config). Hindsight's only imposed rules: model_list is non-empty, each entry has a model_name, and requests route against the first entry's model_name. This removes _LITELLM_PROVIDER_PREFIX (provider→prefix translation), _build_model_list (flat→nested rewrite), and _build_fallbacks (auto-wired ordered fallback). Users now write LiteLLM-native configs and pick their own routing strategy — ordered fallback via 'fallbacks', load-balancing via shared model_name + 'routing_strategy', rate-limit awareness via rpm/ tpm, and so on. The docs link to LiteLLM's reference rather than recapitulating it. Renames: ENV_LLM_LITELLMROUTER_CHAIN -> ENV_LLM_LITELLMROUTER_CONFIG llm_litellmrouter_chain -> llm_litellmrouter_config _parse_llm_router_chain -> _parse_llm_router_config LLMProvider(litellmrouter_chain=) -> LLMProvider(litellmrouter_config=) The dataclass fields change shape from list[dict] to dict (JSON object). Net reduction across the touched files: ~165 lines. * docs: regenerate hindsight-docs skill from updated configuration.md * refactor(api): drop all shape validation on litellmrouter config, use fixed 'default' entrypoint The previous version still inspected the user's config in two places: the parser checked model_list/model_name shape, and __init__ pulled primary_model_name out of model_list[0]. Both are gone. The parser now only verifies the env var is parseable JSON. Whatever the user supplies — dict, list, missing keys, weird shapes — flows through. LiteLLM Router is authoritative about the shape and raises its own errors at construction time if something's wrong. The provider no longer extracts a 'primary' name from the input. Instead it always issues completions against model_name='default' — the single Hindsight-imposed convention. Users put one entry with that name in their model_list as the entrypoint and use any names they want for fallback/load-balance/weighted-pool members. This avoids both pre- validation footguns and any dependence on Router's internal API (model_names, model_list attributes) that could shift between versions. Docs and tests updated to match. The CI matrix already used 'default'. * docs: regenerate hindsight-docs skill * ci(test): cap retain max_completion_tokens for litellmrouter matrix row gpt-4.1-nano caps OpenAI completion at 32768 tokens, but Hindsight's default DEFAULT_RETAIN_MAX_COMPLETION_TOKENS is 64000. The 'openai' matrix row passes because OpenAICompatibleLLM has model-specific token capping; LiteLLMLLM (and the new LiteLLMRouterLLM by inheritance) don't. That's a pre-existing limitation orthogonal to this PR — the cap-aware behaviour lives in OpenAICompatibleLLM and intentionally doesn't apply to LiteLLM-routed calls. Lower retain max_completion_tokens via env in the litellmrouter job so CI exercises the Router path end-to-end instead of dying on a provider-side BadRequestError that's not the thing we're testing. * fix(api): cap LiteLLM-routed max_completion_tokens to model registry limit Hindsight defaults retain_max_completion_tokens to 64000 — fine for high-capacity models, but breaks against models with smaller caps (gpt-4.1-nano: 32768; gpt-4o-mini: 16384). OpenAICompatibleLLM already caps via a hardcoded string-match table; LiteLLMLLM and the new Router provider didn't, so a default Hindsight install pointed at a small model would fail with provider BadRequestError. Cap pre-emptively using LiteLLM's own per-model registry (litellm.get_max_tokens). For LiteLLMLLM the cap is self.model. For LiteLLMRouterLLM the cap is the min across all configured deployments, computed once at __init__ — this way a single max_completion_tokens value works no matter which deployment Router picks (primary, fallback, weighted-pool member). Unknown models contribute no cap. Reverts the temporary CI workaround that lowered HINDSIGHT_API_RETAIN_ MAX_COMPLETION_TOKENS=32000 for the litellmrouter row — Hindsight should work out of the box. * docs: shorten litellmrouter config section, add models.mdx pointer Move the discoverability pointer into models.mdx alongside the existing LiteLLM tip, where users browsing for model options will find it. Strip the configuration page entry to its essentials: env-var table, one ordered-fallback example, and the three short caveats. Defer routing details to LiteLLM's docs rather than recapitulating them. |
||
|
Nicolò Boschi
|
3f08b115f8 |
docs(zai): document z.ai provider and add default model (#1532)
* docs(zai): document z.ai provider and add default model Follow-up to #1529. Adds z.ai (Zhipu GLM series) to the provider list, example blocks, default-model table, and `.env.example`. Also wires `zai` into `PROVIDER_DEFAULT_MODELS` so the new docs entry actually matches what the engine resolves when only the provider is set. * docs(zai): use glm-4.5-flash as default (free tier) glm-4.5-air requires a paid balance on z.ai; flash is on the free tier and works as a sensible default. Air is still listed in the example as the paid-tier upgrade. |
||
|
Nicolò Boschi
|
a81892096a |
fix(config): default openai-codex model to gpt-5.4 (#1357)
* fix(config): default openai-codex model to gpt-5.4 gpt-5.2-codex was deprecated by OpenAI and is rejected by the Codex API on current ChatGPT Pro tiers. Switch the default to gpt-5.4, which is in the active model list. Closes #1344 * fix(config): use gpt-5.4-mini as openai-codex default |
||
|
Evo
|
4282e8423a |
docs(models): document litellm-sdk embeddings provider (#1336)
* docs(models): document litellm-sdk embeddings provider * docs(models): mirror litellm-sdk embeddings provider in sidecar |
||
|
Nicolò Boschi
|
461b00d4d9 |
fix(llm): omit tool_choice="auto" and add deepseek as first-class provider (#1294)
* fix(llm): omit tool_choice="auto" and add deepseek as first-class provider
DeepSeek's reasoner pathway (which deepseek-v4-flash enters by default
with thinking mode) returns HTTP 400 for any tool_choice value, including
"auto". Since omitting tool_choice is semantically equivalent to "auto"
per the OpenAI API spec, we now omit it whenever the caller passes "auto",
which fixes reflect for deepseek-v4-flash without changing behaviour for
compliant providers.
Also promotes DeepSeek to a first-class provider: provider="deepseek"
auto-configures base_url=https://api.deepseek.com and the default model
to deepseek-v4-flash. Documented in configuration.md and .env.example.
* docs(deepseek): add to LLMProvidersGrid, default-models table, and config examples
The LLMProvidersGrid component on the Models page is the canonical visual
list of supported LLM providers; it was missing DeepSeek. Also add it to
the provider default-models table and the per-provider configuration
example block in models.mdx so the page is internally consistent.
* docs: single-source-of-truth for LLM providers (data file + table component)
Adds hindsight-docs/src/data/llmProviders.tsx as the canonical list of
supported providers with id, label, icon, and default model. Both
LLMProvidersGrid (icon grid on the Models page) and the new
LLMProvidersTable component (used in models.mdx for the default-models
table) consume it, so adding a provider now means editing one file
instead of three.
While converting, also added the providers that were missing from the
icon grid: Vertex AI, OpenAI Codex, Claude Code, OpenRouter.
* fix(docs-skill): render LLM provider grid + table in agent skill mirror
The agent-facing skill at skills/hindsight-docs/ is plain markdown — the
MDX-to-MD converter in scripts/generate-docs-skill.sh was leaving
<LLMProvidersTable /> and <LLMProvidersGrid /> as literal JSX, breaking
the verify-generated-files CI check and hiding the supported-providers
data from agents that rely on the skill.
Move the provider data out of llmProviders.tsx into llmProviders.json so
both the React components and the Python skill generator read from the
same source. Teach the converter to render <LLMProvidersTable /> as a
markdown table and <LLMProvidersGrid /> as a bullet list, sourced from
that JSON. Adding a provider is still one-file: edit llmProviders.json.
* chore(pipecat): apply ruff format
Files added in
|
||
|
Nicolò Boschi
|
d0b2ab9ad2 |
feat(reranker): add SiliconFlow provider; share Cohere-compatible HTTP client (#1019)
* feat(reranker): add SiliconFlow provider and share Cohere-compatible HTTP client Closes #859. Adds a `siliconflow` reranker provider for SiliconFlow's Cohere-compatible `/rerank` endpoint, and refactors ZeroEntropy plus the Cohere custom-base_url code path onto a shared `_CohereCompatibleRerankClient`. Setting `HINDSIGHT_API_RERANKER_COHERE_BASE_URL` now routes the `cohere` provider through the same HTTP client, making it a generic entry point for any Cohere-compatible rerank host (Azure AI Foundry, Jina, Voyage, self-hosted BGE, ...). * fixup: update cohere tests for shared HTTP client + regen docs skill + ruff format |
||
|
Nicolò Boschi
|
61a8014f9d |
docs: 0.5.0 release notes, changelog, and blog post (#907)
* docs: add 0.5.0 release notes and changelog * docs: include all commits since v0.4.22 and add recall perf to blog * docs: include all commits since v0.4.22 and add recall perf to blog * docs: add openrouter default model to provider table * docs: reorder blog sections, fix code snippets, remove paperclip * docs: add hermes integration docs link * docs: fix broken anchor in blog post TOC |
||
|
Nicolò Boschi
|
f74b577e02 |
feat: add built-in llama.cpp LLM provider for local inference (#933)
* feat: add built-in llama.cpp LLM provider for fully local inference Add `llamacpp` as a new LLM provider that manages a llama-cpp-python server subprocess. Auto-downloads Gemma 4 E2B Q4_K_M (~3.5 GB) on first use and runs inference locally via Metal/CUDA with no external services needed. - New provider: `HINDSIGHT_API_LLM_PROVIDER=llamacpp` - Singleton server shared across retain/reflect/consolidation - Configurable: model path, GPU layers, context size, grammar enforcement - User-extensible via `HINDSIGHT_API_LLAMACPP_EXTRA_ARGS` - Flash attention + prompt caching enabled by default - LLM provider cleanup on shutdown (stops subprocess) - hindsight-embed: `--ui` flag on `daemon start`, removed FORCE_CPU on macOS - Docs: configuration.md, models.mdx, providers grid updated * chore: regenerate docs skill and update lockfile for local-llm dep |
||
|
Volodymyr Prypeshniuk
|
07de798c3b |
feat(google): add support for google embeddings and reranker (#863)
* Add support for google embeddings gemini/vertex and google reranker via vertex search api * Add reference docs |
||
|
Nicolò Boschi
|
02b0f7799d |
docs: add Volcano Engine as supported LLM provider (#715)
* docs: add Volcano Engine as supported LLM provider Follow-up to #714. Add Volcano Engine (ByteDance) to the documentation: - LLM providers grid component - Provider list in configuration docs - Provider example with base URL and default model - Default model table in models page * chore: regenerate docs skill references |
||
|
Nicolò Boschi
|
db70fdbe5e |
feat: add LiteLLM LLM provider for Bedrock and 100+ providers (#679)
* feat: add LiteLLM LLM provider for Bedrock and 100+ providers Add a new `litellm` LLM provider that uses the LiteLLM SDK for chat completions and tool calling, enabling AWS Bedrock and 100+ other providers for Hindsight's core engine (retain, recall, reflect). - New LiteLLMLLM provider in engine/providers/litellm_llm.py - Registered in factory, valid providers list, and no-api-key set - Refactored API key validation to use requires_api_key() helper - Added boto3 dependency for Bedrock auth - Updated docs: configuration, models, monitoring, providers grid * feat: add bedrock as first-class LLM provider alias Add `bedrock` as a dedicated provider name that auto-prepends the `bedrock/` prefix to model names and delegates to LiteLLMLLM under the hood. This makes Bedrock support more discoverable — users set `HINDSIGHT_API_LLM_PROVIDER=bedrock` with plain Bedrock model IDs. * test: add Bedrock to CI provider tests - Add bedrock/us.amazon.nova-lite-v1:0 to MODEL_MATRIX in test_llm_provider.py - Add AWS credential check in should_skip_provider() - Pass AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION_NAME secrets to test-api job - Update default bedrock model to amazon.nova-2-lite-v1:0 * fix: regenerate docs skill files and bump memory test timeout - Regenerate skills/hindsight-docs references after docs changes - Bump test_llm_provider_memory_operations timeout to 600s for slower providers like Bedrock via LiteLLM * test: skip bedrock lite models in memory operations test Nova Lite has a 10K output token limit which is too low for fact extraction (requires 64K). The api_methods test (completion, tools, structured output) already validates the provider works correctly. * test: use Nova Pro for bedrock CI tests to cover full memory pipeline Nova Lite only supports 10K output tokens, too low for fact extraction. Switch to Nova Pro which supports the full 64K output needed for retain/reflect operations. This ensures bedrock is tested on all Hindsight functionalities, not just basic API methods. * test: switch bedrock CI to Nova 2 Lite (supports 64K output tokens) Nova v1 models (Pro, Lite) have a 10K output token limit which is too low for fact extraction. Nova 2 Lite supports 64K+ output tokens, enabling full memory pipeline testing (retain + reflect). |
||
|
Octopus
|
1f1462a5f6 |
feat: upgrade MiniMax default model from M2.5 to M2.7 (#606)
* feat: upgrade MiniMax default model from M2.5 to M2.7 MiniMax has released MiniMax-M2.7, their latest model with a 1M context window (up from 204K). This updates the default model across config, docs, and examples. M2.5 remains fully compatible for users who prefer it. - Update PROVIDER_DEFAULT_MODELS to MiniMax-M2.7 - Update .env.example and documentation references - Add test_minimax_provider.py with M2.7 and backward compat tests * chore: remove test file per review feedback --------- Co-authored-by: PR Bot <pr-bot@minimaxi.com> |
||
|
Nicolò Boschi
|
06200f1752 |
docs: revamp sidebar with icon grids and language support (#563)
* docs: revamp sidebar with icon grid components and language support - Merge Clients and Integrations sections into the developer sidebar (removed top-level SDKs navbar item) - Reorder sidebar: Architecture → API → Clients → Integrations → Hosting - Unify icon system using react-icons (LuXxx/SiXxx) via customProps.icon - Add uppercase section titles with increased spacing and reduced indentation - Rename Node.js → "JavaScript / TypeScript" with TypeScript icon - Add reusable IconGrid and SupportedGrids components (ClientsGrid, IntegrationsGrid, LLMProvidersGrid) - Use grids in FAQ, Models, Overview, and Quick Start pages - Convert developer/index.md, models.md, faq.md to MDX for JSX support * fix: use inline style for label color to prevent link color inheritance * fix: label visibility and rename JavaScript/TypeScript to TypeScript * feat: add HTTP client to grid and OpenAI Compatible to LLM providers grid |