mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Nicolò Boschi
1d5f852809
* test(system): blackbox system-test suite driven only through the public API The api-slim suite is ~500 files, each covering one mechanism, which leaves composition bugs — consolidation wiping facts, a delta refresh missing a backdated window, a transfer dropping evidence — with nothing watching them. This adds the layer that does: tests that drive a real `hindsight-api` process over HTTP through the published Python client, with no engine access and no SQL. Determinism comes from a stub server implementing the OpenAI chat-completions/embeddings APIs and a Cohere-compatible rerank endpoint. The server under test is pointed at it with ordinary environment variables (`HINDSIGHT_API_LLM_BASE_URL`, `..._EMBEDDINGS_OPENAI_BASE_URL`, `..._RERANKER_SILICONFLOW_BASE_URL`), so no production code changes are needed and the real provider transport — the OpenAI client, JSON repair, retries, structured output — is exercised for real rather than replaced by a fake. Design decisions worth knowing: - Rules match on a named pipeline step, not the request body. Nothing on the wire identifies the caller (the json_schema name is the constant "response", and the soft json_object path sends no schema at all), so a step is a short anchor phrase from its prompt — owned by `steps.py` so a prompt edit is one line, not thirty red tests. - An unmatched call fails the test and prints the rule to paste in. `MockLLM` synthesizes plausible facts from its input when it doesn't recognize a call, which is why tests using it pass without proving anything. - Requests are validated strictly. A fake you control drifts permissive, and the provider bugs that have actually hurt here (Bedrock rejecting response_format, Azure 400ing on prompt_cache_key) are all "the provider refused our request". - Background work stays on. Consolidation runs in the worker after retain returns; tests wait for it via the operations API rather than disabling it, because that asynchronous half is where the composition bugs live. - The server runs from a scratch dir holding an empty `.env`, since a discovered `.env` deliberately overrides the ambient environment (#2961) and would otherwise silently replace the whole test configuration. CI (`test-system`) needs no provider secrets, so unlike every `test-api` job it also runs on fork PRs — and with embeddings and reranking both stubbed, nothing loads sentence-transformers, so it skips torch and the HuggingFace cache too. * test(system): pin the whole recall payload, not just a keyword With the LLM, embedder and reranker all stubbed, a recall is a pure function of its input — so assert it as one. Ranking order, the rendered fact text, the document/chunk identity composite, the temporal fields, the empty envelope sections, and each of the four scores are now pinned. The three retrieval components are reproducible to the bit across runs; only `final` drifts (~1e-9), because it folds in recency measured against wall-clock now, so it gets a tolerance and a comment saying why. The previous assertion ("Berlin appears somewhere") passed just as happily with fusion inverted, the reranker contributing nothing, or the temporal fields silently stopping being parsed. * ci: skip the two unconditional jobs for a system-tests-only change Iterating on hindsight-system-tests/ costs ~6 minutes of CI against a 36-second test. Of the 96 jobs, 91 already skip for a change confined to that package; the remainder were build-docs (2.2 min) and verify-generated-files (3.6 min), both deliberately unconditional. Both now hang off a new `outside-system-tests` filter rather than a positive one, so they still run for every other change in the repo and step aside only for a PR touching nothing but the system-test package — which no generated file is produced from and the docs site never reads. The filter is a lone negation on purpose: paths-filter builds one matcher per pattern and ORs them, so the natural-looking ['**', '!dir/**'] pair matches every file and leaves the filter permanently true. Verified against picomatch directly before committing.