mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Nicolò Boschi
2161c4e815
* ci: add pre-commit hook to keep skills/hindsight-docs in sync The CI verify-generated-files job has been failing on ~82% of recent runs because PRs touch hindsight-docs/src/pages/changelog/ or hindsight-docs/static/openapi.json without re-running ./scripts/generate-docs-skill.sh, leaving the committed skills/hindsight-docs/references/ copy stale. Catch the drift locally instead. The hook regenerates and, if the working tree diverges from the index after regen, fails the commit with a clear message pointing the author at `git add skills/hindsight-docs/`. The pre-commit dispatcher (.githooks/pre-commit) already iterates every *.sh in scripts/hooks/, so the new file is picked up automatically. * fix(retain): stop mutating caller-provided content dicts PR #1398 (memory pressure) added an in-place pop of the "content" key on contents_dicts after building combined_content, to release per-item strings the engine no longer needs. Because the engine forwarded the caller's dict objects all the way through (memory_engine → _retain_batch_async_internal → orchestrator.retain_batch), the pop reached back through the same references and stripped the key from the caller's input. Any code path that holds onto the contents list after retain_batch_async returns then trips KeyError: 'content'. This is what was making test_extensions.py::TestOperationHooksParameters:: test_retain_pre_hook_receives_all_parameters fail intermittently on main (the streaming path triggers the pop; non-streaming paths skip it). Fix: - memory_engine.py: take an engine-owned shallow copy of contents after the validator hook so the orchestrator can mutate freely without leaking to the caller. Strings are shared by reference, so the copy adds only ~150 bytes of dict overhead per item — negligible vs the multi-MB strings. - orchestrator.py (_streaming_retain_batch): clear combined_content immediately after handle_document_tracking / upsert_document_metadata in all three first-batch paths (no-facts skip, mini-batch DB work, post-loop fallback). Once tracking persists the document, nothing reads combined_content again, so releasing it shrinks the lifetime of the per-document text from "until function returns" to "until DB write completes" — recovering the bulk of #1398's memory savings without the caller-mutation side effect. nonlocal declarations on _process_db_batch and _run_mini_batch_db_work are required because Python infers combined_content as local once any branch assigns to it. Memory profile vs PR #1398: - #1398 benchmark shape (caller releases its reference at call time): identical sustained, brief 2x peak during the combined_content + per-item-strings overlap window before tracking completes. Other PR #1398 savings (chunks, batch lists, sanitized_content) untouched. - HTTP / FastAPI callers (request body holds strings until the handler returns): no observable change — those strings were going to live through the request anyway.
23 lines
802 B
Bash
Executable File
23 lines
802 B
Bash
Executable File
#!/bin/bash
|
|
# Pre-commit hook: keep skills/hindsight-docs/ in sync with hindsight-docs/
|
|
# sources. CI's verify-generated-files job regenerates and fails the build on
|
|
# drift; this hook catches it locally first so PR authors don't ship a commit
|
|
# that needs a follow-up "regenerate docs skill" patch.
|
|
|
|
set -e
|
|
|
|
REPO_ROOT="$(git rev-parse --show-toplevel)"
|
|
|
|
"$REPO_ROOT/scripts/generate-docs-skill.sh" >/dev/null
|
|
|
|
if ! git diff --quiet -- skills/hindsight-docs/; then
|
|
echo ""
|
|
echo " ❌ skills/hindsight-docs/ was out of sync with hindsight-docs/ sources."
|
|
echo " The generator just refreshed it — stage the regen and re-commit:"
|
|
echo " git add skills/hindsight-docs/"
|
|
echo ""
|
|
git diff --stat -- skills/hindsight-docs/ | sed 's/^/ /'
|
|
echo ""
|
|
exit 1
|
|
fi
|