15 Commits

Author	SHA1	Message	Date
Sanderhoff-alt	0478c09c41	perf(retain): optimize embedding_to_pgvector via zero-copy orjson (#3815) ... Retain and import paths convert float embeddings to pgvector vector literals for asyncpg binding (insert_facts_batch, compute_semantic_links_within_batch, update_memory_unit_embedding). The baseline implementation used a Python generator: "[" + ",".join(repr(float(value)) for value in embedding) + "]" For 500 facts (768,000 floats at 1536d), this allocated 768,000 PyFloat objects and 768,000 PyUnicode strings, taking ~220 ms CPU time and ~15 MB heap memory. The optimized implementation leverages np.frombuffer on PackedEmbedding (array('f')) for zero-copy buffer views, and orjson.OPT_SERIALIZE_NUMPY to format floats directly into the output byte buffer using Rust Ryu SIMD: * Promotes numpy to explicit direct dependency across hindsight-api and dev; * Formats shortest float32 representation (byte-identical Postgres storage); * Isolates _repr_literal fallback helper for non-finite and non-float inputs; * Unifies _dumps_or_repr_fallback with single payload parameter and no option branching; * Streamlines embedding_to_pgvector into a concise polymorphic dispatcher; * Seamlessly supports array('f'), list[float], tuple, ndarray, and str. Measured on Apple Silicon via vector-serialization-bench (best of 5 repeats): workload baseline prod speedup peak alloc single_bge_384 (1x 384d) 0.136 ms 0.040 ms 3.4x 36K -> 13K single_openai_1536 (1x 1536d) 0.489 ms 0.085 ms 5.8x 144K -> 50K batch_20_gemini_768 (20x 768d) 4.580 ms 0.514 ms 8.9x 356K -> 185K batch_200_openai_1536 (200x 1536d) 92.64 ms 9.45 ms 9.8x 6.1M -> 3.4M batch_500_large_doc (500x 1536d) 221.78 ms 23.84 ms 9.3x 15.0M -> 8.4M batch_200_raw_list (200x 1536d) 87.70 ms 11.48 ms 7.6x 6.1M -> 6.0M Throughput increased from 3.3 Mfloat/s to 32.5 Mfloat/s (~9.8x speedup on typical retain batches), with ~44% peak memory reduction on 500-fact batches. Includes unit tests in test_packed_embeddings.py covering bit-identical float32 roundtrips, custom non-serializable objects, non-f array fallthrough, tuples, ndarrays, and non-finites.	2026-08-31 12:50:04 +02:00
Nicolò Boschi	9fcb7ca7ac	perf(tokenizer): replace tiktoken with quicktok and default to o200k_base (#3788) ... Token counting is on the hot path of both retain and recall. Recall counts once per candidate fact, per candidate chunk, per source fact and per reranker document; retain counts whole documents. All of it went through `len(encoding.encode(text))` — which builds a full Python list of ids only to take its length. Measured on this repo's own text with the microbenchmark added here, against tiktoken 0.12.0 on a 14-core M-series, both on o200k_base: workload tiktoken quicktok speedup peak alloc 200 ranked facts 4.39 ms 0.75 ms 5.8x 3 KiB -> 1 KiB 500 source facts 6.92 ms 1.12 ms 6.2x 2 KiB -> 1 KiB 50 candidate chunks 12.02 ms 1.57 ms 7.7x 21 KiB -> 1 KiB 100 reranker documents 10.84 ms 1.55 ms 7.0x 12 KiB -> 1 KiB one 77k-token document 36.08 ms 3.00 ms 12.0x 2.8 MB -> 1 KiB Summed across the four counting stages one recall runs: 34.2 ms -> 5.0 ms. Three things make this worth a dependency change rather than a micro-opt: * `count()` returns an int without materialising the ids, so counting a large document allocates nothing. tiktoken has no count-only API — `encode_to_numpy` reaches the same 1 KiB but none of the speed, and is measured here too. * ids are byte-identical to tiktoken's; the benchmark asserts that on adversarial inputs before it times anything. * the vocabularies ship inside the wheel, so nothing is downloaded at runtime. That removes the tiktoken pre-download from the Docker build (both stages) and from scripts/dev/setup.sh — air-gapped deployments no longer need it baked in. The dependency risk is maintenance, not correctness, so it is contained: engine/token_encoding.py is the only module that imports quicktok, every call site routes through get_token_encoding() / count_tokens(), and its one dependency (numpy) was already in the tree. Replacing it means rewriting that file and nothing else. This also removes the last direct tokenizer import that had escaped the seam (`__import__("tiktoken")` in reflect/prompts.py). Removes #3756's workaround. count_tokens_windowed existed to bound the memory of counting a large retain body, encoding a megabyte at a time and accepting an approximate answer because a fixed character cut can split a token. count() allocates nothing at any size AND is exact, so the windowing, its helpers and its six call sites are gone — those callers now get an exact count. The test file keeps the property that made #3756 worth fixing (allocation does not track the input), now asserted against count_tokens itself. Default encoding moves to o200k_base, selectable with HINDSIGHT_API_TOKENIZER_ENCODING (server-level: budgets are only comparable between banks if they are all counted the same way). o200k_base is what current OpenAI models tokenize with. On English and code it counts within a fraction of a percent of cl100k_base, but on non-Latin scripts it is far closer to what a model actually charges — a mixed-script line with emoji is 19 tokens under cl100k_base and 13 under o200k_base. Since these counts back budgets that stand in for a context window, the closer vocabulary is the more honest one. Set cl100k_base to reproduce the previous counts exactly. Call sites that only need a number now call count(); the ones that need ids (query truncation, chunk truncation, reranker truncation, prompt fitting) still encode, but only after a count shows the text does not fit. The chunk-budget loop also stopped encoding each oversized chunk twice.	2026-08-25 15:02:26 +02:00
Nicolò Boschi	fe5c25d64c	fix(mental-models): store delta documents as verbatim markdown blocks (#3361, #3273) (#3622) ... * fix(mental-models): store delta documents as verbatim markdown blocks (#3361, #3273) A knowledge page could come back with a whole table welded onto one line, in sections no delta operation had named, and never recover. The delta refresh was blamed, but the damage was done one refresh earlier. `structured_content` was only the source of truth on the delta leg. The full leg stored the LLM candidate markdown verbatim in `content` while deriving the structure from it with `parse_markdown`, a typed-block parser that flattened anything its union could not express -- nested lists, list continuation lines, blockquotes, hard line breaks, horizontal rules, HTML, indented code, table alignment, a table row missing an outer pipe. 15 of 16 common constructs lost information, and every loss was a fixed point, so no later refresh could undo it. The two columns disagreed by construction, and the next delta refresh published the degraded one over the whole document. Schema v2 stores each block as a verbatim markdown fragment plus an id. Nothing parses a table, so nothing can flatten one. `parse_markdown` is deleted; `split_markdown` replaces it and recognises only ATX headings and blank lines, both fence-aware, which makes it lossless -- asserted as a property over a 26-case corpus of exactly the constructs v1 destroyed. Blocks are now addressed by id rather than by index (#3273). An index has to be counted by the model, and an off-by-one lands in range, silently overwrites an unrelated block, and is recorded as a success. An id is copied, not derived; one that does not resolve -- or that names a block in a different section -- is skipped and reported. Operation payloads are plain markdown strings, so the model no longer has to emit a typed block union either. `content` is now always the render of `structured_content`, on both legs, so the two can no longer drift apart. Also here: - `parse_llm_json` escapes `\n \r \t \b \f` inside JSON string values instead of blanking them. A model writing a markdown table into a string often forgets to escape its line breaks, and replacing them with spaces delivered the table already collapsed. Other control characters keep the previous treatment. - A model that adds a table row as its own block would render a broken table, so the prompt asks for `replace_block` and bare rows landing directly after a table are folded into it. - Migration `d1e2f3a4b5c6` clears v1 blobs. They are a lossy projection of the row's own `content`, so there is nothing to convert: the next refresh re-imports the structure from the markdown, losslessly. `content` is untouched. - The Gemini eval fixture pinned `gemini-2.0-flash`, which the provider has retired (404), so the whole eval class was dead. Verified against a real model: `test_document_survives_many_delta_rounds_intact` runs five delta rounds feeding one new fact each, asserting after every round that content is the render of the structure, that no line welds a table separator to other cells, that sections no operation named are byte-identical, and that a section never named across the whole run is unchanged at the end. Run four times, 20 real rounds, green. It is what caught the orphan table row. * fix(mental-models): write the structure whenever content is written `create_mental_model(content=...)` and `create_knowledge_page` inserted the markdown and left `structured_content` NULL, so a model authored as markdown had no structure until its first delta refresh -- and that refresh was then the one to derive it, silently reshaping a document nobody had asked it to touch. `update_mental_model(content=...)` was worse: it could set the markdown while leaving the *previous* document's structure in place, so the two columns described different documents until the next refresh papered over it. Nothing enforced the pairing; the refresh path just happened to pass both. Both writes now go through `canonical_document()`, which splits the authored markdown and hands back the structure together with its render. The insert stores both, and the update derives the structure whenever a caller supplies content without one. A refresh still passes both explicitly -- there the structure is authoritative and the markdown is already its render -- and is untouched. The derivation is hoisted above the embedding computation so the embedding, the history snapshot and the UPDATE all see one text. `content` is therefore the render of `structured_content` from the first byte rather than from the first refresh, which is what the assertion churn in this commit is: authored markdown now comes back canonicalised, so a document that was stored as "v1" reads back as "v1\n". Also makes the migration test rerunnable: a pg0 instance survives between runs and alembic will not replay a migration on a DB already stamped past it, so seeding into it would have left the rows untouched and the test asserting nothing. * feat(reflect): answer with a document, render the markdown from it The mental-model refresh asked the agent for markdown and worked out the document's structure by reading that markdown back. Reading LLM markdown back is where #3361 destroyed tables, and it is unnecessary here: the refresh knows it is producing a document, so it can ask for one. `done()` gains a document mode. Instead of an `answer` string it takes a `document` -- an ordered list of sections, each with a heading, a level and its blocks -- and the markdown that gets stored and shown is rendered from it. The model no longer writes the markdown that gets persisted, and nothing parses markdown to find out what the model meant. `answer` is not merely discouraged in that mode, it is absent from the schema, so there is no escape hatch back to prose. The shape is deliberately flat -- an array of sections holding arrays of block strings, no unions. A tool schema goes to the provider verbatim and not every provider accepts `oneOf` (Gemini rejects it), and a shape the model can fill without thinking is one it fills correctly. `document_from_sections` is tolerant, because a tool call is still model output: a missing heading, a `##` the model prefixed anyway, an out-of-range level or a non-string block is coerced rather than rejected. A block holding several blank-line-separated fragments is split into one block each, so the document keeps the granularity delta operations address even when the model packs a whole section into one string. Downstream is unchanged: the rendered markdown still flows on as `text`, so structured-output extraction, the length rewrite and the HTTP response all behave as before. The one place the two could drift is the length rewrite, which edits the text after the fact -- there the structure is re-derived from the rewritten markdown, which is lossless and keeps the invariant that the stored text is exactly what the stored structure renders. Splitting markdown is now only an import path: a model created from authored markdown, a restored export, or a run that produced plain text anyway (a provider that dropped the tool call, the iteration-limit answer). Verified against a real model: all five `hs_llm_core` refresh evals pass with the agent emitting structure, including the five-round stability run. The ordered list that a previous run had rewritten now survives untouched, and the new table row lands inside the table rather than beside it. * test(benchmarks): compare two builds on how a document survives being edited Neither half of "is the new pipeline better" was measurable before this. The unit tests prove the mechanics in isolation and the refresh evals prove one build behaves, but nothing compared a build against another one on the thing that actually broke: a document rewritten by an LLM over and over. The harness talks HTTP only, so the same code drives a server built from any revision. That is what makes an A/B possible without a feature flag inside the code under test: run it once per build, compare the two artifacts. Every document from every round is stored, and metrics are recomputed at comparison time, so sharpening a metric costs nothing instead of another few hundred LLM calls. Two things it measures, deliberately separately. Structural, no LLM: collapsed tables (the detector from #3361), rows, nesting, hard breaks, fences and quotes lost, sections that drifted with no operation naming them, plus pipeline health and latency. Damage is counted only in sections no operation named -- a refresh that rewrites a section it targeted may legitimately restructure it, and scoring that as corruption would punish the model for doing its job. Content, judged: each round declares what must be true afterwards and what must no longer be stated, checked one claim at a time so a miss points at a fact rather than at a score; plus a blind pairwise preference between the two builds' final documents, judged in both orderings so position bias cannot decide it. Three cases, and the third is the point. `api-reference` carries every fragile construct; `onboarding-playbook` carries none, because a change that fixes tables while degrading ordinary prose is not an improvement and that is where it would show; `release-runbook` puts a table with a missing outer pipe in a section the fact stream never touches. Both details are load-bearing: a well-formed table never triggered the bug, and a model asked to edit a malformed table tends to rewrite it correctly, repairing the damage before it can be measured. The reported failure was in sections the operations never named, where nothing could repair it. Token usage is not reported. The stored reflect_response does not carry it, and a column that is always zero reads as "this is free" rather than "this is not measured here". * test(benchmarks): harden the judging, and say what the A/B measured Running the benchmark against main and the branch turned up three problems in the benchmark itself, all of which would have made its verdict untrustworthy. The runner slept a fixed interval instead of awaiting the async operations it submitted, so its first results were five rounds of "Generating content..." scored as though they were documents. Retain, create and refresh are all submit-and-poll; it now polls, and refuses outright if the seed it is about to measure is still a placeholder. Damage was attributed to the whole document rather than to the sections nobody asked to change, which scored a model deliberately restructuring a section it targeted as if the machinery had corrupted it. Damage is now measured only in untouched sections, and the metrics are recomputed from the stored documents at comparison time, so this sharper reading could be applied to results already collected instead of paying to re-run them. A single judge call decided each claim, and one pedantic reading moved a build's score: a document describing an operation as "synthesises stored memories" was scored as not supporting "answers questions over stored memories" — the same operation, in the wording the source fact itself used. Claims are now decided by majority of three, and that claim was rewritten to test the fact rather than one phrasing of it. A claim a correct document can fail measures the corpus, not the pipeline. The report prints mean document length beside the preference column, because judges favour longer documents and a preference that tracks that column should be read sceptically rather than counted. The prompt change is the finding that landed back in the product: stating a document's structure is more clerical than writing prose, and the model got terser at it — measurably shorter documents that the judge liked less. Document mode now says plainly that the structure is the shape of the answer, not a budget for it. Baseline recorded in baseline_report.json: 45 refresh rounds per build from identical seeds. main lost 3 tables to collapse, 9 table rows, 6 levels of list nesting and 5 hard line breaks across 6 damaged rounds, and drifted 14 sections nobody had named. The branch lost nothing and drifted nothing. Content came out level — 100% recall and zero stale claims on both sides. * fix(mental-models): give the delta leg the document's own token budget `max_tokens` was enforced in exactly one place: a rewrite of the *synthesis* answer when it came back longer than the budget. In delta mode that answer is only context for the operations call and never becomes the document, so the document that actually gets stored was never measured against the budget at all. A delta refresh only adds. The document-evolution benchmark measured ~20 tokens of growth per round across 45 rounds, monotonic, which crosses the 4096-token knowledge-page default after a couple of hundred refreshes — and knowledge pages refresh after every consolidation. The configured budget was quietly ignored for the entire life of a page after its first full build. Truncating the document here would delete knowledge nobody asked to delete, so the budget is stated instead: the delta call is told the document's current size against its budget, and when it is over, asked to make room with the same operations it uses for everything else — on content that is superseded or duplicated, never by dropping the facts it is integrating and never by summarising a section that is still current. Below 80% of the budget nothing is said at all. Every refresh records document_tokens and document_budget, and going over adds a warning, so a page that keeps growing is visible rather than merely large. Also closes the one path where the model still wrote markdown that got stored: the over-budget trim. In document mode it is now asked for a document, so the structure survives the trim instead of being re-derived from prose the model wrote. A response that is not JSON falls back to the previous split, which is lossless — the worst case is the old behaviour, not a lost answer. The real-LLM eval is the part that could not be mocked: told that a document is over budget, does a model reclaim space or append anyway? It drops the twelve archived sections and keeps the current process and the checklist — 434 tokens to 39 against a 200-token budget, stable across four runs. The assertions check where the space came from, because getting under budget by deleting current content would pass a naive shrink check and be worse than going over. * test(mental-models): audit every trigger flag on the delta leg `max_tokens` looked wired up — read from the model, passed to reflect, enforced by a rewrite — and was still ignored for the document that actually got stored, because in delta mode the thing it capped never becomes the document. Reading the code is how that was missed; nothing asserted the flag at its destination. So every flag is now exercised through a real delta refresh and asserted where it lands: retrieval options at the reflect call, document options in the delta prompt or the persisted row. Full mode is covered by the surrounding modules; this is the leg where a flag goes to die. Fifteen flags checked. Fourteen were already honoured. The audit pins them so a future change to either leg cannot quietly drop one: - retrieval: fact_types, exclude_mental_models, exclude_mental_model_ids, the model's own id (a model must not feed on its previous version), include_chunks, recall_max_tokens, recall_chunks_max_tokens, the model's tags, tags_match - document: max_tokens, response_schema (extracted from the merged document, not from reflect's delta-only answer), keep_trace, mode - and the whole trigger surviving a create/read round trip, since a flag that does not persist is not honoured either The fifteenth is documented behaviour that reads as a bug from outside, so it is pinned as intended rather than "fixed": `tag_groups` overrides flat tags entirely, dropping the model's own tags and forcing `tags_match` to `any`, because each group carries its own match mode. A `tags_match` set alongside groups is deliberately not forwarded. The default for a tagged model with no `tags_match` is `all_strict`, not `any` — a model scoped to tags must not widen its own scope by default. Scheduling flags (`refresh_cron`, `refresh_after_consolidation`) are honoured outside the refresh executor — the maintenance loop and the consolidation hook — and keep their existing coverage there. * test(benchmarks): type the structural summary, and cover the flag main just added Two findings from reviewing the branch against a fresh main. `_structural_summary` returned a raw dict of known keys, which the project standards forbid for exactly the reason it bit here: the report read it with `summary["rounds"]` and nothing would have caught a renamed metric until the table rendered wrong. It is a `StructuralSummary` model now, and the side-by-side table renders `model_dump()` so a metric added later appears without being listed twice. Main added a sixteenth trigger flag while this branch was in flight (`min_refresh_interval_seconds`, #3621). It gates automatic refreshes rather than shaping one, so it is honoured in the submit path and covered there — but the round-trip test enumerates the whole trigger on purpose, because a field that round-trips as None looks like the flag being ignored rather than like a storage bug. Adding it keeps that list exhaustive. * fix(mental-models): teach the retraction prompt the schema it emits into CI caught what the rebase brought: main added an unsay pass (#3618) whose prompt documents the operation vocabulary a second time, in prose, and it still told the model to say `{"op": "remove_block", "section_id": "...", "index": N}` with typed `block` payloads. Under the id-addressed schema those ops fail validation and are dropped, so a retracted fact would keep being stated and nothing would say why — the unsay feature silently doing nothing. The prompt now describes the schema it actually emits into: blocks addressed by `block_id`, block payloads as markdown strings, and the note about emitting removals in descending index order deleted, because ids do not shift when a sibling is removed and telling a model to order by position invites it to think in positions again. Guarded structurally rather than by one more test. The op vocabulary is written down twice — Pydantic models the applier validates against, and prose in each system prompt — and a test for the prompt that drifted does not exist by construction, since it is the one nobody wrote. `test_delta_prompt_schema_parity` asserts over *every* prompt carrying an operations vocabulary that each op exists, that no shape names a field the schema rejects, that no v1 typed block survives, and that blocks are addressed by id; plus a check that a prompt asking for `{"operations": [...]}` cannot be left off the list. Reverting the prompt fails three of them. The rest is the same schema change reaching tests the rebase brought in: canned ops in the outcome matrix moved to `text`, the retraction tests resolve a real `block_id` out of the document the prompt shows them (which is what a model does, and what a hardcoded index cannot express), the stale `parse_markdown` monkeypatch points at `structured_document_from_stored`, and four assertions on authored content now expect the canonical render.	2026-08-20 11:35:32 +02:00
Nicolò Boschi	613a699e9f	fix(consolidation): eliminate duplicate observations via interleave dedup recall (#1907) ... Round-robin interleave fusion for consolidation dedup recall (guarantees the semantic-#1 'twin' a slot so the LLM updates instead of duplicating), unified 'reranking' strategy param (cross_encoder/rrf/interleave), case-sensitive exact-dup guard, obs-dedup tool + benchmark wired into the perf dashboard (English dataset). Near-dup observation rate 4% -> 0% on the English hermes transcript (1/10 and 1/4), coverage 89% -> 94%, no false merges.	2026-06-03 17:37:57 +02:00
Nicolò Boschi	efe5ff8494	feat(perf): publish perf-test results to external dashboard (#1474) ... * feat(perf): publish perf-test results to external dashboard repo Adds `--benchmark-output-dir` to perf-test, which emits two JSON files in github-action-benchmark format: latency.json (smaller-is-better: durations + recall p50/p95/p99/mean) and throughput.json (bigger-is- better: items/queries/memories per sec). The Performance Tests workflow now publishes both to vectorize-io/hindsight-continuous-performance- monitor's gh-pages branch on each scheduled run. Iteration mode (TEMP — search "TEMP" to revert before merge): push trigger on this branch, default scale=small, locomo skipped unless manually dispatched. Setup needed (one-time): - PAT with Contents:write on the dashboard repo, stored as secret PERF_DASHBOARD_TOKEN. - After the first run creates gh-pages there, enable Pages on that repo (Settings → Pages → gh-pages branch). * fix(perf): wipe benchmark working dir between latency and throughput publishes github-action-benchmark clones the dashboard repo into a fixed ./benchmark-data-repository directory and doesn't clean up, so the second invocation in the same job fails with 'destination path already exists'. * feat(perf): replace github-action-benchmark with custom dashboard publisher Drops the two benchmark-action steps (and the dead `--benchmark-output-dir` flag + `_to_benchmark_entries` helper in system_perf.py) in favour of a single `scripts/benchmarks/publish-perf-results.sh` step. The script: 1. Reads the perf-test JSON output. 2. Enriches it with commit metadata (subject, author, author_date, commit URL, PR URL via `gh api commits/<sha>/pulls`). 3. Clones the dashboard repo's gh-pages branch using PERF_DASHBOARD_TOKEN. 4. Writes data/<timestamp>-<short_sha>.json and prepends the run to data/index.json (newest first). 5. Commits and pushes (with one rebase-retry on push rejection). The matching custom static site lives on gh-pages of vectorize-io/hindsight-continuous-performance-monitor (separate commit in that repo). * perf(workflow): publish dashboard on workflow_dispatch too * feat(perf): publish workflow run URL and LoComo results to dashboard Perf script now embeds workflow_run.{id,url} in each enriched run JSON and the manifest entry, sourced from default GitHub Actions env vars (GITHUB_RUN_ID + GITHUB_REPOSITORY). LoComo gets its own publish script (publish-locomo-results.sh) and a new step in the locomo job. The script strips per-question detailed_results (kept in the workflow artifact) before pushing — keeps each run small enough for git. Output lands at: data/locomo/<timestamp>-<short_sha>.json data/locomo-index.json The matching dashboard page (locomo.html) is in the dashboard repo. * perf(workflow): revert iteration-mode TEMP markers Restores the production defaults that were temporarily flipped while iterating on the dashboard: - drop the push trigger on feat/perf-dashboard - default scale: small → large - default locomo_skip: true → false - locomo job condition: workflow_dispatch-only → inputs.locomo_skip != true Scheduled cron now runs the full suite + LoComo daily and publishes to the dashboard.	2026-05-06 15:12:11 +02:00
Nicolò Boschi	9c33a7c730	feat(perf): add system performance test runner (#1201) ... * feat(perf): add system performance test runner and CI workflow Add `uv run perf-test` command that orchestrates retain throughput and recall latency benchmarks using mock LLM + pg0 for deterministic, LLM-independent baselines. Wraps existing recall_perf/retain_perf building blocks without duplicating benchmark logic. Also fixes _RRFReranker in recall_perf.py to include the cross_encoder attribute now required by the engine's combined scoring path. * feat(perf): add run-perf-test.sh script * feat(perf): use run-perf-test.sh in CI, remove run-retain-perf.sh Replace ad-hoc retain perf wrapper with the new system perf test script in CI workflow and docs. The standalone retain_perf.py is still available for ad-hoc document benchmarking. * feat(perf): add suite input to workflow dispatch	2026-04-22 17:37:58 +02:00
Nicolò Boschi	aefb3fcf4d	fix: improve async batch retain with large payloads (#366) ... * fix: improve async batch retain with large payloads * fix: improve async batch retain with large payloads * api * api * api * api * api * Clean up perf benchmark: keep only Python files - Remove README.md and PERFORMANCE_FINDINGS.md - Remove results/ JSON files (gitignored) - Remove test_data/ directory - Keep only __init__.py and retain_perf.py * docs: explain automatic batch optimization for async retain - Add section explaining Hindsight automatically handles batch sizing - Users don't need to manually tune batch sizes with async mode - Hindsight splits large batches (>10k tokens) into optimized sub-batches - Include example showing best practices * docs: remove emojis and code example from performance page * fix: correct OperationDetails type to match API response - Change optional fields to use | null instead of ? - Fixes TypeScript compilation error in control plane build * fix: use discriminated union for OperationDetails type - Support both success and error states properly - Fixes TypeScript error when setting error state * fix: use unique document_ids in batch retain examples - Each item in a batch must have unique document_id - Update both Python and JavaScript examples - Fixes test-doc-examples CI failure * chore: trigger CI * fix: test mocking and duplicate document_ids in examples - Mock _get_pool() in test_async_retain_tags.py to avoid _initialized error - Set _initialized = True on mocked MemoryEngine instances - Fix duplicate document_ids in retain.py and retain.mjs examples * fix: properly mock async pool/connection and fix more duplicate document_ids - Use AsyncMock for pool.acquire() to fix 'can't be used in await' error - Fix duplicate document_ids in retain-async examples (retain.py and retain.mjs) - Remove batch-level document_id parameter that caused duplicates * ci: collect all doc example failures and show summary - Run all Python/Node.js/CLI examples regardless of individual failures - Collect failure list and display summary at the end - Show pass/fail count and list of failed files - Exit with failure only after running all examples * refactor: extract doc example testing to standalone script - Create scripts/test-doc-examples.sh to run all examples - Collects logs of failed examples separately - Shows full error logs only for failures at the end - Clean summary with pass/fail counts - Proper exit codes - Replaces inline bash in CI workflow * fix: doc examples - duplicate document_ids and error handling - retain.py: move document_id to item level to avoid duplicates - documents.mjs: add error handling for getDocument to show clear error message * fix: update tests for duplicate document_id validation - test_async_retain_tags: verify operation structure instead of exact UUID - test_delete_bank: use unique document_ids (team-doc-1, team-doc-2)	2026-02-16 12:51:42 +01:00
Nicolò Boschi	b43ef98686	feat: consolidation performance benchmark and optimization (#227)	2026-01-29 11:24:15 +01:00
Nicolò Boschi	a3ad76d165	many improvements	2025-11-26 18:22:25 +01:00
Nicolò Boschi	3e618cb001	papers and fixes	2025-11-14 14:07:41 +01:00
Nicolò Boschi	588065182a	polish + cli + helm + standalone	2025-11-11 12:35:20 +01:00
Nicolò Boschi	ada9562cc2	lot of improvements	2025-11-10 15:02:51 +01:00
Nicolò Boschi	5d32e79955	fixes	2025-11-07 14:14:50 +01:00
Nicolò Boschi	fcc5250656	bm25 and re-rankers	2025-11-07 10:22:59 +01:00
Nicolò Boschi	39801d9f8b	local setup and speed	2025-11-05 15:35:07 +01:00