mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Nicolò Boschi
dbe0ffb989
`pending_consolidation` counted every fact with `consolidated_at IS NULL`, including the ones stamped `consolidation_failed_at` that the consolidator's own candidate query (`reads.find_unconsolidated`) excludes on purpose. The gauge therefore had a floor no amount of work could clear: it sat above `?consolidation_state=pending` by exactly `failed_consolidation`, and an operator could not tell a real backlog from an abandoned residue. `pending` now carries the consolidator's predicate, so the two buckets are disjoint and a bank with no live backlog reaches zero. The same predicate was missing in five more places: - `get_bank_stats` keeps a second copy of the freshness SQL for the `writes_memory_rows_in_sql` path — fixing only `counts.py` would have left the default Postgres path wrong. - `hindsight.consolidation.backlog` had the same floor, which made "backlog > 0 for N minutes" unalertable on any bank holding a residue. - reflect's `tool_search_observations` derives `is_stale` / `freshness` from this count, so a residue told the model the observations were stale on every call, for ever (fixed transitively via `get_bank_freshness`). - the control plane's consolidation card computed `done = total - pending`, which would have counted the failed rows as done once pending got strict. - the benchmark runner waits for this count to reach 0, so one permanently failed fact burned the full 3000s timeout. Everything else that answers "what is left to consolidate" already excluded them: the memories list filter, `count_unconsolidated`, and the `banks_needing_consolidation()` maintenance routine — so scheduling was never spinning on the residue.