mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Sanderhoff-alt
bc06bd051f
Replaces the O(N^2) pairwise loop in _find_intrabatch_similar_pairs with a prefix-filtering set-similarity join, returning exactly the pairs the double loop returned. Each name's trigrams are sorted rarest-first and only the leading |A| - ceil(t*|A|) + 1 are indexed; a pair sharing none of those tokens cannot clear the cutoff. Cutting each prefix from its own size is what makes the pruning lossless — ascending set-size order is for the size filter, not for correctness. 3.9x faster at the 250-name cap and 8x at 1000 on distinct names; 2.5x over 9,525 real per-document batches harvested from LoCoMo and LongMemEval. Two shapes are slower and are now benchmark workloads rather than footnotes: batches under ~50 names (tens of microseconds) and batches of mutually similar names, where the filter prunes nothing and costs ~1.2x. _INTRABATCH_MAX_NAMES stays at 250 — that second shape costs ~156ms at 500 with no await in the join. Equivalence is pinned against the loop it replaced over randomised batches at 14 cutoffs, including exactly-achievable Jaccard ratios where a float t*|A| a hair above a whole number would shorten the prefix and drop a pair on the cutoff. Verified further on 60,000 set-level trials and the 9,525 real batches at six cutoffs: zero mismatches. Also memoises candidate trigram sets in _resolve_from_candidates, filled lazily so candidates the scoring loop never reaches cost nothing and no work happens ahead of the _SCORING_YIELD_EVERY yield points (#3211).
17 lines
497 B
Bash
Executable File
17 lines
497 B
Bash
Executable File
#!/bin/bash
|
|
# Microbenchmark entity resolution and in-batch deduplication on retain paths.
|
|
#
|
|
# Usage:
|
|
# ./scripts/benchmarks/run-entity-resolver-bench.sh
|
|
# ./scripts/benchmarks/run-entity-resolver-bench.sh --repeats 10
|
|
# ./scripts/benchmarks/run-entity-resolver-bench.sh --json /tmp/entity_bench.json
|
|
|
|
set -e
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
|
|
|
cd "$PROJECT_ROOT/hindsight-dev"
|
|
|
|
exec uv run entity-resolver-bench "$@"
|