fix(corpus): normalize diversify() position term so similarity can compete

The greedy score mixed incommensurate scales: cosine similarity bounded
to [-1, 1] against an absolute list index that grows with the pool. For
a candidate j positions later to be preferred at the default
diversity=0.5, its similarity advantage had to exceed j -- impossible for
the non-negative cosines real footage embeddings produce. diversify()
therefore returned the input order verbatim, placing exact-duplicate
clips in adjacent edit slots, the one thing its docstring promises to
prevent. The threshold where the knob started working also depended on
pool size (0.66 at 4 candidates, 0.95 at 11).

Normalize the position term to [0, 1] so both terms share a scale. The
documented endpoints hold exactly as before: diversity=0 returns input
order, diversity=1 picks the most mutually dissimilar. Enumerating the
position also drops the O(n^2) remaining.index() lookup per candidate.

Closes #392
This commit is contained in:
0xDevNinja
2026-07-16 18:44:14 +05:30
parent f8d94632ea
commit 7ad68f28ec
2 changed files with 135 additions and 2 deletions

View File

@@ -407,14 +407,21 @@ class Corpus:
best_i = -1
best_score = -1e9
picked_mat = self.clip_embeddings[np.array(picked)]
for i in remaining:
# Normalize the position term to [0, 1] so it lives on the
# same scale as cosine similarity. An absolute index grows
# with the pool, drowning the similarity term: at the 0.5
# default a candidate one slot later needed a similarity gap
# > 1.0 to be preferred — impossible for non-negative
# cosines — so diversify() degenerated to input order.
denom = max(1, len(remaining) - 1)
for pos, i in enumerate(remaining):
sim_picked = float(np.max(self.clip_embeddings[i] @ picked_mat.T))
# We want LOW similarity, so we negate.
score = -sim_picked
# Diversity weights how hard we penalize similarity. At
# diversity=1 we always pick the most different; at
# diversity=0 we just take them in input order.
score = diversity * score + (1.0 - diversity) * (-remaining.index(i))
score = diversity * score + (1.0 - diversity) * (-pos / denom)
if score > best_score:
best_score = score
best_i = i