mirror of
https://github.com/countbot-ai/CountBot.git
synced 2026-09-14 20:46:47 +08:00
235c6c0e78
Chunk-level ask used to inject the top-6 chunks into the generation prompt unconditionally. On the 60-question eval, all 10 negative questions (answer not in the corpus) were still answered, and the top-1 BM25 scores of negatives overlap 7/10 with positives, so no score threshold can gate this reliably. The ask path now routes through a lightweight LLM grader (single call, ~200-400 tokens) before generation: - all relevant -> generate as before - partially relevant -> filter to the graded-relevant chunks, generate - none relevant -> rewrite the query once with LLM, re-search and re-grade; still none -> refuse explicitly instead of hallucinating Every LLM step degrades safely: no provider, grading failure or unparseable output falls back to the previous plain-generation behaviour, so nothing breaks when the flag is on but no provider is configured. Only active under COUNTBOT_RAG_CHUNKS=1. No new dependencies. Tests: 12 new (three-way routing, refusal after one rewrite, retry succeeds, grading failure/unparseable/no-provider fallbacks, grader output parsing contract). Full suite: 138 passed. Refs: #107