Fuse vector and full-text results by rank instead of asking the model to choose

[jasperblues]

Retrieval currently asks the model to choose between vector and full-text search. It has now been observed choosing wrong in both directions, and each mistake was expensive. Fusing both and ranking the union removes the decision instead of trying to instruct it better.

Current state

ToolishRag exposes vectorSearch and textSearch as separate tools, and the tool descriptions tell the model when to use each. On a 40-question benchmark corpus:

• With one wording, the model called full-text 22 times against vector's 1 — including for pure prose questions with no identifier in them. It then required every word, matching zero chunks, and retrieval returned empty.
• With the corrected wording it inverted to 1 against 7, barely using full-text at all — including for questions where an exact-wording match is what would answer them.

Both are routing failures. Neither is obviously fixable by better instructions: the categories the guidance uses ("semantic queries", "exact-keyword matches") are not ones a model can reliably recognise, because every question has keywords in it.

Why fusion rather than better routing

The two retrievers fail on different inputs and succeed on different inputs, which is the textbook case for running both. The obstacle has always been that their scores are not comparable — BM25 sums per-term contributions and is unbounded before normalization; cosine is bounded and calibrated differently. Measured on one real corpus, query-to-chunk cosine tops out around 0.75 while normalized BM25 for the same corpus reaches 0.62–0.79 for questions and 0.29–0.55 for exact identifier matches. Merging those by score means whichever retriever happens to produce bigger numbers wins, regardless of relevance.

Reciprocal rank fusion sidesteps that entirely: it combines on Σ 1/(k + rank) and uses only ordering, which is the one thing the two retrievers genuinely share. It is what Elasticsearch, Vespa and Weaviate do for hybrid retrieval, for this reason.

Direction

One search tool that runs both retrievers and returns the fused ranking, replacing the routing choice rather than supplementing it. Keep it behind a switch so the routed surface remains available and the two can be compared on the same corpus.

Worth measuring, not assuming: fusion has a cost — two retrievals per call — and it may lose to a well-routed vector-only search on a corpus where full-text rarely helps. The benchmark is the arbiter.

Outcome

Either evidence that fusion beats routing on a real corpus, and a default worth changing, or a recorded result that routing is good enough and the extra retrieval is not worth paying for.

[jasperblues]

Measured on a 40-question benchmark (Australian Venture Capital Act, 42-document corpus), three passes per variant. Fusion does not justify itself here, and the reason is instructive.

variant	passes	mean	target-doc recall	median latency
routed, EXPRESSION	32, 32, 31	31.7	37/40	6.6s
routed, LITERAL	32, 31, 32	31.7	37/40	6.5s
fused (RRF)	30, 30, 29	29.7	37/40	5.1s

Consistently ~2 questions worse, with non-overlapping ranges, and consistently ~20% faster — the model makes one call rather than deliberating over two tools.

Why it loses

RRF weights agreement, so a chunk ranked first by full-text and 30th by vector outranks vector's own top hit: 1/90 + 1/61 = 0.0275 against 1/61 = 0.0164. On prose questions full-text's first result is frequently junk, and fusion promotes it into the top 8 at the expense of a good vector hit.

That is the mechanism working as designed, not a bug — but on a corpus where one retriever is reliably better for most questions, "both retrievers vote" costs more than the routing mistakes it prevents.

What it did fix

The routing problem was real: the same model called full-text 22 times against vector's 1 under one wording, then 1 against 7 under the next. Fusion removes that variance entirely. It just does not pay for itself in answer quality here.

Worth knowing before anyone retries this

Target-document recall was identical at 37/40 across all three variants. Fusion is not finding less, it is ranking differently. The residual failures are a vocabulary gap between question and evidence — "count as law" vs "legislative instrument" ranks 80th by vector and 239th by full-text — which no merge strategy reaches. Query expansion is the lever there, not fusion.

Implementation lives on an unpushed branch. Leaving this issue open: fusion may still win on a corpus where the two retrievers are more evenly matched, or with a weighting that favours the stronger retriever rather than treating both equally.

[rishiii183]

I think I can solve this

[jasperblues]

@rishiii183 Thanks, however we'll do this one internally as there's a specific way that we want to solve it. It is incubating on an internal project. Once it is up will welcome feedback.

[rishiii183]

@jasperblues alright jasper