He compared techniques that used:

- BM25 only

- Jev only

- LLM + Jev

The latter worked best.

But I wonder whether LLM on its own might have gotten close. If you're going to use an LLM for each call then it seems reasonable to test whether or not adding Jev improves performance.

EDIT: I should have had more coffee before writing this comment. As make3 points out below, I forgot that in the LLM+Jev setup, the LLM only needs to run once per query, whereas in the LLM-only setup the LLM would run once per item/segment being evaluated.

A reranker has to be fast. LLMs are not.

Importantly though, LLMs can generate questions in parallel, and there can be much fewer questions (q) than segments to evaluate against (n), so the cost of jev on questions (jev(q * n) + llm(q)) vs the cost of putting all the segments in the LLM can be smaller (llm(n)), (llm(n) >> llm(q) + jev(n * q) if n >> q)

The real benchmark that's missing for me is MaxSim on a real embedder from MTEB.