Remix.run Logo
▲ PaulHoule 25 minutes ago

Length-dependence has been a thing in vector search since bag-of-words!

https://en.wikipedia.org/wiki/Okapi_BM25

was one of very few useful results of the first decade of TREC research... And the main thing it addresses is balancing various factors which could cause a full text search engine to privilege either long or short documents.