I worked on large scale RAG systems before and can say people vastly underestimate full text search and vastly overestimate embeddings. FTS is really easy, portable and scalable and gets you very far, the 80/20 rule applies. Embeddings appear to be nice and magic but when you really get into them you notice: semantic similarity isn’t as good as you think and certainly it won’t make everyone happy. You will inevitably end up having to re-embed more or different chunks of your text to accommodate more and more precise embedding search - at which point you’ll go the last mile and do reranking etc etc all the while having to support the operational burden of vector search. Then you turn around and build a search query with 500 keywords and sure it’s painful but it just works, accommodates all use cases, scales and is overall less annoying to maintain.

▲

andai 2 minutes ago | parent | next [-]

Re: the rube goldberg machine of diminishing returns

https://www.anthropic.com/engineering/contextual-retrieval

This is from two years ago, but I think it's still SotA?

▲

josh_p an hour ago | parent | prev | next [-]

I worked on getting an address database into elasticsearch years ago when it was still using modified tf-idf. Customers wanted FTS where a lot of the queries would be something like "100 First Ave, NY" or "200 2nd St, MN".

It was one of the most fun projects I've worked on in my career so far. I got a learn a lot about how US and international addresses worked, so many edge cases, and got to really understand how customers were using the existing search to make sure they weren't adding any duplicates to the database. Token filters and synonyms were neat and figuring out the right indexing strategy was a lot of fun.

It was a lot more work to get it right for most of the use-cases our customers had than just "throw it into ES and be done". That would probably have been fine for the 80/20 case, like you said, but I agree that the bulk of the work is going to be fine-tuning the search solution, whatever technology you're using.

	▲	oever an hour ago \| parent [-]
		What's your opinion on nominatim? I find that it gives up quickly when there's one or two typos in an address. It nails your examples.

▲

jameshart 2 hours ago | parent | prev | next [-]

I think people also overestimate the need for full text search when the one doing the querying is an LLM. If your underlying data is structured records, like a customer database, while humans might not have time or skills to figure out that when they want to search by phone number they need to do a join from the contacts table to the users table and normalize the phone number to look up first, making it best to just surface phone numbers as part of the data that is full/text-indexed… an agent is quite happy to handcraft the right SQL to find records that match on a specific field, given the right SKILLS.md and schema information. Turning fuzzy searches into exact DB lookups is a great way LLMs can augment users.

(Obviously this doesn’t apply to searching actual rich document data - for that, go all in on text search, embedding, etc)

▲

mdp2021 an hour ago | parent | prev | next [-]

> people vastly underestimate full text search

It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.

	▲	locknitpicker 18 minutes ago \| parent [-]
		> It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours. It is, if people don't even stop to think if they need synonyms, periphrases, or mistaken neighbours. As the blog post points out, more often than not you don't, particularly if your primary usecase is to search for technical keywords or codenames.

▲

shay_ker an hour ago | parent | prev | next [-]

How long have "large scale RAG systems" really existed in the first place? I'm always surprised at this, given how new all this really is, relatively speaking.

▲

lacedeconstruct 3 hours ago | parent | prev | next [-]

I thought text search was always the first thing you try, then fuzzy search, then you go for RAG

▲

wongarsu 2 hours ago | parent | next [-]

It's not like a simple embedding search takes that much longer to implement. Especially on short descriptions where you don't have to deal with chunking. And if you let an LLM write the code it's even less of a difference. Combine that with embedding search promising to solve all your search problems, and I understand why people often skip over full text search and go straight to embeddings

▲

ozim 2 hours ago | parent | prev [-]

I think Bitwarden implemented some vector search in their password search feature ... totally annoying it gives me back all kinds of stuff that I don't care.

I want fuzzy search like 95% of time and then I might consider having additional list of things that can be suggested by vector search.

	▲	gwerbin an hour ago \| parent \| next [-]
		Bandcamp has had legendarily bad semantic search for as long as they've been around. It's often completely impossible to find an artist or album or song even when you type the exact name.
	▲	a1o 2 hours ago \| parent \| prev [-]
		A good UI could do these and also exact match, give some point system to the results, then order them and perhaps use a bold highlight to reflect what parts of the input query reflected in each result.

▲

quijoteuniv an hour ago | parent | prev | next [-]

On my last go at making my own rag i still got better results by collecting the data and uploading to a project in open(butclosed)ai. My own rag, used by an agent was giving poorer results, and even the agent prefered (derailed)to not use it and look for the info itself rather than using the rag

	▲	idontneedcoffee an hour ago \| parent [-]
		I would be really grateful if someone could battle-test my frankenstein(lmdb + roaring bitmaps + to-be-removed lance with a bitmap-based virtual fs-like tree on top of your data) outside of its original narrow use-case

▲

mmargenot an hour ago | parent | prev | next [-]

And you get bm25 for free with so many modern setups! I do still love to experiment with tuning semantic search for your specific corpus via various kinds of embeddings, but bm25 is hard to beat.

▲

bensyverson an hour ago | parent | prev | next [-]

Yes, and don’t forget, LLMs are very good at tagging, so it’s not even that painful to backfill the corpus.

▲

kaon_2 3 hours ago | parent | prev | next [-]

Can you elaborate? We have technicians searching in different languages. Also our knowledge base is often in different languages. I just don't see how full text search can work? Maybe in a problem space like a wiki where people always know what to search for?

▲

tantalor 3 hours ago | parent | next [-]

FTS like Elasticsearch supports cross-language (also called multi-language) search.

▲

jon-wood 3 hours ago | parent | prev [-]

Instinctively this feels like a two phase problem - start with some machine translation into a single spoken language and index that, then when people are querying do the same thing. When returning search results show them in the original language.

▲

whilenot-dev 3 hours ago | parent | next [-]

Why not create indexes for multiple languages, as that would also avoid double translation issues (e.g. GER [query] → ENG [index] → GER [document])?

▲

kaon_2 3 hours ago | parent | prev [-]

Yes we've tried. It works. But jargon is hard. RAG with embeddings works all the same. The LLM doesn't mind receiving sources in Italian, french and German, and then outputting the answer in Japanese while providing the verbatim German jargon term in brackets

	▲	jameshart 2 hours ago \| parent [-]
		Embedding search is effectively machine translation into a single common ‘language’ - embedding space - and then searching that; cleaner and less lossy than translating everything into English for searching, but harder to debug when it goes wrong.

▲

clevergadget 2 hours ago | parent | prev [-]

I don't know what level of quality is required for this site but RAG is trash its just trash. its magic beans.