Can you elaborate? We have technicians searching in different languages. Also our knowledge base is often in different languages. I just don't see how full text search can work? Maybe in a problem space like a wiki where people always know what to search for?

▲

tantalor 3 hours ago | parent | next [-]

FTS like Elasticsearch supports cross-language (also called multi-language) search.

▲

jon-wood 3 hours ago | parent | prev [-]

Instinctively this feels like a two phase problem - start with some machine translation into a single spoken language and index that, then when people are querying do the same thing. When returning search results show them in the original language.

▲

whilenot-dev 3 hours ago | parent | next [-]

Why not create indexes for multiple languages, as that would also avoid double translation issues (e.g. GER [query] → ENG [index] → GER [document])?

▲

kaon_2 3 hours ago | parent | prev [-]

Yes we've tried. It works. But jargon is hard. RAG with embeddings works all the same. The LLM doesn't mind receiving sources in Italian, french and German, and then outputting the answer in Japanese while providing the verbatim German jargon term in brackets

	▲	jameshart 2 hours ago \| parent [-]
		Embedding search is effectively machine translation into a single common ‘language’ - embedding space - and then searching that; cleaner and less lossy than translating everything into English for searching, but harder to debug when it goes wrong.