I built this for local RAG https://github.com/kbrisso/byte-vision it uses llama.cpp and Elasticsearch. On a laptop with 8 GB GPU it can handle a 30K token size and summarize a fairly large PDF.

▲

busssard 3 hours ago | parent [-]

elasticsearch is the true limitation of rag systems...

	▲	kbrisso 3 hours ago \| parent [-]
		The vector search works great once you figure it out. I wanted to focus on writing the application and not have to rewrite a document store.