I think that if I were to build this I might skip the LLM and just setup a nice interface to the source files using qmd style search. [0]

That way the tiny model isn't acting as a potentially incorrect interpreter but you still get semantic search to help you find the right documents quickly.

I suppose there is something to be said for a good summarization, I just worry that these tiny models aren't really GOOD summarizers.

[0] https://github.com/tobi/qmd

Awesome, I didn't know qmd style search, thank you!

I'm all up for using the better/simpler task for the job. That's been a thing lately also with the Jev/decision models hype, most of those things can either already be done by a pretrained classifier[0], or you can start with a bigger model and then train your smaller/adhoc one when you have some data (it doesn't even take much for most applications, you can get about 95% accuracy on email classification with 50-100 examples)

0: http://jeffyclassify.com/blog/when-llms-fail-at-classificati...