Does anyone have experience using SMLs for RAG (either as query rewriter or as generator for the final answer)?

I'd like to work with a corpus offline (internal university research data) and I'm hoping I can get everything done without the data leaving the premises.

I guess the biggest bottleneck is going to be for the context window size which won't be able to fit too many result "hits."

Any info or advice would be appreciated.