The real benefits of LLMs is to give me a good summary of these documents. Still, using these abstractions is probably not worth it.

Well, a summary. It's impossible to evaluate the quality of the summary without actually reading the document you're trying to avoid reading.

LLMs are a symptom, not a solution. Just another over-hyped bullshit (those downvoting will agree in few years or never as they are the ones riding the hype) that its only concern is to boost company stocks. Google is the proof, their search started to suck immediately when they added AI into concept. It promotes bullshit, while it doesn't hit the really relevant results, even if you specify them into the details. If AI would have any real value, it would never be given out for free and this is the case: they have a solution and they are searching for a problem it solves.