LLMs are still absolutely horrid at analyzing PDFs so the results they are getting must be chock full of errors..

Gareth321 20 hours ago | parent | next [-]

Are they? I've had excellent success. The confusing part of this is that there are two types of PDF. The first is a "normal" digital PDF. The second is a scanned PDF. The first can essentially be read like a document. LLMs have no issues with this. It's the second kind of PDF where the constraint becomes the vision capability, and this is very impressive with Astra. I've had no issues with either. I imagine there could be issues with unusually dense and/or misaligned text on scanned PDFs, but I have not tested this.

The bottom line, though, is that PDF OCR is usually regarded as a solved problem. LLMs won't usually do the recognition itself. It will farm it out to established tools which are very good.

▲

QuantumGood 4 hours ago | parent | next [-]

The more data in the PDF, the more nines you need in the OCR accuracy. Plenty of 5/S, O/0 and other issues exist at frequencies that cause problems.

▲

gf000 19 hours ago | parent | prev | next [-]

Well, I would argue about the first part. Even if they contain "native" text that can be extracted, in most cases their order will be messed up and it is often crucial for correct parsing.

So in many cases the visual way is the only one that works correctly, the textual one is just a shortcut that may be walkable in certain cases.

▲

Otterly99 15 hours ago | parent | prev [-]

It depends on what you called solved.

If the goal is to only extract the unstructured text from the document, it is definitely solved. Extracting a more natural structure like paragraph separation, tables, header, footers (what is referred as document intelligence) is much more complicated and not fully solved, but I would say almost.

	▲	jorvi 12 hours ago \| parent [-]
		Yup, this. It is actually one of my test cases for LLMs: take the weekly discount PDFs of all the big supermarkets and process each of them, creating a nice table per supermarkt, converting discounts like 1+1 and only listing discounts that are interesting value. I then share that with a bunch of people. All models fail this, even the really expensive ones. Even with harness, examples and proper insistent instruction, they'll mix up items and their related discount, which category the item should be in, which page they are on, skipping over items etc. As said above, you can OCR it, but at that point you're not processing a PDF, you're processing an image. And yes, I know the underlying raw PDF data is messy, but that's why it's such a good test.

▲

alansaber 18 hours ago | parent | prev | next [-]

Yes but this is more an artifact of how much PDF sucks as a file format than LLMs being crap.

▲

sumedh 19 hours ago | parent | prev | next [-]

Which models are you using.

	▲	drtgh 16 hours ago \| parent [-]
		All the models output mixed data (errors). The larger is the context, the bigger the probability to happen. And this is an inherent property of LLMs (the output is a statistical concatenation of text, the LLMs are an vectorial database filled statistically). If you connect an LLM to pilot a plane with passengers, prepare the obituaries, because would happen sooner or latter, an undesired index got mixed and produced a mixed/wrong output. But here, the bigger mistake would be to let an LLM execute commands without human revision, aimed by the deceptive marketing around the A"I" (LLMs).

▲

davidguetta 16 hours ago | parent | prev [-]

they were in 2023