Untrusted ... but does it have any value at all when you can't be sure that a lot of it is hallucinated? After all, LLMs are not very good with numbers.

▲

defrost a day ago | parent [-]

You're correct that I can't be sure as I don't work at KPMG and haven't had any contact with their piles of documents, existing practices, or TaxBot summaries.

What I do know as a fact is that KPMG are self reporting satisfaction with their in house work on putting such a thing together.

The 'proof' will be the next five years of application to corporate clients.

> After all, LLMs are not very good with numbers.

The assumption, always, should be that neither are interns.

Hence why draft summaries should be reviewed and sanity checked by senior experienced people.

I would assume (based on my prior work summarizing large volumes of data for mineral and energy resources domain) that any report produced would have references back to source documents and pages making the task of cross checking the product simple and relatively straightforward.

▲

Neywiny 19 hours ago | parent [-]

I think the concern is more than what it gathered, I think there's a lot of skepticism over it missing something. The same way so many AI tools just ignore commands, imagine it just ignoring a few sentences. Maybe like:

> We'll sell you our company for $100. But, you have to do a hand-stand and spin around 5 times.

If the AI only puts the first sentence in the summary, you could see how it'd be a bad day for the client. Any human would go "huh that's weird, I'll make sure that's noted in the summary" but in my experience, AIs just don't have that feeling.

▲

defrost 19 hours ago | parent [-]

What's being ignored, it seems, is this is explicitly an in-house tool for a first draft summary to be reviewed by an in-house accountant prior to a final presentation to a client.

> imagine it just ignoring a few sentences.

Sure. Just like the risk every such human intern | associate | junior prepared similar draft report already carries today and in the past.

One would hope that as a company at risk of litigation and carrying the can for bad advice that an AI reduced draft such as this would be proof read by a senior expert in house who would trace back every "We'll sell you our company for $100." to the _original_ context via an embedded hyperlink in the draft.

It's certainly the way in which things were done when generating summaries of tens of thousands of documents for mineral and energy clients looking to invest at least $50 million in advancing projects for return.

▲

Neywiny 19 hours ago | parent [-]

You've missed my point. I don't think any human who has a job at a law firm would ignore a sentence like that. I think any AI I've used has ignored explicit instructions of moderate severity. I'm not worried it'll hallucinate things into existence, I'm worried it'll ignore them out. Can't summarize without throwing away words. I don't trust it to choose the right ones.

	▲	defrost 7 hours ago \| parent [-]
		And you've missed mine. I don't think any human at any law firm, medical practice, major resource company, etc. that deals with volumes of documentation in the course of making multi million deals would _trust_ an associate / intern pool or an AI to create a perfect product that can be passed directly to a client without any form of checking and verification. It's a _given_ that there will be shortfalls and errors and the procedures need to be sufficient to embrace an error prone distillation phase and a circle back and verify phase. At least in my experience to date. It's clear from the article that KPMG feel much the same way.