> saying LLMs can help with their accounting, help them close deals by crunching the numbers in seconds, write complex code for them etc etc.

Why do you think the results of this paper contradict these claims at all?

▲

grey-area 3 days ago | parent [-]

A machine which confabulates and cannot count is not a good fit for accounting tasks. They’ll make all sorts of subtle errors which are difficult for humans to notice.

	▲	stratos123 2 days ago \| parent [-]
		That wouldn't even necessarily be true if models really "couldn't count", since software exists - if an LLM is making an Excel spreadsheet rather than doing everything manually, it's both much harder for it to mess up and easier to notice and recover. It's even less true given that what this paper actually tests is "LLMs don't have a literally perfect accuracy when you make them do increasingly big problems with zero thinking". (Confabulation is IMO a much bigger problem, but it's unrelated to architecture - it's an artifact of how models are currently trained.)