Mass attempted replication of old findings of LLMs with new models

dwroberts 5 hours ago | parent | next [-]

> Mostly capability failures that newer models no longer make: counting letters

Have LLMs fundamentally actually changed or improved to resolve problems like this, or have the big models been aggressively tuned / trained on specific examples so it’s less embarrassing?

Because I’ve noticed that Google’s stuff always seems to generate code to do counting tasks, even when not asked for, so I wondered if the LLM itself even generated the answer or it’s just being instructed to use a tool? (Which IMO is a workaround)

	▲	dcrazy 3 hours ago \| parent \| next [-]
		The LLMs have indeed gotten better at representing numbers and performing arithmetic, but they can still do it wrong. I don’t consider it a workaround for an LLM to delegate arithmetic to a calculator.
	▲	cyanydeez 4 hours ago \| parent \| prev [-]
		Inthink finetuning and prompts and intermediary engineering harnesses all obscure idependent verificant of raw LLM capability. So yes, they probably spend half theor budget trying to keep suspension of disbelief.

▲

_davide_ 10 hours ago | parent | prev [-]

Seems like this claim is expired in 3...2...1...