> Mostly capability failures that newer models no longer make: counting letters

Have LLMs fundamentally actually changed or improved to resolve problems like this, or have the big models been aggressively tuned / trained on specific examples so it’s less embarrassing?

Because I’ve noticed that Google’s stuff always seems to generate code to do counting tasks, even when not asked for, so I wondered if the LLM itself even generated the answer or it’s just being instructed to use a tool? (Which IMO is a workaround)

The LLMs have indeed gotten better at representing numbers and performing arithmetic, but they can still do it wrong.

I don’t consider it a workaround for an LLM to delegate arithmetic to a calculator.

Inthink finetuning and prompts and intermediary engineering harnesses all obscure idependent verificant of raw LLM capability.

So yes, they probably spend half theor budget trying to keep suspension of disbelief.