Remix clone Hacker News

> there are some new capabilities that are big, but they are still fundamentally next-token predictors

Anthropic recently released research where they saw how when Claude attempted to compose poetry, it didn't simply predict token by token and "react" to when it thought it might need a rhyme and then looked at its context to think of something appropriate, but actually saw several tokens ahead and adjusted for where it'd likely end up, ahead of time.

Anthropic also says this adds to evidence seen elsewhere that language models seem to sometimes "plan ahead".

Please check out the section "Planning in poems" here; it's pretty interesting!

https://transformer-circuits.pub/2025/attribution-graphs/bio...

▲

percentcer 5 hours ago | parent [-]

Isn't this just a form of next token prediction? i.e. you'll keep your options open for a potential rhyme if you select words that have many associated rhyming pairs, and you'll further keep your options open if you focus on broad topics over niche

▲

DennisP 4 hours ago | parent | next [-]

Assuming the task remains just generating tokens, what sort of reasoning or planning would say is the threshold, before it's no longer "just a form of next token prediction?"

▲

pertymcpert 2 hours ago | parent | prev | next [-]

It doesn't really make explain it because then you'd expect lots of nonsensical lines trying to make a sentence that fits with the theme and rhymes at the same time.

▲

throwuxiytayq 4 hours ago | parent | prev [-]

In the same way that human brains are just predicting the next muscle contraction.

▲

alfalfasprout 4 hours ago | parent [-]

Except that's not how it works...

	▲	Workaccount2 3 hours ago \| parent [-]
		To be fair, we don't actually know how the human mind works. The most sure things we know is that it is a physical system, and that does feel like something to be one of these systems.