I hadn't seen the token/Joules comparison with human speech before. Humans are still 22x more efficient, which is not that far considering the rate of progress in this area.

▲

Phemist 6 hours ago | parent | next [-]

The 20W number includes EVERYTHING else the brain does. The chips/models are literally only producing tokens. Let's see an LLM drive a robot harness and have the robot produce speech, as well as move through 3D space, keep track of metabolic needs, etc. etc. etc. before we compare efficiencies. That is even assuming the tokens are of equal quality. This comparison is currently Apples and Oranges.

▲

phoghed 6 hours ago | parent | next [-]

kind of a moot point if you can't get your brain to not do everything else. I think it's a fun comparison, even if it's not a 100% equivalence.

▲

cmrdporcupine 4 hours ago | parent | prev | next [-]

Right, I can do the talked about ~3 tok/sec output and drive a car, hold my bladder, and eat chips at the same time.

Take that, Jalapeno!

▲

falcor84 2 hours ago | parent [-]

For what it's worth, LLMs don't really suffer from incontinence, so at least that part is pretty much a solved problem.

	▲	pantalaimon an hour ago \| parent \| next [-]
		They sometimes leak their system prompt
	▲	undersuit 12 minutes ago \| parent \| prev [-]
		So why are their water cooling systems filled with leak detectors? /s

▲

CooCooCaCha 5 hours ago | parent | prev [-]

And the brain is literally only producing electrochemical signals.

I don’t see how tokens can’t produce speech or track metabolic needs. You can talk to chatgpt can’t you? Or do you mean literally talking? Because that’s not a brain function, that’s the mouth, vocal chords, and lungs.

	▲	Phemist 5 hours ago \| parent [-]
		> I don’t see how tokens can’t produce speech or track metabolic needs. It probably could, but the point is this would require additional tokens, blowing up the comparison. The token output of LLMs and "token output" of speech are simply at different abstraction levels. Hence my comparison to the LLM brain driving the robot harness to produce speech etc. This would be more comparable, and also look significantly worse than "only" the 22x less efficient number.

▲

nojs 3 hours ago | parent | prev | next [-]

> Humans are still 22x more efficient, which is not that far considering the rate of progress in this area.

Based on a human output rate of 3.3 tok/s, which seems questionable as a means of comparison

▲

xyzsparetimexyz an hour ago | parent | next [-]

I do believe that this is the trade off. We are more efficient but slower in terms of thinking (at the same level of intelligence). Some animals go much further in terms of that trade off, see https://en.wikipedia.org/wiki/Portia_(spider) for example.

▲

falcor84 2 hours ago | parent | prev [-]

What exactly are you questioning?

	▲	nojs 2 hours ago \| parent [-]
		The claim that tok/s independent of quality is a useful comparison (I can get thousands of tok/s on a suitable small model), and secondarily that humans can’t output “tokens” faster than than in some sense, which I am less confident about

▲

DoctorOetker 5 hours ago | parent | prev | next [-]

I couldn't source the parameters from the screenshot or the nearby graphs, but from the nearby graphs you can see that at concurrency C=1, tokens/Joule (vertical axis) has totally plummeted, and obviously concurrent inference is much more efficient by batching. Divide the memory by the bandwidth and thats how long it takes to dump the full RAM contents through the chip. Do you want to do this once per token for a single conversation, or do you want to progress multiple conversations if you're going through all the weights anyway? The peak in the graphs is easily 22x more efficient than the low bottom right part on the graphs. So in batched mode its already more efficient than human speech.

▲

walrus01 an hour ago | parent | prev | next [-]

Fairly amazing when you think about it, like human intellect can run on a bowl of rice and a chicken yakitori skewer.

▲

jstummbillig 5 hours ago | parent | prev | next [-]

At just inference! Which both a human and a model can not do without training, but while training rounds to zero for the model, for humans it scales linearly.

I am relatively certain we have already squarely been beaten in net efficiency at scale.

▲

plasticchris 6 hours ago | parent | prev | next [-]

Probably not when you consider the training cost and upkeep expenses, not to mention the depreciation…

▲

kemiller 6 hours ago | parent | prev | next [-]

I wonder how that stacks up if you consider all the time you have to keep the body alive when it’s not actively producing “tokens”.

	▲	jdiff 3 hours ago \| parent [-]
		Careful, let's not put the whole matrix into stasis outside of business hours. Productivity is not the only reason to let these meatbags burn oxygen.

▲

saagarjha 4 hours ago | parent | prev | next [-]

You’re missing the factor for intelligence/token.

▲

danishanish 6 hours ago | parent | prev [-]

I mean, surely when quality is accounted for the difference is significantly higher

	▲	GaggiX 6 hours ago \| parent [-]
		Or maybe significantly lower.