Quirks from the article:

1) The performance numbers are simulated

2) The details say that the performance numbers require 2.7x the overall power, so around 13.5W. Power needed for all the memory is not explicit in the numbers.

3) The compiler is also simulated and it's a prototype, so it's not a drop-in replacement for anything like Nvidia ISAAC

4) 5W seems to refer to sub-4B models, so not the 100B param ones.

▲

NewJazz 2 hours ago | parent | next [-]

The chip is designed to run models with 20 billion to 100 billion parameters (approximately 3 billion to 20 billion active parameters)

Doesn't seem like 100B dense models are even on the table, only MoE models. Kinda misleading.

	▲	gunalx an hour ago \| parent [-]
		Not really. Models of that size is reasonably often moe. It is also 100b even if it only has 20b active at a time.

▲

sampton an hour ago | parent | prev | next [-]

13.5W is Jetson nano territory.

▲

idiotsecant 2 hours ago | parent | prev [-]

I think the human brain uses something on the order of 10-20W so 13.5W seems like there's probably some room to improve on a 4b model!

▲

dinfinity an hour ago | parent | next [-]

Statements like these often overlook that a human brain is pretty slow for a lot of tasks. It's hard to make direct comparisons, but for language (related cognitive work) we're at maybe a few tokens per second.

I wonder how even a team of humans (a meat mixture of experts) would perform on the AI benchmarks, measured in energy use per intelligence index/benchmark point.

	▲	egberts1 44 minutes ago \| parent [-]
		"Few tokens per seconds"?! Human brain is an advanced Next Concept Model.

▲

delichon 2 hours ago | parent | prev [-]

423. Could y'all please start prefixing comments with the number of joules you consumed in generating it, so that I can decide if it's worth reading?

	▲	Terr_ 2 hours ago \| parent [-]
		Flashbacks to dial-up internet era, where any website would, parenthetically, indicate how big a linked file was with an estimated download time.