| ▲ | Intelligence per Watt: Measuring Intelligence Efficiency of Local AI(arxiv.org) |
| 55 points by pythonic_hell 2 days ago | 7 comments |
| |
|
| ▲ | jmiskovic 10 minutes ago | parent | next [-] |
| Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winter time the GPU also doubles as a 300W in-house heater. Not to mention avoiding personal data collection and re-selling. |
|
| ▲ | api an hour ago | parent | prev | next [-] |
| Unless I misread it, are they saying local GPUs use less energy? That’s surprising, almost unbelievable, due to batching. Local is usually not batched. |
| |
| ▲ | stymaar an hour ago | parent [-] | | Small models are much smaller than frontier models though, which is how they end up consuming less energy despite low batch count. (Though with local models growing strong agentic capabilities, batching becomes a reality with local models as well). |
|
|
| ▲ | iLoveOncall 3 hours ago | parent | prev [-] |
| > We propose intelligence per watt (IPW), task accuracy per unit of power Stupid metric. It's not because a model is better performing that it necessarily requires more energy or compute. |
| |
| ▲ | utopiah 2 hours ago | parent | next [-] | | I don't think that's what they are saying. In fact if they did the metric would be pointless. Rather they are saying by estimating that value on different architectures, one can find more efficient ones. They use open model to be able to remove unknowns. They aren't advocating for one model or another, only more efficient architectures. | | |
| ▲ | the8472 21 minutes ago | parent [-] | | Intelligence per Joule would be more appropriate in many cases. If a model can do the same work but takes 10 times as long as a bigger one that can still be useful (e.g. due to memory constraints), but at the same wattage it burns 10 times the energy.
Even more so on mobile devices. |
| |
| ▲ | _diyar 2 hours ago | parent | prev [-] | | > We propose miles per hour (MPH), distance travelled per unit of time Stupid metric. It‘s not because you spend more time that you travel farther. s/ |
|
