This isn't a hardware feat, this is a software triumph.

They didn't make special purpose hardware to run a model. They crafted a large model so that it could run on consumer hardware (a phone).

▲

pdpi 6 hours ago | parent | next [-]

It's both.

We haven't had phones running laptop-grade CPUs/GPUs for that long, and that is a very real hardware feat. Likewise, nobody would've said running a 400b LLM on a low-end laptop was feasible, and that is very much a software triumph.

▲

bigyabai 5 hours ago | parent [-]

> We haven't had phones running laptop-grade CPUs/GPUs for that long

Agree to disagree, we've had laptop-grade smartphone hardware for longer than we've had LLMs.

	▲	pdpi 4 hours ago \| parent [-]
		Kind of. We've had solid CPUs for a while, but GPUs have lagged behind (and they're the ones that matter for this particular application). iPhones still lead by a comfortable margin on this front, but have historically been pretty limited on the IO front (only supported USB2 speeds until recently).

▲

smallerize 6 hours ago | parent | prev | next [-]

The iPhone 17 Pro launched 8 months ago with 50% more RAM and about double the inference performance of the previous iPhone Pro (also 10x prompt processing speed).

	▲	6 hours ago \| parent [-]
		[deleted]

▲

SV_BubbleTime 4 hours ago | parent | prev | next [-]

>triumph

It’s been a lot of years, but all I can hear after reading that is … I’m making a note here, huge success

	▲	GorbachevyChase 4 hours ago \| parent \| next [-]
		There’s no use crying over every mistake. You just keep on trying until you run out of cake.
	▲	breggles 4 hours ago \| parent \| prev [-]
		It's hard to overstate my satisfaction!

▲

anemll 3 hours ago | parent | prev [-]

both, tbh