new | show | ask | jobs Github

adastra22 a day ago

NPUs aren't meant for LLMs. There are a lot more neural net tech out there than LLMs.

▲

aleph_minus_one a day ago | parent [-]

> NPUs aren't meant for LLMs. There are a lot more neural net tech out there than LLMs.

OK, but where can I find demo applications of these that will blow my mind (and make me want to buy a PC with an NPU)?

▲

adastra22 a day ago | parent | next [-]

Apple demonstrates this far better. I use their Photos app to manage my family pictures. I can search my images by visible text, by facial recognition, or by description (vector search). It automatically composes "memories" which are little thematic video slideshows. The FaceTime camera automatically keeps my head in frame, and does software panning and zooming as necessary. Automatic caption generation.

This is normal, standard, expected behavior, not blow your midn stuff. Everyone is used to having it. But where do you think the computation is happening? There's a reason that a few years back Apple pushed to deprecate older systems that didn't have the NPU.

▲

adgjlsfhk1 a day ago | parent [-]

I've yet to see any convincing benchmarks showing that NPUs are more efficient than normal GPUs (that don't ignore the possibility of downclocking the GPU to make it run slower but more efficient)

	▲	adastra22 a day ago \| parent \| next [-]
		NPUs are more energy efficient. There is no doubt that a systolic array uses less watts per computation than a tensor operation on a GPU, for these kinds of natural fit applications. Are they more performant? Hell no. But if you're going to do the calculation, and if you don't care about latency or throughput (e.g. batched processing of vector encodings), why not use the NPU? Especially on mobile/edge consumer devices -- laptops or phones.
	▲	imtringued 17 hours ago \| parent \| prev [-]
		https://fastflowlm.com/benchmarks/ https://fastflowlm.com/assets/bench/gemma3-4b.png

▲

jychang a day ago | parent | prev [-]

Best NPU app so far is Trex for Mac.