There is a lot of buzz around this topic but i am really curious about what's actually useful day to day when running AI/LLMs locally on your PC. A few things I'm trying to understand:

Which models are you running locally and on what hardware? What use cases have actually worked out considering the speed and memory limitation? At what point did local start making more sense than just hitting an API? What's still not there yet?

Feels like we're at an interesting inflection point but hard to cut through the noise on what's working v/s just setting it up

Local is best when you're iterating fast on prompt templates or need to run the same task hundreds of times—API costs add up quickly there.

For a long time it has been a win to do clustering, retrieval, ranking, classification, anomaly detection and other non-generative tasks with local models from the BERT family. These are so much better than all the things I tried from 2000-2020. ModernBERT for feature generation plus classical algorithms from scikit-learn works so well, plus scikit-learn has great tools for eval, model selection, probability calibration and all the other things that make the difference between "write another arXiv paper" vs "deploy a product into production"

If you read the splogs you'd think it is all about the "zero-shot" scenario where you write a prompt asking what decision you want the model to make. Yeah, it is amazing that it does so well with so little input, but side by side "many-shot" models that learn a decision surface in hyperspace beat the pants off "few-shot" models that look at nearest neighbors and "zero-shot" models that do whatever they do. It's not even close.

The reckoning for generative AI may near and it might be like something right out of "Where's your Ed at?" When people see honest prices for current models I think a lot of people might decide the juice isn't worth the squeeze. The story in the next few year is not going to be bigger models that only Elon Musk models but a relentless fight to get costs down.