| ▲ | thewebguyd 2 hours ago |
| Local AI was/is bound to happen, eventually. It'd be smart of Nvidia to get ahead of it. Non-techy consumers may never do it, but at some point businesses are going to start asking when do they stop paying per token and start running models themselves. Right now the hardware is cost prohibitive, but I doubt that'll always be the case. Eventually the hardware will get cheaper and more available, and Nvidia seems to be betting on that. They don't care where inference happens, so long as it happens on Nvidia hardware. |
|
| ▲ | h14h 2 hours ago | parent | next [-] |
| IMO it's only a matter of time before "self-hosting local AI" is as complicated as installing an app and clicking a download button. And when that happens, the pitch to non-techy users is "Free ChatGPT you can use offline with zero privacy risk". Once hardware accessibility and LLM efficiency advance to the point that this becomes feasible, I suspect it'll result in a much bigger hit to the cloud AI market than many expect. |
| |
| ▲ | ribosometronome 15 minutes ago | parent | next [-] | | That workflow has been around for awhile now. I'm sure there are others but LM Studio has a model browser in app that effectively simplifies things to hitting download and hitting launch. The complexity tends to be in that there's a lot of models to choose from and also knowing how to set up whatever tool you're using with a local model. None of it's particularly hard, unless you start trying to customize settings. I think the bigger hang up is that they're still slower and less capable than the frontier models, especially at the hardware specs most home users are likely to have. | |
| ▲ | adamrezich 2 hours ago | parent | prev [-] | | Why is it only a matter of time? The AI-as-a-service companies are going to continue to improve their products by improving both the part that could be reproduced in a self-hosted setup, but also the “secret sauce” they put on top of that to make it a better product. There is no incentive for this “secret sauce” to be something that can be reproduced for self-hosting, is there? | | |
| ▲ | h14h an hour ago | parent | next [-] | | I think a major incentive could be to sell hardware. If Apple is able to get their hands on a local LLM capable of covering a significant % of what people use ChatGPT for, the pitch they can offer is: "Free, private, offline ChatGPT so long as your laptop has X GB of RAM" Beyond that, I wouldn't underestimate the incentive of "because I can". The "secret sauce" you refer to is effectively just a DB & a while loop that feeds text to a bunch of tensors. If an indie dev decides they want to release something that dismantles the OpenAI & Anthropic moats, there really isn't all that big of a technical barrier stopping them. | | |
| ▲ | bigyabai an hour ago | parent [-] | | LLM inference decode is heavily dependent on memory speed, not just having lots of memory. You can't say "X amount of ram" because the memory bandwidth on an M1 is 68.3 GB/s versus the 614 GB/s of an M5 Max, or a 4090's 1.01 TB/s over GDDR6X. This basically creates a bottleneck at the oldest/cheapest Apple Silicon machines, which are already crippled for context prefill. | | |
| ▲ | h14h 34 minutes ago | parent [-] | | Thanks for clarifying -- I was oversimplifying. But honestly, obsoleting a huge number of otherwise great Apple Silicon machines is something Apple would moment consider a major "pro" of building a compelling local AI stack. With how much speculation around the difficult time Apple has had getting people to upgrade from M1, I'm sure they'd jump at such an opportunity. | | |
| ▲ | bijowo1676 10 minutes ago | parent [-] | | this might be a way for Apple to milk product revenue for many years. - Please buy our new Macbook pro M5 that gives you 20 tokens/s on local 80B LLM next year
- Please buy our new Macbook pro M6 that gives you 25 tokens/s on local 80B LLM milking product revenue in perpetuity by offering meaningful marginal improvements, while keeping same architecture will be the golden goose for Apple +plus if it allows to segment market by wallet size into poor/middle/rich classes, thats even better |
|
|
| |
| ▲ | thewebguyd an hour ago | parent | prev [-] | | What secret sauce? We already have open source tooling for tool use, web browsing, and code execution/computer use. Open weight models will win in the end. AIaaS might keep an edge with multi-modal agentic workflows, but for 80% of general use cases, no "secret sauce" needed, the open weight models are already there, and tooling is constantly getting better. The bottleneck is the cost of local hardware right now. |
|
|
|
| ▲ | artyom 2 hours ago | parent | prev | next [-] |
| I'm from the times when you had to purchase a separate chip to perform floating point math. It was called a math co-processor. [1] After a few generations (and over a decade) that was indistinguishable from the CPU chip itself. It's a long hyperbole, I know, but I think local inference is inevitable; and the big fishes know it. Will that be a complex technical setup? An appliance? An additional chip in your motherboard? So transparent it's burned right into the CPU? Those are just implementation details. We're probably just one generational breakthrough away from it. [1] https://en.wikipedia.org/wiki/X87 |
| |
| ▲ | postalrat an hour ago | parent [-] | | Like the math co-processor it might end up just being new instructions for the cpu to handle ai related math. |
|
|
| ▲ | smrtinsert an hour ago | parent | prev [-] |
| > Non-techy consumers may never do it They will. As some point in the future, people will want everything, they'll prompt full movies because they're bored and want to watch something. |
| |
