| ▲ | tripleee 2 hours ago | ||||||||||||||||
By on device you mean running locally on a mobile device like an iPhone? If so we're a really long way from a chatbot that's engaging to talk to. There's just not enough processing power to do it No-one's going to want to talk to a 7B model for more than 3 minutes | |||||||||||||||||
| ▲ | post-it 42 minutes ago | parent | next [-] | ||||||||||||||||
I think we're a breakthrough away. I don't think processing power is an issue. A 7B model has a lot of cruft (like knowledge of code syntax) that a character sim doesn't need. It should be possible to extract just the relevant parts of a larger and more powerful model and run it with the same resources you would run a general 7B model. We just don't have the ability to isolate and extract weights like that yet. | |||||||||||||||||
| ▲ | Tuna-Fish 2 hours ago | parent | prev | next [-] | ||||||||||||||||
Yes, current gen hardware was not built to run such models, and are obviously inefficient and bad at it. Significant investment is being spent on producing hardware that is more suitable for running mid-sized (100B-ish) models. Starting with flash chips with wide enough interfaces to provide the weights (NAND flash can be ~5x more energy-efficient for reads than DRAM), and going into more exotic solutions like compute-in-memory. | |||||||||||||||||
| ▲ | toomuchtodo 2 hours ago | parent | prev [-] | ||||||||||||||||
Yes, iphone, ipad, mac. Maybe your phone makes the inference requests to either Apple servers (requiring iCloud+ and credit based?) or your Mac (Tailscale equivalent). My assumptions are that inference costs continue to decline (and/or the difficulty level of routing requests declines) and hardware acceleration continues into the future. These assumptions might be wrong, but that is the idea the thesis is based around. (have built workflows and harnesses that currently target LLM provider APIs under the assumption I'll be able to point "locally-ish" eventually, my career had a strong focus on distributed computing about 10-15 years ago, think managing data and work clusters for LHC detector data processing, and I grew up on hobbyist distributed.net and BOINC projects, all the lenses I see this problem through; queues, jobs, sync vs async performance optimization, etc; recovered sysadmin|infra/devops engineer a lifetime ago) | |||||||||||||||||
| |||||||||||||||||