| ▲ | schmorptron 4 hours ago |
| It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably |
|
| ▲ | batperson 4 hours ago | parent | next [-] |
| The future of inference is likely in ASICs, so we'll get the inverse, a bit less capable than frontier but super fast models. Like this 14k tok/s beast https://chatjimmy.ai/ from Taalas (who got acquired by AMD recently). GPT-6-astra runs at like ~40 tok/s, I have a hard time imagining what could be accomplished with that type of model at 10k+ tok/s when in the hands of the public. Will certainly make cybersecurity a challenge for older systems. |
| |
| ▲ | 2 hours ago | parent | next [-] | | [deleted] | |
| ▲ | SPascareli13 2 hours ago | parent | prev | next [-] | | Like how crypto used ASICS but then didn't because the scaling of consumer hardware made it obsolete? | | |
| ▲ | wtallis 2 hours ago | parent | next [-] | | To the extent that cryptocurrency moved off ASICs, it was because of interest shifting to different cryptocurrencies that were specifically designed to be harder to mine on an ASIC than Bitcoin's compute-heavy, memory-light hashing. I'm not sure there's any reason to expect a similar shift from LLMs. The hardware used for training doesn't dictate what hardware needs to be used for inference, and nobody's going to design an LLM architecture with an overt intention to make it better suited to GPUs and hard to target with ASICs. | | |
| ▲ | SPascareli13 17 minutes ago | parent [-] | | Yet it doesn't seem that ASICs will have any particular advantage over consumer hardware since AI is very memory heavy, which is (right now) expensive no matter how you package it. And the compute is just simple matrix multiplication, which is almost entirely what GPUs were meant to do anyway. |
| |
| ▲ | actionfromafar 2 hours ago | parent | prev [-] | | Am I missing some joke here? |
| |
| ▲ | domhudson 4 hours ago | parent | prev [-] | | This is incredible! Are there other big players in this space (freezing models to silicon)? | | |
|
|
| ▲ | redox99 4 hours ago | parent | prev | next [-] |
| At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize. |
| |
| ▲ | Izmaki 4 hours ago | parent | next [-] | | This is what people forget when they see slow performance: at 1 t/s it's still roughly the equivalent of having another person work for you at no extra cost besides the initial purchase/sign-on-bonus. Frontier models are amazing, but what will really be useful for us is having models and hardware so efficient that you can run useful LLMs locally. One of my favourite LLMs to this day is still my jail-broken gemma4 12b because it's small enough to run on my computer, but also 100% local and free as in liberty. | | | |
| ▲ | mhaberl 3 hours ago | parent | prev | next [-] | | I would agree, but I want to add that I have real issues with combination of opencode plus slow inference (4-5tok/s). I get weird interruptions. I can only guess its related to some kind of timeouts in the harness or something. Its not a problem of the model of course, but it seems impractical atm. I wonder if anyone else had this kind of thing happening. | | |
| ▲ | ygjb 2 hours ago | parent | next [-] | | For OSs harnesses this seems like a good thing to point a paid model at fixing if you don't want to dig in yourself? | |
| ▲ | redox99 3 hours ago | parent | prev [-] | | I think pi handles it better |
| |
| ▲ | Capricorn2481 3 hours ago | parent | prev [-] | | > At 1t/s it's still faster than humans for a lot of tasks Which tasks? I think you're underestimating how token hungry current proposed workflows are. | | |
| ▲ | redox99 3 hours ago | parent | next [-] | | Anything you do right now? A typical 10 minute prompt "simply" becomes about 7 hours long. (40t/s vs 1t/s). | | |
| ▲ | Capricorn2481 3 hours ago | parent [-] | | I don't really do anything at my job that could be done in a single prompt, and certainly not something that would take me 7 hours. A 7 hour task would take multiple iterations with how LLMs are right now. You said it would do overnight what would take a human half a week, so I'm curious what tasks you are doing where AI is 3x faster than you even at 1t/s | | |
| ▲ | redox99 3 hours ago | parent [-] | | Your prompts are probably very underspecified then. Frontier models one shot the majority of my prompts. UI is kind of the exception, there I do have to ask for a lot of tweaks. | | |
| ▲ | Capricorn2481 3 hours ago | parent [-] | | Maybe. Do you have an example of a prompt you've done recently? I don't think it's valuable at all to try and craft a specific prompt to "one-shot" a task, I'm way faster just doing small asks and guiding it. At 1t/s, how would you even know your prompt was insufficient? It would take you 7 hours to see it. | | |
| ▲ | redox99 3 hours ago | parent [-] | | Stuff along the lines of implement controller service and tests for the following endpoints: - list of many endpoints with the JSON they receive and return and description of what they need to achieve Stuff you could probably do in a single work day if you lock in and enter flow state, but in a typical job takes like half a week. And the vast majority of times the AI one shots it with no bugs, where I would have copy paste errors or dumb stuff I'd need to fix before it's shippable. | | |
|
|
|
| |
| ▲ | coldtea 3 hours ago | parent | prev [-] | | Doesn't matter which task. Compare it with about 40-50t/s an LLM oneshots with, and it, and whatever task now takes X time, takes X * 40-50 with this. |
|
|
|
| ▲ | KumaBear 24 minutes ago | parent | prev | next [-] |
| running my own locally. I just set the tasks to start when systems go idle over x. Read and copy only to external drive projects, codes, ect for review. I review the reports the changes and apply them myself or correct them. Is it slower than say throwing it into fable yes. But I don't have to be monitoring it 24/7 |
|
| ▲ | IgorPartola 4 hours ago | parent | prev [-] |
| My parents learned to program on punch cards. They told me it was a day of preparing the program, an hour of running it, just to get a syntax error. |
| |
| ▲ | 4 hours ago | parent | next [-] | | [deleted] | |
| ▲ | bluedino 3 hours ago | parent | prev [-] | | Write the program, punch the cards, send the cards to another building to be loaded, program runs, printout comes out in another building, somehow this takes 2-3 days | | |
| ▲ | lurker919 3 hours ago | parent [-] | | Coding is the new punch card slots now. My children will listen in awe about how typing and testing used to take hours or even (gasp!) days. |
|
|
