| ▲ | walrus01 7 hours ago |
| My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much. edit: I think this is a fantastic project in general concept, and look forward to seeing more efforts towards the general idea of being able to run a 350B to 900B size model locally, even if as slow as 1 tok/s, on hardware that ordinary people can afford. Anything along the general concept of "we have fast read NVME SSD storage, we have a big ass model on local disk, we'll read it at 11GB/tok as we need it, not try to load the whole thing". |
|
| ▲ | JumpCrisscross 4 hours ago | parent | next [-] |
| The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028. (I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.) |
| |
| ▲ | walrus01 2 hours ago | parent | next [-] | | For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise. Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add. | | |
| ▲ | Abishek_Muthian 8 minutes ago | parent [-] | | Which CPU gen are you suggesting, is there any writeup on such setup where <10K (not incl. power bill) cpu only rig is giving usable token speeds on latest SoTA open weights models? In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment. Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control. Kudos to OP though, They had clear goals and they achieved it. |
| |
| ▲ | paytonjjones 3 hours ago | parent | prev [-] | | Today's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say. | | |
| ▲ | anon373839 3 hours ago | parent | next [-] | | This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway. | | |
| ▲ | tharkun__ 2 hours ago | parent | next [-] | | Caveat: I have not been able to try that model locally, so no personal experience. Running this locally at usable speeds would be cost prohibitive for personal coding use for me. But if we can believe you that it's doing what a Claude model was doing a year ago then I'd say: OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do. | |
| ▲ | 3836293648 3 hours ago | parent | prev | next [-] | | Could you expand more on what you do with qwen3.6? Because I couldn't get the denser 27B version to do trivial "take this pattern, repeat it over a single file with minimal thought, just slightly beyond what I can do with sed" reliably. | | |
| ▲ | kgeist 3 hours ago | parent [-] | | How was qwen3.6 launched? The thing is, everyone has their own variant of "qwen3.6 27b" depending on the launch parameters, ranging from "SOTA in its class" to "completely broken" |
| |
| ▲ | dyauspitr an hour ago | parent | prev [-] | | > I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway. What could be more valuable than outputting the exact thing you asked for? |
| |
| ▲ | stingraycharles 19 minutes ago | parent | prev [-] | | Looking at how critical we are about today’s models, vs where we were last year, and I don’t expect anyone to be content with Fable-class models in 2028. Expectations seem to be rising at a faster rate than models can improve. |
|
|
|
| ▲ | codazoda 5 hours ago | parent | prev | next [-] |
| I’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me. |
| |
| ▲ | alexhans 4 hours ago | parent | next [-] | | Having a thin python/ts orchestrator and workers that pick up tasks from the directories like events and decide whether to make deterministic calls and wait is pretty standard albeit custom way of doing things in this space where you're bottlenecked by the concurrent call your workers/agents can make. The hard thing is always keeping complexity low and being ZeroOps. | | |
| ▲ | antondd 2 hours ago | parent [-] | | Are there any frameworks/scaffolding/harnesses or general resources on this you can share? I’d love to learn more | | |
| ▲ | walrus01 2 hours ago | parent [-] | | Most any ticketing system can integrate with ordinary IMAP and smtp email flow, so you can really use any agent that can "do" inbound and outbound email to talk to a self hosted ticket queue. |
|
| |
| ▲ | digitaltrees 2 hours ago | parent | prev [-] | | This is actually really smart. It would be like working with a team of humans. I have a 3 Mac Studio set up and built an IDE / harness (propelcode.app) and would be interested in contributing if you’re open to collaboration |
|
|
| ▲ | vforno 7 hours ago | parent | prev | next [-] |
| In the readme you can see benchmark which everyone with different hardware is running Colibrì, and I have to say I've seen great times! I'm always doing more to improve! |
| |
| ▲ | walrus01 7 hours ago | parent [-] | | I have a 16-core system with 256GB RAM here I could try it with but regretfully it's so old the CPUs aren't AVX2 capable. Otherwise it makes a fairly good llama-server test system for CPU only stuff. Oh well. Time to upgrade (painful to the wallet these days). | | |
| ▲ | vforno 7 hours ago | parent [-] | | Maybe we can see some integration! | | |
| ▲ | fuzzfactor 4 hours ago | parent [-] | | If you get good at extracting remarkable performance from the most lesser of instruments enough to pull their own weight regardless, just imagine what it can be like when such a practitioner gets behind the keyboard of a world-class Steinway. And just does what they do best. Without ever having touched such a capable instrument themself. On a level playing field the expression of virtuosity can outshine those who have never known any instrumental limitations at all :) When pulling way more than your own weight happens like for few others. There should be an award for getting the most out of the electronics rather than trying to reach orbit by building the tallest pile of e-waste. First Prize right before your eyes ! Grande praise ! And just starting to ascend toward an unconquered summit that others find forbidding ;) Or they find uninteresting since the limit naturally lies on firm earth somewhere below the stratosphere. | | |
|
|
|
|
| ▲ | charcircuit 5 hours ago | parent | prev | next [-] |
| For most projects the more practical solution is to use clouds offering GLM 5.2 for free. 1 token per minute is minuscule compared to their rate limits for free usage. |
| |
| ▲ | joerawr 3 hours ago | parent [-] | | But it's about the journey not the destination. My current running local LLMs train of thought... |
|
|
| ▲ | bigiain 5 hours ago | parent | prev | next [-] |
| > on hardware that ordinary people can afford These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd? sigh |
| |
| ▲ | 3836293648 3 hours ago | parent | next [-] | | Maybe not afford new, but they probably already had it from before the current crisis? | |
| ▲ | walrus01 4 hours ago | parent | prev | next [-] | | The very boring pair of two 16GB ddr5 6000 I had in my newegg shopping cart went from $399 to $475, so increasingly the answer will be "no". | | |
| ▲ | bigiain an hour ago | parent | next [-] | | I bought whole Intel N100 mini pc with 16GB of DDR5 in it in 2023 for $AUD289 (so about $US200). I got a 16GB (DDR4) SODIMM in 2022 for $AUD88 ($US60). | |
| ▲ | fuzzfactor 4 hours ago | parent | prev [-] | | Maybe that's a measure of the self-fulfilling dollar incentive toward "renting" someone else's RAM in the future rather than trying to actually own such an outlandishly luxury item :\ |
| |
| ▲ | fuzzfactor 4 hours ago | parent | prev [-] | | Ideally this engineer's approach will yield better performance on lesser equipment in the future, if they keep up the good work after they get more-capable gear to experiment with as time goes by :) |
|
|
| ▲ | joshsantiago01 5 hours ago | parent | prev [-] |
| [flagged] |
