Can I ask your setup? I’ve been tasked with taking a beefy machine (vague on details on purpose, the specs are legit) and trying to figure out how to run on-prem inference using open models. Gated in that I cannot use non-US models. I don’t think I have a handle on how to set it up for multiuser use without nuking TPS or context length. The current power users set and forget tasks overnight and expect results in the morning. Which… I have mixed feelings about, but that is the bar set by paying all the money for GitHub copilot enterprise.

Any tips?

▲

nejch 13 hours ago | parent | next [-]

With setup you mean HW or the SW stack?

We used to run GLM-5 class models but have now changed to smaller ones as we're able to serve more concurrent users with our limited hardware (DeepSeek-v4-Flash-0731, Qwen-3.8-27b). We run on the order of hundreds of parallel requests right now, Qwen with data parallelism and DSv4F with P/D disaggregation, but will probably continue to tweak this.

Qwen 3.8 27b in my experience is more than capable of churning out features overnight with the right tools (don't rely on its world knowledge, give it tools like playwright and github MCP for upstream context and search, and give it goals to work on). Bonus with another model like DSv4 as an adversarial reviewer.

▲

kouteiheika 18 hours ago | parent | prev [-]

> Gated in that I cannot use non-US models.

This is incredibly silly. First I would try to convince whoever has decreed this to reconsider. The only SOTA open-weight models are non-US, and rather than being afraid of potential Chinese backdoors secretly planted in open weight models you should be afraid of LLMs simply just doing something dumb. To give you an analogy: no one has ever demonstrated a backdoor in a popular model which maliciously runs 'rm -Rf *'; plenty of people have encountered a model running 'rm -Rf *' by mistake. You need to protect yourself against the model doing something stupid anyway, and, paradoxically, a US open-weight model makes this more likely due to them being weaker.

If you need a precedent then you can cite Mistral[1] hosting GLM-5.2 on their platform, or plenty of US-based providers on Open Router hosting Chinese models.

[1]: https://docs.mistral.ai/models/zai-glm-5-2

▲

irishcoffee 6 hours ago | parent [-]

Yeah look, I don’t have a say in it. Also, the whole Chinese putting backdoors into routers hasn’t helped any.

https://www.reuters.com/world/asia-pacific/researchers-disco...

	▲	kouteiheika 4 hours ago \| parent [-]
		Did you actually try to convince them? Again, this requirement is incredibly silly. Here, look at this (it's a link to AA; I shortened it since the link's huge): https://tinyurl.com/2dhzdrt6 Right now, the best US-based open-weight model (Inkling) has 42 points and is 975B parameters. The best Chinese open-weight model has 60 points. Also, note, Qwen3.8-27B on low thinking beats Inkling here and gets 43 points. Of course, AA scores are definitely not bulletproof and there is a certain amount of benchmaxxing going on, but the point stands -- it is incredibly silly to limit yourself to US-only models, just because of an imagined threat that doesn't exist. Let me state this in stronger terms: whoever decided this for you is clueless. They might have the authority to override you, but it's your job to tell them that the only thing this achieves is essentially pointing a gun at your own foot and pulling the trigger.