These people have zero idea what they're doing. Not a single mention of pipeline parallelism that would actually make the setup useful to run a big model.

▲

itkovian_ 4 hours ago | parent | next [-]

I can’t stand it. Very engineering-y over specified formal language around a complete lack of core understanding. Is damaging other people read this and try to learn things from it.

▲

schaefer 4 hours ago | parent | prev | next [-]

can you point to a write up that discusses what you're talking about?

because I would read it.

▲

kelmoran 4 hours ago | parent | prev [-]

I feel like you would want to run 8 smaller models separately for quantity of raw output. 1 big model is slow and isnt guaranteed to make no mistakes.

	▲	monster_truck 4 hours ago \| parent \| next [-]
		That's not quite how it works. Throwing Deepseek V4 Flash on 4 of these would net you something like >200tk/s for 16 concurrent requests, that's 600 million _output_ tokens a month. Guess what happens when you use 8
	▲	teaearlgraycold 4 hours ago \| parent \| prev [-]
		Doesn't really mean anything without a specific use case to guide model selection. The thing is Qwen 3.8 27B can be ran on far far cheaper hardware. If you're spending the big bucks on these rigs you probably made the wrong choice if you aren't using models that require all of that VRAM.