> running it on a no GPU, but tons of RAM server

Or from SSD using something like Colibri[1]. Not going to be quick, but at least runable.

[1]: https://github.com/JustVugg/colibri

It's a great concept but I think it would cross the line from 'very slow' to 'so slow it's unusable' at this size. Even if we say you have an NVME SSD that does 7GB/s reads, that's dramatically slower than being able to hold the whole thing in DRAM. Like the difference between 1.3 tok/s in RAM vs 0.1 tok/s with a colibri-like method.

edit: the results I have seen from people trying colibri with fast consumer grade PCI-E 4.0 NVME SSD are 0.1 tok/s on models that are <700B in size, things that are well under 800GB on disk. With something that's 3T in size it'll probably be a lot slower than hat.

▲

zozbot234 4 hours ago | parent | next [-]

For single stream inference of a MoE model, the size of active sparse parameters will matter a lot more than total parameters. This is generally around half of the reported active parameter count - the other half being a dense subset that can be easily cached in VRAM even on fairly modest consumer setups. So the achievable performance may be quite a bit better than a naïve assessment might suggest.

▲

magicalhippo 6 hours ago | parent | prev | next [-]

It claims to support using multiple devices RAID-0 style, which should boost performance, but yea probably not very useful for most.

But still fun you can run it at home.

▲

LtdJorge 4 hours ago | parent | prev [-]

On a server machine you can have more than 100GB/s of NVMe if you parallelize (RAID 0 and the like). But it's still gonna be noticeably slower.

	▲	walrus01 4 hours ago \| parent [-]
		1536GB of DDR4 ECC server RAM is somewhere between $4000-6000 USD used right now, by the time you put in parallel enough NVME SSD to approach good speeds, you'd be approaching that (and also likely running out of PCI-E bus lanes directly attached to the same motherboard to reasonably do so).