Remix.run Logo
marcuskaz 5 hours ago

The 256GB option is +$4,000 - the overall price for 512GB setup would probably be $20k!

notnullorvoid 4 hours ago | parent [-]

It needs to be a little less than double the overall price for 256GB option, otherwise it's better to get 2 256GB and link them.

Zylokloto 4 hours ago | parent [-]

you can't link onchip memory.

notnullorvoid 4 hours ago | parent [-]

True, but you can link them up over thunderbolt or Ethernet. If your goal is to run local LLMs, not all weights need to live on the same computer. You can segment the workload by layers and pass the activations along the lower bandwidth interconnect with a small perf penalty. Also you get double the CPU/GPU cores allowing for better multi user/agent performance.

Zylokloto 3 hours ago | parent [-]

Ah sorry you are right I missread what you meant by it.