make it 4x40 raid-0 ssds to achieve 40 tps.

or 40 macbooks with each 4 ssd. to get 40 tps.

npn a day ago | parent | next [-]

yeah raid or not you still get the hard limitation by the pcie lanes

it is even worse with 40 macbooks.

if 40 macbooks is all that take to serve a 1TB model with decent speed then you would see everyone selling the models for very cheap right now.

▲

Argonautlabs a day ago | parent | prev [-]

Bandwidth doesn't multiply like that here, and we measured it rather than assumed it. A MoE layer needs 16 expert reads and can't proceed until the slowest one lands, so a layer costs the max over its reads, not the sum.

Going from one drive to four (13.6 → ~33 GB/s of combined ceilings) took decode from ~52% to 100% of our number — not 4× — with

Every drive already at 90–100% of its own ceiling. RAID-0 was one of the first things tried and it lost: striping makes every read touch every drive, so the slowest drive sets every barrier.

What moves this is per-read latency and read scheduling, and for long prompts not re-reading each layer's experts eight times.

Numbers in results/SCALING.md and results/PREFILL.md.

▲

lowbloodsugar a day ago | parent [-]

Would the 40 Mac’s work with pipelining though?

	▲	adrian_b 16 hours ago \| parent [-]
		No matter how many external drives you gather, the data coming from them must be squeezed through the peripheral interfaces of the Apple SoC. So your CPU, made by Apple, Intel, AMD etc., has a number of PCIe lanes and a number of USB/Thunderbolt ports for connecting peripherals. Those have an aggregated throughput, which sets an upper limit for the amount of data that can be read per second from all the peripheral devices. In a given computer, usually not all the lanes and ports of the CPU are actually connected, so the limit may be even lower. In desktop PCs and mini-PCs, usually only 4 + 4 = 8 PCIe lanes are available for SSDs, and when there are more SSD sockets they share some of those lanes. A much higher SSD throughput could be achieved in a desktop PC by using the GPU connector with an SSD adapter for M.2 SSDs, which has 16 PCIe 5.0 lanes, with a 64 GByte/s throughput. Taking out the GPU might actually be OK for doing AI inference, because a beefy CPU like a Ryzen 9950X should be able to keep up with a reading throughput of 88 GB/s from 6 SSDs (2 on the motherboard and 4 on the add-on PCIe card), while computing inference in the INT8 or BF16 formats, so the absence of the GPU would not reduce the inference speed when it is limited by the speed of reading the weights.