Is there an actual use for this in a homelab? Isn't 25gbps enough that no SSD will saturate it?

Please be careful with units. There are high-end consumer SSDs that will deliver data at 14+GB/s, which is much higher than 25Gb/s.

▲

MarkSweep 2 hours ago | parent | prev | next [-]

The second sentence of the article:

> We have a number of these now and have been running our RDMA network backend for the 8x NVIDIA GB10 cluster for several months now, using one.

I have a friend who uses something similar for his 4 NVIDIA GDX Spark cluster.

▲

avidiax 2 hours ago | parent [-]

I see, so only if you are running an AI cluster. And what is it transferring between the nodes? Partial computations?

▲

ericd 2 hours ago | parent [-]

Essentially, yes. The DGX Sparks, at 128 gigs for ~$4700 are one of the cheapest ways to get enough high-ish speed memory cobbled together to run one of the more capable open weight models to run at home/for a small biz. This switch lets you combine the memory of 2-4 of them.

	▲	bitbckt 2 hours ago \| parent [-]
		4-8 of them. 2 Sparks can operate without a switch.

▲

toast0 an hour ago | parent | prev | next [-]

I purchased my SSD based on price and I have a AM4 APU so I'm limited to pci-e 3, but a quick check with crystaldiskmark says I'm hitting about 3.6 GB/s on sequential reads in seq1m q8t1 ... That would saturate 25gbps (assuming you have the right access patterns). seq1m q1t1 does 2.3 GB/s on my system, close enough to 25 gbps.

By my math, pci-e 5.0 should do 4x the throughput, so if you had 4 pci-e ssds you wanted to serve over the network and you had the pci-e lanes, you could saturate 400 gbps.

I'm pretty happy with my little baby network with a handful of 10G and 2.5G ports, and I'm cheap so if I made a big upgrade it'd probably be something used that has more 10g ports plus a 40G I could DAC to the adjacent home server ... but my home servers are barely powerful enough to push 10g anyway. (Something like a DCS-7050TX-48 goes for ~ $100 and would be a big upgrade in capacity I don't use over the S3300-28X I currently run)

▲

hgoel 2 hours ago | parent | prev | next [-]

DGX Sparks have 200Gbps interlinks for clustering. Apparently switches are the only practical way to scale such clusters beyond 2 nodes even though each Spark has 2 such ports.

	▲	FireBeyond 42 minutes ago \| parent [-]
		Well, the problem is that you want each node to be able to directly connect to each other, so you run into a mesh situation. https://www.youtube.com/watch?v=QJqKqxQR36Y shows 8 running at 100gbps and clustered.

▲

FireBeyond 44 minutes ago | parent | prev | next [-]

Yes, AI... this video does a good job walking through: https://www.youtube.com/watch?v=QJqKqxQR36Y

(tl;dw - guy connects first four DGX and then another 4 "DGX-alikes". Uses 400gbps breakout cables (you can split interfaces, TIL) so he has 2 ports on the first switch, and then gets this one, I believe. You can buy cables that are 1x400 to 4x100gig, etc.

▲

corvad 2 hours ago | parent | prev [-]

I mean at lot of what a homelab is experimenting. For example I have an absurd amount of communication 56GbE Mellanox switches but never really saturate it.

	▲	bobdvb 27 minutes ago \| parent [-]
		Yup, I have a pair of CRS504, so I can learn MLAG and my three node cluster doesn't really tickle the sides of that.