Cool! So for marginally less in cost and power usage than the numbers I quoted, you can get 2 more machines than with the RDMA setup. And you’ve still not solved the thing that I called out as the most important drawback.

▲

nicky_nickell 15 hours ago | parent [-]

how significant is the latency hit?

▲

angoragoats 15 hours ago | parent [-]

The OP makes reference to this with a link to a GitHub repo that has some benchmarks. TCP over Thunderbolt compared to RDMA over Thunderbolt has roughly 7-10x higher latency, ~300us vs 30-50us. I would expect TCP over 200GbE to have similar latency to TCP over Thunderbolt.

Put another way, see the graphs in the OP where he points out that the old way of clustering performs worse the more machines you add? I’d expect that to happen with 200GbE also.

And with a switch, it would likely be even worse, since the hop to the switch adds additional latency that isn’t a factor in the TB5 setup.

▲

wmf 14 hours ago | parent | next [-]

You're ignoring RoCE which would have the same or lower latency than RoTB. And I think macOS already supports RoCE.

	▲	angoragoats 4 hours ago \| parent [-]
		MacOS does not support RoCE.

▲

Hikikomori 5 hours ago | parent | prev [-]

Switch probably does cut through so it starts forwarding the frame before its even fully received.