Curious about the >2 case. You get a Quantum X800 switch? Or just a fast Ethernet switch? These are quite cool because of the amount of VRAM on them (though I find the “unified RAM” marketing unnecessarily confusing) but I avoided them because I wasn’t sure I could network them. A B300 node is $500k but I think I’d do MI355X now at that size because new Claudes will be able to port easily.

The advantage of the station is gradual improvement. Though tbh I wonder if I should just settle for putting a few more normal H100 NVL and trying RoCE. Post is at least inspiring of ideas if nothing else.

There is something for macOS that uses Thunderbolt for high speed interconnect. Obviously it doesn't scale beyond a few nodes but I think that's a decent and fairly approachable middleground

Edit: This github.com/exo-explore/exo