Woah, this is exciting. I'm traveling but I have a 5090 lying around at home. I'm eager to give it a go. Docs are here: https://docs.tinygrad.org/tinygpu/

I hope it'll work on an M4 Mac Mini. Does anyone know what hardware to get? You'll need a full ATX PSU to supply power, right? And then tinygrad can do LLM inference on it?

▲

999900000999 17 hours ago | parent | next [-]

You can buy a cheap GPU enclosure for about 100$ off ali express.

Takes a standard PSU. However, Mac Minis don't have occulink. So you might be a bit limited by whatever USB C can do.

Now if Intel can get there Arc drivers in order we'll see some real budget fun.

https://www.newegg.com/intel-arc-pro-b70-32gb-graphics-card/...

32 GB of VRAM for 1000$. Plus a 500$ Mac Mini.

▲

Fnoord 17 hours ago | parent | next [-]

Those $100 ones don't come with a cage. If you do want a cage, you'll end up with $180 in total, with zero warranty.

Article mentions: "Apple finally approved our driver for both AMD and NVIDIA"

Does not mention Intel (GPUs). Select AMD GPUs work on macOS, but...

Macs (both Intel and ARM) support TB, but eGPU only work on Intel Macs, and basically only with AMD.

Good news is for medium end gaming choices are solid, and CUDA works on AMD these days.

▲

999900000999 16 hours ago | parent [-]

Fortune favors the bold my friend.

I own one of these, the cage is just a piece of plastic. Anyway, I don't think 80$ is that big of a difference here. I can't really afford a 4k Nvidia GPU. Intel is my only hope.

▲

Fnoord 16 hours ago | parent [-]

Almost twice the price and simply more accurate info regarding price and features.

Brand is TH3P4G3. Egpu.io has decent eGPU comparisons.

I wouldn't want all that dust in my GPU fans, prefer that near my case fans. I also don't like it given I got cats and want to store/box hw. I do use the eGPU in the fuse box. If I had a larger house, I'd use a server rack.

I was recently in the market for an eGPU but for a different niche (not eGPU/eNPU/eTPU but getting a HBA via TB to connect a LTO-6 drive via SAS). I went for a Sonnet instead, very low profile and small. I also bought an Asus one. Slightly bigger, came with more fans but TB4 instead of TB3 on the Sonnet. The cages are aluminium. Those eGPU were second hand (also without warranty but quicker S&H than Chinese New Year) but came with PSU. As you also gotta buy a PSU for it which came with the eGPUs I mentioned. For me no biggie, as I got a decent PSU lying around.

▲

jasomill 14 hours ago | parent [-]

I've been using a Sonnet eGPU box with Nvidia GPUs (1070/3070) on an Intel NUC for about 5 years, and it works great.

One nice thing about the Sonnet eGPU boxes is that they use standard SFX PSUs that are inexpensive to replace if they fail.

For LTO, I'm cheap, and iSCSI over a dedicated 2.5 Gbps Ethernet link is fast enough for my aging FC LTO-5 drives and spinning rust backup disks.

	▲	serf 14 hours ago \| parent [-]
		I used Sonnet egpu box on a similarly equipped Dell XPS and it had so many little issues that it sold me off of eGPUs over Thunderbolt entirely. Sleep broke across all OSs, if sleep didn't break the GPU wouldn't get powered on with the laptop. If one side lost power during an outage (the gpu side, the laptop has a battery..) it would require an elaborate voodoo ritual of cycling both of them on and off until they 'caught' each other. It would cause the rest of the USB ports on the laptop to reset and drop comms with peripherals once or twice a week, necessitating a rain-dance restart. when Oculink first started showing up I gave up all together and just said "fuck it i'll try it again in a few years.". It worked fine when it worked fine, but the patches in between were not worth my time. I blame Dell and their thunderbolt controllers entirely for the issue, but it left such a bad taste in my mouth that I would have a really tough time buying the newest Sonnet box to try it out. Now I have a desktop machine and don't fall into that market. I ended up throwing that card (an rtx 3xxx) into a dell rackmount and have been happy with that card ever since. to your point though: the non proprietary PSU was a nice feature, but in reality the expansion card for PCI->Thunderbolt or whichever interface you're using can be bought on alibaba for like 20-30 bucks and the PSU is worth another 30-40 bucks , a generic white-label 650w. I think if I did it over i'd just do that and make an enclosure, but the Sonnet boxes aren't too bad a value by the numbers.

▲

16 hours ago | parent | prev [-]

[deleted]

▲

manmal 17 hours ago | parent | prev | next [-]

Maybe I’m lacking imagination. But how will a GPU with small-ish but fast VRAM and great compute, augment a Mac with large but slow VRAM and weak compute? The interconnect isn’t powerful enough to change layers on the GPU rapidly, I guess?

▲

zozbot234 17 hours ago | parent | next [-]

> But how will a GPU with small-ish but fast VRAM and great compute, augment a Mac with large but slow VRAM and weak compute?

It would work just like a discrete GPU when doing CPU+GPU inference: you'd run a few shared layers on the discrete GPU and place the rest in unified memory. You'd want to minimize CPU/GPU transfers even more than usual, since a Thunderbolt connection only gives you equivalent throughput to PCIe 4.0 x4.

▲

manmal 17 hours ago | parent [-]

But isn’t the Mac Mini the weak link in that scenario?

▲

zozbot234 17 hours ago | parent [-]

It has way more unified memory than your typical dGPU.

	▲	manmal 14 hours ago \| parent [-]
		Yes obviously. That VRAM is also slower and has weak compute attached. Loading to the external GPU will slow things down too much.

▲

arjie 17 hours ago | parent | prev | next [-]

My Mini is actually the smallest model so it actually has "small but slow VRAM" (haha!) so the reason I want the GPU for are the smaller Gemmas or Qwens. Realistically, I'll probably run on an RTX 6000 Pro but this might be fun for home.

▲

GeekyBear 17 hours ago | parent | prev [-]

We've seen many recent projects to stream models direct from SSD to a discrete GPU's limited VRAM on PCs.

How big a bottleneck is Thunderbolt 5 compared to an SSD? Is the 120 Gbps mode only available when linked to a monitor?

▲

manmal 17 hours ago | parent [-]

That’s what, 14GB/s? The GPU‘s VRAM can do 100x that.

▲

GeekyBear 17 hours ago | parent [-]

A discrete consumer GPU card doesn't have enough fast RAM to run a very large model that hasn't been quanitized to hell.

That's why all the projects streaming models into the GPU from an SSD popped up recently.

	▲	manmal 14 hours ago \| parent [-]
		Yes. There’s just no way to get above 1t/s that way with a large model.

▲

lowbloodsugar 17 hours ago | parent | prev [-]

“Lying around”. I’ve got an unopened 5090 in a box that I know will suffer the same fate, so I’m sending it back. So privileged to have the money to impulse buy a 5090 and yet no time to actually do anything with it.

	▲	c-c-c-c-c 2 hours ago \| parent [-]
		You should see his ferrari.