Bad for your power bill though.

Maybe so, but even then a second-hand blade server is more cost-effective than a Raspi Cluster.

> I can get a Xeon E5-2690V4 with 28 threads and 64GB of RAM for about $150.

If the goal is a lot of RAM and you don’t care about noise, power, or heat then these can be an okay deal.

Don’t underestimate how far CPUs have come, though. That machine will be slower than AMD’s slowest entry-level CPU. Even an AMD 5800X will double its single core performance and even walk away from it on multithreaded tasks despite only having 8 cores. It will use less electricity and be quiet, too. More expensive, but if this is something you plan to leave running 24/7 the electricity costs over a few years might make the power hungry server more expensive over time.

For $3000 you can get 3x used Epyc servers with a total of 144 cores and 384 GB memory, with dual-port 25Gbe networking so you can run them in a fully connected cluster without a switch. It will have >20x better perf/$ and ~3x better perf/W.

That combo gives you the better part of a gigabyte of L3 cache and an aggregate memory bandwidth of 600 GB/s, while still below 1000W total running at full speed. Plus your NICs are the fancy kind that let you play around with RoCEv2 and such nifty stuff.

It would also be relevant to then also learn how to do stuff properly with SLURM and Warewulf etc. instead of a poor mans solution with Ansible playbooks like in these blog posts.

Power and noise - old server hardware is not something you want in your home.

Commodity desktop cpus with 32 or 64GB RAM can do all of this in a low-power and quiet way without a lot more expense.

It will probably consume $150 worth of electricity in less than a month, even sitting idle :-\

On Aliexpress those Xeon+mobo+ram kits are really cheap.

Source? That seems like something I would want to take advantage if at the moment...

The VMs quickly get expensive if you leave them running though.

The desktop equivalent of your 10 T3 Micro instances is about $600 if you buy new. For example a Lenovo ThinkCentre M75q Gen 2 Tiny 11JN009QGE has 8x3.2GHz processor with hyperthreading. That's 16 virtual cores compared to the 20 vcpus of the T3 instances, but with much faster cores. And 16GB RAM allows you to match the 1GB per instance.

If you don't have anything and feel generous throw in another $200 for a good monitor and keyboard plus mouse. But you can get a used crap monitor for $20. I'd give you one for free just to be rid of it.

That's a total of $800, or 33 days of forgetting to shut down the 10 VMs. Maybe half that if you buy used.

Granted not everyone has $800 or even $400 to drop on hobby projects, renting VMs often does make sense

You can rent a beefy vm with an H100 for $1.50 / hr

I regularly rent this for a few hours at a time for learning and prototyping

Are you comparing 10 VM with 1 shared core with a 144 core solution?

If you want to run something like GNS3 network simulation on a hosting service's hardware you'll either have to deal with hiring a bare-metal server or deal with nested virtualisation on other people's VM setups. Network simulation absolutely drinks RAM, too, so just filling an old Xeon with RAM starts to look very attractive in comparison to cloud providers who treat it an expensive upsell.

Oracle's free tier is pretty generous too.

Plus you still have access to the whole lot when your ISP goes down (maybe less of a problem than it used to be, but not unheard of)

I disagree. Most of what I've learned about systems comes from debugging the weird issues that only happen on real systems, especially real hardware. The book knowledge is like, 20-30% of it.

>SSD + HAT + PSU + Case + Cooler

Zero of any of that is needed. The new Pi "works best" with a cooler sure but at standard room temps will be fine for serving web apps and custom projects and things. You do not need an SSD. You do not need a HAT for anything.

Apparently the Pi 5 8gb is $120 though WTF.

What personal web site or web app or project can't run just fine on a Pi Zero 2 though? It's a little RAM starved but performance wise it should be sufficient.

Other than second-hand mini PCs, old laptops also make great home servers. They have built in UPS!

> SSD + HAT + PSU + Case + Cooler (+ maybe a uSD)

The only 100% required thing on there is some sort of power supply, and an SD card, and I suspect a lot of people have a spare USB-C cable and brick lying around. A cooler is only recommended if you're going to be putting it under sustained CPU load, and they're like $10 on Amazon.

What do you mean by Cooler? Raspberry pi doesn't need a fan.

Also the other peripherals you consider are irrelevant, since you would need them (or not), in other setups. You can use a pi without a PSU for example. And if you use an SSD, you have to consider that cost in whatever you compare it to.

>I went this route myself and have figuratively and literally shelved a bunch of Pis

>and I have dozens of them,

Reread my post? I meant specifically that Pis are great for the 1 to 2 range. with 3 pis you should change to something else. So I'm saying they are good at the 100$-200$ budget, but bad anywhere above that.

With all these caveats in mind, a raspberry pi seems to be an incredibly poor choice for distributed computing

Exactly. This build sounds like the proverbial "1024 chickens" in Seymour Cray's famous analogy. If nothing else, the communications overhead will eat you alive.

I sure hope no one does this seriously expecting to save some money. I enjoy the videos on "catastrophically ill-advised" build-outs. My primary curiosities that get satisfied by them are:

1) How much worse / more expensive are they than a conventional solution?

2) What kinds of weird esoteric issues pop up and how do they get solved (e.g. the resizable BAR issue for GPU's attached to RPi's PCIe slot)

This is the right take. You might be able to get decent (2-3x less than a GPU rig) token generation, which is adequate, but your prompt processing speeds are more like 50-100x slower. A hardware solution is needed to make long context actually usable on a Mac.

Given the RAM limitations of the first gen Ryzen AI MAX, you have no choice but to go heavy on the quantization of the larger LLMs on that hardware.

Huh? My maxed out Mac Studio gets 60-100 tokens per second on 120B models, with latency on the order of 2 seconds.

It was expensive, but slow it is not for small queries.

Now, if I want to bump the context window to something huge, it does take 10-20 seconds to respond for agent tasks, but it’s only 2-3x slower than paid cloud models, in my experience.

Still a little annoying, and the models aren’t as good, but the gap isn’t nearly as big as you imply, at least for me.

Yeah but slower memory compared to the M3 Ultra. There's a big difference in memory bandwidth, which seems to be a driving factor for inferencing. Training on the other hand, it's probably a lot faster.

To quote @swiftonsecurity - https://x.com/swiftonsecurity/status/1650223598903382016 ;

> DO NOT TAKE HOME THE FREE 1U SERVER YOU DO NOT WANT THAT ANYWHERE A CLOSET DOOR WILL NOT STOP ITS BANSHEE WAIL TO THE DARK LORD AN UNHOLY CONDUIT TO THE DEPTHS OF INSOMNIA BINDING DARKNESS TO EVEN THE DAY

ahah and pricey power wise.

Currently the cloud providers are dumping second gen xeon scalables and those things are pigs when it comes to power use.

Sound wise its like someone running a hair dryer at full speed all the time and it can be louder under load.

Buy a 3/4u case for $100 and put whatever board you want in it with standard PC fans and a decent cpu cooler. Dead silent.

True! They aren't quiet. I keep mine in a well-ventilated room that doesn't typically have people in it.

Not true. Have one running in the closet and never hear it.

Only if it’s a 1U. 2U units idle at silent.

Just to add context — I've been experimenting on my 2nd channel (Level 2 Jeff) with titles that are straight/barebones exactly describing the content of the video, vs a slight bit of clickbait (never untrue, but certainly more intriguing and not describing the exact topic of the video).

The ones that are dead straight with no clickbait are 10/10 (the worst performers), and usually by a massive margin. Even with the same thumbnail.

The sad fact is, if you want your work seen on YouTube, you can't just say "I built a 10 node Raspberry Pi blade cluster and ran HPL and LLMs on it".

Some people are fine with a limited audience. And that's fine too! I don't have to write on my blog at all—I earn negative income from that, since I pay for hosting and a domain, but I hope some people enjoy the content in text form like I do.

I was just at VCF Midwest this past weekend, and I can assure you I am on some of the lower echelons of people who know about niche hardware.

I do get to see and play with a lot of interesting systems, but for most of them, I only get to go just under surface-level. It's a lot different seeing someone who's reverse engineered every aspect of an IBM PC110, or someone who's restored an entire old mainframe that was in storage for years... or the group of people who built an entire functional telephone exchange with equipment spread over 50 years (including a cell network, a billing system, etc.).

Youtubers have armies of sycophants (check their video comments if you dare). Not saying they even court them, something to do with video building a stronger parasocial relationship than a text blog I think.

> more experience with niche hardware than literally anyone is a stretch.

This is why I said "almost anyone." If I changed your words, I could disagree with you as well.