So what does one of these full time data center jobs look like, day to day? If I’m a software engineer I feel like I’d have to move and get a pay decrease to actually work at one of these? I mean until AI finally puts me out of a job. I guess I wouldn’t really be qualified to work one of these jobs?

The blokes I meet who work 8x5 from a data center tend to spend their days installing hardware, deinstalling hardware, providing remote hands (usually cabling, sometimes console access)building racks, managing power supplies and maintaining asset registers. And escorting idiots like me around when things get technical.

If you've never had an opportunity to spend time in a datacenter as a software developer, that's unfortunate but also far too common. What things look like on the inside vary company to company. Generally you're in an OSHA-abiding environment, so safety shoes, ear and eye protection, sometimes gloves.

There's a variety of roles. Security, electricians, HVAC engineers, generally some type of site foreman-ask role, logistics (depending on the size of the place), and technicians (for a lack of a better word, feels like every place calls them something different). There's a variety of roles that often float between sites or oversee many sites, depending again on the scale of the place. AWS is huge. Bigger than you're imagining, so there's quite a few levels deep and include real estate folks as well as construction roles. If you go and look at job postings, you'll even see roles for nuclear engineers at some companies.

But generally what you're talking about here are what I'm calling the technicians. They're responsible for wheeling racks into place (depending on the company they may also be responsible for unloading the trucks). Cabling is nearly always outsourced these days (though not the design of the cables), so rolling a rack into place generally involves securing it to the floor and connecting power, data, and more often than not now-a-days liquid cooling.

The other part of their job is "troubleshooting" failed hardware. Again, really depends on the company. Big big shops have "dumbed down" troubleshooting as much as they can - for a lot of reasons. You don't have to pay folks as much because they're thinking and doing less, the more time they spend troubleshooting the longer the server is offline, and if there's no troubleshooting there's not much for them to screw up. I'm sure there are some great places to be a tech where you get to rip apart servers and bust out the multimeter, that to my understanding is not how the hyperscalers who actually hyper-scale do it.

There's some cleaning, parts management, destroying broken hard drives, shoveling snow off the roof (no lie), and a variety of other odd tasks.

If you ever have the opportunity to check out one of those places it can be a riot and a real eye opener. Depends again on the company though, some of those places have insane security (metal detectors, badge+pin, turnstile door procedures) which make visits super un-fun if they're even allowed outside of legit business reasons. Other companies... well I'm glad that's not where I store my data.

Back "in the day" (2005 give or take a handful of years) techs would often write their own automation and even build some simple services.

And yes, the jobs don't pay particularly well depending upon what it is. Electricians and such command decent wages, but the security guards and techs don't make crazy amounts. I think folks doing contract cabling can come out ahead.

Anyhow, SWEs are wildly insulated from the realities of what things look like on the ground. Maybe that's a good thing, IDK.

Why would a datacenter employ an on site software engineer though? Anyway amazon already has plenty of those in house.

Outside of construction I don't believe datacenters employ many people locally.

It's not easy, actually. You will likely need to be a licensed electrician or a licensed plumber. Both occupations require around 4000 hours of apprenticeship.

Some states don't need a license for low-voltage work, so you might be able to do data wiring.