The complexity is what gets you. One of AWS's favorite situations is

1) Senior engineer starts on AWS

2) Senior engineer leaves because our industry does not value longevity or loyalty at all whatsoever (not saying it should, just observing that it doesn't)

3) New engineer comes in and panics

4) Ends up using a "managed service" to relieve the panic

5) New engineer leaves

6) Second new engineer comes in and not only panics but outright needs help

7) Paired with some "certified AWS partner" who claims to help "reduce cost" but who actually gets a kickback from the extra spend they induce (usually 10% if I'm not mistaken)

Calling it it ransomware is obviously hyperbolic but there are definitely some parallels one could draw

On top of it all, AWS pricing is about to massively go up due to the RAM price increase. There's no way it can't since AWS is over half of Amazon's profit while only around 15% of its revenue.

> If you're using something like ECS or serverless, AWS gains nothing by optimizing the servers to make your code run faster - their hard work results in less billed infrastructure hours.

If ECS is faster, then you're more satisfied with AWS and less likely to migrate. You're also open to additional services that might bring up the spend (e.g. ECS Container Insights or X-Ray)

Source: Former Amazon employee

Just this week a friend of mine was spinning up some AWS managed service, complaining about the complexity, and how any reconfiguration took 45 minutes to reload. It's a service you can just install with apt, the default configuration is fine. Not only is many service no longer cheaper in the cloud, the management overhead also exceed that of on-prem.

It's about fitting your utilization to the model that best serves you.

If you can keep 4 "Java boxes" fed with work 80%+ of the time, then sure EC2 is a good fit.

We do a lot of batch processing and save money over having EC2 boxes always on. Sure we could probably pinch some more pennies if we managed the EC2 box uptime and figured out mechanisms for load balancing the batches... But that's engineering time we just don't really care to spend when ECS nets us most of the savings advantage and is simple to reason about and use.

Agreed. There is a wide price difference between running a managed AWS or Azure MySQL service and running MySQL on a VM that you spin up in AWS or Azure.

> your infra is so efficient and cheap that even paying 4x for it rather than going for colocation is well worth it because of the QoL and QoS.

You don’t need colocation to save 4x though. Bandwidth pricing is 10x. EC2 is 2-4x especially outside US. EBS for its iops is just bad.

The problem is that clouds have easily become 3 or 5 times the price of managed services, 10x the price of option 3, and 20x the price of option 4. To say nothing of the fact that almost all businesses can run fine on "pc under desk" type situations.

So in practice cloud has become the more expensive option the second your spend goes over the price of 1 engineer.

Just curious, what is the spec you pay $6000/year for? Where/what is the line between rent vs buy?

> I’m a bit scared of managing the services on my own

I see it from the other direction, when if something fails, I have complete access to everything, meaning that I have a chance of fixing it. That's down to hardware even. Things get abstracted away, hidden behind APIs and data lives beyond my reach, when I run stuff in the cloud.

Security and regular mistakes are much the same in the cloud, but I then have to layer whatever complications the cloud provide comes with on top. If cost has to be much much lower if I'm going to trust a cloud provider over running something in my own data center.

You sum it up very neatly. We've heard this from quite a few companies, and that's kind of why we started our ours.

We figured, "Okay, if we can do this well, reliably, and de-risk it; then we can offer that as a service and just split the difference on the cost savings"

(plus we include engineering time proportional to cluster size, and also do the migration on our own dime as part of the de-risking)

I've just shifted my SWE infrastructure from AWS to Hetzner (literally in the last month). My current analysis looks like it will be about 15-20% of the cost - £240 vs 40-50 euros.

Expect a significant exit expense, though, especially if you are shifting large volumes of S3 data. That's been our biggest expense. I've moved this to Wasabi at about 8 euros a month (vs about $70-80 a month on S3), but I've paid transit fees of about $180 - and it was more expensive because I used DataSync.

Retrospectively, I should have just DIYed the transfer, but maybe others can benefit from my error...

I’m wondering if it makes sense to distribute your architecture so that workers who do most of the heavy lifting are in hetzner, while the other stuff is in costly AWS. On the other hand this means you don’t have easy access to S3, etc.

> I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …)

Out of interest, how old are you? This was quite normal expectation of a technical department even 15 years ago.

No amount of money will make me maintain my own dbs. We tried it at first and it was a nightmare.

That is why the the second "if" is there.

You can get locked into cloud too.

The lock in is not really long term as it is an easy option to migrate off.

An interesting question, so time for some 100% speculation.

It sounds like they probably have revenue in the €500mm range today. And given that the bare metal cost of AWS-equivalent bills tends to be a 90% reduction, we'll say a €10mm+ bare metal cost.

So I would say a cautious and qualified "yes". But I know even for smaller deployments of tens or hundreds of servers, they'll ask you what the purpose is. If you say something like "blockchain," they're going to say, "Actually, we prefer not to have your business."

I get the strong impression that while they naturally do want business, they also aren't going to take a huge amount of risk on board themselves. Their specialism is optimising on cost, which naturally has to involve avoiding or mitigating risk. I'm sure there'd be business terms to discuss, put it that way.

Who are you thinking of?

Netflix might be spending as much as $120m (but probably a little less), and I thought they were probably Amazon's biggest customer. Does someone (single-buyer) spend more than that with AWS?

Hertzner's revenue is somewhere around $400m, so probably a little scary taking on an additional 30% revenue from a single customer, and Netflix's shareholders would probably be worried about risk relying on a vendor that is much smaller than them.

Sometimes if the companies are friendly to the idea, they could form a joint venture or maybe Netflix could just acquire Hertzner (and compete with Amazon?), but I think it unlikely Hertzner could take on Netflix-sized for nontechnical reasons.

However increasing pop capacity by 30% within 6mo is pretty realistic, so I think they'd probably be able to physically service Netflix without changing too much if management could get comfortable with the idea

Fair point!

5 - Datacenter (DC) - Like 4, except also take control of the space/power/HVAC/transit/security side of the equation. Makes sense either at scale, or if you have specific needs. Specific needs could be: specific location, reliability (higher or lower than a DC), resilience (conflict planning).

There are actually some really interesting use cases here. For example, reliability: If your company is in a physical office, how strong is the need to run your internal systems in a data centre? If you run your servers in your office, then there's no connectivity reliability concerns. If the power goes out, then the power is out to your staff's computers anyway (still get a UPS though).

Or perhaps you don't need as high reliability if you're doing only batch workloads? Do you need to pay the premium for redundant network connections and power supplies?

If you want your company to still function in the event of some kind of military conflict, do you really want to rely on fibre optic lines between your office and the data center? Do you want to keep all your infrastructure in such a high-value target?

I think this is one of the more interesting areas to think about, at least for me!

To put it plainly: We deploy a Kubernetes cluster on Hetzner dedicated servers and become your DevOps team (or a part thereof).

It works because bare metal is about 10% the cost of cloud, and our value-add is in 1) creating a resilient platform on top of that, 2) supporting it, 3) being on-call, and 4) being or supporting your DevOps team.

This starts with us providing a Kubernetes cluster which we manage, but we also take responsibility for the services run on it. If you want Postgres, Redis, Clickhouse, NATS, etc, we'll deploy it and be SLA-on-call for any issues.

If you don't want to deal with Kubernetes then you don't have to. Just have your software engineers hand us the software and we'll handle deployment.

Everything is deployed on open source tooling, you have access to all the configuration for the services we deploy. You have server root access. If you want to leave you can do.

Our customers have full root access, and our engineers (myself included) are in a Slack channel with you engineers.

And, FWIW, it doesn't have to be Hetzner. We can colocate or use other providers, but Hetzner offer excellent bang-per-buck.

Edit: And all this is included in the cluster price, which comes out cheaper than the same hardware on the major cloud providers

Instead of using the Cloud's own Kubernetes service, for example, you just buy the compute and run your own Kubernetes cluster. At a certain scale that is going to be cheaper if you have to know how. And since you are no longer tied to which services are provided and you just need access to compute and storage. you can also shop around for better prices than Amazon or Azure since you can really go to any provider of a VPS.

Is that for the virtual private network? I heard some people say that you actually get higher bandwidth if you're using the public network instead of the private network within Hetzner, which is a little bit crazy.

I wonder how much companies pay yearly in order to avoid having an employee pick up a drive from a local store, drive to the data center, pull the disk drive, screw out the failing hard drive and put in the new one, add it in the raid, verify the repair process has started, and then return to the office.

> ancient skills https://youtu.be/ZtYU87QNjPw?&t=10

It baffles me that my career trajectory somehow managed to insulate me from ever having to deal with the cloud, while such esoteric skills as swapping a hot swap disk or racking and cabling a new blade chassis are apparently on the order of finding a COBOL developer now. Really?

I can promise you that large financial institutions still have datacenters. Many, many, many datacenters!

There are a bunch of ways to manage bare metal servers apart from Nix. People have been doing it for years. Kickstart, theforeman, maas, etc, [0]. Many to choose from according to your needs and layers you want them to manage.

Reality is these days you just boot a basic image that runs containers

[0] Longer list here: https://github.com/alexellis/awesome-baremetal

Indeed! We've yet to go down this route, but it's something we're thinking on. A friend and I have been talking about how to bring Nix-like constructs to Kubernetes as well, which has been interesting. (https://github.com/clotodex/kix, very much in the "this is fun to think about" phase)

This is what we do, I gave a talk about our setup earlier this week at CfgMgmtCamp: https://www.youtube.com/watch?v=DBxkVVrN0mA&t=8457s

Option 4 as well, that's how we do it at work and it's been great. However, it can't really be "someone on the team knows Nix", anyone working on Ops will need Nix skills in order to be effective.

I'm a NixOS fan, but been using Talos Linux on Hetzner nodes (using Cluster-API) to form a Kubernetes Cluster. Works great too!