Remix.run Logo
▲ rickreynoldssf a day ago

Apple runs their own cloud.

▲etc-hosts a day ago | parent | next [-]

Amazon's internal direction on how they scale out AWS and build new features is heavily guided by "what will it do to one of our biggest customers that has a secret code name but everyone knows its Apple".

▲swiftcoder a day ago | parent | prev | next [-]

> Apple runs their own cloud.

That's not the entirely of their cloud exposure, however. A ton of their stuff runs in AWS, I wouldn't be surprised if they had things on other cloud providers as well.

▲mixdup a day ago | parent [-]

The Microsoft outage wasn't an Azure outage anyway, it was specifically tied to Exchange Online. No one running their own stuff in Azure had any issues during that event

▲gr33nq a day ago | parent [-]

Mostly true, but not entirely. Azure Communication Services (ACS) was down alongside Exchange Online and the other Microsoft 365 applications. I verified this by testing upwards of a dozen times throughout the outage window. There was absolutely no indication made by Microsoft though that ACS was impacted in any way.

▲blinding-streak a day ago | parent | prev | next [-]

Apple is a massive Google Cloud customer for storage (iCloud) and has been since at least 2018.

▲lrvick a day ago | parent | prev [-]

A giant cluster of Puppet managed RedHat computers managed like it is 2007, mixed with Google Cloud.

Apple has literally no idea what they are doing with Linux and cloud.

▲mingus88 a day ago | parent | next [-]

They also have private compute cloud, not running Linux

They also use AWS and run servers in China using whatever cloud provider the Party uses

Like you’d expect a trillion dollar global tech enterprise to be, they run a ton of stuff geared toward whatever meets their business objectives

▲prmoustache a day ago | parent | prev | next [-]

I am not sure what is supposed to be wrong with puppet managed linux servers.

▲lrvick 18 hours ago | parent | next [-]

Production servers should be hardened, immutable, and remotely attestable appliances. Everyone should be able to prove every line of code in memory at all times, with access to secrets gated on signed quorum approval from multiple reviewers. They should not have dev tools, ssh, a shell, and certainly not capable of blindly modifying themselves at runtime if a single admin of a Puppet control server asks them to.

Such tactics mean a single compromised engineer laptop means a compromise of production, and thus a data breach is all but inevitable.

I would disqualify anyone seriously suggesting Salt, Puppet, Ansible or otherwise advocating for live-mutating production servers in a sysadmin interview today.

▲prmoustache 12 hours ago | parent | next [-]

I understand and subscribe to some of the things you say but:

> Such tactics mean a single compromised engineer laptop means a compromise of production, and thus a data breach is all but inevitable.

I hope you don't believe that an immutable image cannot be compromised. The commits done to the build system come from the same engineer laptops and, in a world where CVE are published ao frequently, said compromised image could be deployed largely in only a few minutes hours.

Also there is something to be said with lack of debugging tools. While I agree that reducing the surface of attack is the way to go, I am pretty sure the reason it takes so long for some of the big infra outages we have experienced from various cloud vendors comes from the inability of engineers to log on some systems, observe and potentially fix temporarily by hand to stop a race condition. There is often a race condition and serpent eating its own tail situation involved and logs might not even reach the monitoring infra anymore.

Going back to manage pets is not the way to go but I am not entirely certain removing ability for a subset of engineers to log on systems is exactly the way to go.

▲lrvick 3 hours ago | parent [-]

> I hope you don't believe that an immutable image cannot be compromised.

Obviously, but I will trust a unikernel with just kernel/init and an unprivileged sandboxed daemon to have dramatically less attack surface than a redhat system. Reducing attack surface is the job.

> The commits done to the build system come from the same engineer laptops and, in a world where CVE are published ao frequently, said compromised image could be deployed largely in only a few minutes hours.

That is what signed commits and signed reviews are for, and a CI that mandates both. I mentioned quorum control and this is the method I always use. I founded the only Linux distribution that does this.

> Also there is something to be said with lack of debugging tools.

In some cases one may wish to expose a custom, unprivileged, and sandboxed ssh daemon that provides basic logging and read only diagnostic tools to admins. This is usually fine. Hardening and observability can co exist. No need for a real ssh and huge traditional desktop linux shell interface attack surface.

But if I want to do major debugging with dangerous tools in prod, which should almost never be required, it should have to be with consent of another team member to add a -new- image to the fleet with those debug tools, temporarily.

This would only be required when the only way to debug is with production secrets, but the default should always be to reproduce on a duplicate system with dummy secrets.

▲solyin 17 hours ago | parent | prev | next [-]

What's the hardened immutable infra you're using these days?

▲lrvick 3 hours ago | parent [-]

Talos Linux, Stagex, and my own operating systems I have custom made for various past employers.

Custom operating systems are not scary. Make a UKI so your entire OS is just a kernel, init, and a target daemon payload. Treat servers like firmware. The less code in memory the better.

That said, I am working on an general purpose hardened server OS to be released soon publicly covered in my last talk https://lance.dev/talks/bsm2026

▲edf13 13 hours ago | parent | prev [-]

Never worked with a legacy system?

▲lrvick 3 hours ago | parent [-]

Yes. I am often the one that ensures they get understood, spun down, and upgraded.

▲Nevermark 20 hours ago | parent | prev [-]

Say “puppet linux servers” really fast several times.

Yeah.

▲ibejoeb a day ago | parent | prev | next [-]

Interesting. This is the first I've heard of broad service interruption at Apple in a while, as I can't even remember the last time. Do you suspect something about puppet or their overall security posture to be implicated in this?

▲Symbiote a day ago | parent | prev | next [-]

Up to this point, it seems to have been working well.

▲lrvick 18 hours ago | parent [-]

[dead]

▲rdtsc a day ago | parent | prev | next [-]

Hell they were more stable than all the other clouds so far. Maybe a giant cluster of redhat computers managed like 2007 is actually pretty robust compared to super kube control plane v42 with 100 add-ons and layers thrown in on top.

▲aprdm a day ago | parent | prev | next [-]

What's the problem with running puppet in 2026 ?

▲electric_mayhem 17 hours ago | parent | prev | next [-]

You say that like every group in Apple does things the same way.

They don't.

▲Prelaunch7096 a day ago | parent | prev [-]

[dead]