Remix.run Logo
▲ prmoustache a day ago

I am not sure what is supposed to be wrong with puppet managed linux servers.

▲lrvick 19 hours ago | parent | next [-]

Production servers should be hardened, immutable, and remotely attestable appliances. Everyone should be able to prove every line of code in memory at all times, with access to secrets gated on signed quorum approval from multiple reviewers. They should not have dev tools, ssh, a shell, and certainly not capable of blindly modifying themselves at runtime if a single admin of a Puppet control server asks them to.

Such tactics mean a single compromised engineer laptop means a compromise of production, and thus a data breach is all but inevitable.

I would disqualify anyone seriously suggesting Salt, Puppet, Ansible or otherwise advocating for live-mutating production servers in a sysadmin interview today.

▲prmoustache 13 hours ago | parent | next [-]

I understand and subscribe to some of the things you say but:

> Such tactics mean a single compromised engineer laptop means a compromise of production, and thus a data breach is all but inevitable.

I hope you don't believe that an immutable image cannot be compromised. The commits done to the build system come from the same engineer laptops and, in a world where CVE are published ao frequently, said compromised image could be deployed largely in only a few minutes hours.

Also there is something to be said with lack of debugging tools. While I agree that reducing the surface of attack is the way to go, I am pretty sure the reason it takes so long for some of the big infra outages we have experienced from various cloud vendors comes from the inability of engineers to log on some systems, observe and potentially fix temporarily by hand to stop a race condition. There is often a race condition and serpent eating its own tail situation involved and logs might not even reach the monitoring infra anymore.

Going back to manage pets is not the way to go but I am not entirely certain removing ability for a subset of engineers to log on systems is exactly the way to go.

▲lrvick 3 hours ago | parent [-]

> I hope you don't believe that an immutable image cannot be compromised.

Obviously, but I will trust a unikernel with just kernel/init and an unprivileged sandboxed daemon to have dramatically less attack surface than a redhat system. Reducing attack surface is the job.

> The commits done to the build system come from the same engineer laptops and, in a world where CVE are published ao frequently, said compromised image could be deployed largely in only a few minutes hours.

That is what signed commits and signed reviews are for, and a CI that mandates both. I mentioned quorum control and this is the method I always use. I founded the only Linux distribution that does this.

> Also there is something to be said with lack of debugging tools.

In some cases one may wish to expose a custom, unprivileged, and sandboxed ssh daemon that provides basic logging and read only diagnostic tools to admins. This is usually fine. Hardening and observability can co exist. No need for a real ssh and huge traditional desktop linux shell interface attack surface.

But if I want to do major debugging with dangerous tools in prod, which should almost never be required, it should have to be with consent of another team member to add a -new- image to the fleet with those debug tools, temporarily.

This would only be required when the only way to debug is with production secrets, but the default should always be to reproduce on a duplicate system with dummy secrets.

▲solyin 17 hours ago | parent | prev | next [-]

What's the hardened immutable infra you're using these days?

▲lrvick 3 hours ago | parent [-]

Talos Linux, Stagex, and my own operating systems I have custom made for various past employers.

Custom operating systems are not scary. Make a UKI so your entire OS is just a kernel, init, and a target daemon payload. Treat servers like firmware. The less code in memory the better.

That said, I am working on an general purpose hardened server OS to be released soon publicly covered in my last talk https://lance.dev/talks/bsm2026

▲edf13 13 hours ago | parent | prev [-]

Never worked with a legacy system?

▲lrvick 3 hours ago | parent [-]

Yes. I am often the one that ensures they get understood, spun down, and upgraded.

▲Nevermark 20 hours ago | parent | prev [-]

Say “puppet linux servers” really fast several times.

Yeah.