Remix.run Logo
▲ Railway Security Disclosure(blog.railway.com)
117 points by passenger an hour ago | 38 comments
▲theturtletalks 31 minutes ago | parent | next [-]

Postiz founder posted on X that his service is being brought down, his keys were already used and Railway was quiet about this until he spoke up:

https://x.com/wickedguro/status/2113015432974405658

▲MattIPv4 38 minutes ago | parent | prev | next [-]

Seen multiple folks in the Railway Discord confirming that their secrets stored in Railway have been used by third parties, so it definitely sounds like the attackers were able to read decrypted environment variables.

▲martinald 25 minutes ago | parent | prev | next [-]

I have a bad gut feeling that every day is going to look a bit like today with endless urgent patches and breaches.

Fwiw it sounds similar to the hack over the weekend that opencode suffered.

▲Fordec 11 minutes ago | parent | next [-]

I fully believe the only sustainable future is that hot patching will need to itself become fully automated. Hot patch now with a minimal allowed amount of line changes to head off the AIs, review it in office hours for auditing and the team is actually awake.

My second belief is that environment variables as a development concept will need to be retired. If it's something that causes a security breach if it by its mere existence on a public git repo, it is information that by design cannot ever pass to a filesystem out of a chain of trust, not by mere development conventions people choose to adhere to.

▲munificent 8 minutes ago | parent [-]

> hot patching will need to itself become fully automated.

Great, even less control over the things we use and more power ceded to corporations.

▲Fordec 5 minutes ago | parent [-]

Due to the small nature of hot patching (eg. bump this dependency, rotate these keys, turn off that config, follow the runbook) it will unlikely need to be a frontier model and is likely able to be done LocalLLM/Jev style decision making rather than having to hand all the keys to frontier model operators.

▲yakz 12 minutes ago | parent | prev | next [-]

It's only just starting. It'll get worse before it gets better.

▲reconnecting 21 minutes ago | parent | prev | next [-]

I think the only way out is to go back to on-prem for everything, like in the 00s.

LAN cables will go up in price soon.

▲solyin 16 minutes ago | parent [-]

How does on-prem help if your product is a SaaS.

▲lawrenceduk 3 minutes ago | parent [-]

You open a SaaS shop where people come to use it

▲Sivart13 10 minutes ago | parent | prev [-]

surely once there's been enough patches the code will be perfect though?

▲fbrncci 8 minutes ago | parent | prev | next [-]

Right just as I take my first day off (for the entire year!) from work to fly out for a vacation. The mother of all security bombs starts dropping all around stacks and services that I am responsible for. How else could have taken time off go for me…

▲paxy 28 minutes ago | parent | prev | next [-]

It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!” as if it absolves you from having to maintain a secure system. If anything it makes the situation more damning. If it was possible to find exploits using AI then your own red teaming should have found them before the attacker did.

▲tokioyoyo 14 minutes ago | parent | next [-]

Defending is so much harder than attacking. You don't even have to focus on one target, as you can just mass-AI-script a lot of attacks wherever it's possible, and see what sticks. Even if you did everything right, there's always the social-engineering angle, or a single compromised device, or just accidental bad upgrade through supply chain. If you can mass-spread the seed, and plant something that "calls home" constantly, you can just have an agent that would start the attack the minute it gets infected, and go deeper.

It's really a wild west right now, have no idea how we fix it. I'm assuming everything will get more concentrated in big co's hands, as having a very robust security is extremely hard.

Also, we have to agree that younger people will start companies. There will be small companies that cannot afford red teams. There will be companies that are just not equipped, and their profit margins cannot sustain constant attacks.

▲arcanemachiner 23 minutes ago | parent | prev | next [-]

> your own red teaming should have found them before the attacker did

Easier said than done. The defender has to do everything right, all the time, to prevent a headline like this. The attacker only needs to be right once, and it's game over. AI or no AI.

▲gkoberger 20 minutes ago | parent | prev | next [-]

Unless they edited the post, they never said this. You should delete this; it's false information.

▲mbreese 9 minutes ago | parent | prev | next [-]

I didn’t actually see that in this post at all. They mention that the end goal of the attackers might be to get to AI inference providers. But I don’t think they are crying “AI was used” here.

Or did I miss something?

▲barrrrald 26 minutes ago | parent | prev [-]

This one doesn’t?

▲fyzix 7 minutes ago | parent | prev | next [-]

How is it possible rotate keys if you disable deployments? You're basically telling us to take our services offline...

▲minraws 27 minutes ago | parent | prev | next [-]

how are the secrets decrypted, I would like some explanation so that I can have some understanding here, I am starting to feel like all these companies are just meme-ing with their security posture.

Like did someone vibe code too hard, how do env secrets leak for everyone...

▲9cb14c1ec0 17 minutes ago | parent [-]

Easy, all the bad guys need to do is access whatever system stores the decryption keys.

▲eecks 11 minutes ago | parent | prev | next [-]

Huh, I've been using Railway tonight and this was the first I've seen of this. Just got an email about it now too at 23:36

▲DrawTR 31 minutes ago | parent | prev | next [-]

Jeez, what a rough past 24 hours for all security folk. Hang in there

▲jacquesm 29 minutes ago | parent [-]

It's going to be a rough decade.

▲rectang 18 minutes ago | parent | next [-]

I really don't know how we are going to deal with accelerated exploit generation via LLMs — it blows up the disclosure processes we've evolved.

The only thing I can think of is hunkering down with static HTML to reduce attack surface, but that's laughably incompatible with many many business requirements.

▲cortesoft 16 minutes ago | parent | prev [-]

It feels like the only solution is going to be proactively running your own AI agents looking for patches and applying them.

▲rectang 10 minutes ago | parent [-]

The issue is where to direct the LLM's attention.

Prompting the LLM to "look for vulnerabilities in PHP's multipart upload handling code" is going to find stuff that "look for vulnerabilities in PHP" won't.

▲AJRF 20 minutes ago | parent | prev | next [-]

Do cloud providers have a lock-down mode where you can press 1 button and stop all spend?

▲sergiotapia 10 minutes ago | parent | prev | next [-]

If you're on Railway do not wait, ROTATE every API key at the source, and do not set new values because Railway is still compromised and not secured.

▲MattIPv4 36 minutes ago | parent | prev | next [-]

Related: https://news.ycombinator.com/item?id=50177413

▲wateralien 28 minutes ago | parent | prev | next [-]

Is this our new AI daily reality?

▲iririririr 17 minutes ago | parent [-]

companies probably have php and Python applications exposed to the web and nobody knows.

the speed of irresponsible deployments is infinite now, by design

▲gbnwl 27 minutes ago | parent | prev | next [-]

The past two days have been awful as someone who uses Railway.

▲Keloran 42 minutes ago | parent | prev | next [-]

Well this is certainly interesting, I wonder how their secrets were being stored, I assumed they were stored in vault and then injected with a sidecar, but I guess not

▲alpineman 20 minutes ago | parent | prev | next [-]

Train wreck

▲esseph 27 minutes ago | parent | prev | next [-]

Fuck, I thought this was about trains

▲lijok 24 minutes ago | parent | prev [-]

Damn I guess all those l33t hiring practices were for nothing

▲rvz 9 minutes ago | parent [-]

Of course, it was a giant scam that tested for nothing and optimized for how to pass a test.

Let's see if companies are ready to hire more people who know what they are doing, against a super-intelligent swarm of millions of agents with cyber capabilities.