Remix.run Logo
Railway (PaaS) global outage(status.railway.com)
58 points by TealMyEal 2 hours ago | 50 comments
ZoneZealot an hour ago | parent | next [-]

That's a big yikes just after promoting themselves in the Jmail thread yesterday https://news.ycombinator.com/item?id=46966562

Of course every service will have outages, it's just funny to see it so soon after saying:

> We're nuts for studying failure at the company [...]

(albeit a different 'failure' context)

tonyhb 43 minutes ago | parent [-]

IDK, it looks like servers were up, connectivity worked well, and some builds were failing. Wouldn't call that a big issue, and the same thing was happening with Vercel due to their git clones etc. yesterday too.

IOW, doesnt look as bad as the title suggests?

justjake an hour ago | parent | prev | next [-]

Hello! Railway founder here

We'll have a post mortem for this one as we always write post mortems for anything that affects users

Our initial investigation reveals this affects <3% of instances

Apologies from myself + the Team. Any amount of downtime is completely unacceptable

You may monitor this incident here: https://status.railway.com/cmli5y9xt056zsdts5ngslbmp

vintagedave an hour ago | parent [-]

Hi Jake. Appreciate your presence here on HN.

This affected a seemingly random set of services across three of my accounts (pro and hobby, depending on if this is for work or just myself.) That ranges from Wordpress to static site hosting to a custom Python server. All of the deployments showed as Online, even after receiving a SIGTERM.

While 3% is 'good', that's an awfully wide range of things across multiple accounts for me, so it doesn't feel like 3% ;) Please publish the post mortem. I am a big fan of Railway but have really struggled with the amount of issues recently. You don't want to get Github's growing rep. Some people are already requesting I move one key service away, since this is not the first issue.

Finally, can I make a request re communication:

> If you are experiencing issues with your deployment, please attempt a re-deploy.

Why can't Railway restart or redeploy any affected service? This _sounds_ like you're requiring 3% of your users to manually fix the issue. I don't know if that's a communication problem or the actual solution, but I certainly had to do it manually, server by server.

justjake 31 minutes ago | parent [-]

Totally! People who see the impact will likely see more impacted than say, 3% of their services. Not all disruption created equal.

We rolled out a change to update our fraud model, and that uses workload fingerprinting

Since, in all likelyhood, your projects are similarly structured, there will be more impacted workloads if the shape of your workloads was in the "false positive" set

Will have more information soon but very valid (and astute) feelings!

huydotnet 2 hours ago | parent | prev | next [-]

Context: This is Railway the PaaS company, not your daily commute vehicle (which is good in general, still bad for many users, like me).

ratorx 44 minutes ago | parent | next [-]

A global train outage would be quite a spectacle, is that even possible?

TealMyEal an hour ago | parent | prev [-]

Yeah, i probs should have made that clear

huydotnet an hour ago | parent | prev | next [-]

Joke about train line aside, I think Railway fits right in the spot that Heroku left.

They have a nice UI, support deploy any kind of backend-involved apps as long as it can be built into a docker container. While many PaaS out there seems to prioritize frontend only apps.

And they have a free plan, so people can just quickly deploy some POC before decide if it's good to move on.

Anyone know if there is any other PaaS that come with a low cost starter plan like this (a side from paying for a VPS)?

Tankenstein an hour ago | parent | next [-]

Render.com has a similar value proposition. I’ve used them and am pretty happy. Railway seems to have more bundled observability built in, that i’d like in render.

ktaraszk an hour ago | parent | prev | next [-]

Yes, have you seen miget.com by any chance? You can start with the free tier, and can have a backend with a database for free (256Mi plan). If you need more, just upgrade. They redefined cloud billing. Worth checking.

czhu12 an hour ago | parent | prev | next [-]

Been building an open source version of railway at https://canine.sh. Offers all the same features without the potential of a vendor lock-in / price gouging.

Onavo 41 minutes ago | parent | next [-]

The docs seem to be non existent. Is the canine yaml documented?

You want docs like this:

https://coolify.io/docs/applications/ci-cd/github/setup-app

https://coolify.io/docs/applications/build-packs/dockerfile

https://coolify.io/docs/applications/build-packs/overview

Plenty of screenshots and exact step by step instructions. Throwing an "example git repo" with no documentation won't get you any users.

Put your shoes into that of a Heroku/Vercel user. DevOps is usually Somebody Else's Problem. They are not going to spend hours debugging kubernetes so if you want to sell them a PaaS built on Kubernetes, it has to be fool proof. Coolify is an excellent example, the underlying engineering is average at best (from a pure engineering point of view it's a very heavy app that suffers from frequent memory leaks, they have a new v5 rewrite but it's been stuck for 2 years) but the UI/UX has been polished very well.

czhu12 a minute ago | parent [-]

Yeah working through documentation still. The goal isn’t so much to replace coolify. Mostly born out of my last start up that ran a $20M business, 15 engineers, with about 300-1000qps at peak.

I think the single VPS model is just too hard to get working right at that scale.

I think north flank would be a better comparison of what canine is trying to do, rather than indie hackers. The goal is not take away kubernetes, but to simplify it massively for 90% of use cases but still give full k8s api for any more advanced features

imiric 37 minutes ago | parent | prev [-]

> Computing is getting cheaper

Heh.

Looks like a great product, although maybe mention some honest reasons to not use it, instead of the passive-aggressive marketing ones.

vcanales an hour ago | parent | prev | next [-]

I use https://github.com/coollabsio/coolify on a VPS for this.

mstank an hour ago | parent | prev [-]

VPS + Dokploy gives you just as much functionality with an additional performance boost. Hostinger has great prices and a one-click setup. Good for dozens of small projects.

jedbrooke 2 hours ago | parent | prev | next [-]

Oof, off topic but the trains were out of service here for my commute last night so I though from the headline this meant that somehow all trains everywhere just stopped working. Glad to see it’s just some Saas product that’s down

Gravityloss 2 hours ago | parent | next [-]

Indeed! Remote bricking of trains is perhaps a thing: https://www.thedrive.com/news/hackers-beat-anti-repair-softw...

esseph an hour ago | parent [-]

It's also mandated by Congress in the US, it's called PTC. (Remote control)

jasoncartwright 2 hours ago | parent | prev [-]

PaaS

jpcompartir an hour ago | parent | prev | next [-]

This is great, not 10 minutes before this outage did I present Railway as a viable option for some small-scale hosting for prototypes and non-critical apps as an alternative to the Cloud giants

ezekg an hour ago | parent [-]

It always happens that way. I guarantee some people migrated from Heroku to Railway and bragged about future stability to the team, only to experience this.

jpcompartir 33 minutes ago | parent [-]

Yeah 100%

This won't change my decision, but it is still impeccable timing

zachrip an hour ago | parent | prev | next [-]

I actually think the title is misleading. I'm not sure actual existing deployments are affected? Seemingly just new ones are not working?

justjake an hour ago | parent | next [-]

Railway founder here. <3%.

That said, we treat this exigently seriously!

Any downtime is unacceptable and we'll have a post mortem up in the next couple hours

TealMyEal an hour ago | parent | prev [-]

it seemed to have been all deployments that had a browser facing interface. id say some cloudflair DNS config messup

vintagedave an hour ago | parent | prev | next [-]

Multiple services are receiving SIGTERM or shutdown signals. See dozens of support messages here: https://station.railway.com/questions/services-down-799f7bc1

Here's a sample log entry:

> 2026-02-11T14:35:11.916787622Z [err] 2026/02/11 14:35:03 [notice] 1#1: signal 15 (SIGTERM) received, exiting

I've had about one third of my Railway services affected. I had no notification from Railway, and logging in showed each affected service as 'Online', even though it had been shut down.

I'm pretty annoyed. I am hosting some key sites on Railway. This is not their first outage recently, and one time a couple of months ago was just as I was about to give our company owner a demo of the live product.

engelo_b 42 minutes ago | parent | prev | next [-]

this is the hidden cost of the move to paas. you trade devops velocity for a massive single point of failure. once the global control plane goes down, your entire risk profile is basically out of your hands.

jszymborski an hour ago | parent | prev | next [-]

Does anyone know if Railway operates its own cloud or if it's running off AWS/GCP/Azure/etc...

jsheard an hour ago | parent | next [-]

They were originally a GCP wrapper but they started colo'ing their own racks about a year ago.

https://blog.railway.com/p/data-center-build-part-one

autonomousErwin an hour ago | parent | prev | next [-]

I think it has it's own "metal" services they're migrating customers to. Afaik they used GCP for "legacy" cloud services.

spollo an hour ago | parent | prev [-]

All of these services are bundling the underlying AWS/GCP/etc resources in an easier to use package.

railway-rahul 29 minutes ago | parent [-]

This is actually not true for us at Railway. It's our own metal.

searls an hour ago | parent | prev | next [-]

lol, just yesterday a friend asked me if he should move his business to Railway from Heroku. Welp.

zachrip an hour ago | parent [-]

I've been using railway a while now, and I've basically never paid them but I would. It's even better than heroku. Super easy to use.

et-al 2 hours ago | parent | prev | next [-]

I thought this was about a global outage regarding actual trains, but it looks like Railway is a Heroku replacement.

lysace an hour ago | parent [-]

Any news is good marketing if you're unknown.

HaZeust an hour ago | parent | prev | next [-]

Wasn't half the HN crowd repping this place yesterday when the Vercel CEO offered to pay for Jmail? Rough lol

iJohnDoe an hour ago | parent | prev | next [-]

Affected by the outage since about 6:15 AM PT this morning. We're still down as of 9:00 AM PT.

Our existing containers were in a failure state and are now are in a partial failure state. Containers are running, but underlying storage/database is offline.

Many questions on their forum are similar to our situation. People wondering if they should restart their containers to get things working again. Worried about if they should do anything, risk losing data if they do anything, or just give everything more time.

I'm glad Railway updated their status page, but more details need to be posted so everyone knows what to do now.

Everyone has outages, it's the way of life and technology. Communication with your customers always makes it less painful and people remember good communication and not the outage. Railway, let's start hearing more communication. Forum is having problems as well. Thanks.

locknitpicker 2 hours ago | parent | prev | next [-]

Is anyone expected to know what railway.com is?

colesantiago an hour ago | parent | prev [-]

What is this 'railway'?

I am assuming that a domain like railway.com should be about trains.

Why does every tech company have to name themselves as a one word .com website and what they do is unrelated and vague to their own name?

Does every tech company think they are Apple and have to register every word in the dictionary and redefine it as a technology company?

Really bad name for a company.

normie3000 an hour ago | parent | next [-]

> Does every tech company think they are Apple and have to register every word in the dictionary and redefine it as a technology company?

Netflix?

thornewolf an hour ago | parent | prev | next [-]

dispell the hate from your heart

colesantiago an hour ago | parent [-]

Questioning things is not 'hate' Mr Wolf.

Liftyee 44 minutes ago | parent | prev | next [-]

Possible empirical justification: Non-tech and more "typical" orgs (train companies...) don't spend lots of money on slick-sounding one-word .com domains.

blibble an hour ago | parent | prev | next [-]

could be worse

could be called "entire" (https://entire.io/)

imiric 32 minutes ago | parent [-]

Worse would be x.com.

lbrito 23 minutes ago | parent | prev | next [-]

A lot of companies have been doing that for a long time

Lotus

Jaguar

Caterpillar

Shell

its a human thing

vimda an hour ago | parent | prev [-]

You don't think "railway" at least conjures ideas about the company? It's not some random word. Not every company needs to be "helps you ship software quickly inc"