| ▲ | Railway Blocked by Google Cloud(status.railway.com) |
| 176 points by aarondf 2 hours ago | 53 comments |
| |
|
| ▲ | dangoodmanUT an hour ago | parent | next [-] |
| It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation. |
| |
| ▲ | somewhatgoated 10 minutes ago | parent | next [-] | | On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year. | |
| ▲ | abrookewood 38 minutes ago | parent | prev | next [-] | | Yep, agree 100%. Such a stupid move on their behalf. | |
| ▲ | jameson 28 minutes ago | parent | prev | next [-] | | What was the reason GCP took down a startup previously? | |
| ▲ | tjpnz an hour ago | parent | prev [-] | | AWS normally contacts you first. | | |
| ▲ | kevin_nisbet 32 minutes ago | parent | next [-] | | Do they? The only anecdotal thing I've seen is we hired a vendor to do a pentest a few years ago, and they setup some stuff in an AWS account and that account got totally yeeted out of existence by AWS if memory serves. | | |
| ▲ | dannyw a minute ago | parent | next [-] | | You should not be conducting unauthorized penetration tests against third party infrastructure providers without permission. That’s assuming your vendor was pentesting AWS systems. If you meant you hired a vendor to pentest your own systems on AWS, that’s of course a totally different matter. | |
| ▲ | alchemism 29 minutes ago | parent | prev | next [-] | | I’m fairly certain you are supposed to contact any vendor before attempting to penetrate hosts with authorization, not the other way around. | | |
| ▲ | coredog64 2 minutes ago | parent [-] | | Having done this for both Azure and AWS, there's a specific ticket that needs to be filed with each provider that documents the scope of your pen test, where you're coming from, and a time frame over which you're doing it (which ISTR was "not more than 24 hours") |
| |
| ▲ | mixdup 10 minutes ago | parent | prev [-] | | Responding to an unknown security tester like that is a selling point, not a cautionary tale |
| |
| ▲ | cherioo an hour ago | parent | prev [-] | | They better do. What is google doing? | | |
|
|
|
| ▲ | binarycleric 27 minutes ago | parent | prev | next [-] |
| How the heck do these things happen, especially with companies with huge monthly spend? At my last job we had some suspicious workloads running on AWS and our TAM reached out to us before taking any action. Who wants to bet this was some AI automation gone wrong and because GCP seems to be allergic to actually contacting a human to get a response, this just sits in some support queue that outsourced workers look at after a few hours just to give a canned response? |
| |
| ▲ | garciasn 21 minutes ago | parent | next [-] | | Nothing surprises me with anything related to support on GCP. While we absolutely do not need them, I have been through no less than 12 different Account Executives over the last 6y and they're all ENTIRELY and COMPLETELY useless. They all introduce themselves, beg me to setup a meeting w/them and some sort of engineering resource(s), and they come to a meeting with a canned slide deck that is so absurdly unrelated to us that I just laugh, and then the next time I hear from them it's because we have a new AE. This is my most recent reply (right after Next '26): > I really appreciate you reaching out; however, we have met with, I dunno at this point, more than a dozen GCP Account reps, execs, technical teams, etc over the years and there's little to no value for us or you, now or in the future. Please do feel free to invest your time on your other clients. We're good; truly. I love GCP and its services; we have been very pleased with it over the years, but the human side of it? Fucking sucks and I just don't see why they even bother. | | |
| ▲ | OptionOfT 11 minutes ago | parent | next [-] | | It's because they're measured on something, unsure which metric, but it's definitely not how helpful they are to you. | | |
| ▲ | YuriNiyazov 7 minutes ago | parent [-] | | Don't know about GCP, but our AE on AWS was also continuously rotating, and as best I can tell, their job was to figure out what we are planning to build, and to ensure that we should always use <INSERT AWS SERVICE DU JOUR> for that, rather than a competitor product or build it ourselves. | | |
| |
| ▲ | idontwantthis 9 minutes ago | parent | prev [-] | | It doesn’t worry you enough that someday you could have a serious problem and they wouldn’t be able to help you? | | |
| ▲ | garciasn 5 minutes ago | parent [-] | | On the list of things that worry me the most about our company's stuff, an issue I cannot solve w/o help from a human at GCP is around #900000042. |
|
| |
| ▲ | guluarte 18 minutes ago | parent | prev [-] | | It's Google. They let you use their services, but the moment you don't fit the norm, they suspend you. |
|
|
| ▲ | BitWiseVibe 21 minutes ago | parent | prev | next [-] |
| As someone who runs some public APIs, the amount of spam from Railway IPs is insane. They have horrible abuse prevention. Hopefully this encourages them to improve their operations. |
|
| ▲ | UrbanNorminal 23 minutes ago | parent | prev | next [-] |
| Is google allergic to humans or something? Cannot they just send an email or call the company before taking a wrecking ball to the entire company's infra? Are they stupid? |
|
| ▲ | bearjaws 26 minutes ago | parent | prev | next [-] |
| I will never leverage GCP in an enterprise setting, it's honestly amazing how hard they fumble the bag. Will be interesting to see when GCP support started working with them, from the updates there was an hour and change from when they identified the issue and GCP support was confirmed. In the cloud space it seems like AWS does nothing and wins. |
|
| ▲ | brokenodo 8 minutes ago | parent | prev | next [-] |
| Well, as a 2 week tenured and very happy Railway customer until now, I am now a Render customer. Somehow DNS cut over within 1 min(!) and live after about 30 minutes of work. Not bad! |
|
| ▲ | codegeek an hour ago | parent | prev | next [-] |
| This is bad. Even their own website is down at railway.com. Looks like total dependency on google cloud. Surprising for a company of their scale with all this VC money. |
| |
| ▲ | choilive an hour ago | parent [-] | | They run a decent amount of their own compute/bare metal server for customer workloads. But likely still had some critical dependencies on GCP. |
|
|
| ▲ | tux 43 minutes ago | parent | prev | next [-] |
| At this point you can’t trust Google anymore, it keeps breaking things. Imagine having Google AI do this thins automatically. Will have apocalypse in in a day. |
|
| ▲ | r_lee an hour ago | parent | prev | next [-] |
| seriously, is it possible to trust GCP with critical data/services at this point if you're not a billion dollar company? I'm exaggerating but someone said they got "auto banned" what if that happens to a small account which hosts some really important data/services there? |
| |
| ▲ | xyzzy_plugh 26 minutes ago | parent | next [-] | | I've managed several accounts with GCP over the years and I've always maintained a great relationship with our contacts there. Some of these accounts were quite small, on the order of <$20k/mo, and even then we were kept abreast of anything that might be cause for concern. I always maintain a standing biweekly meeting with at least someone on the other side (account exec, technical staff, whatever) and I've yet to be blindsided by anything. Is Google's communication good? No, not particularly. The only way something like TFA happens is if the relationship is neglected (by one or both parties). I'm not saying Railway did something wrong, but there are usually many flags and opportunities to correct long before drastic actions. I get the impression that Railway plays fast and loose with a lot of their limits and resources and that Google may not be a fan of that. Edit: would also like to say that if you put all your resources in one GCP project you are going to have a bad time. If you organize stuff over many projects it is very unlikely that they will ever take account wide action. I've had issues with, for example, a particular tenant's behavior, but it never jeopardized the other tenants. | |
| ▲ | Avicebron 35 minutes ago | parent | prev | next [-] | | > what if that happens to a small account which hosts some really important data/services there? Pray to @dang that you will make the front page of HN? | |
| ▲ | throwaway85825 34 minutes ago | parent | prev | next [-] | | Even if you are a billion dollar company you still have problems like the Australian pension did. Google is just that bad. | |
| ▲ | chi_features 25 minutes ago | parent | prev | next [-] | | https://blog.railway.com/p/series-b Agreed. Railway are probably not far off a billion dollar company though! | |
| ▲ | ttoinou 26 minutes ago | parent | prev [-] | | Railway isnt far from being a billion dollar company, no ? |
|
|
| ▲ | padolsey 28 minutes ago | parent | prev | next [-] |
| Does anyone know how this even happens inside the walls of google? Is it an automated process? How is such a (presumably) high revenue account just magically blocked without human intervention? I'm quite perplexed. |
| |
| ▲ | jpollock 20 minutes ago | parent [-] | | There would have been efforts to contact them, but it would have been via their contact method, aka the email they set it up with. Common ways this happens? They are using a credit card to run their business with no backup payment method. Then the company's contact person is on vacation. Sign up for terms. It will get you payment terms! | | |
| ▲ | mbreese 8 minutes ago | parent | next [-] | | Yeah, I'm not sure what to think here. We know Google is not the best at customer service and has automated account suspensions. But, what I'm curious about here is why this happened. Railway hosts applications for customers. An uneducated guess for some possible reasons: 1) one of those customers hosted something they shouldn't have 2) railway had something spawn that took up too many resources 3) Or their account balance was too high 4) Or something... But all of this probably culminates in someone needed to read an email that was missed. Scaling a customer infrastructure setup like Railway is hard. This is one of the non-technical hard parts - how to make sure your account with your primary vendor is safe. But, I'm willing to wait to pass judgement here until more information is available. I'm sure the post-mortem will have lessons. I'd like to know more. | |
| ▲ | scratchyone 12 minutes ago | parent | prev [-] | | Honestly still insane to nuke a high-volume client's business after a single payment issue. There would be no reason for Google to believe that a single hiccup like that is evidence that they won't get paid and have to cut account access immediately. |
|
|
|
| ▲ | orliesaurus 37 minutes ago | parent | prev | next [-] |
| I wonder if someone has exploited a weird Google-safety automated process to report something on Railway which caused Google to block the whole thing. |
|
| ▲ | jefborges 39 minutes ago | parent | prev | next [-] |
| Railway is back, but I’m not sure if I can trust keeping my projects there, so I’m going to migrate to another company. |
|
| ▲ | redanddead 12 minutes ago | parent | prev | next [-] |
| one of the many reasons companies are cloud agnostic and dont want to get locked in |
| |
| ▲ | fh67 4 minutes ago | parent [-] | | Yeah but until you find that the new cloud provider won't approve your compute quota or doesn't have enough capacity in the region or you hit fraud flags for stagnant account spinning up lots of compute. |
|
|
| ▲ | parineum 9 minutes ago | parent | prev | next [-] |
| There's a lot of, what seems to me, unfounded blame being directed at Google for this. Isn't railway the company that just blamed Anthropic for deleting their prod database? |
|
| ▲ | gnabgib 2 hours ago | parent | prev | next [-] |
| Dupe - join the discussion started an hour ago instead of query string work (12 points, 4 comments) https://news.ycombinator.com/item?id=48200827 |
| |
| ▲ | aarondf 2 hours ago | parent [-] | | I added the qs because it defaulted to a story from 3 months ago. |
|
|
| ▲ | ChrisArchitect 28 minutes ago | parent | prev | next [-] |
| Earlier: https://news.ycombinator.com/item?id=48200827 |
|
| ▲ | isninkhamiss an hour ago | parent | prev | next [-] |
| github got way more noise for less |
|
| ▲ | rvz an hour ago | parent | prev | next [-] |
| Let me guess… Googler running AI agent in production that blocked this startup’s account. |
|
| ▲ | rekabis 2 hours ago | parent | prev [-] |
| TL;DR: putting all your eggs into one basket is bad, man. |
| |
| ▲ | canpan an hour ago | parent | next [-] | | How to handle domains? The rest is easy, but your domain registrar blocking you sounds like a pain. My current solution is to use a local small provider, just for the domain. Then if there is a problem with your play account it is out of any blast radius. | | |
| ▲ | FlamingMoe an hour ago | parent | next [-] | | What do you mean by local small provider? A registrar on main street? | |
| ▲ | truekonrads an hour ago | parent | prev [-] | | MarkMonitor | | |
| ▲ | Barbing 30 minutes ago | parent [-] | | Any changes since acquisition? Looks like they were sold at the beginning of the year to a company without a Wikipedia page whose parent company doesn’t have one either https://en.wikipedia.org/wiki/Markmonitor Acquired in November 2022 by Newfold Digital, it was later announced that the firm would be sold to Com Laude, a company owned by PX3 Partners.
-Edit-Private equity apparently https://px3partners.com PX3 stands for purpose, passion, and performance. It is a pan-European private equity firm with headquarters in London. It invests behind transformative themes and targets companies operating within select segments of the business services, consumer and leisure, and industrials sectors with strong business fundamentals.
|
|
| |
| ▲ | binarycleric 26 minutes ago | parent | prev [-] | | Same applies to all the companies betting the farm on AWS. |
|