The interesting and yet-to-be-explained part is why google flagged the account?

Put all the timestamps you want in the post mortem about what you observed, but you haven't addressed the root cause.

The "this doesn't make sense" part of the story likely has a real explanation that nobody wants to reveal yet.

This exact thing happened to me when I ran https://www.fatherly.com/ circa 2017. Google just shut down our account without notice. We were spending like $10k/month. It also locked us out of our premium support account, so we couldn't even get anyone there to notice that they'd locked us out.

After about 8 hours, a random Google support tech said it was because we were mining bitcoin, which was laughably untrue. We had CPU usage graphs and logs for the whole time and there was no spike. At around 12 hours, they turned it back on, said it was "misconfiguration of our abuse detection" and gave us like $100 in credit.

Absurd. Say what you will about AWS, they would never do that to a customer without a rep reaching out to you first. I have not trusted GCP since.

▲

titzer 5 hours ago | parent [-]

Google thinks everything should be replaced with automation.

Remember knowledge cards? Prior to the LLM AI revolution, they had an extraordinarily crappy AI system digest the entire internet to figure out the wrong facts about stuff and then present it to users as solid truth, with no human review and no way to report inaccuracies.

They just don't care. If the task requires a person to look at a thing and tell if it's right, they only do that for like 5 examples and then train a classifier, then deploy said classifier without thinking twice because "at internet scale" or whatever crap.

	▲	dylan604 5 hours ago \| parent [-]
		Google is the epitome of expecting happy path results to always be the end result. I could absolutely see someone writing this knowledge card system, but then realizing how much work it would be to edit it with some PM not wanting to say the project was a failure and needing serious amounts of human effort to correct and just releasing it as is. Gotta earn those KPIs for that next promotion, and then it's someone else's problem!

▲

neya 4 minutes ago | parent | prev | next [-]

Top comments as usual are buried in deep hate for Google, I doubt that will pressure anyone at Railway to address this.

▲

Aperocky 6 hours ago | parent | prev | next [-]

Shouldn't Google answer this if they are unhappy with this incident report? Are we even sure that Railway knows?

	▲	e40 6 hours ago \| parent \| next [-]
		I seriously doubt Railway knows. That's the MO for Google and others, suspend account without explanation.
	▲	tomComb 6 hours ago \| parent \| prev \| next [-]
		They can't - that would violate the privacy rights of their customer. They need to tell Railway and Railway needs to tell us, or Railway can tell us that Google is refusing to tell them. Either way, we need to hear about this from Railway.
	▲	SoftTalker 6 hours ago \| parent \| prev \| next [-]
		The report at this point is pretty much just a timeline of what happened. No explanation of why, no accusations, no blame. A PR piece, to Railway's customers, reassuring them that "we're not ignoring this." Now the lawyers are huddling. IMO there won't be a lot more said publicly by either side, at least until any threat of lawsuits for damages is settled.
	▲	kyrra 5 hours ago \| parent \| prev [-]
		the Railway PM doesn't say they weren't told. It just sort of glosses over this. I would be interested to know if they were told (or not).

▲

array_key_first 6 hours ago | parent | prev | next [-]

I don't think you're typically told why for these things, and it's mostly automated from what I can tell. The automated systems make mistakes but more importantly they're completely opaque. Nobody, not even Google, knows how they work exactly.

▲

potatoman22 6 hours ago | parent | next [-]

Google should know why a human accepted the automated suggestion, or if and why there wasn't any human oversight in the first place.

	▲	okanat 4 hours ago \| parent \| next [-]
		Google knows and wants that there is no oversight. Don't do business with any big tech, if you don't want this kind of incidents.
	▲	advisedwang 4 hours ago \| parent \| prev [-]
		Google knows why there is no human oversight: because that is expensive (both in terms of the labor doing review and the ongoing fraud likely happening while the human review happens).

▲

llmslave 5 hours ago | parent | prev [-]

For big accounts, like railway, zero chance this was a handsoff fully automated ban

	▲	mjcl 2 hours ago \| parent \| next [-]
		Really? This isn't the first time their automation took down a big customer (UniSuper in 2024) by accident. In that case the automation actually deleted the resources and GCP had to recover them.
	▲	x0x0 5 hours ago \| parent \| prev [-]
		That assumes a competent org. If this were aws, I fully believe that. At gcp it's entirely plausible.

▲

2 hours ago | parent | prev | next [-]

[deleted]

▲

marginalx 5 hours ago | parent | prev | next [-]

Who is the "You" in "you haven't addressed the root cause"? If you are asking Railway to spend effort doing this rather than simply moving away from GCP, I'm not sure why they would unless they want to sue GCP to recover damages to brand and long term customer retention.

The moment GCP shut off without any forewarning, its done deal, no need to ask any further questions.

▲

croes 6 hours ago | parent | prev [-]

That‘s the point where Google tells you they won’t tell you the exact reason because of security reasons

▲

apple4ever 6 hours ago | parent | next [-]

Exactly this, which is the problem with all modern accounts. No person to talk to so you can understand what happened and maybe fix it.

▲

tcdent 6 hours ago | parent [-]

They most definitely have a person to talk to. They're not the largest Google Cloud user by far, but they are large enough to have human account reps.

	▲	xp84 6 hours ago \| parent [-]
		And those reps might not be told what the reason is.

▲

realusername 6 hours ago | parent | prev [-]

They also don't want to tell you because then they have to put rules and cannot ban people arbitrarily.

Giving reasons is putting accountability on Google and they don't want that.