Remix.run Logo
ad_hockey 8 hours ago

Minor point, but one of the complaints is a bit odd:

> curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing.

It's an API. Where would you type DELETE to confirm? Are there examples of REST-style APIs that implement a two-step confirmation for modifications? I would have thought such a check needs to be implemented on the client side prior to the API call.

kokada 4 hours ago | parent | next [-]

I don't think this is a minor point. It seems clear by this point that the author is clueless how even API works and are just trying to shift blame for third-parties instead assuming that they're just vibecoding their whole product without doing proper checks.

Yes sure, there seems to be lots of ways this issue could have been mitigated, but as other comments said, this mostly happened because the author didn't do its proper homework about how the service they rely their whole product works.

whartung 3 hours ago | parent [-]

It's also moot.

If the API replied "Are you sure (Y/N)?" the AI, in the mode it was in, guardrails completely pushed off the side of the road, it would have just said "Yes" anyway.

If you needed to make two API calls, one to stage the delete and the other to execute it (i.e. the "commit" phase), the AI would have looked up what it needed to do, and done that instead.

It's a privilege issue, not an execution issue.

kokada 2 hours ago | parent [-]

Exactly, that just reinforces the fact that the author is just blaming others instead of getting any valuable insights about this "postmortem analysis".

easton 6 hours ago | parent | prev | next [-]

AWS actually has a thingy on some services called “deletion protection” to prevent automation from accidentally wiping resources the user didn’t want it to (you set the bit, and then you need to make a separate api request to flip the bit back before continuing).

I think it’s designed for things like Terraform or CloudFormation where you might not realize the state machine decided your database needed to be replaced until it’s too late.

chrisandchris 6 hours ago | parent | next [-]

And then, someone added IAM so you could actually restrict your credentials from deleting your database.

First mistake is to use root credentials anyway for Terraform/automated API.

Second mistake is to not have any kind of deletion protection enabled on criticsl resources.

Third mistake is to ignore the 3-2-1 rule for backups. Where is your logically decoupled backup you could restore?

I am really sorry for their losss, but I do have close to zero empathy if you do not even try to understand the products you're using and just blindly trust the provider with all your critical data without any form of assessment.

throwaway041207 5 hours ago | parent | prev | next [-]

GCP Cloud SQL has the same deletion protection feature, but it also has a feature where if you delete the database, it doesn't delete backups for a certain period of days. If someone is reading this and uses Cloud SQL, I highly suggest you go make sure that check box is checked.

andy81 4 hours ago | parent | prev | next [-]

Agents will happily automate away intentional friction like a confirm prompt, even if you organise it as multiple API calls.

The fix needs to be permissions rather than ergonomics.

causal 6 hours ago | parent | prev | next [-]

There's also a cooldown period on some deletes (like secrets) to make sure you don't accidentally brick something

jeremyccrane 5 hours ago | parent | prev [-]

This should be the solution. All destructive actions require human intervention.

Someone1234 3 hours ago | parent | next [-]

If we take that literally, then just remove all destructive API endpoints. Because then, it they no real purpose, you cannot automate the removal of anything.

I think some other suggestions are saner (cool-down period, more fine-grain permissions, delete protection for certain high-value volumes). I don't think "don't allow destructive actions over the API" is the right boundary.

gizmondo 3 hours ago | parent | prev [-]

A human representing the company should be physically present in the provider's office to perform such an action or what? Otherwise you would just grant your agent a way to impersonate a human.

noxvilleza 6 hours ago | parent | prev | next [-]

> Are there examples of REST-style APIs that implement a two-step confirmation for modifications?

A pattern I've seen and used for merging common entities together has a sort of two-step confirmation: the first request takes in IDs of the entities to merge and returns a list of objects that would be affected by the merge, and a mergeJobId. Then a separate request is required to actually execute that mergeJob.

Ekaros 7 hours ago | parent | prev | next [-]

User is an idiot for using AI Agent. But I am not saying that it is not also badly designed system. Soft delete or something like should be standard for this type of operations. And any operator should know well enough to enable it for production.

kikimora an hour ago | parent | prev | next [-]

You won’t, but API implementation can and should mark a volume as pending deletion and keep it for a while. Like AWS does with keys and some other things.

mdavid626 6 hours ago | parent | prev | next [-]

In AWS eg. bucket can be deleted only when empty. Deleting all files first is your confirmation.

lelanthran 5 hours ago | parent [-]

> In AWS eg. bucket can be deleted only when empty. Deleting all files first is your confirmation.

That wouldn't have helped in this case - the agent made a decision to delete, so if necessary it would have deleted all the files first before continuing.

The question that comes to mind is "how are people this clueless about LLM capabilities actually managing to rise to be the head of a technology company?"

BarryMilo 4 hours ago | parent [-]

How are people still deluded enough about this economic system to believe rank implies competence?

WhyNotHugo 2 hours ago | parent | prev | next [-]

AWS has deletion protection for databases, and you have to make a separate call to disable it first. Deletion is rejected if you don’t disable that protection.

joegibbs 3 hours ago | parent | prev | next [-]

I suppose could implement it by requiring a deletion token that is returned when making a deletion request which doesn't have its own deletion token, but why would you? That's something for the frontend to handle.

gus_massa 5 hours ago | parent | prev | next [-]

Assuming the API has some secret spot to write DELETE, wouldn't the chatbot just send DELETE and make the protection only delay the disaster for 10 seconds?

powera 8 hours ago | parent | prev | next [-]

He (or ChatGPT) is throwing spaghetti at the wall. Not having the standard API key be able to delete the database (and backups) in one call makes sense. "Wanting a human to type DELETE as part of a delete API call" does not.

jeremyccrane 5 hours ago | parent [-]

In the user interface for Railway, all destructive actions require multiple confirmations, plus typing "apply destructive changes". Why would an API key (regardless of its scope) be able to delete without confirmation?

lelanthran 4 hours ago | parent | next [-]

> Why would an API key (regardless of its scope) be able to delete without confirmation?

What do you think an API is for? There's no user sitting at the keyboard when an API is called so where would that confirmation come from? It can't come from the user because there is no user.

fetzu 5 hours ago | parent | prev | next [-]

Isn’t the point of an API to have two computers talk to each other? As in “if I want safeguards for humans, it would be my responsability to put them BEFORE calling that API”?

lelanthran 5 hours ago | parent | prev | next [-]

> Why would an API key (regardless of its scope) be able to delete without confirmation?

How do you see this working? Any confirmation would be given by the agent.

jbxntuehineoh 4 hours ago | parent | prev [-]

... because that's how every other cloud provider API works? the AWS console makes you confirm before deleting a bucket; DeleteBucket does not

6 hours ago | parent | prev | next [-]
[deleted]
dr_hooo 6 hours ago | parent | prev | next [-]

I read this as "the agent should have asked for confirmation before running".

dymk 3 hours ago | parent | prev | next [-]

The whole tweet is AI slop, I doubt the human hitting "post" read through it all that closely. If they did, maybe they'd also go "Wait, that's nonsense".

IceDane 3 hours ago | parent | prev [-]

This person is a card-carrying moron and has no idea how anything works. Even if we concede that maybe there should be some grace period or soft deletions or whatever..

Also, the post is 100% written by an LLM, which is ironic enough on its own. But that then makes it a bit more curious that you find this argument in this slop, because any LLM would say so. But if you badger it enough, it will concede to your demands, so you just know this clown was yelling at his LLM while writing this post.

He really should've thrown this post at a fresh session and asked for an honest, critical review.