The worst blunder I made was when I explored cloud resources to improve the product's performance.

I created a GCP project (my-app-dev) for exploring how to scale up the cloud service. I added several resources to mock the production, like compute instances/cloud SQL/etc, then populated the data and run several benchmarks.

I changed the specs, number of instances and replicas, and configs through gcloud command.

  $ gcloud compute instances stop instance-1 --project=my-app-dev
  $ gcloud compute instances set-machine-type instance-1 --machine-type=c3-highcpu-176 --project=my-app-dev
  $ gcloud sql instances patch db-1 --tier=db-custom-32-131072 --project=my-app-dev

  $ gcloud projects list
  PROJECT_ID     NAME       PROJECT_NUMBER
  my-app-test    my app     123456789012
  my-app-dev     my app     234567890123     <- the dev project I was working on
  my-app         my app     345678901234     <- the production (I know it's a bad name)

Of course I checked every commands. I couldn't YOLO while working on cloud resources, even in dev environment. But I focused on the subommands and its content and didn't even think it had changed the project ID along the way.

It continued to suggest more and more aggressive commands for testing, and I approved them brain-deadly...

  $ gcloud sql instances patch db-1 --database-flags=max_connections=500 --project=my-app
  $ gcloud compute instances delete instance-1 --project=my-app
  $ echo 'DELETE FROM users WHERE username="test";' \
      | gcloud sql connect my-db 
   --user=user --database=my-db --project=my-app
  $ wrk -t4 -c200 -d30s \
      "http://$(gcloud compute instances describe instance-1 \
      --project=my-app \
      --format='get(networkInterfaces[0].accessConfigs[0].natIP)')"

Fortunately, it didn't do anything irreversible. But it was one of the most terrifying moments in my career.

This is part of the reason deployments to production cloud environments should:

1. Only be allowed via CI/CD

2. All infra should be defined as code

3. Any deployment to production should be a delayed process that also has a human-approval step in the workflow (at least one, if not more)

(Exactly where that review step is placed depends on your organisation - culture, size, etc.)

And anyone that does need to touch production should do so from an isolated VM with temporary credentials. Developers shouldn't routinely have production access from their terminal. This last aspect is easy and cheap to set up on AWS. I presume it's also possible in Google Cloud.