https://www.pingcap.com/trust-hub/security/security-update-o...

I find it really baffling how much they are downplaying this incident on their statement.

”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”

That does not appear to be true.

> PingCAP's engineers paste customers' production credentials into Jira tickets.

Holy hell, I've heard of this happening, but I've never personally seen it in my 10 years as an SRE. That's absolutely wild.

Coming this close on the heels of the security audit by NCC [1] is pretty embarassing for all parties.

[1]https://www.pingcap.com/press-release/tidb-ncc-group-securit...

If true, this is next level incompetence from PingCap. Apparently the attackers set up a replication pipeline for a month?!

> We even stood up a replication feed of their own control-plane database using credentials committed in their own repositories — GCP service-account keys and an Azure service principal — and let their infrastructure deliver it to us. Every six hours, for the last twenty-two days of the campaign, from 8 September to 30 September, that pipeline faithfully re-exported every table of the operational heart of TiDB Cloud — tenants, users, API keys, cluster credentials, changefeeds, migrations — and pushed it out.

Pingcap's statement:

    With support from outside counsel and third-party forensic specialists, there is no evidence that data stored in TiDB was impacted. We keep on investigating and will provide updates on any new findings.

AWS private rates gives PingCAP ~47% off list price. Wonder which services got the biggest discounts? Unsurprisingly for a db company the cross-az traffic is 15% of their bill.

Man I recently executed a migration from Aurora to Planetscale. I seriously considered PingCap but decided against it because I thought they cloud pricing was too complex. I wonder if my personal information is in the leak.

> Summary: "The root password of the cluster can be reset unconditionally"

But what matters is that it was all memory safe and had no data races.

Why is Baidu among their clients

what is happening today

and how are people not talking about this more

https://www.moxfive.com/blog/who-is-fulcrumsec-inside-the-cl...

First the Google Cloud outage [0], then the CA private key leak [1]...

...and now this...

> Today we are making the PingCAP breach public. We possess the complete TiDB Cloud control plane, its billing and finance estate, its GCP storage estate and its AWS cost and billing estate, its Jira, its CRM, its analytics, its source code, and a four-year log of every connection every customer ever made to its product, from the IP addresses of their application servers.

This is an entire leak of ALL PingCap TiDB's customers AND data.

All of this in a single day. It is over.

[0] https://news.ycombinator.com/item?id=50163961

[1] https://news.ycombinator.com/item?id=50166579

It's written in Rust, so it's safe.