Remix.run Logo
827a 10 hours ago

I said this when this finding was originally posted and I'll say it again: This is by far the worst security incident Google has ever had, and that's why they aren't publicly or loudly responding to it. It's deeply embarrassing. They can't fix it without breaking customer workflows. They really, really want it to just go away and six months from now they'll complete their warning period to their enterprise contracts and then they can turn off this automated grant. Until then they want as few people to know about it as possible, and that means if you aren't on anyone's big & important customer list internally, and you missed the single 40px blurb they put on a buried developer documentation site, you're vulnerable and this will happen to you.

Disgusting behavior.

zarzavat 10 hours ago | parent | next [-]

It's not a security incident because it makes Google money. It's extra revenue. They are embarrassed all the way to the bank.

At some point, when it appeared 2 months ago on HN and they still did nothing about it, intentionality can be assumed.

bombcar 10 hours ago | parent [-]

This is exactly it - and the normal "resolution" is a class-action lawsuit but no doubt their terms and conditions forbid that.

However, anyone affected should probably pollute their docket with lawsuits anyway.

100ms 10 hours ago | parent | prev | next [-]

This is only a little billing leakage, Operation Aurora in 2009 was 100x worse

827a 10 hours ago | parent [-]

It's actually much more than a billing leak [1]; again, most people don't know how bad this is, because Google is trying to keep it hush-hush. These keys don't just grant access to Gemini completions; they grant access to any endpoint on the generative AI google cloud product. This includes: seeing all of the files that google cloud project has uploaded to gemini, and interacting with the gemini token cache.

[1] https://trufflesecurity.com/blog/google-api-keys-werent-secr...

tantalor 10 hours ago | parent | prev | next [-]

What does this have to do with security?

827a 7 hours ago | parent [-]

Billing control is security, to be clear, but beyond that: The key permissions that enable anyone to generate text also grant access to all GCP Generative AI endpoints in the project they were provisioned in. That includes things like Files that your system might have uploaded to Gemini for processing, and querying the Gemini context caches for recent Gemini completions your system did. Both of these are likely to contain customer-facing data, if your organization & systems use them.

If you're hearing this and your gut reaction is This can't be real; We're on the same page. Its a staggering issue that Google has categorically failed to respond to. They automatically added this permission to existing keys that they knew their customers were publishing publicly on the internet, because the keys are legitimately supposed to be public for things like client-side Firebase access & Google Maps tile rendering.

They did not notify customers that they were doing this. They did not notify customers after this issue was reported to them months later by Truffle. They did not automatically remove the additional key grants for customers. They continue to push guidance targeted at novices like "just put the Gemini key behind a proxy (that's also publicly exposed on the internet)", which might solve the unintentional files and caching endpoint leaks but doesn't solve the billing issue. They denied that Truffle's initial report was even valid, until Truffle used the Internet Archive to find a Google internal key from 2023, published for a Google Maps widget or something, before Gemini was even released, that was still active, and used it to demonstrate to Google that, hey, anyone can use this key to get Gemini completions on the house, is there anyone driving this ship??" Google fixed the permissions on that specific key. And did nothing else.

7 hours ago | parent [-]
[deleted]
JackSlateur 10 hours ago | parent | prev [-]

And this is why we invented segmentation, and everybody that are still not doing that are paying now and this is fine

Google is not the only culprit here;