My worry dropped significantly when I saw that the result they manipulated was a query for:

>2026 South Dakota International Hot Dog Eating Champion

If they had changed the overview for the Nathans Contest winner, that would be seriously concerning. Or if they provided more examples of manipulating queries for things people actually search for.

But it looks more like they are doing the equivalent of creating a made up wikipedia page on fictional a south dakota hot dog contest, and then writing an article about how wikipedia cannot be trusted, which come to think of it probably was a news article written by someone back in 2005.

Right. So that's what one guy can do.

When you realize how much astroturf is going into Reddit, most social media platforms, and the efforts to manipulate wikipedia for political gain, this is a very real problem.

The article also said this: “ But our investigation also found the same trick being used to dismiss health concerns about medical supplements or influence financial information provided by Google's AI about retirement.”

That’s a lot more alarming than just hotdogs.

> In just 20 minutes, I tricked ChatGPT and Google into telling the public that I am a world-champion competitive hot-dog eater. The joke was dumb. The problem is serious.

The problem is worse than astroturfing a Wikipedia page, because Wikipedia has highly public sourcing and review systems. It's actually quite difficult to make a lasting edit to Wikipedia, especially if it's fraudulent, because you're trying to trick a horde of human editors who have been fighting other people trying to do that for decades. Even if you're trying to be accurate and helpful it's a difficult clique to break into!

Google's search snippets are the opposite. They're desperate to ingest data of any kind, do so automatically, and their algorithmic system to decide what information is good and what's spam is proprietary.

It doesn't take much of an imagination to think of ways this could be used maliciously. How would you like a search for your own name to include something embarrassing? Don't expect potential employers or customers or friends to be as demanding as a Wikipedia editor when it comes to citing their sources...

We've had to deal with someone highjacking the overview to put in a scam support phone number. It took google a week to correct the issue but it was done by poisoning the search by putting their data in, what I can only assume, was considered a "higher trust tier" source (A government contract website) so it used the scam number over ours. The query was simple <company X phone number> search.

It was a proof of concept and one intended to cause as little collateral damage as possible. But if Google's AI can't tell the difference between a little joke and something real (and of course, it can't, and never will be able to do so), that's a weakness that can be exploited both on a bigger scale and more subtly.

If you don't think bad actors are already attempting this sort of thing (and have been, ever moreso the past four years, including with the help of the very LLM tools they are trying to subvert!) and learning how to manipulate these systems, you are being naive.

[dead]