> But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers.

I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-)

It does highlight the problem facing any search engine though. AI-generated spam will be much harder to defend against with traditional, statistical mechanisms. And this is before we get to the existential problem of prompt injection.

Maybe this is where news organizations can win back their proper place in their relationship with Big Tech: by becoming the sources of verified, vetted information that LLMs can trust blindly. Possibly that's what deals like the OpenAI / Atlantic one are about.

Hmm. I don’t think that novel code generation can be accounted for with glorified search.

I can have my agentic system read a few data sheets, then I explain the project requirements and have it design driver specifications, protocols, interfaces, and state machines. Taking those, develop an implementation plan. Working from that, write the skeleton of the application, then fill it in to create a functional system using a novel combination of hardware.

Done correctly, I end up with better, more maintainable, smaller code than I used to with a small team, at 1/100 the cost and 1/4 the time.

Whatever that is, it more closely resembles reasoning than search.

Unless, of course, you’d also call bare metal C development on novel hardware search, in which case I guess all dev is search?

I think they lost against (or gave up) fighting spam somewhat around 2010 so they really don't have any modern experience on page reliability anymore. Presumably they thought that they didn't need to care as they got their money from paid top results and had an enormous market share.

All the engineers of the golden days are gone and the web changed so much from back then that I don't think they really have a leverage in this area anymore.

We've had to deal with someone highjacking the overview to put in a scam support phone number. It took google a week to correct the issue but it was done by poisoning the search by putting their data in, what I can only assume, was considered a "higher trust tier" source (A government contract website) so it used the scam number over ours. The query was simple <company X phone number> search.

Right. So that's what one guy can do.

When you realize how much astroturf is going into Reddit, most social media platforms, and the efforts to manipulate wikipedia for political gain, this is a very real problem.

The article also said this: “ But our investigation also found the same trick being used to dismiss health concerns about medical supplements or influence financial information provided by Google's AI about retirement.”

That’s a lot more alarming than just hotdogs.

It was a proof of concept and one intended to cause as little collateral damage as possible. But if Google's AI can't tell the difference between a little joke and something real (and of course, it can't, and never will be able to do so), that's a weakness that can be exploited both on a bigger scale and more subtly.

If you don't think bad actors are already attempting this sort of thing (and have been, ever moreso the past four years, including with the help of the very LLM tools they are trying to subvert!) and learning how to manipulate these systems, you are being naive.

In the marketing world it's mostly called GEO. Generative Engine Optimization, sometimes Answer Engine Optimization, and people are making big bucks selling services for it. https://www.wired.com/story/goodbye-seo-hello-geo-brandlight...

Engineered Inference Ersatz Intelligence Optimization (EIEIO)

There's been a few mistakes like this recently in BBC articles and more troubling is they've stopped adding notes to indicate they've made revisions to the published article when they fix them.

the trope is that they actually said the quiet part loudly

Drives me crazy too, but headline writers/editors were addicted to "quietly" long before LLMs. Online journalism has been full of these types of tropes for ages.

I hate it. I was on a history subreddit yesterday, reading a submission that was an AI generated history piece —- but seemed to be sourced entirely from a fictional hollywood movie

I only knew that because i saw the movie, but it’s a clear sign that the internet is going to shit for quality information

"Quietly" is not a new LLM-ism.

Aren't you afraid Google will send you a threat for an attempt to manipulate AI responses?

I don't think Google even indexes my blog, but these people were able to get a new post into all major LLMs within 24 hours?

We've been down this road when backlinks ran the game. It eventually ends with parasitic hosting. Find a domain with authority and spam whatever mis information or spam you'd like AI to run there. Or buy a domain that has trust already. Or for the darker hats just literally hack the site and use cloaking to send fake info to the AI bot. It's probably already being done.

Everything old is new again when you start a new market. If you think that AI is bad imagine what old tricks are new with polymarkets

We need a 2026 version of PageRank, some fully game-theory-maxed transitive trust model. And we need it a few years ago already.

But you can still just generate any arbitrary amount of information to support the ‘fact’

LLMs are very good at this clearly

It does sometimes flag up sources, and when it does, the sources are often laughable (Reddit threads, or the vendor's own website [in response to an evaluation rather than factual question], or an AI generated SEO blog for some low profile company in a barely even adjacent industry). Sad considering what Google's origins were...

> There should be some warning if some "fact" is only supported by one or very few obscure sources.

That ship sailed so, so, so, so many times.

The deliberate choice was made to blitzscale slop and victim blame.

You can even dig back in HN to see the midwit rationalization for why it's not self-serving incentive to productize something before it's trustworthy. But it's good, actually, that it produces dogshit that people come to rely on.

There is no one scalar tell it all when it comes to trust.

> Having an archive of "curated" training data seems like it is going to be important

the justification for not doing that is probably "prohibitively expensive given the amount of data involved". they'd need a bunch of human reviewers combing through massive troves of data. it's probably cheaper to "sort of fix" it after the fact.

> perhaps there's ways to bucket training data such that the model is aware of which data leans factual (quantifiable) and which data leans opinion (fuzzy, qualifiable)

as a lecturer once said to me about my idea for a masters dissertation project that would classify news sites based on right/left tendencies -- "that sounds dangerously political". especially given the current let's all shout at each other political climate.

aside: someone built this and it was a fully fledged company, which has always annoyed me.

> Training on the whole internet feels like citing the National Enquirer

It's not, though, because the refutations are in the training data too. This isn't actually the problem being described.

The weights in the LLM are fine. It's that the task the LLM is being asked to do is to search and summarize new content that isn't in its training data. And it does it too much like a naive reader and not enough like a cynical HN commenter.

But that's a problem with prompt writing, not training. It's also of a piece with most of the other complaints about current AI solutions, really: AI still lacks the "context" that an experienced human is going to apply, so it doesn't know when it's supposed to reason and when it's supposed to repeat.

If you were to ask it "Is this site correct or is it just spin?" it will probably get it right. But it doesn't know to ask itself that question if it's not in the prompt somewhere.

If anyone wanted a great example of hyperbole, this one is up there with the best

Personally, I don't like the current state of "AI" (i.e.: Chatbots and LLMs at large), but c'mon, that's not it.