> This is the fundamental problem: AI can generate the form of security research without the substance.

I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.

▲

Sharlin 3 days ago | parent | next [-]

Unfortunately, to a majority of the population approximately 100% of LLM output seems entirely legitimate.

▲

dafelst 3 days ago | parent | next [-]

I agree wholeheartedly, and this is the core problem - many of the people evangelizing LLMs for a particular task (especially investors and AI gold rush "entrepreneurs") do not have enough expertise in that particular field to effectively evaluate the quality of the output. It sure looks the part though, and for those with a shallow understanding, it is often enough.

▲

captainkrtek 3 days ago | parent | prev | next [-]

That, combined with the confidence any of its output is communicated back to the user.

▲

acchow 3 days ago | parent | prev | next [-]

I’ve been trying ChatGPT for transit directions on Shanghai’s metro and it has been absolutely terrible. Hallucinating connections and routes.

But all of it’s responses definitely seem convincing (as it has been trained to do)

▲

jimbokun 3 days ago | parent | prev | next [-]

Except for things they happen to know something about.

▲

gdulli 3 days ago | parent [-]

Unfortunately, too few people are making the obvious leap from "LLMs aren't great for topics I have expertise in" to "maybe that means LLMs aren't actually great for the other topics either."

	▲	dtech 3 days ago \| parent \| next [-]
		We as humans aren't good at it. Before AI it was already coined as the "Gell-Mann Amnesia" effect
	▲	3 days ago \| parent \| prev \| next [-]
		[deleted]
	▲	3 days ago \| parent \| prev [-]
		[deleted]

▲

Rebuff5007 3 days ago | parent | prev [-]

And a sizable portion of the population believe vaccines don't work and/or have 5G!

I feel like I'm watching a tsunami about to hit while literally already drowning from a different tsunami.

▲

godelski 2 days ago | parent | prev | next [-]

There's another term for this that I think should catch on: Cargo Culting

Everything looks right but misses the underlying details that actually matter.

There is a larger problem that I think we like to pretend that everything is so simple you don't need expertise. This is especially bad in our CS communities where there's a tendency of thinking intelligence in one domain cleanly transfers to others. In this respect I generally advise people not to first ask LLMs what they don't know but what they are experts in. That way they can properly evaluate their responses. Least we all fall for Murry Gelmann amnesia lol

https://en.wikipedia.org/wiki/Cargo_cult

▲

lukev 3 days ago | parent | prev | next [-]

"Form without substance" is quite possibly the most accurate way to describe LLMs in general. Extremely powerful (and extremely useful) pattern finders and replicators... and little else.

	▲	8note 3 days ago \| parent [-]
		its certianly why they pair well with tool calls - substance without form

▲

jsheard 3 days ago | parent | prev | next [-]

The other fundamental problem is that to a grifter, it's not a fundamental problem for the output to be plausible but often wrong. Plausible is all they need.

	▲	gdulli 3 days ago \| parent \| next [-]
		That's an important one. Another fundamental problem with plausible output is that it makes a manager, or a junior, or some other unsophisticated end user think the technology is almost there, and a reliably correct version is just around the corner.
	▲	jacquesm 3 days ago \| parent \| prev [-]
		Indeed. The elderly in my family are seeing a substantial uptick of AI generated stuff that looks extremely plausible. Fortunately they're old but not stupid, so far nobody has fallen for any of these but I have to admit: they look good enough to pass a first casual inspection.

▲

fisf 3 days ago | parent | prev | next [-]

It's also a fundamental problem of security research. Lot's of irrelevant, highly contextual "vulnerabilities", submitted to farm internet points (driven by a broken cve system). AI only amplifies this.

▲

seanmcdirmid 3 days ago | parent | prev | next [-]

No, it is the problem of any ceremonial barrier in existence. If substance wasn’t required in the first place, people were faking it already.

▲

everybodyknows 3 days ago | parent | prev | next [-]

A parallel to AI-slop has existed for generations now out here in meatspace: Administrative/legal people on the periphery of a technical field (though possibly alas, at the top of the org's command chain) who do not at all understand what technical terms signify, but having seen hundreds of sentences produced by real experts, become able to themselves string together plausible-looking assertions.

	▲	breakpointalpha 3 days ago \| parent [-]
		We call these people "compliance flies". Any large enough organization gathers them en mass to cloud real development work with "compliance."

▲

spwa4 3 days ago | parent | prev [-]

But the problem is fundamentally slop, and grifters. It is possible to generate valid security bug reports with an AI agent, and there certainly is value in that. I'd even say a lot of value.

But of course producing fake ones is far easier and cheaper.