Isn't it mostly the medium that's problematic? With an issue tracker it's easier to close as duplicate

Aurornis 9 hours ago | parent | next [-]

An open visibility tracker would be a goldmine for finding new exploits before a fix is even available.

From what I’ve seen many of the AI bug search operators are newer to security research. They’re burning their tokens trying to find kernel bugs as their claim to fame before other people with AI tools find them first. They don’t spend time de-duplicating their own bugs.

Some of them may not be coming from real people. There are honeypot repos that are entirely fake and only have folders of simple files with clear security problems. They collect automated reports they get from all of the AI bots that people are running.

▲

smallerfish 9 hours ago | parent [-]

So make it a closed issue tracker with a public email gateway. Get Anthropic to donate LLM time to classify and combine incoming reports.

▲

throwaway85825 9 hours ago | parent [-]

If the LLM hallucinates bugs what makes you think any classification won't be hallucinated?

▲

quuxplusone 8 hours ago | parent [-]

The issue highlighted in Linus's message isn't that the LLM is hallucinating fake bugs; it's that 100 people running the same LLM on the same codebase find the same real bug 100 times, and if they all send it to the private security mailing list, it's (1) unmanageably high volume and (2) stupid security theater [because by definition any bad actor with the same LLM would find that bug — it's effectively public at that point].

	▲	throwaway85825 4 hours ago \| parent [-]
		You don't need an LLM to deduplicate bugs, just categorize by files affected. The real security problem is LLMs have a ~499/500 false positive rate and the new 'security research' post this slop and DDoS the mailing list.

▲

dgellow 8 hours ago | parent | prev | next [-]

You still spend time identifying duplicates and doing triage. That can be very significant for a project like Linux.

Interestingly enough doing that type of triage is something LLMs are actually great at

	▲	2 hours ago \| parent [-]
		[deleted]

▲

cduzz 9 hours ago | parent | prev | next [-]

If the AI is awesome at identifying security bugs in the linux kernel, it likely can also identify if the thing it's found is similar to something that is already found in the security mailing list?

Or, put another way -- what flags the duplicate? The filer or the system? If my cheese factory is measured by the volume of cheese instead of the quality, I'll churn out the cheese even if it's sloppy duplicated cheese. And that is the case if a person has to flag a new ticket as "same as this" or not.

What's that law that says that any sufficiently large problem turns into a moderation problem?

▲

crote 9 hours ago | parent | next [-]

The problem is that the tech companies are paying their research/marketing departments for headlines that go "Researcher uses powerful new Saga 6.2 release to find 597 kernel vulnerabilities! (Can your company afford NOT getting their $1000/month subscription?)", not for headlines that go "Researcher spends $50.000 to find 597 bugs, then spends $25.000 figuring out 540 of them are duplicates".

Unless the kernel community starts banning & publicly shaming repeat offenders, there's zero incentive for them to put any effort in filtering out duplicates. They are mostly doing it for marketing after all, not out of a genuine interest in making the kernel better.

▲

fiedzia 8 hours ago | parent | prev | next [-]

> it likely can also identify if the thing it's found is similar to something that is already found in the security mailing list?

It can not because this mailing list is not public.

▲

flumes_whims_ 8 hours ago | parent | prev [-]

> “AI detected bugs are pretty much by definition not secret, and treating them on some private list is a waste of time for everybody involved – and only makes that duplication worse because the reporters can't even see each other's reports.”

	▲	cduzz 8 hours ago \| parent [-]
		Ah; so it _is_ a tool problem. It is _also_ a moderation problem. One could ban orgs that flood the zone with AI generated trash, but is there some potential middle ground where there are sets of filters to identify duplicated bugs, and possibly just internally dump "AI spam" to a lower queue? This seems like the sort of problem I'd addressed in the 90s with killfiles and spamassassin. In other words, can't the ingestion just go through some filters to shield the humans at the end of the pipe?

▲

Cthulhu_ 8 hours ago | parent | prev | next [-]

While true, security reports should be treated as confidential until a patch is widely available.

▲

stonogo 8 hours ago | parent | prev [-]

And with a mailing list you don't even have to do that! The problem doesn't really change, because you have to figure out whether it is a duplicate before you can mark it as duplicate, and that's the 'managing' part of 'unmanageable'.