The best case scenario for AI companies is, people receive those bug reports, look at the model that produced it and not even look at the details, just apply the fix mindlessly

This gives Anthropic a staggering amount of power. Oh it came from Mythos? We will just lose time trying to analyze it, better apply the fix ASAP

▲

stingraycharles 2 hours ago | parent [-]

> The best case scenario for AI companies is, people receive those bug reports, look at the model that produced it and not even look at the details, just apply the fix mindlessly

Do people maintaining serious software do this, though?

	▲	nextaccountic 17 minutes ago \| parent \| next [-]
		The problem is that serious software is drowning in AI vulnerability reports. There is not enough manpower to analyze them properly. And if you ignore the reports (like curl is doing in their 1-month vacation), malicious actors will just exploit them. At some point it's inevitable to just rubber stamp whatever is coming from AI. The actual, underlying problem is that software is buggy and current programming languages aren't fit for writing reliable software. There's a wide gap between the state of art in formal verification, and what is actually practiced in the industry. It's because of this general unreliability that AI has a large supply of vulnerabilities to find. The situation will only get better if software becomes reliable and written in solid foundations. My guess is that AI will be even more useful to verify software (something like, write Lean or Coq proofs that the software is not vulnerable, things like that), rather than finding vulnerabilities piecemeal but still letting software be written in unsuitable languages, with no formal verification to prevent bugs from sneaking through.
	▲	pixl97 an hour ago \| parent \| prev [-]
		Define 'serious'. There is a lot of software in serious places written by very unserious people.