The problem are the lazy idiots using the software. This is the reason we need capable humans in the loop for anything important.

Here we have a bad joke. The system flags it. The school sends it to the police. The police detain and interrogate the kid. Everyone is treating the determination of a complex automated system as their own determination. We also have every actor treating this as a credible threat. For this to be credible, you have to have the means to accomplish it. They gave a timeline. You know you have time to investigate before making an arrest. Problem is, nobody cares.

▲

yard2010 5 days ago | parent [-]

If we have to rely on competent humans, we are doomed.

	▲	giantg2 5 days ago \| parent \| next [-]
		Any system will have failures. The goal is to eliminate some of those failures through multiple checks. The automated system can help by bringing up possible issues that couldn't be monitored by humans. But then it should be humans reviewing these alerts for false positives. It should be much like using a security scanning tool. Findings will come up and you need someone to disposition them. Some will need no actions false positives or issues under your risk tolerance, but then others need appropriate responses. It seemes in the cases like the example, you have a intern running the scan tool and turning every finding into a high vulnerability because they don't know any better than to blindly trust the tool.
	▲	1718627440 5 days ago \| parent \| prev [-]
		We always rely on humans being competent. If we forget that and treat everything as being correct then we are doomed.