I think we are about to enter a world where security compromises occur at least an order of magnitude more often due to AI. The fact that AI is sometimes wrong benefits the attacker because they can just automate attacking with agents until they breach a system and the failed attempts don't matter.

Nicholas Carlini has a good talk about the risks on his homepage https://nicholas.carlini.com/ (can't link to the talk directly for some reason)

▲

swatcoder 5 hours ago | parent [-]

The corollary is that service access will become increasingly penalized, authenticated, and finicky as the most general first line defense against automated "informed brute force" attackers will be to slow or ban access that looks like them.

If you've got a lot of roving bandits with fancy lockpicks, you're village needs to harden the walls, reduce the number of entry points, identify and document entrants more universally, check under more carriages and inside more trunks for stowaways, etc. And with all that comes delays, imprivacy, accusations, false positives.

The low trust world that we're headed towards isn't going to be nearly as fun.

	▲	Buttons840 an hour ago \| parent [-]
		Building on your analogy: Currently we spend a lot of time and effort making sure our walls appear nice and inviting, with lots of convenient entry points where advertisements can be placed. In the future we will build ugly walls with a few well guarded entrances. What I mean is, look at how much time and effort we spend on making software services beautiful and branded, and so little time spent on security. In the future we'll probably have fewer webpages and more hardened text-based APIs that end users interact with via their AI agents.