You're not wrong, but something can work well enough to still be useful despite falling short of the idea of a proof or any formal definition.

Let's say that 95% of individual humans can pass it and only 2% of bots. For someone maintaining a website, who has to decide between using this system and shutting down their site because of the increased costs, that may very well be good enough

That’s a pragmatic and understandable argument. And for an individual hobbyist site owner, that’s fine. Are we okay with excluding 1 person in 20 from the services of a midsized organization? What if they’re integral to the workplace? Or a major transport provider without differentiated competitors? What if the organization is a state government?

I'm one of the 5% apparently, cloudflare thinks I'm a bot. I suppose you'll exempt me in exchange for all my ID documents and bank statements?