Because it will become unpractical. It’s like saying your SO shouldn’t have access to your bedroom, or the maid should only have access to a single room. Instead what you do is having trusted people and put everything important in a safe.

In my case, I either use apt (pipx for yt-dlp), or use a VM.

I don't agree that the only options are "give it almost everything" or "give it nothing and now it's a huge pain in the arse". Which seems to be what you implied. I do think there are better middle grounds where an app almost always works out of the box but also can't access almost everything on the system. There are also UI changes that can help deal with this like the Android security prompts do.