If it's technically.implemented uniformely without thought of user access patterns, then yes it's a bad idea causing friction and frustration.

If I drag&drop a file in an app or it's icon, or by right click. The permission would be implicit.

For configuration files they could be part of allow rules, since they are so common if they follow the XDG specification.

For saving files, flatpak apps could be allowed to save anywhere in the users home sub-directories (not home top-level) as long as it doesn't overwrite any existing file. There are downloads, public, documents pictures, videos, that are barely used by default.

There is plenty of room for better desktop based integration, while of course lacking funding and care for users (thinking of particular devs in this case).

Currently it's a mishmash of separately using flatseal (and equivalents) and having your fingers crossed if you're likely to exfiltrate or not your data.