This is why you need either a rootless container setup or user namespaces to remap the container user to irrelevant host users. https://docs.docker.com/engine/security/userns-remap/

Weak that this isn't the default.

Is there a mitigation for Mac? Can you do the same with eg Lima or is this just a Docker thing?

User namespaces significantly rise the risk of exploits and many setups disable them. One may argue that Docker should have used them when they were available, but that would break too many useful setups involving privileged containers.