The Google drive is also hosted by Google on Google's servers. They already have access to everything in there.

I think the concern is that this might somehow enable a privacy policy they weren't aware of that permits training over the entire Drive. However, I think the primary reason for this is that these products generally would like to store data on the user's Google Drive but Google Drive doesn't have super granular permission structure to be able to set up a partitioned directory for the app alone. I actually think that might be a good thing to work on next?