You could implement it this way:

- the first time you select a directory it must be empty

- you can drag files in there afterwards

- the directory gets whitelisted for future use

Probably has bad usability, but would be more secure.