Basically all up to the training data, as things often are.

I wonder if that might be another reason to just completely disable this feature and not make it a permission: otherwise people could use it to build trainingsets for geoguesser models.

You still need some smarts, since the picture you just took won't be in the training data.