You can use "Include file/location" in your ~/ssh/config.

I don't understand, though, why you would not want to init a git repo in ~/ssh? What am I missing? It's not like "having version control" is the same as "upload it".