I have no ~/.config/git directory, ~/.config/git/config might take higher precedence, but ~/.gitconfig works too.

Yes you may have to create the directory :)