I live in a city (a quite small one) and if I say that I'm going into the city, then it means I leave my neighbourhood (it's in the same city) and I'm going to the city centre.

Same. I'm in Berlin, I may say "I'm going to the city/town/village" for different parts of the city-state.