The city of London literally has a city also named London inside it and yeah, "Where are you going?" "The City" means clearly that tiny part. There is another small older city inside London, Westminster, but if you meant Westminster you'd say "Westminster" never "The City" in general.