Calling DotA just a custom map is a bit of a stretch. That was merely the packaging. These "custom maps" had various scripting capabilities that made them more than just some terrain.

Also custom maps are mods by definitions anyways, with the exception of games where the creation of maps is a component of gameplay.

I mean, to those who played them, 'custom map' is basically just a term of art indicating the things you said. In the parlance of the mid-2000s WC3 scene, you would call them custom games or custom maps.

Or, if you were slightly older, you might call them UMS, as they were in Starcraft. Short for "Use Map Settings", indicating that the game logic should come from the scripts and triggers in the map file rather than the built-in logic for ladder games.