This gets into spoiler territory, but the walls in Attack on Titan weren't made with human labor. The first hint that something funky is going on was at the end of the first season finale, but we don't get the full story about the walls until much later.

It also was built when Marley was still very much a massive empire. So it wasn't limited to the manpower inside the walls. This is clear from how many of it's actual construction materials were used.