"This argument seems like an arbitrary limitation of the human mind, which I don't think holds up. "

And why not?

Evolution did not reward us for thinking about spaced out concepts, but for coming up with new ways to get food, outsmart the prey, build tools.

That thinking in 4D is helpful for building tools is a new thing so we evolution did not optmized for it (yet).

Most of our thinking now is about how we made rocks think, gods, and fiction. None of those are created by evolution