you're completely missing the point.

Even if it's 'not hard' your brain has to compensate for switching to another realm/space and that takes energy and time especially if you haven't used that particular space for a long time.

This is backed by science. Go read up on short-term working memory and crystallized memory.

All this will add up the maintenance costs, so it had better be a good trade off.