you read very far into a relatively normal phrase "go to visit" meaning "when i visit them". Some people just say "visit" instead of "hang out" or "bum around with."

eta: it's like some people say they have to do errands.