Having a spatial understanding from an ASCII map, while doing long term planning. Just try making an AI play nethack or similar

Convert it to an image on the fly to feed it into a vision language model and I expect it would work just fine.