Deepmind where experimenting with this https://github.com/google-deepmind/lab a few years ago.

Having AI agents learn to see, navigate and complete tasks in a 3d environment. I feel like it had more potential than LLMs to become an AGI (if that is possible).

They haven't touched it in a long time though. But Genie 3 makes me think they haven't completely dropped it.