In theory yes, though depending on the complexity of your game you may need to bundle a lot of userspace libraries and other programs along with your kernel to make it work. Most graphical applications expect a display server like X11 or Wayland to talk to, at minimum.

Yeah, that's the hard part (but also the appeal). How minimal can I go and still have a single-use system. Maybe a holiday project...