So could it actually turn around, like a full 360, and the image would stay the same? It looks super cool but the videos I saw just pan a little one way or the other

It could in theory. The model generates a depth image per frame, so each pixel becomes a small 3D point. It also assumes that the 3D scene is static. From this, you can then simply register all the frames into a huge 3D point cloud by unprojecting the pixels to 3D and render it anyway you like (using a classical 3D renderer) and it will be consistent.

Though, a problem is that if the generated video itself has inconsistent information, e.g., the object changes color between frames, then your point cloud would just be "consistently wrong". In practice this will lead to some blurry artifacts because you blend different inconsistent colors together. So when you turn around you will still see the same thing, but that thing is uglier and blurrier because it blends between inconsistent coloring.

It will also be difficult to put a virtual object into the generated scene, because you don't have the lighting information and the virtual object can't blend its color with the environment well.

Overall cool idea but obviously more interesting problems to be solved!