I suppose that's an option yeah, but when people envision turning this kind of thing into a VR holodeck I think they're expecting unbounded exploration and interactivity, which precludes pre-baking everything. Flattening the scene into a diorama kind of defeats the point.

I actually would rather it be a 3d model so that I don't need to believe they're microwaving a goddamn full size whale for 45 minutes (worth of electricity)

You just need to prerender the 3d world. If its truly exportable as a 3D model, rerendering it real time based on input is trivial