>imagine whole movies shot in this format. The fly-on-the-wall fantasy of movies, without being locked in place for the whole shot.

That would be terrible. Framing is the major expressive feature in cinematography, and any interactive format needs a lot more thought put into it than just having a free camera. Literally the worst of both worlds.

Light field video streaming is a thing, however it's pretty niche. OTOY pioneered holographic codecs and light field videos with some degree of freedom and a sense of depth more than a decade ago.

Not terrible, per se, but not a movie. There are other formats that fit the task better like immersive theater and theater in the round.

This has been extensively explored with VR and games, though.