Client-server multiplayer games are already kind of a very specialized type of video playback if you squint a bit (you're transmitting entities rather than pixels).

This method of multiplayer you propose is inferior in basically every way: you can't do client-side prediction to make inputs feel smoother, and non-trivial scenes will surely take up more bandwidth than just transmitting entity deltas.