I've found that WebM works much better because of the structure of the data in the container. I've also gone down similar routes using outdated tech and even inventing my own encoder and decoders trying to smooth things out but what I've currently found is the best current approach is using WebM because it is easier to lean on hardware encoder and decoders including across browsers with the new WebCodecs APIs. What I've been working on is a little different than what is in this post but but I'm pretty sure this logic still stands.