for video there is another constraint - time

hardware encoders (like the ones in GPUs) typically work realtime-ish, so they do minimal exploration of encoding space

you also have the one-pass/two-pass thing which is key for unlocking high quality compression