Playback is 100% handled by the device. The primary (and essentially only) benefit of H.264 is that almost every device in the entire world has an H.264 hardware decoder builtin to the chip, even extremely cheap devices.

AV1 hardware decoders are still rare so your device was probably resorting to software decoding, which is not ideal.