What about VLC's own built-in versions of decoding libraries (I think, from the FFmpeg project)? Is there a scenario here where we may have to deal with malicious MP4 files?

All media containers are potentially hostile. Any offset, extent, or reference has to be considered hostile user-provided input.