google "metadata in MKV video file" seems to show that MKV already supports something like that, and mkv is already a container format. One problem would be that some collections of video acquired from dozens of sources/people who did the encode won't by any means be 100% standards compliant MKV.