I wish it'd be easy to plug it to a s3 backend and thumbnails etc. ephemeral things could just be on disk.

Yes, that would be the obvious solution. Database and thumbnails on disk, everything else on s3.