Modifications can be licensed differently but that takes extra work. If I release a project with the MIT license at the top of each file and you download my project and make a 1-line change which you then redistribute, you need to explicitly mark that line as having a different license from the rest of the file otherwise it could be interpreted as also being MIT licensed.

You also could not legally remove the MIT license from those files and distribute with all rights reserved. My original granting of permission to modify and redistribute continues downstream.