The delete key thing comes from how Windows descends from DOS which always had the cursor ON a character (blinking box kind of thing) whereas the Mac had the cursor as a thin line BETWEEN characters.

Technically DOS could have had "backspace" (delete the character BEFORE the cursor), "delete" (delete the character on/under the cursor), and "forward delete" (delete the next character AFTER the cursor).