It is end-to-end encrypted, but aren't chat history backups stored in plain text on some server?

These two things are mutually exclusive. They can store encrypted backups on a server somewhere, but unless they have the decryption keys from both parties (E2E) then no, it can't be stored in plain-text.