If you're using Signal, of all things, that's probably a reasonable security/usability trade off.

Granted, I'm sure there are a lot of people out there who just use Signal because one of their more security-conscious friends/family members told them to install it, and so you're probably right for those people.

But, frankly, I can't see how else they could do it. Offering an unencrypted option, or weakening the encryption (by storing a copy of the key on Signal's servers) would make Signal not Signal.