Then don't do that?

Install vanilla LineageOS or GrapheneOS with no GApps. Then install Signal from https://signal.org/android/apk/