Is there some "no means no" additional setting that could be added where someone has to go into settings that would prevent that?

I fear that with the notifications pop up asking me this I might hit the wrong button and woops turn it on.

Try from inside the signal app itself instead of system settings? On android Signal has an option at hamburger menu > Settings > Notifications > Notifications (toggle switch)

Oh... hmm, two toggles actually. One at Settings > Notifications > Calls > Notifications toggle, and the other at Settings > Notifications > Messages > Notifications toggle