Yeah, if they refuse then that is also a signal about how much they truly value you (and not an insignificant one either).

If someone refuses, then you can frame that it limits candor and vulnerability. If they still prefer to have it on, then you can simply be less open with them.