Does that mean Signal has access to my contacts? So much for privacy!

It does not, actually. Signal only gets the contact number if you click on the button.

	▲	crtasm 3 days ago \| parent [-]
		In my case I allowed Signal to access my contacts and these options only appear for people who use it, don't know what happens if you deny the access.