If you aren't saving people's phone numbers in your own contacts, signal isn't storing them in group chats (and even if you are, it doesn't say which number, just that you have a contact with them).

Signal doesn't share numbers by default and hasn't for a few years now. And you can toggle a setting to remove your number from contact discovery/lookup entirely if you are so inclined.