I think I and most vegans also wouldn't eat it. It has nothing to do with rudeness or even the specific ethics of that situation or something. Just I wouldn't be able to physically stomach it. I would feel guilty but I wouldn't be able to eat it.

Best course of action imo is watch yourself in that moment, understand that people are going above and beyond for you even though they don't fully understand you, they are trying to accept you. I'd go to the bathroom, try to reason with myself that no animals are being killed specifically for you, the accidental touch won't give any flavour to your dish and it's all in your head. There's no ethical issues whatsoever in that situation.

If after that you still can't make yourself eat it then you should apologize, explain it to them, tell them you tried to make yourself but couldn't and I bet you'd get a LOT more sympathy.