I do it regularly. But, I am not rude about it, I do not insult the meeting nor its organizer. I used the sentence "Hi, am I necessary for the X meeting?" Followed by quick reason. Quick reason would be:

- "I have many meetings already and worry about not meeting deadline".

- "I do not have strong opinions either way, so am fine getting just outcome".

- "I do not have knowledge to be useful".

- "It seems like I wont be useful".

Basically, you can guess these from the topic alone. Most of the time, the response is something like "of course no problem" sometimes followed by short explanation why I was called in. Occasionally they say they indeed want me there.