The polite way to do this is on receiving the invite, ask for clarification of what's expected from you at the meeting. Combine that with pointing the inviter at someone else who is also invited and you can politely express that your presence is not necessary.

Or just counter-schedule a meeting in the same slot.