Don't take it personally; not inviting them after a few times is enough.