Except that's not what "confidence man" means. It means that you gain their confidence/trust, then betray it.

A con man often uses the illusion of confidence to gain trust, though that's not the only way. The reverse also works: gain their trust by seeming unconfident and incapable, and thus easily taken advantage of.