However, many (most?) specialists won't see you without a referral unless they are self pay.

Whether you need a referral typically depends on your insurance plan, not the doctor you want to see.

Yes. They don't have time to deal with hypochondriacs who just show up. Unless you are an existing patient asking for followup, or have a referral, many will not schedule you or will maybe give you an appointent six months from now.