My dad once told me that being smart was a curse in some ways, but he framed it as having a World view that few others had.

I think however your take on it is closer: when you're smart, you're told you're smart (both as a positive and as a negative - I was bullied for it), it just becomes part of your self dialogue and identity and it gets in the way.

It's taken years of therapy for me to get to the point where I realised none of that really mattered as much as having great relationships with people I liked, and going for long walks in green spaces and not thinking about anything.

RE self dialogue: I think it is also self-reinforcing in a positive way. "I am competent and capable of this" is certainly less distracting internal dialogue than "maybe I'm not smart enough" to children. I believe most adults eventually get better at ignoring this type of thing altogether, since it isn't really useful or productive.