I think it can be an error labeling people as "fast" or "slow". I had similar self doubts to the OP during my PhD, where so many people around me would say they "got" a concept and I was just feeling behind. But a few years in, while desperately trying to "catch up" I realized a good portion of the time I was just misinterpreting. Even those tasks aren't as well defined as the OP suggests.

There's another dimension that often is not acknowledged: depth. People have different thresholds at where they're comfortable talking about a topic or saying they "understand". I also don't think there's a strong correlation with the person's intelligence, if anything, a slight bias towards "slower" people being smarter.

  - Dumb people might have low thresholds as they are unaware of depth. 
  - Smart people will have low thresholds because they do better thinking out loud or are just saying they think they have enough to launch off of. 
  - Dumb people might be slow because they haven't thought about the thing very much. 
  - Smart people might be slow because they are considering different depths.

We all think fast and slow at times (intended), and we're all 4 people in the above list on different topics. I think we should just make sure we're judging people at the right race. The trouble is despite standing in front of you, talking face to face, you don't know if in that time they've run a few meters or a few kilometers. I think we'd all do better if we worried a little less about speed. If your destination is nowhere, you get there in the same time regardless of your speed.

>“Your 100m sprint time isn't going to tell us much about your 400m time”

…

Statistically it should actually tell us a lot about your 400m sprint time.