Exactly. I'm not able to memorize multiplication tables or even listen to math lectures - I can't hold more than one or two numbers in my head before they start disappearing. However, I am capable of learning and using math to an advanced level, at least temporarily.

How? The way I think of numbers is actually quite similar to what is currently known as "new math". It's not taught well because almost no teacher was raised with it (and thus lack the ability to think in new math "natively"), but it is based on something real.