My milage does vary. I swore off caffeine for a decade. Then I discovered that stims restore some cognitive function that I had written off.

The difference is that my anxiety is more interesting (to me) than distressing. I can sometimes leverage it as a mechanism for change.

Granted - this also possible because my anxiety (currently) falls within a range. Turn it up a ½doz notches and I probably won't be mining it for usefulness.

Have you tried tea instead? Its effect is markedly different from coffee. THere should be no anxiety, but still contain that cognitive benefit.