I stopped drinking coffee for a month and didn't notice any difference in my anxiety/mood/etc, so I returned to my regular schedule (3-4 large cups in the mornings).

Same here, and it made me wonder, why did I go back to drinking coffee?

My best explanation is that there are effects, I just suck at self-awareness :)