The math isn't that difficult once you grok mod math. It's like time, like doing addition and subtraction on a clock. What's 10 + 4 on a clock? 4 hours past 10 is 2.

The math stays difficult after basic discrete concepts and gets more difficult as you go. :)

It's straightforward to get yourself to a place where you can do cryptographic things and feel somewhat comfortable with what's happening. Truly understanding it to the point where you can reason safely about it is deceptively harder.