8 years ago I graduated with a bachelors degree in pure math and theoretical physics. 2 years later I settled down and decided to just become a software engineer building CRUD apps. Since graduating I've consistently looked for a job that explicitly had software and pure math requirements, but I've continuously come up short.

In theory, I think two good avenues for you to check out would be: 1) building physics and game engines at a company like Unity; 2) building the knowledge engine behind something like WolframAlpha/Mathematica. It's possible you'll need graduate school experience to be competitive for these jobs.

I'd guess that no matter what you're going to end up doing some kind of programming. I don't think formal education is required for this.