What different routes are there to publish research besides academia? I would love to work on publications but it is not practical for me to return to an institution right now.

I've never seen a math journal that requires academic credentials or affiliation to submit a paper, and I've published several math papers without an academic affiliation. You can put your employer as your affiliation or even "Independent Researcher". The hard part is writing the papers themselves. Getting a paper in a decent math journal as an outsider is rare, not because the journals ban or are biased against outsiders, but just because it's rare for an outsider to write a decent math paper.

The sibling comments are good. Some additional info:

It's hard, there are a lot of i to dot and t to cross (and remember the j and f too).

The paper must have the correct length, a relevant introduction, conclusions and a lot of other form stuff, detailed but not too much detailed, the correct references, ...

And you must solve an interesting problem. The famous problems are too hard, a lot of people already tied to solve them. And "interesting" means what the community consider to be interesting, that sometimes it's what is interesting for the application and sometimes is something completely different.

And must be correct, in my university we got a few independent people (5?) claiming to have solved the Goldbach conjeture a few years ago. Unsurprisingly all had big mistakes.

My suggestion is to partner with someone already publishing math papers. Do you have a friend that is mathematician? Do you have a friend that has a friend that is mathematician? If your friends work in an uninteresting topic, you can ask for help to contact someone else that is working in something you like.

Be aware that there are abusive people, but most horror cases I heard are about experimental physics where a few extra hands are more helpful, mathematician usually don't lock people in the basement until they finish the experiments.

Once you understated how to publish, which journals are good, which journals are run by idiots, you can try to publish independently.

I don't know what your current industry is, but Google publishes their research and collaborates with academic institutions is one that immediately comes to mind. The government (NASA, NSA for one), tech companies (IBM, Microsoft, etc.), medical companies, aerospace/defense (JPL), just to name a few. I am sure there are way more than I could think of and I am sure others will care to fill in as well.

Research is not exclusive to academia.