I've also been hosting all of my own infrastructure including email. I have only used small cheap VPSs and one thing I have learned is that the reputation of the IP address you are assigned may or may not have a history of delivering mail, and it's just luck if you get an IP that hasn't been blacklisted or known for SPAM in the past. I have put off migrating to a new hosting provider and actually just finished that task.

If you set up a new VPS for sendmail email, the first thing you should do is look up the IP address you have been assigned to assess it's reputation. If it's bad, you may be able to be assigned a new IP through some providers, but you can always just spin up a new VPS, check it's IP, and if it's clean, use it, otherwise drop the VPS, create a new one, and check the IP.

Another option is to buy your own CIDR range. The smallest subnet you can buy is a /24, but your addresses will be portable between ISPs giving you complete control over your IP reputation, and the ability to warm up IPs for sending when an existing IP you are sending from starts to get a negative reputation.

Always make sure you set your rDNS or PTR record to match the A record for the mail server's FQDN.

Set up opendkim, opendmarc, and policy-spf or equivalent, make sure your DNS is all aligned properly, and set your DMARC policy to enforce. You can start off enforcing on a small percentage of outgoing email, and if that looks good, set to reject 100% of DMARC failures.

My stack is postfix and dovecot with opendkim, opendmarc, policy-spf, with postgres to hold domains, mailboxes, aliases, etc and some simple scripts to get postfix and dovecot to use the postgres db. I used postgres for reasons not directly related to sending email, so the same thing can be accomplished using something like SQLite or even flat text files. I ended up writing a perl program years ago to manage the whole stack from a single cli command.

I'm not suggesting my stack is anything special. I use the daemons I use because I have been using them for years and understand them very well. If you want something more than just a TUI, there are a lot of options. I still use spamassassin, but I know there are newer alternatives.

The main driver behind how I chose the stack I use is knowing exactly how every part of it works.

Running your own email server without understanding how all of it works is what people mean when they say that you should not host your own email server. It's not enough to copy/paste from a tutorial, because you will run into issues in which something isn't working right, but since you just copy/pasted you have no idea how to fix it.

I worked for an online publishing company that sent a mixture of quality content to double-opt-in subscribers and transactional emails, and moving traffic around on your IPs if you have more than one to maintain reputation. We sent over 500 million emails a month using a combination of self-hosting our MTAs, and also using ESPs in some scenarios in which even the best IP/domain reputation practices aren't enough to get inboxed. Always set up feedback loops and maintain your lists if you are sending newsletters. Set google postmaster tools.

One of the reasons I run my own mail server is that I create aliases any time I sign up for an account on a website that are unique to that site. github@example.com would be an alias that forwards to an actual mailbox that is unknown to the rest of the world. This way I can tell who has sold my email address, or had their mailing list compromised, and I can simply remove the alias to stop any spam going to the real inbox. Doing that can reveal some pretty interesting things about how your information is bought and sold by data brokers.