Running your own mail servers to do the volume emitted by Sendgrid would indeed be on the level of starting your own medium sized business. Getting IP allocation, swip'ing them out to divisions of your company or your customers and paying into whitelists for all the "free" email providers like Google et al would be a massive up front cost.

Running your own mail server for personal email is an afternoon of setup DKIM, DMARC, SPF, FCrDNS and such, setup of your MTA/IMAP/WEB preferences, tuning some filters, setting up aliases, accounts for family and with time the tuning work eventually slows down and then it's just maintaining accounts, aliases and the occasional rules to block problem networks and domains. With time you may find some servers that require lowering security or filters but that is also very easy.