It's moslty due to how the protocol works. Sending and Receiving are asynchronous and may or may not follow the same path.

When sending the other side is the server, when receiving, you are the one. The protocol also has the concept of relaying, where message can go trough intermediary nodes.

So you can set one server for receiving email (yours) and another one for sending (smtpgo).

For actually reading, the server for receiving need to deliver the message (either built-in feature or through a program) to the correct mailbox. That would probably means writing it on disk somewhere. And then you need to use a mail user agent (MUA) to read that file. Protocol like imaps and pop3 are for transfering your inbox to another computer, not for receiving or delivering email.