The reason you wouldn't do that is IP doesn't give you a mechanism to share an IP address with multiple processes on a host, it just gets your packets to a particular host.

As soon as you start thinking about having multiple services on a host you end up with the idea of having a service id or "port"

UDP or UDP Lite gives you exactly that at the cost of 8 bytes, so there's no real value in not just putting everything on top of UDP