It's been a while, but I used to work at WhatsApp and we used Erlang distribution heavily. I understand the clusters have gotten really huge since I left.

It's super handy. There's no security barrier between nodes. It's a headache if your network is unreliable.

For a chat app, messaging someone becomes a series of steps:

a) look up if they're online (send a message to the presence database service)

b) if you got a process id back, that's the process connected to the user, so send it the message. The process could be on the same machine or not, but the sending api is the same. This is the special part: few other environments make arbitrary messaging between processes/threads/tasks/whathaveyou so pervasive.

c) if you don't get a process id back, the user is offline; send the message to the offline database.