I do it too, but I "buffer" one person's speech while I process the other's, and vice versa. Do you process both at once?

I wonder if reading techniques (i.e. human TTS) are also a bit like this. You have to "read ahead" and maintain a mental buffer, in order to parse the sentences so that you can put emphasis in the correct places, timing etc. So the eyes and mind are ahead of the speech.