I like this and felt like some of it was much more fluid; but was I alone in feeling like the interjected "uh-huh" or "yeah?" moments felt a little jarring?

Almost felt a bit *uncanny valley* for what "natural" conversation is supposed to be like. If the "uh huh" isn't timed correctly, it'll feel like a zoom call with lag.

Every second of the interaction is uncomfortable to me, but I also have extreme difficulty with video calls with humans. The latency completely breaks my mind.

I agree. Many times those little interjections don't feel natural. It's impressive, but there's still a lot of room for improvement.