I think the raw tokens embeddings shouldn't have been included in the PCA (past the first or second graph). Since they're disjoint from the conversation embeddings, you're essentially losing a dimension and compressing all the conversations into a plane. There may well have been much more obvious structure in the language embeddings that you'd be able to see.