No UX needs to be perfect for everyone, but this doesn't sound trivial to make reliable.

First things that came to mind:

  - facial hair
  - getting people to learn to make bigger mouth movements and not mumble
  - we're constantly self-correcting our speech as we hear our voice. This removes the feedback loop.
  - non english languages (god forbid bilingualism)
  - camera angles and head movement