The exact same concern already existed without LLMs. It is called social engineering, and has been a known risk for a while.