There's an HN thread from yesterday in which people are extolling the ability of these vectorial databases to practice law because most of them don't understand how LLMs work. They assume that LLMs "understand" what they're being asked and what they're regurgitating.

Lane Kiffin almost destroyed LSU's football program acting on legal advice from ChatGPT. A video game publisher owes the former owners of a studio it acquired $200+ million because he based his actions on legal advice from ChatGPT. In the past week alone, California has disciplined over a dozen attorneys for LLM hallucinations because they used LLMs (mostly ChatGPT) to produce their legal pleadings.

And that's in an area where there are multiple safeguards to catch the issues before they become permanent problems. There's absolutely no justification for using AI in warfare, where mistakes tend to be pretty final.