I've had LLMs send me down complete rabbit holes for questions that are very specific. Here's an example:

We use glowroot (and open source JAVA APM). I was trying to compile it on my mac and some of the protobuf Maven plugins threw an issue. I gave copilot the entire pom.xml and the specific error and the versions being used. It send me on a complete wild goose chase and hallucinated like crazy even suggesting version upgrades to versions that do not exist or recommending parameters that have no use in the plugin.

Long story short, I just went to the github issues page of the maven plugin and searched and someone had posted a solution. Again, the solution wasn't new. It was suggested around apple started using ARM for their laptops. It was there in github and yet copilot hallucinated.

So, I don't feel too confident of coding assistants. Yes, they do a decent enough job to get your boilerplate done. But they're hopeless to resolve specific issues.