Something I've noticed is LLMs seem to be able to answer questions on everything, in quite a lot of detail. But I can't seem to get them to actually do anything useful, you basically have to hand hold them the entire way to the point they don't really add value. I'm sure there is plenty of research in to this, but there does seem to be a big difference between being able to answer questions, and actual intelligence.

For example I have some product ideas in my head for things to 3D print, but I don't know enough about design to come up with the exact mechanisms and hinges for it. I've tried the chatbots but none of them can really tell me anything useful. But once I already know the answer, they can list all kinds of details and know all about the specific mechanisms. But are completely unable to suggest them to me when I don't mention them by name in the prompt.