I was just curious asked chatgpt:

How can I make a really great steak myself? What to look out for, what to do. Give me detailed instructions.

I got a really good answer (including - cut, temperature and reverse searing), so I probably won't even need understanding. (Oh - and yes, I can cook - the answer mirrors what I already know).

Yes, it was a very, very poor analogy, because cooking a steak is a well known and well documented thing where LLMS produce good output. They fail when you get off the beaten path and try to do something that's not "mainstream".

AI has proved several times they can prove some complex math things that humans have been trying for decades so solve. They have not failed and have clearly done complex things humans cannot. Since humans have not done it before you cannot call it on the beaten path.