If I ask a SOTA model to just implement some functionality, it doesn’t necessarily do so using a great architectural approach.

Whenever I ask a SOTA model about architecture recommendations, and frame the problem correctly, I get top notch answers every single time.

LLMs are terrific software architects. And that’s not surprising, there has to be tons of great advice on how to correctly build software in the training corpus.

They simply aren’t great software architects by default.

You know that if you ask the LLM correctly you get top notch answers, because you have the experience to judge if the answer is top notch or not.

I spend a couple of hours per week teaching software architecture to a junior in my team, because he has not the experience to not only ask correctly but also assess the quality of the answer from the LLM.