Engineering uses repeatable processes to produce expected results. Margin is added to quantifiable elements of a system to reduce the likelihood of failures. You can't add margin on a black box generated by throwing spaghetti at the wall.

You can. We know the properties of materials based on experimentation. In the same way, we can statistically quantify the results that come out of any kind of spaghetti box, based on repeated trials. Just like it's done in many other fields. Science is based on repeated testing of hypotheses. You rarely get black and white answers, just results that suggest things. Like the tensile strength of some particular steel alloy or something.

> Engineering uses repeatable processes to produce expected results

this is the thing with LLMs, the response to a prompt is not guaranteed to be repeatable. Why would you use something like that in an automation where repeatability is required? That's the whole point of automation, repeatability. Would you use a while loop that you can expect to iterate the specified number of times _almost_ every time?

Practically everything engineers have to interact with and consider are equivalent to a software black box. Rainfall, winds, tectonic shifts, material properties, etc. Humans don't have the source code to these things. We observe them, we quantify them, notice trends, model the observations, and we apply statistical analysis on them.

And it's possible that a real engineer might do all this with an AI model and then determine it's not adequate and choose to not use it.