> How it is made should be irrelevant in the process.

Is that true? Almost all answers have value based on how they were reached, especially when (like this case) there's no trivial and objective verification step.

For example, suppose you live next to a volcano, and I sell you some software that predicts when it will erupt. One day you dig into it and realize I just hard-coded "not today" [0] which is 99.999% accurate.

I think you would be quite angry, and justifiably so... But why would that be if "not today" was correct, and "how it's made is irrelevant to the process?"

Tying it back to the court case, we expect professional witnesses to look at the details and then reach a conclusion, and to reach out based their own professional expert knowledge of cause and effect... Not to pick a conclusion and try to make the details fit, nor to offload executive function to an LLM. Even if they render the same boolean verdict, we care about what process was taken.

[0] https://xkcd.com/221/

Totally agree on your example:

> One day you dig into it and realize I just hard-coded "not today" [0] which is 99.999% accurate.

It only becomes visible when you do the research.

The justice process in this example just has to check and verify the inputs. There is no way around it. Today it is this expert, tomorrow another one.

When someone is caught that's an ethical issue and there could be more structural consequences as well. Like blocking the person as an expert or other means. As the expert becomes untrustworthy.

“How” as in “what tool” is a different “how” as in “with what intent”. LLM was used to produce a bunch of arguments supporting a pre-intended outcome. Doesn’t matter much whether it was a machine or a human (and whether it was “expert” himself or a ghostwriter), but it does matter - a lot - how the outcome was the pre-selected and the task was to cherry-pick the facts to support it.