The main problem with this paper is that this is not the work that federal judges do. Technical questions with straight right/wrong answers like this are given to clerks who prepare memos. Most of these judges haven't done this sort of analysis in decades, so the comparison has the flavor of "your sales-oriented CTO vs. Claude Code on setting up a Python environment."

As mentioned elsewhere in the thread, judges focus their efforts on thorny questions of law that don't have clear yes or no answers (they still have clerks prepare memos on these questions, but that's where they do their own reasoning versus just spot checking the technical analysis). That's where the insight and judgement of the human expert comes into play.

▲

arctic-true 7 hours ago | parent [-]

This is something I hadn’t considered. Most of the “mechanical” stuff is handed off to clerks - who, in turn, get a ringside seat to the real work of the judiciary, helping to prepare them to one day fill those shoes. (So please don’t get any ideas about automating away clerkships!)

	▲	sjudson 7 hours ago \| parent [-]
		Right. Clerks do the grunt work of this sort of analysis, which could easily be handed off to agents. They do this in order to get access to their real education: preparing and then defending to the judge the memos on those thorny legal questions. It would probably be a good thing for both clerks and judges to automate the sort of analysis this paper considers (with careful human verification, of course). That's not where the meat of anyone's job actually is.