Remix.run Logo
▲ otterley 8 hours ago

As an attorney, I am not particularly impressed about a 20% pass rate on a benchmark. The name of the game for litigation practice is about being prepared for novel situations, not relitigating past ones.

▲ctolsen 7 hours ago | parent | next [-]

To be fair to them, on this particular benchmark (Harvey's Legal Agent) 20% puts it almost top of the pack of _any_ model, and certainly top for any remotely in its price class.

Opus 5.5 scores 4% on it.

▲henryfjordan 5 hours ago | parent | prev | next [-]

AI is coming for the transactional work long before it will come for litigation. In 50 years, nobody is going to have a human write their will, for instance.

▲NoboruWataya an hour ago | parent | next [-]

Writing a will isn't really "transactional", there is no negotiation involved. And wills are already a highly commoditised legal product. (So I agree with you that will drafting is ripe for disruption by LLMs, but don't think this indicates a broader disruption of transactional work or the profession generally.)

▲HDThoreaun 3 hours ago | parent | prev | next [-]

50 years is way too long a time horizon to predict much of anything

▲nprateem 5 hours ago | parent | prev [-]

In 50 years time 95% of legal complaints will be decided by AI

▲ozgrakkurt 7 hours ago | parent | prev | next [-]

It is amusing how many people are arguing what litigation is here, probably none of these people have a law education.

Imagine a bunch of lawyers arguing what programming is. Even a bunch of programmers arguing what programming is abnoxious at worst and a mental exercise at best.

▲tclancy 5 minutes ago | parent | next [-]

How does that track though? If, as a programmer, I claim programming is really the tasks a principle engineer does at Google or Netflix, is that accurate or survivorship bias from someone who made it to the top of the chain?

If lawyerin’ was always doing something novel, it wouldn’t be much of a set of laws. The bulk of lawyerin’ in the US is, I would guess without a whit of substantiation, a lot of small claims and standard form type things. Not discounting the value of the profession in any way, but if one job based on standard codes can have its basic functions replaced by AI, why not another?

▲hansvm 6 hours ago | parent | prev | next [-]

That's how I feel reading HN's opinions on the recent math results. <5% of the responses even understand the point they're supposed to be arguing. AI might take over mathematics. It might not. The discussion regarding that topic is almost devoid of any relevant information though.

▲otterley 5 hours ago | parent [-]

I'd extend that to the typical HNer's understanding about practically any subject. A lot of bluster and supposition, little first-hand knowledge or expertise.

▲caaqil 6 hours ago | parent | prev | next [-]

> probably none of these people have a law education.

Domain knowledge is informative sometimes but mainly overrated: the success of LLMs proves just that. What most lawyers do is fairly standardized and routine, efforts to invoke mystery cases or "novel" is an attempt at resisting the inevitable. The "the only British food in Palo Alto" equivalent of unique skills.

▲idiotsecant 10 minutes ago | parent [-]

>What most lawyers do is fairly standardized and routine

True of...literally every occupation.

▲jester997 6 hours ago | parent | prev [-]

Lawyers would be better suited to argue what programming is because they know logic. There’s not much to algorithms: do this, then do that.

Programmers typically don’t know law outside of which license allows them to get code for free.

▲konart 4 hours ago | parent | next [-]

> There’s not much to algorithms: do this, then do that.

Yeah, and how many of drivers or software engineers can fix their car's engine? Just a few. Even though in priciple there's not much to it.

▲jester997 21 minutes ago | parent [-]

It’s a joke…

▲KellyCriterion 5 hours ago | parent | prev [-]

a) I have never met a Lawyer who was able to write a single code line

b) I have met Lawyer colleagues who didnt know how build a =SUM() in Excel

So, I doubt....

▲otterley 5 hours ago | parent | next [-]

> a) I have never met a Lawyer who was able to write a single code line

That doesn't mean they don't exist. I went to law school with many former software engineers, many of whom are now prosecuting patents on behalf of high tech companies, and some of whom went back to engineering (like me).

"I have never met ______" is of no consequence to anyone; it's an argument style best avoided unless you're talking about deities or space aliens.

▲jester997 11 minutes ago | parent [-]

I met a space alien! His name is Grok and he talks to me when I’m sleeping.

▲jacquesm 3 hours ago | parent | prev [-]

There are a couple right here on HN.

▲alansaber 7 hours ago | parent | prev | next [-]

Objectively a 20% pass rate is not impressive. Yet, there is still utility to AI research and drafting.

▲throwaway89864 7 hours ago | parent [-]

Wouldn't be surprised, if a human attorney would get 5%, while copy-pasting ChatGPT answers, report in the results a week past the deadline, with a $12,500 retainer charged upfront.

▲newsclues 7 hours ago | parent | prev | next [-]

If professional AI can do all the work that isn’t novel, you free up human talent for edge cases and then improve your AI with new information.

How much of your work is truly novel and unique? Not 100%

▲otterley 6 hours ago | parent [-]

Totally fair. However, even if the LLM would come to the same conclusion and produce the same content that a competent attorney would, the attorney must still thoroughly vet the work for accuracy. So there is potentially time saved in research and drafting, but it’s not the miracle some might characterize it as.

An 80% miss rate on a benchmark suggests to me that automation has a very long way to go, even for the drafting and research phases.

▲jacquesm 3 hours ago | parent [-]

And fixing a broken draft can be more work than writing a proper draft in the first place.

▲ 7 hours ago | parent | prev | next [-]
[deleted]
▲linuxftw 7 hours ago | parent | prev | next [-]

What are you talking about? 90% of litigation is convincing the judge that existing case law that clearly applies to your case does apply to your case.

▲seizethecheese 7 hours ago | parent | next [-]

I'm guessing what they're saying is that nothing goes to litigation if it's a clear pattern-match to existing cases, so actual litigation practice is only novel cases.

▲ajcp 7 hours ago | parent | prev | next [-]

Isn't that just in the US/Commonwealth judicial system? I could be completely wrong on how others, like non-adversarial judicial systems, work and/or how pervasive the US system is or is not.

EDIT: specifically around the use of judicial precedent in case-law.

▲treetalker 4 hours ago | parent [-]

Look at all these benighted folks who believe that American judges adhere to precedent when it’s brought to their attention!

▲dolebirchwood 7 hours ago | parent | prev [-]

90% of litigation is pressuring a settlement.

▲intrasight 7 hours ago | parent | next [-]

> 90% of litigation is pressuring a settlement.

90% of legal work where there is a plaintiff and defendant is pressuring a settlement. My wife, an attorney, does not consider that to be "litigation".

▲dolebirchwood 6 hours ago | parent [-]

As a former litigator, I would consider that to be litigation the moment a complaint is filed in court, regardless of whether the dispute is settled without trial.

But, for your sake, it's probably best that you accept your wife's definition.

▲xmprt 7 hours ago | parent | prev [-]

And most of the leverage you have in negotiating a settlement is by referencing prior case law. If something is novel, there's too much variance to get a good settlement.

▲dolebirchwood 6 hours ago | parent [-]

No, most of the leverage you have is how much time and money is on your side vs. the opposing party.

▲xmprt 5 hours ago | parent [-]

Money doesn't mean anything if you don't have a case to begin with. More money/time just makes the lawyers win because they end up siphoning most of the settlement. And more money with crappy lawyers will always lose to less money on smart lawyers who are able to use AI to find 100 examples of prior cases going your way. That's the whole point of AI disrupting the field.

▲jhassell 7 hours ago | parent | prev | next [-]

[flagged]

▲drivebyhooting 7 hours ago | parent | prev [-]

How much novelty can there be? If attorneys were as diligent and generous with “open sourcing” their data as SWEs had been, could LLMs benefit from all that training data?

▲keoneflick 7 hours ago | parent | next [-]

> How much novelty can there be?

As much novelty as there is in human interactions.

▲74948494i4 7 hours ago | parent | prev [-]

> How much novelty can there be?

Infinite

▲Synthetic7346 7 hours ago | parent [-]

In the same sense there could be infinite CRUD apps?

▲oblio 7 hours ago | parent | next [-]

Law and code are only superficially similar as law always requires human interpretation plus law is an interface to reality. Code can stay contained in its nice, little, abstract box.

▲drivebyhooting 7 hours ago | parent | next [-]

So every lawsuit dispute, divorce, traffic injury, etc… is a completely novel case that requires depths of creativity far beyond simple CRUD apps or solving 377 open math problems.

Makes sense.

▲dfdydx 7 hours ago | parent | prev [-]

Isn’t code an interface to reality as well?

▲oblio 7 hours ago | parent [-]

A lot of code is an interface to other code. Nginx is how many layers removed from actual physics?

▲benatkin 6 hours ago | parent | prev [-]

In the same sense as infinite jest.