I would love to see these benchmarks split up across "generic" tasks and novel tasks.

For example, redlining an OEM agreement with my vendor. They pretty much all use the same template, and the objections are always the same. I tried using Claude first, and even my lawyer was impressed that it found things they missed. Then I used Claude to redline that type of contract and the other side said "your lawyer was surprisingly thorough". I had my lawyer do the final check, but saved a bunch of money doing the first three rounds with Claude instead. And my lawyer basically just said "looks good to me" by the time they reviewed it.

Or even writing a simple will or estate document.

It seems like LLMs are great for the kind of generic work you'd usually pawn off to a new associate or paralegal, but not for the important things that partners usually do.

> It seems like LLMs are great for the kind of generic work you'd usually pawn off to a new associate or paralegal

This seems to be the case with most industries that LLMs are impacting, coding, data input, legal etc.

Whilst it's great to get the time saving from not delegating to a junior (LLMs are much faster at the basic level 1 work) there's going to be this big knowledge gap that you really only gain from actually doing the mundane boring shit.

A lot of this _should_ be automated away, but we really need to think about how the next group of people are going to learn the skills and nuance to applying them. I don't really have an answer to that one other than still giving that work to junior staff, but they will then just utilise an LLM to help complete the job, and rightfully so, I'd have done exactly the same thing.

I think OpenAI tried doing this with some early chatgpt settings with "learning mode" or whatever that is. But an LLM just telling you what to do will never teach someone, there was something about having to go out and hunt for the answer through the sea of crap responses or wrong SO questions that really taught you something. How do you replicate that learning with an LLM and how do you apply that to different industries like Law?

I did much the same with a placement agency. Even uninformed about the law beyond the handful of things I worry about, it became clear the company had simply welded a bunch of different boilerplate, from different countries, without actually putting a real lawyer in the mix. Or just having a junior look at it.

Weirdly, I wonder the net effect of me helping them accidentally tighten their terms to something enforceable.

Hmm, Claude, make a note to just figure out if the doc holds together before I bother.

> And my lawyer basically just said "looks good to me" by the time they reviewed it.

Last year when i mentioned this application of LLMs somewhere on Reddit (in the course of arguing that white -collar workers are an endangered species) it was met with scorn by commenters who insisted that machines could never match the skill and care needed to write and review contracts and the like, at least not to the level demanded by the legal profession. Well maybe the human standard isn't that high to begin with; my dad was just telling me last night the story of how his lawyer had urged him to cave to a plaintiff's demands because the judge who was hearing the case "gets bored and doesn't like reading the documents." (The judge's son is currently running for the office of county prosecutor, lol.)

I've spent quite a lot of time with corporate counsels over the past dozen years and one of my consistent findings is that they -- except in areas of liability and intellectual property -- almost always defer to the sponsoring business executive in defining terms and specifying the details of the contract. They are present primarily to keep biz leaders and their counterparties operating within an acceptable risk profile but usually don't have strong opinions on much of anything else.

So, to me it isn't curious that your experience with Claude has been positive. Provided you know what you're doing and you aren't asking Claude to go too far outside the expected norm, there wouldn't typically be a compelling reason for most contract attorneys to push back.

Where I work, we're trying to let Legal use AI for the first pass of things for like financial "onboarding docs" we negotiate with clients. Some tweaks the client wants we probably don't care about, some need careful review by a proper lawyer. It would be great if we could run it, and then on the ones with only minor tweaks we don't care about get the doc signed immediately.

I'd worry that professionals have learned to look for certain problems, and the LLM may sometimes be keeping the problem while transmuting its signature to evade detection.

Kind of like how they can take crazy ideas and dress them in better vocabulary.

so they eliminate your talent pipeline and cut down on billable hours? is this good?

how much 'novel' tasks are there in law?

The Legal Department at my work did not have the same success you did.

Every AI tool they tested completely shit the bed and ended up requiring more time for contract review than a human review, and completely failed to spot any of the issues that were actually relevant to our industry. The AIs were also for some reason really focused on lack of consideration, even though that was not an issue with any of the tested contracts.

> It seems like LLMs are great for the kind of generic work you'd usually pawn off to a new associate or paralegal, but not for the important things that partners usually do.

This seems to be the common thread everywhere LLMs end up being useful. As the sole software developer? Pure trash, but as "tons of juniors we can shuffle boring tasks at": great. As the sole designer and maintainer of a website? It'll look like 150% slop and no will buy anything, but as a designers friend to create what they dictate: even a designer can throw together a good looking website quickly. Etc etc.