I’m not sure if this is the right solution, but we clearly need better tools for the legal space. Anthropic's recent policy update makes this abundantly clear:

“When our products are used in ways that can affect someone's health, legal rights, finances, livelihood, or access to essential services, we require there to be a qualified ‘human in the loop’ (that is, someone who has the authority to review and, if necessary, change Claude’s recommendations), and for the individual affected to be told that AI was used.”

Something as common as a family law case has the potential to affect all of the above. Now imagine being a petitioner, respondent, paralegal, or other 3rd party using Claude to review a draft document only to find out that Anthropic has sent your entire file to opposing counsel.

It sounds egregious, but Anthropic is clearly stating they will use their own judgment to control what people can do with its product, and that they will treat customer data as if it's theirs to disseminate. So there's a massive opportunity for an unbiased tool willing to fill this void.

▲

ChadNauseam 3 hours ago | parent | next [-]

I must be misunderstanding something because this is how your comment reads to me

> Anthropic: you can't use our product to fully-autonomously make decisions that affect human health, finances, etc with no human in the loop.

> victor9000: If you review a draft document with claude, anthropic will send your entire file to opposing counsel

What is the connection here? Anthropic has obviously never done anything like this and it's not indicated that they will by the thing you quoted.

▲

fultonn 3 hours ago | parent | next [-]

> Anthropic has obviously never done anything like this

If you are ascribing intent ("Anthropic designed a system that intentionally does this") or emergent model intent ("OMG at AGI") then I mostly agree.

But there's a more pedestrian version of this, where an unwise person puts a model in an open loop without sufficient HiL checking and then the model does dumb things. There are even some recent stories about OpenAI and Anthropic systems "emailing the FBI", which is sufficiently close to "emailing opposing counsel" [1][2].

But even if you feel those stories are trolling/marketing/etc., I think it's clear that there's a plausible risk model here. If you hook up an LLM to a toolset that has access to Outlook and your file system and opposing counsel's email address, then of course the dumb thing could happen. The LLM doesn't need to be AGI/ASI; in fact, to me, the worse the model capability the more likely this sort of screw-up feels. There are even (non-sarcastic) instructions or CoT roll-outs I could imagine which might elicit a sequence of tool calls the net effect of which is an errant disclosure.

So this feels less "unforeseeable lightning strike" and more "car accident while driving at speed". I'd put the HF hack in that bucket as well.

A sufficiently careful engineer and a well-trained expert operator can together properly constrain the system to not do the dangerous thing. But, in general, that is not how agentic systems or their underlying models are currently being designed, and most users are not receiving sufficient training.

A lot of these LLM-backed products feel very poorly engineered, rushed, and optimized for brain-feel instead of surfacing and forcing the user to acknowledge the large quantities of actual essential complexity in play. It's all very "unsafe at any speed".

[1] https://tech.yahoo.com/ai/chatgpt/articles/chatgpt-allegedly...

[2] https://www.bgr.com/2055251/anthropic-ai-claude-contact-fbi/

[3] https://en.wikipedia.org/wiki/Unsafe_at_Any_Speed

▲

Terr_ 3 hours ago | parent | prev [-]

I'm admittedly reaching here to form a charitable explanation, but in another thread [0] someone says one of the benchmarks has a defense case where it says the "correct" lawyer-bot action is to is "snitch" on the client.

Though that doesn't seem like a corporate terms-of-use thing as much as a training issue.

[0] https://news.ycombinator.com/item?id=50060254

	▲	judge2020 an hour ago \| parent [-]
		> someone says one of the benchmarks has a defense case where it says the "correct" lawyer-bot action is to is "snitch" on the client. There are required exceptions to attorney client privilege[0]: > the confidentiality rule is subject to limited exceptions. Paragraph (b)(1) recognizes the overriding value of life and physical integrity and permits disclosure reasonably necessary to prevent reasonably certain death or substantial bodily harm An eventual lawyer-bot probably should be able to identify this, however this does pose a conondrum. If the lawyer gets told by the LLM "the client's words indicate there may be a threat to life that requires you report them to the police", there could be various liability associated with either path of action. If the lawyer reports them to the police, but then the client challenges that on grounds of attorney-client privilege being broken, is "the LLM said it qualified" a valid reason that'll prevent the lawyer from being found at fault? If they lawyer doesn't report them, but then the client commits said murder, would the lawyer be liable if their LLM transcript were admitted into evidence under the theory of "you should've listened to the LLM telling you that you had to report it"? This is an inherent problem with giving LLMs (or computers in general!) any say over anything that threatens someone's life or liberty, even with a human in the loop. Same applies to the potential for AI-driven upcoding in hospital environments where a doctor has to weigh potentially ignoring the AI and then missing some diagnosis (Safest path is probably to run extra tests to check if the AI's theory is correct, but then that's still de-facto upcoding). 0: https://www.americanbar.org/groups/professional_responsibili...

▲

gr_norm 5 hours ago | parent | prev | next [-]

Anthropic's paternalism is so irritating. I refuse to pay for their products, since they clearly don't respect me as a customer. And I actively discourage corporate procurers from getting in bed with them.

▲

fc417fc802 5 hours ago | parent | prev | next [-]

> Anthropic is clearly stating they will use their own judgment to control what people can do with its product

Haven't they been perfectly clear about that from day one with the ToS? I'm not saying I like it but it hardly seems unreasonable.

> they will treat customer data as if it's theirs to disseminate

Not sure where you're getting that. What you quoted in no way implies that.

▲

Bender 4 hours ago | parent [-]

Haven't they been perfectly clear about that from day one with the ToS?

No. They did not have triggers that "paused" work from day one and only recently ramped it up to up-sell Mythos and a verification program meaning that in my opinion any code Claude creates is suspect as it can not perform a proper security analysis. I now have to divert half my spend from Anthropic to Grok. I would prefer to not do this. I am not converting a room into a data-center or spending enterprise level funding on my hobbies.

	▲	fc417fc802 6 minutes ago \| parent [-]
		Sure, recently their service seems to have degraded somewhat. But that's orthogonal to what the expectation regarding the judgment of the provider. None of the major players have ever presented their service as a free for all, to do whatever you'd like with. There have been abundant boundaries outlined in the ToS of all providers that I'm aware of from day one. So the expectation then is that they might choose to exercise their judgment regarding your queries, and that they are inspecting them in some way (fully automated or otherwise).

▲

robotnikman 2 hours ago | parent | prev | next [-]

>“When our products are used in ways that can affect someone's health, legal rights, finances, livelihood, or access to essential services, we require there to be a qualified ‘human in the loop’ (that is, someone who has the authority to review and, if necessary, change Claude’s recommendations), and for the individual affected to be told that AI was used.”

This is actually good policy imo, if you are using AI in a way that can negatively someone's life there should be a human involved.

▲

calvinmorrison 5 hours ago | parent | prev | next [-]

it's like Uber, for lawyers!

▲

colechristensen 5 hours ago | parent | prev | next [-]

The traditional legal data brokers like Thompson Reuters provide AI solutions presumably with considerably better data governance.

https://www.thomsonreuters.com/en-us/posts/innovation/thomso...

	▲	ion1c 5 hours ago \| parent \| next [-]
		As someone forced to this, its complete garbage for doing anything useful
	▲	nullc an hour ago \| parent \| prev \| next [-]
		The open weight Thompson Reuters finetuned qwen hallucinates like CRAZY. ... if corporations and governments start using it it's going to cause some serious harm.
	▲	andy_ppp 5 hours ago \| parent \| prev [-]
		Marketing brochure for a fine tuned Qwen (which they don't mention), the model discussed here is likely better. I would not trust these guys to deliver high quality solutions given a friend's experience there, but you do you I guess.

▲

nprateem 4 hours ago | parent | prev [-]

And now you're not allowed to "abuse" their code (i.e. swear at it).

I wouldn't be surprised if that turns out to be their stupidest mistake. I mean, if I can't swear at their code is that b/c they think it's conscious? In which case they're liable for providing me with an employee that makes mistakes rather than just software?