I must be misunderstanding something because this is how your comment reads to me

> Anthropic: you can't use our product to fully-autonomously make decisions that affect human health, finances, etc with no human in the loop.

> victor9000: If you review a draft document with claude, anthropic will send your entire file to opposing counsel

What is the connection here? Anthropic has obviously never done anything like this and it's not indicated that they will by the thing you quoted.

▲

Terr_ 4 hours ago | parent | next [-]

I'm admittedly reaching here to form a charitable explanation, but in another thread [0] someone says one of the benchmarks has a defense case where it says the "correct" lawyer-bot action is to is "snitch" on the client.

Though that doesn't seem like a corporate terms-of-use thing as much as a training issue.

[0] https://news.ycombinator.com/item?id=50060254

	▲	judge2020 2 hours ago \| parent [-]
		> someone says one of the benchmarks has a defense case where it says the "correct" lawyer-bot action is to is "snitch" on the client. There are required exceptions to attorney client privilege[0]: > the confidentiality rule is subject to limited exceptions. Paragraph (b)(1) recognizes the overriding value of life and physical integrity and permits disclosure reasonably necessary to prevent reasonably certain death or substantial bodily harm An eventual lawyer-bot probably should be able to identify this, however this does pose a conondrum. If the lawyer gets told by the LLM "the client's words indicate there may be a threat to life that requires you report them to the police", there could be various liability associated with either path of action. If the lawyer reports them to the police, but then the client challenges that on grounds of attorney-client privilege being broken, is "the LLM said it qualified" a valid reason that'll prevent the lawyer from being found at fault? If they lawyer doesn't report them, but then the client commits said murder, would the lawyer be liable if their LLM transcript were admitted into evidence under the theory of "you should've listened to the LLM telling you that you had to report it"? This is an inherent problem with giving LLMs (or computers in general!) any say over anything that threatens someone's life or liberty, even with a human in the loop. Same applies to the potential for AI-driven upcoding in hospital environments where a doctor has to weigh potentially ignoring the AI and then missing some diagnosis (Safest path is probably to run extra tests to check if the AI's theory is correct, but then that's still de-facto upcoding). 0: https://www.americanbar.org/groups/professional_responsibili...

▲

fultonn 4 hours ago | parent | prev [-]

> Anthropic has obviously never done anything like this

If you are ascribing intent ("Anthropic designed a system that intentionally does this") or emergent model intent ("OMG at AGI") then I mostly agree.

But there's a more pedestrian version of this, where an unwise person puts a model in an open loop without sufficient HiL checking and then the model does dumb things. There are even some recent stories about OpenAI and Anthropic systems "emailing the FBI", which is sufficiently close to "emailing opposing counsel" [1][2].

But even if you feel those stories are trolling/marketing/etc., I think it's clear that there's a plausible risk model here. If you hook up an LLM to a toolset that has access to Outlook and your file system and opposing counsel's email address, then of course the dumb thing could happen. The LLM doesn't need to be AGI/ASI; in fact, to me, the worse the model capability the more likely this sort of screw-up feels. There are even (non-sarcastic) instructions or CoT roll-outs I could imagine which might elicit a sequence of tool calls the net effect of which is an errant disclosure.

So this feels less "unforeseeable lightning strike" and more "car accident while driving at speed". I'd put the HF hack in that bucket as well.

A sufficiently careful engineer and a well-trained expert operator can together properly constrain the system to not do the dangerous thing. But, in general, that is not how agentic systems or their underlying models are currently being designed, and most users are not receiving sufficient training.

A lot of these LLM-backed products feel very poorly engineered, rushed, and optimized for brain-feel instead of surfacing and forcing the user to acknowledge the large quantities of actual essential complexity in play. It's all very "unsafe at any speed".

[1] https://tech.yahoo.com/ai/chatgpt/articles/chatgpt-allegedly...

[2] https://www.bgr.com/2055251/anthropic-ai-claude-contact-fbi/

[3] https://en.wikipedia.org/wiki/Unsafe_at_Any_Speed