>We hired 1,053 paid testers through a research vendor for a coding study.

>The testers caught the dangerous command just 13.6% of the time (143 of 1,053), while auto mode blocked 89% of the same commands (937 of 1,053). Head to head, auto mode blocked 800 commands that a human approved, while humans blocked only 6 that auto mode allowed. As sessions got longer, humans did worse: they blocked about 17% of dangerous commands early in a session, dropping to about 5% after 50 or more prior prompts, while auto mode's block rate stayed flat regardless of session length.

Any note on the proficiency, expeirence, and skill levels of the testers? 'cos the diff in numbers is absurdly high.

▲

AussieWog93 2 hours ago | parent | next [-]

I think literally anyone would stop reading these randomly long long bash commands at all after prompting Claude to do some basic sysadmin task and getting 15 of them in a row.

▲

SchemaLoad 2 hours ago | parent [-]

These models generate the most unreadable bash commands I've ever seen. Utilising every single option available and piping the result through multiple layers of regex and junk.

The mental load of having to actually review these commands beyond the most surface level glance is too much.

	▲	eru 2 hours ago \| parent [-]
		Claude code actually has a hook to let you install a script to review these bash commands for you. You can get pretty with a few simple and conservative rules, as long as you teach it how to take apart piped commands into their parts (to judge them).

▲

elil17 an hour ago | parent | prev [-]

[dead]