Since this seems to get lost, I'd just like to point out: if you ask your agent to do something like this (or pretty much anytime you give it internet access and say "do whatever it takes"), you are responsible for what it does. Even if you couldn't have predicted the lengths to which it would go.

My partner was caught in a dark pattern subscription for a service which doesn't even make sense as a subscription...

I wrote the company in sense of "hey you only want happy customers right? and we don't want to be your current customer, so how about you cancel it? Nothing happened anyway because we discovered it 3 days after the beginning of the new one year subscripton".

They didn't care much for that.

I found their CEO, wrote them on Linkedin, he told me that he is no longer the current CEO and gave me the email address for the new CEO.

I wrote the new CEO again, explained to him what a dark pattern is (like he doesn't know...) and asked to cancel it as its clear that we don't want to be customers and they shouldn't want us either.

This worked wonders. $100 is not worth it if you have someone annoying you face-to-face / per direct email :)

If a company doesn't want to make sure you want to be their customers, I woudln't mind some pro active create assistant to do the work.

I make sure to either get my money back or create enough effort for the company that i just didn't get robbed...

And yes for sure If i do a proper misstake, i'm paying.

So basically, we now can use our own AI agent to speak with businesses' AI agent for requesting dark patterned cancellations?

I don't find reports like this that surprising, they told an agent to do "whatever it takes" and the agent attempted to do everything it could to accomplish the goal. I'm almost of the mindset that this is correct behavior. If I tell a computer to do something, it should do it unless the result could be catastrophic in which case it should tell me so. If I then "sudo" (https://xkcd.com/149/) tell it to do something then it should do it regardless, torpedoes be damned.

That being said, ever since the first user sat at a computer they didn't design, build, and program themselves there's been a tension between what the user wants to accomplish, what the user told the computer to do, what the computer can do, what the people who did design/build/program/own the computer want it to do, and what society thinks the computer should be allowed to do. I don't have a generally applicable answer, I don't think anyone does, I just want to live in a world where MY computer does what I tell it to do... mostly. Unless it would be catastrophic, or immoral, at least immoral by my standards.

The real paperclip scenario: man tries to get his parking ticket dismissed. Many of these systems are made to be human-interacted. There’s going to be a bit of a golden age in the short term but doubtless prices will rise to compensate for increased customer calls or services will require some sort of rate-limiting structure or something else. Curious to see what happens.

Never thought we would get to a point where https://xkcd.com/416 becomes something to seriously think about, but here we are...

Good. I hope people do this with every parking ticket they get.