Remix.run Logo
api 2 hours ago

In my opinion at least, any attempt to intentionally engineer addiction is morally wrong and should be at the very least civilly if not criminally actionable (if it can be proven).

It's mind control. That phrase is often associated with woo woo conspiracy theory, but I don't think that's hyperbolic. If you are engineering addiction, what you're doing is attempting to run a black hat security attack against the human central nervous system to violate a person's will and autonomy. That is the definition of mind control. Properly framed I think it's a form of assault, especially when deployed with deliberate concealment of motive or against children, the elderly, the mentally ill, anyone not strong enough to resist it.

The medium doesn't matter: addictive drugs, slot machines, web sites, social media apps, games. In that list I actually think explicitly labeled gambling stuff like Las Vegas is the least evil since at least in that case it's kind of up front and overt about what it is. I mean, it's a casino. WTF do you think a casino does? Hard drugs are kind of the same at this point. The thing that makes this case so awful is that people genuinely didn't know. Nobody including myself understood just how effective addiction engineering in a digital medium could be, and these companies were secretive about just how hard-core they were getting about studying and engineering this. They hired psychologists. There's stories of studies putting people in FMRI machines. It was a deliberate R&D program with the (secretive) stated goal of addiction.

The next thing important to understand is that businesses are structurally incapable of self-policing something with a profit incentive. I didn't say unwilling. I said incapable. Businesses, like all "corporate" entities (non-profits, governments, etc.) operate with a certain set of constraints, social customs, and rules, and for a business one of those is that they must earn revenue and/or attract investment to survive. If addiction results in revenue, then any business that refuses to engineer addiction will fail.

This is also why regulation of things like child labor, forced labor, safety requirements, etc. are required. If a safe workplace reduces profits than any company implementing a safe workplace loses to those who do not, etc.

BTW this is something a lot of "online Marxist" type anti-capitalists don't understand when they vilify business leaders. A business that doesn't do the profitable thing stops being a business. It's outcompeted by those who do the thing. The only way to stop these things is to universally police them, which levels the playing field and allows businesses who don't do the bad thing to keep operating. It's an instant race to the bottom because a business cannot not race.

So yeah... lots of words but I think this is an important issue and it's good that it's happening. I really hope an example gets made here, or that this at least raises awareness.

I think it's critically important that we start dealing with digital addiction engineering as an issue before the AI funding bubble pops. When that happens all the AI companies will have to look for other sources of revenue. Addiction engineering online became a thing after the dot.com bubble popped for exactly that reason. We already have AI driven addictionware in the form of digital partner apps and such, but if the AI investment gold rush stalls and we have a mass push for revenue among AI companies then with no policing here this will get really really bad.

A final moral to the story, I think, is that science f'ing works.

Like I said, it shocked me to see how much addiction a web site or a mobile app could create. Learning that it was deliberately engineered makes it less surprising.

Science works. When you apply the scientific method to an engineering goal, progress gets made, often dramatic progress. If the laws of physics and the nature of the system you're studying will allow it, you can get so far that the result can be shocking. Look at the hydrogen bomb. Look at aviation, space flight, AI, modern medicine. There's good and bad in there of course cause people do both good and bad things.

Which is why saying "we are going to apply science to make our product more addictive" should be criminal conspiracy. "We are going to apply science to do X" where X is some form of assault should be criminal. "We didn't think it would work that well" is a poor defense because I think at this point we should have learned that science works.

Gabrys1 2 hours ago | parent [-]

By your definition, almost any ad is mind control and should be forbidden. At least any effective ad. And I don't disagree at that conclusion. There's a difference between telling people you have a product and making people want to have your product

api an hour ago | parent | next [-]

I kept saying "policing" here because unfortunately this is the absolute worst kind of evil for a society to have to deal with: the kind that is not absolutely black and white.

Murder is a tidy kind of evil. If you murder someone that's bad and you go to jail, and murder is pretty decisive. The victim is no longer alive. The black and white nature kills all the nuance around "why did you do it?" It doesn't matter why you killed them. You killed them so you committed murder.

This, however, has shades of grey, and that sucks. If I write a novel and try to make it entertaining and make it a page turner by making chapters punchy and short, is that "engineering addiction?" What if I design a game and try to make it fun and end up making it addictive?

You end up with stuff like the famous and infamous statement about porn: "I can't define it but I know it when I see it." That sucks because to police something you need definitions and lines.

Moral problems like this suck because they require deeply nuanced thinking and a continuous social discourse around where to draw the line, and we hate that. People like absolutes. So one problem you tend to get with moral problems like this is bifurcation of the discourse into moralist crusaders with a zero-tolerance policy versus libertines who think it's not a problem. Both of these extremes lead to abuse and immoral actions. The crusaders end up being worse tyrants than the problem, while the libertines leave the sheep to be devoured by wolves.

But that's what this is. It's a complicated grey problem that mandates a continuous ongoing social discourse and a never-ending series of court cases. But it exists, so we have to deal with it, so we have to do that.

It means we have to think. Ugh.

To answer your question directly: yeah, I think a big part of the ad industry falls here, and that's good. What remains is ads that are informative, playful, or entertaining, and those are the kids of ads we don't tend to hate so much. Some of them can even be legitimate art.

The best way to start with problems like this is: start by drawing a line just to the right of the stuff that is unambiguously terrible.

If you create a scientific R&D program to study the human brain and human psychology with the express purpose of engineering addiction and then fail to communicate this fact to the people you're trying to addict, that's IMO pretty damn unambiguously evil. When the government did this it was called MKULTRA and there were congressional hearings over it and people shredded documents and hid under rocks in fear of what they were doing coming to light, which kind of shows that the people doing it knew it was bad. So let's start by banning that kind of shit even if it's happening in the private sector with private money.

Then once you've drawn that line, re-assess. If the problem is still terrible and pervasive, consider drawing another line a little further to the right. Repeat.

scotty79 an hour ago | parent | prev [-]

> almost any ad [...] should be forbidden

I wholeheartedly agree with this. There should be one central catalog of products and services and all commercial information should be available to "pull" and never get "pushed". To be able to sell any product or service you should be legally required to register it in the catalog with detailed description (including manuals, also technical manuals) and up to date price.