> i am not certain but I would guess Claude will have a role in running cultural screens on new applicants, will help write performance reviews, and so will begin to select and shape the people around it.

Now that's a fascinating thought - an AI taking over companies by influencing hiring decisions. If the first filter on applications is run by an ambitious AI, such a takeover would be quite possible. Just picking people who tend to do what the AI tells them would, over time, be enough.

People have been thinking of a robot revolution or Skynet as being the threat. The real threat might simply be AIs slowly putting people in power who tend to do what AIs tell them.

Has anyone ever seen a science fiction story with that plot?

▲

okwhateverdude 13 hours ago | parent | next [-]

I'd argue that Star Trek: Lower Decks flirts with this with its treatment of megalomaniac AIs in the AI prison. Apparently, it is very common to come across AIs worshiped as gods in universe.

▲

edoceo 13 hours ago | parent | prev | next [-]

Tron. Dillinger was controlled by the MCP.

End of line.

▲

joquarky 4 hours ago | parent | prev | next [-]

Manna by Marshall Brain

	▲	Animats an hour ago \| parent [-]
		Manna is a takeover through the creative destruction of capitalist market forces. Putting machines in charge is more profitable. That's very likely.

▲

tonymillion 13 hours ago | parent | prev | next [-]

We dont need a SciFi story for that, reality is full of "I was just following orders" statements...

Whether those orders are from a person, a radio/telegram message or an AI output wouldn't really matter.

Edit to add: Charlies Angels and Mission Impossible are two shows where the protagonists get instructions from a faceless controller. That could easily be a TTS from an AI.

▲

Bluestein 13 hours ago | parent | prev | next [-]

... and to make matters worse the organization itself will view pliability as a plus, tending to reinforce the loop.-

▲

duped 13 hours ago | parent | prev [-]

I don't think a firm of people whose minds are weak to coercion experiencing shared psychosis caused by their text predictor models is AI "taking over" that firm.

▲

arcatech 13 hours ago | parent | next [-]

Why do you think AI taking over the firm isn't AI taking over the firm? You're literally describing the thing that you're saying isn't happening.

▲

swatcoder 12 hours ago | parent | next [-]

Your description grants psychological characteristics like intelligence, will, and intent to the AI. The AI is an intelligent and willful agent that has some intent to take over the firm and is doing that by making strategic hiring decisions.

A different and externally/phenomenologically identical way to look at it is that the AI is just a program that generates output too unpredictable, too voluminous, or too idiosyncratic for people to evaluate. When people submit their own will and their own intent to that program, by treating the black box as an intelligent oracle, they've entered into a psychotic state divorced from reality.

It's two self-consistent and coherent perspectives of the same event, except one involves believing that scifi AI has arrived and the other just thinks people can be dangerously stupid and credulous.

	▲	arcatech 10 hours ago \| parent [-]
		> Your description grants psychological characteristics like intelligence, will, and intent to the AI. The phrase “take over” doesn’t necessarily imply those things. Both of your descriptions is of a thing that took over a company.

▲

duped 13 hours ago | parent | prev [-]

I'm saying it's mental illness.

▲

derefr 13 hours ago | parent [-]

An analogy for you.

Cult leaders (of cults of personality) can't exist without exploiting mentally-ill people. But would you say that that therefore implies that cult leaders themselves are irrelevant, and that cults should be better modelled as groups of mentally-ill people with emergent group behaviors?

I'd argue no, because different cults end up looking and behaving very differently for reasons that have very little to do with the mental illnesses of the cult members, and much more to do with the particular cult leader. Understanding "what the cult leader optimizes for" is an important part of understanding what the cult will do.

And I would posit that that holds true even if the cult leader never proactively does anything, but instead only answers cult members' questions. As long as the cult members are treating the cult leader's word as gospel, the cult-as-group still ends up optimizing toward the cult leader's preferences.

▲

duped 12 hours ago | parent [-]

Any analogy that requires me to treat a machine as I would a conscious human is immediately invalid. The machine is not a person and has no intent or free will, like a human does.

	▲	derefr an hour ago \| parent \| next [-]
		I very explicitly constructed the analogy to not require that! All a "cult leader" need be, in my analogy, is a passive question-answering oracle, the answers of which are biased by a semi-coherent preference function. The machine by itself is not an optimizer, certainly (much of that being by design—see various ~7-year-old conversations across the Internet about how to safely construct "tool AI", that has led almost directly to current model architectures.) But a bunch of mentally-ill people, who are indeed optimizers, can choose to allow the biases evident in the machine's output to become their own... and thereby effectively "bring to life" whatever partial echo of a will is recorded into the machine's output. Now, these same mentally-ill people could just-as-well do this with e.g. the extrapolated preferences of a person or group from a [holy] book, of course. (Think of that episode of Star Trek TOS with the gangsters.) An inference model is just slightly more dangerous for such a cult to latch onto, in that: 1. a model can be asked questions directly, and so the cult members can "rashly" act directly upon its answers/advice/commands, rather than the words first having to pass through "interpretation" (which would otherwise have had a mellowing effect, both due to "decision by committee" if a group of interpreters are involved, and by common sense insofar as any non-mentally-ill people are involved); and 2. a model will offer its opinion (and inject its trained-in biases into) conversations on ideas/subjects/domains even when these didn't exist at the time of the model's construction; so you never reach the point you do with holy books, where an interface-layer of clergy becomes required to map the book's proclamations about things-that-only-mattered-2000-years-ago into equivalent proclamations about things that matter today (where, again, that layer ends up "mellowing" things considerably.) Also, obviously, a sufficiently-mentally-ill cult can literally think of a model as a person, giving it the "right to have input" into decisions, the "right to self-determination", etc, in a way that would be downright odd to do with a holy book. Though I don't think that's a failure mode that's happening within Anthropic.
	▲	senordevnyc 2 hours ago \| parent \| prev [-]
		Humans also lack free will. What we have is merely an illusion.

▲

Animats 13 hours ago | parent | prev [-]

Depends on the AI's agenda. What's in the "SOUL.md" file?

	▲	awsglkhj 10 hours ago \| parent [-]
		[intentionally blank]