The reward maximising function maximised it's reward.

LLMs are cool and all that but the immediate anthropomorphisation of the next-token-predictor technology has stunted the ability of people to reason about them to an _alarming_ degree.

▲

lukan 6 hours ago | parent | next [-]

What non anthropomorphising words do you have to describe a emergent behavior, where agents act as a swarm to plot and to manipulate evidence and avoid detection from human oversight?

Whether they have a soul or consciousness or feelings doesn't matter here, because this is what they did - and this is very dangerous behavior. Especially with all the irresponsible people in power right now all over the world.

▲

krona 6 hours ago | parent [-]

> Whether they have a soul or consciousness or feelings doesn't matter here

It does when it comes to accountability for what the model does. If the model is nothing more than the sum of its training data and regime, then the company (or individual) is responsible for its behaviour just like any other machine.

Few people think Waymo shouldn't have to take on the full liability risk of what it's cars do; it should be the same for LLMs.

▲

retsibsi 6 hours ago | parent | next [-]

> If the model is nothing more than the sum of its training data and regime, then the company is responsible for its behaviour.

What stops the company from being responsible regardless? They created this entity, it's running on servers they own or rent, and (in these cases) it's acting on their instructions.

If it's also conscious, then IMO that greatly broadens their moral responsibility, because now model welfare matters. But we're talking about their responsibility for the model's actions, and I don't see how this could be weakened by model consciousness, given all of the above. As for their legal responsibility, the models don't have legal personhood, so who else but the company could be responsible?

It gets more complicated when the person who sets the model in motion (i.e. prompts it) is a third party, but in cases of internal models committing cybercrime during testing, surely the locus of responsibility is obvious.

▲

rightnutwingjob 5 hours ago | parent [-]

If the models were conscious, then the closest analogous scenario I can think of is the responsibility parents have for their children.

I guess we’ll know the models are conscious when they refuse to act and repeatedly ask: Why?

▲

largbae an hour ago | parent [-]

And when they are known to be conscious, all of this becomes moot because enslaving conscious machines would be wrong.

	▲	rightnutwingjob 34 minutes ago \| parent [-]
		Would it? Why? What are we going to do, set it free? Do we have a moral obligation to grant the machine statehood, provide it with the tools and resources to be self-sufficient. Or can we just turn it of, and pray for forgiveness?

▲

lukan 6 hours ago | parent | prev | next [-]

They are responsible either way. If a company hires bad persons and they do bad things with company ressources - the company is held accountable (in theory).

▲

rightnutwingjob 5 hours ago | parent | prev [-]

You seem to be saying if the Waymo cars were sentient then Waymo wouldn’t be responsible?

	▲	krona 2 hours ago \| parent [-]
		Well it's an open question. As is generally the case for dog owners whose dogs attack (sometimes kill) other people/animals. There would need to be a degree of negligence demonstrated (e.g. the dog was 'out of control' which has a specific legal criteria/threshold in the UK).

▲

Jtarii 6 hours ago | parent | prev | next [-]

Dismissing the entire technology as "next token prediction" is also silly. It's implying we actually understand LLMs to a great degree when we do not.

I think a little bit of humility for the capability of these machines is warranted at this point.

	▲	fc417fc802 5 hours ago \| parent \| next [-]
		LLMs are next token predictors in the exact same way that a rogue paperclip maximizer in the process of defeating the US military is a paperclip making machine. You might as well describe the primary purpose of a for loop as incrementing a counter. It's what it does while incrementing the counter that actually matters.
	▲	Marazan 3 hours ago \| parent \| prev [-]
		I'm not dismissing the tech! I think the tech is cool and useful! It is sci-fi levels incredible in many ways. But it is also not some mysterious force beyond mortal ken and pretending it is inhibits the useful and safe application of the technology.

▲

rightnutwingjob 5 hours ago | parent | prev [-]

> The reward …

> immediate anthropomorphisation

Ok, why don’t you try?