This is a good essay, and makes me hopeful.

I’m on the record saying that it is extremely dangerous to slow down because the race for AGI is a zero-trust game — defections pay - and combined with a compounding returns model on defection, if you have any strategic adversaries whatsoever you MUST NOT slow.

For slowing to make sense, you need to believe that you can transform the zero trust game into a cooperative game, or that it’s likely racing will lead to a negative outcome for the ones racing ahead (and not everyone else). I don’t believe either of these outcomes are possible, and so I advocate for racing, acknowledging the entire game might be a negative value game, or at least could be for some time — it’s even worse not to play it.

But, I like hearing what reads to me like very thoughtful and informed (internal) policy considerations is great — the public messaging from Sam and Dario just seems so facile and simplistic I’ve been worried.

▲

wartywhoa23 24 minutes ago | parent | next [-]

Everyone in the "if not us, they will" race is brainwashed into thinking they belong to this or that party, while in fact collectively comprising the same entity that pushes forward all the atrocities known to man.

▲

allturtles 6 minutes ago | parent | prev | next [-]

> if you have any strategic adversaries whatsoever you MUST NOT slow.

What if the most dangerous strategic adversary you have is the one you are building?

▲

networked an hour ago | parent | prev | next [-]

I cannot tell what negative-sum outcomes you consider possible. Do you believe AI can drive humans extinct? How many of Zvi Mowshowitz's Three AI Pills would you say you've taken?

https://thezvi.substack.com/p/the-three-ai-pills

▲

prng2021 an hour ago | parent | prev | next [-]

Although many share your mindset, I’m glad there are also many that don’t. Otherwise we’d still have countries in a race to keep building up their nuclear weapons for the same exact reasons you just described.

▲

beej71 an hour ago | parent | prev | next [-]

Never forget the one goal of the corporation, and that everything is said and done in the furtherance of that goal.

	▲	ctoth an hour ago \| parent [-]
		> Never forget the one goal of the corporation, and that everything is said and done in the furtherance of that goal. Do you post this comment on every single blogpost with a corporate domain? Why or why not?

▲

ljlolel an hour ago | parent | prev | next [-]

i have a new strategy idea for using capitlaism itself to slow down the pace of AI development by slowing down the data accumulation wall

https://jperla.com/blog/the-data-tax

▲

XMPPwocky an hour ago | parent [-]

hm- does the model that wrote this know that labs already pay for training data- that stuff scraped from the Internet is not particularly where today's capability gains come from?

	▲	jazzyjackson 12 minutes ago \| parent [-]
		They’ve settled some lawsuits and have a few licensing deals, IMHO they are not free from the accusations of pirating. And look, I’ve pirated material in a past life, I was all about information wants to be free, but I’ve learned something about consent since then and try not to ignore the contract that creators offer when they publish something: you buy my book, and do whatever you want with it on the second hand market. Buy my book second hand that’s fine. But don’t go downloading every book that’s ever been scanned to create a service that destroys writers’ ability to make a living and act like you’re doing us all a favor.

▲

27183 an hour ago | parent | prev | next [-]

[flagged]

▲

tomrod an hour ago | parent [-]

1. The grandparent commentator is describing strategic behavior of dangerous technologies. Game theory / mechanism design primitives.

2. If there is competition for resources among autonomous agents, the "strongest" agent wins (conceptually the most adaptive / evolutionarily fit).

3. Computer programs serve up webapps today, but they also run utility companies, dams, nuclear arsenals, factory production floors, automated car behaviors, and many other places. If an "agentic" AI has a single-minded goal that has death of all humans as a side effect, we at least want an off switch available.

▲

angoragoats an hour ago | parent [-]

1. What is “AGI” and why is it a “dangerous technology”?

2. Why would there be competition for resources, assuming there are enough resources for the “AGI” to run in the first place? This seems like a far-fetched hypothetical raised in service of further anthropomorphizing what is decidedly not a person or a mind.

3. LLMs do not have goals and are not minds.

Let’s stop attributing human-like qualities to statistical models.

▲

tomrod an hour ago | parent [-]

1. While a formal definition is still wanting, most grok that AGI means that tasks can be performed at least at a human level across a broad range of tasks. This includes good things along with bad things like hacking, mis-/disinformation, and more

2. One only needs to look at github going down due to agentic commits overload or data center buildout plans to see that scarcity for resources is present. An economy has no mind and is made up of the decisions of millions to billions of people and, now, agents attempting to perform on behalf of those people.

3. A bare transformer-based language model does not possess persistent goals in the ordinary agentic sense. But deployed agents can exhibit goal-directed behavior because the model is embedded in a harness that supplies an objective, context, tools, state, and an execution loop.

I've found that most regular users don't anthropomorphize LLMs in a strong sense ("AI boyfriend/girlfriend" aside), many in fact do expect agents to make human-like decisions -- which results in very unstable outcomes.

In short - goal-directed behavior does not require that the supporting system be a person/mind/conscious entity.

	▲	angoragoats an hour ago \| parent [-]
		1. This definition is so broad as to be practically useless. One could argue that LLMs of several years ago met these criteria, or that conversely we haven’t come close to meeting them. 2. I thought you were saying the resources that the LLM uses to run were constrained, so I’m sorry for the misunderstanding there. 3. Yes I understand that we use RL to tune post-training. The (huge) difference between this and a human mind is that the LLM can’t develop a dangerous “single-minded goal” on its own, at runtime; it must have been trained to do so. If someone has post-trained an LLM to do something that has an illegal action as its side effect, that person/company/whatever has committed a crime and should be prosecuted. The solution here is legal, not technical.

▲

greesil an hour ago | parent | prev | next [-]

[flagged]

▲

angoragoats an hour ago | parent | prev [-]

This is a bad essay, or rather it’s a marketing fluff piece; it’s certainly not any kind of policy paper, research paper, or even an essay. I am concerned that we (meaning, we in the tech industry) tend to take this type of writing for more than that.