The ad hominems are tiresome. You think the people who are explicitly trying to replace all humans are any less weird and problematic?

The rationalists long predicted that misaligned AI would break out of training environments, this has happened. They believe (among other things) that this will continue to happen, and that creating powerful AI that is aligned and controllable is not currently possible.

You should draw a line in the sand and decide what evidence you'd need to see to convince you that their perspective is worth considering.

You don't have to agree with all of their beliefs and predictions (I don't), but you should evaluate their arguments on merit, not on a characterization of some weird subset of people who happened to read a convincing blog post.

▲

lukewarm707 a day ago | parent [-]

i read AI 2027 at the time and thought it was nonsensical. however i was not aware of the conflict of interest running through the philosophy itself, the safety institutes "independently" monitoring ai, and the former public benefit orgs like openai and anthropic creating ai (now de facto fairly evil for profit companies).

it explains so much now.

it would explain why redwood and METR were chosen and apparently willing to immediately write a whitewash report for openai blaming the agents.

it explains why anthropic is infantilising its users, anthropomorphising its product when there is no scientific consensus of llm consciousness, or accepting of outlandish and unreal claims about human extinction when they are lacking in evidence.

who knows, it may also explain and fund the recent media blitz by jacob coxon, which has led to the recent virality of ai safety.

▲

jmoggr a day ago | parent [-]

sure, yeah, I've seen how the sausage is made on some of this stuff, and its not pretty.

However this only claims that they are not good at creating effective and trustworthy orgs, it does not address their claim that it is not possible to crate powerful, aligned, and controllable AI. Both things be true.

	▲	lukewarm707 a day ago \| parent [-]
		for what it is worth, i find current frontier ai impossibly stupid. you ask what is the date, it doesn't know. if you set a system prompt to repsond in 5 sentences after 100k context, it can't, the context rot just doesn't permit it. tool calling is not intelligence, it doesn't fix the underlying issue. it is uncontrolled in the sense that a rock is uncontrolled if you throw it at someone's head. it isn't good enough for anything important. you would not trust it to run an alarm in a power station. of course it is not general ai. ask an llm to drive a car. you need to see in fractions of a second ideally. the llm might brake after 20 seconds. i think llms are doomed here. we could come up with a different architecture but then we would be back in the land of fantasy. we are stupid but hopefully not that stupid. we are dealing with something computer based, luckily we can survive without, with some thinking about it. i would start there instead of with the AI part.