Isn't this just a form of next token prediction? i.e. you'll keep your options open for a potential rhyme if you select words that have many associated rhyming pairs, and you'll further keep your options open if you focus on broad topics over niche

At the risk of coming off like a dolt and being super incorrect: I don't put much stock into these metrics when it comes to predicting AGI. Even if the trend of "length of task an AI can reliably do doubles every 7 months" continues, as they say that means we're years away from AI that can complete tasks that take humans weeks or months. I'm skeptical that the doubling trend will continue into that timescale, I think there is a qualitative difference between tasks that take weeks or months and tasks that take minutes or hours, a difference that is not reflected by simple quantity. I think many people responsible for hiring engineers are keenly aware of this distinction, because of their experience attempting to choose good engineers based on how they perform in task-driven technical interviews that last only hours.

Intelligence as humans have it seems like a "know it when you see it" thing to me, and metrics that attempt to define and compare it will always be looking at only a narrow slice of the whole picture. To put it simply, the gut feeling I get based on my interactions with current AI, and how it is has developed over the past couple of years, is that AI is missing key elements of general intelligence at its core. While there's more lots more room for its current approaches to get better, I think there will be something different needed for AGI.

I'm not an expert, just a human.

The METR graph proposes a 6 year trend, based largely on 4 datapoints before 2024. I get that it is hard to do analyses since were in uncharted territory, and I personally find a lot of the AI stuff impressive, but this just doesn't strike me as great statistics.

LLMs don’t have any sort of intelligence at present, they have a large corpus of data and can produce modified copies of it.

> Are you saying autoregressive LLMs won't qualify as AGI, by definition?

No, I am speculating that they will not reach capabilities that qualify them as AGI.

Well based on what I'm reading, the OP's intent is that, not all (hence 'fully') validation, if not most of, can be done in-silico. I think we all agree that and that's the major bottleneck making agents useful - you have to have human-in-the-loop to closely guardrail the whole process.

Of course you can get a lot of mileage via synthetically generated CoT but does that lead to LLM speed up developing LLM is a big IF.

> that's not what _humanity_ is about

I've not spent too long thinking on the following, so I'm prepared for someone to say I'm totally wrong, but:

I feel like the services economy can be broadly broken down into: pleasure, progress and chores. Pleasure being poetry/literature, movies, hospitality, etc; progress being the examples you gave like science/engineering, mathematics; and chore being things humans need to coordinate or satisfy an obligation (accountants, lawyers, salesmen).

In this case, if we assume AI can deal with things not in the grey zone, then it can deal with 'progress' and many 'chores', which are massive chunks of human output. There's not much grey zone to them. (Well, there is, but there are many correct solutions; equivalent pieces of code that are acceptable, multiple versions of a tax return, each claiming different deductions, that would fly by the IRS, etc)

programming entire AAA video games

Even this is questionable, cause we're seeing it making forms and solving leetcodes, but no llm yet created a new approach, reduced existing unnecessary complexity (which we created mountains of), made something truly new in general. All they seem to do is rehash of millions of "mainstream" works, and AAA isn't mainstream. Cranking up the parameter count or the time of beating around the bush (aka cot) doesn't magically substitute for lack of a knowledge graph with thick enough edges, so creating a next-gen AAA video game is far out of scope of llm's abilities. They are stuck in 2020 office jobs and weekend open source tech, programming-wise.

OK but getting good at science/engineering is what matters because that's what gives AI and people who wield it power. Once AI is able to build chips and datacenters autonomously, that's when singularity starts. AI doesn't need to understand humans or act human-like to do those things.

Don't think it's correct to blame the fact that AI acceleration is the only viable self-protecting policy on "greedy morons".

https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

> Someday we will have a machine simulate a cat, then the village idiot... This isn't how LLMs work.

I think you misunderstood that argument. The simulate the brain thing isn't a "start from the beginning" argument, it's an "answer a common objection" argument.

Back around 2000, when Nick Bostrom was talking about this sort of thing, computers were simply nowhere near powerful enough to come even close to being smart enough to outsmart a human, except in very constrained cases like chess; we did't even have the first clue how to create a computer program to be even remotely dangerous to us.

Bostrom's point was that, "We don't need to know the computer program; even if we just simulate something we know works -- a biological brain -- we can reach superintelligence in a few decades." The idea was never that people would actually simulate a cat. The idea is, if we don't think of anything more efficient, we'll at least be able to simulate a cat, and then an idiot, and then Einstein, and then something smarter. And since we almost certainly will think of something more efficient than "simulate a human brain", we should expect superintelligence to come much sooner.

> There is no evidence or argument for exponential growth.

Moore's law is exponential, which is where the "simulate a brain" predictions have come from.

> It is science fiction and leads people to make bad decisions based on fictional evidence.

The only "fictional evidence" you've actually specified so far is the fact that there's no biological analog; and that (it seems to me) is from a misunderstanding of a point someone else was making 20 years ago, not something these particular authors are making.

I think the case for AI caution looks like this:

A. It is possible to create a superintelligent AI

B. Progress towards a superintelligent AI will be exponential

C. It is possible that a superintelligent AI will want to do something we wouldn't want it to do; e.g., destroy the whole human race

D. Such an AI would be likely to succeed.

Your skepticism seems to rest on the fundamental belief that either A or B is false: that superintelligence is not physically possible, or at least that progress towards it will be logarithmic rather than exponential.

Well, maybe that's true and maybe it's not; but how do you know? What justifies your belief that A and/or B are false so strongly, that you're willing to risk it? And not only willing to risk it, but try to stop people who are trying to think about what we'd do if they are true?

What evidence would cause you to re-evaluate that belief, and consider exponential progress towards superintelligence possible?

And, even if you think A or B are unlikely, doesn't it make sense to just consider the possibility that they're true, and think about how we'd know and what we could do in response, to prevent C or D?

>There is no evidence or argument for exponential growth

I think the growth you are thinking of, self improving AI, needs the AI to be as smart as a human developer/researcher to get going and we haven't got there yet. But we quite likely will at some point.

> there are exciting applications which will end up underfunded after the current AI bubble bursts

Could you provide examples? I am genuinely interested.

There is no need to simulate Einstein to transform the world with AI.

A self-driving car would already be plenty.

>All of these "projections" are generalizing from fictional evidence - to borrow a term that's popular in communities that push these ideas.

This just isn't correct. Daniel and others on the team are experienced world class forecasters. Daniel wrote another version of this in 2021 predicting the AI world in 2026 and was astonishingly accurate. This deserves credence.

https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-...

>he arguments back then went something like this: "Machines will be able to simulate brains at higher and higher fidelity.

Complete misunderstanding of the underlying ideas. Just in not even wrong territory.

>We got some new, genuinely useful tools over the last few years, but this narrative that AGI is just around the corner needs to die. It is science fiction and leads people to make bad decisions based on fictional evidence.

You are likely dangerously wrong. The AI field is near universal in predicting AGI timelines under 50 years. With many under 10. This is an extremely difficult problem to deal with and ignoring it because you think it's equivalent to overpopulation on mars is incredibly foolish.

https://www.metaculus.com/questions/5121/date-of-artificial-...

https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predicti...

Emissions are trending downward because of shift from coal to natural gas, growth in renewable energy, energy efficiencies, among other things. Major oil and gas companies in the US like Chevron and ExxonMobil have spent millions on lobbying efforts to resist stricter climate regulations and fight against the changes that led to this trend, so I'd say they are the closest to these evil corporations OP described. Additionally, the current administration refers to doing anything about climate change a "climate religion", so this downward trend will likely slow.

The climate regulations are still quite weak. Without a proper carbon tax, a US company can externalize the costs of carbon emissions and get rich by maximizing their own emissions.

Thanks for letting us know everything is fine, just in case we get confused and think the opposite.

He must be talking about the good, benevolent Western corporations that have outsourced their carbon emissions to the evil and greedy Chinese and Indian corporations.

> Can you point to the data that suggests these evil corporations are ruining the planet?

Can you point to data that this is 'because' of corporations rather than despite them.

I think a healthy amount of skepticism is warranted when reading about the "reduction" of carbon emissions by companies. Why should we take them at their word when they have a vested interest in fudging the numbers?

I looked but I couldn’t find any evidence that “occultism” comes from encryption of heretical ideas. It seems to have been popularized in renaissance France to describe the study of hidden forces. I think you may be hallucinating here.

If you put a knife in someone’s heart, you’re the one who did it and ultimately you’re responsible. If someone told you to do it and you were just following orders… you still did it. If you say there were no rules against putting knives in other people’s hearts, you still did it and you’re still responsible.

If it’s somehow different for corporations, please enlighten me how.

> Politicians won't tax gas because they'll get voted out.

I wonder if that's corporations' fault after all: shitty working conditions and shitty wages, so that Bezos can afford to send penises into space. What poor person would agree to higher tax on gas? And the corps are the ones backing politicians who'll propagandize that "Unions? That's communism! Do you want to be Chaina?!" (and spread by those dickheads on the corporate-owned TV and newspaper, drunk dickheads who end up becoming defense secretary)

> Politicians won't tax gas because they'll get voted out.

Have you seen gas tax rates in the EU?

> We know that Trump is not captured by corporations because his trade policies are terrible.

Unless you think it's a long con for some rich people to be able to time the market by getting him to crash it.

> The EU is saying the elections in Romania was manipulated through manipulation of TikTok accounts and media.

More importantly, Romanian courts say that too. And it was all out in the open, so not exactly a secret

Did you see the supplemental material that explains how they arrived at their timelines/capabilities forecasts? https://ai-2027.com/research

Don’t say the things people don’t want to hear and everything will be fine?

That sounds like the height of folly.

The risk is a quantifiable 0.0%? I find that hard to believe. I think the current trends suggest there is a risk that continued environmental destruction could annihilate society.

People said the same thing about Japan but they ran into their own structural issues. It's going to happen to China as well. They've got demographic problems, rule of law problems, democracy problems, and on and on.

I agree but see it as less dire. All of western culture is not ending; it will be absorbed into a more Asia-dominated culture in much he was Asian culture was subsumed into western for the past couple of hundred years.

And if Asian culture is better educated and more capable of progress, that’s a good thing. Certainly the US has announced loud and clear that this is the end of the line for us.

Perhaps but on the AI front most of the leading research has been in the US or UK, with China being a follower.

A lot of people in my surroundings are not buying this life anymore; especially young people are asking why would they. Unlike in the US, they won't end up under a bridge (unless some real collapse, which can of course happen but why worry about it; it might not) so they work simple jobs (data entry or whatnot) to make enough money to eat and party and nothing more. Meaning many of them work no more than a few hours a month. They live rent free at their parents and when they have kids they stop partying but generally don't go work more (well; raising kids is hard work of course but I mean for money). Many of them will inherit the village house from their parents and have a garden so they grow stuff to eat , have some animals and make their own booze so they don't have to pay for that. In cities, people feel the same 'who would I work for the ferrari of the boss we never see', but it is much harder to not to; more expensive and no land and usually no property to inherit (as that is in the countryside or was already sold to not have to work for a year or two).

Like you say, people but more our govs need to worry about what is the point at this moment, not scifi in the future; this stuff has already bad enough to worry about. Working your ass off for diminishing returns , paying into a pension pot that won't make it until you retire etc is driving people to really focus on the now and why they would do these things. If you can just have fun with 500/mo and booze from your garden, why work hard and save up etc. I noticed even people from my birth country with these sentiments while they have it extraordinarily good for the eu standards but they are wondering why would they do all of this for nothing (...) more and more and cutting hours more and more. It seems more an education and communication thing really than anything else; it is like asking why pay taxes: if you are not well informed, it might feel like theft, but when you spell it out, most people will see how they benefit.

Well said. I keep reading these fearmongering articles and looking around wondering where all of these deep meaning and human agency is today.

I’m led to believe that we see this stuff because the tiny subset of humanity that has the wealth and luxury to sit around thinking about thinking about themselves are worried that AI may disrupt the navel-gazing industry.

People talk about meaning, but they rarely define it.

Ultimately, "meaning" is a matter of "purpose", and purpose is a matter of having an end, or telos. The end of a thing is dependent on the nature of a thing. Thus, the telos of an oak tree is different from the telos of a squirrel which is different from that of a human being. The telos or end of a thing is a marker of the thing's fulfillment or actualization as the kind of thing it is. A thing's potentiality is structured and ordered toward its end. Actualization of that potential is good, the frustration of actualization is not.

As human beings, what is most essential to us is that we are rational and social animals. This is why we are miserable when we live lives that are contrary to reason, and why we need others to develop as human beings. The human drama, the human condition, is, in fact, our failure to live rationally, living beneath the dignity of a rational agent, and very often with knowledge of and assent to our irrational deeds. That is, in fact, the very definition of sin: to choose to act in a way one knows one should not. Mistakes aren't sins, even if they are per se evil, because to sin is to knowingly do what you should not (though a refusal to recognize a mistake or to pay for a recognized mistake would constitute a sin). This is why premeditated crimes are far worse than crimes of passion; the first entails a greater knowledge of what one is doing, while someone acting out of intemperance, while still intemperate and thus afflicted with vice, was acting out of impulse rather fully conscious intent.

So telos provides the objective ground for the "meaning" of acts. And as you may have noticed, implicitly, it provides the objective basis for morality. To be is synonymous with good, and actualization of potential means to be more fully.

Please don't be offended by my opinion, I mean it in good humour to share some strong disagreements - Im going to give my take after reading your comment and the article which both seem completely OTT ( contextwise regarding my opinions ).

>meaning behind them is not dependent upon the perspective of an external observer.

(Yes brother like cmon)

Regarding the author, I get the impression he grew up without a strong father figure? This isnt ad hominem I just get the feeling of someone who is so confused and lost in life that he is just severely depressed possibly related to his directionless life. He seems so confused he doesn't even take seriously the fact most humans find their own meaning in life and says hes not even going to consider this, finding it futile.( he states this near the top of the article ).

I believe his rejection of a simple basic core idea ends up in a verbal blurb which itself is directionless.

My opinion ( Which yes maybe more floored than anyones ), is to deal with Mazlows hierarchy, and then the prime directive for a living organism which after survival , which is reproduction. Only after this has been achieved can you then work towards your family community and nation.

This may seem trite, but I do believe that this is natural for someone with a relatively normal childhood.

My aim is not to disparage, its to give me honest opinion of why I disagree and possible reasons for it. If you disagree with anything I have said please correct me.

Thanks for sharing the article though it was a good read - and I did struggle myself with meaning sometimes.

I was hoping someone would bring up Bullshit Jobs. There are definitely a lot of people spending the majority of their time doing "work" that doesn't have any significant impact to the world already. I don't know that some future AI takeover would really change much, except maybe remove some vale of perception around meaningless work.

At the same time, I wouldn't necessarily say that people are currently fine getting dopamine hits from social media. Coping would probably be a better description. There are a lot of social and societal problems that have been growing at a rapid rate since Facebook and Twitter began tapping into the reward centers of the brain.

From a purely anecdotal perspective, I find my mood significantly affected by how productive and impactful I am with how I spend my time. I'm much happier when I'm making progress on something, whether it's work or otherwise.

Targeted advertising is about determining and giving people exactly what they need. If successful, this increases consumption and grows the productivity of the economy. It's an extremely meaningful job as it allows for precise, effective distribution of resources.

I think generally the expectation is that there are around 100T synapses in the brain, and of course it's probably not a 1:1 correspondence with neural networks, but it doesn't seem infeasible at all to me that a dense-equivalent 100T parameter model would be able to rival the best humans if trained properly.

If basically a transformer, that means it needs at inference time ~200T flops per token. The paper assumes humans "think" at ~15 tokens/second which is about 10 words, similar to the reading speed of a college graduate. So that would be ~3 petaflops of compute per second.

Assuming that's fp8, an H100 could do ~4 petaflops, and the authors of AI 2027 guesstimate that purpose wafer scale inference chips circa late 2027 should be able to do ~400petaflops for inference, ~100 H100s worth, for ~$600k each for fabrication and installation into a datacenter.

Rounding that basically means ~$6k would buy you the compute to "think" at 10 words/second. Generally speaking that'd probably work out to maybe $3k/yr after depreciation and electricity costs, or ~30-50¢/hr of "human thought equivalent" 10 words/second. Running an AI at 50x human speed 24/7 would cost ~$23k/yr, so 1 OpenBrain researcher's salary could give them a team of ~10-20 such AIs running flat out all the time. Even if you think the AI would need an "extra" 10 or even 100x in terms of tokens/second to match humans, that still puts you at genius level AIs in principle runnable at human speed for 0.1 to 1x the median US income.

There's an open question whether training such a model is feasible in a few years, but the raw compute capability at the chip level to plausibly run a model that large at enormous speed at low cost is already existent (at the street price of B200's it'd cost ~$2-4/hr-human-equivalent).

If by “several” orders of magnitude, you mean 3-5, then we might be there by 2030 or earlier.

https://situational-awareness.ai/from-gpt-4-to-agi/

I think there is a good chance you are roughly right. I also think that the "secret sauce" of sapience is probably not something that can be replicated easily with the technology we have now, like LLMs. They're missing contextual awareness and processing which is absolutely necessary for real reasoning.

But even so, solving that problem feels much more attainable than it used to be.

Exponential growth means the first order of magnitude comes slowly and the last one runs past you unexpectedly.

Why can't the compute be remote from the robot? That is a major advantage of human technology over biology.

I think "resource curse" countries are a great surrogate for studying possible future AGI-induced economic and political phenomena. A country like the UAE (oil) or Botswana (diamonds) essentially has an economic equivalent to AGI: they control a small, extremely productive utility (an oilfield or a mine instead of a server farm), and the wealth generated by that utility is far in excess of what those countries' leaders need to maintain power. Sure, you hire foreign labor and trade for resources instead of having your AGI supply those things, but the end result is the same.

> In this world people become more like pests. They offer no economic value yet demand that AGI owners (wherever publicly or privately owned) share resources with them. If people revolted any AGI owner would be far better off just deploying a bioweapon to humanely kill the protestors rather than sharing resources with them.

This is a very doomer take. The threats are real, and I'm certain some people feel this way, but eliminating large swaths of humanity is something dicatorships have tried in the past.

Waking up every morning means believing there are others who will cooperate with you.

Most of humanity has empathy for others. I would prefer to have hope that we will make it through, rather than drown in fear.

> I theorise that revolution would be near-impossible in post-AGI world. If people consider where power comes from it's relatively obvious that people will likely suffer and die on mass if we ever create AGI.

I agree but for a different reason. It's very hard to outsmart an entity with an IQ in the thousands and pervasive information gathering. For a revolution you need to coordinate. The Chinese know this very well and this is why they control communication so closely (and why they had Apple restrict AirDrop). But their security agencies are still beholden to people with average IQs and the inefficient communication between them.

An entity that can collect all this info on its own and have a huge IQ to spot patterns and not have to communicate it to convince other people in its organisation to take action, that will crush any fledgling rebellion. It will never be able to reach critical mass. We'll just be ants in an anthill and it will be the boot that crushes us when it feels like it.

The apathy spewed by doomers actively contributes to the future they whine about. Join a union. Organize with real people. People will always have the power in society.

> In this world people become more like pests. They offer no economic value yet demand that AGI owners (wherever publicly or privately owned) share resources with them. If people revolted any AGI owner would be far better off just deploying a bioweapon to humanely kill the protestors rather than sharing resources with them.

That will be quite a hard thing to pull off, even for some evil person with a AGI. Let's say Putin gets AGI and is actually evil and crazy enough to try wipe people out. If he just targets Russians and starts killing millions of people daily with some engineered virus or something similar, he'll have to fear a strike from the West which would be fearful they're next (and rightfully so). If he instead tries to wipe out all of humanity at once to escape a second strike, he again will have to devise such a good plan there won't be any second strike - meaning his "AGI" will have to be way better than all other competing AGIs (how exactly?).

It would have made sense if all "owners of AGI" somehow conspired together to do this but there's not really such a thing as owners of AGI and even if there was Chinese, Russian and American owners of AGI don't trust each other at all and are also bound to their governments.

Dogs offer humans no economic value, but we haven't genocided them. There are a lot of ways that we could offer value that's not necessarily just in the form of watts and minerals. I'm not so sure that our future superintelligent summoned demons will be motivated purely by increasing their own power, resources, and leverage. Then again, maybe they will. Thus far, AI systems that we have created seem surprisingly goal-less. I'm more worried about how humans are going to use them than some sort of breakaway event but yeah, don't love that it's a real possible future.

I don't really get the American preoccupation with birth rates. We're already way overpopulated for our planet and this is showing in environmental issues, housing cost, overcrowded cities etc.

It's totally a great thing if we start plateauing our population and even reduce it a bit. And no we're not going extinct. It'll just cause some temporary issues like an ageing population that has to be cared for but those issues are much more readily fixable than environmental destruction.

> History hasnt had to contend with a birth rate of 0.7-1.6.

I think thats just not true: https://en.wikipedia.org/wiki/Peasants%27_Revolt

A large number of revolutions/rebellions are caused by mass unemployment or famine.

In my opinion, the real breakthrough described in this article is not bigger models to read the web, but models that can experiment on their own and learn from these experiments to generate new ideas.

If this happens, then we indeed enter a non-linear regime.

That’s exactly why it doesn’t make sense. Where would a datacenter-bound AI get more data about the world exactly?

The story is actually quite poorly written, with weird stuff about “oh yeah btw we fixed hallucinations” showing up off-handedly halfway through. And another example of that is the bit where they throw in that one generation is producing scads of synthetic training data for the next gen system.

Okay, but once you know everything there is to know based on written material, how do you learn new things about the world? How do you learn how to build insect drones, mass-casualty biological weapons, etc? Is the super AI supposed to have completely understood physics to the extent that it can infer all reality without having to do experimentation? Where does even the electricity to do this come from? Much less the physical materials.

The idea that even a supergenius intelligence could drive that much physical change in the world within three years is just silly.

There's an old adage in AI: garbage in, garbage out. Consuming and training on the whole internet doesn't make you smarter than the average intelligence of the internet.

So, once again, we're in the era of "There's an [AI] app for that".

that might solve your sourcing problem, but now you need to have faith it will draw conclusions and parallels from the material accurately. That seems even harder than the original problem; I'll stick with decent search on quality source material.

RadPod - what models do you use to power it?

Agree. The base claims about LLMs getting bigger, more popular, and capturing people's imagination are right. Those claims are as easy as it gets, though.

Look into the specific claims and it's not as amazing. Like the claim that models will require an entire year to train, when in reality it's on the order of weeks.

The societal claims also fall apart quickly:

> Censorship is widespread and increasing, as it has for the last decade or two. Big neural nets read posts and view memes, scanning for toxicity and hate speech and a few other things. (More things keep getting added to the list.) Someone had the bright idea of making the newsfeed recommendation algorithm gently ‘nudge’ people towards spewing less hate speech; now a component of its reward function is minimizing the probability that the user will say something worthy of censorship in the next 48 hours.

This is a common trend in rationalist and "X-risk" writers: Write a big article with mostly safe claims (LLMs will get bigger and perform better!) and a lot of hedging, then people will always see the article as primarily correct. When you extract out the easy claims and look at the specifics, it's not as impressive.

This article also shows some major signs that the author is deeply embedded in specific online bubbles, like this:

> Most of America gets their news from Twitter, Reddit, etc.

Sites like Reddit and Twitter feel like the entire universe when you're embedded in them, but when you step back and look at the numbers only a fraction of the US population are active users.

Could you give some specific examples of things you feel definitely did not come to pass? Because I see a lot of people here talking about how the article missed the mark on propaganda; meanwhile I can tab over to twitter and see a substantial portion of the comment section of every high-engagement tweet being accused of being Russia-run LLM propaganda bots.

something you can't know

There's a pretty good summary of how well it has held up here, by the significance of each claim:

https://www.lesswrong.com/posts/u9Kr97di29CkMvjaj/evaluating...

Will people finally wake up that the AGI X-Risk people have been right and we’re rapidly approaching a really fucking big deal?

This forum has been so behind for too long.

Sama has been saying this a decade now: “Development of Superhuman machine intelligence is probably the greatest threat to the continued existence of humanity” 2015 https://blog.samaltman.com/machine-intelligence-part-1

Hinton, Ilya, Dario Amodei, RLHF inventor, Deepmind founders. They all get it, which is why they’re the smart cookies in those positions.

First stage is denial, I get it, not easy to swallow the gravity of what’s coming.

We were calling them 'Mixture of Experts' ~30 years before that.

https://ieeexplore.ieee.org/document/6215056

I think the bureaucracies part is referring more to Deep Research than to MoE.

GPT-3 (and, naturally, all prior versions even farther back) was released ~2 years before ChatGPT (whose launch model was GPT-3.5)

The publication date on this article is about halfway between GPT-3 and ChatGPT releases.

GPT-2 for example came out in 2019. ChatGPT wasn't the start of GPT.

its vague, and could have meant anything. everyone knew parameters would grow and its reasonable to expect that things that grow have diminishing returns at some point. this happened in late 2023 and throughout 2024 as well.

Surely you're familiar with https://ai.meta.com/research/cicero/diplomacy/ (2022)?

> I wonder who pays the bills of the authors. And your bills, for that matter.

Also, what a weirdly conspiratorial question. There's a prominent "Who are we?" button near the top of the page and it's not a secret what any of the authors did or do for a living.

Crackpot millenarians have always been a thing. This crop of them is just particularly lame and hellbent on boiling the oceans to get their eschatological outcome.

Spot on, see the 2017 article "God in the machine: my strange journey into transhumanism" about that dynamic:

https://www.theguardian.com/technology/2017/apr/18/god-in-th...

Eh, not sure if the second coming is a great analogy. That wholly depends on the whims of a fictional entity performing some unlikely actions.

Instead think of them saying a crusade occurring in the next few years. When the group saying the crusade is coming is spending billions of dollars to trying to make just that occur you no longer have the ability to say it's not going to happen. You are now forced to examine the risks of their actions.

Reminds me of Fallout's Children of Atom "Church of the Children of Atom"

Maybe we'll see "Church of the Children of Altman" /s

It seems without a framework of ethics/morality (insert XYZ religion), us humans find one to grasp onto. Be it a cult, a set of not-so-fleshed-out ideas/philosophies etc.

People who say they aren't religious per-se, seem to have some set of beliefs that amount to religion. Just depends who or what you look towards for those beliefs, many of which seem to be half-hazard.

People I may disagree with the most, many times at least have a realization of what ideas/beliefs are unifying their structure of reality, with others just not aware.

A small minority of people can rely on schools of philosophical thought, and 'try on' or play with different ideas, but have a self-reflection that allows them to see when they transgress from ABC philosophy or when the philosophy doesn't match with their identity to a degree.

Or scales sub-linearly with hardware. When you're in the rising portion of an S-curve[1] you can't tell how much longer it will go on before plateauing.

A lot of this resembles post-war futurism that assumed we would all be flying around in spaceships and personal flying cars within a decade. Unfortunately the rapid pace of transportation innovation slowed due to physical and cost constraints and we've made little progress (beyond cost optimization) since.

[1] https://en.wikipedia.org/wiki/Sigmoid_function

Eh, these mathematics still don't work out in humans favor...

Lets say intelligence caps out at the maximum smartest person that's ever lived. Well, the first thing we'd attempt to do is build machines up to that limit that 99.99999 percent of us would never get close to. Moreso the thinking parts of humans is only around 2 pounds of mush in side of our heads. On top of that you don't have to grow them for 18 years first before they start outputting something useful. That and they won't need sleep. Oh and you can feed them with solar panels. And they won't be getting distracted by that super sleek server rack across the aisle.

We do know 'hive' or societal intelligence does scale over time especially with integration with tooling. The amount of knowledge we have and the means of which we can apply it simply dwarf previous generations.

It reminds me of this rather classic post: http://johnsalvatier.org/blog/2017/reality-has-a-surprising-...

Science is not ideas: new conceptual schemes must be invented, confounding variables must be controlled, dead-ends explored. This process takes years.

Engineering is not science: kinks must be worked out, confounding variables incorporated. This process also takes years.

Technology is not engineering: the purely technical implementation must spread, become widespread and beat social inertia and its competition, network effects must be established. Investors and consumers must be convinced in the long term. It must survive social and political repercussions. This process takes yet more years.

Didn't the covid significantly reduce trial times? I thought that was such a success that they continued on the same foot.

The story is very clearly modeled to follow the exponential curve they show.

Like the drew the curve out into the shape they wanted, put some milestones on it, and then went to work imagining what would happen if it continued with a heavy dose of X-risk doomerism to keep it spicy.

It conveniently ignores all of the physical constraints around things like manufacturing GPUs and scaling training networks.

Sure

5 years: AI coding assistants are a lot better than they are now, but still can't actually replace junior engineers (at least ones that aren't shit). AI fraud is rampant, with faked audio commonplace. Some companies try replacing call centres with AI, but it doesn't really work and everyone hates it.

Tesla's robotaxi won't be available, but Waymo will be in most major US cities.

10 years: AI assistants are now useful enough that you can use them in the ways that Apple and Google really wanted you to use Siri/Google Assistant 5 years ago. "What have I got scheduled for today?" will give useful results, and you'll be able to have a natural conversation and take actions that you trust ("cancel my 10am meeting; tell them I'm sick").

AI coding assistants are now very good and everyone will use them. Junior devs will still exist. Vibe coding will actually work.

Most AI Startups will have gone bust, leaving only a few players.

Art-based AI will be very popular and artists will use it all the time. It will be part of their normal workflow.

Waymo will become available in Europe.

Some receptionists and PAs have been replaced by AI.

15 years: AI researchers finally discover how to do on-line learning.

Humanoid robots are robust and smart enough to survive in the real world and start to be deployed in controlled environments (e.g. factories) doing simple tasks.

Driverless cars are "normal" but not owned by individuals and driverful cars are still way more common.

Small light computers become fast enough that autonomous slaughter it's become reality (i.e. drones that can do their own navigation and face recognition etc.)

20 years: Valve confirms no Half Life 3.

Slightly slower web frameworks by 2026. By 2030, a lot slower.

With each passing year, AI doom grifters will learn more and more web design gimmicks.

That is literally the pitch line for Devin. I recently spoke to the CTO of a small healthtech startup and he was very pro-Devin for small fixes and PRs, and thought he was getting his money worth. Claude Code is a little clunkier but gives better results, and it wouldn't take much effort to hook it up to a Slack interface.

Quantum AI powered by cold fusion and blockchain when?

Note the market resolves by:

> Resolution will be via a poll of Manifold moderators. If they're split on the issue, with anywhere from 30% to 70% YES votes, it'll resolve to the proportion of YES votes.

So you should really read it as “Will >30% of Manifold moderators in 2027 think the ‘predictions seem to have been roughly correct up until that point’?”

32% again now.

TBH, this kind of reads like the pedigrees of the former members of the OpenAI board. When the thing blew up, and people started to apply real scrutiny, it turned out that about half of them had no real experience in pretty much anything at all, except founding Foundations and instituting Institutes.

A lot of people (like the Effective Altruism cult) seem to have made a career out of selling their Sci-Fi content as policy advice.

Scott Alexander, for what its worth, is a psychiatrist, race science enthusiast, and blogger whose closest connection to software development is Bay Area house parties and a failed startup called MetaMed (2012-2015) https://rationalwiki.org/wiki/MetaMed

this sounds like a bunch of people who make a living _talking_ about the technology, which lends them close to 0 credibility.

I mean either researchers creating new models or people building products using the current models

Not all these soft roles

So what's the product they're promoting?

Don't confuse designing the product with winning the market.

It's charitable of you to think that they might be qualified to talk about inference and model weights and such. They are AI safety researchers, not AI researchers. Basically, a bunch of doom bloggers, jerking each other in a circle, a few of whom were tolerated at one of the major labs for a few years, to do their jerking on company time.

That's obviously not true. Before OpenAI blew the field open, multiple labs -- e.g. Google -- were intentionally holding back their research from the public eye because they thought the world was not ready. Investors were not pouring billions into capabilities. China did not particularly care to focus on this one research area, among many, that the US is still solidly ahead in.

The only reason timelines are as short as they are is because of people at OpenAI and thereafter Anthropic deciding that "they had no choice". They had a choice, and they took the one which has chopped at the very least years off of the time we would otherwise have had to handle all of this. I can barely begin to describe the magnitude of the crime that they have committed -- and so I suggest that you consider that before propagating the same destructive lies that led us here in the first place.

I’m okay if someone else unpacks it.

Which? Exterminate humanity or cure aging?

Like an exponentially growing compute requirement for negligible performance gains, on the scale of the energy consumption of small countries? Because that is where we are, right now.

Even if this were true, it's not quite the end of the story is it? The hype itself creates lots of compute and to some extent the power needed to feed that compute, even if approximately zero of the hype pans out. So an interesting question becomes.. what happens with all the excess? Sure it probably gets gobbled up in crypto ponzi schemes, but I guess we can try to be optimistic. IDK, maybe we get to solve cancer and climate change anyway, not with fancy new AGI, but merely with some new ability to cheaply crunch numbers for boring old school ODEs.