> "The singularity is a specific belief that we will achieve AGI

That is one version of it, but not the only one. "John von Neumann is the first person known to have discussed a "singularity" in technological progress.[14][15] Stanislaw Ulam reported in 1958 that an earlier discussion with von Neumann "centered on the accelerating progress of technology and changes in human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue""[1]. A time when people before it, would be unable to predict what came after it because it was so different. (And which I argue in another comment[2] is not a specific cutoff time, but a trend over history of the future being increasingly hard to predict over shorter and shorter timeframes).

Apart from AGI, or Von Neuman accelerationism, I also understand it as augmenting human intelligence: "once we become cyborgs and enhance our abilities, nobody can predict what comes next"; or artificial 'life' - "if we make self-replicating nano-machines (that can have Darwinian natural selection?), all bets about the future are off"; or "once we can simulate human brains in a machine, even if we can't understand how they work, we can run tons of them at high speeds".

> and usher in a golden age. (Either for itself or us. That part's a bit under contention, from my understanding.)

Arguably, we have built weakly-superhuman entities, in the form of companies. Collectively they can solve problems that individual humans can't, live longer than humans, deploy and exploit more resources over larger areas and longer timelines than humans, and have shown a tendency to burn through workers and ruin the environment that keeps us alive even while supposedly guided by human intelligence. I don't have very much hope that a non-human AGI would be more aligned with our interests than companies made up of us are.

[1] https://en.wikipedia.org/wiki/Technological_singularity

[2] https://news.ycombinator.com/item?id=46935546

Just think of this as risk management.

If AGI doesn't happen, then good. You get to keep working and playing and generally screwing off in the way that humans have for generations.

On the other hand if AGI happens, especially any time soon, you are exceptionally fucked along with me. The world changes very rapidly and there is no getting off Mr Bones wild ride.

>Why is previous progress a guaranteed indicator of future progress?

In this case, because nature already did it. We're not just inventing and testing something whole cloth. And we know there are still massive efficiencies to be gained.

For me the Concorde is an example of how people look at stuff incorrectly. In the past we had to send people places very quickly to do things. This was very expensive and inefficient. I don't need to get on a plane to have an effect just about anywhere else in the world now. The internet and digital mediums give me a presence at other locations that is very close to being there. We didn't need planes that fly at the speed of sound, we needed strings that communicate at the speed of light.

how do you take over the world if you have access to 1000 normal people? if AGI is by the original definition (long forgotten by now) of surpassing MEDIAN human at almost all tasks. How the rebranding of ASI into AGI happened without anyone noticing is kind of insane

That’s completely wrong. There was barely any progress in previous millennia. There was even economic Nobel prize for showing why!

Lol, wut?

The world can produce more things cheaper and faster than ever and this is an economic decline? I think you may have missed the other 6 billion people on the planet getting massive improvements in their quality of life.

The bell curve of IQ and being stupid probably don't have much to do with each other.

Think of stupidity as the consequences of interacting with ones environment with negative outcomes. If you have a simple environment with few negative outcomes, then even someone with a 80 IQ may not be considered stupid. But if your environment rapidly grows more complex and the amount of thinking you have to do for positive outcomes increases then even someone with a 110 IQ may find themselves quickly in trouble.

> I look at the trajectory of LLMs, and the shape I see is one of diminishing returns

It seems even more true if you look at OpenAI funding thru 2022 initial public release to how spending has exponentially increased to deliver improvements since. We’re now talking upwards of $600B/yr of spending on LLM based AI infrastructure across the industry in 2026.