Compute will always be the bottleneck even if this were true.

As a statement of fact divorced from context, this is of course true, but it's worth putting it in context of what small-medium scale models have been achieving recently. Many of the most recent releases from Chinese labs are almost on par with trillion parameter models from less than a year ago (edit: despite being small enough to usably run on prosumer hardware). It seems clear parameter efficiency can still be improved dramatically.

In which case, maybe we don't need as much compute as we might expect. I hesitate to say "to reach a singularity" because it's kind of hard to define how that works out. Even intelligence probably hits some scaling limits eventually (e.g. speed of light related restrictions on how far it can scale, or how quickly it can expand).

▲

Fordec 6 hours ago | parent | prev | next [-]

If humans can figure out to optimize to circumvent bottlenecks, I have no doubt each new bottleneck will also get routed around, just now automated.

▲

mrbungie 6 hours ago | parent | next [-]

We are not in an everything-has-an-API world yet, and it'll for sure take some time to get there.

	▲	Fordec 5 hours ago \| parent [-]
		For sure. Anyone who thinks that we're in the end state of what progress can be made simply lacks imagination. This is all going to keep changing and iterating for the rest of our natural lives. The only constant is change.

▲

HenrikPontoppid 6 hours ago | parent | prev | next [-]

Yes. In other words: the singularity. I'll only believe it when I see it though.

▲

Fordec 6 hours ago | parent [-]

I'm coming around to not liking the term singularity, it implies an endpoint or finish line rather than something that just keeps continuing and evolving.

▲

piloto_ciego an hour ago | parent | next [-]

I've done a lot of thinking about this since I first used ChatGPT to write some BS jinja2 templates hours after I first play with it. I said to my friend then (who scoffed at me) that "man, this is incredible, I think we're in the foothills of the singularity! This is insane! Sure it's stupid now but I can't believe this is even possible!" That friend is so black pilled and bitter he now hates AI. Whatever, I can't fix that, but the current progress is astounding.

But thinking about the geometry of this problem helps understand why people aren't adjusting well to this. While we're walking on the curve, we look at the rate of change of the curve and say, "well, yeah, of course, dy/dx is 5 at this point and was 1 at the point a few years ago, because the curve is getting steeper" but we're always going to feel this way as things rip off into the stratosphere because dy/dx(e^x) = e^x.

From standing on the curve the curve is notably seeper, but the steepness totally makes sense to you. It's only when you look back 10 years or so that you think "wait a second, holy hell, I couldn't have imagined this!"

The first time I really (I mean really) thought about the singularity and AI was in roughly 2014. I mean, yea, I'd thought about things before that, but yeah, before the Humans need not apply video, I'd never actually given it much thought. I think back to myself 10 - 15 years or so ago, when I was just starting to tackle real programming projects, and was just starting to get decent at writing code, there is no way on earth that I would have imagined that a little over a decade later, Navier-Stokes would be solved by a computer program and the vast majority of my work would be playing sooth sayer to increasingly complicated piles of linear algebra.

▲

supern0va 5 hours ago | parent | prev [-]

It doesn't imply that. The singularity is just the inflection point.

	▲	jsLavaGoat 4 hours ago \| parent \| next [-]
		Singularity and inflection point are incompatible mathematically and in the plain sense, it really is focused on a particular moment and always has been, hence the term. And it's definitely supposed to imply some kind of historical discontinuity not a change in convexity.
	▲	Fordec 4 hours ago \| parent \| prev [-]
		Which assumes the presence of an inflection point that keeps inflecting rather than revert to an S-curve. The growth model is not borne out yet to declare what shape it is.

▲

dakolli 6 hours ago | parent | prev [-]

How do you automate the mines to get the raw materials to make the compute from, and build additional fabs that take a almost a decade to stand up. You're actually delusional.

▲

Fordec 6 hours ago | parent | next [-]

Hello good sir from the 1700s pre-industrial revolution who doesn't think that mines and factories can be automated.

	▲	dakolli 2 hours ago \| parent [-]
		The factories that supply the equipment, maintain the equipment, the energy inputs, the financials of those mines are not automated. People on hackernews are actually some of the dumbest people on the internet. This place is worse than lesswrong.

▲

7373737373 4 hours ago | parent | prev | next [-]

Some mines are already heavily automated: https://youtube.com/watch?v=_Z9w-mUoUsY

https://youtube.com/watch?v=SRuht0QIprs

▲

a2ff6eeb0 4 hours ago | parent | prev [-]

https://en.wikipedia.org/wiki/Lights_out_(manufacturing)

Scroll down to the existing examples section.

▲

monster_truck 5 hours ago | parent | prev | next [-]

Based on the leaps in local inference speed in the past month, which have been absurd, I'm p confident we're going to whiplash from compute constrained to storage constrained.

Bit apples to oranges, but it reminds me of all the fiber we installed in the late 90s, certain that per-strand capacity increases were years or decades out, only to get massively rugged

▲

Fordec 5 hours ago | parent [-]

I expect the investments into AI driven mathematic discoveries that underpin compression efficiency will be a key investment area. Particularly at the data center scale rather than per device or per file level.

	▲	monster_truck an hour ago \| parent \| next [-]
		It's not going to be enough. The naive approach of a project I've been working on was pushing >10gbps over the local network, after a ton of work I got it back down under 1... and now it's processing so much more shit that I'm almost past 5 again! It compresses at >3:1 but the latency hit isn't suitable. I get the impression the only reason there is renewed interest in photonics is because DCs are simply out of room (and power) to rack more servers and switches.
	▲	xtracto 5 hours ago \| parent \| prev [-]
		pi-fs will solve all our data compression problems.

▲

Miner49er 7 hours ago | parent | prev | next [-]

Eventually recursive self-improvement includes reducing bottlenecks.

▲

glenstein 6 hours ago | parent | next [-]

Which is to say, scalable and open-ended capability of ramping up physical infrastructure.

I don't know that that's achievable yet. Though the era of increasingly advanced and automated robotics seems to be around the corner which could create a cycle, vicious or virtuous depending on how you feel about it.

▲

10xDev 7 hours ago | parent | prev [-]

Eventually the bottleneck might be people themselves.

	▲	ccozan 4 hours ago \| parent [-]
		Improbably, the real bottleneck is energy.

▲

lijok 4 hours ago | parent | prev [-]

And the goalposts move again