Remix.run Logo
▲ Legend2440 4 hours ago

>I mean something beyond current AI is needed.

Every time someone says 'current AI is not enough, we need new architectures', it turns out that current AI was enough and all we really needed was more scaling.

▲rhdunn 21 minutes ago | parent | next [-]

Do we know that that is the case?

We know the architectures of the open weight models as they include architecture diagrams, support in projects like llama.cpp, etc.

Companies like Anthropic and OpenAI don't release information about their models (aside from general things like context window size, whether they support reasoning, etc.). A lot of what we know is speculation. From press releases Anthropic have stated that they are getting Claude to assist with designing new iterations of the models, but no specific details.

The open weight models and other arXiv papers/research are iterating on and experimenting with the base/core transformer architecture:

- RoPE/NoPE/etc. for context window scaling;

- dense vs mixture of experts models;

- Mamba-based State Space Models (falcon models, etc.);

- reasoning, combining transformers with a residual stream, Attention Residuals (Kimi), etc.

- activation functions: ReLU; SwiGLU; etc.

- etc.

▲phailhaus 3 hours ago | parent | prev | next [-]

Not in this case, what we needed was a genius mathematician to explore with Claude. Which is exactly what Terry said: "So maybe some combination of humans and AIs in 2050 or something will make further progress."

▲bko 2 hours ago | parent | next [-]

I remember when, for a brief period of time, the best chess players were human + computer. There were a lot of Ted talks and think pieces reflecting on this reality. This was called centaur chess.

This lasted for 2005–2014. By 2015, the advantage humans added was negative.

I imagine this will be the case here as well, but I highly doubt it'll take that long.

▲burnerRhodov3 an hour ago | parent [-]

yes, for every specific task. but life is not like chess. life is like playing a million different chess games simulaniously, all of the time. Being a gentle hand that knows when AI is onto something and when it's torturing itself in a loop. He calls it out in the pdf, the aha moments he experiences, that lets him further guide the progress.

▲cman1444 36 minutes ago | parent [-]

Math, specifically, is a bit more like chess though I think.

▲Legend2440 3 hours ago | parent | prev | next [-]

It looks like >99% of the work is the LLM. He even admits as much.

The genius here is Claude.

▲srcreigh 2 hours ago | parent | next [-]

Anthropic would have claimed the result for themselves, without Levant, if they could. They didn’t. A strong mathematician was evidently required.

It doesn't make sense to compare the quantity of work. Humans are significantly more efficient than AI, so little work can go a lot further.

I’m sure for every incredible insight which Levant credits to Claude, there’s 100x+ more bad insights that Levant had to filter out.

▲dontlikeyoueith an hour ago | parent [-]

> I’m sure for every incredible insight which Levant credits to Claude, there’s 100x+ more bad insights that Levant had to filter out.

This is certainly my experience of using Claude every day, in a much less cutting edge domain.

And Claude itself cannot tell the difference between the two. The best it can do is use some sort of external constraint (e.g. verification) to filter them.

▲notsylver 3 hours ago | parent | prev [-]

Company selling shovels says shovels did all of the work

▲pyronite 2 hours ago | parent | next [-]

Human who could claim credit claims AI did all of the work.

▲ClarityJones 2 hours ago | parent [-]

I also use Claude and it never produced this result for me.

▲ipsum2 2 hours ago | parent [-]

Did you also have access to unreleased models by Anthropic?

▲arshxyz 2 hours ago | parent | prev [-]

Not sure anyone would have been able to write 250k pages of Lean without the shovel

▲dzhiurgis 2 hours ago | parent [-]

Real men used to be able to fly to moon on 10kb of memory (and 100 billion dollars)

▲underclocked 20 minutes ago | parent [-]

And they weren't able to prove the Riemann hypothesis

▲mitthrowaway2 2 hours ago | parent | prev [-]

"Maybe 24 years from now we'll make progress" is a little different than "actually it will be solved a month from now with current techniques"

▲boc an hour ago | parent | prev | next [-]

This was literally Dario's contribution to the AI race.

He just did the math and realized you just needed more and more compute to get more intelligence. People keep doubting it but it's showing no sign of being incorrect.

▲MiloLeo 17 minutes ago | parent [-]

When you say more compute are you referring more to just inference compute or training?

ChatGPT was the turning point of realizing we needed just more compute

▲alomaki an hour ago | parent | prev | next [-]

current AI might not be optimal, but still a powerful unoptimal could create optimal AI

▲kridsdale1 3 hours ago | parent | prev [-]

I was downing some extra bitter coffee while I read that.