Remix.run Logo
▲ MiiMe19 40 minutes ago

The Alpoge proof is like 250k pages, I doubt anyone knows what is going on with it, even Alpoge.

▲CapmCrackaWaka 32 minutes ago | parent | next [-]

If his proof is 250,000 pages, then I am extremely dubious. Just today, I was working with about 1000 lines of code and asked opus 5.4 to make a change. It introduced an obvious bug, that actually ended up doing the exact opposite of what I asked to do( print out less lines of logging, instead of more).. even if he has access to some kind of super LLM, I am extremely skeptical that it can output a 250,000 page document that is coherent and bug free, no matter what this guy says.

▲ThePhysicist 12 minutes ago | parent | next [-]

I would assume they do not produce the proof at once, and Lean is different from programming language (as far as I understand) in the sense that you can check if a program is correct or not within the Lean model, so it seems ideally suited for AI as they can experiment and extend the proof without ever producing a bug. I assume the real difficulty is checking that what you have proven is actually what you wanted to express but I would assume this is easier.

Makes me wonder if general programming languages are too expressive for AI making it too easy to express conflicting or incorrect ideas, we might need something like Lean but for general purpose programming. I'm aware there are some formal systems for that but I think more along the lines of general purpose programming.

▲ 7 minutes ago | parent | prev | next [-]
[deleted]
▲cakeface 9 minutes ago | parent | prev | next [-]

Did you deliberately pick Opus 5.4 over Opus 5.5? That was the biggest thing that stood out to me in your comment. Opus 5.5 has been an all time banger for me.

▲redbluered3 26 minutes ago | parent | prev [-]

You don't have access to the secret model the labs keep for themselves. Neither can you spin up thousands of subagents and burn billions of tokens a day.

Proof can be total bunk but underestimating the labs is silly.

▲lossolo 37 minutes ago | parent | prev [-]

So how do we know if it does what it does? Anyone that is using frontier models for a few years, will tell you that they try to reward hack whenever they can.

▲Frost1x 29 minutes ago | parent | next [-]

That’s the purpose of Lean. It’s kind of like asking: how do you know your CPython is running all the correct binary?

Well, you kind of don’t, as it’s beyond human comprehension (in theory you could trace everything down to a certain level) but there are layers and layers of build abstraction that you assume are correct, compilers, runtime environments, and mature libraries that are checked and validated. Not a perfect analogy but that’s kind of the gist of it.

▲lossolo 22 minutes ago | parent [-]

OK, but if I write code, I can verify it in a few simple steps. What about 250k pages of proof? How can I verify that page 133,432 has the correct assumptions and that it isn't reward hacked just to make lean code pass? How it's done?

▲cman1444 7 minutes ago | parent [-]

You can personally verify every binary digit of your code in a few simple steps?

▲m11a 35 minutes ago | parent | prev [-]

In theory, if you formulated your proposition correctly, and Lean says it is proved, then the proposition is true. This works because of the Curry-Howard correspondence -- proofs can be represented as computation.

▲lossolo 17 minutes ago | parent [-]

OK, so if I understand correctly, I formulate a proposition in natural language, then an LLM creates 250k pages of proof based on it, then it takes those 250k pages/lines of proof and creates 15 million lines of code to prove those 250k pages. Is that correct? If so, how can I be sure that my proposition is correctly translated into those 250,000 pages? Isn't it possible that the LLM can reward hack here in the same way I've seen it do in some cases where it struggles to pass the test and get its reward? For example, it tries to change assumptions without telling me because something doesn't work or it removes something or changes the test etc.

▲chunky1994 3 minutes ago | parent [-]

The 250k pages is the 15M LOC of lean proof. The code is the proof.