I actually still don't see the source for them trying several times, but we can take that for granted. Regardless, as I said:

1. It's labeled as "moderately interesting"

2. They said that they expect an expert could solve it in 1-3 months

3. They had already come up with the solution that the AI had but weren't convinced it would have worked

So how big was the gap here, do you think?

▲

famouswaffles 3 hours ago | parent | next [-]

Yes, a "moderately interesting" Open problem.

I can't think of any chores that would take an expert months to complete. I can't think of any chores that I've completed but was then 'unconvinced could work'. Please sit down and think about what you are saying here. Are we still talking about chores ?

One of the more strange phenomena with machines getting better and the incessant need (seemingly driven by human exceptionalism) to downplay each result, is that you just end up belittling humans in the process.

This is significant. Your analogy is wrong. It's fine to admit it.

▲

staticassertion 2 hours ago | parent [-]

Writing a complex parser or certainly a compiler is a 1 - 3 month project, for example.

Again, I'm not trying to downplay this, but to frame this accurately. I think an AI being able to build a parser/ compiler is cool too.

> One of the more strange phenomena with machines getting better and the incessant need (seemingly driven by human exceptionalism) to downplay each result, is that you just end up belittling humans in the process.

I don't believe in human exceptionalism at all, don't attribute positions to me.

▲

famouswaffles 2 hours ago | parent [-]

>Writing a complex parser or certainly a compiler is a 1 - 3 month project, for example.

1. Estimating time completion of something that has been done multiple times before and an open problem that has not yet been solved is a different matter entirely. 1 to 3 months is an educated guess and more likely than not, an underestimate.

2. I do not think months long complex compilers and parsers are being routinely completed by LLMs as your original comment implied. Regardless, they are different classes of problems.

▲

staticassertion 2 hours ago | parent [-]

I don't get what either of your points is intended to demonstrate. Let's revisit the first post I replied to:

> It's deeply surprising to me that LLMs have had more success proving higher math theorems than making successful consumer software

As far as I can tell, they absolutely have not had more success in this area relative to making successful consumer software.

▲

famouswaffles an hour ago | parent [-]

Well we are kind of arguing past each other aren't we ?

"More success" is a bit vague in this instance but building a compiler that would take a programmer 1 to 3 months is not comparable to this result regardless of whatever similarity exists in time completion estimates. That's the point.

You can publish a paper (and in fact the researchers plan to) off this result. A basic compiler is cool but otherwise unremarkable. It's been done many times before.

You are leaning too hard on how long the researchers (who again did not manage to solve the problem in their attempts) estimated this would take and the "moderately interesting" tag of again, what was still an open research problem.

This, alongside a few math and physics results that have cropped up in the last few months is easily more impressive than the vast majority of work being done with LLMs for software.

▲

staticassertion an hour ago | parent [-]

> "More success" is a bit vague in this instance but building a compiler that would take a single programmer 1 to 3 months is not comparable to this result regardless of whatever similarity exists in time completion estimates. That's the point.

I guess we just disagree on this. It's not clear to me that these are totally different in terms of what they represent.

> You can publish a paper (and in fact the researchers plan to) off this result. A basic compiler is cool but otherwise unremarkable.

Publishing papers means very, very little to me. I can publish a paper on a programming language, you know that, right?

> You are leaning too hard on how long the researchers (who again did not manage to solve the problem in their attempts) estimated this would take and the "moderately interesting" tag of again, what was an open research problem.

I obviously estimate my "leanings" as being appropriate. I'm just using the researchers direct quotes. Factually, they had already come up with the approach that ultimately panned out. Factually, they estimated that a human could do this in some timeframe. What am I overly leaning on here?

> This, alongside a few math results that have cropped up in the last few months is easily more impressive than the vast majority of work being done with LLMs for software.

I think both are impressive, I don't know that I would draw some sort of big conclusions about it at this point. I definitely wouldn't draw the conclusion that AI is better at formal mathematics than producing software.

▲

famouswaffles an hour ago | parent [-]

>Publishing papers means very, very little to me. I can publish a paper on a programming language, you know that, right?

We both know that you are not getting that published in a reputable journal without a lot of effort beyond merely 'publishing the language I created', but sure, I'm sure you can get something on arxiv.

>I obviously estimate my "leanings" as being appropriate. I'm just using the researchers direct quotes. Factually, they had already come up with the approach that ultimately panned out.

This really should not be hard to understand.

1. One is something that has been done many times before and the other an unsolved problem. It doesn't take a genius to see one estimate is likely much stronger than the other. If your point hinges on comparing them directly, it's pretty weak.

2. A moderately interesting open research problem is not the same thing as a moderately interesting problem and you seem to be conflating the two.

▲

staticassertion an hour ago | parent [-]

> We both know that you are not getting that published in a reputable journal without a lot of effort beyond merely 'publishing the language I created'. But sure, you can get something on arxiv.

lol what? There are papers on programming languages all the time.

> 1. One is something that has been done many times before and the other an unsolved problem. It doesn't take a genius to see one estimate is likely much stronger than the other.

Building a compiler for a new programming language, building net new code, etc, is all stuff that was unsolved / had not been done before.

> 2. A moderately interesting open research problem is not the same thing as a moderately interesting problem and you seem to be conflating the two.

Feel free to explain the difference, I guess.

▲

famouswaffles an hour ago | parent [-]

>lol what? There are papers on programming languages all the time.

Sure and have you read them ? They're the results of many months or years of research and development so I really don't know what point you think you are making here.

>Building a compiler for a new programming language, building net new code, etc, is all stuff that was unsolved / had not been done before.

Okay but that's not taking a month or two or being asked of LLMs x10000 every day so thanks for making my point I guess.

>Feel free to explain the difference, I guess.

No thanks. If you don't understand it that's fine. This has run its course anyway.

	▲	staticassertion an hour ago \| parent [-]
		Yeah idk what you're going on about lol

▲

famouswaffles 3 hours ago | parent | prev [-]

Also, the full write up does not say the researchers solved it.

	▲	staticassertion 2 hours ago \| parent [-]
		> I had previously wondered if the AI’s approach might be possible, but it seemed hard to work out. They didn't solve it, that's fair. They did consider the approach already.