Remix.run Logo
stavros 8 hours ago

Then test Bob on what you actually want him to produce, ie novel problems, instead of trivial things that won't tell you how good he is.

Why is it a problem of the LLM if your test is unrelated to the performance you want?

troupo 7 hours ago | parent | next [-]

How can Bob produce novel things when he lacks the skills to do even trivial things?

I didn't get to be a senior engineer by immediately being able to solve novel problems. I can now solve novel problems because I spent untold hours solving trivial ones.

stavros 7 hours ago | parent [-]

Because trivial things aren't a prerequisite for novel things, as any theoretical mathematician who can't do long division will tell you.

sgarland 7 hours ago | parent | next [-]

I would love to see someone attempt to do multiplication who never learned addition, or exponentiation without having learned multiplication.

There is a vast difference between “never learned the skill,” and “forgot the skill from lack of use.” I learned how to do long division in school, decades ago. I sat down and tried it last year, and found myself struggling, because I hadn’t needed to do it in such a long time.

thepasch 4 hours ago | parent | next [-]

> There is a vast difference between “never learned the skill,” and “forgot the skill from lack of use.”

This sentence contains the entire point, and the easiest way to get there, as with many, many things, is to ask “why?”

ipaddr 6 hours ago | parent | prev [-]

Most people learn multiplication by memorizing a series of cards 2x2,2x3.. 9x9. Later this gets broken down to addition in higher grades.

Jensson 6 hours ago | parent | next [-]

Most people learn multiplication by counting, it has been in basic mathbooks since forever. "1 box has 4 cookies. Jenny ha 4 boxes of cookies. How many cookies do Jenny have?" and so on, the kids solve that by counting 4 cookies in every of the 4 boxes and reaching 16. Only later do you learn those tables.

sgarland 4 hours ago | parent | prev [-]

That’s definitely not how I learned it, nor how my kids have learned it. I vividly remember writing out “2 x 3 = 2 + 2 + 2 = 3 + 3.” I later memorized the multiplication table up to 12, yes, but that was not a replacement of understanding what multiplication was

Warwolt an hour ago | parent | prev | next [-]

That's not a good analogy. A good mathematician isn't necessarily dealing with calculations, i.e. long division, but rather with proof.

No-ones becomes a good mathematician without first learning to write simple proofs, and then later on more complex proof. It's the very stuff of the field itself.

Folcon 7 hours ago | parent | prev | next [-]

There's a difference between needing no trivial skills to do novel things and not needing specific prerequisite trivial skills to do a novel thing

troupo 7 hours ago | parent | prev [-]

Ah yes. The famous theoretical mathematicians who immediately started on novel problems in theoretical mathematics without first learning and understanding a huge number of trivial things like how division works to begin with, what fractions are, what equations are and how they are solved etc.

Edit: let's look at a paper like Some Linear Transformations on Symmetric Functions Arising From a Formula of Thiel and Williams https://ecajournal.haifa.ac.il/Volume2023/ECA2023_S2A24.pdf and try and guess how many of trivial things were completely unneeded to write a paper like this.

stavros 7 hours ago | parent [-]

Seems that teaching Bob trivial things would be a simple solution to this predicament.

sumeno 6 hours ago | parent [-]

That's what the program he just took was supposed to be for, learning not output. You've just reinvented the article from first principles, congrats

HauntingPin 5 hours ago | parent | next [-]

Sometimes I wonder how deeply some people actually read these articles. What's the point of the comments if all we're doing is re-explaining what's already explained in such a precise and succint manner? It's a fantastic article. It's so well-written and clear. And yet we're stuck going in a circle repeating what's in the article to people who either didn't read it, or didn't read it with the care it deserves.

thepasch 4 hours ago | parent | prev [-]

> That’s what the program he just took was supposed to be for, learning not output.

If you send a kid to an elementary school, and they come back not having learned anything, do you blame the concept of elementary schools, or do you blame that particular school - perhaps a particular teacher _within_ that school?

skydhash 7 hours ago | parent | prev [-]

What people forget about programming is it is a notation for formal logic, one that can be executed by a machine. That formal logic is for solving a problem in the real world.

While we have a lot of abstractions that solve some subproblems, there still need to connect those solutions to solve the main problem. And there’s a point where this combination becomes its own technical challenge. And the skill that is needed is the same one as solving simpler problems with common algorithms.