Great article. I like the simple point about the hypothetical IQ test sent one week in advance. It makes a strong case about time being the true bottleness. I think this same idea could be applied to most tests.

Implicit in the design of most tests is the idea that a person's ability to quickly solve moderately difficult problems implies a proportional ability to solve very difficult problems if given more time. This is clearly jumping to a conclusion. I doubt there is any credible evidence to support this. My experience tends to suggest the opposite; that more intelligent people need more time to think because their brains have to synthesize more different facts and sources of information. They're doing more work.

We can see it with AI agents as well; they perform better when you give them more time and when they consider the problem from more angles.

It's interesting that we have such bias in our education system because most people would agree that being able to solve new difficult problems is a much more economically valuable skill than being able to quickly solve moderate problems that have already been solved. There is much less economic and social value in solving problems that have already been solved... Yet this is what most tests select for.

It reminds me of the "factory model of schooling." Also there is a George Carlin quote which comes to mind:

"Governments don't want a population capable of critical thinking, they want obedient workers, people just smart enough to run the machines and just dumb enough to passively accept their situation."

I suspect there may be some correlation between High IQ, fast thinking, fast learning and suggestibility (meaning insufficient scrutiny of learned information). What if fast learning comes at the expense of scrutiny? What if fast thinking is tested for as a proxy for fast learning?

What if the tests which our society and economy depend on ultimately select for suggestibility, not intelligence?

>most people would agree that being able to solve new difficult problems is a much more economically valuable skill than being able to quickly solve moderate problems that have already been solved

Do most people agree with that? I agree with that completely, and I have spent a lot of time wishing that most people agreed with that. But my experience is that almost no one agrees with that...ever...in any circumstance.

I don't even think society as a whole agrees with this statement. If you just rank careers according to the ones that have the highest likelihood of making the most money, the most economically valuable tend to be the ones solving medium difficulty problems quickly.

"Implicit in the design of most tests is the idea that a person's ability to quickly solve moderately difficult problems implies a proportional ability to solve very difficult problems if given more time."

I used to share that doubt, especially during my first semesters at university.

However, my experience over the decades has been, that people who solved moderately difficult problems quickly were also the ones that excelled at solving hard and novel problems. So in my (little) experience, there is a justification for that and I'd be definitely interested (and not surprised) to see credible evidence for it.

> I like the simple point about the hypothetical IQ test sent one week in advance.

It’s a simple point but an incorrect one.

If you can work on it for a week, it’s no longer an IQ test. Nobody is saying that the questions on an IQ test are impossible. It’s the fact that there are constraints (time) and that everybody takes the test the same way that makes it an IQ test. Otherwise it’s just a little sheet of kinda tricky puzzles.

Would you be a better basketball player if everyone else had to heave from 3/4 court but you could shoot layups? No, you’d be playing by different rules in an essentially different game. You might have more impressive stats but you wouldn’t be better.

> Implicit in the design of most tests is the idea that a person's ability to quickly solve moderately difficult problems implies a proportional ability to solve very difficult problems if given more time. This is clearly jumping to a conclusion. I doubt there is any credible evidence to support this.

I think this approach is effectively testing if a student studied the material. It makes the correlation between memorization and understanding. Recall a piece of information is fast if avaliable.

Its a commonly expressed experience among university students that learning memorization techniques and focusing on solving previous exams is a disproportionately effectively way to pass courses.

It's technically more impressive to pass the exam by never doing a single similar problem before and deriving a solving method or forumla that wasn't memorised.

I took deliberate effort to avoid looking at previous exam question for a course until the week before, since it cased good grades at little value to me long term.

> I suspect there may be some correlation between High IQ, fast thinking, fast learning and suggestibility (meaning insufficient scrutiny of learned information). What if fast learning comes at the expense of scrutiny? What if fast thinking is tested for as a proxy for fast learning?

Precisely. Speaking from experience, in school, every claim that I was supposed to accept and reproduce on an exam or in homework was met with a gut response: "Is this really true? Is so, why? How do you know?". I wanted to verify the information and know the justification for believing it, the reason something was true. What's more, I had trouble with the coherence of the claims being made. The physics we are taught in school, for example, raises very serious metaphysical questions. This produced in me a spirit of rebellion. I felt a certain vague disgust for the way things were taught that frustrated my motivation. In some sense, it didn't feel like truth was being treated seriously. The ceremony of education, with all its trappings, is all that was treated seriously. "Getting the grade", not understanding something was what it was all about. It felt like an acrobatics contest and a game of one-upmanship.

Now, sometimes, the justification for a claim was obvious, at least given certain premises (these are often left tacitly assumed, often implied: the danger), but that's not always or perhaps even usually the case. Even in math, a science that can be done from the armchair, we are given formulas and methods that are supposed to be taken on faith and simply used. Through repetition, we are supposed to become better at identifying situations in which we can apply them. But where do these formulas and methods come from? What do they tell us, and how do we know?

And I emphasize "faith": there is no way the valedictorian has verified everything he or she was taught or knows the justification for them. A "good student" keeps up, and since scrutiny and analysis take time and skill - time no student is given especially as the workload piles up, and skill no student possesses - a faithful student, a student who obediently accepts what he or she is being told. You can imagine that blind faith would produce the "perfect student". (Curiously, we are simultaneously commanded to "question everything" - except questioning everything, of course - but then required not to actually practice that advice.)

Now, you could argue that students are too young to understand the justifications for the claims being made, and in practice, we are always relying on faith in some authority. Few people realize how much faith we rely on in our lives. Society entails a certain epistemic deference, even if merely practical or perfunctory. In practice, it is unrealistic not to rely on faith. Faith has its proper place.

Someone might also say that students could be bracketing the information they are receiving. They may simply be entertaining it as a possibility in good faith and playing with it, until verification becomes possible or necessary. Maybe. But given the intellectual immaturity of students, and the obedience at the top, I suspect there is at least a superficial assent given to what they are taught. Otherwise, school is a game to be played, one that, we are told, is an instrument for climbing the ladder of social status. The content doesn't matter. What matters is that you play by the rules of the game and that you play by them well. When you do that, the kingmakers and status granters will throw you a few golden chips and elevate you in the eyes of society. You will be in.

Sounds cynical. After all, wouldn't an institution that wants to select for wisdom also create barriers? Of course, regardless of how effective they are. But the differences cannot be ignored. The intent and purpose are different, for one. The means of selection are another. Education is bureaucratized. We think that bureaucracy will create a "level playing field", eliminating the biases and favoritism that "personal judgement" is bound to entail. But who designs the bureaucracy? What does it actually select for? And does it not often commit the fallacy of confusing features of the method for features of the real?

We're obsessed with rank, and bureaucratic methods make us obsessively so. We imagine there is a sharper slope and a smaller peak than there really is. There is a slope, to be sure. I am no egalitarian. But come on.

Anyway, for all that rambling, what are some of the morals here?

I suppose my first point is that education ought to be focused on first principles first. It ought to be focused on understanding and truth and learning the competence of being able to get there, as that is the whole point. The trivium and quadrivium of old did this. People think of the Middle Ages as some kind period hostile to education. They think it was like the Prussian-style of education (from which modern education gets a lot of its ideas), oriented toward mindless obedience and unquestioned submission to the state. Nothing could be further from the truth. Universities were renowned for open discussion and debate, perhaps most famously in the form of the disputation. The Scholastics were famous for intellectual rigor, a rigor that puts to shame the pompous pretensions of the so-called Enlightenment that never missed the opportunity to erect straw men of the Medievals to ridicule.

Second, rewards and penalties are selection mechanisms. We get the behavior we reward and we get less of the kind we punish. Habits are like this, too: indulging a habit of overeating reinforces and magnifies the habit, while restraint has the opposite effect. What does our education system feed? What does it starve? We should ask this question ceaselessly.