In my country, we have public exams to get into Uni, with the ones for high demand majors being very competitive, but performance in that exam is not a good predictor of academic performance.

The guy who got into my uni class as #2 in that exam dropped out after a few semesters because he couldn't beat calculus. The #4 took several extra semesters to graduate despite not working/not interning. Several others in the top third struggled through. We had _maybe_ 2 or 3 guys who straight-A'd the entire major.

I myself got in as #17 and still failed a few courses. Thankfully no one cared throughout my professional career.

> I don't buy the notion that tests do not test relevant skills.

> In my long career I've noticed a strong correlation between SAT scores and academic performance as well as job performance.

A test doesn't need to test the relevant skills for that, it just needs to test _something_ that correlates with academic performance and job success.

Do you also think LLM leaderboards accurately reflect the capabilities of the models being tested? If you do, then I can easily point you to numerous academic papers pointing out the various flaws in many leaderboards (from poorly designed benchmarks like bABI and the original SQuAD, to data contamination, and more).

In that same way, any test, including the SAT and GRE have flaws. They can be gamed in ways similar to LLM leadeboards: test prep makes you better at them. That's one of the main reasons universities moved away from SAT; they were afraid that it disenfranchised lower socioeconomic status students (and it does to some degree). The issue is that the test is positively correlated with success in an undergraduate program, so they threw out the baby with the bathwster. The real issue is that the SAT is not able to distinguish the capabilities among students to the degree it purports to.

And if you want an anecdote to match all yours, the first time I took a GRE practice test, I got a 3 on the writing. Not because I'm poor at writing, but because I didn't really know what they were looking for. After reading a test prep book, I got a 4.5 on my next practice test and a 5 on my final practice test. When I finally took the actual GRE, I got 6 on the analytical writing. Trust me, nothing changed in my writing ability over that time. In fact, I didn't even practice the skill except through those three practice tests. Clearly the test was not capable of determining my real ability to make an argument; it merely tested my ability to adapt my writing to what was supposedly being tested.

Interestingly, the vast majority of universities that got rid of the GRE requirements for PhD programs are not going back on that. Turns out that the students with the highest GRE scores are the ones most likely to drop out of their STEM PhD. [1]

[1]: https://journals.plos.org/plosone/article?id=10.1371/journal...

> I don't buy the notion that tests do not test relevant skills. In my long career I've noticed a strong correlation between SAT scores and academic performance as well as job performance.

SAT tests intelligence (aptitude), not skills. Which is why it correlates with job performance, where intelligence can (over some time) matter as much or more than a starting point of relevant skills.

Back in the dark ages of the 80s when I was taking the SAT and ACT, these tests were considered good predictors for the first semester's performance.

That's it.

I did well on both tests and did well on my first semester. It's the semesters after where my performance tanked because I didn't have some of the work habits that solid B students had. (I will also be clear and say that at least some of the problem is attributable to the university and how it handled advisors. My advisor was completely useless and let me schedule for _way_ too much hard stuff.)

There was also a really good predictor of how one would do on the SAT or ACT: NoBitH. The Number of Bathrooms in the House.

Yup. The SAT and ACT at the time were better measures of economic advantage than innate intelligence. I have no reason to believe that this isn't still the case, especially since they're more entrenched in the system than ever.

Given that I'm not responding to any claim about the efficacy of math tests... it's actually your statement which is wholly irrelevant to this discussion.

Because I'm not trying to make a universal claim about all tests.

For one semester.

Maybe two.

So you yourself has just given an example how easy it is to temporarily get good at "test-filling", by simply preparing for that.

The question is whether a given test measures anything relevant - did your friend become a better programmer for doing 6 weeks of leetcodes? E.g. what kind of experience did he gain about large code bases and how to handle those? Continuing your analogy, would you fly with a pilot who drilled on taking off a bunch of time, but never practised flying in a storm?

I'm not saying leetcodes or exams are useless, but Goodhart's Law apply.

This has to be a joke...

Ground school most certainly does not involve a lot of math, it's not like there's any calculus or algebra involved... it's basic arithmetic. Furthermore it's categorically false that you need to pass ground school before you're allowed to fly.

Are you just making things up?

>A teacher that cannot explain how calculus works cannot teach it to anybody.

This is a strawman argument, I never made anything that could even remotely be interpreted as this.

>I've seen too many coders using bubble sort because they don't know enough to look for a better algorithm.

This is committing a very basic logical fallacy. The fact that someone who is incompetent likely can't pass a test is not the same claim as someone who can't pass a test is likely incompetent.

Hopefully you are able to identify this logical mistake that you're committing and revise your position accordingly.