Remix.run Logo
▲ tgma 20 hours ago

This is a classic fallacy: that interview should look the same as the job. There is no reason[1] your evaluation pipeline has to necessarily resemble the task you are hiring for. You could test people on random cognitive tasks and predict their job performance.

[1]: Purely technical. There may be social reasons or legal consequences like "disparate impact" theories coming at you.

▲mlyle 18 hours ago | parent | next [-]

Sure, random cognitive tasks are correlates of technical job performance.

But there's plenty of people with high intelligence and problem solving that aren't good programmers, so we probably want to try to hone in on that a little bit instead of picking something with positive correlation.

There are also tall people who are not great basketball players, and vice versa. If you can't do anything but measure height, maybe you should pick for your team that way... but if you aren't so restricted there's surely better ways to go.

▲tkamat29 17 hours ago | parent | next [-]

Why would you want to hone in on finding "good programmers" though? Programming is not the end-all be-all skill of being a good software engineer, this was true even before AI but is even more true now.

Leetcode is popular because:

1. It gauges problem solving ability and pattern recognition, which are essential to any job, and particularly useful for software engineers.

2. It is language agnostic - most jobs would rather have a strong engineer that doesn't know their particular language/tech stack than a weak engineer that is familiar with the stack.

3. It has objective evaluation criteria (correctness + time/space complexity).

It's certainly not perfect, but I have yet to see an alternative that accomplishes all of the above. The one thing I can see potentially complementing it is the new style of AI-assisted assessment where you are dropped into an unknown codebase, and asked to make fixes/features that are assisted by an AI agent. This solves the language-agnostic issue with pre-AI versions of these interviews, where you had to have memorized specific tech stack features or language syntax in order to accomplish the task. Now AI can take care of that stuff, and the interviewee can focus on the problem solving aspect.

▲timr 17 hours ago | parent | next [-]

> It gauges problem solving ability and pattern recognition, which are essential to any job, and particularly useful for software engineers.

Proof of claim required. Leetcode primarily measure's someone's ability to memorize leetcode problems, and secondarily measures their ability to vomit this knowledge up in a stressful situation.

There's an entire test-prep industry surrounding this interview method. Any undergrad will tell you that the way you pass leetcode interviews is to "grind". You'd have to be oblivious to not realize that the system has been gamed to hell and back.

Since you are just asserting things, I will assert a simpler explanation: the primary reason Leetcode is popular is because it is cheap. If you have the problem that you have too many applicants and don't want to spend money interviewing them all, you have to come up with an efficient filter that doesn't violate hiring laws. You don't particularly care if the false negative rate is high, so a test of superficially relevant trivia is what you do. Even better if the trivia can be validated by computer!

Setting aside the fact that almost no companies actually had this problem outside of FAANG, thanks to AI, the approach has gone from "somewhat relevant" to "completely irrelevant" within a couple of years. It's time to take it out back and shoot it.

▲01CPMD3174 14 hours ago | parent | prev | next [-]

Leetcode is popular because it gives hiring teams an objective 1)getting people to self select out of the process and 2)shortlisting the people who do opt-in.

I’m sure the companies know that Leetcode is not a great indicator for likelihood to succeed in the role. If they had a perfect information about people, then would stop interviewing at all and just take the best people who applied.

They don’t have perfect information. The information is highly asymmetric. The companies are just trying to not be efficient and not get burned.

It sucks for candidates. But it makes sense for companies when they thousands of applicants they know effectively nothing about.

▲filipdobro 16 hours ago | parent | prev | next [-]

Leetcode interviewing is popular because it barely requires any effort by the interviewer. It is a small, localized knowledge check of your DSA skills. No other profession requires you to invest this much time into the interviewing process. How about we look at past achievements, contributions and certifications, like in any other field?

▲TeMPOraL 14 hours ago | parent [-]

> How about we look at past achievements, contributions and certifications, like in any other field?

In other fields, those things aren't nearly as meaningless and easy to game as they are in ours.

▲filipdobro 10 hours ago | parent [-]

I don't agree with that. No mechanic shop will test if you actually know how to change the brakes of a car. They have to gauge it from the interview process without doing the actual work.

▲mlyle 4 hours ago | parent [-]

The mechanic shop will figure out you don't know what you're doing really, really quickly though. Whereas when we hire someone it's often much longer before we get quality feedback.

▲jjav 14 hours ago | parent | prev [-]

> Leetcode is popular because:

Well, no. It rewards ability (and free time) to repeatedly work through problems and memorize the solutions. And it rewards acting skills so you can perform like you're discovering solutions on the fly, that you've actually memorized.

That's all it does.

▲tgma 14 hours ago | parent | prev [-]

Weeding out smart non-programmers is quite easy. Obvious by their resume and their non-coding interviews. I bet in an unlikely scenario that you fail to detect that, you probably have a pretty good hire who'll learn in a heartbeat.

What you really want to weed out, at least at top-tier companies who can afford to be more selective, is people who have a resume, but are not that great at fundamentals. Most loud complaints seem to be of this nature: guy had N years of experience at previous company/is author of popular software, but couldn't get through basic linked list/recursion/BFS--as if this is not an acceptable bare minimum expectation of a professional in the field.

Often people are too polite to say it out loud, but realistically, that is exactly when it is working as intended.

▲miroljub 13 hours ago | parent [-]

It's not. In 99,99% of programming jobs you will never have to implement a linked list. And yet, interviewers pretend it's important to be able to write it on the blackboard.

▲tgma 13 hours ago | parent [-]

Not the point. If you have way more applicants than roles and you are paying well, given the choice between a programmer who cannot implement a linked list [really?] and one who can, you would hire the one who has that ability, even if all they do all day is to stare at Claude Code.

▲montag 15 hours ago | parent | prev | next [-]

Leetcode is and has always been a conveniently disguised IQ test.

▲tptacek 6 hours ago | parent | next [-]

People love saying this, but it doesn't make any sense. There are software firms that, within the past 15 years, broadly screened all candidates with actual IQ tests. There's no reason to disguise them at all. The reason nobody in the industry does this is that we all know what a debacle it would be to screen primarily on IQ tests rather than "can people do the job effectively".

▲menaerus 14 hours ago | parent | prev | next [-]

It takes some IQ to solve leetcode problems but better performance of human A in comparison to human B does not say anything about their absolute IQ capabilities. It's a conveniently disguised test for attracting highly motivated individuals with average or more than average IQ and nothing else.

▲JustFinishedBSG 8 hours ago | parent | prev | next [-]

It's an absolutely terrible intelligence test.

It's basically just a grinded leetcode test.

▲oreally 6 hours ago | parent | prev [-]

[flagged]

▲oreally 6 hours ago | parent | prev | next [-]

Wrong.

Literal psychologists have done the research on this: A good, predictive interview is a small representative test of what people do in the normal job. Only us techbros seem arrogant enough to want to be the exception to decades of research.

▲timr 19 hours ago | parent | prev | next [-]

> This is a classic fallacy: that interview should look the same as the job. There is no reason[1] your evaluation pipeline has to necessarily resemble the task you are hiring for.

Well then. If the "classic fallacy" is that job interviews should reflect the actual job, then what does it really matter what we do? Let's just have a big pub quiz. That measures "intelligence", too.

> You could test people on random cognitive tasks and predict their job performance.

Show me a single piece of evidence that memorizing code golf answers from a book is a valid proxy for intelligence, and we'll talk.

(Hint: You'll have better luck convincing me that it's a stealth way of discriminating against older, more experienced applicants who are further from undergrad and less willing to re-memorize Cracking the Coding Interview for the 150th time. Because it actually is.)

In any case, I do not understand this...Stockholm syndrome...that some programmers seem to have for the idiotic interview system.

▲boorang 18 hours ago | parent | next [-]

> (Hint: You'll have better luck convincing me that it's a stealth way of discriminating against older, more experienced applicants who are further from undergrad and less willing to re-memorize Cracking the Coding Interview for the 150th time. Because it actually is.)

I hadn't actually thought about it this way, but it makes sense. I'm in my early 50s and hella experienced and talented (if i do say so myself), but I refuse to do any interviews that require me to do any more preparation than researching the company I'm applying to.

▲timr 17 hours ago | parent [-]

I think it's a common story for anyone who has been around a while: you have a career (possibly even working for big, famous names) develop a long resume of accomplishments, and some dinky startup with a staff of teenagers rejects you via email because you didn't solve a leetcode hard problem while singing and dancing on zoom.

It's obviously absurd, and while I almost understand it for a company that doesn't know you, you'll get hit with this stuff when you've been referred internally, by people you worked with in the past.

▲tgma 19 hours ago | parent | prev [-]

Obvious strawman: is "coin flipping tournament" a "random cognitive task?"

There is a century of data and studies that suggest cognitive ability is positively correlated across tasks. Conducting IQ tests for hiring is practically illegal in the US. Leetcode is the closest you get to an IQ test without tripping some laws.

▲timr 18 hours ago | parent | next [-]

Yes, I changed it to a better metaphor before you wrote your comment. I had originally written "pub quiz", then changed it to "coin flipping contest" to emphasize the stupidity of the task, then changed it back, anticipating the criticism. The argument does not change -- the claim that testing coding trivia is a test of intelligence is a bald assertion.

Please use your intelligence to see the argument I am making, instead of nitpicking the example.

> Conducting IQ tests for hiring is practically illegal in the US. Leetcode is the closest you get to an IQ test without tripping some laws.

No, it isn't. Disparate impact is. Any test that has a disparate impact on a protected class is de facto illegal, but there's case law surrounding IQ tests.

By the way...ironically enough, Griggs v. Duke Power established that any intelligence test must have relevance to the job responsibilities, which sort of blows a hole in your entire theory of the case that this is a "classic fallacy".

https://en.wikipedia.org/wiki/Griggs_v._Duke_Power_Co.

▲kelipso 17 hours ago | parent [-]

> the claim that testing coding trivia is a test of intelligence is a bald assertion

I think you have to let this one go. SAT, LSAT, GRE, ASVAB all correlate well with IQ scores. Leetcode should too, after the interviewee does a few weeks of practice (just like all those other tests).

▲timr 17 hours ago | parent [-]

> Leetcode should too, after the interviewee does a few weeks of practice (just like all those other tests).

I mean, you're just making stuff up, but ok.

▲tptacek 18 hours ago | parent | prev [-]

Not only is it not practically illegal, as the sibling commenter notes, but several household-name corporations openly IQ test candidates. General cognitive testing is presumptively lawful in white collar employment.

▲tgma 18 hours ago | parent [-]

Your pivot from "IQ test" in your first sentence to "general cognitive testing" in second is the whole point.

Leetcode is just one step away from that (and much more litigation-proof).

▲tptacek 18 hours ago | parent [-]

IQ tests are general cognitive tests. If your entire argument is that IQ tests are fine so long as you call them something else, you're not really saying anything.

Leetcode is not a general cognitive test, obviously.

▲tgma 18 hours ago | parent [-]

IQ tests are general cognitive tests. The opposite is not necessarily true.

The whole point of the multibillion dollar test industry is to fudge a "noisy" IQ[ish] test and make it look like it won't have disparate impact (validation "studied") and buy you some ass-covering ability.

> Leetcode...

"one step away"

▲tptacek 17 hours ago | parent [-]

Leetcode is a domain-specific test. It is the opposite thing to a general cognitive test. The entire point of IQ is the positive manifold: the notion that general cognitive tests are all monitoring the same latent variable.

If you're saying, like, the Wonderlic test is fine but IQ testing isn't, all you're saying is that the "IQ brand" is problematic. That's wrong, but also it's not saying anything interesting. Call it whatever you want, it's the same business premise.

If you're saying that Leetcode is, like the Wonderlic, a direct proxy for IQ, that's just clearly false.

▲tgma 17 hours ago | parent | next [-]

[flagged]

▲ 17 hours ago | parent | prev [-]
[deleted]
▲Mawr 15 hours ago | parent | prev [-]

> This is a classic fallacy: that interview should look the same as the job.

I think I'm gonna get a stroke reading this. The classic fallacy is the nonsense you're spouting — that there's anything besides the job itself that is relevant to it. You're hiring people for x, so the only thing you care about is their ability to do x. This is obvious.

The perfect test is just hiring someone on the spot and seeing how they perform. We don't do this because it's expensive but obviously it would be the perfect test, measuring exactly what you're hiring for, by definition.

So then any test that goes further and further away from that ideal becomes worse and worse at predicting job performance. The trick is figuring out a good enough test that's still reasonably quick and cheap to do.

Deliberately going out of your way to test your canditates for something unrelated to the job is insanely stupid.

It's essentially just arrogance. Arrogantly asserting that you know for sure your particular "generic" test actually selects for ~all traits you want while also rejecting ~all traits you don't. I assure you, the chances of you having a good enough model of reality to do this are laughable. Pure arrogance.

Again, easiest to see by thinking of the opposite example to before — the worst test possible would of course be something completely unrelated, so idk, having a chef play ping-pong. I'm sure you could convincingly argue it tests something, but why in the world would you not just have him make a dish, since you know, that's what you're hiring him to do? (it's not actually by the way, now that's a classic fallacy — that the job is the mechanical subset of it and not the job as a whole. Soft skills, anyone? sigh).

I've seen someone suggesting that since a lot of programmers like playing Factorio, why not interview people by having them play Factorio? Ah, because then you're selecting for Factorio players. Do you have a good model of what makes a good Factorio player? Hell, do you have a good model of what makes a good performer at your job? You absolutely do not. Why select for Factorio players when you can select for your-job players?

Another common example of this utter arrogance is thinking that a candidate's performance during a highly stressful, unnatural, and artificial classic interview grilling environment is indicative of their perfomance in the every day feeling-secure-in-their-job state of mind. Oh, look, this fool I'm basically abusing from my position of power is stammering and squirming, must mean he's a good-for-nothing loser! Hah, another fraud found!

So sure, keep putting people in as much stress as possible and asking them how many golf balls fit in an airplane. BTW, I walk out of such interviews immediately. Have fun with your new Leetcode grinder performative actor hires though — you get what you test for.

▲tgma 15 hours ago | parent [-]

> ...you get what you test for.

Multiple trillion dollar companies has not been that bad.