Remix.run Logo
perlgeek 5 months ago

Company A wants to hire an engineer, an AI could solve all their tech interview questions, so why not hire that AI instead?

There's very likely a real answer to that question, and that answer should shape the way that engineer should be assessed and hired.

For example, it could be that the company wants the engineer to do some kind of assessment whether a feature should be implemented at all, and if yes, in what way. Then you could, in an interview, give a bit of context and then ask the candidate to think out loud about an example feature request.

It seems to me the heart of the problem is that companies aren't very clear about what value the engineers add, and so they have trouble deciding whether a candidate could provide that value.

juujian 5 months ago | parent | next [-]

The even bigger challenge is that hiring experts in any domain requires domain knowledge, but hiring has been shifted to HR. They aren't experts in anything, and for some years they made do with formulaic approaches, but that doesn't cut it anymore. So now if your group wants to get it done, and done well, you have to get involved yourself, and it's a lot of work on top of your regular tasks. Maybe more work because HR is deeply involved.

ghaff 5 months ago | parent | next [-]

>hiring has been shifted to HR

Well, unless you know sufficiently senior people. But I suspect that is a deeply unsatisfactory answer to many people in this forum.

My long term last, only technically-adjacent, job came through a combination of knowing execs, having gone to the same school as my ultimate manager, and knowing various other people involved. (And having a portfolio of public work.)

ctkhn 5 months ago | parent | prev | next [-]

I saw this at the big corporate (not faang/tech) place I work at. Engineers run and score interviews, but we don't make the final decision. That goes to HR and the hiring manager who usually has no technically background.

BobaFloutist 5 months ago | parent | prev | next [-]

HR are experts in HR, which is to say they have a broader view of the institutional needs and legal requirements of hiring staffing than you do. It's always annoying when that clashes with your vision, but dismissing their entire domain is unlikely to help you avoid running into that dynamic again and again

whiplash451 5 months ago | parent | prev | next [-]

> hiring has been shifted to HR

Not everywhere. At my company, HR owns the process but we -- the hiring tech team -- own the content of interviews and the outcomes.

bongoman42 5 months ago | parent | prev [-]

I've never seen hiring completely in the domain of HR. HR filters incoming candidates and checks for culture fit etc, but technical competency is checked by engineers/ML folks. I can't imagine an HR person checking if someone understands neural networks.

michaelt 5 months ago | parent | prev | next [-]

> Company A wants to hire an engineer, an AI could solve all their tech interview questions, so why not hire that AI instead?

Interview coding questions aren't like the day-to-day job, because of the nature of an interview.

In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law.

It also has to be a problem that's solvable within about 40 minutes.

The problem needs to test the candidate meets the company's hiring bar - while also having enough nuance that there's an opportunity for absolutely great candidates to impress me.

And the problem has to be possible to state unambiguously. Can't have a candidate solving the problem, but failing the interview because there was a secret requirement and they failed to read my mind.

And of course, if we're doing it in person on a whiteboard (do people do that these days?) it has to be solvable without any reference to documentation.

gopher_space 5 months ago | parent | next [-]

> In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law.

If you send me a rubric I can pre-load whatever you want to talk about. If you tell me what you're trying to build and what you need help with, I can show up with a game plan.

You need to make time for a conversation on the intricacies of voucher calculation and global sales tax law if you want to find people jazzed about the problem space.

qudat 5 months ago | parent | prev | next [-]

> In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law.

Proving if they are technically capable of a job seems rather silly. Look at their resume, look at their online works, ask them questions about it. Use probing questions to understand the depths of their knowledge. I don't get why we are over-engineering interviews. If I have 10+ years of experience with some proof through chatting that I am, in fact, a professional software engineer, isn't that enough?

janoc 5 months ago | parent | prev [-]

>Interview coding questions aren't like the day-to-day job, because of the nature of an interview.

You have missed his point. If the interview questions are such that AI can solve them, they are the wrong questions being asked, by definition. Unless that company is trying to hire a robot, of course.

rurp 5 months ago | parent | prev | next [-]

One of the best interviews I've encountered as a candidate wasn't exactly a pair programming session but it was similar. The interviewer pulled up a webpage of theirs and showed me a problem with it, and then asked how I would approach fixing it. We worked our way through many parts of their stack and while it was me driving most of the way we ended up having a number of interesting conversations that cropped up organically at various points. It was scheduled for an hour and the time actually flew by.

I felt like I got a good sense of what he would be like to work with and he got to see how I approached various problems. It avoided the live coding problems of needing to remember a bunch of syntax trivia on the spot and having to focus on a quick small solution, rather than a large scalable one that you need more often for actual work problems.

nottorp 5 months ago | parent | prev | next [-]

Problem is, company A doesn't need an engineer to solve those interview questions but real problems.

placardloop 5 months ago | parent | next [-]

“Real problems” aren’t something that can be effectively discussed in the time span of an interview, so companies concoct unreal problems that are meant to be good indicators.

okdood64 5 months ago | parent | prev [-]

This is the answer.

Let's not pretend 95% of companies are asking asinine interview questions (though I understand the reasons why) that LLMs can easily solve.

bitwizeshift 5 months ago | parent | prev | next [-]

Tech interviews in general need to be overhauled, and if they were it’d be less likely that AI would be as helpful in the process to begin with (at least for LLMs in their current state).

Current LLMs can do some basic coding and stitch it together to form cool programs, but it struggles at good design work that scales. Design-focused interviews paired with soft-skill-focus is a better measure of how a dev will be in the workplace in general. Yet, most interviews are just “if you can solve this esoteric problem we don’t use at all at work, you are hired”. I’d take a bad solution with a good design over a good solution with a bad design any day, because the former is always easier to refactor and iterate on.

AI is not really good at that yet; it’s trained on a lot of public data that skews towards worse designs. It’s also not all that great at behaving like a human during code reviews; it agrees too much, is overly verbose, it hallucinates, etc.

lanstin 5 months ago | parent | prev | next [-]

I want to hire people who can be given some problem and will go off and work on it and come to me with questions when specs are unclear or there's some weird thing that cropped up. AI is 100% not that. You have to watch it like a 15 year old driver.

diob 5 months ago | parent | prev | next [-]

It's because coding interview questions aren't so much assessing job skills as much as they are thinly veiled IQ tests.

I think if it was socially acceptable they'd just do the latter.

tptacek 5 months ago | parent | next [-]

Plenty of companies administer IQ tests. The reason everyone doesn't is that it doesn't work well.

vasco 5 months ago | parent | prev [-]

A lot of companies have IQ like tests, in particular big consulting companies like McKinsey and so on.

Imnimo 5 months ago | parent | prev | next [-]

A company wants to hire someone to perform tasks X, Y and Z. It's difficult to cleanly evaluate someone's ability to do these tasks in a short amount of time, so they do their best to construct a task A which is easy to test, and such that most people who can do A can also do X, Y and Z.

Now someone comes along and builds a machine that can do A. It turns out that while for humans, A was a good indicator of X, Y and Z, for the machine it is not. A is easy for the machine, but X, Y and Z are still difficult.

This isn't a sign that the company was wrong to ask A, nor is it a sign that they could just hire the machine.

dahart 5 months ago | parent | prev | next [-]

This is a great point. Though what if the answer is that the company can hire that AI to solve a significant fraction of its actual problems? People who do the assessments and decide what features should look like are often called managers (product, engineering, etc.).

For a while I’ve been skeptical that the rate of hiring of engineers would change significantly because of LLMs, but I’m starting to feel like maybe I’m wrong and it’s already changing and companies are looking toward AI to lower costs and require fewer humans. In that case they are probably still going to want people who are technically exceptional - maybe even more so - but are able and willing to create, integrate, and babysit AI generated code, and also do PM and EM style feature management.

If companies are slowing hiring due to AI, I would expect interviews to get worse before they get better.

siva7 5 months ago | parent | prev | next [-]

> For example, it could be that the company wants the engineer to do some kind of assessment whether a feature should be implemented at all, and if yes, in what way. Then you could, in an interview, give a bit of context and then ask the candidate to think out loud about an example feature request.

So a Product Manager?

perlgeek 5 months ago | parent | next [-]

Maybe.

Maybe now, or maybe in a year or two, AI coding tools will be good enough that a single semi-technical person can be Product Manager for a small product, and implement all the feature through AI/LLM tools.

Probably not for something of the complexity of Google Maps, but for a simpler website with some interactive elements, that could work.

But then, this was just an example. There can be lots of reasons that companies still need engineers, my point was that they need to think about these reasons, and then use these reasons to decide how to select their engineers.

camdenreslink 5 months ago | parent | prev [-]

In most companies every engineer above a junior level is expected to pass features and bugfixes through their common sense filter and provide feedback. Product managers and designers aren't infallible and sometimes lack knowledge about the system or product that an engineer might have.

You can't just take requirements and churn out code without a critical eye at what you're doing.

johnrealtoast 5 months ago | parent | prev [-]

[dead]