I have what I think is a good analogy for this problem.

In the Olden Days [tm], jobs were advertised through recruiters, physical media (eg the paper) and connections. You had to review applications and conduct interviews. The cost of applying was relatively high, your reach was relatively low and the investment per applicant was relatively high. So imagine that there were enough jobs for everyone in a simplified model. 10 people applied for 10 jobs. It's not the same 10 people for each job. But there was a decent ROI on effort. It kinda just worked.

Fast forward to now and the cost of applying is essentially zero in terms of registering interest and submitting a CV. And you apply for a lot more jobs so instead of 10 applicants for 10 jobs, you have 200 applicants for 200 jobs. Still the same applicant to job ratio but way more inefficient for everyone involved. Applicants can't put in the same effort for 200 applications that they did for 10 and employers can't review 200 applications the same way they did 10.

So what happens? Employers, who have the power in this relationship, put up roadblocks in the name of efficiency. Now you have to survive ATS before ever going in front of a human. That ATS uses inscrutable logic that may filter you out for not including enough keywords or some other specious reason. You now have hiring assignments.

The net effect is that an applicant puts in 200 applications, get automatically filtered out from 180 of them and then has to do upwards of 20 take home assignments.

Plus there are more and more layers added. More rounds of interviews. Phone screens. Remote interviews then on-site interviews. All of this wastes time and, like you allude to, I don't think it's effective. But it's a natural response to the illusion of choice.

Let me give an online dating analogy. In years gone by, you'd rely on meeting people in person. Now, less so. And speaking in a strictly heteronormative sense, how it tends to go is that women on average have hundreds of choices and men have on average far fewer. A gender imbalance plays into this.

So what does a woman in this scenario do? They start adding filters to just make the numbers more manageable. Height, salary, location, same interests and so on. So the net effect is that that a lot of people are indepndently applying filters and filtering down to a pool with a lot of crossover. Conventionally attractive men, for example, will tend have far more options.

So I think the same happens with hiring. If you're a Big Tech company, you start adding filters. Did this person go to a top school? What internships did they have? Do these things matter for on-the-job performance? Barely (IMHO). But what you'll probably find is that a handful of people have a ton of options while others struggle. And it's simply the product of employers trying to make their applicant pool manageable but they're all doing it in very similar ways.

And I honestly don't know what the solution is.