This isn't an uncommon lament, despite what the author thinks.

The problem is that running "studies" on what the author and others are asking for is effectively impossible. You can't get 25 teams of suitably-random professionals to build a non-trivial program one way, and 25 to build it another, and then do statistically-significant analysis of them, because that would be a staggeringly expensive study... and that is just for one such study, which would then be, you know, wrong, incorrectly analyzed, controversial, biased, etc. The expense of getting a representative set of such studies, independently conducted, enough to actually settle a question, beggars the imagination.

Even such studies as have been performed are almost all invalidated by virtue of being run on inexperienced students. I'm not really that interested in whether inexperienced students do or do not do well with some methodology for the most part. What happens with professionals?

In the meantime, all we've got is experiences. Contrary to popular belief, science does not mandate that we therefore curl up into a ball and cry ourselves to sleep at night. We just have to do our best. Berating other people for not pouring billions of dollars into the simplest of studies won't help much. They're still not going to, and you still have to go to work, sit down, and figure out how you're going to accomplish some task, even it you don't have double-blinded meta-analyses from decades of studies to pull from.

This is so close to my sentiment that I had to double check to see if I wrote it.

And to explore running studies a bit further. The second time you build a system goes so much better because you already know all of the weird edge cases. And the third better still because your failures in the second time has cured you of some of your hubris.

Even if you somehow bankrolled 50 repeat projects and did the statistics etc correctly, you're still going to get some weird artifacts because some of those teams have people who did the thing before. You'll learn the wrong lesson when the real lesson is "make sure Bob is working on Bluetooth because he's done it 10 times before."

Starting with people with no experience is likewise not interesting because nobody really cares what lessons you learn by turning a bunch of muggles loose on something difficult.

What you need to bankroll is 50 teams worth of people who spend their entire careers testing out a hypothesis. (And even then you probably need to somehow control their professional communities in some way because again who cares what some small group of people approaches a problem when you could instead have people who go out and learn things from other people.)