But it's stupid. If i smack myself in the head with a hammer is that proof hammers shouldn't be relied on?

If you smack yourself in the head with a hammer and it injures you that's evidence that smacking people in the head with hammers is bad and shouldn't be done, right?

Here we’re at the origin of the tool and get to watch how many people hit themselves in the head before we learn this collective wisdom.

There’s a gap between what the tool will allow you to tell it to do, and what it’s good at. The feedback mechanism to tell the difference is deficient compared to a hammer.

No, but it would be proof you didn't get the point of the paper.

Are there start-ups led by idiots suggesting that smacking yourself in the head with a hammer will help treat your diabetes?

If not, then perhaps there's a problem in your analogy.