I think it's largely the mindset of moving fast and breaking things that's at fault. If say ship it at "good enough", it will not behave well.

Building a competent well-behaved crawler is a big effort that requires relatively deep understanding of more or less all web tech, and figuring out a bunch of stuff that is not documented anywhere and not part of any specs.