How exactly other websites “gave it all away”? Also examples you list are websites putting some explicit rule eg in their robots.txt or filtering web crawlers. This is all a reaction to existing situation, so Reddit for sure has been scrapped before Reddit realised what was happening.