| ▲ | bicepjai 13 hours ago | |
Even after providing firebase endpoint, crawlers come to the site ? | ||
| ▲ | Bender 12 hours ago | parent | next [-] | |
Most crawlers have no concept of what that is. They will follow links to this site and then follow links out of this site even after being told not to [1]. The majority of crawlers follow zero rules, RFC's, etc... The few platforms that do follow standards and rules are akin to a law abiding citizen in Mos Eisley. [1] - rel="nofollow" | ||
| ▲ | dang 10 hours ago | parent | prev | next [-] | |
Oh my god. It's the crawlpocalypse. | ||
| ▲ | busymom0 12 hours ago | parent | prev [-] | |
Unfortunately, the firebase API is very bad as they even acknowledge that in their github page. | ||