Does it suck if you offer a product and someone’s looking for it and the agent can’t look on Amazon but they can look on your small side maybe that’s good

▲

Geezus_42 an hour ago | parent | next [-]

Not it the site can't stay up because it's constantly being hammered by bots.

▲

edukite an hour ago | parent | prev [-]

Dude, I had a 1Gb of nginx logs daily then I blocked them and it dropped to 10mb weakly.

	▲	dawnerd 36 minutes ago \| parent \| next [-]
		Legit what one of our clients problem was. Normal web traffic wasn’t an issue and the logs were just a few megs. Yeah there was a problem where they were not rotating but once meta started crawling them 24/7 ignoring the robots.txt it kept filling their vps up. An annoying problem to fix once a server is completely full and you can’t even ssh in.
	▲	sva_ 40 minutes ago \| parent \| prev [-]
		30GB a month isn't too terrible? I guess it depends - for example my provider includes 10TB egress per month, so 30gb would be negligible if it's static content. I'm not sure what others do, but I often click on the links that the agents pull up as sources so the websites do get more discoverable. What really sucks is e.g. Gemini that pulls images without being able to go to the source (unless it changed recently.)