Well-behaved agents will obey robots.txt and not fall into the trap.