| ▲ | Bender 6 hours ago |
| still have the ability to disable the crawler via robots.txt flags Never rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting first RST from web daemon and second RST from the OS. |
|
| ▲ | an hour ago | parent | next [-] |
| [deleted] |
|
| ▲ | saghm 4 hours ago | parent | prev | next [-] |
| Yeah, calling this "disabling" it is inaccurate, it's just "asking nicely". |
| |
| ▲ | amelius 4 hours ago | parent [-] | | Yeah it reminds me of the typical copyright notice found in a book, even decades ago: > All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher. For AI, wasn't the book first "reproduced or transmitted" into a training system? How would we reformulate it today? | | |
| ▲ | mitxela 3 hours ago | parent [-] | | Legal notices are designed to be as scary as possible so they usually say you can't do things that you actually can do. | | |
|
|
|
| ▲ | xmcp123 3 hours ago | parent | prev | next [-] |
| I have been writing bots and scrapers for almost 20 years. Not once have I had someone I work for in this capacity even ask about robots.txt |
| |
| ▲ | xena an hour ago | parent | next [-] | | How much would I have to pay you for you to change careers permanently? | |
| ▲ | kazinator an hour ago | parent | prev | next [-] | | robots.txt can used as a reward mechanism. If an automated visitor that we would normally block begins by accessing robots.txt that's a good sign and we give that a plus score or whatever. If subsequent accesses from that visitor comply with robots.txt, we let them continue (perhaps with some rate checks). Conversely, if an obviously automated visitor does not fetch robots.txt, that can be an instant ban, kind of like a reverse honey pot, if you will. So it can have some nonzero value in the overall context; it's just silly to think of it as a blocking mechanism. In short, robots.txt documents what access is allowed; it states the policy. Enforcing it the job of something else. Someone you work for in this capacity shouldn't ask about robots.txt because that's the sort of detail for which they have you to deal with. | |
| ▲ | salawat 2 hours ago | parent | prev [-] | | You are part of the problem, sounds like. | | |
| ▲ | appplication 2 hours ago | parent [-] | | This implies the solution is every actor in the world willfully complying which doesn’t sound very realistic if you ask me. Similar to saying world hunger would be solved in everyone who could donated $100 to those in need. Sure it may be true but foot stomping about why isn’t it organically happening is missing the point. | | |
| ▲ | gherkinnn an hour ago | parent [-] | | There is a clear difference between the global issue of famine and actively choosing to work for an arsehole corp. | | |
| ▲ | appplication an hour ago | parent [-] | | The problem itself does not matter. If the nature of any solution relies on willful global compliance, it’s not going to be very effective at whatever it’s trying to solve. | | |
|
|
|
|
|
| ▲ | cowlevel 5 hours ago | parent | prev [-] |
| Wouldn't feeding them false data be even better? |
| |
| ▲ | usrnm 5 hours ago | parent | next [-] | | That could consume as much, if not more, resources as returning real data | | |
| ▲ | literalAardvark 4 hours ago | parent [-] | | Returning some text files from the server fs is _wayyy_ lighter than letting the bot go through the search, middleware, and db while crawling your stuff. | | |
| ▲ | yonatan8070 3 hours ago | parent [-] | | Or just sending it a tiny zip bomb, or even just some sh*t right out of /dev/random |
|
| |
| ▲ | post-it 3 hours ago | parent | prev | next [-] | | I'm not convinced that it's possible to poison AI. Real data all correlates, fake data is all unique in its own way. I suspect feeding AI fake data is a waste of time. | |
| ▲ | zeryx 4 hours ago | parent | prev | next [-] | | Angel banning is better, 100% agreed on giving it false data that the scraper builders can't easily detect. Give it seemingly correct and useful but totally wrong answers | | |
| ▲ | throwaway613746 4 hours ago | parent [-] | | Would you mind explaining what "angel banning" is? I've never heard this term before and even Google doesn't seem to know what it is. | | |
| ▲ | LoganDark 3 hours ago | parent [-] | | I would assume it's exactly what they described, banning something by pretending to still helpfully assist them while silently undermining them instead. |
|
| |
| ▲ | esseph 5 hours ago | parent | prev [-] | | We've already been using AI to target military strikes without human intervention. Make of that what you will. |
|
