I'm increasingly thinking that the only solution may be to have a search index with human editorial control. As much as I think Google deserves scorn for how badly they have made the search experience on the web, we have to remember that they're fighting a battle against SEO and spam. It feels like they have mostly given up. Any automatic solution will always be an arms race where they will be one step behind. The only way to get ahead is to ensure that all content has to pass through human moderation before it can be included -- that's the only way I can think of to break the cat and mouse game dynamics at play here.

Google, Facebook and others will never do this because it's expensive. Their cost savings is our toxic waste.

So what does a viable solution look like? I'm not sure. Maybe a system with distributed, volunteer moderators like Wikipedia. It's not perfect, but that model has held up remarkably well over its lifespan and has seemed to mostly hold its own against increasingly sophisticated attacks.

▲

cosmic_cheese 2 days ago | parent | next [-]

Figuring out how to scale it may be a challenge, but it may be worth considering something along the lines of the directory sites that used to be the main method of discovery on the web prior to the rise of search engines, which itself can be searched.

I would personally find this useful at least, particularly when diving into unfamiliar topics. In a lot of those situations, a handful of high quality links can preempt a litany of more granular searches.

This might be an area where a traditional, respectful search engine style crawler backed by a small cheap LLM paired with human curation would work well. The crawler+LLM finds sites, attempts to compute some quality score, figures out where they'd fit in the directory's hierarchy, and then submits them for human review and approval. If a poor quality link makes its way through, users can flag it for review and if curation agrees it gets booted.

▲

gtowey 2 days ago | parent [-]

I'm leaning against the user flagging model of content moderation because it keeps the incentive for bad actors to try to game the system. Bad content, even briefly included can still do a lot of damage. It's like spam email -- even though something like 99.9% gets filtered out, that 0.1% is still valuable enough for spam senders to keep trying. If the problem of spam is just a numbers game, then bad actors will just increase the volume to compensate.

	▲	cosmic_cheese 2 days ago \| parent [-]
		The problem is that it can also be gamed in the opposite direction. Anything automated gives a way for hostile third parties to disappear links at will, which can be similarly damaging especially if timed right. YouTube and many other sites with automated DMCA handling for example have major issues with this.

▲

underlipton 2 days ago | parent | prev | next [-]

>we have to remember that they're fighting a battle against SEO and spam

We have to remember that they decided to side with the SEOmongers rather than crush them, because it was better for Google's bottom line.

▲

IAmBroom 2 days ago | parent | prev | next [-]

> Google deserves scorn for how badly they have made the search experience on the web, we have to remember that they're fighting a battle against SEO

Where did you get this idea from? They actively promote it.

▲

OtomotO 2 days ago | parent | prev | next [-]

On one hand they fight SEO and spam, on the other they take advertising money to manipulate the order of search results...

▲

ileonichwiesz 2 days ago | parent | prev [-]

The scale of this would need to be unfathomable. Doesn’t YouTube get something like 100 years of video uploads every day? You’d need hundreds of thousands of people working full time to review that much footage manually.

	▲	gtowey 2 days ago \| parent [-]
		Isn't that kind of the point though? We don't need 100 years of video uploads every day. We need fewer, higher quality videos. If 90% of those were likely to get rejected, such as all the low-effort AI spam videos generated today, then there wouldn't be much incentive to submit them in the first place. The volume of uploads would drop. This is only a problem if you're an advertising platform who needs an endless stream of new crap to recommend to your users to try to keep them on the platform as long as possible. This is why I keep landing on moderation-before-inclusion as the standard. Review systems that flag bad content means that bad content still gets seen, and often still has enough of an effect to make it worthwhile for bad actors to churn out. IMO that's part of explanation of why there's 100 years of videos uploaded to YouTube every day. Even videos that get taken down get some exposure; increasing the volume of crap is a solution for their short life span. It seems like a self-reinforcing problem there, even if there were an honest attempt at moderation. The old content doesn't go away, but YouTube is incredibly bad at making it effectively searchable. They don't want you to find the highest quality videos on a topic, they want you to plug in to the endless new stream of slop which just recapitulates things which were done better before. This isn't really an "old man yelling at kids to get off his lawn" take that everything older is better. Instead I'm saying that the Wikipedia approach to help users find and collaborate on good content to make it even better or more comprehensive is preferable to the approach that the only way to be seen is to endlessly publish new content, which is meant to obscure what came before it on a topic. It's like if you walked into a library, but the only thing visible was a "new releases" shelf and searching the stacks was nearly impossible. That would make it basically a matter of luck if you found anything relevant, and makes deep research impossible.